HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
Authors: Enmao Diao
Jie Ding
Vahid Tarokh
Venue: ICLR 2021 Poster (2021)
Keywords: Federated Learning
Internet of Things
Heterogeneity

Abstract: Federated Learning (FL) is a method of training machine learning models on private data distributed over a large number of possibly heterogeneous clients such as mobile phones and IoT devices. In this work, we propose a new federated learning framework named HeteroFL to address heterogeneous clients equipped with very different computation and communication capabilities. Our solution can enable the training of heterogeneous local models with varying computation complexities and still produce a single global inference model. For the first time, our method challenges the underlying assumption of existing work that local models have to share the same architecture as the global model. We demonstrate several strategies to enhance FL training and conduct extensive empirical evaluations, including five computation complexity levels of three model architecture on three datasets. We show that adaptively distributing subnetworks according to clients' capabilities is both computation and communication efficient.

Published as a conference paper at ICLR 2021
HETEROFL: COMPUTATION AND COMMUNICATION EFFICIENT FEDERATED LEARNING FOR HETEROGE- NEOUS CLIENTS
Enmao Diao
Jie Ding
Department of Electrical and Computer Engineering
School of Statistics
Duke University Durhm, NC 27705, USA
University of Minnesota- Twin Cities Minneapolis, MN 55455, USA ding Smmmm.mmmeeeemm
enmao. www.@duke..eeeeoooo edu
Vahid Tarokh
Department of Electrical and Computer Engineering Duke University
Durhm, NC 27705, USA tarohhttrokkcuuur
ABSTRACT
Federated Learning (FL) is method of training machine learning models on pri- vate data distributed over large number of possibly heterogencous clients such as mobile phones and IoT devices. In this work, we propose new federated learning framework named HeteroFL. to address heterogeneous clients equipped with very different computation and communication capabilities. Our solution can enable the training of heterogeneous local models with varying computation com- plexities and still produce a single global inference model. For the first time, our method challenges the underlying assumption of existing work that local models have to share the same architecture as the global model. We demonstrate several strategies enhance FI. training and conduct extensive empirical evaluations, in- cluding five computation complexity levels of three model architecture on three datasets. We show that adaptively distributing subnetworks according to clients' capabilities is both computation and communication efficient.
INTRODUCTION
Mobile devices and the Internet of Things (loT) devices are becoming the primary computing source for billions users worldwide (Lim et al. 2020). These devices generate a significant amount of data that can be used to improve numerous existing applications (Hard al., 2018). From the privacy and economic point of view, due to these devices' growing computational capa- bilities, it becomes increasingly attractive to data and train models locally. Federated learning (FL) (Konečný et al., 2016 McMahan et al. 2017) is a distributed machine learning framework that enables number of clients to produce global inference model without sharing local data by aggre- gating locally trained model parameters. A widely accepted assumption is that local models have to share the same architecture as the global model (Li et al. 20206) to produce a single global inference model. With this underlying assumption, we have to limit the global model complexity for the most indigent client to train its data. In practice, the computation and communication capabilities each client may vary significantly and even dynamically. It is crucial to address heterogeneous clients equipped with very different computation and communication capabilities.
In this work, propose new federated learning framework called HeteroFL to train heteroge- neous local models with varying computation complexities and still produce a single global infer- ence model. This model heterogeneity differs significantly from the classical distributed machine learning framework where local data are trained with the same model architecture (Li et al., 2020b) Ben-Nun & Hoefler 2019). It is natural to adaptively distribute subnetworks according clients'

---

Published as a conference paper at ICLR 2021
capabilities. However, to stably aggregate heterogeneous local models to single global model un- der various heterogeneous settings is not apparent. Addressing these issues thus a key component of our work. Our main contributions of this work three-fold.
We identify the possibility of model heterogeneity and propose an easy-to-implement framework HeteroFL that can train heterogencous local models and aggregate them stably and effectively into a single global inference model. Our approach outperforms state-of- the-art results without introducing additional computation overhead.
Our proposed solution addresses various heterogencous settings where different propor- tions of clients have distinct capabilities. Our results demonstrate that even when the model heterogeneity changes dynamically, the learning result from our framework is still stable and effective.
We introduce several strategies for improving FL training and demonstrate that our method is robust against the balanced non-IID statistical heterogeneity. Also, the proposed method can reduce the number of communication rounds needed to obtain state-of-the-art results. Experimental studies have been performed to evaluate the proposed approach.
2 RELATED WORK
Federated Learning aims to train massively distributed models at a large scale (Bonawitz et al., 2019). FedAvg proposed by McMahan et al. (2017) is currently the most widely adopted FL base- line, which reduces communication cost by allowing clients to train multiple iterations locally. Ma- jor challenges involved in FL include communication efficiency, system heterogeneity, statistical heterogeneity, and privacy (Li et al., 2020b). To reduce communication costs in FL, some studies propose to use data compression techniques such as quantization and sketching (Konečný et al.) 2016 Alistarh al., 2017, Ivkin et al. 2019), and some propose to adopt split learning (Thapa) et al. 2020). To tackle system heterogeneity, techniques of asynchronous communication and active sampling of clients have been developed (Bonawitz al., 2019 Nishio & Yonetani] (011)) Statis- tical heterogeneity is the major battleground for current FL research. A research trend is to adapt
the global model accommodate personalized local models for non-IID data (Liang et al. 2020).
e.g., by integrating FL with other frameworks such as assisted learning (Xian et al. [2020). meta-
learning (Jiang et al. 2019: Khodak al.) 2019), multi-task learning (Smith et a 2017). transfer learning (Wang et al. 2019 Mansour et al. 2020), knowledge distillation (Li & Wang] (0199) and
lottery ticket hypothesis (Li et al.J 2020a). Nevertheless, these personalization methods often in- troduce additional computation and communication overhead that may not be necessary. Another major concern FL is data privacy (Lyu et al. 2020). as model gradient updates can reveal sensitive information (Melis et al. 2019) and even local training data (Zhu et al. 2019 Zhao et al. 2020).
To our best knowledge, what we present is the first work that allows local models to have different ar- chitectures from the global model. Heterogeneous local models can allow local clients to adaptively contribute to the training of global models. System heterogeneity and communication efficiency can be well addressed by approach, where local clients can optimize low computation complex- ity models and therefore communicate a small number of model parameters. To address statistical heterogeneity, we propose "Masking Trick" for balanced non-lID data partition in classification problems. We also propose a modification of Batch Normalization (BN) (Ioffe & Szegedy| 20155) as privacy concern of running estimates hinders the usage of advanced deep learning models.
HETEROGENEOUS FEDERATED LEARNING
3.1 HETEROGENBOUS MODELS
Federated Learning aims train global inference model from locally distributed data X (Xi,...,XX) across m clients. The local models are parameterized by model parameters (WI.... ). The server will receive local model parameters and aggregate them into global model W, , through model averaging. This process iterates multiple communication rounds and can be formulated as W = J >m W at iteration t. At the next iteration, W is transmitted to subset of local clients and update their local models as W!+1 W g-
2

---

Published as a conference paper at ICLR 2021
Global model parameters W
Local model parameters W?
Local model parameters W/2
Local model parameters W!
Figure 1: Global model parameters W are distributed to m 6 local clients with p computa- tion complexity levels.
In this work, we focus on the relaxation of the assumption that local models need to share the same architecture as the global model. Since our primary motivation is reduce the computation and communication complexity of local clients, we consider local models to have similar architecture but can shrink their complexity within the same model class. To simplify global aggregation and local update, it is tempting to propose local model parameters to be subset of global model pa- rameters W C WT However, this raises several new challenges like the optimal way to select subsets of global model parameters, compatibility of the-state-of-ar model architecture, and mini- mum modification from the existing FL framework. We develop Heterogeneous Federated Learning (HeteroFL) to address these issues in the context of deep learning models.
variety works show that we can modulate the size of deep neural networks by varying the width and depth of networks (Zagoruyko & Komodakis 2016 Tan & Le |e| 2019). Because we aim to reduce the computation complexity of local models, we choose vary the width of hidden channels. In this way, we can significantly reduce the number of local model parameters, while the local and global model architectures are also within the same model class, which stabilizes global model aggregation.
We demonstrate our method of selecting subsets of global model parameters W, for single hidden layer parameterized by Wg E R Fig. where dy and ki are the output and input channel size of this layer. is possible to have multiple computation complexity levels WP C C wi as illustrated in Fig. = k.. It follows that the size of local model parameters |W?" ,2(6-1)|W, and the model [W71 I
| Let r the hidden channel shrinkage ratio such that di = yP d, and
shrinkage ratio R
2( With this construction, we can adaptively allocate subsets of global model parameters according to the corresponding capabilities of local clients. Suppose that number of clients in each computation complexity level is (m...... m. Specifically, we perform global aggregation in the following way.
m
1
my
WP
WP,
WP
WP-
WP-1 WP,...
m
m
mp
m- m--ma.p
W7 \ W
Σ
W W?
(2)
m2p
W = W/ WT
W?)
For notational convenience, we have dropped the iteration index L. We denote the WP' as ma- trix/tensor. The W d km] denotes the upper left submatrix with size of dm x km- Also, Wp- WP-11 Wp.++1 denotes the set of elements included in WP-1,t+1 but excluded in WP.++1
We exemplify the above equations using Fig. I The first part of Equation (T) shows that the smallest part of model parameters (blue, p 3) is aggregated from all the local clients that contain it.
3

---

Published as a conference paper at ICLR 2021
In the second part of Equation ( the set difference between part p 1 (orange) and p (blue) of model parameters is aggregated from local clients with computation complexity level smaller than p 1. In Equation (2). the red part of model parameters can be similarly aggregated from m m2p = my clients. In Equation (3). the global model parameters W is constructed from the union of all disjoint sets of the partition. In summary, each parameter will be averaged from those clients whose allocated parameter matrix contains that parameter. Thus, model of an intermediate complexity will have parameters fully averaged with all the other larger models but partially with smaller models (according to the corresponding upper left submatrix).
Several works show that wide neural networks can drop tremendous number of parameters per layer and still produce acceptable results (Han et al. Franklee & Carbin 2018). The intuition is thus to perform global aggregation across all local models, at least on one subnetwork. To stabilize global model aggregation, we also allocate fixed subnetwork for every computation complexity level. Our proposed inclusive subsets of global model parameters also guarantee that smaller local models will aggregate with more local models.
Thus, small local models can benefit more from global aggregation by performing less global aggre- gation for part of larger local model parameters. We empirically found that this approach produces better results than uniformly sampled subnetworks for each client or computation complexity level.
3.2 STATIC BATCH NORMALIZATION
After global model parameters are distributed active local clients, we can optimize local model pa- rameters with private data. It is well-known that the latest deep learning models usually adopt Batch Normalization (BN) to facilitate and stabilize optimization. However, classical FedAvg and most recent works avoid BN. A major concern of BN is that it requires running estimates of representa- tions at every hidden layer. Uploading these statistics to the server will cause higher communication costs and privacy issues Andreux et al. (2020) proposes to track running statistics locally.
We highlight an adaptation of BN named as static Batch Normaliztion (sBN) for optimizing pri- vacy constrained heterogeneous models. During the training phase, sBN does not track running estimates and simply normalize batch data. We do not track the local running statistics as the size of local models may also vary dynamically. This method is suitable for HeteroFL as every communi- cation round is independent. After the training process finishes, the server sequentially query local clients and cumulatively update global BN statistics. There exist privacy concerns about calculating global statistics cumulatively and we hope to address those issues in the future work. We also em- pirically found this trick significantly outperforms other forms of normalization methods including the InstanceNorm (Ulynnov el al., (2016), GroupNorm (Wu & He 2018), and LayerNorm (Ba et al. 2016) as shown in Table 4 and Table 5
3.3 SCALER
There still exists another cornerstone of our HeteroFL. framework. Because we need to optimize local models for multiple epochs, local model parameters at different computation complexity lev- els will digress to various scales. This known phenomenon was initially discussed by the dropout (Srivastava et al. 2014). To directly use the full model during the inference phase, inverted dropout with dropout rate q scales representations with - during the training phase. In practice, dropout is usually attached after the activation layer the selection of subnetworks is performed with masking- Our method directly selects subnetworks from the subsets of global model parameters. Therefore, we append Scaler module right after the parametric layer and before the sBN and activation layers. The Scaler module scales representations by - during the training phase. After the global aggre- gation, the global model can be directly used for inference without scaling. To further illustrate this point, we include comprehensive ablation study in Tables used in our HeteroFI. framework can be formulated as
4 and
A typical linear hidden layer 5
y DBB/sBNSSaler(A... WP. 6
(4)
where y is the output, o(-) denotes a non-linear activation layer, e.g ReLUO,, and WP, b9, are the weight and bias for local model m computation complexity level p. With all the practical meth- ods mentioned above, we propose the complete pseudo-code for our HeteroFL framework in Algo- rithm I The local capabilities information I.m is an abstraction of the computation and communica-
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: HeteroFL: Heterogeneous Federated Learning
Input: Data X; distributed on M local clients, the fraction C of active clients per communication round, the number of local epochs E, the local minibatch size B, the learning rate 7, the global model parameterized by W the channel shrinkage ratio r, and the number of computation complexity levels P. System executes:
Initialize Wy and local capabilities information LI.K for each communication round 0, .,... do
Mr t max .M, S - random set of M, clients
for each client m E St in parallel do
Determine computation complexity level p based on Lm
I'm < (f--1), dm <- Fmmdg km +- Tinkg
Wt + W/f: dmss km]
W +- ClientUpdate Tmm W
end
for each computation complexity level do W Wp-11t11 \ Wp.+1 M. Mp.r.e Σ Mr M wp-let+ Whett
end
W Up W Update LIKK;") (Optional)
WP-1.++
\ Wgtt
end
Query representation statistics from local clients (Optional) ClientUpdate rm. W:):
rm,
Bm f Split local data Xm into batches of size B for each local epoch from I to E' do for batch im E Bm, do
Wm <- Wm m V)WE m. Tmi dm)
end end
Return W to server
tion capabilities of local client m. Once this information is communicated to the server, the server can know the model complexity that should be allocated to the client. We can also optionally update learning rates to facilitate optimization and local capabilities information if changing dynamically.
4 EXPERIMENTAL RESULTS
We trained over 600 individual models for exploring and demonstrating the effectiveness of method. We experimented with MNIST and CIFAR1O image classification tasks and the WikiTexx2
language modeling task (LeCun al. 1998 Krizhevsky al. 2009 Merity et al. 2016 Devlin et al. 2018).
Our experiments are performed with three different models including CNN for MNIST, a preacti- vated ResNet (PreResNet18) (He et al. (016) for CIFAR 10 and a Transformer (Vaswani et al. 2017) for WikiText2. We replace BN in CNN and PreResNet18 with our proposed sBN, and attach the Scaler module after each convolution layer. To study federated optimization. we adopt data partition the same as (McMahan et al. 2017 Liang al. 2020). We have 100 clients, and the fraction C of active clients per communication round is 0.1 throughout our experiments. For IID data partition, we uniformly assign the same number of data examples for each client. For balanced non-IID data partition, we assume that the label distribution is skewed, where clients will only have examples at most from two classes and the number of examples per class is balanced. We note that there exist other kinds of non-IID data partition, e.g., the unbalanced non-IID data partition where clients may hold unbalanced labeled dataset and the feature distribution skew where clients may hold different features. We conduct masked language modeling task with 15% masking rate and uniformly
5

---

Published as a conference paper at ICLR 2021
assign balanced data examples for each client. It needs to point out that each client will roughly have 3000 different words in their local dataset, while the total vocabulary size is 33278. The details regarding hyperparameters and model architecture can be found in of the Appendix.
To study the effectiveness of our proposed HeteroFL framework, we construct five different com- putation complexity levels {a, b, c, d,ee) e) with the hidden channel shrinkage ratio r 0.5. We have tried various shrinkage ratios, and we found that it is most illustrative to use the discrete complexity levels 0.5, 0.25, 0.125, and 0.0625 (relative the most complex model). For example, model 'a' has all the model parameters, while models "b' to 'e' have the effective shrinkage ratios 0.5, 0.25, O. 125, and 0.0625. We note that the complexity of 'e" is close to a logistic regression model. Our experi- ments indicated that the ratio can be arbitrary between (0, 1] and dynamically change. In practice, using a dictionary of discrete complexity levels are convenient for coordination purposes.
Each local client is assigned an initial computation complexity level. We annotate Fix for exper- iments with a fixed assignment of computation complexity levels. and Dynamic for local clients uniformly sampling computation complexity levels at each communication round. We perform dis- tinct experiments for Fix and Dynamic assignments. All the figures are based on the Fix scenario, where we considered models of different sizes, and each client is allocated with a fixed size. All the tables are based on the Dynamic scenario, where we randomly vary the allocation of clients' model complexity, and the ratio of the number of weak learners is fixed to 50%.
The x-axis of figures represents the average model parameters. When 10% clients use the model 'a' and 90% use the model ee", the average number of model parameters is 0.1 x (size of model 'a') + 0.9 x (size of model 'e'). We interpolate this partition from 10% to 100% with step size 10% to demonstrate the effect of proportionality of clients with various computation complexity levels.
To demonstrate the effect of dynamically varying computation and communication capabilities, we uniformly sample from various combinations of computation complexity levels. For example, model "a-b-c-d-e' means that we uniformly sample from all possible available levels for every active client at each communication round. We show the number of model parameters, FLOPs, and Space (MB) to indicate the computation and communication requirements of our methods. For example, since calculates computational metries by averaging those of
we uniformly sample levels, model model and e. The ratio is calculated between the number of parameters of a given model with respect to its 100% global model.
We compare our results to other baseline methods like Standalone, FedAvg, and I.-FFeAvg gath- ered from (Liang et al., 2020p. Standalone means there is no communication between clients and the server. In our experimental studies, we considered more complex models compared with the existing work. In particular, the baseline models in LG-FedAvg used MLP (on MNIST) and CNN (on CIFAR10). In terms of the number of parameters, (on CIFAR10) are comparable with those baselines. In terms of the FLOPs, our model 'd-e' (on MNIST and CIFAR10) can be compared with those baselines. The single-letter models 'a', 'b', 'c', 'd', 'e' are our implementations of the FedAvg equipped with the sBN and Masking CrossEn- tropy. The takeaway of Table 2 is that a weak learner that can only train small model 'e' (on (IFAR10)(770099) can boost its performance to 'c-e" (86.88%) "b-e" (89.10%) or 'a-e' (90.29%), which are close to the scenario where all the learners are strong, namely c(87.55%) b(89.8 (189.82%) or a(91.99%). In particular, in 'c-e', 'b-e", or 'a-e', half of the clients are trained with larger models 'c', "b', or 'a', while the other half are trained with the model 'e'. Only the aggregated global models "c", 'b', or 'a' are used during the testing stage. Although weak clients train smaller models 'e', they will test with the largest models 'c", 'b', or 'a" to gain better performance.
models 'a-e' (on MNIST) and 'b-e'
Full results including other possible combinations can be found in appendix in Table method is robust to dynamically varying model complexities. it worth noting that our method does not incur any additional computation overhead and can be readily adapted to existing applications.
7 Finally, our
We also perform experiments for balanced non-IID data partition and provide a simple trick to achieve comparable results. As mentioned earlier, most state-of-the-art results ofbalanced non-IID datasets suggest the personalization of local models to achieve better local results (Smith et al. 2017 Liang et al. 2020 Li et al.] 2020a). Here, the Local results assume that the training data distribution and test data distribution for each local client are the same, and assign zero probability for those classes that presented a client during training. The Global results were calculated from the global model applied to the test data directly. The Local results were cumulatively averaged from

---

Published as a conference paper at ICLR 2021
90
90
88
88
Accuracy
Accuracy
86
86
84
84
a-b
82
b-c
82
80
a-d
b-d
80
b-e
0.2
0.4
0.6
1.0
0.5
1.0
1.5
2.0
2.5
Number of Model Parameters
Number of Model Parameters
(a)
(b)
88
84
86
83
82
Accuracy
84
Accuracy
81
82
80
80
79
c-d
78
d-e
77
0.8
Number of Model Parameters
le5
Number Model Parameters (d)
145
Figure 2: Interpolation experimental results for CIFARIO (IID) dataset between global model com- plexity ((a) (b) b. (c) c, (d) d) and various smaller model complexities.
the performance of each data example on each local client. Zhao et al. (2018) showed that the failure of non-IID FL related the weight divergence among local model parameters trained locally for many iterations. The weight divergence mostly occurs in the last classification layer of networks.
Thus, instead of full Cross- Prooss-Etrrpyy Loss for all classes, we are motivated to train each local model only with their corresponding classes. In this way, each local model will train sub-task given lo- cally available label information. Specifically, we mask out the output of the model before passing it Cross-Entropy Loss, which named as Masked Cross-Entropy Loss. We experimented with several different ways masking, we find replacing the last layer outputs that are not associated with local labels with zero achieves both stable and comparable local and global results. When aggregating local model parameters, we do not aggregate the untrained parameters in the last classi- fication layers. Either the server infer this information implicitly, the local clients report which classes they have to the server explicitly. We provide a comprehensive ablation study in Ta- bles The results show that Masked Cross-Entropy Loss significantly improve local performance and moderately global performance of balanced non-lII data partition task. Since our primary focus address model heterogeneity, we leave the analysis of this trick future work. We show the results of Fix experiments in appendix in Fig. Table The results show that our method performs comparably those with personalized local models. Our method is readily adaptable, free of computation overhead, and only rely the single global model for testing local and global results. It allows local clients switch to another subtask simply by changing its mask without querying the server for others' personalized models.
Dynamic non-IID results are also included in
We show the learning curves 50% Fix and Dynamic assignments in appendix in Fig. I The learning curves show that the optimization of HeteroFL for the IID dataset is stable and efficient. Our method achieves better results with a fewer number communication rounds, e.g., 800 for Heterofl and 1800 for LG-FedAvg (Liang et al. 2020p. We empirically discover gradient clipping stabilizes the optimization of HeteroFL as it prevents small models from gradient explosion. We therefore adopt universal learning rate for heterogeneous local models. It is also perceivable

---

Published as a conference paper at ICLR 2021
Accuracy
Model
Ratio
Parameters
FLOPs
Space (MB)
Non-IID
Local
Global
1.00
1.6 M
80.5 M
5.94
99.53
99.85
98.92
a-e
0.50
40.5 M
2.98
99.46
98.96
a-b-c-d-c
0.27
416
21.6 M
1.59
99.46
99.85
98.29
b
1.00
391 K
20.5 M
1.49
99.53
99.87
99.10
b-e
0.51
199
10.4 M
0.76
99,51
99.67
98.51
b-c-d-e
0.33
131 K
6.9 M
0.50
99.52
99.88
98.99
1.00
99 K
5.3 M
0.38
99.35
99.56
96.34
c-e
0.53
K
0,20
99.39
99.79
97.27
c-d-c
0.44
0.17
99.31
99.76
97.85
d
1.00
25 K
1.4
0.10
99.17
99.86
97.86
d-e
0.63
16
0.06
99.19
99.63
97.70
1.00
7 K
400 K
0.03
98.66
99.07
92.84
Standalone (Liang et al. 2020
1.00
633 K
1.3 M
2,42
86.24
98.72
30.41
FedAvg [iang et al. 2020
1.00
633 K
1.3 M
2,42
97.93
98.20
98.20
IG-FedAvg (1 Liang
2020)
1.00
633 K
1.3 M
2.42
97.93
98.54
98.17
Table Results of combination of various computation complexity levels for MNIST dataset. Full results can be found in Table
Accuracy
Model
Ratio
Parameters
FLOPs
Space (MB)
Non-IID
IID
Local
Global
1.00
9.6
330.2 M
36.71
91.19
92.38
56.88
a-e
0.50
165.9
18.43
90.29
92.10
59.11
a-b-c-d-e
0.27
2.6
88.4 M
9.78
88.83
92.49
61.64
b
1.00
2.4 M
83.3 M
9.19
89.82
93.83
55.45
b-e
0.51
1.2
42.4 M
4.67
89.10
90.68
59.81
b-c-d-e
0.33
801 K
27.9 M
3.05
87.92
91.90
59.10
1.00
604 K
21.2 M
2.30
87.55
91.09
55.12
c-e
0.53
321 K
11.3 M
86.88
91.83
63.47
c-d-e
0.44
265 K
9.4
1.11
85.79
91.49
55.42
1.00
152 K
5.5 M
0.58
84.21
90.77
61.13
d-e
0.63
9S
3.5 M
0.36
82,93
90.89
56.16
1.00
38 K
1.5 M
0.15
77.09
89.62
54.16
Standalone Liang et al. 2020)
1.00
1.8
3.6 M
6.88
16.90
87.93
10.03
FedAvg [Lang et al.
2020
1.00
1.8 M
3.6 M
6.88
67.74
58.99
58,99
LG-FedAvg (Linng et al.
2020
1.00
1.8 M
3.6 M
6.88
69.76
91.77
60.79
Table 2: Results of combination of various computation complexity levels for CIFARIO dataset.
Full results can be found in Table g
that aggregation of model parameters trained with non-lID data makes the optimization less stable. Results of Dynamic show that global aggregation of dynamically varying computation complexities is stable.
5 CONCLUSIONS AND FUTURE WORK
We propose Heterogeneous Federated Learning (HeteroFL), which shows the possibility of coordi- natively training local models much smaller than a global model to produce a single global infer- model. Our experiments show that FL can be made practical by introducing HeteroFI.
8

---

Published as a conference paper at ICLR 2021
Model
Ratio
Parameters
FLOPs
Space (MB)
Perplexity
1.00
19.3 M
1.4 B
73.49
3.37
a-e
0.53
10.2 M
718.6 M
38.86
3.75
a-b-c-d-e
0.37
7.2 M
496.6 M
27.55
3.55
b 1.00
9.1 M
614.8 M
34.74
3,46
b-c 0.56
5.1 M
342.0 M
19.49
3,90
b-c-d-e
0.46
4.2 M
278.7 M
16.07
3.64
1.00
4,4 M
290.1 M
16.92
3.62
c-c 0.62
2.8 M
179.7 M
10.57
3.89
c-d-e
0.58
2.6 M
166.7 M
9.85
3.66
d 1.00
2.2 M
140.7 M
8.39
3.83
d-e
0.75
1.7 M
105.0 M
6.31
3.90
1.00
1.1 M
69.3 M
4.23
7.41
Table 3: Results of combination of various computation complexity levels for WikiText2 dataset. Full results can found in Table
Accuracy IID
Model
Normalization
Scaler
MNIST
CIFARIO
None
99.2
81.3
IN
99.5
87.7
GN
N/A
99.5
81.0
LN
99.5
77.3
sBN
99.6
91.7
None
98.6
58.1
IN
97.4
66.4
GN
N/A
98.7
62.6
LN
98.6
53.7
sBN
98.7
77.0
None
99.5
80.1
X
sBN
99.0
90.1
None
99.2
80.4
IN
99.5
86.6
GN
99.5
76.0
LN
99.4
71.7
sBN
99.5
90.1
Table 4: Ablation Study of IID scenarios. The single-letter models 'a' and 'e" are FedAvg equiped with various normalization methods. The sBN significantly outperforms other existing normaliza- tion methods, including the InstanceNorm (IN), GroupNorm (GN) (the number of group G=4), and LayerNorm (LN). Scaler is used for HeteroFL to train models of different sizes and moderately improve the results.
and sBN and Mased Cross-Entropy Loss, as HeteroFL fully exploits local clients' capabilities and achieves better results with a fewer number of communication rounds. We demonstrate our results with various model architectures, including CNN, PreResNet18. 8, and Transformer, and show that our method is robust to statistical heterogeneity and dynamically varying local capabilities. A future direction is to distinct model classes well model heterogeneity. Also, the proposed methods may be emulated to address heterogeneous few-shot learning, multi-modal learning, and multi-task learning.
ACKNOWLEDGMENTS
This work was supported by the Office of Naval Research (ONR) under grant number N00014-18 1-2244, and the Army Research Office (ARO) under grant number 1911N2-20-110022..

---

Published as a conference paper at ICLR 2021
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-clicien sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems, pp. 1709-1720, 2017.
Mathicu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric W Tramel. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 129-139- Springer, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 160.006000 2016.
Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR), 52(4):1-43, 2019.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp. Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, al. Towards federated learning at scale: System design. arXiv preprint arXiv: 190.:v/902.0004: 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1eXxv:880330333 2018
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 111.:::8111.3660 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03100 2015.
Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Ion Stoica, Raman Arora, et al. Communication- efficient distributed sgd with sketching. In Advances in Neural Information Processing Systems, pp. 13144-13154, 2019.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning, arXiv preprint arXiv: 2019.
Mikhail Khodak, Maria-Florina Balcan, and Ameet S Talwalkar. Adaptive gradient-basse based meta- learning methods. In Advances in Neural Information Processing Systems, pp. 5917-5928, 2019.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. tradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) (1221)))22822222 1998.
Ang Li. Jingwei Sun, Binghui Wang. Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: Personalized and communication-effiiee federated learning with lottery ticket hypothesis on non-iid datasets. arXiv preprint arXiv:2008.0333 2020a.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv: 1910.03508 2019.
10

---

Published as a conference paper at ICLR 2021
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50 60, 2020b.
Paul Pu Liang Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 2020.
Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang. Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 2020.
Lingjuan Lyu, Han Yu, and Qiang Yang. Threats federated learning: A survey. arXiv preprint arXiv:2003.02133, 2020.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications federated learning. arXiv preprint arXiv:2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficin learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 JEEE Symposium on Security and Privacy (SP), pp. 691-706. IEEE, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: (60960084933 2016.
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC--22-201 IEEE International Conference on Communications (ICC), pp. 1-7. IEEE, 2019.
Virginia Smith, Chao-Kat Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in Neural Information Processing Systems, pp. 4424 4434, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1): 100)11999999999 1958, 2014.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 1rXiv:1905.1 2019.
Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit Camtepe. Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004.1208 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 1607.0002 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
Kangkang Wang, Rajiv Mathews, Chlos Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252 2019.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Xun Xian, Xinran Wang, Jie Ding, and Reza Ghanadan. Assisted learning: framework for multi- organization learning. NeurIPS 2020 (spotlight). 2020.
Sergey Zagoruyko and Nikos Komodakis. 160.0:1605 2016.
Wide residual networks.
arXiv preprint

---

Published as a conference paper at ICLR 2021
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint (rXiv:2001.026100 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv: 800000088, 2018.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, pp. 14774-14784, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
The appendix contains supplementary experimental results. Table show the ablation study of Non-IID experiments. Compared results of IID experiments shown Table Table shows the ablation study Maksed CrossEntropy which is shown beneficial for balanced Non-IID data partition. In Table 6 we show the hyperparameters adopted in our experiments. In Figure and we show the Fix complexity assignments of MNIST and WikiText with iid data partition experiments. From Figure 6 to we show the Fix complexity assignments for all balanced Non-IID data partition experiments. Figure(itv 11 we show the learning curve of experiments with Dyanmic complexity assignments. The complete results for all experiments with Dyanmic complexity assignments can be found in Table 8 and
Accuracy Non-III)
Model
Normalization
Scaler
Masked CrossEntropy
MNIST
CIFAR10
Local
Global
Local
Global
None
97.4
97.4
42.6
42.8
N/A
x
sBN
99.4
99.4
53.4
53.7
None
99.7
95.6
91.7
58.5
99.8
98.7
88.4
43.7
GN
N/A
/
99.7
98.3
91.2
58.2
LN
99.8
98.3
89.9
54.2
sBN
99.9
98.6
92.1
59.2
None
96.2
96.0
38.9
38.2
N/A
x
sBN
90.1
90.1
40.7
40.4
None
99.5
96.5
86.6
48.9
IN
98.5
89.8
83.7
37.0
GN
N/A
99.2
92.2
83.5
36.7
99.3
94.0
82.6
40.0
sBN
99.3
94.2
90.1
52.9
None
96.8
96.9
37.8
37.4
X
X
sBN
99.2
99.2
41.0
41.3
None
99.4
95.7
89.1
52.8
sBN
99.8
98.0
90.7
57.7
a-e
None
X
97.3
97.3
34.6
34.4
sBN
99.3
99.3
46.0
46.7
None
99.5
95.8
90.3
55.6
IN
99.8
98.7
87.0
34.4
GN
99.5
96.2
88.7
49.7
LN
99.5
96.2
78.5
25.0
sBN
99.8
98.2
92.8
60.4
Table 5: Ablation Study of Non-IID scenarios. The Masked CrossEntropy is used for Non-IIID experiments. significantly improves the local performance and moderately improves global per- formance. Single letter model 'a' and 'e" are FedAvg equipped with various normalization methods. The sBN significantly outperforms other existing normalization methods, including InstanceNorm (IN), GroupNorm (GN) (the number of group G=4), and LayerNorm (LN). Scaler is used for Het- eroFL to train models of different sizes and moderately improve the results.
13

---

Published as a conference paper at ICLR 2021
Data Model Hidden size
MNIST CNN
CIFARIO PreResNet18 [64, 128, 256, 512]
WikiText2 Transformer
[64. 128, 256, 512] 5
[512. 512, 512, 512] 1 100
Local Epoch E
Local Batch Size
10
10 SGD
Optimizer
Momentum
0.9
Weight decay
5.00E-04
Learning rate "
0.01
0.1
0.1
IID
200
400
100
Communication rounds
non-III
400
200
I1D
[100)
[150, 250]
[25, 50]
Decay schedule (0.1)
non-III)
[200]
[300, 500]
[50, 100)
Emhedding Size
256
Number of heads
8
N/A
Dropout
0.2
Sequence length
64
Table 6: Hyperparameters and model architecture used in our experiments.
99.6
99.5
99.4
Accuracy
Accuracy
99.3
99.3
99.2
a-b
99.2
a-c
b-c
99.1
a-d
b-d
99.1
a-e
b-e
99.0
0.2
0.6
1.6
0.5
2.0
2.5
3.5
Number of Model Parameters
1c6
Number Model Parameters (b)
le5
(a)
99.2
99.4
99.3
99.0
99.2
Accuracy
98.9
99.1
98.8
99.0
98.7
98.9
c-d
98.6
98.8
C-e
d-e
98.5
0.2
0.4
0.6
0.8
1.0
1.00 1.25
1.50 1.75
2.00
2.25
2.50 lel
Number of Model Parameters
165
Number of Model Parameters (d)
Figure 3: Interpolation experimental results for MNIST (IID) dataset between global model com-
plexity ((a) (b) b. (c) c, (d) d) and various smaller model complexities.
14

---

Published as a conference paper at ICLR 2021
6.0
a-b
6.0
b-c
b-d
5.5
a-d
5.5
b-e
Perplexity
5.0
a-e
Perplexity
5.0
4.5
4.5
4.0
4.0
3.5
3.5
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
3
5
7
Number Model
let
Number Model Parameters
leb
(b)
c-d
d-e
C-e
6.5
Perplexity
Perplexity
5.0
5.5
4.5
4.5
4.0
3.5
1.5
2.0
2.5
3.0
3.5
4.0
4.5
1.2
1.4
1.6
1.8
2.0
2.2
Number of Model Parameters
Number of Model Parameters (d)
366
(c)
Figure 4: Interpolation experimental results for WikiText2 (IID) dataset between global model com-
plexity ((a) a. (b) b, (c) c. (d) d) and various smaller model complexities.
15

---

Published as a conference paper at ICLR 2021
100.00
99.75
98
99.50
Accuracy
96
Accuracy
99.25
99.00
94
98.75
a-c
b-c
92
98.50
b-d
a-e
98.25
b-e
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Number Model Parameters
146
Number of Model Parameters
(b)
99.8
99.5
99.0
99.6
Accuracy
Accuracy
98.5
98.0
99.4
97.5
99.2
97.0
c-d
96.5
99.0
96.0
0.2
0.4
0.6
0.8
1.0
1.00
2.50 Number Model Parameters (d)
1.25
1.50 1.75
2.00
2.25
Number Model Parameters
125
164
Figure 5: Interpolation experimental results for MNIST (non-lID, Local) dataset between global model complexity ((a) a, (b) b. (c) c, (d) d) and various smaller model complexities.
16

---

Published as a conference paper at ICLR 2021
93
92
91
92
90
91
Accuracy
89
88
87
a-b
86
b.c
b-d
85
a-e
b-e
84
0.2
0.4
0.6
0.8
1.0
1.5
Number Model Parameters
101
Number of Model Parameters
(a)
(b)
92
91.5
91
Accuracy
Accuracy
89
90.5
88
90.0
C-d
89.5
86
c-e
d-e
89.0
1
2
6
0.6
0.8
1.0
1.2
1.4
Number of Model Parameters
Number of Model Parameters (d)
Figure 6: Interpolation experimental results for CIFARIO (non-IID, Local) dataset between global model complexity ((a) a, (b) b. (c) c, (d) d) and various smaller model complexities.
17

---

Published as a conference paper at ICLR 2021
100.0
97.5
95
95.0
Accuracy
90
Accuracy
90.0
a-b
87.5
a-c
b-c
80
85.0
b-d
a.e
b-e
82.5
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Number of Model Parameters
le6
Number of Model Parameters
97
96
90
Accuracy
Accuracy
85
94
93
80
92
75
70
90
C-e
d-e
65
0.2
0.8 Number of Model Parameters
0.4
0.6
1.0
1.00 1.25
1.50 1.75
2.00
2.25
2.50
Number of Model Parameters
Figure 7: Interpolation experimental results for MNIST (non-IID, Global) dataset between global model complexity ((a) a, (b) b. (c) c, (d) d) and various smaller model complexities.
18

---

Published as a conference paper at ICLR 2021
60
56
Accuracy
55
50
50
48
a-c
45
b-c
a-d
b-d
b-e
40
0.2
0.6
1.5
Number Model Parameters
107
Number of Model Parameters
(a)
(b)
62
54
60
52
Accuracy
50
Accuracy
48
46
56
44
54
c-e
d-e
40
3
0.6
0.8
1.0
1.2
1.4
Number of Model Parameters
Number of Model Parameters (d)
Figure 8: Interpolation experimental results for CIFARIO (non-lID, Global) dataset between global model complexity ((a) a, (b) b. (c) c, (d) d) and various smaller model complexities.
19

---

Published as a conference paper at ICLR 2021
100
100
99
Test Accuracy
Test Accuracy
98
97
97
a-b
a-b
8-C
a-C
96
3-d
a-d
95
50
100 125
175
25
100 150 175 Communication rounds
Communication rounds
(a) Fix, IID
(b) Dynamic, IID
100
100
99
99
Test Accuracy
Test Accuracy
98
98
97
97
a-b
a-b
a-c
96
a-d
96
a-e
95
95
100
350
200
300 350 400
Communication rounds
Communication rounds
(c) Fix, non-lID, Local
(d) Dynamic, non-lID, Local
100
100
90
90
Accuracy
MMM
Test Accuracy
80
80
a-b
a-b
60
a-d
60
a-d
a-e
0 50 100 150 200 250 300 350 400 Communication rounds (e) Fix, non-IID, Global
0
50
100 150 200 250 300 350 400 Communication rounds
(f) Dynamic, non-IID, Global
Figure 9: Learning curves of MNIST datasets with 50% Fix and Dynamic computation complexity assignments.
20

---

Published as a conference paper at ICLR 2021
100
100
90
90
Test Accuracy
80
80
Test
70
70
a-b
3-C
3-C
60
3-d
60
a-d
50
50
50
100
150 200 300 350 400 Communication rounds
50
100
150 200 250 300 350 400 Communication rounds
(a) Fix, IID
(b) Dynamic, IID
100
100
90
90
Test Accuracy
Test Accuracy
80
80
70
70
3-b
a-b
a-c
60
60
a-d
a-e
a
50
50
100
200 300 400 500 600 700 800 Communication rounds
100
700
Communication rounds
(c) lx, non-lID, Local
(d) Dynamic, non-lID, Local
70
70
60
60
Test Accuracy
Test Accuracy
40
40
a-b
a-b
20
10
a-d
10
a-d
a-e
100 200 300 400 500 600 700 800 Communication rounds (e) Fix, non-IID, Global
100 200 300 400 500 600 700 800 Communication rounds
(f) Dynamic, non-IID, Global
Figure 10: Learning curves of CIFARIO datasets with 50% Fix and Dynamic computation complex- ity assignments.
21

---

Published as a conference paper at ICLR 2021
20.0
20.0
a-b
a-b
17.5
17.5
8-c
a-c
15.0
a-d
15.0
a-d
Test Perplexity
Test Test Perplexity
12.5
a-e
12.5
a-e
10.0
10.0
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
25
50
75
100
125
150
175
200
0
25
50
75
100
125
150
175
200
Communication rounds
Communication rounds
(a) Fix
(b) Dynamic
Figure 11: Learning curves of WikiText2 datasets with 50% Fix Dynamic computation com- plexity assignments.
Accuracy
Model
Ratio
Parameters
FLOPs
Space (MB)
Non-IID
IID
Local
Global
1.00
1.6
M
5.94
99.53
99.85
98.92
a-b
0.63
974.1 K
50.5 M
3.72
99.54
99.96
99.10
a-c
0.53
827.9 K
42.9 M
3.16
99.52
99.89
99.12
a-d
0.51
791.1 K
41.0 M
3.02
99.54
99.81
98.37
0.50
781.7 K
40.5 M
2.98
99.46
99.89
98.96
0.44
682.4 K
35.4 M
2,60
99.53
99.90
98.72
a-b-d
0.42
657.8 K
34.1 M
2.51
99.52
99.78
98.02
a-b-e
0,42
651.6
33.8 M
2.49
99.54
99.95
98.92
a-ed
0.36
560.4
29.1 M
2.14
99.57
99.95
99.34
0.36
554.1K
28.7 M
2.11
99.52
99,94
98.43
ade
0.34
K29.6
M
2.02
99.57
99.80
98.92
abb-
0,33
518.1K
26.9 M
1.98
99.54
99.80
99.03
a-b-c-
0.33
513.4 K
26.7 M
1,96
99.46
99.69
97.53
a-b-de
0.32
495.0 K
25.7 M
1.89
99.49
99.85
98.66
a-c-d-e
0.27
421.9 K
21.9 M
1.61
99.54
99.84
98.80
e--e---e
0.27
415.8
21.6 M
1.59
99.46
99.85
98.29
1.00
391.4K
20,5 M
1.49
99.53
99.87
99.10
b-c
0.63
245.1
12.9 M
0,94
99.49
99.87
99.05
b-d
0.53
208.3
11.0 M
0.79
99.44
99.85
98.95
be
0.51
199.0 K
10.4 M
0.76
99.51
99.67
98.51
bec-d
0.44
171.9K
9.1 M
0.66
99.54
99.84
98.98
bee
165.6
0.63
99.51
99.85
98.20
bdde
0,36
141.1K
74M M
0.54
99.48
99.89
98.72
b-c-d-e
0.33
130.5 K
6.9 M
0.50
99.52
99.88
98.99
1.00
98.9 K
5.3 M
0.38
99.35
99.56
96.34
c-d
0.63
62.1 K
3.4M
0.24
99.38
99.92
99.05
e-e
52.8
99.39
99.79
97.27
c-d-c
0.44
43.6
2.4
0.17
99.31
99.76
97.85
1.00
25.3
1,4 M
0.10
99.17
99.86
97.86
d-e
0.63
15.9
909.5 K
0.06
99.19
99.63
97.70
1.00
K.6
400.5 K
0.03
98.66
99.07
92.84
Standalone (Liang et al. (2020)
1.00
633.2 K
1.3 M
2.42
86.24
98.72
30.41
FedAvg (Liang et al. 2020
1.00
633.2 K
1.3 M
2.42
97.93
98.20
98.20
LG-FedAvg Liang
2020
1.00
633.2 K
1.3 M
2.42
97.93
98.54
98.17
Table 7: Results of combination of various computation complexity levels for MNIST dataset.
22

---

Published as a conference paper at ICLR 2021
Accuracy
Model
Ratio
Parameters
FLOPs
Space (MB)
Non-IID
IID
Local
Global
1.00
9.6
330.2
36.71
91.19
92.38
56,88
a-b
0.63
206.8
22.95
90.60
91.35
59.93
a-c
0.53
5.1 M
175.7 M
19.50
90.59
92.83
60.25
a-d
0.51
4.9 M
167.9 M
18.64
90.28
91.78
56.54
a-c
0.50
4.8 M
165.9 M
18.43
90.29
92.10
59.11
a-b-c
0.44
4.2
144.9 M
16.07
89.70
90.41
54.16
a-b-d
0.42
139.7
15.49
89.98
90.29
$1.79
a-b-e
0.42
4.0 M
138.4 M
15.35
89.79
90.79
62.17
a-c-d
0.36
3.5 M
119.0 M
13.20
89.47
89.82
53.13
a-c-e
0.36
3.4 M
117.6 M
13.05
89.35
93.59
57.30
ade
0.34
3.3
112.4 M
12.48
88.75
91.11
56.74
abb-c-
0.33
110.1
12.19
89.33
91.32
54.50
a-b-c-e
0.33
3,2 M
109.1 M
12.09
89.37
92.52
61.56
a-b-d-e
0.32
3.1 M
105.1 M
11.65
89.40
91.80
56.78
a-c-d-e
0.27
2.6 M
89.6 M
9.93
88.42
91.50
62.15
a-b-c-d-e
0.27
2.6 M
88.4 M
9.78
88.83
92.49
61.64
1.00
2.4
83.3 M
9.19
89.82
93.83
55.45
b-c
0.63
1,5
52.3 M
5.75
89.00
89.96
52.29
b-d
0.53
1.3 M
44.4 M
4.88
89.18
91.78
51.07
be
0.51
1.2
42.4 M
4.67
89.10
90.68
59.81
bed
0.44
1.1 M
36.7 M
4.02
88.35
92.79
58.09
bee
0.42
1.0 M
35.3 M
3.88
87.98
91.98
58.28
bede
0.36
866.3
M
88.06
91.94
54.02
bec-le
0.33
800.7 K
27.9 M
3.05
87.92
91.90
59.10
1.00
603.8 K
21.2 M
2.30
87.55
91.09
55.12
c-d
0.63
377.8 K
13.4 M
1.44
86.75
91.58
54.61
0.53
321.1 K
11.3 M
1.22
86.88
91.83
63.47
c-d-e
0.44
264.6 K
85.79
91.49
55.42
1.00
151.8 K
5.5
0.58
84.21
90.77
61.13
d-e
0.63
95.1 K
3.5 M
0.36
82,93
90.89
56,16
1.00
38.4 K
1.5 M
0.15
77.09
89.62
54.16
Standalone (Liang et al.
2020)
1.00
1.8
3.6 M
6.88
16,90
87.93
10.03
FedAvg [Liang cf al.
2020
1.00
1.8
3.6 M
6.88
67.74
58.99
58.99
LG-FedAvg (Liang et al
2020
1.00
1.8 M
3.6
6.88
69.76
91.77
60.79
Table 8: Results of combination of various computation complexity levels for CIFARI0 dataset.
23

---

Published as a conference paper at ICLR 2021
Model
Ratio
Parameters
FL.OPs
Space (MB)
Perplexity
a
1.00
19.3 M
1.4 B
73.49
3.37
a-b
0,74
14.2 M
991.4 M
54.12
3.31
0.62
11.8 M
829.0 M
45.20
3.71
a-b-c
0.57
10.9 M
757.6 M
41.72
3,42
0.56
M
754.3
40.94
3.74
a-b-d
0.53
10.2 M
707.8
38.87
3,53
a-e
0.53
10.2 M
718.6 M
38.86
3.75
a-h-e 0.51
9.8 M
684.0 M
37.49
3,47
b 1.00
9.1 M
614.8 M
34.74
3,46
a-b-c-d 0.45
603.4
33.39
3.61
a-c-d
0.45
599.6
32,93
4.08
a-b-c-e
0.44
8.5 M
585.5
32.34
3.50
a-c-e
0.43
8.3 M
575.8 M
31.54
3.65
a-b-d-e
0.41
7,9 M
548.2 M
30.21
3.64
a-
0.39
$26.0
28.70
4.02
a-b-c-d-e
0.37
496.6
27.55
3.55
b-c
0.74
6,8 M
452.4
25.83
3,45
a-c-d-e
0.35
6,8 M
467.0 M
25.76
3.92
b-d
0.62
5.7 M
377.7 M
21.57
3.70
b-c-d
0.58
5.2
348.5 M
20.02
3.47
0.56
342.0
3,90
b-c-c
0.54
324.7
18.63
3,46
1.00
4,4 M
290.1 M
16,92
3.62
b-c-d-e 0.46
4.2 M
278.7 M
16.07
3.64
b-d-e
0.45
4.1 M
274.9 M
15.79
3,92
0.75
215.4
12.66
3,46
c-e 0.62
179.7
3.89
c-d-e
0.58
2.6 M
166.7 M
9.85
3.66
d 1.00
2.2 M
140.7 M
8.39
3,83
d-e 0.75
1.7 M
105.0 O M
6.31
3,90
1.00
69.3 M
4.23
7.41
Table 9: Results of combination various computation complexity levels for WikiText2 dataset.
24


Reviews:
[{"review_id": "lb7-g2Q3-jN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE:\nThe authors have consistently improved their argumentation over the course of the review and addressed my concerns sufficiently. I have increased my score and recommend acceptance.\n\n\nORIGINAL REVIEW:\nThe authors propose three elements:\nA way to approach model heterogeneity across clients with different resource constraints, a 'masking trick' for approaching non-iid-ness and a modification to BatchNormalization in the Federated Learning setting.\n\nIn order to receive differently sized models that still allow to be aggregated systematically on a central server, the authors propose to deterministically prune away neurons/feature maps of NNs to effectively create networks of different widths. While a more powerful end-device receives, computes with and updates all feature maps, a less powerful device computes only with a fraction of those feature maps. This idea seems novel and interesting to me and I commend the extensive experimental study. \n\nFor the proposed 'static BN', the authors propose to not worry about running estimates across all clients until convergence of the model. I disagree with both, the proposition that 'static BN' is something new compared to normal BatchNormalization, nor that it inherently solves the issue with BN in the federated setup. Firstly, BN in section 3.1 of the original paper suggests using moving averages only as a means to estimate test-time performance during training. At convergence, the final model requires re-computing statistics on the whole dataset, identical to what is proposed in this work. Secondly, apart from the problem of finding a global statistics estimate at the end of training, the usage of mini-batch statistics during training is used as an alternative to the global estimates since the global estimates are too expensive to compute during training. In centralised training, a random mini-batch represents the global data-set well enough. In non-iid data settings, a random mini-batch from a client does not serve as a good estimate of the global data statistics, only for the statistics of its local data-set. \nI therefore do not understand how the proposed solution of 'sBN does not track running estimates and simply normalize [sic] batch data' addresses the issue that a client-level mini-batch does not represent global statistics. \nA common approach to dealing with BN is to simply replace BN with, for example Group Normalization (https://arxiv.org/pdf/1910.00189.pdf) (Also see Section 5 for a discussion of the issues of BN in FL). \n\nI am therefore asking the authors to explain the exact differences of sBN to normal BN with respect to re-estimation of global statistics at the end of training and to elaborate on the issue of using mini-batch statistics as an estimate of global data-set statistics. If indeed there is a difference to normal BN, I would like to see explicit experimental results that compare normal BN to sBN and, ideally, to GroupNormalization as a way to circumvent the BN issue. \nMaybe I am misunderstanding in the sense that the 'Masking Trick' also somehow alleviates the non-iid data issues with BN. If that is the case, I would like to see an explicit ablation study that distinguishes between the two. \n\n\nWith respect to non-iid data and the proposed 'masking trick' the authors cite Zhao et al. stating that the weight divergence mostly occurs in the last classification layer of networks. Inspecting Figure 2 of that paper, this conclusion can be drawn only for one of the three experiments at display. I agree that this is a minor point and the proposed trick sounds reasonable and interesting to me. In oder to see its effectiveness, however, there needs to be an ablation study with and without that trick. I cannot find such an experiment in the paper. Furthermore it should be stated that label skew is just one of the possible sources of non-iid-ness in FL. Lastly, the authors mention that the masking trick 'allows local clients to switch to another subtask simply by changing its mask...'. I have troubles understanding what is implied here. From a client's perspective there is only its local label-distribution. If a client is assumed to be new to the federation of clients, the new client would receive the un-masked global model presumably. In which setting would a client require a new mask (from another client?)\n\nExperimental Evaluation:\nThe authors present a large range of experiments for different scenarios and levels of heterogeneity between clients.\nI do have issues understanding the results precisely though. The authors do not mention what the x-axes in Figure 2 represent. Is the y-axis local or global accuracy? In combination with Tables 1 and 2, I am confused. If Standalone and FedAvg have 633K parameters respectively, how does a 100% model a (first row in Table 1) have 1.6M parameters? Presumably, experiments were conducted with the same full, 100% CNN model architecture. Alternatively, the hyper parameters in Table 4 in the Appendix do not make sense to me. An alternative interpretation would be that these are the amount of parameters communicated until convergence - but then again the hyperparamters are inconclusive and additionally, the space requirements of 100% model a should still be the same as FedAvg. Or does this column describe the amount of communication at 32bit float precision? \nIf the authors chose a different architecture for their baselines, then the results are inconclusive. \nI am assuming that 'Standalone' refers to no communication between clients, but that needs to be specified! \nAlso I am assuming that 'Local' assigns zero probability on $p(y=c|x)$ for those classes $c$ that are not present on a client during training. Again, this is not explicitly specified. Are the reported results averaged across clients? Are they weighted by the amount of data in the local test-sets?  In the conclusion, the authors state that their method achieves better results with fewer number of communication rounds. I can no-where see a comparison of communication rounds. \n\nThe authors mention two scenarios: Fix and Dynamic and explicitly say 'We annotate Fix for experiments with a fixed assignment of computation complexity levels and Dynamic for local clients uniformly sampling ...\". I cannot find this annotation anywhere and am therefore confused which setting the results correspond to. \n\nI am missing one axis of evaluation: In a heterogeneous (Fixed) setting with, for example, 50% a and 50% level e, how is the average local performance on devices with model a and model e separately. In the text, the authors describe 100% model e achieves 77.09% accuracy and a 50-50 mix with model a achieves 89%. But that does tell me nothing about how much the (weak) clients with model e improved through the increased power in these other 50% devices. In the next sentence the authors claim that 'HeteroFL can boost client's performance with low computation and communication capabilities'. But reporting the average could also allow for the conclusion that only clients with higher compute power and a larger model achieve higher performance. \n\nConclusion\nThe paper proposed three elements, a heterogeneous modelling approach, sBN and the masking trick. Apart from confusion in motivation and explanation, the experimental section requires most attention in my opinion. Things are simply very unclear to me.  Terms, axes and results need to be properly discussed. Furthermore, the effects of HeteroFL, sBN and the masking-trick need to be independently studied, otherwise no conclusion can be drawn on the effectiveness of the individual ideas. I would recommend the authors to re-focus their paper on the heterogeneous training idea alone and leave sBN, which I don't understand at the moment, and the non-iid remedy trough masking to another paper. \n\nI believe that the idea of dynamically adjusting the model width to the local compute capabilities in the way the authors present it is promising. However I need to be convinced that less powerful clients can meaningfully contribute to the global model and have higher performance compared to training a small-sized global model in the first place. I encourage the authors to revisit their motivation for elements of this work (HeteroFL, sBN and masking), refocus, and fix their experimental discussion. I believe that there is enough merit to this idea and paper to be accepted with major effort during the rebuttal. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YL51Fy8HSRT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new federated learning framework called HeteroFL, which supports the training of different sizes of local models in heterogeneous clients. Clients with higher computation capability can train larger models while clients with less computation capability train smaller models, and all these model architectures belong to the same model class. This approach dramatically benefits clients with limited computation capability and fully exploits their computation power. \n\nStrengths:\n1. The paper is well-motivated. The communication and computation problem does exist in federated learning, which made the proposed approach practical to apply. \n2. The idea is novel enough. As far as I know, no other papers exploit the potential of model heterogeneity in federated learning.\n3. The experiment is solid and rigorous enough to support the main idea. The authors use three different models and three corresponding datasets to conduct the experiments. The results show that model heterogeneity is promising, which outperforms state-of-the-art federated learning approaches.\n4. The paper is well written. The authors organize the whole paper concisely and comprehensively, which makes the paper easy to read.\n\nWeakness:\n1. Although static batch normalization and Masked Cross-Entropy Loss are not the main contributions of this paper, I think it's more persuasive to prove their effectiveness by comparing it with baseline experiments instead of simply applying it.\n2. The experiment setting should be elaborated more, such as the hyperparameters.\n\nAlso, I have some questions regarding the paper content:\n1. Why the parameter size of Standalone, FedAvg, and LG-FedAvg is smaller than many other models in Table 2? Intuitively, Standalone, FedAvg, and LG-FedAvg use complete model architecture, so the parameter size should be the largest. However, it is less than models like a and b in Table 2.\n2. How did you choose the shrinkage ratio? Have you compared the results of different shrinkage ratio? And also, Have you tried other methods of shrinking models?", "extra_scores": {}, "extra_text": {}}, {"review_id": "71Xpb3GeOG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework named HeteroFL to address heterogeneous clients equipped with very different computation and communication capabilities.\n\nThe proposed methods were examined on three datasets for image classification and language modelling tasks.\n\nExperimental results show that the proposed method boosts accuracy of the baseline FedAvg.\n\nHowever, there are various undefined notation and missing details in the paper. Therefore, the paper is not easily readable and requires readers to make assumptions to fill the gaps. Some of the proposed strong claims should be supported and verified either theoretically or experimentally. In addition, experimental analyses should be improved by comparing the proposed method with the other related work.\n\nMore detailed comments and suggestions are as follows:\n \n- Please use different notation for matrices and sets. For instance, $W$ denotes both a matrix/tensor and a set in the paper, which causes a problem in equations (1)-(3).\n\n- In equation (1), does W_i^p denote a set or a matrix/tensor? \n\n- What do W^t_g[: d_m, : k_m] and W^{p−1,t+1}_ g \\ W^{p,t+1}_g denote?\n\n- Could you please elaborate why \"small local models can benefit more from global aggregation by performing less global aggregation for part of larger local model parameters.\"? That is, what are the benefits, and how do you assure that this approach enables to have these benefits?\n\n- Please explain how you calculate statistics of hidden representations from local data after the model converges more precisely. How do you determine the convergence criteria?\n\n- It is stated that \"Local models only upload their statistics for once after optimization is completed.\" What are these statistics and how are they used when they are uploaded to the server?\n\n- r^{p-1} is a scalar constant. Therefore, Scaler module scales feature activations by a constant. How does this help training?\n\n- Please define \\phi() (activation layer) more clearly. Do you refer to a non-linear activation function?\n\n- Please define local capabilities information and L_{1:k} and L_m.\n\n- Please describe details of architectures of the CNN, PreResNet18 and Transformer used in the experiments.\n\n- Please explain what \"Standalone\" denotes in the tables.\n\n- Please explain how you construct complexity levels in detail. \n\n- Please explain the masking operation used in Masked Cross-Entropy Loss. More precisely, how do you mask out the output?\n\n- The proposed method follows similar motivation and approaches of split learning and federated-split learning methods. \nCould you please elaborate similarity/difference between your proposed methods and these methods? \nA comparative experimental analysis would be also helpful to explain the novelty over these methods.\n\nFollowing the rebuttal:\n\nI checked comments of other reviewers and response of authors. Most of my questions were addressed in these responses. Therefore, I improve my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2pvqnGhpIXi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a novel FL algorithm named HeteroFL (the name might sounds weird to some peoples) and 3 different simple methods to improve FL in heterogeneous conditions (i.e. both in term of clients and data partitioning). These tricks are: 1. A revised batchnormalisation; 2. a pre-activity scaling; 3. a masked loss (i.e only consider local classes)  to help with non-IID datasets. All these modifications have been tested on 3 different datasets and 2 different tasks. From the results, we can see that the proposed approach works better. Although, it is not clear from where the benefit comes. \n\nMy biggest concern with this paper lies in the complexity of the problem raised vs the lack of analysis of the proposed solutions. Appart from HeteroFL that is definitely a very nice idea, the small tricks try to address very important concerns related to FL. 1. What do we do with running statistics?; 2. How do we manage highly non-iid partitioning?. Unfortunately, the given solutions appear to be small \"tricks\" or \"fixes\" without any real theoretical or empirical insights. In my opinion, each problem should be detailed and discussed in a standalone paper. In this extent, reading this paper is a little bit like: \"We present this method (HeteroFL), that works pretty well if we add these little fixes to very important problems\". But we have no-idea on how these little fixes actually help HeteroFL (and thus could also help FedAVG, FedPROX etc etc). \n\nHowever, it is worth noting that the core idea of this paper: HeteroFL, is a very simple and elegant way to deploy FL on an heterogeneous client set. \n\nPros:\n+ HeteroFL is a very nice idea to deploy FL with an heterogeneous set of clients, and it seems to work well.\n+ While the reasons aren't clear, the 3 proposed methods to stabilise and enhance the training process could help with bigger FL questions.\n+ The paper is self-contained.\n\nCons:\n- some claims are wrong: \"the clients with the lowest capabilities will not be the bottleneck of FL’s performance\" -> Well, according to the results, adding smaller clients (i.e. less powerful) always harms the performance. \"The results show that HeteroFL can boost clients’ performance with low computation and communication capabilities by allowing the training of heterogeneous models with larger computation complexities.\" -> This sentence is unclear and leads to a wrong statement. HeteroFL isn't boosting the performance of small clients. Indeed, we can expect that small clients aren't even able to train the model, so HeteroFL is allowing us to train on such clients, but it is not boosting the performance. According to the results, smaller models = worst performance.\n- sBN, scaling and masked loss \"seem\" to help, but it absolutely unclear to which extent. Also, why are the statistics finally uploaded to the server for inference (isn't this a leak of information ?). The theoretical motivation of the Scaler with the dropout analogy isn't clear at all. I'm pretty sure that it is mostly a matter of re-phrasing. \n\nRemarks:\n- Eq. 1, 2 and 3 are a bit hard to read. Is the \"\\\" symbol used to denote integer divisions ? From a first read, it is not clear how weights are aggregated in intermediate complexity levels.\n- The explanation of the effect of the proportionality is really unclear. Therefore, it is very hard to understand what Figure 2 (and the others) are about ... \"To demonstrate the effect of proportionality of clients with various computation complexity levels, we interpolate from 10% to 100% with step size 10% of global model proportionality. For example, a − b means to interpolate between a and b models starting from 10% of clients assigned level a and 90% of clients assigned level b to 100% level a clients. \" -> This is very hard to understand. What is global model proportionality ?\n- Most of the Figures aren't black and white compatible (all the curves are almost impossible to distinguish).\n- Section 3 should be splitted in 4 parts. One for each proposed method.\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had a number of concerns which seem to have been addressed by the authors in the discussion phase.  All the reviewers are in favor of accepting the paper. The paper provides an interesting/novel idea for federated learning with heterogenous clients/devices. 
Decision: Accept (Poster)