FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction
Authors: Samiul Alam
Luyang Liu
Ming Yan
Mi Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
Model Heterogeneity
Large ModeIs

Abstract: Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. Empirically, we show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout. Lastly, we evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex.

FedRolex: Modee--eteo Federated Learning with Rolling Sub-Model Extraction
Samiul Alam 1:2 Luyang Liu Ming Yan*.,, Mi Zhang
Miichiaan State University, "The Ohio State University, Google Research, "The Chinese University of Hong Kong, Shenzhen
alamsami @mau edu, Juyangliudgooglee.omm yanmingdcuhk edu.cn, mizhang. 16osu. edu
Abstract
Most cross-device federated learning (FL) studies focus on the modee-hhoooeneous setting where the global server model and local client models are identical. How- ever, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, partial training (PT)-based approach that enables model-heterogeneond FL and can train global server model larger than the largest client model. At its core, FedRolex employs rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-ar PT- based modee-hheteoogeneous FL methods (e.g. Federated Dropout) and reduces the gap between
and model-homogencout FL, especially under the large-model large-datast regime. addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex ex on an emulated real-world device distribution to show that TedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code available at: https://gituub.com/AAof7 MI.Sys-Lab/FedRRley
Introduction
Federated learning (FL) is a machine learning paradigm that trains models from distributed clients with private data under the coordination of a central server 2). In this work, we focus on cross- edge devices. The majority of existing
device FL where clients are usually cross-device FL studies focus on the model-homogeneeus setting (16), in which the server model and the client models across all the participating client devices are identical. However, model- homogeneous FL are confronted with two fundamental constraints: (1) device heterogeneity is more realistic consideration when deploying FL systems in real-world applications: different client devices could have very diverse on-device resources and are only capable of training models with capacities that match their on-device resources. Having the same model all the devices would, unfortunately, exclude clients with low-end devices who would otherwise make unique contributions model training from their own local data; (2) state-of toffttheart machine learning has moved towards large models [7] such as Transformer [8] Restricting server and client models to be the same inevitably causes model-homogenae resource constraint of client devices.
FL to fail train such large models due the
To relax the fundamental constraints of hodee-homooeneous FL, proposed where heterogeneous models with different capacities across the server and the clients are trained during the federated training process. One primary challenge in hodee-hhetroogeneuus FL is
FL was
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Comparison of FedRolex with model-homooenneou and model-helerogeneous FL methods
Model Heterigeneity
Aggregation Scheme
Sub- Sab-model
Need of Public Data
Server Model
Compatihility with
Extraction Scheme
Size
Secure Aggregation
FedAng
No
Client Model
Yes
No
Client Model
Yes
SCAFFOLD
No
No Unlabeled
Clieni Model
Yes
FeiBE X
Clieni Model
No
FeaGKTT
No
Largest Client Model
No
FedDF
Yes
Knowledge Distillation
Unlabeled
Unlabeled
Largest Model
angest Client Model
No
No
Fed-ET [+
Unlabeled
2:
Largest Client Model
No
Federated Dropout [3
Random
No
Largest Client Model
Yes
FjORD
HeseroFL
[5]
[
Yes
Portial Training
Static
Static
No
No
Largest Client Model
Largest Model
Yes
Yes
FedRolex (Our Appronch)
Rolling
No
Largest Client Model
Yes
the aggregation of heterogeneous client models. To address this challenge, knowledge distillation (KD))based approaches have been proposed [2H in which the client models serve as teachers and the server ensembles the knowledge distilled from the individual client models, However, KD-based approaches general require public data on the server to achieve competitive model accuracy. whereas the desired public data may not be always available in practice. Moreover, since KDDbased approaches need the individual client models (whole models or prediction layers) or their outputs to be sent to the server, they are incompatible with secure aggregation protocols [16], which limits their privacy guarantee, To remove the dependency on public data and ensure compatibility with secure aggregation, partial training (PT) approaches such random sub-model extraction (Federated Dropout (3) and static sub-model extraction (HeteroFL 04,, FjORD (5]) were proposed. In these approaches, each client trains smaller sub-model extracted from the larger global server model, and the server model is updated by aggregating those trained sub- models. However, the fundamental issue of existing PT-based methods is that the sub-models are extracted in ways (either random or static) such that the parameters of the global server model are not evenly trained. This makes the server model vulnerable to client drif! induced by the inconsistency between individual client model and server model architectures a unique challenge of model-heterogeeeene FL.
In this work, we propose PT-based model-heeerogeenouu FL approach named FedRolex to tackle the fundamental issue of existing methods. The key difference between FedRolex and existing PT-based methods is how the sub-models are extracted for each client over communication rounds in the federated training process. Specifically, instead of extracting sub-models in either random or static manner, FedRolex proposes rolling sub-mode extraction scheme, where the sub-model is extracted from the global server model using a rolling window that advances in each communication round. Since the window is rolling, sub-modek from different parts of the global model are extracted in sequence in different rounds. As a result, all the parameters of the global server model are evenly trained over the local data of client devices.
The proposed rolling sub-model extraction scheme, though simple, has equipped FedRolex with multifold merits compared to prior arts (Table[]): (I) FedRolex enables different parts of the global server model to be evenly trained, which mitigates the client drift induced by model heterogeneity. (2) Contrary to static sub-model extraction approaches (HeteroFL, FjORD), FedRolex is able to train a global server model that is larger than the largest client model, enabling FL to benefit from the superior performance brought by large models. It echoes some concurrent efforts in developing FL primitives to support training large server models in cross-device settings, e.g. Federated Select [17] (3) Compared to random sub-model extraction (Federated Dropout), we show in theoretical statistical analysis in Section 3 and Appendix A. the global server model is trained more evenly by FedRolex as the expected number of rounds for FedRolex going through all the parameters of the global model for at least certain times is smaller than that of Federated Dropout, (4) FedRolex only needs transmit the sub-model that is needed by given client instead of the full server model the client. This allows clients to contribute to federated training under resource constraints and reduces communication overheads (Appendix A.6 (5) Lastly, FedRolex is fully compatible with existing secure aggregation protocols that enhance the privacy properties of FL systems.
We evaluate the performance of FedRolex under two regimes: i) small-model small- -mall-dataset regime (most existing cross-device FL. studies use this combination), and ii) large-model large-dataset regime
In hodel-hhooggeeeuss FL, client drift is primarily induced by data heterogeneity across clients. In model- model-bbterogeneouus FL, model heterogeneity across clients is another critical source that induces client drift.

---

(this combination echos recent efforts on pushing the frontier of cross- cross-device FL towards training
large server models on large-scale datasets [T8 21)) We highlight five of our findings: (1) FedRolex
consistently outperforms state-of-the-an PT-based model-hhtterogeneuur FI. methods under both
small-model small-dataset and large-model large-dataset regimes ($4.1). (2) FedRolex reduces the gap between large-dataset regime ($4.2). (3) With FedRolex, under both regimes, having a small fraction of
and model-homogeneous FI., especially under large-model
large-capacity models could significantly boost the global model accuracy ($4.3). (4) FedRolex is able to train a global server model that is larger than the largest client model and outperforms Federated Dropout in terms of global model accuracy ($4.4). (5) Using an emulated real-world device distribution, we show that FedRolex enhances the inclusiveness of FL and boosts the performance of low-end devices that would otherwise not benefit from FL (4.5)
2 Related Work
Knowledge Distillation (KD)-based Model- Hete---eetroggseouus FL. One primary approach for model- heterogeneous FL in cross-device settings is based on knowledge distillation (KD) [22]. In partieular, FedDF [10 distills knowledge from a set of classifiers trained with private data from federation of client devices. The logit outputs of each classifier against an unlabeled public dataset are then used to train a student model at the server with KD, Similarly, DS-FL [I]] utilized an unlabeled public dataset at the server and proposed a distillation- based supervise FL approach to enhance performance by pseudo-labeling the public data. FedGKT ( proposed group knowledge transfer in which knowledge is transferred to a large model in the server from clients without public data. Fed-ET 12 proposed a weighted consensus distillation scheme with diversity regularization that enables the training of a large server model with smaller client models. KD-based approaches, however, have several limitations: they often require public data to achieve competitive model accuracy. This is because model accuracy is dependent on the size of public data as well as the domain similarity of public data with client data [10 12 23). Furthermore, KD- methods use client model weights partially or entirely as teachers to transfer knowledge to the server, they are incompatible with secure aggregation protocols, making them vulnerable to backdoor attacks [0].
Partial Training (PT)-based Modee-Heterogeneeuu FL., To address the limitations of KD-based approaches, partial training (PT) has emerged as another solution for model-heterogeeneuss FL.. Depending on how the sub-models are extracted from the global server model, existing PT-based methods can be in general categorized into two groups: random sub-model extraction and static sub-model extraction. Specifically, inspired by the dropout technique commonly used in centralized training 124) Federated Dropout 13 proposed to randomly extract sub-models from the global model. Though easy to be integrated into existing FL frameworks, as reported in [25), Federated Dropout becomes less effective when the data heterogeneity is high and the client cohort is small
due to its randomness in selecting sub-models. In contrast, HeteroFL [14] and FjORD [15) proposed
static extraction schemes where sub-models are always extracted from designated part of the global server model. However, such a static extraction strategy has two primary drawbacks. First, the global server model is restricted to the same size as the largest client model. As such, the size and capability of the global model are implicitly restricted by the resources of client devices, making it not able to train large models due to resource bottlenecks at client devices. Second and more importantly, under static extraction, depending on their resource demands, different sub-models can only be trained on clients whose on-device resources are matched. As consequence, part of the global server model cannot be trained on data at low-end client devices, causing different parts of the global model to be trained on data with different distributions. This would degrade the performance of the global model, especially under high data heterogeneity. In this work, we propose rolling sub-model extraction scheme that tackles the issues of both random and static sub-model extraction methods.
3 Methodology
3.1 Formulation of Model-Heterogeneee FL
Let denote N client devices with non-IID (non-iden)) connndentically and independently distributed) local data D {D1, D2, .... D Hodel-homogeneouu FL trains global model of parameter 0 by solving the following optimization problem:
N
min F (()) Σ (()
(1)
0
3

---

with
1 10
O) &
1(0:
Mp
where D, dn,, dn.3... An m.. is the set of local data samples of client n and pn. is its corresponding weight such that pr. 2 0 and Σn Pn 1.
In comparison, in model hetteel-eetrggeeouus FL, clients train local models with heterogeneous capacities G {31, .2.... AN J. and the local objective function of the n client becomes
(O,)
m2 Σ
1
1(On kn,
Here, Br denote the model capacity of client n. and we define it as the proportion nodes extracted from each layer in 0 for client π. The size of 0n depends on 3n, and the parameter 0, is obtained by selecting sub-model from the global model which can change from one round to another. If On changes, the objective function also changes. For simplicity, we use the same notation for the loss function for all clients and rounds, though they differ between clients and rounds. The key to model- hedeo--hetroogeneuus FL is selecting 8n from the global model 9 given model capacity Sn.
3.2 FedRolex: Mede--Heetsogeeneous FL with Rolling Sub-Model Extraction
As partial training (PT))based approach, each client, FedRolex trains only a sub-model extracted from the global server model and sends the corresponding sub- model updates back to the server for update aggregation. To help understand how FedRolex ex works, for simplicity, Figure Taareeilllune Iillustrates three rounds of federated training of FedRolex on two participating heterogeneous clients, where one trains large-capaciny sub-model (left) and the other trains a small-capacity one (right). At the high level, at each round, the server extracts sub-models of different capacities from the global model and sepa- rately broadcasts them to the clients that have the cor- responding capabilities. The clients train the received sub-models on their local data and transmit their het- erogeneous sub-model updates to the server. Lastly, the server aggregates those updates, and the result of the ag- gregation is used to update the global model for the next round. The pseudocode of FedRolex is in Algorithm
Global Server Model
Round f
Round
Round j +
The key to the design of FedRolex involves two design choices. In the following, we describe them in detail.
Large-capacity Client Model
Small-capacity Client Model
(1) What sub-models to be extracted for each client across different rounds? At the server, FedRolex uti- lizes rolling window extract the sub-model from the global model. The rolling window advances in each round, and loops over all parts of the global model in sequence across different rounds. This process iterates such that the global model is evenly trained until convergence.
Figure 1: Overview of the rolling sub- model extraction scheme in FedRolex.
Taking Figure 1 extracted from the global model is {a, d, c, d) and {c, d, e}, respectively. In round j + 1, the rolling window advances one steffl the cargeccapacity and small-capacity client model becomes (b, c, d. e} and (d, e, a), respectively. Similarly, in round j + 2, the rolling window advances one step further, and the large-capacity and small-capacity client model becomes {c, e, a} and {e, a, b), respectively.
example: in round j, the large-capacity and small-capacity client model
Such a rolling sub-model extraction scheme can be formalized as follows. Let 0.5) denote the parameters of the sub-model extracted from the global model for client in round K; denote the
"The step size is hyperparameter of FedRolex. Please refer Appendix A.4 for our ablation study it.

---

total number of nodes in layer of the global model, and S denote the node indices of layer i of the global model that belongs to the extracted sub-model for client n in round j. Then the layer of the sub-model extracted by the rolling sub-model extraction scheme for client in round is given by:
|0 = (i. - 1, 1....................... . [3.K:] 1 (3.3 - 1....,, K. - -3 (0. 1.......... (0.1......++1.... 1. -1 K,}
if [8n S K₁ else.
(4)
where j j mod KG-
(2) How to aggregate heterogeneous sub-model updates update the global model? FedRolex employs straightforward selective averaging scheme with no client weighting to aggregate heteroge- neous sub-model updates sent from the clients to update the global mode Specifically, computes the average of the updates for each parameter of the global model separately based on how many clients in a round updated that parameter. The parameter remains unchanged no clients updated
Taking Figure D again as an example: in round the updates for and b are obtained from the large-capacity model and the update for from the small-capacity model only. In contrast, since c and d are part of both models, the update is computed by taking the average from both models.
Algorithm 1: FedRolex
Initialization (((),,) / NN
12 Subroutine clientStep (0l"), D,
Input :D, Bn Vn E N, Output:0'
13
mn f- len (n.) for k + to m. do
14
Server Executes
15
On - 0n Vi(0,) dn,,))
for j - 0 to J - do
16
end
Sample subset M from N
17
return On
Broadcast (o5)
m, to client E M
Vi, from Equation m.
7
for each client m E M do clientStep (8 Dm end
10
Aggregate g(++)) [i,b]
according to Equation 10)
end
3.3 Comparison with Random and Static Sub-model Extraction Schemes
Existing sub-model extraction schemes can be grouped random-based (Federated Dropout) and static-based (HeteroFL. FjORD) methods. In this section, we describe the differences between them and the proposed rolling-based based scheme employed in FedRolex. For comparison purpose, the pseudocodes of both Federated Dropout and HeteroFL are included in Appendix[A.3 A.
3.3.1 Comparison with Random Sub-Model Extraction Scheme
In random sub-model extraction scheme, in each round, the sub-mode models extracted from the global model a random manner. As such, the layer of the sub-mode extracted by the random sub-model extraction scheme for client n in round j is given by:
{kk integer ke E [0, K; - 1] for c [8,
(5)
where a total number of 8,, K, nodes are randomly chosen from the global model.
Discussion: As shown in Figure FigureBleft). similar to the proposed rolling-based scheme, the sub-models extracted across different rounds by the random-based scheme have different architectures. However,
'In Appendix A.3 we did an ablation study on three client weighting schemes and compared them with the non-weiee weighting (selective averaging) scheme. We find that the performance of the three weighting schemes is not significantly better than the non- weighting scheme. Please refer to Appendix A.3 for details.
5

---

Model
Global Server Model
Round
Round j+1
Small-capacity Client Model
Large-capacity Client Model
Small-capacity Model
Large-capacity Model
diobal-modeh-cpacity
Sub-model Extraction
Static Sub-model Extraction
Illustration of how sub-models are extracted by random sub-model extraction scheme (Left) and static sub-model extraction scheme (Right) over two rounds.
Figure 2:
due to its randomness in selecting sub-models in each round, the global model is trained less evenly, making more vulnerable to client drift. In short, although the expected value of the frequency for updating each index is the same for all the indices, their exact frequencies are not the same due to randomness. Consequently, the candom-based scheme cannot balance the update frequencies of different parts of the global model, and it inevitably takes more rounds to update the whole global model. Moreover, as we show in Appendix [ the expected number of rounds for Federated Dropout selecting all sub-models at least times in the order of logg() + I(m 1) log log I, which is larger than that of FedRolex, mI.
3.3.2 Comparison with Static Sub-Model Extraction Scheme
In static sub-model extraction scheme, in each round, the sub-mut sub-models are always extracted from designated part of the global model. As such, the layer i of the sub-model extracted by the static sub-model extraction scheme for client in round is given by:
1 = 1.2, 13, 1}.
(6)
Note that s(i) does not depend on j. In other words, as shown in Figure is extracted for each client in every round. Moreover, the client model with smaller capacity and client model with larger capacity are not independent. As shown FigureZrright), the small-capacity model {a, b, 3 is part of the large-capacity model {a, b, d). which again, is a part of the global- capacity model {a, b, c, d. These are the two key differences between both the madom-baseed and the proposed rolling-based scheme.
2 right,, the same sub-model
Discussion:
Given that, the static-based scheme, however, has two primary drawbacks, First, cover the whole global model, there must be clients to train the full-size global model {a, b, c, d, e}. As such, the global model is restricted the same size as the largest client model. Second, as shown in Figure right,, while a, b and will be trained on data on all three types of clients, d will not be trained on data on small-capacity clients, and e will only trained on data on dobal--odeel-cpacee clients. As a consequence, different parts of the global model are trained on data with different distributions, which inevitably degrades the global model training quality.
Experiments
Datasets and Models. We evaluate the performance of FedRolex under two regimes. Under small-model small small-dataset regime, we train pre-activated ResNet18 (PreResNet18) 88 models [26] on CIFAR-10 10 and CIFAR- (2 We replace the batch Normalization PreResNet18 with static batch normalization [4 28 and add scalar module after each convolution layer [14). Under large- model large-dataset regime, we use Stack Overflow (29] and followed
train a modified 3-leyer
Transformer (8) with vocabulary of 10, 000 words, where the dimension of token embeddings is

---

128, and the hidden dimension of the feed-forward network (FFN) block is 2048. We use ReLU activation and use 8 heads for the multi-head attention where each head is based on 12-dimensional (query, key, value) vectors. The statistics of the datasets are listed in Table 2
Table 2: Dataset statistics.
Dataset CIFAR-10
Train Clients
Train Examples
Validation Clients
Validation Examples
Test Clients
Test Examples
100
50,000
NIA
N/A
N/A
10,000
CIFAR- 100
50,000
10,000
Stack Overflow
342,477
135,818,730
38,758
16,491,230
204,088
16,586,035
Data Heterogeneity. For CIFAR-10 and 110, we followed HeteroFL [4 to model non-IID distributions by restricting each client to have L labels. In our evaluation, we consider two levels of data heterogeneity. For CIFAR- 10, we define low data heterogeneity. For CIFAR- 100, we use L. 20 as high data heterogeneity and I, 50 as low data heterogeneity. These two levels roughly correspond to Dirichlet distribution Dirx (a) with equal to 0.1 and 0.5, respectively. For Stack Overflow, the dataset is partitioned over user IDs, making the dataset naturally non-lID distributed.
2 as high data heterogeneity and L. 5 as
Model Heterogeneity. Without loss of generality, in our evaluation, we consider five different client model capacities
{1. 1/2, 1/4. 1/s, 1/16} where for instance, 1/2 means the client model capacity is half of the largest client model capacity (full model). To generate these client models, for ResNet 18, we vary the number of kernels in convolution layers and keep the nodes in the output layers the same. For Transformer, we vary the number of nodes in the hidden layer of the attention heads.
We compare FedRolex against both sfae-oo-the-an P'T-based model model-hhetroggenouss FL methods including Federated Dropout [33] and HeteroFL [4 as well as state-of-the oftt-off-theart KD-based model-heterogeeneuu FL methods including FedDF [10], DS-FL [I]] and Fed-ET [12] 5 To ensure a fair comparison, all the PT-based baselines are trained using the same learning rate, number of communication rounds, and multi-step learning rate decay schedule. The details of the schedule for
Baselines.
each dataset and experiment are described in Appendix AA.
Configurations and Platform. For CIFAR-10 10 and CIFAR-100, apply bounding box crop 30 to augment the images. In each communication round, 10% of the clients are randomly selected from a pool of 100 clients. For Stack Overflow, we followed 2 to use 10% dropout rate to prevent over- fitting, and 200 clients are randomly selected from a pool of 342, 477 clients in each communication round. The details of the hyper-parameters for model training are included in Appendix (A.7] We implemented FedRole ex and PT--baed baselines using PyTorch 31 and Ray and conducted our experiments 8 NVIDIA A6000 GPUs.
Evaluation Metrics. We use global and local model accuracy as our evaluation metrics. Specifically, global model accuracy is defined as the server model accuracy the test set; and local model accuracy is defined as the accuracy of the server model on each of the client's local datasets. For CIFAR-10 and CIFAR- 100, we report the classification accuracy. For Stack Overflow, next word prediction accuracy which includes both out-of-vocabulat (OOV) and end-of-sentence (EOS) tokens. We using three different seeds for Stack Overflow.
report the
experiments using five different seeds for CIFAR-10 and CIFAR- 100 and
4.1 Performance Comparison with State-of-the-Art Model-Heeteoggeneous FL Methods
First, we compare the performance of FedRolex with state-of-the-art PT and KD-based model- heterogeneous FL methods. For a fair comparison, we followed the experimental settings used in prior arts where the distributions of client model capacities are uniform and the global server model is the same as the largest client model.
Evaluation Results: Table summarizes our results. We have two observations. (1) In comparison with state-of-the-art P'T-based methods, under the small-model small-dataset regime, FedRolex consistently outperforms HeteroFI. and Federated Dropout under both low and more challenging high data heterogeneity scenarios. In particular, under high data heterogeneity, Federated Dropout
*We did not compare with FJORD because code not open-source and we could not reproduce their results following the paper.
We did not compare with FedGKT
9 as it is only compatible with CNN models.
7

---

Table 3: Global model accuracy comparison between FedRolex, PT and KD-based model- heterogeneous FL methods, and homooeneoo FL. methods. Note that the results of KD-based methods were obtained from [12]. For Stack Overflow, since KD-based methods cannot be directly used for language modeling tasks, their results are marked as N/A.
High Data Heterogeneity
Low Data Heterageneity
Method
Stack Overflow
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-I00
FedDF
73.81 (2 0.42)
31.87 0.46)
76.55 (1 0.32)
37.87 (#: 0.31)
N/A
KD-based
DS-FL
65.27 (= 0.53)
29.12 (+ 0.51)
68.44 (# 0.47)
33.56 (# 0.55)
NIA
Fed-ET
78.66 (# 0.31)
35.78 (# 0.45)
81.13 (# 0.28)
41.58 (# 0.36)
N/A
HeteroFL Federated Dropout
63.90 (# 2.74)
52.38 (= 0,80)
73.19 (# 1.71)
57.44 (# 0.42)
27.21 (# 0.22)
PT-based
46.64 (# 3.05)
45.07 0.07)
76.20 (+ 2.53)
46.40 (+ 0.21)
23,46 0.12)
FedRolez
69.44 (# 1.50)
56.57 (# 0.15)
84.45 (# 0.36)
$8.73 (#: 0.33)
29.22 (# 0.24)
Homogeneous (smallest)
38.82 (# 0.88)
12.69 (# 0.50)
46.86 (#: 0.54)
19.70 (tt 0.34)
27.32 (# O. 0.12)
Homogeneous (largest)
75.74 (# 0.42)
60.89 0.60)
84.48 0.58)
62.51 (# 0.20)
29.79 0.32)
which extracts sub-model randomly has worse performance than FedRolex and HeteroFL. which both extract sub-models in deterministic manner. Under large-model large-dataset regime, FedRolex also outperforms both HeteroFL and Federated Dropout. These results together demonstrate the superiority of FedRolex under both regimes. (2) In comparison with state-of-the-art KD-based methods, TedRolex only performs worse than Fed-ET and FedDF on CIFAR-I0 under high data heterogeneity, but outperforms all the KD-based methods on the more challenging CIFAR- which has a larger number classes than CIFAR-10 10 under both low and high data heterogeneity scenarios. It important to note that KD-based methods leverage public data boost their model accuracy while FedRolex does not.
4.2 Performance Comparison with Homoggenos FL Methods
We also compare the global model accuracy of FedRolex with two model-homogeneouss cases where all the clients have the largest capacity model (33 = {1)) and the smallest capacity model
(3 (1/16)), representing the upper and lower-bound performance, respectively.
Evaluation Results: As listed Table 3 compared with other PT-based methods, FedRolex reduces the gap in global model accuracy between model-heterogeneous and upper-bound model- homogeneous settings. In particular, FedRolex is on par with the upper-bound model-homogeneous case for Stack Overflow, whereas both HeteroFL and Federated Dropout perform worse than the model homogeneous case using the smallest model. This result indicates that with FedRolex, we will not be constrained to only using high-end devices to achieve competitive global model accuracy. Note that Fed-ET achieves a higher global model accuracy than the homel--hmmoennous upper bound on CIFAR-10 under high data heterogeneity, which showcases the advantage of using public data.
4.3 Impact of Client Model Heterogeneity Distribution
In our previous experiments, the distributions of model capacities across client devices are set to be uniform. In this experiment, we aim to understand the impact of the client model heterogeneity distribution. To do so, without loss of generality, we use two client model capacities 3 = {1. 1/16] and vary the distribution ratio between the two (denoted as p) where p represents the case in which all the clients have the largest capacity model (33 = {1]) and p = represents the case in which all the clients have the smallest capacity model (B (2/16))
Evaluation Results: Figure 3 shows how global model accuracy changes when p varies from 0 to 1 for CIFAR- 10, CIFAR- 100 and Stack Overflow. We have three observations. (1) For CIFAR- 10 (Figure f)), there is large gap in global model accuracy between high and low data heterogeneity for a wide range of p (from 0.1 to 1). This is because CIFAR-10 is relatively simple task and hence the global model accuracy is bottlenecked by the level of data heterogeneity instead of model capacity. This result indicates that having more high-capacity models in the cohort has only limited contribution to global model accuracy. (2) For the more challenging CIFAR- gap in global model accuracy is much lower between high and low data heterogeneity. In contrast to CIFAR- 10, the global model accuracy is bottlenecked by the highest capacity of the models rather than the level of data heterogeneity. (3) For both regimes (Figure (i) vs. Figure 3 we observe
F the
8

---

Global Model Accuracy
Global Model Accuracy
Global Model Accuracy for Stack Overflow
CIFAR10
CIFAR100
80
30
Accurcyy
70-
Accurscy
50
Apoumayy
25
80-
20
00
80
15
low heterogeneily
- low heterogeneily high heterogenelty
high haterogereity
30
10
0.2
0.8
(iI)
(iii)
Figure 3: Impact of client model heterogeneity distribution on global model accuracy for (i) CIFAR-
10, (ii) CIFAR-100, and (iii) Stack Overflow.
that having small fraction of large-capacity models significantly boosts the global model accuracy, but keeping increasing the ratio of large-capacity models has limited contribution to the accuracy.
4.4 Performance on Training Larger Server Model
Similar to Federated Dropout, one advantage of FedRolex over static sub-model extraction methods (HeteroFl. and FjORD) is that FedRolex is able train global server model that is larger than the largest client model. In this experiment, we aim to evaluate the performance of FedRoles ex on training larger server models. To do so, we consider the case where the size of the global server model is = {2, 4, 8, 16} times the size of client models. For simplicity, all client models have the same size.
Evaluation Results: Figure () and Figure (i) compare FedRolex with Federated Dropout in terms of global model accuracy when y for CIFAR- 10 and CIFAR-100. respectively. As shown, although the global model accuracy drops for both FedRolex and Federated Dropout when 7 increases, especially from to 4, FedRolex consistently achieves higher global model accuracy than Federated Dropout across y {2, 4, 8, 16) under both low and high data heterogeneity. For Stack Overflow (Figure 4 fii)), the global model accuracy has a much smaller drop when y increases. This demonstrates the superiority of using large models large-scale datasets for training larger server models.
Global Model Acouracy for CIFAR-10
Global Model Accuracy for CIFAR-100
Global Model Accuracy for StackOverflow
100
Accuracy (%)
Accuracy (%)
20
50
18
10
5
10
15
10
15
1.0
15
Low Heterogeneiity FedRalex
Low Heterogeneity su FedRolev
FedRolex
High Heterogeneity Federated Dropout
High Heteropenelty Federated Dropout
Federated Drapout
(ii
(iii)
Figure 4: Performance on training larger server model when the server model is times the size of the client model for (i) CIFAR- 10, (ii) CIFAR- 100, and (iii) Stack Overflow.
4.5 Enhance Inclusiveness of FL in Real-world Distribution
primary vision of FedRolex is enhance the inclusiveness of FL. To demonstrate this, in this experiment, we use real-worke household income distribution to emulate real-world device distribution. Specifically, we retrieve household income distribution information from Bureau (33] We map 3, = 1/16 with the income group with earning less than $75, 000 and assign proportions of remaining groups in $25, 000 increments with increasing values of 9n.. Detailed mapping of this distribution to the corresponding income distribution is provided in Figure in Appendix A.7

---

Table 4: Performance of FedRolex under emulated real-world device distribution.
High Data Heterogeneity
Low Data Heterogeneity
Dataset
Method
Local Accuracy
Global Accuracy
Local Accuracy
Global Accuracy
Homogenenus (smalleest)
K5,90 (+ (1.66)
38.82 0.88)
66.122 (+ 0.52)
46.86 (# 0.54)
CIFAR-I0
Homogeneous (lurgest)
95.54 (# 0.26)
75.74 (# 0.41)
93.54 (#: 0.44)
84.48 (# 0.58)
FedRolex
94.05 (# 1.01)
63.17 (# 1.45)
91.03 0.36)
80.14 0.52)
Homogeneous (smallest)
34.51 (1: 0.56)
12.69 0.50)
33.22 0.10)
19.70 (2: 0.34)
CIFAR-100
Homogeneous (lurgest)
81.99 (# 0.78)
60.89 (# 0.60)
76.43 (# 0.54)
62.51 (# 0.20)
FedRolex
73.33 (2 0.96)
45.78 (# 1.71)
66.31 (4 0.34)
48.44 (# 0.51)
Homogeneous (smallest)
27.32 0.12)
Stack Overflow
Homogeneous (lurgest) FedRolex
29.79 0.32)
29.55 (# 0.41)
CIFARIO
CIFARLO High Heterogeneity
CIFAR100 Low Heterogenelty
CIFAR100 High Heterogeneity
Law Heterogeneity
20
15
Courte
30
70
80
100
20
40
60
80
Accurryy
Actitety
Accuory
Figure 5: Local model accuracy distribution of FedRolex (orange color) vs. the smallest model- homogeneous case (blue color) for CIFAR-10 10 and CIFAR-100 under low and high data heterogeneity.
Evaluation Results: Table shows both the global and local model accuracies of FedRolex for CIFAR-10 10 and CIFAR-100 100 as well as the global model accuracy on Stack Overflow under the emulated real-world device distribution. Again, we compare with two homoee-hoooeneouss cases where all clients have the smallest and largest model capacities, representing lower and upper-bound accuracy, respectively. We make two observations. (1) Looking at the global model accuracy, FedRolex consistently outperforms the lower-bound model-homogeeeeuus case across CIFAR- 10, CIFAR-100, and Stack Overflow. This result indicates that FedRolex enhances the inclusiveness of FL and improves the accuracy of the global model, which would otherwise not be able to achieve. (2) Looking the local model accuracy, FedRolex significantly outperforms the lower-bound model- homogeneous case on CIFAR-10 10 and CIFAR-100 under both low and high data heterogeneity. This result indicates that FedRolex effectively boosts the performance of low-end devices, which would otherwise not benefit from FL. A detailed illustration of how local model accuracy distribution of individual clients shifts when FedRolex ex is used compared to the smallest model honelllooooeeeeuus case with the same client outreach is shown in Figure 5
Conclusion
We presented FedRolex, a partial training (PT)-based model-hhterogeneouu FL approach that is able to train a global server model larger than the largest client model. FedRolex ex proposed a rolling sub-model extraction scheme that enables parameters of the global server model to be evenly trained to mitigate client drift induced by model heterogeneity. We provided a theoretical statistical analysis of its advantage over Federated Dropout, Our experimental results show that FedRolex ex consistently outperforms state-of-the-ar PT-based methods across models and datasets at both small and large scales. Moreover, we demonstrated its performance on an emulated real-world device distribution and show FedRolex contributes to making FL more inclusive.
Limitations and Future works. In this work, we provided statistical analysis of FedRolex. Full convergence analysis of FedRolex is not trivial and is left for future work. In addition, the goal of this work is train global server model using federation of heterogeneous client models. Determining what models to deploy onto each client after the global server model is trained is a separate task, especially when the global server model is large. We will pursue it as our future work.
6 Acknowledgement
We thank the reviewers for their helpful comments. This work was partially supported by NSF PFI:BIC-1632051, CNS-181455 DMS-2012430 and a Google Computing Platform (GCP) grant.
10

---

References
[1] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2) 2):122)1111 1-000, 2021.
[2] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi. Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang. Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization, 2021.
[3] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication wwt.ctaiiioonnniinnttt learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[4] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings Machine Learning and Systems, 2:429 4500, 2020.
[5] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[6] Hong-Yop You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint -xXix:20099019744 2020.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0725 2021.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[9] Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Feder- ated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33: 14068-14080, 2020.
[10] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351-2363, 2020.
[11] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-bas cemi-supervised federated learning for communication--cfiiceett collaborative training with non-iid private data. arXiv preprint arXiv:2008 2010006800000 2020.
[12) Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. International Joint Conference Artificial Intelligence (IJCAI), 2022.
[13] Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.072100 2018.
[14] Enmao Diao, Jie Ding, and Vahid Tarokh. Heteroff: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.0126. 2020.
11

---

[15] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis. Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
[16] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv: /6/ 1.0444482 .04482, 2016.
[17] Zachary Charles, Kallista Bonawitz, Stanislav Chiknavaryan, Brendan McMahan, al. Fed- crated select: A primitive for communication-and memory-efficient federated learning, arXiv preprint arXiv:2208. 09432, 2022.
[18] Jae Hun Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Theertha Suresh, Shankar Kumar, and Rajiv Mathews. Scaling language model size in cross-device federated learning. arXiv preprint arXh::2204.0977 2022.
[19] Zachary Charles, Kallista Bonawitz, Stanislav Chiknavaryan, Brendan McMahan, and Blaise Agúlea y Arcas. Federated select: A primitive for communication- and memory-efficient federated learning, 2022. URL httpt:tttss//...oooo org/ www.:oog//////aeeeeeooeoe ab0///0/./2.....//////////// 094322
[20] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16070-16084, Curran Associates, Inc., 2020, URL http:::/ neurips. cc/ Paper pdf
[21] Tien-Ju Yang, Dhruv Guliani, Françoise Beaufays, and Giovanni Motta. Partial variable training for efficient on-device federated learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4348-4352. IEEE, 2022.
[22] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge a neural network. arXiv preprint
2(7), 2015.
[23] Samuel Don Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gor- don Wilson. Does knowledge distillation really work? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
URL htt.hhtttt//pppeeeiieeeeee ottp:///eeeeeeeeeeeroooooddddda.
[24] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 19(1) 1958, 2014.
[25] Gary Cheng, Zachary Charles, Zachary Garrett, and Keith Rush. Does federated dropout actually work? In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3387-3395, 2022.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[28] Mathieu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric W Tramel. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Rep- resentation Transfer, and Distributed and Collaborative Learning, pages 129-139. Springer, 2020.
[29] TFF. Tensorflow federated stack overflow dataset. Online: https://www. tensorflow. orgfiederated/aii_docs/s www.ghgalafffiiaooooooaassssseee 2019.
12

---

[30] Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe V Le. Learning data augmentation strategies for object detection, In European conference on computer vision, pages 566-583. Springer, 2020.
[31]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. http: pdf
1ibrary
[32] Philipp Moritz, Robert Nishihara. Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging (Al) applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577, 2018.
[33] U.S. Census Bureau. Percentage distribution of household income in the u.s. in 2020. In Statista, September 2021. Retrieved May 18, 2022, wtomhttps:////ttt tronhattps//rw..sstaiiiaaaa statista. coonnttatssics/////ooooo of 203183/percentage- ditpeeisttstiiiiiiroiooooooaaott www.oonooo-oooouoooooooooooooooe hwwshholl----oo income--nnn--inn the-
[34] Sheldon M. Ross. Introduction Probability Models. Academic Press, San Diego, CA, USA, 11th edition, 2014.
[35] Donald Newman. The double dixie cup problem. The American Mathematical Monthly, 67 (1):58-61) 1960.
[36] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv: 1912.0088, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "lHzfU7_OEtN", "reviewer": "Reviewer_2zgM", "summary": "In this paper, the authors target an important problem in federated learning (FL) and propose a simple partial training (PT)-based technique called FedRolex to enable model-heterogeneous FL. Compared to existing methods, the core innovation in FedRolex is a rolling sub-model extraction scheme, where the sub-model is extracted from the global model using a rolling window that advances in each communication round. As such, parameters of the global model are evenly trained other than trained either randomly or statically in existing methods. This simple innovation has been demonstrated empirically to be the key factor that contributes to the superior performance of FedRolex over state-of-the-art methods. ", "questions": "- Can the proposed technique generalize to other data types such as languages and the corresponding language models?\n- Can the proposed technique generalize to a larger cohort size which is more practical in real-world deployments?\n", "limitations": "The authors discussed the limitations in section 5.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The review of the existing literature is solid. Table 1 summarizes the key differences between the proposed technique and existing works across a number of important dimensions. This is very helpful to understand the contributions of the paper.\n- The proposed technique is simple yet powerful. Such simplicity helps the readers to factor out tricks that are not essential.  \n- The experiments are designed with careful thought and the conclusions drawn from the experimental results are quite exciting, especially enhancing the inclusiveness of FL in real-world distribution. \n\nWeaknesses:\n- The experiments are limited to vision datasets (i.e., CIFAR-10, CIFAR-10). It is not clear whether the proposed technique can generalize to other data types such as languages.\n- The cohort size considered in the experiments is quite small. In practice, a much larger cohort size is normally incorporated in cross-device FL.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MH5WBbl2PEc", "reviewer": "Reviewer_ic9g", "summary": "In this paper, the authors propose to sample the sub-models in a rolling way and find the simple method improves the performance of the global model compared with baselines.\n", "questions": "* KD-based baselines perform much worse than PT-based methods on CIFAR-100, could the authors provide the details of the chosen proxy data?\n* Though sampling the sub-network in a rolling way outperforms baselines in the given experimental settings, it is not clear whether FedRolex outperforms baselines with a higher client participation rate. \n", "limitations": "This paper does not have a limitation subsection. But I don't think the paper will have potential negative societal impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Overall, the paper provides an interesting perspective for understanding the limitations of HeteroFL and Federated Dropout. \n* The authors not only evaluate the PT-based methods but also compare **FedRolex** with KD-based methods.\n* In the appendix, the authors give statistical analysis to further analyze the proposed method.\n\nWeakness:\n* Compared with Federated Dropout and HeteroFL, FedRolex only modifies the sampling method for Federated Dropout. The idea is not novel enough.\n* The experimental results do not list the performance of the simplest model in FedRolex. For the heterogeneous methods, the performance of each group should be listed.\n* The authors only provide the statistical analysis to prove that the index is chosen equally randomly, the convergence analysis is not provided. \n* The authors should provide the communication cost and computation overhead to reach the target accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CznujbdzLMH", "reviewer": "Reviewer_5GU1", "summary": "The authors proposed FedRolex, a model-heterogeneous federated learning framework that breaks the constraint of standard federated learning based on homogeneous model. The authors proposed a rolling technique to extract heterogeneous submodels from a shared global model, which has been shown to achieve much better performance in comparison with other model-heterogeneous federated learning methods such as Federated Dropout and HeteroFL in both high and low data heterogeneity levels.", "questions": "-\tWould FedRolex work on modern large models such as Transformer?\n-\tCan the authors elaborate more on the concept of client drift to make the paper self-contained?\n", "limitations": "The authors discussed the limitations in section 5.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors did a good job in reviewing the existing works, especially the ones that are recently published. \n- This is a timely and important topic to work on, and the authors have demonstrated that a simple submodel rotation technique can significantly bridge the gap between model homogeneous and model heterogeneous settings.\n- The work has a potential to greatly improve the fairness of federated learning. \n\nWeaknesses:\n- The primary weakness of the work is that although the work claims that the proposed FedRolex could train large server models, the model studied in the current version is ResNet18. The work would be much more convincing if FedRolex could train modern large models such as Transformers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8YlqFSRnPnX", "reviewer": "Reviewer_6GHV", "summary": "In this paper, the authors study the model heterogeneous problem in federated learning. The authors propose a simple but effective method, which adopts a rolling window to generate small sub-models for local training. Extensive on real-world datasets verify the effectiveness of the proposed method.", "questions": "Refer to the weakness.", "limitations": "Refer to the weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The model heterogeneity problem studied in this paper is important for the applications of federated learning in real-world scenarios.\n2. The proposed method is simple to implement.\n3. This paper is well-written and easy to follow.\n\nWeakness:\n1. The proposed method can only be used to train CNN-based models and is difficult to be applied to train other SOTA models, such as transformers.\n2. The proposed method reduces computation costs for low-resource clients by only updating a small part of model parameters in a training round. The proposed method still needs different clients to train an intelligent model with the architecture. However, in some scenarios, it may be difficult for the low-resource clients to save and load the model that can be trained on the rich-resource clients. It seems that the proposed model cannot tackle such scenarios.\n3. The computation resources in the rich-resource clients seem not be fully exploited by the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept