ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
Authors: Hui-Po Wang
Sebastian U Stich
Yang He
Mario Fritz
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
progressive learning

Abstract: Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data. However, training is resource-intensive for edge devices, and limited network bandwidth is often the main bottleneck. Prior work often overcomes the constraints by condensing the models or messages into compact formats, e.g., by gradient compression or distillation. In contrast, we propose ProgFed, the first progressive training framework for efficient and effective federated learning. It inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. We theoretically prove that ProgFed converges at the same asymptotic rate as standard training on full models. Extensive results on a broad range of architectures, including CNNs (VGG, ResNet, ConvNets) and U-nets, and diverse tasks from simple classification to medical image segmentation show that our highly effective training approach saves up to $20\%$ computation and up to $63\%$ communication costs for converged models. As our approach is also complimentary to prior work on compression, we can achieve a wide range of trade-offs, showing reduced communication of up to $50\times$ at only $0.1\%$ loss in utility. 

Under review as a conference paper at ICLR 2022
PROGFED: EFFECTIVE, COMMUNICATION, AND COMPUTATION EFFICIENT FEDERATED LEARNING BY PROGRESSIVE TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data. However, training is resource- inttensvee for edge devices, and limited network bandwidth is often the main bottleneck. Prior work often overcomes the constraints by condens- ing the models or messages into compact formats, e.g., by gradient compression or distillation. In contrast, we propose ProgFed, the first progressive training frame- work for efficient and effective federated learning. inherently reduces computa- tion and two- way communication costs while maintaining the strong performance the final models. We theoretically prove that ProgFed converges at the same asymptotic rate standard training full models. Extensive results broad range of architectures, including CNNs (VGG, ResNet, ConvNets) and U-nets, and diverse tasks from simple classification medical image segmentation show that our highly effective training approach saves up to 20% computation and up to 63% communication costs for converged models. As approach is also compli- mentary to prior work on compression, we can achieve a wide range of trade-offs, showing reduced communication of up 50x at only 0.1% loss utility.
INTRODUCTION
Federated Learning (FL) (McMahan et al. 2017)) has led remarkable advances in the development of extremely large machine learning systems. Federated training methods allow multiple clients (edge devices) to jointly train a global model without sharing their private data with others. Despite the progress, training methods in FL still suffer from high communication and computational costs, as edge devices are often equipped with limited hardware resources and limited network bandwidth.
Prior literature has studied various compression techniques to address the computation and com- munication bottlenecks. We can divide these methods into three main categories (see also Table I) (i) Compression techniques that represent gradients (or parameters) with fewer bits to reduce com- munication costs. Prominent examples are quantization (Alistarh et al.) 2017 Lin al., 2018 Fu al. 2020) sparsification (Stich et al. 2018 Konečný al. 2016). (ii) Model pruning techniques that identify (much smaller) sub-networks within the original models to reduce computational cost at inference (Li & Wang 2019 Lin et al. 2020). And (iii) knowledge distillation (Hinton et al.) 2015)
techniques, that allow the server to distill the knowledge from the clients with hold-out datasets (Li
& Wang 2019 Lin al. 2020). However, these methods can sometimes degrade performance.
Progressive learning (Karras et al., 2018 is a well-known technique that has been used in image generation to stabilize the training. The primary idea is to first train the shallower layers on simpler tasks (e.g., images with lower resolution) and gradually grow the network to tackle more complicated tasks (e.g., images with higher resolution). By applying progressive training in federated learning setting, we can benefit from highly reduced resource demands (the computation and communication load is drastically reduced when training shallow models). By growing the models to their original size, we can recover (and often surpass) the accuracies reached by standard training. Moreover, we observe that progressive training facilitates the learning process and stabilizes training. Despite the appealing features, no previous study has systematically investigated exploiting progressive learning to reduce the costs in federated learning.

---

Under review as a conference paper at ICLR 2022
Table Comparison of ProgFed to other compression schemes.
Computation Reduction
Communication Reduction
Datasct
Technique
Efficiency
Message Compression
Model Pruning
Vonly for inference)
Model Distillation
ProgFed (Ours)
In this work, we propose ProgFed, the first progressive learning framework that reduces both com- munication and computation costs in FL. In our approach, we divide the model into several over- lapping partitions and introduce lightweight local monitoring heads to guide the training of the submodels. During training, the model capacity is gradually increased, until it reaches the full model of interest. This, of course, reduces the computation and communication costs, since the shallow sub models have much fewer parameters than the full model. We show theoretically that ProgFed training converges at the same asymptotic rate as standard training on full models. Due the nature of progressive learning, our method can reduce computational overheads and provides two-way communication savings (both from server-to-clien and client-to-seree directions). We demonstrate that our method can reach impressive performance (matching the baselines) using much quantization can be applied on top of progressive training. We experimentally show
fewer communication. ProgFed is compatible with classical compression- and sparsificatinn-wiie that combing these features enables a higher compression ratio and may motivate more advanced compression schemes based on progressive learning.
We summarize our main contributions as follows.
We propose ProgFed, the first progressive learning framework to reduce the training resource demands (computation and two-way communication). We theoretically prove that ProgFed con- verges at the same asymptotic rate standard training the full model.
We conduct extensive experiments on various datasets (CIFAR-I0/10) and architectures (VGG-16/19, ResNet- 18/152, ConvNets, 3D-Unet) to show that with the same number of epochs, our method can save around 25% computation cost, up to 32% two-way communication costs in federated classification, and 63% in federated segmentation without sac- rificing performance.
EMNIST and BraTS)
In addition, our method helps the models around 2x fewer communication costs in classifi- cation and 6.5x fewer costs in U-net segmentation to achieve practicable performance (2 98% of the best baseline). This is beneficial for combating limited training budgets in federated learning. Lastly, we show that our method complements classical compression and appears robust against compression errors. permits higher compression ratio and may motivate more advanced compression schemes based on progressive learning. With these combined techniques, we are able to reduce communication up to 50x only 0.1% loss utility.
2 RELATED WORK
In this section, we review work on cost reduction and leave progressive learning in Section D
Message compression. Fruitful literature has studied message compression (e.g. on gradients or model weights) to reduce the communication costs in distributed learning. The first category
focuses solely on client-to-server compression (Alistarh et al. 2017 Wen et al. 2017 Lin et al.,
2018 Bernstein et al., 2018 Stich et al., 2018 Konečný et 2016 Karimireddy
al. 2019 Fu
et al. Stich To name few, Konečny et al. (2016) reduce the costs by sending sparsified gradients and compressing them with probabilistic quantization. Alistarh et al. (2017); Wen et al. (2017) prove the convergence of probabilistic quantized SGD. SignSGD (Bernstein et al. 2018) significantly compresses the gradients with only one bit, while Karimireddy et al. (2019) fix the biased nature of SignSGD with error-feedback mechanism and generalize to other compression schemes. Compression on server-to-cliem communication has been shown non-trivias and attracted many recent focuses Yu et al. 2019 Tang al. 2019 Liu et al., 2020 Philippenko & Dieuleveut, 2020). Instead of dedicated designs, our method inherently reduces two-way communication costs and complements existing methods we will show in Section 4
2

---

Under review as a conference paper at ICLR 2022
G1
G
E
E
E
Eg
Es
B
Ey
G
Progressive Growing
Progressive Growing
(a) feed-forward networks
(b) U-nets (Symmetric).
Figure I: overview of ProgFed (a) feed-forwane networks and (h) U-nets (Symmetric). We progressively train a deep neural network from the shallower sub-models, e.g. e.g.M M consisting of main block Eg and head G1 2) gradually expanding to the full model M = M (Eq. 1t. Note that the local heads G, in feed-forward networks are only used for training sub-models and discarded when progressing to the next stage.
Model pruning, Model pruning removes redundant weights to address the resource con- straints (Mozer & Smolensky, 1989 LeCun et al., 1990 Frankle & Carbin 2019 Lin al. 2019). There are two categories: unstructured and structured pruning. Unstructured methods prune indi- vidual model weights according to certain criteria such as Hessian of the loss function (LeCun al.) 1990 Hassibi & Stork [993) and small magnitudes (Han et al.] (0015). However, these methods cannot fully accelerate without dedicated hardware since they often result in sparse weights. In contrast, structured pruning methods prune channels or even layers to alleviate the issue. They of- ten learn importance weights for different components and only keep relatively important ones (Liu al. 2017 Yu et al. 2018, Mohtashami et al. 2021 Li et al. 202T). ). Despite the efficiency, model pruning usually happens at inference and does not reduce training costs while our method achieves computation- and communication-efficee
even during training.
Model distillation. Another line of research for model compression is model distillation (Buciluă et al. 2006 Hinton et al. 20155), which requires a student model to mimic the behavior of teacher model (Polino et [ Sun al. 2019). Recent work has investigated transmitting logits rather than gradients (Li & Wang 2019 Lin et al.) 2020 He & Annavaram, 2020 Choquette-Chne
et al.) 2020), which significantly reduces communication costs. However, these methods either require additional query datasets (Li & Wang 2019; Lin et al. 2020) cannot enjoy the merit of datasets from different sources (Choquette-Cnne -hhettte-Choooe et al. 2020). In contrast, our work reduces the costs while
retaining the dataset efficiency.
PROGFED
Federated learning is distributed learning framework in which huge amount of clients train local models independently, and a central server aggregates the client updates to learn global model. Motivated by progressive learning, we propose ProgFed that progressively expands the network from a shallower one to the complete model. Our method is effective and efficient for multi-client training while we consider only one client in theoretical analysis for conciseness. The proposed model splitting and progressive growing are illustrated in Figure | and the federated optimization scheme is summarized Algorithm| [ We now present the proposed method in detail below.
3.1 PROPOSED METHIO
Model Structure. We now describe the proposed training method. For given machine learning model M. i.c. function M(-x): R= + æ² that maps n--dienssional input to & logits for pa- rameters (weights) Idd we assume that the network can be written as a composition of blocks (feature extractors) E; along with task head Gs. namely,
M:=
O E, = Gs Es Eg Eg 1=1
Note that the E, 's could denote e.g., ... stack of residual blocks simply single layer. The learning task solved by minimizing a loss function interest C E² + R (e.g., cross-entropy) that maps the predicted logits of M real value, i.e. minimization of f(x) = L M(x).
3

---

Under review as a conference paper at ICLR 2022
Progressive Model Splitting. To achieve progressive "rseressiveeeee learning, first divide the network M into S stages, denoted by M for s E {l......S S}} associated with the split indices. We additionally introduce local supervision heads for providing supervision signals. Formally, define
== Ö E.
where G. is a newly introduced head. Each head G., for s < S consists of only a pooling layer and a fully-conmnn cally-connected layer in our experiments for ford-forward networks. The motivation that simpler heads may encourage the feature extractors E, to learn more meaningful representations. Note that the sub-model M IR" + & produces the same output size as the desired model M; therefore, its corresponding loss (xxx == L a MM can be trained with the same loss criterion L as the full model.
Training of Progressive Models. We propose to train each sub-model MM' for T. iterations (a cer- tain fraction of the total training budget) and gradually grow the network from M to M M.. When growing the network from stage s to s + 1. we pass the corresponding parameters of the pretrained blocks E,, S 8, to the next model M and initialize its blocks Ey+1 and Gs+1 with random weights. Once the progressive training phase is completed, we continue training the full model M in an end-to-end manner for the remaining iterations. The length T. of each progressive training phase is a parameter that could be fine-tuned individually for each stage (depending on the application) for best performance. However, as practical guideline that we adopted for all exper- iments in this paper, we found that denoting roughly half of the total number of training iterations T to progressive training, and setting T. 23 for S. Ts- works well across all considered training tasks.
T($+1) such that T Σ9 Ts
23
Extension to U-nets. In addition to feed-forward networks (Figure aa), we show that our method can generalize to U-net (Figure feed-forward networks, the encoder sends both the final and intermediate features to the decoder. Therefore, we propose to grow the network from outer to inner layers as shown in Figure Tb) and retain the original output heads as G; We refer the strategy as the Symmetric strategy. In contrast, we propose another baseline, the Asymmetric strategy, which adopts the full encoder at the beginning and gradually grows until reaches the full decoder. For this strategy, we also adopt several temporal heads for earlier training stages. As we will show in Section 4.3, the Symmetric strategy significantly outperforms the Asymmetric strategy, which supports the notion of progressive learning.
U-net typically consists of an encoder and a decoder. Unlike
Practical considerations. We empirically observe that learning rate restart (Loshchilov & Hutter, 2017) clearly facilitates training in the centralized setting. This is because sub-models may overfit the local supervision while learning rate restart helps the sub-models with the newly added layers escape from the local minima and quickly converge to a better minima. On the other hand, warm- up (Goyal et al. 2017) for the new layers plays an important role in federated learning. Model weights often take a longer time to converge in federated learning, which makes the newly added layers introduce more noise to the previous layers. With warm-up, the new layers recover the per- formance without affecting previous layers. For instance, the existence of warm-up leads around 2% difference (53.23 vs. 51.09) on CIFAR-100 with ResNet-18.
3.2 THEORETICAL ANALYSIS
In this section we prove that progressing training converges to stationary point at the rate ( ie. with the same asymptotic rate as SGD training on the full network. For this, we extend the analysis from (Mohtashami et al. 2021) that analyzed the training of partial subnetworks. However, in our case the networks are not subnetworks, M of M=+1 (as the head is not shared), and we need to extend their analysis to progressive training with different heads.
Notation.
We denote by x* the parameters of sa s E (I,...,S} and abbreviate x = E Bd for convenience. For 8 S i S let æ'Ig. and (x'(xx) E. denote the projection of the parameter x' and gradient (xe) to the dimensions corresponding to the parameters of E, to E, (without parameter of Es+1 to E, and without head G.). In iteration the progressive training procedure updates the parameters of model f", min(S, 1+ 1). For convenience, we do not explicitly write the dependency of s on t below, and use the shorthand 2; to denote the corresponding model at timestep f. We further define x, such that XJJ.. 2/15, and x, XoJoş the complement.
4

---

Under review as a conference paper at ICLR 2022
Algorithm Progfeed-PPooresssie training Federated Learing setting
I: Input:
initialization xi. model .M(-, xo); iteration budgets T, T,, number of stages S, 1, desired number of local updates Output: parameters Xy and trained model M((xx))
learning rate
for 1..... +: then 4: if mS.
b switch from M to M after T, iterations > initialize new block Ex+1 and new head G copy parameters shared blocks E Ex.
5:
initialize parameter x randomly
x, f- X
(the old head G, is discarded)
end
9:
Sample subset clients
v standard Federated Learning on active model M
10:
for each active client E do initialize x x;
11:
send x; to active clients
12:
if warm-up up needed after growing then freeze x -, and warm-up x"
13:
warm up the newly added layers
14:
end if for j
15:
== l.....JJ do
4 Local SGD updates compute (mini-batch) gradient 9 client c's data
16:
x.i+11 x2. not
17:
end for
18:
D. Xx,J - xx11
19:
end for
20:
x = x + 1e 2011 A.
aggregate updates from the clients
21:
end for
Assumption (L-smoothness), The function f: IRd - is differentiable and there exists a con- stant L such that
V - v f < L | y
(3)
We assume that for every input x", we can query an unbiased stochastic gradient g"(x*) with [x*)( (x*)). We assume that the stochastic noise is bounded.
Assumption (bounded noise). There exist a parameter o² 2 0 such that for any E [l....,,]:
E [xlg V (x*)) < 0²
Vx
(4)
The progressive training updates x = x Ng" (x²) with a SGD update on the model xf. With the assumptions above, which are standard in the literature, prove the convergence of sub- models M as well as the model of interest M.
Theorem 1. Let Assumptions [ and hold, and let the stepsize in iteration be % = ouy with Fa )
}²}. Ore min {1,
}.
V f(xe) E. V [.)] }. Then it holds for any
7 min
[V/"(>)))
T-1 af (x(())) < after at most the following number of iterations T: -
(
0² + LFo.
o
(5)
Let := max q ==
then + - [Vx(2)) < after at most the
following iterations T: VX*(X;))
A0²
92
④ + LFo,
where Fg :- f(xo) (ming fx)).
Theorem shows the convergence of the full model M. The convergence is controlled by two fac- tors, the alignment factor a; and the norm discrepancy 9t- The former term measures the similarity between the corresponding parts of the gradients computed from the sub-models and the full model (note that a, = the last phase of training on f f). The latter term g measures the magnitude discrepancy (in Figure we display the evolution of ai during training for example task, note

---

Under review as a conference paper at ICLR 2022
ResNet-11
RosNet:13
VGG-16
VG5-19
80
611
60
Accracy (55)
40-
20
Erd-ooend
20
Er Erd44osnd
Erd-to-and
Erd4o-ind
20000
00000
G0000
200000
400000
10000 20000
30000
10000 20000 3000 40000
GFLDGS
Figure 2: Accuracy (%) vs. GFLOPs on CIFAR-100 in the centralized setting.
that = in the last phase of training). We would like to highlight that the convergence crite- rion in the first statement is lower bounded by the average gradient in the last phase of the training, 22 Ei=//2 V fx,) 2 I T E f (x()) || (this is due to our choice of the length of the phases, with Ts 2 7/2). This means, that progressive training will provably require at most twice many iterations but can reach the performance of SGD training on the full model with much cheaper per-iteration costs.
4 EXPERIMENTS
4.1
SETUP
We now describe the main implementation details and provide supplementary details in Section
We consider four datasets: CIFAR-10 (Krizhevsky et al. 2009 CIFAR-100 (Krizhevsky et al. 2009). EMNIST (Cohen et al. 2017). and BraTS (Menze et al. 2014 Bakas et al. 2017] The former three are for image classification, while the last one is medical image dataset for tumor segmentation. We conduct centralized experiments for analyzing the basic properties of our method while considering practical applications in federated settings. For the centralized settings, we train VGG-16. VGG-19 (Simonyan Zisserman 2014). ResNet-18, and ResNet- 152 (He et al. 2016) on CIFAR- 100 (100 clients, IID). For the federated settings, we train ConvNets on CIFAR- 10 and EMNIST (3400 clients, non-Ill)), ResNet- on CIFAR-100 (500 clients, non-IID), and 3D-Unet (Sheller et al. 2020) on the BraTTS dataset (10 clients, IID). Note that we follow Hsich et al. (2020) to replace batch normalization in ResNet-18 18 with group normalization.
Datasets, tasks, and models.
Implementation. We implement all settings with Pytorch Paszke et al. 2019). For the experiments in the centralized settings, we implement models based on Vries & Taylor (2017), where
all experiments for 200 epochs and decay the learning rates in [60, /20, 160) epochs by a factor of 0.1. We additionally adopt warm-up (Goyal et al.. 2017) and learning rate restart (Loshchilov) & (outtrr[[11] in our method to better fit in progressive learning. For federated classification, we follow federated learning benchmarks in (McMahan al.) 2017 Reddi al. (021] implement CIFAR-10, CIFAR-100, and EMNIST, respectively. For federated tumor segmentation, we follow for
3 for EMNIST and all the other datasets and T, as the practical guideline described in Section 3 We adopt 5 and 25 warm-up epochs for federated EMNIST and federated CIFAR-100, respectively. We summarize the
Sheller et al. (2020) for the settings and data splits. We set
parameters (and number of clients) in Table
4.2 COMPUTATION EFFICIENCY
We first analyze the computation efficiency of our method in the centralized setting (where all data is available on single device) to study the effect of the progressive training in isolation, before moving the federated use cases. We average the outcomes over three random seeds and we consider four architectures on CIFAR-100, including VGG-16, VGG-19, ResNet-18, 18, and ResNet- 152. As shown in Table 2 our method performs comparably to the baselines (that train on the full model) after 200 epochs while consuming fewer floating-pont point operations per second (FLOPs) and training wall-clock time.

---

Under review as a conference paper at ICLR 2022
Reehart:33 Visis
Thepss Pettmerte (M)
200.2
C
Drats 1yymeetrid
09.3
Brats Heymeettr
00.2
Gk
Cost Roduction
Cost Production Puctor
Figure 3: Computation cost reduction {98%. 99%. Figure 4: Communication cost reduction {98%, 99.95%, best} compared the baseline (training full models) performance in the centralized setting.
99%, 99.95%, best) compared to the baseline perfor- mance in the federated setting.
EMMST
CIFAR-10
CIPAR-100
Bra
10
62-
42-
30
15
Baseline
22
Daseline
20
Beseline
Deseline
Asymmetric Symmetric
Ours
Ours
Ours
10
2003
Con mww.iaaiiiiioccoommmmmmmmm 16+1
Figure 5: Communication Accuracy (%) in federated settings on EMNIST (3400 clients, non-IID), CIFAR-10 (100 clients, IID), CIFAR-100 (500 clients, non-III), and BraTSS (10 clients, ITD).
To analyze the efficiency, we report the performance when consuming different levels of costs. Figure showws that our method (orange lines) consistently lies above end-to-end training on the full model (blue lines), meaning that our method consumes fewer computation resources to improve the models. Moreover, we visualize (98%, 99%, 99.95%. best) of the performance of the converged baseline (analysis with larger range presented Figure 12p. TigureBiinicatt improves computation efficiency across architectures. In the best case, our method can accelerate training up to 7x faster when considering limited computation budgets. We also observe that VGG models improve more than ResNets. A possible reason might be that due to local supervision, sub-models enjoy larger gradients compared to end-to-end training, while it rarely benefits ResNets since skip-connections could partially avoid the problem.
that our method
4.3 COMMUNICATION EFFICIENCY
We experiment in the federated setting to verify the communi- cation efficiency of our method. In particular,
Table 2: Results on CIFAR-100 in the centralized setting.
Accuracy
Reduction
consider classification tasks on three datasets, EMNIST, CIFAR-IO, and CIFAR-100, and tumor seg- mentation tasks on the BraTS -13.02%
End-to-end
Ours
Walltime
FLOPs
ResNet-18
76.08 1 0.12
75.84 + 0.28
-24.75%
-1.60%
ResNet-152
77.77 1 0.38
78.57 1 0.33
-22.75%
-19.68%
dataset. We follow the standard Sec- tion [. to train the models and average the results over three random seeds. Results Table indicate that our method achieves comparable results on EMNIST and outperforms the baselines on all the other datasets. In addition, our method saves 20% to 30% two-way communication costs in classification and up to 63% costs in segmentation. The result simultancously confirms the effectiveness and efficiency of our method.
VGG16 (bn)
71.79 0.15
71.54 0.45
-14.57%
VGG19 (bn)
70.81 + 1.18
70.90 + 0.43
-22.10%
-14.43%
protocol as described
We compare the performance different communication costs in Figure 5 We observe that our method is cattuuuiiaaiioeffffiieee over every cost budget, especially when the model parameters are not evenly distributed across sub-models. For instance, 3D-Unet has most of its parameters in the middle part of the model, making our Symmetric update strategy extremely efficient. On the other hand, the Asymmetric strategy shows marginal improvement since it starts from the heaviest portion of the model. The finding aligns with the motivation of progressive learning: learning from simpler models might facilitate training. More analysis on segmentation is in Section C.4
7

---

Under review as a conference paper at ICLR 2022
Table Federated ResNet- on CIFAR-IOK with compression. LQ-X denotes linear quantization followed by used bits representing gradients, and SP-X denotes sparsification followed by the percentage of kept gradients (See Table TO for standard deviations).
LQ-8 +SP-25
LQ-8
Float
LQ-8
LQ-4
LQ-2
SP-25
SP-10
+SP-10
Accuracy
Baseline Ours
52.54 53.23
49.40
49.55
47.26
51.23
51.79
49.67
50.25
53.07
52.32
52.87
52.00
51.86
52.19
52.24
Compression Ratio (%)
Baseline
100
25.00
12.50
6.25
25.00
10.00
6.25
2.50
Ours
77.10
19.28
9.64
4.82
19.28
7.71
4.82
1.93
mewiie faraans (84)
100-
Dans
DunsLo0
noaace Son aarraaoo (3)
100
Dunsggz
Dunu5110
50-
Durs-L04
50-
DursLQQ-55735
Durs-SF35
Durs-Q0-55 5710
BevelitaeL00
Dase Iniii---P53 5P25
Besetite 5575
60-
Basetine 5
30-
30
50
150
Seduction Furtor
Cord Reduction Fertor
(a) Moderate compression
(b) Intensive compression
Figure 6: Relative performance vs. communication cost reduction with federated ResNet-18 CIFAR-100
with (a) modest compression and (b) intensive compression.
Lastly, we analyze the cost reduction when achieving (98%, 99%, 99.5%, best} of the performance of the converged baseline. This experiment studies the behavior of method when only granted limited budgets. Figure 4 (and Figure [3] in the appendix that displays larger range) presents that except for the Asymmetric strategy, our method improves communi- cation efficiency across all datasets. In particular, achieves practicable perfor- mance with 2x fewer costs in classifica- and up to 6.5x fewer costs in tumor segmentation. We also observe that the communication efficiency improves more when consid- cring lower budgets. This property benefits when the time and communication budgets are lim- ited (McMahan et al. 2017).
Results in federated settings. We report accuracy (%) classification and Dice scores (%) for segmentation, followed by cost reduction (CR) as compared to the base-
Table 3:
lines.
Baseline
Ours
CR
EMNIST
85.75 0.11
85.67 0.06
-29.49%
CIFAR-10
84.67 = 0.14
84.85 0.30
-29.70%
CIFAR-100
52.54 + 0.44
53.23 + 0.09
-22.90%
BraTS (Aym.)
86.77 = 0.45
87.66
0.49
-5.02%
BraTS (Sym.)
86.77 = 0.45
87.96 + 0.03
-63.60 %
4.4 COMPATIBILITY
We show that our method complements classical compression techniques including quantization and sparsification. We train several ResNet-18 on CIFAR-100 in the federated setting and apply linear quantization and sparsification following Konečný et al. 2016). Specifically, we consider bits, bits, and bits for quantization (denoted by LQ-X), 25% and 10% for sparsification (denoted by SP-X), and their combinations. Table 4 demonstrates the results of our method and the baseline equipped with various compression techniques. Our method clearly outperforms the baselines in all settings. It indicates that our method is more robust against compression errors, compatible with classical techniques, and thus permits higher compression ratio.
In addition, we visualize (50%, 80%, 90%, 99.5%, best) relative performance against com- munication cost reduction in Figure 6 We observe that method is more efficient across all percentages in every pair (Ours vs. Baseline, plotted the same color). Besides, the baseline fails to achieve comparable performance in many settings, e.g., the ones with quantization, while
8

---

Under review as a conference paper at ICLR 2022
Table Comparison between update strategies on CIFAR-100 with ResNet-18,
Baseline
Ours
Layerwise
Partial
Mixed
Random
Acc (%) Ratio
76.08
-0112
75.84-10.28
72.40-10.16
74.70.1..04
75.04.11.26
74.38
1
0.86
R
0.88
method retains comparable performance even with high compression ratios. Interestingly, even with additional compression, our method still facilitates learning at earlier stages. For example, Ours- LQ8+SP25 achieves comparable performance around 50x faster than the baseline, 60x faster to achieve 80%, and more than 200x faster to achieve 50% of performance. Overall, these properties grant our method to adequately approach limited network bandwidth and open up the possibility of more advanced compression schemes.
4.5 ANALYSIS OF PROGFED
Effeet of norm discrepancy. As discussed in Section 3.2 the convergence rate of the full model controlled by norm discrepancy, namely 9. As qe approaches 1, the convergence rate will be closer to the convergence speed of the sub-models. We empirically evaluate the norm discrepancy on CIFAR-100 100 with R8 in the centralized setting. Figure shows that the norm discrepancy decreases as the sub-models gradually recover the full model. It suggests that spending too much time on carlier stages may hurt the convergence speed while it offers a higher compression ratio. This outlines the trade-off between communication and training efficiency.
Comparison between update strategies. As described in Section 3 ProgFed progressively trains the network from the shallowest sub-model M to the full model M. We verify our update strategy by comparing it with var- ious baselines in the centralized setting. end-to-end training; Layerwise se only updates the latest layer E, while still passing the input through the whole model M; Partial is similar to our method but ac- quires supervision signals from the last head Gs; Mi xed combines Partia and Ours, trained on supervision from both G, and Gş; Random randomly chooses a sub-model M update, rather than follows progressive learning.
Norm discrepancy
Stages
Paselines
2-
100
125
203
Figure 7: Norm discrepancy.
Table 5 presents the performance and the computation cost ratio. We make several crucial observations. (I) Ours and Random do not pass the input through the whole network, making them consume fewer computation costs. (2) Layerwiss greedily trains the network but achieves the worst performance, which highlights the importance of end-to-end fine-tuning. (3) Ours outperforms Random in both costs and accuracy, verifying the necessity progressive learning. We also note that our method does not require additional memory space (compared to Random) and is easy to implement.
5 CONCLUSION
Beyond prior work on expressing models in compact formats, we show a novel approach to modify- ing the training pipeline to reduce the training costs. We propose ProgFed and show that progressive learning can be seamlessly applied to federated learning for communication and computation cost reduction. Extensive results on different architectures from small CNNs to U-Net and different tasks from simple classification to medical image segmentation show that our method communication- and computation-efficii that a progressive learning scheme has even led to improved performance in vanilla learning and more robust results when learning is perturbed e.g. in the case of gradient compression, which high- lights progressive learning as a promising technique in itself with future application domains such learning and advanced compression schemes.
especially when the training budgets are limited. Interestingly, we found

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have provided all assumptions in the main paper and attached the proof of Theorem | in the appendix. All implementation details and datasets have been discussed in the main paper and the appendix. We attached the source code as a supplementary material and will publicly release the source code upon acceptance.
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficii sgd via gradient quantization and encoding. Advances Nearul Information Processing Systems (NeurIPS), 30:1709-1720, 2017.
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 4(1):1-13, 2017.
Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Rus- sell Takeshi Shinohara, Christoph Berger, Sung Min Martin Rozycki, et al. Identifying the machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv: 1811 1.02629, 2018.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In Pro- ceedings of the International Conference on Machine Learning (ICMI.,, pp. 736-745. PMLR, 2020.
Jeremy Bersstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Com- pressed optimisation for non-convex problems. Proceedings of the International Conference on Machine Learning (ICML). pp. 560-569. PMLR, 2018.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil, Model compression. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541, 2006.
Christopher A Choquette-Choo Natalic Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, and Xiao Wang. Cape learning: Confidential and private collaborative learning. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist hand- written letters. 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921-2926. IEEF, 2017.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:/7088 001.7708..44 04552, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2019. URL https:
//openreview.. net.
Fangcheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang. Yingxia Shao, Zhang, and Bin Cui. Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In Proceedings of the International Conference on Machine Learning (ICML),
330443314. PMLR, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis. Lukasz Wesolowski. Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (706.0277,, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
Chaoyang He and Murali Annavaram. Group knowledge transfer: Federated learning of large cnns at the edge.
Advances in Neural Information Processing Systems (NeurIPS), (33), 2020.
Chaoyang He, Shen Li, Mahdi Soltanolkotahi, and Salman Avestimehr. Pipetransformer: Automated elastic pipelining for distributed training of transformers. arXiv preprint a:Xiv:2102.03166, 2021.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Geoffryy Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint WrXiv:153300253/ 2015.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 4387- 4398. PMLR, 2020.
Sai Praneeth Karimireddy, Quentin Rehjock, Schastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In Proceedings of the International Conference on Machine Learn- ing (ICML), pp.
PMLR. 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Jakuh Konečný, Brendan McMahan, Felix Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint orXiv: 161000594, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS). pp. 598 605, 1990.
Changlin Li, Guangrun Wang. Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8607-8617, 2021.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint 1011:3910.03381, 2019.
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement hierarchical representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In Proceedings of the International Conference Learning Representations (ICLR). 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated lcarning. In Advances Neural Information Processing Systems (NeurIPS), 2020.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the com- munication bandwidth for distributed training. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Xiaorui Liu, Yao Li, Jiliang Tang, and Ming Yan. A double residual compression algorithm for efficient distributed learning. In Proceedings the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 333-143. PMLR, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2736-2744, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. cttpuuuniiiaiioffffffiiiee learning of deep networks from decentralized data. In Proceedings of the Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1273-1282. PMLR, 2017.
Bjoem H Menze, Andras Jakab, Stefan Bauer. Jayashree Kalpathy- Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, al. The multimodal brain tumor image segmentation benchmark (brats). IEFE transactions on medical imaging, 341104:19332004, 2014.

---

Under review as a conference paper at ICLR 2022
Amirkeivan Mohtashami, Martin Jaggi, and Sebastian U Stich. Simultaneous training of partially masked neural networks, arXiv preprint
2021.
Michael C Mozer and Paul Smolensky. Skcletonization: A technique for trimming the fat from network relevance assessment. In Advances in Neural Information Processing Systems (NeurlPS), pp. 107-115, 1989.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, -eeh-perfrrance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. pp. 8026 8037. Curran Associates, Inc., 2019.
Constantin Philippenko and Aymeric Dieuleveut. Bidirectional compression in heterogeneous settings for distributed federated learning with partial participation: tight convergence guarantees. arXiv preprint arXiv:2006. 14591, 2020.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Sashank Reddi, Zachary Charles, Manzil Zaheer. Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Micah Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Sebastian U. Stich. On communication compression for distributed optimization on heterogeneous data, arXiv preprint (tt::20000003885 2020.
Scbastian Stich, Jean-Baptins Cordonnier, and Martin Jaggi. Sparsified SGD with memory. Advances in Neural Information Processing Systems (NeurIPS), pp. 4452-4463. 2018.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 4323-4332, 2019.
Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang. and Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass ertor-compensated compression. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine- Sorkine-Homunng, Olga Sorkine- and Christopher Schroers. A fully progressive approach to single-image resouutino In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 864 873, 2018.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terggrad: ternary gra- dients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Rongliang Wu, Gongjie Zhang, Shijian Lu,, and Tao Chen. Cascade ef-gan: Progressive facial expression edit- ing with local focuses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5021-5030, 2020.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang, Slimmable neural networks. In Proceed- ings of the International Conference on Learning Representations (ICLR), 2018
Yue Yu, Jaxxiag Wu, and Longbo Huang. Double quantization for coammunicaaiionfffccieee distributed opti- mization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF OF THEOREM 1
In this section we prove Theorem ! The proof builds on (Mohtashami et al. 2021 that considered training of subnetworks, but not the progressive learning case.
Lemma 2. Let x, denote the weights of the full model, and x; the weights of the model that is active in iteration Note that it holds E, = x E, as per the definition in the main text. It holds,
22 o²
F.f(+++1) S [xx,) of [V}* 1|2
Let's abbreviate gi = (x;)). By the update equation x x ng it holds X E - 788jE. With the L-smoothness assumption and the definition of dzz.
Proof.
L E
(x(++++)) S [(xx) 7o (V)()))) L,) +
Bi
7:21
Bi|E, EgiF 2 + Eggq F. 13
= -((x)) - 7th
Eigi +
E(
2
s s(()) TO / E ( +
E(Igg" - -ggil1 | (x() 1.. 1=.1
I.
72 0²
$ f(x)) 7 (Vf(x)) R., (x.))
F.. 11 (x(((x)))
$ fx(x) 1p
"t L) V 15,1 2
22 0² 2
20g
V V (x)) F. 112 LاL 0²
$ f(x,)
2 t
2
S [(x)) - 2
| (x,) |.. 1|2
7²L
2
Where in the last equation we used the facts that an, 1 and 7/ 027y
We now prove Theorem
Proof. We first define F, == Ex((,,) - (min f(x)). By rearranging Lemma we have
Fo ( 1|
P. Fr 7 0²
Next, with telescoping summation, we have
2(Fo FT-1)
2Fo
Eag [V
+ 7Lo <
Ty + Lo
(9)
Ty
We now can prove the first of part Theorem by setting the step size to be O(min{ 1 T, in (Mohtashami et al. 202T.
(,FT)
}
To prove the convergence the model of interest (the second part),
V
1|
T
(10)
1. 1| 5 T
of VV/
0-
where
9: 44
and
q max Ct-
(11)
Vf*(x²))
[,
(E[T]
13

---

Under review as a conference paper at ICLR 2022
Table 6: Parameters for federated experiments
Dataset
#clients
#clients per spoch
batch size
#epochs
EMNIST
3400
68
20
1500
CIFAR-10
100
50
2000
CIFAR-100
500
40
20
3000
BraTS
10
10
100
#epoch_pec per per_ecient
#stages (S)
T.
#epochs htctsssffrwwannnnnn warmup
EMNIST
1
3
250
5
CIFAR-10
5
250
CIFAR-100
1
4
375
25
BraTS
3
25
By definition y 2 qr for all [T], reach the last inequality and combine it with the first part of the theorem.
T af [
(12)
q²
Using for as the new threshold, we immediately prove the second part.
B IMPLEMENTATION DETAILS
We describe details of the datasets used in Section and present the hyper- parame-prrmeters in Table
CIFAR-10. We conduct experiments CIFAR-10 datasets for federated learning, following the setup previous work (McMahan et al. 2017). The dataset is divided into 100 clients randomly, namely iid distributions for every client. We adopt the same CNN architecture with 122,570 param- eters.
1-0.--00. We follow the federated learning benchmark of CIFAR-100 proposed in (Reddi al. 202T) conduct the experiments CIFAR-100 We use ResNet--18/152 (batch norm replaced with group norm (Hsich et al.) (0200) and VGG-16/-19 in the centralized setting, while only considering ResNet-18 18 in the federated experiments. This setup allows to evaluate the federated learning systems on non-lID distributions, where we use the splits as suggested in (Reddi et al., 2021
EMNIST. We follow the benchmark setting in (Reddi et al. 2021) experiment. There are 3,400 clients and 671,585 training examples distributed in a non-iid fashion. The models are eventually evaluated on 77,483 examples, resulting in a challenging task.
BraTS. addition to image classification, we conduct experiments on brain tumor segmentation based (Sheller et al., 2020). We train 3D-Unet the BraTS2018 dataset, which includes 285 MRI scans annotated by five classes of labels. The network has 9,451,567 parameters. The training is randomly partitioned into clients. All clients participate every training round and locally train their models for three local epochs. This setting matches the practical medical applications. Institutions often own relatively stable network conditions, and the data are rare and of high resolution.
Architectures. ConvNets for EMNIST and MNIST consist of two convolution layers, termed Convl and Conv2, followed by two fully connected layers, termed FCI and FC2. To apply pro- gressive learning with S = 3, we set Convl, Conv2, FC1 to be the three stages, namely E, and FC2 to the final head, namely Gs. As for VGGs, we divide the whole networks into five compo- nents according to the max-pooling layers. We combine the first two be E₁ and set the others to be the remaining E, under the setting S 4. To apply ProgFed to ResNets He et al. (2016) we first replace the batch normalization layers with group normalization. By convention, ResNets have
14

---

Under review as a conference paper at ICLR 2022
60
60
Accursior 1%5
Baseline
Baseline
Partial
Portial
Layerwise
layerwise
Mixed
Mixed
Ours
0-
20000
40000
50000 GFiops
80000
100000
50
100
200
Epochs
Figure 8: Performance vs. computation costs and Performance vs. epochs when comparing our method to different updating strategics.
00
Mocuracy (35
00
End-tt-end Curs
End-to-end
Erd- fnd--to-ee and
Endito-ard
Ours
Ours
Ours
50
100 tpochs
150
200
50
300
150
200
50
100 Fpachs
150
230
0
50
300
150
200
Eperhs
Ppochs
Figure Accuracy (%) vs. Epochs on CIFAR-100 centralized setting.
five convolution components, i.e. Convl, Conv2.x, Conv3_x, Conv4_x, and Conv5_x. We combine Convl I and Conv2_x to be Ei and all the other components to be the remaining E;. It thus matches our setting.
C MORE RESULTS
C.I COMPARISON BETWEEN UPDATE STRATEGIES
As described in Section 4.5 we compare ProgFed other baselines. We additionally report the performance vs. computation costs and performance vs. epochs in Figure where Ours reaches comparable performance while consuming the least cost.
C.2 COMPUTATION EFFICIENCY
We present more experiments in the centralized setting to prove the computation efficiency of our method. Figure presents accuracy vs. epochs with four architectures on CIFAR-100. The result
0.9
0.8
DICE (8)
0.7
0.5
Baseline
0.5
Asymmetric
Symmetric
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Computation cost (GFLOPs)
1e6
Figure 10: DICE (%) vs. computation costs BraTS.
15

---

Under review as a conference paper at ICLR 2022
1.3
0.5
Figure 11: Computation accelcration at different percentage of performance. The orange har indicates the best performance of our method.
200
ResNet18 ResNett
100
VSS116
Relitve Peformencess.
90-
VGG19
Reldhhve Performancess
ENNITT CIFAR-10
BraTS (8ymmetric)
BroTT (Asymmabis kcl
100
Figure 12:
Computation cost reduction at (50%, Figure 13: Communication cost reduction at (50%, 60%. 70%, 80%, 90% 98%, 99%, 99, 99.95%. best) of 60%. 70%, 80%, 90% 98%, 99%, 99.95%. best) of the baseline performance the centralized setting.
baseline performance the federated setting.
indicates that our method converges comparably faster to end-to-end training in practice. Figure T1 presents Figure 3 in bar charts. Similar to Figure 3 our method improves across architectures while VGGs benefit even more from our method. Figure I0 presents the computation costs of 3D-Unets the BraTS dataset. We make the first observation that tumor segmentation requires heavy computation. Interestingly, even though the earlier stages of Symmetric consume much fewer communication costs (Figure 5). they require more computation costs than Asymmetric.. It might root from the higher resolution of feature maps that Symmetric keeps and thus lead to trade-off' between communication and computation costs.
Figure extend Figure larger range (50%, 60%, 70%. 80%. 90% 98%, 99%, 99.95%. best). The result shows that our method benefits across models and is especially efficient when training budgets are limited.
C.3 COMMUNICATION EFFICIENCY
We present more experiments in the federated setting to prove the communication efficiency of our method. To complement Figure we additionally visualize performance communication costs and performance vs. epochs in Figure 14, Although our method causes performance fluctuation in some datasets, the performance recovers very quickly. Figure 15 presents Figure 4 in bar charts. The results show that our method saves considerable costs in almost all settings except for EMNIST 95%. It is because both baseline and our method improve fast the beginning while our method stands out in the latter phase of training (e.g. after 98%). The result also supports that our method improves across datasets.
We present Figure 4 in a region that models are applicable. Here, we plot the figure with a larger range (50%, 60%, 70%, 80%, 90% 98%. 99%, 99.95%, best) in Figure [3] The result is consistent, showing that our method benefits across datasets, and is efficient when granted limited training budgets.
16

---

Under review as a conference paper at ICLR 2022
Bazelinc
39
Gaseline
Baseline
Communaattion 1604
Commur oodee 2001
Coech
(a) NenvNet on EMNIST
(b) ConvNet on CIFAR-10
Baseline
Ours
00
Baseline
Asymmettic
Asymmmtticc
Symmeticc
Symmetric
1220
2003
2000
5000
265
1205
1003
2000
2308
300
1202
2003
(c) ResNet- 18 CIFAR-100
(d) 3D-Unet BraTS
Figure 14: Accuracy vs. computation costs and accuracy vs. epochs in the federated setting. (a)()Cc) shows the result for three classification tasks; (d) shows the result for the segmentation task, where two update strategies Symmetric and Asymmetric adopted for 3neUnee.
TNNNST
EIPA-10
CHAR-136
FrATS (Symmetrr)
7
15
13
60%
69%
09 5%
59.9%
38%
90%
5%5%
100.2%
65%
43%
60%
09.
5/6
103.7%
65%
43%
96%
99.5%
100.%
Figure 15: Communication cost reduction at different percentage of performance. The orange bar indicates the best performance of our method.
C.4 VISUALIZATION FEDERATED SEGMENTATION
We visualize the outputs of 3D-Unet (see Section 4.3) in Figure 16 The result matches the number reported in Table 3 that the models perform similarly after they have converged. However, the com- munication cost consumption is disparate. Symmetric only consumes 36.40% while the others either marginally improve it. We visualize the results in Figure 17 when granted
do not any limited communication budgets. Note that even with the same cost, Symmetric and Asymmetric may stay different stages because Symmetric starts from the outer part, consisting of significantly fewer parameters. We find that Baseline and Asymmetric fail to compress the models with 0.18% since their models are of size 34 MB (i.e. 0.908%). while only Symmetric achieve Interestingly. Sym- metric has produced promising results at Q) 18%8 of costs. It suggests that our method significantly facilitates learning even given limited communication budgets. Meanwhile, we hope these findings could inspire more further work medical learning problems.
Table 7: Raw results CIFAR-100 with four architectures in the centralized setting (to complement Table 2
End-to-End
ProgFed (Ours)
Seed1
Seed2
Seed3
Mean
Std
Seed1
Seed2
Seed3
Mean
Std
ResNet-15
75.95
76.12
76.17
76.08
0.12
75.57
76.13
75.83
75.84
0.28
ResNet- 152
77.69
77.44
78.19
77.77
0.38
78.95
78.46
78.31
78.57
0.33
VGG16 (bn)
71.94
71.77
71.65
71.79
0.15
71.36
72.05
71.21
71.54
0.45
VGG19 (bn)
69.47
71.25
71.70
70.81
1.18
71.30
70.45
70.95
70.90
0.43
17

---

Under review as a conference paper at ICLR 2022
Table Raw results in federated settings on EMNIST. CIFAR-IO, and CIFAR-100 (to complement Table 3
EMNIST
Seed1
Seed2
Seed3
Mean
Std
Baseline Ours
85.63
85.77
85.85
85.75
0.11
85.65
85.62
85.73
85.67
0.06
CIFAR-10
Seed
Seed2
Seed3
Mean
Std
Baseline Ours
84.62
84.82
84.56
84.67
0.14
84.64
84.73
85.19
84.85
0.30
CIFAR-100
Seed1 51.79
Seed2
Seed3
Mean
Std
Baseline Ours
51.86
52.58
52.08
0.44
53.33
53.16
53.21
53.23
0.09
Table 9: Raw results on BraTS in the federated setting (to complement Table
Seed
Seed 2
Seed 3
Mean
Std
Baseline
87.29
86.51
86.51
86.77
0.45
Ideal (Ours)
88.19
87.22
87.57
87.66
0.49
Idea2 (Ours)
87.92
87.98
87.98
87.96
0.03
Table 10: Federated ResNet-18 on CIFAR-100 with compression. LQ-X denotes linear quantization followed by used bits representing gradients, and SP-X denotes sparsification followed by the percentage of kept gradi- ents. complement Table
Float
LQ-8
LQ-4
LQ-2
Accuracy (%)
Baseline
52.54 + 0.44
49.40 + 0.75
49.55 + 0.59
47.26 + 0.29
Ours
53.23 + 0.09
53.07 + 1.00
52.32 + 0.15
52.87
0.54
Compression ratio (%)
Baseline Ours
100
25.00
12.50
6.25
77.10
19.28
9.64
4.82
LQ-8
LQ-8
SP-25
SP-10
+SP-25
+SP-10
Accuracy (%)
Baseline
51.23 + 0.56
51.79 + 0.10
49.67 1.58
50.25
1.03
Ours
52.00 + 0.19
51.86 + 0.23
52.19 + 0.03
52.24 + 0.12
Compression ratio (%)
Baseline Ours
25.00 19.28
10.00
6.25
2.50
7.71
4.82
1.93
18

---

Under review as a conference paper at ICLR 2022
Table 11: Results with FedAvg and FedProx EMNIST and CIFAR-100)
EMNIST
CIFAR-100
End-to-end-FedAAg
85.75
52.54
ProgFed+FodAAg
85.67
53.36
End-to-end+FeeProx
86.36
53,25
ProgFed++FedPox 86.03
52.30
C.5 RAW NUMBERS AND MORE STATISTICS
Table g
and present the numbers of the experiments in Table 2 and over three random
seeds. Table 10 presents the standard deviations of Table over three random seeds.
C.6 GENERALIZABILITY OF PROGFED BEYOND FEDAVG
We show that ProgFed can generalize to other federated optimizations beyond FedAvg. In this section show that ProgFed can generalize advanced optimizations. We combine our method with FedProx and conduct experiments on EMNIST and CIFAR-100 with ConvNets and ResNet- 18, respectively. Table [T] shows that both end-to-end training and ProgFed improves over FedAvg on EMNIST when applying FedProx. On the other, our method with FedProx shows little improvement on CIFAR- 100 while end-to-end training benefits from FedProx (52.54 vs. 53.25). However, despite the improvement from FedProx, it remains comparable to ProgFed with FedAvg.
D MORE RELATED WORK
We discuss more related work in this section.
Progressive Learning. The core idea of progressive learning (PL) train the model from easier tasks (e.g.. low-resolution outputs or shallower models) to difficult but desired tasks (e.g., high- resolution outputs or deeper models).
was originally proposed to stabilize the training process and has been widely considered in vision tasks such image synthesis (Karras et al. 2018), image super-resolution (Wang et al. 2018), facial attribute editing (Wu et al. (2200) and representation learning (Li et al. (019). One of the most representative methods PGGAN (Karras et al. 2018).
which trains GANs first for generating 4x4 images with a shallow network and progressively extends the network for generating images of size 1024x1024 In addition stabilizing the training process, PL naturally reduces the computation demands since it typically employs smaller models before reaches the full model. This even benefits federated learning since the message size also decreases. Despite the benefits, little work has investigated applying PL to federated learning. We note that although some works (He et al.) 202T]] Belilovsky al.) 2020) also consider partially training, they often seek solutions to fixing layers rather than considering training difficulty, and the application in general federated tasks remain unexplored.
19

---

Under review as a conference paper at ICLR 2022
Rasseligg - (1009)
Syymmpes) (94.98%))
Symmetrio (36.40%)
Iaput -
Ground Truth
144
Figure 16: Visualization of federated segmentation. From left to right: Imput Ground Truth, Baseline, Asym- metrie, and Symmetric updating strategies. Despite the comparable performance, Symmetric consumes signifi- cantly fower communication costs.
pertoni
taput (i.e. 0.908%).
Ground Truth Truth
(b) R 0.18%
~ 9.40%
(d) 40%
Figure 17: Segmentation results under { 0.18% ~ 9.40% ~ 36.10% of communication costs of the converged haseline. From top bottom: baseline, Asymmetric (Ours), and Symmetric (Ours). Only Symmetric achieve 0.18% 7536 MB) compression ratio, since the size of the other models is already around 34 MB
20


Reviews:
[{"review_id": "uAk2eFfnqpM", "reviewer": "Reviewer_1N8v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is an interesting work on accelerating the training process of federated learning.  The reviewer still has several concerns about the current submission:\n(1) It seems that the main benefit of progressive training for federated learning comes from the early stage during training.  Hence, we recommend the authors do more ablation studies on the hyperparameter $S$. \n(2) For different stages,  how do the authors set the hyperparameters, such as learning rate.  Since in different stages, the depth of subnetworks is differents,  the hyperparameter tuning techniques may highly influence the training speed and final performance.\n(3) Theorem 1 provides the convergence of the proposed ProgFed.  However, the convergence results do not indicate the benefit of progressive training for federated learning. \n(4)  The experiments should be strengthened.  The main optimizer in ProgFed is fedavg.  The authors should incorporate the proposed ProgFed with several other types of local opitmizers, such as fedprox, scaffold, etc., to further show progressive training can be a versatile and strong technique for accelerating federated learning. ", "summary_of_the_paper": "The authors adopt the progressive training technique to accelerate the training process federated learning. The authors also establish the convergence rate of the proposed algorithm. Extensive experiments demonstrate the efficacy of the proposed algorithm.  ", "summary_of_the_review": "see the comment above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E70sEHFDGtL", "reviewer": "Reviewer_3Huv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Novelty & Significance:\n\n- The idea of progressive training is not new, and the authors cite in particular Karras et al 2018. Given the importance of the topic in the paper, I would have appreciated a paragraph devoted to it in the related work section.\n- Theoretical results follow a previous proof structure but seem to bring some novelty. Their significance is not that big insofar as the paper is concerned with deep learning, for which convergence rates do not play a huge role (in particular, the learning rate schedule is chosen independently of the rate derived). Further, the final result is that the convergence is at worst twice slower than the baseline, which is a simple consequence of the choice of spending at least half of the training budget on the full network.\n- The experimental results show that the idea can bring some computational and communication savings, in a variety of contexts (cross-device/silo), and is compatible with other approaches.\n    - Re computational gains, the benefits seem highly dependent on the architecture (cf fig 3), which reduces its significance.\n    - The proposed approach seems only to bring very modest gains with respect to a « random » approach where one would train random sub-networks (cf Table 5). This reduces the significance of the results.\n\n# Writing & Clarity:\n- The paper is well written and the main idea is way conveyed.\n- However, notations could be simplified (e.g. page 4 the notation $x_{t, E_s^c}$ is introduced but never used) to facilitate understanding\n- I would suggest using a log range for the compression factor axis on figs 6a and 6b.\n\nHere are a few minor typos:\n- Page 4, paragraph training of progressive model, there is a typo for $T_S$, I think the correct value should be $T (S+1) / 2S$\n- Circular notations for the theoretical analysis (end of page 4) for $s = \\min(S, t/T_s)$\n- End of sec 4.5, missing « is » easy to implement\n- Proof of lemma 2, page 13, the first update equation is probably missing a $\\gamma_t$ before $g_t^s$ \n\n# Quality & Correctness:\n\n## Theoretical results:\n\n- I did not check the details of the proof of Theorem 1.\n- Regarding the results, the choice of learning step $\\alpha_t$ seems to be hardly usable in practice, as it relies on the knowledge of both the full gradient and the gradient of the sub-network.\n\n## Empirical results\n\nComputation reduction;\n- I am not sure to completely understand the metric used in the x-axis of Fig 2, in terms of unit used. Is it the total number of FLOP (i.e. the number of floating point operations over time integrated along time)? \n- Why isn’t BraTS reported as well for the computation cost reduction benchmark?\n\nCommunication reduction:\n\n- Which architecture is used for the results reported in Fig 4 and 5 for the classification tasks? Additional results are reported in Appendix for varying architectures, but it would be nice to precise it in the main text\n- For multiple experiments, the authors average results over 3 seeds, which is a good effort towards reproducibility. Would it be possible to perform 5 runs in order to get stds ? Otherwise, to have access to the individual values?\n- There are some instabilities at each step transition (cf Fig 5), and I suspect that warm-up seems to play some role here. The paper is silent about this aspect there were no ablation studies or investigation on this particular ingredient. I would have appreciated a focus on this part rather than e.g.", "summary_of_the_paper": "This paper introduces the idea of progressive training of deep networks for federated learning, where only some layer blocks are trained initially and the remaining blocks are progressively added, in order to reduce both local computation cost and communication cost in the early stages (as there are fewer weights). The authors provide a convergence proof of this approach, showing the same asymptotic convergence rate as plain training.\nNumerical experiments are conducted in EMNIST, CIFAR-10/100 and BraTS, split either in an IID or non-IID fashion and representing either a cross-device or cross-silo setting. Numerical experiments show a highly architecture-dependent reduction in the computation cost in the centralised setting. In the federated setting, communication gains significant communication gains are also achieved. Finally, the authors show that progressive training is compatible with other compression methods, increasing the gain in communication cost.", "summary_of_the_review": "This paper introduces an existing scheme into the federated setting, and provides an experimental proof of the gains it brings in terms of local computations and communication. Despite some minor issues which can be fixed, I think this paper could be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "68Wvf8_G6wA", "reviewer": "Reviewer_3zVc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please see \"Summary Of The Paper\" for the overview of the paper.\n\nThe paper introduces \"ProgFed\", which is an algorithm designed for federated learning scenarios. The authors use theoretical and empirical results trying the show the superiority of ProgFed over other federated learning methods when applying to neural networks. However, I think that there are several weaknesses in this paper.\n\n1. The first weakness (major weakness) comes from the  \"algorithm\" part. If I understand correctly, the novelty of ProgFed is applying the progressive training technique into the federated learning setting. However, progressive training is an existed method trying to speed up the neural network training in the single machine setting. In the single machine case, it is easy to apply progressive training techniques with different optimizers, e.g. RMSProp, ADAM, SGD, or other optimization algorithms. In the federated learning setting, it is also natural to apply progressive training with other federated optimization algorithms, e.g. local methods like FedAvg (Local-SGD), SCAFFOLD, FedPAGE, or compression schemes like quantization, Sign-SGD, etc. For ProgFed,  if I understand correctly, it just applies the progressive training technique with the Local-SGD algorithm. Thus the first weakness comes from the algorithmic novelty perspective.\n\n2. The second weakness (major weakness) comes from the model and the theoretical analysis. The data heterogeneity is the most important aspect that differentiates federated optimization from distributed optimization. If the data on different clients are not iid, then the gradients on different clients are not the same. In ProgFed algorithm (Algorithm 1) Line 16, the algorithm uses local update $x_{c,j+1}^s = x_{c,j}^s - \\eta g_c^s(x_{c,j}^s)$ for stage $s$, client $c$, local step $j$. Here, $\\nabla f_c(\\cdot)$ should be different in the federated learning setting. However, in the assumption part and the proofs, I do not see any assumptions to control the data heterogeneity (like $(G,B)$-BGD). Based on the fact that FedAvg and Local-SGD need assumptions like $(G,B)$-BGD to guarantee the convergence, I believe that either the model does not consider data heterogeneity (which is a main drawback for the federated learning applications), or the proof needs to modify in order to support data heterogeneity.\n\n3. The third weakness (minor weakness) comes from the empirical part. For the experiments, the authors compare ProgFed with very straightforward optimization (end-to-end) and several compression schemes. However, as I stated previously, ProgFed actually consists of two modules, progressive training and Local-SGD, which behave at a different level. Directly comparing ProgFed with end-to-end training does not show the effectiveness of ProgFed, I suggest the authors add more experiments, such as ProgFed vs Progressive training + simple SGD, to demonstrate the effectiveness of local-steps with the existence of progressive training, or ProgFed vs Progressive training + quantization, to demonstrate whether the progressive training combines better with local steps methods or compression methods, or ProgFed VS Local-SGD, to show the power of progressive training. Besides, I think the data split for different datasets can be moved to the main content.\n\nPlease point out if my understanding is wrong, and I will consider raising my score if my concerns and questions are addressed or answered. ", "summary_of_the_paper": "The paper introduces \"ProgFed\", which is an algorithm designed for federated learning scenarios. \n\nProgFed applies the idea of progressive training. Instead of training the neural network from end to end, it first trains the shallow layers, and then gradually train the deep layers, hoping that: 1) training the shallow layers first will learn some simple and meaningful features which will help training the deep layers, 2) and thus reduces the communication cost for the federated learning applications. Besides, the ProgFed algorithm also applies the local steps (Local-SGD or FedAvg) in order to decrease the communication cost during the model training. The ProgFed algorithm works not only for the feed-forward neural nets but also the U-nets, which consist of an encoder and a decoder.\n\nFor the theoretical part, the authors show that ProgFed has similar convergence results compared with SGD.\n\nFor the empirical (experimental) part, the authors use experiments trying to show the superiority of ProgFed over other existed federated learning methods, e.g. the trivial end-to-end training procedure, the communication compression methods (e.g. quantization)", "summary_of_the_review": "In summary, I have the following concerns:\n\n1. The proposed ProgFed algorithm seems a little bit incremental, since it seems like an \"A+B\" method where A is progressive training technique and B is FedAvg (local-SGD).\n2. For the model or the theoretical analysis, either the model does not consider data heterogeneity or the theoretical analysis needs to be modified.\n3. (minor concern) ProgFed is a combination of two techniques, and comparing with compression schemes like quantization is not very fair.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0F51JG2wUKO", "reviewer": "Reviewer_5a4S", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission is good; the paper is well-written, the idea is simple and seems to work well in practice. Furthermore, the experimental evaluation is thorough and considers multiple datasets and architectures. There is also a nice bonus that the proposed method is orthogonal to more traditional approaches that target communication cost reduction. For this reason, I am leaning towards recommending acceptance.\n\nHaving said that, there are some things that I would like some input from the authors:\n- It seems that currently you throw away the auxiliary heads obtained during the progressive training phase. This seems like a waste; why not reuse these heads for inference? For example, one could argue about some “voting” between the heads or even “early exiting” of datapoints if an early head is confident enough about the prediction (this would reduce the computational complexity at inference time as well).  Does the performance of each head become worse after the end-to-end fine-tuning?\n- While the provided proof is welcome, unless I am missing something, it seems to be in the centralised setting. Can it be extended to the federated setting? As the main motivation of this work is the federated setting, it makes more sense to prove convergence in such scenarios. \n- In the experiments you report the mean after three experiment runs. Why not show the standard deviation / error as well? This will allow one to better judge the significance of the results. Currently it is a bit unclear.\n- In Figure 2 it is weird that the end-to-end training has a bit of instability that is not present in progressive training. Could you elaborate on why this is the case? Furthermore, at Figure 5 ProgFed seems to have quite some aggressive dips in performance (I guess when switching stages). This is especially pronounced at CIFAR 10. Could the authors expand upon this and whether such instabilities can be hindering in practice?\n- Do you have any indication or intuition as to why progressive training helps with quantization / compression?\n- For section 4.5; does the “Layerwise” strategy use only one head, i.e., the one at the end? If yes, why not use a separate head (i.e, such as ProgFed)? How does Layerwise + end-to-end fine-tuning perform compared to ProgFed? Furthermore, the difference between ProgFed and “Random is small”, so some standard error indication would help.  Finally, the “Random” strategy is missing from Figure 8 and it would be great if you could elaborate on what is the extra memory that “Random” needs.\n\nAs for other minor things:\n- The plots such as the one at Figure 3 are kind of confusing and not very clear. I would suggest that the authors convert them to bar plots (such as the ones they have in the appendix) instead. \n- There is a reference that is listed twice (the “Sparsified SGD with memory”)\n- “Layerwise” referred to as “Laywise”\n", "summary_of_the_paper": "The issue that this work attempts to solve is that of computation and communication efficiency in federated learning. This is realised via ProgFed, a method that progressively trains neural network blocks of layers in $S$ “stages”. For the first $S - 1$ stages only a subset of layers is selected to be optimised, with the help of an optional (depending on the architecture) additional head to provide the training signal (which is discarded after that particular stage is finished). At the last stage, the authors propose to fine-tune the entire architecture in an end-to-end manner. Due to the specific nature of this form of training, the first $S-1$ stages of training require less compute (due to not needing to perform the forward / backward passes on the layers not selected) and also less communication (as the clients and server only need to communicate the specific set of layers that is optimised). The authors prove that such a training procedure converges in a rate similar to the one of standard end-to-end training and demonstrate in extensive experiments the benefits of ProgFed. As an extra bonus, the authors also show that ProgFed is more amenable to compression / quantization of the federated messages, thus showing that the benefits of these, more traditional, approaches are additive to the ones of ProgFed.", "summary_of_the_review": "Good submission with a simple idea that seems to work well in practice. Extensive evaluation and nice compatibility with more traditional compression and quantization. There are some points that I would like some input from the authors which, if turned out positive, would strengthen this work further. For this reason, I am leaning towards acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: As pointed out by some reviewers, the proposed method basically puts progressive training in the federated context. The theoretical analysis only concerns the centralized or non-federated setting and thus give no insight or guidelines for progressive training in federated learning. The main advantage of saving communication mainly comes from the simple observation that less parameters are computed and communicated during each round before the full end-to-end stage. However, this may cause extra overhead in hyper-parameter tuning including number of stage, learning rate schedules and stage-wise warmup. Despite its potential effectiveness in practice, the current version of the paper falls short of the acceptance bar due to the weakness in novelty and relevant theory for federated learning.
Decision: Reject