CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization
Authors: Kilian Pfeiffer
Martin Rapp
Ramin Khalili
Joerg Henkel
Venue: Accepted by TMLR ()

Abstract: Devices participating in federated learning (FL) typically have heterogeneous communication, computation, and memory resources. However, in synchronous FL, all devices need to finish training by the same deadline dictated by the server. Our results show that training a smaller subset of the neural network (NN) at constrained devices, i.e., dropping neurons/filters as proposed by state of the art, is inefficient, preventing these devices to make an effective contribution to the model. This causes unfairness w.r.t the achievable accuracies of constrained devices, especially in cases with a skewed distribution of class labels across devices. We present a novel FL technique, CoCoFL, which maintains the full NN structure on all devices. To adapt to the devices’ heterogeneous resources, CoCoFL freezes and quantizes selected layers, reducing communication, computation, and memory requirements, whereas other layers are still trained in full precision, enabling to reach a high accuracy. Thereby, CoCoFL efficiently utilizes the available resources on devices and allows constrained devices to make a significant contribution to the FL system, preserving fairness among participants (accuracy parity) and significantly improving final accuracy.

Ramin Khalili Published in Transactions on Machine Learning Research (06/2023)
CoCoFL: Communication- and Computation-Aware Feder- ated Learning via Partial NN Freezing and Quantization
Kilian Pfeiffer Karlsruhe Institute of Technology
kilian. dilaan-feifferrit.ccmm
Martin Rapp Karlsruhe Institade of Technology
martin.rapppkii..commm edu
ramin.khallf@@haauees.com
Huawei Research Center Munich
Jörg Henkel Karlsruhe Instätade of Technology
henkelBkitt.c edu
Reviewed on OpenReview: https: //tps://oll opwrrevee.......e..m net/ forum? id- XJIg4b0bku
Abstract
Devices participating in federated learning (FL) typically have heterogeneous communica- lion, compulation, and memory resources. However, in synchronous FL, devices need to finish training by the same deadline dictated by the server. Our results show that training smaller subset of the neural network (NN) at constrained devices, i.c., dropping neu- rons/filters proposed by state of the art, is inefficient, preventing these devices to make an effective contribution to the model. This causes unfairness w.r.t the achievable accuracies of constrained devices, especially in cases with a skewed distribution of class labels across de- vices. We present a novel FL technique, CoCoFL, which maintains the full NN structure on all devices. To adapt to the devices' heterogeneous resources, CoCoFL freezes and quantizes selected layers, reducing communication, computation, and memory requirements, whereas other layers still trained in full procision, cnabling to reach high accuracy. Thereby, CoCoFL efficiently utilizes the available resources devices and allows constrained devices to make significant contribution to the FI. system, preserving fairness among participants (accuracy parity) and significantly improving final accuracy.
Introduction
Deep learning has achieved impressive results in many domains (He et al. 2016 Huang et al. 2017 Young et al. 2018), and is also being applied in embedded systems such as mobile phones or internet of things (IoT) devices (Dhar et al. 2021). With recent hardware improvements, these devices are not only capable of performing inference
pre-trained model, but also of on-device training. Hence, federated learning (FL) (McMahan al. (017)) has emerged as an alternative to central training. An FL system compriscs many devices that each train deep neural network (NN) on their private data, and share knowledge by exchanging NN parameters via server. Distributing learning through FL brings many benefits, most importantly preserving the privacy of the end users.
Devices in real-world systems have limited computation, communication, and memory resources for train- ing, varying across devices. For instance, smartphones that participate in an FL system have different performance and memory (o.g., different hardware generations), and the conditions of their wircless com- munication channels vary (e.g.. due fading |Goldsmith] (2005)). Similar observations can be made in IoT
systems (Bhardwaj al. 2020 As stated by prior art. (Rapp et. al. 2022 Xu et. al. 2021
Diao et. al.

---

Published in Transactions on Machine Learning Research (06/2023)
design time run time
NN profiling Configurations
Heuristic
com. volume
Device
FL Server
quantized trained
train time
Figure 1: Overview of CoCoFL. At design time, different configurations of w.r.t. communication, computation, and memory in training, At run time, a heuristic selects a suitable configuration on each device w.r.t. the device's constraints.
layers are profiled
2020 Horvath et al. p021p, to enable efficient learning in such systems, FL needs to adapt to the per-dovice constraints, i.e., harduwrre-uware FL. Although different. techniques are proposed, the common idea in these state-of-the the-art solutions is to reduce the complexity by training subsets of the NN model on less capable de- vices, to match the required resources for training to the actual resource availability. While these techniques enable constrained devices to participate in the training, they do not effectively learn from their data, i.e., they do not preserve fairness (accuracy parity (Shi et al. 2021)). This is especially critical with non inde- pendent and identically distributed (non-iid) data, where the data differs statistically between devices (Hsu et al. 2019). , Our evaluation results show that existing solutions perform poorly in non-iid cases, such that in some settings, simply excluding constrained devices from training reaches higher accuracies. We attribute this in part to the fact that updates at constrained devices are less relevant to the overall learning objective, as they train much smaller subsets of the model, and also to the inability of these solutions to efficiently use the available resources in constrained devices.
In this paper, we propose a new technique CoCoFL (Fig. D. that allows all devices to calculate gradients based on the full model, irrespective of their capabilities, through partial freezing and quantization of the model at constrained devices. We show that quantizing frozen layers but keeping trained layers at full precision results in large reduction in resource roquirements, while still enabling efficient learning at devices. This combination has not been exploited so far. Freezing layers reduces the required gradient computations, the storage of intermediate activations, and the size of the parameter update, while quantization further speeds up the computations of frozen layers. Thereby, our solution adjusts the complexity of training to the resources available at each devico. Partial freezing and quantization opens up a large design space, whore cach layer can be frozen or trained on each participating device. The selection of trained layers has a significant impact on the required resources and on the accuracy. We introduce a houristic that allows for server- independent selection of layers w.r.t. local resource availability at run time, based on design-time profiling of the performance of devices. We demonstrate that our solution reaches significontly higher accuracy in iid and non-iid data, when compared with the state of the art, significantly improving FL. systems.
In summary, we make the following novel contributions:
We empirically show that in many scenarios, state-of-the-art subset-based techniques do not reach better accuracies than simply excluding less capable devices (a straightforward baseline). We ob- serve this throughout various datasets (e.g., CIFAR10, XChest, and Leaf benchmark data), data distrributions and NN topologies (e-g., ResNet, DenseNet, and Transformers).
Compared to the state of the art, in these scenarios, enable increased fairness of contribution and higher final accuracies in FL with helerogeneous resources by allowing less capable devices to do training based on the full NN structure. This is achieved by the following technical contributions.
We introduce a novel partial freezing and quantization technique to adjust to computation, commu- nication, and memory constraints of devices that allows to train full layers of NNs.
We introduce CoCoFI4] based on partial freezing and quantization, with simple, yet effective, heuristic to select locally on each device which layers to freeze or train based on the available communication, computation, and memory resources.
The code available at http:://gttabb..m///ttt//oos hctpp//bbcccmmm/1///oo/ot
2

---

Published in Transactions on Machine Learning Research (06/2023)
2 System Model and Problem Definition
System Model: We target distributed system comprising a server that is responsible for coordination and devices C that act as clients. Each device E C has exclusive access to its local data De. Training is done iteratively using FL. in synchronous rounds r. In each round, a subset of the devices (O) C C is selected. Each selected device downloads the latest model parameters w(r) from the server, performs training on its local data for a pre-defined round time T, and then uploads the updated model parameters to the server. The server averages all received updates (FedArg (McMahan et al. 2017 build w(++1) for the next round:
w (((1)) Exectrs Dej Execoe) [Da] w!"
The server discards updates that arrive late (straggler), i.e., devices must upload their updates in time.
Device Model: Devices are heterogeneous w.r.t. their computation (performance), memory, and commuu nication constroints. The performance of device (how long the training of an NN takes) depends on its hardware (number of cores, mieroarchitecture, memory bandwidth, etc.), software (employed deep learning librarios, etc.), and training configuration (topology, amount of data, etc.). Similarly, the available mcm- ory M, of device e depends on its hardware, while the required memory during training depends on the software and training configuration. Some of these configurations are fixed by the FL system (NN topology, etc.), while others are fixed by the device (hardware, software, amount of data), but some configuration A!") can be adjusted per device per round. In our case, A(r) A describes the subset of all NN layers that are trained in round by device c, with A being the set of all configurations (see Section 5). The training time of device for any A A is represented by the function te A-ER. The required memory during training is represented by m. A-R. We obtain te and me through profiling our technique on real hardware (measuring the training time and peak memory usage for different A).
The communication channel between devices and servers is commonly asymmetric: The download link from the server to the devices can be neglected due to the commonly high transmit power of base stations (Yang et al.] 2020). The upload link from devices to the server is subject to heterogeneous channel quality, discussed in Section | Therefore, we model the communication constraint s!r)) of a device in round r as a limit in the number of bits that can be uploaded to the server at the end the round. In our case, all layers not contained in A(r) are frozen (and quantized). Their parameters do not change, hence, do not need to be uploaded to the server. We represent the size of the parameter update for any A E A by a function s A-NN that is independent of device characteristics. Function s can be derived analytically or by counting parameters per layer.
Problem Definition: Our main objective is to maximize the final accuracy ace of the server model (R after R rounds under communication, computation, and memory constraints, by selecting per-device per- round the set of trained layers A!?):
maximize (R) VeEC VISSR s.t. (c(A A ma(AA))MM. A ((A
(2)
We also evaluate fairness (accuracy parity Shi et al. 2021 as a secondary metric, by mcasuring the device- or group specific accuracy using data that reflects each device's 's or group's distribution of local data De-
3 Related Work
We divide the related work into works that employ a similar mechanism settiiiaatioo/reziiing and works that target a similar problem (computation/ /ow.:////oommmmmiioiioooooo/oooo
constraints in FL.).
Quantization and Freezing in Centralized Training: Most works on quantization largel the inference, with full-precision training. Naive training on quantized parameters leads to training stagnation as small gradients are rounded to zero (Li et al., [007]]. To solve this, one branch of works performs stochastic rounding (Gupta al., (2015). However, stochastic rounding prevents convergence to the local minimum in the final phase of training with low learning rate [Li al.] 2017), reducing the accuracy. Another
3

---

Published in Transactions on Machine Learning Research (06/2023)
branch of work uses full-precision copy of the parameters an accumulator (Micikevicius et al. 2018 Calculating the parameter gradient based on quantized activations and parameters induces instability to the learning processes, which requires a lower learning rate to maintain convergence, slowing down the training, but also still resulting in a lower accuracy (Guo) 2018). In summary, achieving fast convergence and high accuracy requires keeping the Irained layers in full precision (aclivalion and parumeters in the forward and backward pass). Goutam et al. (2020) stochastically freeze layers of an NN to speed up training, keeping the frozen layers full precision which limits the achievable speedup. Also, due to its stochastic nature, it is not applicable to hard computation constraint. All these works either apply quantization freezing, None of the eristing works has erploited the symbiosis between freezing and quantization, where only frozen layers are quantized to maintain good convergence properties.
Communication, Computation, and Memory Constraints in FL: Most works on resource-constrainet FL. have targeted communication, extensively studying compression, quantization, and sketching of param- eter updates (Shi al. 2020; Thakker et al., 2019p. All perform regular training of the full NN in full precision, requiring full computation and memory resources, and only reduce the size of the parameter up- date. They are orthogonal to ours, i.e. applicable on top of CoCoFL. The work in Chen et al. (2021) detects NN paramcters that have stabilized, freezes them, and excludes them from synchronization to reduce the required communication. However, this technique can not cope with a given communication constraint. Re- cently, a preliminary work (Yang et al. 2022 proposed freezing layers to save communication and memory in FL. It exploits that frozen layers do not require storing activations for computing gradients, and do not need to be uploaded to the server. This technique has later been combined with quantization during download and upload (Ro al. 2022), but unlike in our CoCoFL, quantization is not used during training, missing out on significant optimization opportunities (c.g., up to 4x lower computation time we will show in our experiments). All these works do not reduce the computation cost of training.
Computation constraints in FL devices have only recently attracted attention. Some employ asynchronous FL (Xic et al. 2020). However, this does not reduce memory requirements and may reduce the convergence stability (McMahan han el al. 2017). FedProx (Li et al.) 2020) dynamicallly drops training data on straggling de- vices, which reduces computations but does not affect communication and memory requirements, and reduces the contribution of less capable devices. DISTREAL (Rapp et al., 2022) employs dropout to dynamically re- duce the size of the trained NN, reducing computations. They still transmit the full NN updates, and, hence, do not reduce communication costs. Several others train subsets of the NN on each device by (temporarily) scaling the width (number of filters/neurong memory. In particular, Helios (Xu et al.) 2021), HeteroFL (Diao et al., 2020), and FjORD (Horvath et al. 2021 proposed to create a separate subset. per each device according to its available resources. However, training very small subsets on weak devices does not enable them to effectively learn from their data, as we show in our evaluation, reducing fairness, hence, the final accuracy. Additionally, the reductions in commu- nication and computation achieved by width scaling are tightly coupled. Consequently, one of them forms the bottleneck, resulting in unexploited resources in the other metric. Finally, Yang et al. (2020) study the trade-off between communication and computation to minimize the overall energy consumption. This work is not applicable to a per-device computation or communication constraint.
of layers. This may save communication, computation, and
In summary, none of the eristing works on resource- conpuree-entttuieedd FL can adapt to per-device communication, computation, and memory constraints, while still effectively learning from all data on all devices. We achieve this through our novel combination of freezing and quanlization.
4 Partial Freezing and Quantization
This section introduces our freezing, layer fusion, and quantization technique to reduce the training com- plexity, which will be used in the CoCoFL. algorithm (Section 5)
4.1 Background: NN Structure and Training
Structure: Common state-of -ftte-- deep NNs like ResNet (He et al. 2016). DenseNet (Huang et al. 2017). MobileNet (Howard et al. 2017 follow similar structure, where convolutional layer (CI.) is followed
4

---

Published in Transactions on Machine Learning Research (06/2023)
forward pass
backward pass
forward pass
backward pass
quantized for- & backward
b1
b2
bg (a)
b1
61
b2
b3 (b)
b1 (c)
b
b3
b2
quantized forward
(a]
(a.)
(b)
(b)
(a)
(b)
(c)
(b)
(b)
training in full precision
Fa
fa
8fa dura
f2
fa
0/s
a
0f2
Bwn dez
dug
calculated in full precision
calculated in full precision
calculated in full precision
Figure The left part shows an NN where only block by is trained, while all others are frozen, and the forward passes bi, b2, and by are quantized (/1, J2, and fa), not requiring backward pass. The right part shows two blocks b2, by being trained, therefore requiring intermediate gradients of the quantized block by. Blocks are labeled (a), (b), and (c) depending on their types.
by a batch normalization (BN) layer, followed by ReLU activation. These layers account for the majority of training time. We label repeating structures like this a block, where in a general case, an NN comprises N blocks. We treat blocks as the smallest entity that is either frozen trained. Note that our technique is also applicable to variants of NNs, (e.g,, multiple skip connections or Transformers as we show in our evaluation) but for the sake of simplicity, the block description follows the ResNet structure.
Training: An update step in NNs comprises a forward and a backward pass. We describe the forward pass as a chain of consecutive operations, where each block has an associated forward function Ii+1 = (xx) with parameters w,. The full forward pass of the NN is calculated as ğ = ff(fv- (x1))) where ğ is the NN's output and 2 is the input. The backward pass consists of several gradient calculations to compute dE for each block i, where E C(g. y) is the optimization criterion with some
the paramcter gradients dm; loss function L. Using the chain rule, the calculations can be split into several gradient calculations. In the general case, the gradients w.r.t. block's parameters can be expressed as
OE OE Dw; Dî
(II k= Ook
0f(()))) 0f((zz) Ow:
(3)
Using the calculated gradients owyy local training with stochastic gradient descent (SGD) obtains updated parameters t; wi 10 OE where n) is the learning rate. When calculating gradients of several blocks,
intermediate gradient computations can be reused.
4.2 Freezing, Fusion, and Quantization of Blocks
Freezing: Freezing a parameter removes the need to calculate its gradients. As by Eq. (3) the number of required intermediate gradients depends on the block's index (c.g., the calculation of dwy requires no intermediate gradients, while dwi requires intermediate gradients from all other blocks). Based on the required per-block operations, we distinguish between three block types (illustrated in Fig,
aE
(a) Frozen block: With no preceding trained block, frozen block only requires forward pass f((z).
(b) Trained block: Trained blocks require forward pass fi(x;), calculation of gradicnts w.r.t. their param-
eters fi(=:), and gradients w.r.t the input 0f((=;) for preceding trained blocks.
de (c) Frozen block with backward pass: With preceding trained blocks, frozen block requires the forward
ows
pass fa(zz) and intermediate gradients w.r.t. the input @f((ms)
dx.
Consequently, freezing blocks reduces the number of per-block operations of frozen blocks (from to 2 or 1), and therefore saves mulliphy-accummulle operalions (MACs), reducing compulation time. Similarly, if layer is not trained, the activation values 2, can be released in memory during the forward pass, reducing the memory footprint. Additionally, parameters of frozen layers do not change throughout an FL round, therefore, do not have to be uploaded.
Fusion: If BN is used for normalization, we fuse the convolution operation with the following BN opera- tion Toffe & Szegody 2015 Jacob et al.] (2018) in frozen layers (we study the application of other normaliza- tion techniques Section 6). A BN layer normalizes each channel to zero mean and unit variance followed by a trainable scale 7/ and bias 3, The statistics of frozen blocks that do not require intermediate gradients
5

---

Published in Transactions on Machine Learning Research (06/2023)
(type (a)) stay constant over time. Hence, we can express the BN operation linear operator with Van being the channel-wise output of the BN operation, Vier. the CL's output, and a small number for stability
VieN
yior.
(7 + 8.), Votte
(4)
where the coefficient of Vier. is a new combined scale, referred to as %, and the second summation term is a new combined bias, referred to as B,. To fuse the BN operation with the preceding CL, we express the CL as Vict. = W,, and plug the output yier into the BN operator: Blax = (f) + ß.. This gives a scaled version of the original kernel W: = - W;W; with new bias 3i. The same can be applied for type (c) layers, with the only difference being that f and o² values are only valid for a limited number of mini-batches In summary, the forward pass of type (a) and type (c) blocks is simplified by fusing three operators, reducing the number of operations.
Quantization: Quantization of operators in NNs is usually used for inference. In contrast, we apply the idea for training; however, we quantize only parts of the NN that are frozen. Note that this is different from quantization aware training, since all trained parameters remain in full precision. Therefore, type (a) and (c) blocks are quantized, i.e., the fused convolution is performed in int8 instead of float32. Additionally, in type (c) blocks, we also quantize the calculation of the intermediate gradients in the backward pass, i.c., the fused transposed convolution. Consequently, the remaining operations in the forward and backward pass of frozen blocks require less time for execution and have a lower memory footprint.
Quantization of operations introduces quantization noisc in the forward pass and the backward pass of frozen layers, thereby affecting the training. Additionally, updating the fused layers' statistics only at the beginning of the round introduces an error. We demonstrate in our experiments that the benefits of increased efficiency w.r.t. training time and memory footprint outweigh the added noise. We quantify the benefits and the effects of quantization noise in detail in an ablation study in Section 6.3 In summary, freezing, fusion, and quantization of selected blocks lower the compulational complexily, while still allowing less capable devices to calculate parameter gradients of other blocks in full precision based on the full NN. This novel combination has not yet been exploited for training.
4.3 Implementation in PyTorch
We implement the presented training scheme in PyTorch 1.10 (Paszke et al. 2019), which supports int8 quantization. While the following description could also be applied to other quantization schemes (e.g., int4, float16), as of now, PyTorch only provides the necessary operators in the backends FBGEMM and QNNPACK for int8. Quantization levels, such as int4, could further lower the training time but at the same time could also have an impact on the accuracy. Contrary to quantization for inference, the layers' input scales, as well as the BN layers' statistics change throughout the training. Our implementation enables real-world training time and memory reduction throngh a combination of on-the-fly scale calculation and statistics from the server.
To preserve a high accuracy, the quantization of a PyTorch tensor æ roquires scale s to optimally utilize the int8 range. We calculate the scale by using
Quantization in the Forward Pass:
32 s<) max([max(z)), mmin())))/127.0.
(5)
Quantized operators (e.g., linear, conv2d, and add) take a quantized tensor स (and quantized weights) as input. In the used backends calculations are done in int8 arithmetic, but the accumulation of the result is done in int16/32. Therefore, each quantized operator requires an output scale 8, to map the int16/32 output to int8. This output scale s, is calculated using Eq. (5). For blocks of type (a), for every mini-batch, we calculate the scale of the input tensor 8q on the fly at the beginning of the first type (a) block. The input gets quantized and stays in the quantized representation throughout its forward propagation through type (a) blocks. For input scales 8,, the scale calculation results in negligible overhead, as æ is already available in its float32 representation. However, the output scales 8o depend on the output of an operation (linear, conv2d, add), therefore, can not be calculated a priori without performing the full operation in float. Because of this PyTorch limitation, the output scale s, and the BN layer's statistics (o² and μ from Eq. (4) are obtained from the server and are only set once per FL round.
6

---

Published in Transactions on Machine Learning Research (06/2023)
Quantization in the Backward Pass: Out of the box, PyTorch's Autograd system does not support quantized backward pass but expects float32 values for each calculated gradient. We implement a custom PyTorch Module for blocks of type (c) based on a custom Autograd Function that encapsulates all quantized operations in one backward call. Due to this limitation, a quantization/dequaaa /dequaniization is required, for cach block each mini-batch. The intermediate gradients' scale Bg is calculated on the fly. These limitations are inherently considered in our experimental results (profiling), i.c., fixing these limitations of Py would further increase the efficacy of CoCoFL. These overheads are minor compared the speedups gained through quantization, since large convolution operations dominate the training time (e.g., we measure a 6% overhead of scale/ /uuantization///eeeeee ponaattiatiom/deuuatattttoo /dequantization of type (c) blocks in MobileNet (Howard et al. 2017) but gain a reduced computation time by a factor of 1.3). Similar to type (a) blocks, the output scales of operators (e.g. transposed convolution) in the backward pass are obtained from the server.
Type (b) blocks (trained blocks) require only minor modifications. In order to acquire the scale of the operations, PyTorch forward and backward hooks are used to calculate 8g. For trained blocks, 8o can be efficiently acquired since trained blocks' operators calculate regular float32 outputs. Together with the trained parameters, these scales are uploaded to the server and averaged alongside the NN parameters. The scales only have to be calculated in the last mini-batch of a training round and result in negligible overhead. To perform fusion, as presented in Section E devices that train a respective block upload their BN statistics. The statistics are averaged alongside the parameters and distributed to the devices that require the respective statistics for fusion.
Transformer-Speciffe Implementation Details: We treat encoder layers as blocks, and quantize linear, layernorm, and ReLU operations for type (a) blocks. Due to PyTorch limitations, the attention mechanism has to remain in float32. In type (c) blocks, linear layers and their intermediate gradient calculations get quantized. Further details are provided in Appendix B
5 Overall CoCoFL Algorithm
Partial freezing and quantization enables to adjust the required ionmunication, computation, and memory resources by selecting which blocks to train or freeze. We present CoCoFL, which enables each device to select the trained /frozee /frozen blocks based on its available resources, and the required changes in aggregation at the server, in order to maximize the accuracy under constrained resources.
Heuristic Configuration Selection: A selection of trained blocks is a training configuration A A. The set of all configurations of an NN with N blocks comprises |A| = 2 configurations (each block is either trained or frozen). In each round, each device can select a separate configuration. Therefore, the total search space in each round is 2 N-1 which is infeasible to explore in its entirety, and impractical as it depends on the parameters like the NN structure. Simplifying the search space by assigning a separate quality measure to each configuration also does not work, since the accuracy after training with a certain configuration depends also on the configurations used by other devices. Therefore, heuristic optimization is required. Simple deterministic hcuristies like selecting configurations that train the maximum number of blocks once show bad performance, as some blocks would never get trained. As another example, round-robin selection would lead to all devices selecting the same configuration, where our observations have shown that this leads to lower accuracy (we provide experimental results in Appendix Cp. Therefore, CoCoFL selects a random configuration on each device based on its available resources. Thereby, the probability that many devices select the same configuration is negligible, while uallt all blocks within a device's capability get trained. An additional benefit of this scheme that no signaling between the server and devices to transmit the available resources and selected configurations is required, which hhhwiss could slow down the overall FL process and prevent scalability.
To be able to select configurations w.r.t. the available resources, we need to quantify the resouree requirements per configuration. We obtain this information through design-time profiling of a real implereentation of our presented freezing and quantization scheme on real devices, but an analytical model of the resources could also be employed. Profiling takes several seconds per configuration. For instance, profiling MobileNet takes 4.7s on average on the x64 target platform. It is, therefore, infeasible to profile all 2 configurations per device, which would take several months. We solve this by only considering configurations À A that
7

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm Each Selected Device (Client) in Each Round
Require:
t.te me, De, TT, Sr, Mc receive w((r) from the server
> profiling information of the device, data, available resources b initial parameters at the beginning of the round
Ar [A (Al) & &' A mc(A) S M. s(A) s S.)
feasible configurations (Eq
Amax [Aj E Ay: VAA E & Ax}
discard non-maxinal configurations
A random choice(Amax)
select random configuration blocks) local training
(Wrrain: (quant) + apply(A (w()))
D apply the configuration
Utrain train model (UPrrain, Mqquant) with local data De send Utrain to the server
> parameter update
Algorithm 2 FL Server (Synchronization and Aggregation)
w()) random initialization
each round r 1.2..... do
((r) seleect devices
broadcast w(r) selected devices c(r)
for each e c(r) receive Barain,e from device c
for each block i do
aggregation
C f {o: itrrain.c contains block x}
b. devices that have trained block
w (r(+1) <
[
1C11
w
Exsc, (frain.e())
train a single contiguous range of blocks. This reduces the search space to | = N(N+11 2 i.e., 17 min
for MobileNet on x64. resources can be estimated much faster, relaxing this restriction could further improve our technique. The run-time algorithm for devices is outlined in Algorithm [ At the beginning of each round, each device determines the set of feasible configurations AySÁ, given its currently available resources. We then discard all configurations that train subset of blocks trained by another feasible configuration, i.., we only keep maximal configurations Am, thereby maximizing the accuracy by fully exploiting the available resources. Each device selects a random remaining configuration, which results in different configurations being trained on different devices without requiring any synchronization between devices. Finally, the selected fusion and quantization configuration is applied, and the NN is trained.
Each device only uploads updates of the blocks that were trained in full precision (irrain.e), hence, not frozen quantized. The server (Algorithm 2 weighs the updates based on the number of devices that have trained each block to account for partial training on the devices. Fig. 1 shows CoCoFL in nutshell.
A Aggregation of Partial Updates:
6 Experimental Evaluation
Partial quantization of NN models results in hardware-specific gains in execution time and memory. Hence, our evaluation follows a hybrid approach, where we profile on-device training loops on real hardware and take the profiling information to perform simulations of distributed systems. This allows for the evaluation of large systems with hundreds or thousands of devices.
6.1 Evaluation Setup
Profiling Setup and Results: We employ two different hardware platforms to factor out potential micro- architecture-dependenn peculiarities w.r.t. quantization or freezing: x64 AMD Ryzen and a Raspberry Pi with an ARMv8 CPU. For each configuration in Á, measure the execution time, maximum memory usage, and upload volume, The measurements specific details about N, | and the used platform are given in Table 1 further details in Appendix A For simplicity in the implementation, we select one skip connection block in ResNet, MobileNet, and DenseNet as the smallest entity that is either trained or frozen. This choice allows for limited implementation overhead, as the structure is repeatedly used in the NNs. The block granularity could be further incrcasod by selective training and freezing of CL layers within skip connection block, but it would require more individual cases to be implemented. Exemplary profiling results of MobileNet x64 are shown in Fig. 3 where all quantities
stored in lookup table for the FL simulations, NN-
8

---

Published in Transactions on Machine Learning Research (06/2023)
norm. upload volume
norm. memory foolprint
CoCoFL (ours)
HeteroFL
FjORD
0.5
0.5
FedAvg
20
0
0.2 0.4
0.6 0.8
0 0.2 0.4 0.6 0.8
norm. training time
norm. training time
Figure 3: Profiling of MobileNet regarding upload volume, training time, and memory footprint, normalized
training the full NN with FedAvg. Each marker represents training configuration. The figure shows that CoCoFL supports the same range w.r.t. to the constraints as HeteroFL and FjORD but enables independent adjustability of computation/mmmony and communication.
Table 1: Hyperparameters FL experiments.
Hyperpaaaameters
DenseNet
MobileNet CIFAR10
MobileNet CIFAR10 GroupNorm
ResNet50 CIFARI00
R8sNet18 FEMNIST
MobileNet XChest
Transformer
CIFAR10/
IMDB/
(CINIC10)
(Shakespeare)
Rounds R
800
1000
1000
800
600
1000
1000
Amt. data [D]
50K (90K)
50K
50K
50K
640.5K
12.7K
40K (200K)
10/100
10/100
10/100
10/100
35/3500
10/100
10/100
n-decay (x0.1)
[700]
6600, 800]
8800
[750]
[400)
[600]
[800]
Weight Decay
0.001
D.01
0.001
0.01
0.01
0.01
Nb. configs [
253
210
210
171
55
210
35
Nb. blocks N
23
21
21
19
11
21
Platform
ARM
x64
x64
x64
ARM
x64
x61
are normalized to FedAvg (full training of the NN). Using the same setup, profile training with subsets of the NN's filters, as employed by HeteroFL and FjORD. In CoCoFL, configuration refers to specific selection of blocks that are frozen, quantized, and fused. In HeteroFL/FjORD, a configuration refers to a specific ratio of filters that are trained and filters that are dropped. As a consequence, the trade-offs vary. Our results show that the combination of freezing, fusion, and quantization allows training with configurations that reduce the execution time by up to 90% and the memory footprint by 89% compared full training of the NN, a similar range as HeteroFL and FjORD. Further, CoCoFL enables independent adjustability of computation/ iitiiattiommmmoryy memory and communication, giving us higher degree of freedom to select configuration that utilizes the resources at the device, therefore, more efficiently utilizing available resources. Contrary to that, training subsets results in a tightly coupled reduction of resources. However, in some cases, CoCoFL is required to pick a non-optimal configuration w.r.t. computation, to satisfy a memory constraint.
FL Setup and Hyperparameters: We evaluate our technique in an FL system, using the profiling results. For each experiment, we distribute the data from the datasets CIFAR10/100 (Krizhevsky Hinton 2009).
FEMNIST (Cohen al. 2017), CINIC1O (Darlow al.) 2018), XChest (Wang et al., 2017), LMDB et al. 2011), and Shakespeare (Caldas et al., 2019) to devices in C. We evaluate ResNet (Gao al. 2020
(Maas
DenseNee Net (Huang et al. 2017), MobileNet (Howard el al. 2017,, and Transformer Vaswani et al. 201 7) NN models. In each round, a subset c(r) is selected for participation. Devices are randomly grouped in three equally sized sets. The set of strong devices is capable of training the full NN and uploading all parameters, with no memory constraints. The round time is set to the time a strong device requires to finish one training round. The set of medium devices has 2/3 of the computational and memory resources of the strong devices. Hence, to match the round time T, the set of medium devices has to select configurations that reduce required computations to 2/3 of strong devices. The set of weak devices has 1/3 of the computation and memory capabilities of the strong devices. We model the communication budgets of medium and weak devices randomly over rounds to simulate an environment with varying communccttio channel quality,
s(r) s.t. Senedum, U( Satrong
2 Sttrong) We compare CoCoFL to several baselines: state-of-the-art HeteroFT.

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Accuracy (Top 1) in % for DenseNet, MobileNet, ResNet18, ResNet50, and Transformer. For XChest the F1 macro score is given (unbalanced data). In almost all scenarios CoCoFL outperforms the baselines, reaching higher final accuracies.
Topulogy
DenseNet
MobileNet
ResNet18
Setting
CIFARIO
CIFARIO
CIFARIO (w. GroupNorm)
FEMNIST
Dirichlet
(iid)
n--id000.1
reB0.1
(idd)
re90.1
(iid)
n.iid00.1
re80.1
(lid)
re00.1
Centralized
87 8:0.2
87.0=0.7
82.54:11
68.120.0
FodAvg, (f. res.)
84.3+0.1
75.6+1.5
74.94333
84.94:0.2
77442277
76.1 0.1
69.4 + 0.5
72.9 + 1.4
86.2-0.1
82.941.2
CoCoFL (ours)
82.0:10.2
71.99:18
68.8-4.6
83.2:0.3
72.442.9
71.3
+ 0.1
61.2
+ 1.4
63.6
- 3.5
85.0=0.1
81.5=0.6
FJORD
78.7+00.1
60.44-2.1
48.8466.8
79.1+:0.3
51.9+7.3
64.4 + 0.6
42.9
+ 1.7
47.0
+ 5.0
8505
69.34-8.3
HeteroFL
7..4+0.3
61.0#24
51,247.4
79,540.2
53.0=7.6
64.8 4 0.1
55.2
+ 0.3
47.5
= 5.0
85.9=0.0
70.945.8
FedAvg
76.5+0.1
60.4+4.2
50.947.55
78.140.4
49.948.7
56.2 0.8
52.8111
45.9
- 4.7
86.1+0.11
64.947.88
Topology
5esNet50
DenseNet
MobileNet(largoc)
TF
TF-S28
Setling
CIFAR100
CINIC10
XChest
IMDB
Shakespeare
Dirichlet
(idd
re600.1
(idd
n..i.660.11
1o900.1
(idd)
re80.5
(id]
(idd)
-re (Leaf)
Centralized
61.62.4
80.5+0.2
94.2#0.22
84.7:0.77
52.9
FedArg (f. rea.)
57.020.3
53.020.6
77.2#0.1
$3.9=233
65.1#111
91.1+0.3
85
93.2+0.2
82.6+0.1
19.1=0.11
19.110.1
CoCoFL (ours)
52.5:1.22
41.8:2.55
73.6:0.1
53.514.33
52.417.3
91.310.3
73.016.4
87.3=3.8
82.510.5
49.3:0.33
49.1=0.3
FJORD
43.620.8
29.6+4.3
65.7
49.2-22
41.1
66.340.9
52.7+3.9
62.4+0.8
78.5+0.7'
42.920.5
43.0+0.3
HeteroFL
45.9
31.0423
69.4.0..
50.4:224
43.44:7.2
69.4+1.0
65.0+0.9
65.41:1.6
79.2:1..3
44.1::0.2
44.11:0.2
Avd
35.2+-0.2
23.7+00.4
67.7+0.4
48.342.5
42.4+47.2
68.2+1.0
67.0+0.66
66.841.4
78.5++0.6
40.5+0033
40.3+0.11
NNo configuration for weak devices available, therefore weak devices are dropped.
and FjORD, which are the closest to our technique, as both allow for per-device reduction of computational resources, as well as upload volume, and memory. Additionally, we compare to a theoretical bound and a straightforward bascline that drops all but the strong devices from FL training.
Centralized: All data is centralized (on one device), serves as a theoretical upper bound. FedAvg (full resources) (McMahan et al.] 2017): FL is applied, but all devices have full (homoge- neous) resources, hence, FedAvg has only one configuration, that is training the full network. This baseline serves theoretical upper bound.
HeteroFLL (Diao et al.) 2020): A FedAvg variant that drops number of the CL filters, defined by a shrinkage ratio, where each ratio represents a training configuration. We set it to the maximum that a device can train.
FjORD (Horvath et al. 2021): Similarly, state-of-the-art FjORD drops CL filters to reduce re- sources. In difference to HeteroFL, each device trains different drop levels (within their capabili- ties). Devices switch each mini-batch between a feasible configuration. The paper proposes to use [20,40] 40, 60, 80, 100%% of the filters, which results in configurations.
FedAvg (McMahan et al. 2017): Devices that can not train the NN (e.g., due to limited memory) are dropped from the training (therefore also their data). The reduced set of devices performs FedAvg. This serves as a naive baseline and is known to be used in production use cases (Yang et al. 2018
We train with the optimizer SGD with an initial learning rate 1) of 0.1. For a fair comparison we do not use momentum, as FjORD is incompatible with a stateful optimizer. The remaining NN-specific hyperparam- eters, learning rate decay, and weight decay are given in Table [ For each FL experiment, we report the average accuracy and standard deviation after R rounds of training using 3 independent seeds. We study several data split scenarios: First, an iid case, where data is randomly distributed to all devices, and hence, every device has about the same number of samples per class. Second, we study a non-iid case, where we vary the non-iid-ness with the value of a of a Dirichlet distribution, similar to Hsu et al. (2019). Hereby, the number of samples per class varies between devices. Thirdly, we consider a scenario where data is resource correlated non-iid (rc-non-iid). This means that information about certain classes is only available on spe- cific device groups, increasing the necessity include them in the FL process (Fig. 4ah. This has recently been identified as relevant use case in real-world deployments (Maeng al., 2022). Similarly, the rate of re-non-iid-ness is controlled with a.
10

---

Published in Transactions on Machine Learning Research (06/2023)
non-iid (a 0.1)
re-non-fid (a 0.1)
MobileNet / CIFAR10 re-non-iid@0.1
10
10
100
FedAvg (full res.)
class index
class index
grong accuracy
80
CoCoFL (ours)
5
5
1000000
60
HeteroFL
40
FjORD
20
FedAvg
0
0
{strong mmedium [weak)
{strongH Mmedium [wekk
(ombined(stronng (trong medium H weak) groups
device index
device index
(a) Demonstration of non-iid data (Ieft) and resource- (b) Group accuracies of MobileNet with CIFARIO and porelated-non--ie data (right). The size of the circles rep- re-non-id@0.1 data. CoCoFL preserves fairness among resents the number of samples.
groups, since weak and medium devices achieve similar group accuracies as if they would have full resources.
Figure 4: Fig. Aa visualizes resource-correlated non-iid, while Fig, 4b shows the effects of this distribution w.r.t. the fairness (group accuracy) in CoCoFL and baselines.
6.2 FL Results
Table 2
presents the accuracy results given in %. For XChest (unbalanced) the Fl macro score is given.
Vision Models: For iid data, CoCoFL performs close to FedAvg (full res.), improving the final accuracy over the basclines by p.p. for DenseNet (CIFAR10), by p.p. for MobileNet (CIFAR10), and by 6.6 p.p. for ResNet50 (CIFAR100). Similar trends can be seen for CINIC10. This clearly indicates that CoCoFL, uses the available resources devices more effectively. An outlier is the FEMNIST dataset, where FedAvg reaches the highest accuracy despite dropping 2/3 of the devices. We attribute this to the high number of redundant samples in the dataset (10K per class compared to with 5K and 500). Contrary to that, if the number of samples is more limited, as it is the case in the XChest experiments (12K samples total), the advantage of CoCoFL over the baselines increases.
The necessity to include less capable devices in the FL training is more clearly visible in cases with re-non- iid. In Table 2 it can be seen that 0.1 results in larger gap between the upper bound and the naive bascline, demonstrating the importance of involving all devices in the training. CoCoFL enables weak and medium devices to contribute to the global model, reaching up 20 p.p. higher accuracy compared with the state of the art. The reason is that CoCoFL allows weak and medium devices to calculate gradients based on the full NN. In the case of DenseNet and MobileNet, FjORD and HeteroFL even perform close or inferior to the naive baseline, which excludes weak and medium devices from the training, failing to preserve fairness (accuracy parity) among devices. Similar conclusions can be driven from ResNet18/50, albeit FjORD and HeteroFFL perform a bit better in these settings compared with FedAvg. For XChest we present results with 0.5 since we observe that 0.1 leads to complete separation of the binary labels, causing all algorithms to fail to learn at all.
Fairness in Re-non-iid Scenarios: To quantify the contribution medium and weak devices make in the training, we calculate the device-specffic accuracy per group (group accuracy), where the class accuracies are weighted by the groups' class densities. As it can seen for MobileNet (re-non-iid@0.1) in Fig. 4bj CoCoFL achieves group accuracics of 79%/70%//68 66%% for strong, medium, and weak devices, hence, close the accuracy parity of FedAvg (with full resources). The baselines HeteroFL and FjORD reach 88%/48%/ meaning less capable devices can not make meaningful contribution to the global model, hence, lowering the fairness among the device groups
Other Normalization Techniques: We replace BN with GroupNorm (w & He 2018 in MobileNet to test the robustness of CoCoFL w.rtt. other normalization techniques in vision tasks. For MobileNet with CIFAR10, we observe that the overall accuracy is lower in all evaluated algorithms (Table 2) However, the general trends are similar to MobileNet with BN, i.., independent from the normalization, CoCoFL outperforms the state of the art Additionally, we evaluate the re-non-iid scenario, where CoCoFT. reaches
11

---

Published in Transactions on Machine Learning Research (06/2023)
group accuracies of 63.7 60.7 and 9.8% for strong, medium, and weak devices, whereas and 81.3 7.997/.. + 13.5%,
FjORD and HeteroFI. reach 80.7 8.5.22/8///..88888888888888 respectively. Thus, CoCoFL provides better fairness independent of the normalization technique.
NLP Models: To show the applicability of CoCoFL to natural language processing (NLP) problems, we adapt our freezing and quantization scheme (Section 4 for Transformer. We study text. classification with the IMDB dataset and next character prediction with the Shakespeare dataset. The Transformer model uses 6 encoder layers, with embedding size of 128, hidden size of 128, 2 attention heads, and single linear decoder layer. For IMDB the sequence length is 512, for Shakespeare 80. For Shakespeare re-non-iid, we follow the non-iid scheme from Leaf (Caldas et al. 2019), such that different plays (total of 25) are distributed over different device groups. The results in Table show that CoCoFL reaches significantly higher final accuracies than the state of the art.
In summary, CoCoFL reaches higher accuracies in almost all presented scenarios. Additionally, CoCoFL preserves fairness (accuracy parily) by enabling constrained devices lo contribule to the global model. We attribute this large accuracy gap w.r.t the baselines to the fact that CoCoFL allows any device to calculate gradients based on the full NN, while still reducing required resources, opposed to state-of offthe-art techniques that calculate gradients on subsets of the filters of the NN.
6.3 Ablation Study
We conduct an ablation study to quantify the gains and the error of quantization and operator fusion of frozen blocks in CoCoFL. For this purpose, we modify the Mobile PobbleNet/CIFAR100 /CIFARIO iid experiment of Section 6.1 Instead of three, we have two groups: 10% strong devices, i.e., no constraints (training the full NN). We label the remaining 90% limited devices, with a computation and memory limit 1, s.t. fimited (A) = fstrong (A) and Minited M-Mstrong- We apply no communication constraint, therefore, Siimited = Sttrogg- Consequently, limited device has to select a configuration A that satisfies both the computation and the memory constraint. Several experiments are conducted, where / is varied between I E [0, 1]. The remaining hyperparameters are kept the same (Table D. We introduce three variants of CoCoFL, where we profile each variant's 's configurations A to measure the execution time and the memory footprint:
CoCoFL where only freezing and no quantization or operator fusion is applied. The set of feasible configurations is denoted as AF.
CoCoFI,FF FF is a variant where freezing and fusion of operators, but no quantization is applied. Con-
figurations are denoted as A
CoCoFL QFF is the mainline variant (on 6.1 The configurations are equivalent to Á.
To quantify the error introduced through fusion and estimation of the statistics as well as quantization noise, we run all three variants CoCoFL", CoCoFL FF and CoCoFL OPF QFF for different values of 1, but each variant uses the configurations in A on the limited devices (ignoring all computation/mat mmmmory gains that come from quantization and fusion). For given I, all variants can train exactly the same configurations, hence, the same number of blocks. This allows studying the introduced errors independently of the gains in performance/mne maneeer/myy..... On the right part of Fig. 5a. the cumulative number of trainable blocks for a given 1 in A is displayed. Using AF, at least = 0.35 is required to train a single block. The accuracy results are visualized in Fig, 5a (left), where the accuracy for different values of I is reported on the left. Overall, the error is mostly below p.pp, with a maximum of 2.3 p.p. for CoCoFT, QFF and 1.7 p.p. for CoCoFI, Note that this analysis ignores the performance and memory gains, which are studicd in the next section.
Quantification of the Error:
Quantification of the Gains: To quantify the gains, we run the experiments with all three variants where each variant uses its own profiling results. Hence, CoCoFL QFF and CoCoFL FF use Å, and A respectively, We mcasure that CoCoFL QFF reaches maximum reduction of 75% of computation time and 60% of memory w.r.t. to CoCoFL F (45% and 28% with CoCoFL FF). Therefore, al a given constraint 1, CoCoFL QFF and CoCoFL FF can train more configurations and hence, more importantly, configurations with more trained blocks. This results in an overall higher accuracy FL as can be seen in Fig. 5b (left), where with the same 6.5p.p. constraint of I 0.25 CoCoFL QFF achieves an increase of the final accuracy of 6.5 p.p. over CoCoFL FF while CoCoFT. has no configuration on the limited devices that satisfies the constraint. At constraint. of l = 0.4,
12

---

Published in Transactions on Machine Learning Research (06/2023)
CoCoFT.
CoCoFT. PF
CoCoFI, QFF
Q Quantification of Error
Q Quantification of the Gains
mmm. trainable blocks
20
IIII trainable blocks
20
80
80
accuracy [%)
15
accuracy [96]
15
10
10
70
70
5
60
60
0 0.2 0.6
0.6
0.2 0.4
0 0.2 0.466.8
1
(a) (Left) final accuracy not utilizing gains (error analy- (b) (Left) final accuracy utilizing the gains, (right) cumu- sis), (right) cumulative number of trainable blocks used lative number of trainable blocks used for training, When training. When ignoring the gains, the introduced utilizing the gains, limited devices can train more blocks errors from quantization and fusion are negligible.
and achieve higher accuracies, outweighing the error.
Figure 5: Quantification of the gains and the error.
CoCoFL QFF achieves a final accuracy increase of p.p. over CoCoFL", while CoCoFL achieves achieves a final accuracy increase of 14.3 p.p. over CoCoFT.F. From the results can be concluded that the more blocks the limited devices can train, the higher the final accuracy. This visualized Fig. 5b (right), where cumulatively the total number of blocks that can be trained for a given constraint is plotted. The figures also show that in the case of the constraint approaching / 1.0, the advantage of quantization and fusion is result in small accuracy losses due to the introduced error.
vanishing, and
In summary, for limited devices, the benefils of fusion and quantization of blocks, i.e., training more blocks with the same available resources, largely outweigh the introduced error. Only when the devices' constroint approaches full resources, the gains do vanish. Overall, quantization and fusion increase the FL system" 's accuracy, as limited devices can make a higher contribution to the model.
7 Conclusion
We proposed CoCoFL that is able to better incorporate knowledge from constrained devices into the FL model, especially in non-iid cases, preserving fairness among participants. Our comparison with the state of the art, based on real hardware measurements, shows that CoCoFL reaches significantly higher final accuracies. We believe that the gains through quantization can even be higher on devices like smartphones that have on-chip integer NN accelerators.
In an FL system, devices are acquiring data through sensing or interaction with the environment. As devices are distributed in the system, they may have access to different types of data. Examples include sensors that sample environments that differ from each other, smartphones that interact with users with different behaviors. This is therefore important to guarantee that we learn from all these devices, regardless of their capabilities, as any piece of the gathered data matters. What is then important is to provide fairness (accuracy parity) among devices, fairness of participation alone, as was the focus of state of the art, is not enough. By approaching accuracy parity among devices, CoCoFL makes FLL systerss applicable to broader range of use cases, especially use cases when the distribution of classes across devices is skewed.
Acknowledgments
This work is in parts funded by the Deutsches Bundesministerium für Bildung und Forschung (BMBF, Federal Ministry of Education and Research in Germany). The authors ackmowledge support by the state of Baden-Wirttemberg through bwHPC.
13

---

Published in Transactions on Machine Learning Research (06/2023)
References
Kartikeya Bhardwaj, Wei Chen, and Radu Marculescu. New directions in distributed deep learning: Bringing the network at forefront of iot design. In Design Automation Conference (DAC). IEEE, 2020.
S Caldas, P Wu, T Li, Koneený, HB McMahan, V Smith, and Talwalkar. Leaf: A benchmark for federated settings. Xrxiv:1812. 01097, 2019.
Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. Communicttion-cffie federated learning with adaptive parameter freezing. In Int. Conference on Distributed Computing Systems (ICDCS). IEEE, 2021.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. Emnist: extension of mnist to handwritten letters. arXiv: 1en:1702.000 05373, 2017.
Luke Darlow, Elliot Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- not imagenet or cifar-10. arXiv:1810.0935 03500,, 2018.
Sauptik Dhar, Junyao Guo, Jiayi (Jason) Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Trans. Internet Things, 2(3), jul 2021. ISSN 2691-1914. doi:
/3450994.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogencous clients. In International Conference Learning Representations (ICLR), 2020.
Mengya Gao, Yujun Shen, Quanquan Li, and Chen Change Loy. arXiv:2002.09160 2020.
Residual knowledge distillation.
Andrea Goldsmith. Wireless Communiications. Cambridge University Press, 2005. doi: 10.1017/ CRR09780518411224
Kelam Goutam, Balasubranarian, Darshan Gera, and R Raghunatha Sarma. Layerout: Freezing layers in deep neural networks. SN Computer Science, 1(5), 2020.
Yunhui Guo. A survey methods and theories of quantized neural networks. arXiv:1805.04752 2018
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning (ICML), pp. 1737 1746. PMLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Advances in Newral Information Processing Systems, volume 34. NeurlPS, 2021.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetio, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1704: 11..174440048800 04861, 2017.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. UrXiv.1999.06395 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2017. doi: (11109/CVPR20111
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic- only inference. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp.
2704-2713, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In International Conference on Neural Information Processing Systems (NeurIPS), pp. 5813-5823, 2017.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet. Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, volume 2, pp. 429-450 450, 2020.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annuual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
Kiwan Maeng, Haiyu Lu, Luca Melis, John Nguyen, Mike Rabbat, and Carole-Jean Wu. Towards fair federated recommendation learning: Characterizing the inter-dependence of system and data heterogeneity. In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 156 167, 2022.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communictiion-cffieet learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1273 1282, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksfi Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Represenlations (ICLR), 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS). NeurIPS, 2019.
Martin Rapp, Ramin Khalili, Kilian Pfeiffer, and Jörg Henkel. Distreal: Distributed resource-aware learning in heterogencous systems. In AAAI Conference on Artificial Intelligence (AAAI). AAAI, 2022.
Jae Ro, Theresa Breiner, Lara MeConnaughey, Mingqing Chen, Ananda Suresh, Shankar Kumar, and Rajiv Mathews. Scaling language model size in cross-device federated learning. In Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022), pp. 6 6-20, 2022.
Yuamming Shi, Kai Yang, Tao Jiang, Jun Zhang, and Khaled B Letaief. CitimunniaationEffiieene Edge Al: Algorithms and Systems. IEEE Communnications Surveys & Tulorials, 22(4):2167 2291, 2020.
Yuxin Shi, Han Yu, and Cyril Leung. A survey of fairness-aware federated learning. xrxi::2111.0188 2021.
Urmish Thakker, Jesse Beu, Dibakar Gope, Chu Zhou, Igor Fedorov, Ganesh Dasika, and Matthew Mattina. Compressing RNNs for loT Devices by 15-38x using Kronecker Products. arXiv:1906. (ef.:.99000966,, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30,
2017.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localiza- tion of common thorax discases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv:1903.08994 2020.
Zirui Xu, Fuxun Yu, Jinjun Xiong, and Xiang Chen. Helios: Heterogeneitzy-warre federated learning with dynamically balanced collaboration. In Design Automation Conference (DAC). LEEE, 2021.
Tien-Ju Yang, Dhruv Guliani, Françoise Beaufays, and Giovanni Motta. Partial variable training for efficient on-device federated learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4348 4352. IEEE, 2022.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv:1812.0200 2018.
Zhaohui Yang, Mingzhe Chen, Walid Saad, Choong Seon Hong, and Mohammad Shikh-Bahaei. Energy efti cient federated learning wireless communication networks. IEEE Trans. Wireless Communications (TWC), 20(3):1935 1949, 2020.
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learning based natural language processing. IEEE Computational Intelligence magazine (CIM), 13(3):55-775 -5,75,--50 2018.
16

---

Published in Transactions on Machine Learning Research (06/2023)
A Profiling Setup Details
Profiling: Profiling of the used NNs is done for CoCoFL as well as HeteroF'L and FjORD to quantify the reduction in training time, maximum memory usage, and upload volume. Two platforms are used: An x86- 64 AMD Ryzen 7 with 64 GB RAM and a Raspberry Pi 4 (Cortex-A72 ARM v8 64-bit) with &GB RAM, The profiling is done for all tested NN architectures to acquire all configurations in A. Similarly, we profile HeteroFL and FjORD where we vary the drop ratio from 0.1 1.0 with 50 linearly spaced steps. We allow freezing of the last layer (linear layer) by but do not apply quantization. Profiling takes (MobileNet) 17 min on x64 and about h on ARM. The profiling procedure measures the following quantities in the training:
Maximum memory: The following training-related parts are included in the memory measurements: First, the model is loaded from disk, second, a training batch is loaded from the disk, and third, an optimizer is initiated. These operations are followed by training of 16 mini- batches. The maximum memory is measured by using the Linux syscall getrusage with parameter QUSAGE_SELL This call returns struct with a variable ru_ mu_axxrs,, that stores the maximum amount of memory, the process required at some point in time. For all measurements, we subtract the maximum memory before the training (e.g., the overhead of the process, PyTorrch, and NumPy imports).
Training time: For the same training procedure, we measure the training time. The time measure- ments include the NN's forward pass execution, setting gradients to zero, the backpropagation step, and optimizer steps. We do not account for mini- batc-beachhleeel switching between configurations for FjORD and model the switching as a zero-overhead operation.
Upload volume: The upload volume can be directly calculated from the size of the NNs' state_dicts by filtering for parameters that required gradient calculations during training.
SOTA Comparison: For HeteroFL and FjORD we enforce the same constraints from Eq. 2 CoCoFL. To ensure this we configure the baselines in the following way:
as for
FjORD (Horvath et al., 2021 In FjORD all devices switch between different configurations for each mini-batch. We use FjORD's drop levels of PFJORD = (20%, 40%,... 100%). A device e with
lower resources can only train with levels Pý, that satisfy the constraint pEjORD = {p, tp(p)) S
T A me(p;) < Mc A S(p;) < Se)-
HeteroFL. DDao el al., 2020): Similarly, in HeteroFL. the ratio of dropped filters in an NN is set through a shrinkage ratio & E 10. where h = results in training of all available fil- ters. Contrary to FjORD, a device uses the same shrinkage ratio throughout the round. For each device c, we set the shrinkage ratio to the maximum value that satisfies the resource con- straints k = maxo<ks1 s.t. te(k) < T A (k) S Me A S(k) S Sc-
B Implementation Details and Hyperparameters
Miscellaneous Hyperparameters for Vision Models: We evaluate our technique in an FL setup where we train the NN models DenscNet40 (Huang et al. 2017), MobileNetV2 (Howard et al. 2017). ResNet50 (Gao
et al. 2020), and ResNet18 (Gao et al. 2020). The image datasets CIFAR10 (Krizhevsky & Hinton 2009
CINICTO (Darlow et al., 2018). CIFAR100 (Krizhevsky & Hinton 2009), XChest (Wang et al.) 2017] and FEMNIST (Cohen et al. 2017) are used, where each individual image has a resolution of 32 x 32 pixels,
with 3 color channels. In the case of FEMNIST, we scale the 28 x 28 grayscale image to 32 x 32 and 3 channels to have the same NN structure, independent of the dataset type. Additionally, we do not split the written digits and numbers by writers, as proposed by [Caldas et al. (2019). but randomly distribute the images to the devices in case of iid to have an equal amount of data on each device. A mini-batch size of 32 is used for all experiments. For XChest (Wang et al. 2017) we sample 12.7K samples from the full available /no dataset and train for finding/In no finding. The images are downscaled to 256 x 256 with 3 color channels. Per round, each active device e E ((r) trains for one local epoch. We apply no data augmentation techniques.
Miscellaneous Hyperparameters for NLP Models: We evaluate our technique in an FL setup using Transformers (Vaswani et al.) (017) with two datasots. In the case of IMDB (Maas et al., 2011 a sentence- piece tokenizer is used with a vocabulary size of 16, 000 to detect if movic review is positive or negative. every character of the alphabet represents possible to-
In the case of Shakespeare (Caldas et. al. 2019
17

---

Published in Transactions on Machine Learning Research (06/2023)
80
0.5
accuracy [%]
= 0.4
75
0.3
70
0.2
65
60
min+tall
min+-max
round-robin-alll
random raxdon-tmxx +axx (CoCoFL)
max
round-robin--tmnn
random++al
Figure 6: Configuration selection heuristic ablation study evaluating with 1 e [0.2, 0.3, 0.4,
ken. For Transformer models the baselines HeteroFL and FjORD scale down the feature embedding and hidden size instead of training with subsets of CNN filters. To adjust to the resource requirements we use 12.5%, 40,, 62.5%, 81.25%, 100%] of the /enbedinnn dim, yet, for TMDB a large memory overhead remains, hence, weak devices have to be dropped from the training.
Centralized Experiments: For centralized experiments, we reduce the number of rounds R by factor of 10, hence, per experiment, train for R epochs over the full dataset. We adjust the learning rate decay steps accordingly.
C Configuration Selection Ablation Study
To verify the robustness of our configuration selection heuristic (Section specifically, the per-device per- round random selection of configuration out of Amax, perform several experiments. We reuse the setting from the ablation study (Section 6.3 using MobileNet with CIFARIO, 10% strong devices and 90% limited devices. The strong devices train the NN end-lo-end. We run several experiments with / E [0.2, 0.3, 0.4, 0.5] to verify that our heuristic is robust within a large range constraints (and available configurations) Firstly, we study the effect of our configuration reduction mechanism. Specifically, we compare:
max: Keeping only maximal configurations Amax CAy,ie. configurations that not subset of other feasible configurations (as used in CoCoFT.).
all: Keeping all feasible configurations Aç
We further compare our random approach against other mentioned baselines, such as
max: Using the configuration that trains the maximum number of blocks within the devico's capa- bilities. The combination of this selection mechanism and both reduction mechanisms (max+-max and mal+ result in the same configurations selected, hence, we only evaluate it once.
min: Training the configuration with the minimum number of blocks.
round-robin: Switching between feasible configurations in round-based manner (all limited de-
vices train the same configuration in round).
random: Randomly switching between configurations (as used in CoCoFL).
We provide the average final accuracy of three independent runs after 1000 rounds in Fig. 6 We observe that almost. all cases, using only maximal configurations (i.e., configurations that are not a subset of another feasible configuration) increases the final accuracy independent of the selection strategy. Further, observe that random+ (as done in CoCoFL), independent of 1, is within the highost-performing selection strategies. It can be observed from the results that training the same blocks on all devices does not improve upon random, as round-robin+ performs worse than random+ Depending on 1, max and min+-max can outperform + but not consistently throughout different values of (.
18


Reviews:
[{"review_id": "dc2M3Jwsr0", "reviewer": "Reviewer_mk4Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The motivation behind the proposed methodology is clear to me and makes sense. The proposed heuristics for configuration selection and the idea of freezing layers instead of removing them (as in subset selection methods) is novel.\n\n2) The authors show strong empirical evidence across a number of datasets and neural net backbones in Table 2 over competing baselines.\n\nWeaknesses:\nSee Requested Changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the Federated Learning (FL) problem in a synchronous setting where the client needs to send an update to a central server within a fixed time constraint per round. Under this setting, the authors present CoCoFL which allows edge devices with limited computational resources to train full neural networks. Along these lines the main contributions are:\n\n1) CoCoFL allows the client devices to train on the full neural network architecture by using only a subset of trainable layers during training. This is achieved by a combination of freezing, fusing and quantization of a subset of neural network blocks to reduce computational overhead. The rest of the blocks are trained at full precision.\n\n2) The authors rely on profiling to select plausible neural network configurations. To prevent combinatorial explosion, the authors propose a heuristic to select configurations which allow training contiguous range of blocks and then selecting a random configuration from the resulting set.\n\n3) The authors provide empirical justification for the method using neural network backbones like DenseNet, MobileNet, ResNet18 etc.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n\n1) The authors select a “block” as the most granular unit on which freezing and quantization are applied. However, a more granular unit might be a single neural layer. Is there a particular reason for selecting the former in favor of the latter apart from combinatorial complexity during configuration selection? If so, please include in subsequent revisions.\n\n2) Though the proposed heuristic (selection of training contiguous blocks) for configuration selection works well empirically, is there a particular reason for the choice of heuristic. If the authors tried some other heuristics which did not perform as well, it might be worth including a small ablation experiment (in the main text or appendix) regarding the same on a couple datasets.\n\n3) The authors discuss layer fusion in context of BatchNorm in Section 4.2. However, its fairly common to use variants like LayerNorm or GroupNorm. Moreover, Can the authors include the relevant analysis in subsequent revision.\n\n4) The paper is well-written and easy to follow. However, it might be better to discuss the fusion and implementation issues for attention blocks in language models in Section 4. The authors currently briefly state some of this in Section 6.2\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JsA4U611WZ", "reviewer": "Reviewer_zCQ5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper clarity, writing and exposition. The paper is easy to follow and a joy to read\n- Convincing experimental results\n- Real-world measurements taken\n- Code provided\n\nWeaknesses:\n- Novelty: The paper is an example for a well-executed study of a combination of existing ideas in the space. From a critical perspective, the major novelty lies in the heuristic for selecting sub-networks to train. TMLR does not require novelty per se, so I believe this is not a concern for publication. There is sufficient and insightful knowledge contribution. I certainly learned something.\n- Some details missing around BN and quantization\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to train a model through federated learning, specific to the situation where client devices are heterogeneous wrt. computing power and upload capacity. Their method, CoCoFL proposes a method which ensures that all devices return an update within a fixed time-window. Given the heterogeneous landscape, this is achieved by selecting specific subsets of the NN to train, while keeping the rest frozen & quantized. The frozen network components need not be communicated, therefore saving bandwidth. The authors propose a heuristic for selecting which network-components to freeze. \n\nEmpirically, the authors show that by enforcing $some$ update by each client within the given time-constraint, the weaker devices' data is represented better in the final model than with alternative methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would encourage the authors to consider:\n\n- BN discussion: The authors use and discuss BatchNorm. Specifically, they discuss \"Fusion\". While not stating it explicitly, it could be misunderstood that this is a contribution to the field - I would encourage the authors to either reformulate or add a citation. The authors claim that \"In summary ... reducing the number of MACs\". While fusion does reduce the number of operations, it does not change the number of MACs (BN is a scalar operation at test-time). Finally, the authors detail an equation for $y_{iCL}$, for the convolution, which includes a bias-term $b_i$. If followed by BN, adding a learnable bias-term to the linear layer is not necessary, as any bias is captured through $\\beta_i$ of BN. These requested change are not critical to securing my recommendation for acceptance. \n- BN in FL: For FL specifically, the use of BatchNorm is somewhat critical. Specifically (as in your experiments), non-iid data across the federation breaks the BN-assumption that mini-batch statistics represent the global statistics. There is empirical evidence that BN with non-iid minibatches doesn't work so well in general (https://arxiv.org/abs/1702.03275) and leads generally to worse performance than alternatives such as e.g. GroupNorm or LayerNorm. Furthermore, afaik it is reasonable to assume that transferring local BN-statistics to the server leads to worse privacy. I would ask the authors to clarify how the server computes the global BN-statistics in their experiments. Replacing BN with alternative operations would strengthen their experimental results and be more closely aligned with common practice in the field.\n- Exposition of quantized computations. I do not understand why the inputs to quantized operators are re-scaled on-the-fly while $s_o$ is transferred from the server. Assuming BN is fused, the input to a frozen layer \\bar{x}\\_{l-1} is equal to $ quant(f(o_{l-1})) $, i.e the output of the previous layer passed through an activation function and then quantized. How does the activation function influence the fact that $o_{l-1}$ needs to be specially treated as opposed to on-the-fly quantization of $f(o_{l-1_)$ as the input to the following layer? Similarly, why is the backwards-scale transmitted from the server? How does the server procure these scales? I feel like I'm missing something here and I would like the authors to provide more detail in order to secure my recommendation for acceptance. \n- In Figure 2 (right hand side), CoCoFL requires a higher memory-footprint than HeteroFL and FjORD for a given normalized training time. Why is that? (Not critical)\n- I believe the first claim you make about CoCoFL on page 2 to be somewhat misleading: \"In this paper, we propose a new technique CoCoFL (Fig. 1) that allows all devices to train the full model, irrespective of their capabilities, [...]\". I would argue that freezing parts of the model makes it such that a limited-resource device does exactly not train the full model. More nuance would be appropriate, such as for example; \"in expectation\". (Not critical)", "broader_impact_concerns": "No concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZXCpnPOLQj", "reviewer": "Reviewer_pD8x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Proposes a novel approach called CoCoFL for resource-constrained federated learning that adapts to per-device communication, computation, and memory constraints while still effectively learning from all data on all devices. \n\n2. Combines partial freezing and quantization to reduce training complexity and enable devices to select trained/frozen blocks based on their available resources to maximize accuracy under constrained resources. \n\n3. Outperforms existing FL methods in terms of accuracy and efficiency and achieves fairness (accuracy parity) by enabling constrained devices to contribute to the global model.\n\nWeaknesses\n1. Why int8 is the best choice? More discussions on FP16, FP8, Brain float should be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new federated learning technique called CoCoFL that adapts to per-device communication, computation, and memory constraints while still effectively learning from all data on all devices. The approach combines partial freezing and quantization to reduce training complexity and enable devices to select trained/frozen blocks based on their available resources to maximize accuracy under constrained resources. The paper presents a heuristic configuration selection algorithm and an aggregation method for partial updates. The approach is evaluated on various datasets and compared to existing FL methods, showing improved accuracy and reduced training time and communication costs. The contributions of the paper include the development of a novel approach for resource-constrained federated learning, the demonstration of improved accuracy and efficiency compared to existing FL methods, and the proposal of a heuristic configuration selection algorithm and an aggregation method for partial updates.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discussions on whether the int8 is the best choices, as we know the weights can be typically quantized to 4 bits; and also why not consider low bitwidth floating point.\n2. For some devices, the processing unit/chip of float32 and int8 are different whether the proposed method will increase the communication overhead and degrade performance.", "broader_impact_concerns": "no obvious concerns on broader impact\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an issue in federated learning, where devices have different resources and need to finish training within the same deadline. Existing methods that drop neurons or filters from the neural network on constrained devices lead to inaccuracies. The authors propose a novel federated learning technique that maintains the full neural network structure on all devices by quantizing selected layers to reduce resource requirements while still allowing high accuracy. 

Initial concerns include the demand that the paper should provide more explanations on why int8 is chosen as the best option and address potential performance issues when using different processing units. The authors argued they use int8 quantization in CoCoFL because it is fully supported by PyTorch, allowing for accurate measurements on real hardware. They acknowledge that other quantization levels may impact accuracy and enable training on less capable devices. However, PyTorch currently lacks support for these quantized operators on GPUs or dedicated accelerators. Questions on architectural choices were addressed in great detail, and additional ablations were provided. 
Decision: Accept (poster)