Resource-Adaptive Federated Learning with All-In-One Neural Composition
Authors: Yiqun Mei
Pengfei Guo
Mo Zhou
Vishal Patel
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
System Heterogeneity

Abstract: Conventional Federated Learning (FL) systems inherently assume a uniform processing capacity among clients for deployed models.  However, diverse client hardware often leads to varying computation resources in practice. Such system heterogeneity results in an inevitable trade-off between model complexity and data accessibility as a bottleneck. To avoid such a dilemma and achieve resource-adaptive federated learning, we introduce a simple yet effective mechanism, termed All-In-One Neural Composition, to systematically support training complexity-adjustable models with flexible resource adaption. It is able to efficiently construct models at various complexities using one unified neural basis shared among clients, instead of pruning the global model into local ones. The proposed mechanism endows the system with unhindered access to the full range of knowledge scattered across clients and generalizes existing pruning-based solutions by allowing soft and learnable extraction of low footprint models. Extensive experiment results on popular FL benchmarks demonstrate the effectiveness of our approach. The resulting FL system empowered by our All-In-One Neural Composition, called FLANC, manifests consistent performance gains across diverse system/data heterogeneous setups while keeping high efficiency in computation and communication. 

Resource-Adaptive Federated Learning with All-In-One Neural Composition
Yiqun Mei
Pengfei Guo
Mo Zhou Vishal M. Patel
Johns Hopkins University
vpatel36)@jhu.cc
Abstract
Conventional Federated Learning (FL. systems inherently assume uniform pro- cessing capacity among clients for deployed models. However, diverse client hardware often leads varying computation resources in practice. Such system heterogeneity results in an inevitable trade-off between model complexity and data accessibility bottleneck. To avoid such dilemma and achieve resource- adaptive federated learning, we introduce a simple yet effective mechanism, termed All-I-One Neural Composition, to systematically support training complexity- adjustable models with flexible resource adaption. models at various complexities using unified neural basis shared among clients, instead of pruning the global model into local ones. The proposed mechanism endows the system with unhindered access to the full range of knowledge scattered across clients and generalizes existing pruning-based solutions by allowing soft and learnable extraction of low footprint models. Extensive experiment results on popular FL benchmarks demonstrate the effectiveness of our approach. The result- ing FL system empowered by our All-In-One Neural Composition, called FLANC, manifests consistent performance gains across diverse system/data heterogeneous setups while keeping high efficiency in computation and communication.
able to efficiently construct
Introduction
The success of deep learning is greatly empowered by the large datasets 2 However, when data scattered across vast number of edge devices, and subject to privacy regulations, deep learning will be unrealistic. To this end, Federated Learning (FL) 3 provides decentralized solution, enabling a number of participants to jointly train a model without exchanging any raw data. Due to such privacy-preserving property, FL is favorable and widely adopted in many applications 6 g I, including but not limited face recognition prediction [13].
autonomous driving 12 and next-word
When designing federated learning systems [14 LS
17 18 it is commonly assumed that the local models share the same architecture as the global one. However, real-world deployments the wild often equip with diverse
3
can rarely satisfy this ideal assumption. Instead, participants devices (e.g., Internet of Things (IoT)) devices, mobile phones, tablets and personal computers), which greatly vary computation resource budgets (including computation capacity, memory, storage, and network bandwidth). Further, even if participants equip with identical devices, the resource budgets may still vary due to e.g., other irrelevant programs consuming an arbitrary portion of the resource, or reduced CPU frequency due to power-saving mode. This ubiquitous presence of system heterogeneity becomes a major bottleneck for real-world FL applications, as it leads to an inevitable trade-off between model complexity and data accessibility service providers have to either sacrifice model capacity to enable training on indigent devices, or exclude them along with their unique data. Predictably, in such dilemma, both choices will eventually harm the performance of the resulting model.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Server
Global Pamments
Locd
Clients
Server
Neural Basis (shared)
Load Medk Parametees
Clients
diate
To dlicre
LOX
1.0X
To clbaa
cliee
0.25%
ipáit
0,25X
(2) HeteroFL and FjORD
(b) FL with All-In-One Neural Composition
Figure I: Comparison of different system-heterogeneous strategies. Schematics are demonstrated
using one hidden layer two different types of devices A and B. (a) HeteroFL [19 and FjORD [20] prune excessive parameters to adapt to a smaller device. In this case, a large portion (more than 90%)
of parameters between 0.25 and 1.0x nets cannot learn from the data on devices type B. (b) our
approach allows all parameters to leverage and learn from the full range of data.
The existing FL methods with heterogeneous clients, Fe 19 and FjORD 20 propose to modify the width of deep neural networks, i.e. to extract low-footprint sub-models by pruning excessive channels. However, find their performance improvement is inconsistent, which could be even worse than naively deploying the weakest network with the vanilla FedAvg [15]. As shown in Figure ! we use an FL setting with two types of devices A and B, where the high-end device type A takes up 25% clients and can run the full model (1.0x in terms of the model width), while the low-end device type takes up the majority (75%) of participants but can only afford 0.25x of the full model. A counter-intuitivy result is observed with standard ResNet- 18 [1] and CIFAR10 [22 dataset: the Top-1 accuracy of both methods decrease from 86% to 84% when increasing model capacity from 0.25 x to 1.0x. Based on this, we conjecture that the extra 0.75x network capacity does not benefit from all participants' knowledge. Even if all participants contribute to the same global model, pruning in fact limits the contribution of weakest devices merely the 0.25x network, leaving the majority of parameters under-optimized. This can be even more severe when there is data heterogeneity, where inadequately trained parameters may lead poor generalization against unseen classes, and hence worse the overall performance.
Given heterogeneous devices, how to unleash the full potential of resource capacity without sacrificing data accessibility? In this paper, we propose All-In-One Neural Composition, a new scheme for run- auntime resource-adee addativee model construction, which formulates networks at different capacities (thus footprint and data transfer cost) as linear combinations of one unified set parameters, namely neural basis. It is designed to be compact, and regularized to be linearly independent, and hence is efficient in training and communication, as well as more faithful model constructions. With such scheme incorporated, the resulting FL. framework (called FLANC) provides systematic support for training sunplexiy--adjustabee neural networks, and thus enables efficient and more effective ardapice federated learning. Different from muning-baseh methods [19 20), it makes knowledge contribution from clients to model parameters unhindered from both computational resource constraints and model architectural complexity. Our method be seen as generalization of existing solutions by extending hard-coded weight masks to soft and learnable extractions of lower footprint models. In addition, similar to previous approaches [19 20), original models can be pre-composed for inference and executed as normal networks without any run-time overhead.
To validate our approach, we conduct extensive experiments under both statistical data heterogeneity (IID and non-llI) distribution) and system heterogeneity (static and dynamic) settings. We evaluate method on commonly used datasets, i.e., Mashion-MNIST" image classification, as well as Shakespeare [24) for next-character prediction. Although the proposed mechanism is simple in its formulation, it is stable in training, while consistently and significantly
[23), CIFARIO and CIFAR100 (2) for
outperforming the previous system-heter heteem--hterogeeoous strategies [9 20
across different architectures.
2
The Proposed Method
2.1 System Heterogeneity
As aforementioned, conventional FL systems assume a uniform processing capacity for all participants. Nevertheless, edge devices in the wild often equip with diverse hardware and may have very different computational resources and data transfer budgets. Thus, it is crucial to properly model system heterogeneity. When reflecting this concept the deep networks, resource constraints can be

---

R-
Ra
Ulax
Local Model Parumeters
UP S,(R, x
Denuse Tyype
Rg
Sy/Rt
Rechpe
S
W
VwWWw
Capacity-spocifee
Nakale
share
Figure 2: All-In-One Neural Composition using the shared neural basis Vshare and capacity-specifi tensor Up Note, in the rightmost reshaping operation, Sp R, Sp/Ry.
expressed in multiple aspects, such as depth (number of layers) and width (number of hidden channels). In this paper, we follow [9 20 and choose adjust width for resource adaptation. Compared to reducing depth, a reduction in width can more effectively reduce parameters and memory footprint during inference, which is beneficial to the edge devices. Meanwhile, the resulting networks belong to the same model class and share similar intrinsic characteristics, which is preferred
for stabilizing training and model aggregation [9 20 25 26
We denote the ratio as a rescaling factor of the number of active channels in layer W E RSXT Both the model size and computational cost of slimmer p-width model will be reduced by p². To model device in the wild, one can assign each device k with maximum affordable capacity ratio ps according its hardware configuration. Since is impractical to exhaustively list all possible values of p due to the diversity of edge devices" we consider a set of representative capacities P (e.g. {0.25, 0.5, 0.75, 13) by pre-clustering participating devices into [P] resource groups according to their capacity.
2.2 All-In-One Neural Composition
Previous research suggests that the weights of fully-connected and convolution layers are usually over-parameterized, and lie on low-rank subspace [27]. This makes it possible to express these layers in the low-rank tensor format using decomposition techniques [28] 29]. Given convolution weight W xSxT with filter size k, input channel number and output channel number T, we can approximate it as W = V U with V E R X R U E RRxT and R < T. The factorization for fully-connected layer is equivalent the k case. Such low-rank approximation is effective for model compression in order to reduce model size and computation cost [30 31
32 33 Similarly,
speculate federated learning with system heterogeneity could benefit from tensor decomposition.
Thus, we introduce All-In-One Neural Composition, that employs low-rank approximation to repre- sent networks in different widths with unified expression. For arbitrary p-width network, decouple its weight W, E R&* xSpxTp shared tensor Vahare and capacity-specific tensor Up. i.e.
W
Vatare Up
(1)
However, Eq. I cannot be directly used as varying not only changes the output channel but also rescales the input channel S by p. which makes sharing V Vahure of default shape k² S R inapplicable. We address this issue by adjusting Vahare to be more fine-grained. We demonstrate this process in Figure 2 Specifically, given Vshare with shape k22 R₁ R2, we select R2 from the common divisors of all possible input channel Sp with p E P. Correspondingly. the capacity tensor Up has the shape R₂ x (Sp/ R1 Tp). Their multiplication result is reshaped weight matrix W, E IR ((R x (S/RR)xX and can be used for the original p-width network.
Eq. in fact means to compose Wp with shared basis Vatare and coefficients Up through linear combination. Specifically, every weight fragment W.pp E R W is explicitly written as the weighted sum of It, bases Vahare {Vylū (I,...,RA, R2}} with coefficients Ui i.e.
Rs Σ 1=1
W.p
U
where Ut.p is the i-th column of Up. and V, E R the j-th basis vector in Vituee
For example, there are 24,000 unique Android devices by 2015.
3

---

Eq. (2) reveals an outstanding property that the knowledge learned by Vstare can be propagated to all parameters. In the context of FL, this enables every parameter to access the full range of knowledge by training the basis Vahare on all devices. In addition, our formulation also reduces both computation and communication cost as the low-rank tensor format inherently more efficient.
Comparison to low-rank model compression. Low-rank factorization is generally applied as post-processing step to the pre-trained weights for model compression [34] 28 30 29] Most methods produce a single compressed model and require access to the labelled data for fine-tuning. Hence, in the context of FL, the standard compression techniques neither address system heterogeneity nor reduce the footprint and data transfer cost, because they are only applied after the training process. Moreover, the requirement of fine-tuning contrast, our method is applied during the training process, and is able to flexibly construct multiple networks and can be stably learned from scratch without post-calibration.
often infeasible for sevvacyprrseevingg applications. In
FL strategies. Current restem-heterogeneee are pruning-bai based and can be viewed as a specific instantiation of Eq. ( by setting Vshare to the largest weight Wpnee and the coefficients Up to be a hard-coded mask with 0 and entries, indicating whether a parameter is pruned or not. From this viewpoint, our method generalizes existing solution by allowing (1) more compact basis Vahare. which ensures high efficiency and reduces data transfer cost; and (2) learnable soft coefficients Up. which makes both access to the full range of knowledge and adaptive utilization possible. Namely, unlike pruning- boasnd methods [19] 20] where the excluded parameters cannot benefit from data on smaller devices, our approach enables every parameter at all width E P to leverage the full range of knowledge from all clients, no matter whether it can be trained on that device or not.
Comparison to pruning-basee
strategies 19 20
2.3 Orthogonal Regularization for Enhancing Representation Capacity
The low-rank tensor format of Eq. (2) can also be interpreted as adaptively searching for parameters from a low-rank subspace S = span { Vahare Spontaneously, we enforce these basis vectors to be linearly independent, they can form a more expressive subspace and are capable to cover a broader range of knowledge. Nevertheless, according to our observation, this preferable property cannot be obtained implicitly with standard training objectives alone.
To this end, introduce an extra orthogonal regularizer, which enforces the columns of VT to be mutually orthogonal with unit norm. We achieve this by reducing the L2 norm of the difference between Gram matrix and identity matrix. To train a p-width network, one can jointly optimize the orthogonal regularizer and the original classification objective, which can be expressed as:
Cp = Cels Vi Viharee Up) +
|| [4] Vahare !1 - 211
(3)
Here for simplicity, we use Vahare and Up to denote the collections of basis and coefficients over all layers. Cots is the classification loss. l is the layer index and is a balancing parameter.
2.4 Federated Learning with All-In-One Neural Composition (FLANC)
In this subsection, we employ our composition technique into the FL framework to enable resource- adaptive federated training. We abbreviate our framework as "FLANC" for convenience.
As aforementioned, the first step of our framework is to obtain a feasible set of device capacity by clustering participants into [P] groups based on their available resource. Meanwhile, the server also assigns each client & with a corresponding capacity ps based on the clustering results. Next, the server initializes the shared basis 1/0 well as all capacity-specifie coefficients U with p E P. worth noting that our server does not need to explicitly maintain the global networks during training. The data transfer cost is further reduced with the efficient tensor format.
communication round & starts by determining a set joining devices K,, After that, the server broadcasts the basis share to each client & E Kg along with the associated resource-specifie coeffi- cients U/ for ad-hoc adaption. On the client side, each client updates the model with their private data for N local iterations. For each training step, Pk- -idth network is adaptively constructed using
To compute the Gram matrix, neural bases first transposed into columns, i.e., ViTue

---

Algorithm 1: Federated Learning with All-In- Aal-In-One On- Neural Composition (FLANC)
Input: share neural basis; Up, .fag..., Up. with p: E P. [P] capacity-specific coefficients; 7
communication rounds; N local iterations
for L <- 0 to - do // Global communication rounds
Server determines set of joining devices K₁;
Server broadcasts the neural basis Vanaee and coefficients Uz, to each client & E Kr.
for k E Kr do // do in parallel among clients
for 0 N do // local iterations for every client
Client composes the pi- pi--width network using Eq. (
Client computes the loss objective using Eq. and then updates Vitk and Upe:
end
Client uploads the updated neural basis and capacity-speeiffi coefficients the server;
10
end
11
Server collects the updates of the shared hasis and coefficients using Eq.
12
end
Eq. (1) Therefore, the basis Vahare can be optimized by all clients with their local data. At the end of training, the clients send the basis and coefficients back to the server.
The server finishes a communication round by updating the shared basis and coefficients using local training results from clients. In our approach, the basis is fully trained on all participating devices. On the other hand, capacity-specific coefficients updated by the devices with the corresponding resource group. To account for this difference, we provide the following aggregation rule:
V++1=
Σ keK.
D share
and U++1 1
Σ kEKr,
D
(4)
Dxc,1
|KRPP
where DK, UkeK, De is the union of datasets of current participants. And Ke,p (k|}: K2, Py p} is a subset of Xc, with the device capacity equals p. The complete workflow of our method is summarized in Algorithm
Experiments
Datasets. In this paper, evaluate method for image classification tasks on three popular
datasets with increasing complexity: Fashion-MNIST (3). CIFARIO and CIFAR100 (22]. Fashion- MNIST is a relatively simple dataset containing 60,000 examples of 10 classes. CIFARIO and CIFAR 100 are the common classification benchmarks with 50,000 training images with 10 and 100 classes respectively. CIFAR 100 is the most challenging dataset each class only has 500 images. Our approach is generally applicable to not only vision tasks, but also natural language processing tasks. To show this, we further conduct experiments on Shakespeare, which is a text dataset built from Shakespeare Dialogues (35), and the task is next-character prediction. More details about these datasets can be found in the supplementary material.
Data Partition. In our experiment, we follow [19 and consider both cases of IID and non-IID data partition, in order comprehensively showcase the performance. For IID partition, uniformly sample a same number of data for each client. For non-lID case, we assume label shifts and distribute subset of classes for each client. Specifically, the number of classes is and CIFAR10, 10, 30 for CIFAR100 For Shakespeare, we follow the partition method in [24 We refer the underlying distribution of raw data (cach speaking role) as the non-III partition and each data point is equally likely to be sampled in the IID partition.
3 for Fashion-MNIST
Model Architecture. Our approach seamlessly work with both convolutional and fully-connected layers. In this paper, we evaluate it across several different architectures. Specifically, we adapt the standard 18 18 [2] for CIFAR10 and CIFARI00. For Fashion-MNISS we adopt simple 4-yayer CNN. For characharr prediction, we apply our approach upon a 1-layer RNN. Further details can be found in the supplementary file.
Client Partition. For all experiments, we follow previous conventions [5] and assume totally 100 clients with 10% of them being active for each communication round. To model system heterogeneity,
5

---

Hatarol FLANC
50.2
47.3
62
40.2
02.5
460
42
37.5
FLANC
FLANC
43
35.2
FLANC
3.25
0.5
0.25
0.75
1.25
0.5
6.75
1.25
0.5
6.75
10
Copurdy 1o
Capaoty M0
Capurty
Capectty
CIFARIO
(b) CIFARIO Non
CIFAR
(d) CIFARIOO
00.5
Pelerrp
56.5
PyCno
FLANC
-Magttve Ku
-1.4
00.5
83
-50
-3.2
retrredd
-22
113R0
FORO
FLARC
TLANC
0.25
0.5
0.75
1.0
0.25
6.75
3.0
0.5
0.5
5.75
10
025
0.75
10
(c) Fashion-MNIST HID
(f) Pashion-MNIST Non-IID
(g) Shakespeare ITD
(h) Shakespeare Non-IID
Figure 3: Test results for static system-heterogenenue setting. Top-1 1 (f) accuracy and negative
perplexity (1) are used for image classification and next-character prediction tasks respectively.
we construct 4 different complexity levels with P = {0.25, 0.5, 0.75, 1.0). To fully demonstrate the effectiveness of our approach, we consider both the static and dynamic capacity distribution. For static setting, we uniformly assign capacity level for each client, and then client keeps this budget throughout the training. For dynamic setting, we allow each client to randomly increase/decrease its capacity at every communication round. This is also realistic scenario as devices' capacity can vary drastically due to other irrelevant system processes or CPU frequency changes led by the battery status.
Baseline Methods and Implementation Details. We compare our method with two state-of-the-art system-hhterooeeneon solutions HeteroFL and FjORD(20], Both approaches are similar in design and are based on model pruning. Other generic FL algorithms [15 14 36 are inapplicable with the presence of heterogeneous devices due to broken assumptions. Since the official code for FjORD [20] absent, we re-implement this method following the author's 's descriptions and use the default parameter settings including the uniform dropout rate. For hyper-parameted tuning, we split subset of 10% training examples as the validation set. After selecting the parameters, validation data are merged back to the training set, and then we retrain the model for final performance evaluation. For our method, the selection of Rj and R2 depends on architecture and tasks. We implement the proposed approach using Py Torch 1.8.2 Nvidia A5000 GPUs. Detailed descriptions of tlyper-parameted and training can be found in the Appendix.
3.1 Main Performance Evaluation
Here we compare our approach with HeteroFL 19 and FjORD[20] on standard benchmarks. All results with static setup are shown Figure Results with dynamic setting are reported Table
In both sestem-heteroogeeouss settings, our approach consistently outperforms both HeteroFL and FjORD on all entries. The improvements is further magnified when dealing with heterogeneous data. Specifically, we found the performance of HeteroFL 19 and FjORD (0) very close in many cases. This is consistent with our expectation both methods based on parameter pruning, even FjORD (20) in addition allows large devices to randomly switch to a smaller sub-model through ordered dropout. However, we do observe consistent improvements with this dropout technique, and in some cases it further harms the performance of the larger models. And both HeteroFL and FLANC are compatible with this technique. Furthermore, when switching from IID to non-IID data partition, both approaches suffer from significant performance drop. For example, on CIFAR 10, the overall performance of both FjORD and HeteroFL drops to 60% even with their largest model. This is mainly because pruning restricts parameters to have a limited access to the full range of data and thus may not transfer unseen distributions. contrast, our approach allows every parameter to leverage full range of knowledge through the shared basis, and hence is more robust to data heterogeneity. As result, our FLANC achieves around 13% and 9% performance gain compared to HeteroFL and FjORD respectively.

---

Table Experimental results with dynamic cesstem-heterogeeeii For image classification, we use the standard Top-1 accuracy (%) as the metric. Perplexity is used as the metric on Shakespeare. The 'M' and 'K' denote x x10% and x 10 respectively.
Top-1t/ / Peplexity LID
Top-If/Papplxitty Non-IID
Communicatiio Cost
Width
HIteroFI. FjORD FLANC
HeteroFt. FjORD
FLANC
p-width
FLANC
CIFARIO with ResNet. 18
x0.25
81.9
85.1
85.1
$9.9
51.7
63.7
0.7M
0.5M
x0.50
85.0
58.4
x0.75
87.4
86.8
88.8
57.5
66.2
70.6
6.3M
2.4M
x1.0
88.1
86.7
89.5
56.7
65.7
73.0
11.1M
4.0M
Avg
85.6
86.3
87.8
56.2
60.5
69.3
5.2M
2.0M
CIFAR 100 with ResNet- 18
x0.25
49.2
52.5
54.2
44.2
46.7
48.3
0.7M
0.5M
x0.50
56.5
54.9
61.8
50.6
47.5
54.7
288
1.2M
x0.75
61.3
55.8
64,2
52.1
49.2
56.5
6.3M
2.4M
x1.00
63.6
Avg
$7.7
54.8
61.4
49,9
48.1
54.2
5.2M
2.0
Fashion- Fashion-MNIST MNNIST with CNN
x0.25
81.6
x..50
90.5
88.6
91.1
83.7
83.1
85.7
106K
63K
x..75
90.5
89.1
91.2
85.5
186K
x1.00
91.1
89.1
91.4
84.6
83.7
86.8
285K
136K
Avg
Shakespoure with RNN
x0.25
5,54
5.54
5.17
5.55
5.54
49K
48K
x0.50
5.07
5.06
4.74
5.09
5.05
4.80
214K
187K
X0.75
4.86
x1.00
4.64
4.67
4.52
4.69
4.73
4.54
787K
740K
5.03
5.04
4.77
5.06
5.07
4.80
373K
348K
In terms of efficiency, all three methods are able straightforwardly reduce the amount of data to transfer and the cost of computation by switching to smaller p-width models. However, benefiting from the efficient tensor decomposition format, our approach can further reduce the model size, resulting in the best communication efficiency.
3.2 Robustness with Imbalanced Device Distribution
In practice, there is usually no guarantee that different types of devices will be equally distributed. Here we conduct experiments compare the robustness towards imbalanced device distributions. To make things more challenging, we also assume there large capacity gap between clients. Specifically, we consider two types clients A and B, where can run the full model with p=1 and B has a limited support of p=0.25. We consider two scenarios where A accounts for 25% and 75% of total devices respectively.
The results are illustrated in Figure 4 It is noted that, with a limited number of available devices of type A, HeteroFL and FjORD cannot effec- tively improve the accuracy by increasing the capacity, but instead suffer from performance degradation. This is not surprising the major- ity of parameters (>90%% is under-trained with con-
7010
only 1/4 portion of all data available. trast, our approach is still able to obtain perfor- mance gain with additional parameters. This proves that our approach indeed is more favor- able for addressing system heterogeneity issues for FL. On the other hand, when the majority of devices can run the original model, the perfor- mance gap between ours and HeteroFL. shrinks as expected, since in this case most of the pa- rameters can directly access all data. In contrast,
(a) A:25% B:75%
(b) A:75% B:25%
Figure 4: Comparison with HeteroFL and FjORD for imbalanced client distributions. Results are reported on CIFARIO under dynamic system het- erogeneity and IID data partition. (a) Majority participants are indigent devices of type B. (b) Ma- jority clients are high-end devices of type A.
7

---

(a) Layer p. 0.25
(b) Layer p, 0.5
(c) Layer 0.75
d Layer 1. 1.0
30
42
111
Layer 3. p
0.25
Layer p
gg Layer p
0.75
Layer
Figure 5: Statistics of mean coefficient value. Results for the first convolution in the corresponding residual layer reported.
(a) Layer (wro Onth)
(b) Layer 2 (w/o Orth.)
(c) Layer 3 (w/o Orth.)
(d) Layer (wio Orth.)
(e) Layer Orth.)
Layer Orth.)
(g) Layer Orth.)
(h) Layer Orth.)
Figure 6: Visualization of the Gram matrix (of every first convolution in each residual layer) when w/ or w/o orthogonal regularization. Gram matrices for Layer 2-4 are shown histograms because they are too large to clearly visualize.
FjORD squeezes its performance into some intermediate points due to dropout. While improves the performance of the smaller model, the benefits from larger capacity become marginal.
3.3 Knowledge Sharing
Here we study the effectiveness of FLANC in terms of knowledge sharing. One can imagine that all values in set of coefficients (a column of Up) are close to zero, the proposed neural composition will degenerate to pruning. To show that this not case, we report the statistics of the mean absolute value for each column in the coefficient matrices Up. As shown in Figure most values are not small and zero-centered, indicating knowledge stored in the basis can be well-shared across models.
3.4 Ablation Study
Orthogonality. Orthogonal regularizer reduces the redundancy and dependency within neural bases and improves the expressiveness of neural composition. To demonstrate its effectiveness, we remove orthogonality regularization and compare the performance on CIFAR10. As shown Figure 7a one can observe that our regularizer constantly improves performance across all model capacities. We further visualize the Gram matrix of the learned basis in Figure 6 It can seen that the Gram matrices of our approach are very close to the diagonal matrices, demonstrating the basis is indeed

---

orthogonal. It is worth noting that such property cannot be learned naturally with purely classification loss.
Basis Size R1. In our implementation, we set the basis size R1 the common divisor of the input channels over all models. Thus, we can compose all networks with one unified basis. To study our model's sensitivity to the choice of R1, we set to 25%, 50%, 100% of the smallest incoming channel, respectively. As shown in Figure [7] using fine-grained basis can slightly improves the accuracy. The performance peaks 50% of the smallest incoming channel.
(a) Orthogonality
(b) Basis size
(c) Number of bases
Figure 7: Ablation study of FLANC. Results are reported on CIFARION with IID data partition and dynamic system heterogeneity.
Number of Bases Rg. Increasing the number of bases inherently enlarges the dimensionality of the subspace and may improve performance by providing better representational capacity. We study its effect and consider R2 to be 25%, 50% and 75% width of the original model, respectively. As shown in Figure Results suggest that a larger number of bases can improve the performance but the benefit becomes marginal when further increasing from 50% to 75%. In addition, using more bases also linearly increases the model size. For our architectures, we select R2 with the best trade-off between the performance and the number of parameters.
Related Works
Data Heterogeneity FL. the de facto FL algorithm (FedAvg (55), clients conduct local training with the received parameters from a central server. Then the server collects all the locally trained models and aggregates them into a new global model. The design of FedAvg [55 relies on the assumption that data is uniformly distributed across clients. However, the assumption does not hold in real-world scenarios where the underlying data distribution is unknown and very likely non-IID. The non-IID distribution of clients' local data is notorious trap in federated learning and results in the non-negligible client drift issue, which can jeopardize the convergence rate and model performance when the data similarity decreases (37]. Tackling the statistical data heterogeneity attracts extensive attention in recent years. Pioneer works, like FedProx [8), introduce additional regularizers on the local training objective to prevent the local models from diverging due to the non- IID data distribution.
Later on, several works propose inter-client variance reduction techniques [36] 16, 33 by amending the client drift issue with the predicted model updating direction. Personalized FL 440 42 43)
is another strategy to allow a model to better fit the local data distribution on a specific client. One can fine-tune the global model on the local data (44), perform MAML-based personalized approaches [40] 41), or achieve the personalization by local batch normalization layers [14). Our proposed method is perpendicular to the above studies and potentially can be combined with them for further improvement.
System Heterogeneity in FL. In contrast to the substantial effort devoted data heterogeneity, addressing FL system heterogeneity is largely under-explored. The majority of existing methods only focus on reducing communication cost, so that the data transfer cost can be affordable by those edge devices with constrained bandwidth and energy consumption. For example, several advanced FL
optimizers 16. 36 45 46 have been proposed to improve the convergence rate, and thus reduce data
transfer cost by decreasing the number of required communication rounds. Other methods [47] 48 49
50 improve communication efficiency by combining quantization [47 49 and sparse training 48
techniques. For example. LotteryFL 43 and FedMask [42) learn personalized sparse sub-networks achieve high communication efficiency. However, all aforementioned approaches still assume uniform processing capacity among all participants, which cannot be guaranteed for real-world scenarios. One promising way to remove this assumption is leverage knowledge distillation (KD) for learning the central model instead of conventional coordinate-wise averaging. While such methods [52]
55, 56) allow different local models, applying KD often requires additional access to some common public datasets to alleviate the difficulty in knowledge transmission [57]. which is impractical in FI.. For the most recent methods, Split-Mix (58) explores model ensemble by re- re-mixing booget-oomppaible sub-networks at inference time, where its effectiveness is

---

potentially constrained by the insufficient capacity of individual sub-models. FedHM [59] explicitly factorizes global models into low-rank sub-models accommodate client capacity, which has shown to be effective in reducing computation and communication costs. However, the SVD scheme preserves feature width before and after decomposition and thus are lack the ability to scale (peak) memory footprint and may harm the performance by introducing additional factorization error. Our method is most related to HeteroFL [19] and FjORD [20], which allow federated training of local models in different widths adaptively "pruned" from a global model. Nevertheless, discussed before, they cannot fundamentally resolve the challenges and still suffer from degraded performance similar to many conventional FL methods.
Compression Techniques. Both our approach and previous heterogeneous--yytne solutions are related to model compression techniques [34] 30 [9 60]
61 62), a classic research field to reduce the run-time storage and the latency of deep neural networks. Some of the most representative techniques
include quantization [62 63]. neural pruning 660 [1. and tensor decomposition 34 30 29] In the
context of FI., these techniques are first studied and demonstrated to be effective reducing the communication cost 47
48 49 50]. To handle system heterogeneity, current approaches [99 20
extend model pruning techniques to extract sub-models to fulfill the resource requirements. However, as discussed, pruning simply drops model parameters, preventing the dropped ones from leveraging complete knowledge scattered across various devices. On the other hand, our method can be connected to tensor decomposition, where we express layers unified neural basis representing shared knowledge and capacity-speciin
coefficients for linear composition. In this way. it not only allows resource-adaptive adaptive model training through composition, but also naturally ensures the complete knowledge effectively propagates to models at various complexities.
Factorization-based FL Methods. In this work, the low-rank nature of deep models provides foundation for using a unified neural basis to express knowledge and compose heterogencous models. Compared to existing factorization-basel approaches [59 64], our approach does not perform explicit low-rank decomposition but instead only uses this notion conceptually, i.e., FLANC directly learns the basis and coefficients from scratch rather than factorizes them from existing kernels.
This brings several advantages compared to previous approaches performing real low-rank decom- position: (1) FLANC avoids approximation errors of factorization, which will potentially harm the performance. Meanwhile, it can be applied to every layer for better complexity scaling. contrast, SVD based factorization inevitably introduces an approximation error that will be accumulated and propagated to subsequent layers. This limits their applicability to the last several layers and reduces practicality for real-world deployment, as early layers can also become computational and memory bottlenecks, for example, when dealing with large images. (2) FLANC can handle system heterogeneity by constructing client models with different widths, whereas factorization methods cannot achieve the same. Explicit factorization requires the resulting tensors to have aligned input and output dimensions (e.g.. m and r x n) with the original kernel (c.g., change run- cun-time width to effectively adjust memory consumption. FedDLR [64] and FedPara [50] further assume a uniform processing capacity over devices, making them infeasible for dealing with system heterogeneity.
ni), which cannot
5 Conclusion
The assumption of uniform processing capacity in conventional federated learning contradicts the varying computation resources in practice. As result, these methods suffer from an inevitable dilemma between model complexity and data accessibility. To overcome this problem, we propose simple yet effective mechanism, termed All-In-Ons Neural Composition to systematically sup- port aisource-adaptive federated learning. Unlike previous methods, our method enables flexible constructions of models in different complexities, and allows unhindered access to the full range of knowledge scattered across clients. Comprehensive experimental results on vision and language tasks demonstrate the effectiveness of our method, manifesting consistent performance gain across different setups.
Acknowledgments and Disclosure of Funding
This work was supported by an ARO grant
10

---

References
[1] Yandong Guo, Lei Zhang. Yuxiao Hu, Xiaodong Hc, and Jianfeng Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pages 87-102. Springer, 2016.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[3] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3)550-60, 2020.
[4] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
[5] Pengfei Guo, Puyang Wang. Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel. Multi-institutional collaborations for improving deep learning-bane based magnetic resonance image reconstruction using federated learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2423-2432, 2021.
[6] Aiden Doherty, Dan Jackson, Nils Hammerla, Thomas Plötz, Patrick Olivier, Malcolm H Granat, Tom White, Vincent T Van Hees, Michael I Trenell, Christoper Owen, al. Large scale population assessment of physical activity using wrist worn accelerometers: the uk biobank study. PloS one, 12(2):e0169649, 2017.
[7] Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcure Informatics Research, 5(1):1-19, 2021.
[8] Yiqun Mei, Pengfei Guo, and Vishal M Patel. Escaping data scarcity for high righurosolution heterogeneous face ballucination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18676-18686, 2022.
[9] Pengfei Guo, Dong Yang. Ali Hatamizadeh, An Xu, Ziyue Xu, Wenqi Li, Can Zhao, Daguang Xu, Stephanie Harmon, Evrim Turkbey, et al. Auto-fedrl: Federated hyperparameter optimization for multi-inse nulti-instiuutioaal medical image segmentation. arXiv preprint 22Xiv:2003.06:: 2022.
[10] An Xu, Wenqi Li, Pengfei Guo, Dong Yang. Holger Roth, Ali Hatamizadeh, Can Zhao, Daguang Xu, Heng Huang, and Ziyue Xu. Closing the generalization gap of cross-silo federated medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20866 200066220755222 2022.
[11] Qiang Meng, Feng Zhou, Hainan Ren, Tianshu Feng, Guochao Liu, and Yuanqing Lin. Improving federated learning face recognition via privacy-agnostic clusters. In International Conference on Learning Representations, 2022.
[12] Yijing Li, Xiaofeng Tao, Xuefei Zhang, Junjie Liu, and Jin Xu. Privacy-preserved federated learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2021.
[13] Andrew Hard, Kurt Partridge, Cameron Nguyen, Niranjan Subrahmanya, Aishanee Shah, Pai Zhu, Igna- cio Lopez Moreno, and Rajiv Mathews. Training keyword spotting models on non-iid data with federated lcarning. arXiv preprint arXiv:2005. 10406, 2020.
[14] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang. Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07622 2021.
[15] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273- 1282. PMLR, 2017.
[16] Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Scbastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[17] Jaehoon Oh, SangMook Kim, and Se- Young Yun. FedBABU: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2022.
11

---

[18] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002. 06440. 2020.
[19] Enmao Diao, Jie Ding, and Vahid Tarokh. Heteroft: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
[20] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Lcontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for henchmarking machine learning algorithms. 2017.
[24] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for foderated settings. arXiv preprint arXiv:1812.01097, 2018.
[25] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018.
[26] Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1803-1811, 2019.
[27] Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predicting paramcters in deep learning. Advances in neural information processing systems, 26, 2013.
[28] Marcella Astrid and Seung-lk Lee. Cp-decomposition with tensor power method for convolutional neural networks compression. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). pages 115- 118. IEEE, 2017.
[29] Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavský, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In European Conference on Computer Vision, pages 522-539. Springer, 2020.
[30] Ting Chen, Ji Lin, Tian Lin, Song Han, Chong Wang, and Denny Zhou. Adaptive mixture of low-rank factorizations for compact neural modeling. 2018.
[31] Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
[32] Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschensky, Andrzej Cichocki, and Ivan Oseledets. Automated multi-stage compression of neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[33] Yong-Deok Deok Kim, Eunhycok Park, Sungjoo Yoo, Taclim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:
2015.
[34] V Lebedev, Y Ganin, M Rakhuba, I Oseledets. and V Lempissky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In International Conference Learning Representations, 2015.
[35] William Shakespeare. The complete works of William Shakespeare. Race Point Publishing, 2014.
[36] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.0426. 2021.
[37] Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non- con-convex optimization. In International Conference on Machine Learning, pages 7184-7193. PMLR, 2019.
12

---

[38] Tian Li. Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020.
[39] Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance reduced local sgd with lower communication complexity. arXiv preprint vrXiv:1912.12800 2019.
[40] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with thcoretical guarantees: A model-agnostic meta-learning approach. Advances in Nearal Information Processing Systems, 33:3557-3568, 2020.
[41] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. arXiv preprint arXiv: 124488 2019.
[42] Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-efliceenn personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 42-55, 2021.
[43] Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li. Yiran Chen, and Hai Li. Lotteryfl: Empower edge intelligence with personalized and communication-efice effficienn federated learning. In 2021 IEEE/ACM Symposium on Edge Computing (SEC), pages 68-79, IEEE, 2021.
[44] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv: 10252, 2019.
[45] Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Reprexentations, 2021.
[46] Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33:5332-5344, 2020.
[47] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communicationn-fflccentt federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pages 2021-2031. PMLR, 2020.
[48] Xinchi Qiu, Javier Fernandez- Vernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Don- ald Lane. ZeroFL. Efficient on-device training for foderated learning with local sparsity. In International Conference on Learning Representations, 2022.
[49] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics, pages 2350-2358. PMLR, 2021.
[50] Nam Hycon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficcen federated learning. In International Conference on Learning Representations, 2022.
[51] Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. arXiv preprint arXiv:21/2.09822 2021.
[52] Hyowoon Seo. Jihong Park. Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. Federated knowledge distillation. arXiv preprint arXiv:201 /. 020367, 2020.
[53] Felix Sattler, Arturo Marban, Roman Rischke, and Wojciech Samek. Communication-efficieen federated distillation. arXiv preprint arXi::20/2.00633 2020.
[54] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances Neural Information Prucessing Systems, 33:2351-2363, 2020.
[55] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint 10X0022010.0101, 2020.
[56] Chuhan Wu, Fangzhao Wu, Ruixuan Liu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Fedkd: Com- munication efficient federated learning via knowledge distillation. arXiv preprint arXiv:2108. 13323, 2021.
[57] Andrei Afonin and Sai Praneeth Karimireddy, Towards model agnostic federated learning using knowledge distillation. arXiv preprint 11x1::2110.15210. 2021.
13

---

[58] Junyuan Hong. Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Efficient split-mix federated learning for on-demand and in-situ customization. In International Conference on Learning Representations, 2022.
[59] Dezhong Yao, Wanning Pan, Yao Wan, Hai Jin, and Lichao Sun. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization. arXiv preprint vxXi:2211.146553 2021.
[60] Tianxing He, Yuchen Fan, Yanmin Qian, Tian Tan, and Kai Yu. Reshaping deep neural network for fast decoding by node-pruning. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 245-249. IEEE, 2014.
[61] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in nearal information processing xystems. 28, 2015.
[62]
Jiwei Yang. Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7308-7316, 2019.
[63] Dongging Zhang, Jiaolong Yang. Donggiangzi Yc, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 365-382, 2018.
[64] Zhefeng Qiao, Xianghao Yu, Jun Zhang, and Khaled B Letaief. Communication--ffiicent federated learning with dual-side low-rank compression. arXiv preprint 1xXi:2104.124166 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Experiment results validate the main claims.
(b) Did you describe the limitations of your work? [Yes] Please see the supplementary material.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofss of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL.?? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)? [Yes] See Sec. 3 More training details are provided in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Error bars are reported in the supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 3 and the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Included the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] Included in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Included in the supplementary material.
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "rCysc4n2BnK", "reviewer": "Reviewer_f2kP", "summary": "Practical federated learning (FL) needs to handle heterogenous resource constraints among participants. Existing solutions mostly rely on pruning a global model in each client, which would cause uneven contributions to the global model weights among different clients. The paper proposes a mechanism that factorizes the global model weights into two low-rank tensors, where one of the tensors is unified across all the clients (called neural basis) and the other tensor varies according to clients’ resource. The paper also proposes a regularization term to encourage the basis vectors to be orthogonal with unit term. The evaluation shows the proposed mechanism outperforms two existing solutions, HeteroFL and FjORD.", "questions": "1. Can you provide more evidence to support the claim that the shared neural basis enables more knowledge sharing than pruning among clients? If the values in the coefficient matrix are small, isn’t it effectively similar to pruning?\n2. How will the proposed method work if each device has a unique complexity level? Will the training still converge?\n", "limitations": "The paper does not provide meaningful discussions on its limitations, and it would be good to see the discussion on the following aspects:\n1. How will the proposed mechanism work with fine-grained complexity levels?\n2. What are the privacy implications of sharing the coefficient matrices?\n3. What if the resource constraint changes within a communication round (e.g., other applications running)?\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\n1. The paper proposes a reasonable solution to an important problem in FL.\n2. The paper proposes a mechanism that generalizes pruning-based solutions to this problem.\n3. The proposed regularization term is interesting.\n\nWeaknesses\n----------------\n1. The paper misses important baselines in this area. Specifically, the paper should also compare its solution against LotteryFL (Li et al., 2020), FedHM (Yao et al., 2021), and FedMask (Li et al., 2021). These three solutions aim to address the same problem, and they report better accuracy than the paper. \n2. The novelty is incremental. The idea of using low-rank factorization to address system heterogeneity has been proposed by FedHM (Yao et al., 2021). The general idea of using low-rank factorization for FL was proposed by FedDLR (Qiao et al., 2021) and FedPara [47]. While the detail of factorization might be different, it is unclear if the proposed factorization technique is better than other existing ones.\n3. There is no analysis on computation and communication complexity. Similarly, there is no convergence analysis. The paper should provide an analysis of the proposed mechanism, and it would be much better to do the same against related solutions in this space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XD09fisWSOf", "reviewer": "Reviewer_83ZB", "summary": "This paper presents the design of All-In-One Neural Composition, which can formulate networks at different complexities with one unified neural basis that's shared among clients. FL systems have suffered from the problem of system heterogeneity where the service providers would either sacrifice model capacity or data accessibility. This paper provides the FLANC framework that provides systematic support to train complexity adjustable neural networks and evaluated on a wide range of statistical data heterogeneity and system heterogeneity. ", "questions": "1. In the evaluation section, the authors varied the $p$ to simulate heterogeneous systems. It's good to also include some real cases including the exact CPU/GPU that might result in the different $p$ (i.e. the limitations of computation power on each device to result in the limitation on model size).", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. Proposed a runtime method to construct the compressed model instead of applying after the training process compared to other low-rank techniques.\n2. Conducted extensive experiments to show the effectiveness of the proposed method. \n\nWeakness:\n1. Didn't include error bars in the evaluation results. The results reported may not be statistically confident without any error bars. The authors could add randomness experiments on some small scale experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9LLTASey98", "reviewer": "Reviewer_dZNT", "summary": "This paper presents FLANC, a model decomposition method for the heterogeneous devices in Federated Learning (FL). The basic idea is to change the model size/complexity by varying the ratio of channels. ", "questions": "How does FLANC perform with regards to other FL aspects such as training rounds and communication overheads?", "limitations": "None", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1.Better supporting of heterogeneous devices (e.g., IoT and mobiles) is an important research topic in FL.\n2.FLANC is generally applicable to other models and datasets. \n3.FLANC shows promising results compared to HeteroFL and FjORD.\n4.This paper is well-written and easy to follow. \n\nWeaknesses\n1.Only the metric of accuracy is evaluated. It is better to consider other important aspects of FL, e.g., communication overheads. \n2.Evaluation results are from one-run experiment only, which is less convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept