Federated Learning of Large Models at the Edge via Principal Sub-Model Training
Authors: Yue Niu
Saurav Prakash
Souvik Kundu
Sunwoo Lee
Salman Avestimehr
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Resource-Constrained Clients
Sub-Model Training

Abstract: Limited compute, memory, and communication capabilities of edge users create a significant bottleneck for federated learning (FL) of large models. Current literature typically tackles the challenge with a heterogeneous client setting or allows training to be offloaded to the server. However, the former requires a fraction of clients to train near-full models, which may not be achievable at the edge; while the latter can compromise privacy with sharing of intermediate representations or labels. In this work, we consider a realistic, but much less explored, cross-device FL setting in which no client has the capacity to train a full large model nor is willing to share any intermediate representations with the server. To this end, we present Principal Sub-Model (PriSM) training methodology, which leverages models’ low-rank structure and kernel orthogonality to train sub-models in the orthogonal kernel space. More specifically, by applying singular value decomposition to original kernels in the server model, PriSM first obtains a set of principal orthogonal kernels with importance weighed by their singular values. Thereafter, PriSM utilizes a novel sampling strategy that selects different subsets of the principal kernels independently to create sub-models for clients with reduced computation and communication requirements. Importantly, a kernel with a large singular value is assigned with a high sampling probability. Thus, each sub-model is a low-rank approximation of the full large model, and all clients together achieve nearly full coverage of the principal kernels. To further improve memory efficiency, PriSM exploits low-rank structure in intermediate representations and allows each sub-model to learn only a subset of them while still preserving training performance. Our extensive evaluations on multiple datasets in various resource-constrained settings demonstrate that PriSM can yield an improved performance of up to $10\%$ compared to existing alternatives, when training sub-models with only $20\%$ principal kernels ($\sim 5\%$ of the full server model.).

Under review as a conference paper at ICLR 2023
FEDERATED LEARNING OF LARGE MODELS AT THE EDGE VIA PRINCIPAL SUB-MODEL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Limited compute, memory, and communication capabilities of edge users create a significant bottleneck for federated learning (FL) of large models. Current literature typically tackles the challenge with heterogeneous client setting or allows training to be offloaded to the server. However, the former requires a fraction of clients to train near-full models, which may not be achievable at the edge; while the latter can compromise privacy with sharing of intermediate representations or labels. In this work, we consider realistic, but much less explored, cross-device Fl. setting in which no client has the capacity to train a full large model nor is willing share any intermediate representations with the server. To this end, we present Principal Sub-Model (PriSM) training methodology, which leverages models' low- rank structure and kernel orthogonality to train sub-models in the orthogonal kernel space. More specifically, by applying singular value decomposition original kernels in the server model, PriSM first obtains set of principal orthogonal kernels with importance weighed by their singular values. Thereafter. PriSM utilizes a novel sampling strategy that selects different subsets of the principal kernels independently to create sub-models for clients with reduced computation and communication requirements. Importantly, kernel with a large singular value is low-rank
assigned with a high sampling probability. Thus, each sub-model approximation of the full large model, and all clients together achieve nearly full coverage of the principal kernels. To further improve memory efficiency, PriSM exploits low-rank structure in intermediate representations and allows each sub- model to learn only a subset of them while still preserving training performance. Our extensive evaluations on multiple datasets in various resource-constraiieed settings demonstrate that PriSM can yield an improved performance of up to 10% compared existing alternatives, when training sub-models with only 20% principal kernels (~~ 5% of the full server model.).
INTRODUCTION
Federated Learning (FL) is emerging as a popular paradigm for distributed and machine learning it allows local clients perform MI optimization jointly without directly sharing local data (McMahan et al. 2017 Kairouz et al., (022). Thus, it enables privacy protection on local data, and leverages distributed local training to attain better global model. This creates opportunities for many edge devices rich in data to participate in the joint training without direct data sharing. For example, resource-limited smart home devices can train local vision or language models using private data, and achieve a server model that generalizes well to all users via FL (Pichai 2019).
Despite significant progress in FL the recent past, several crucial challenges still remain when moving to the edge. In particular, limited computation, memory, and communication capacities prevent clients from learning large models for leveraging vast amounts of local data the clients. This problem is getting increasing attention in current literature (Diao et al.) [022]] Horvath et al.) 2021 Yao et al.) 2 Vepakomma et al., 2018 He et al. 2020). For example, recent works propose a sub-model training methodology by assigning clients with different subsets of server model depending on their available resources (Diao et al. 2021 Horvath et al. 2022 Yao et al. 2021). However, these works have underlying assumption that some of the clients have sufficient resources to train nearly full large model. In particular, methods like FedHM ( Yao et al. 202T) that adapt low-rank compression to FL ineur more memory footprint for intermediate representations, for small

---

Under review as a conference paper at ICLR 2023
sub-models. As result, server model size is limited by the clients with maximum computation, memory, and communication capacities. To overcome resource constraints on clients, other works (Vepakomma et al., 2018 He et al. 2020) change the training paradigm by splitting a model onto server and clients. The computational burden on the clients is therefore relieved as the dominant part of the burden is offloaded to the server. However, such a methodology requires sharing of intermediate representations and/or labels with the server, which directly leaks input information and potentially compromises privacy promises of FL.
Unlike prior works, this work targets an even more constrained and realistic setting at the edge, in which no client is capable of training large model nor is willing to share any intermediate data and/or labels with the server. To this end, we propose Principal Sub-Model (PriSM) training, which at high level, allows each client to only train a small sub-model, while still enabling the server model to achieve comparable accuracy as the full-model training.
The comnrrsoone of PriSM is the models' inherent low-rank structure, which is commonly used in reducing compute costs (Khodak et al. 2021; Denton et al. 2014). However, naive low-rank approximation in FL (aao et al., 2021),
60-
Block1-1
(Required principal kemelss
Block1-2
where all clients only train top-k kernels, incurs a notable accuracy drop, especially in very constrained settings. In
50
Block2
Block2-2
Figure [] we delve into the matter by showing the num- ber of principal kernels required in the orthogonal space to accurately approximate each convolution layer in the first two ResBlocks in R8 (He et al. 2016) dur-
40
50% kemels
ing FL training]] We observe that even at the end of the FL training, around half of the principal kernels are still needed to sufficiently approximate each convolution layer. We have similar findings for the remaining convolution layers (See Sec 4.3). Therefore, to avoid the reduction in server model capacity, it is essential to ensure that all server-side principal kernels are collaboratively trained on clients, especially when each client can only train a very small sub-model (e.g., < 50% of the server model).
32
100
200
200
400
Communication round
Figure 1: Number of principal kernels in the orthogonal space required to accu- rately approximate each of the two con- volution layers in the first two ResBlocks in ResNet- 18 during FL training. Blocki- j indicates j-th convolution layer in i-th ResBlock. Each of these convolution layers has 64 kernels.
Based on the above observations, PriSM employs novel probabilistic strategy select subset of kernels and cre- ate sub-model for each client as shown in Figure More
specifically, PriSM first converts the model into orthogonal space where original convolution kernels are decomposed into principal kernels using singular value decomposition (SVD). To approximate the original server model, PriSM utilizes a novel sampling process, where principal kernel with a larger singular value has a higher sampling probability. The probabilistic process ensures that all sub-models can together provide nearly full coverage of the principal kernels, thus reaching the near full-model training performance with reduced costs on local computation and communication during sub-model aggregation. PriSM further improves memory efficiency by exploiting low-rank structure in intermediate activations and allows each client to learn only subset of these representations while still preserving training performance. Thus, computation, memory, and communication bottlenecks at the edge are effectively resolved.
We conduct extensive evaluations for PriSM on vision and language tasks under settings where no client is capable of training the large full model. In particular, we consider both resource constraints and heterogeneity in system capacities as well as data distribution. Our results demonstrate that PriSM delivers consistently better performance compared other prior works, especially when participating clients have very limited capacities. For instance, on 18/CIFRR 10, we show that PriSM only incurs around 2% and 3% accuracy drop for i.i.d and highly non-i.i.d datasets under a very constrained setting where all clients train sub-models with only 20% of the principal kernels, accounting for 5% of the full server model. Compared to other solutions. PriSM improves the accuracy by up to 10%. Furthermore, we provide detailed insights into the performance gains attained by PriSM via 1) analyzing server model's rank structure during training; 2) profiling the kernel sampling process; 3) breaking down costs in the system.
See See 4.3 for further details, especially for calculating the required number of principal kemels.
2

---

Under review as a conference paper at ICLR 2023
Cone
singile valacs
Bavrss Nome Rel
Back Nome Rell
Cone
SVD
Back Nomai RdA.
Back Norrs KeUU
Cons
kernoss
principal kernels
sub-model
Figure 2: Creating clients' sub-models. PriSM samples a subset of principal kernels to create a client's sub-model that enjoys less computation and communication overhead. The singular value based sampling scheme ensures every sub-model approximates the full large model, and all sub-models together provide nearly full coverage of the principal kernels. PriSM further improves memory efficiency by allowing each sub-model to learn only subset of intermediate representations.
RELATED WORKS
Factorized Models, Training neural networks with layer factorization has been extensively studied in prior literature (Denton et al. 2014: Khodak al. 2021 Jaderberg et al. 2014 Novikov et al. 2015 loannou et al. 2016). Specifically, these works are based on the observation that well-trained neural networks have inherently low-rank structure and exhibit prnge-correlaions. across kernels. Hence, one can down-size the model with a low-rank approximation to provide significant reduction in computations thus speeding up training. Furthermore, this make model training more affordable for
devices. addition to models' low-rank structure, works such as (Niu et al., 2022) also exploit low-rank structure in intermediate representations to reduce computation overhead.
Federated Learning. While federated learning opens the door for collabo- rative model training over edge users having rich (but private) data, the computation, memory, and communication footprint prohibits training of large models at the resource- www.rrce-ootttainneddd clients. To address these resource limitations in federated learning, a number of works have been proposed the
Resource-Constrr
literature (Diao et al., 2021 Horvath et al. 2021
Yao et al. 2021 Diao et al. 2021 Horvath et al.
2021 Yao et al. 2021 Vepakomma al. 2018 Poirot et al. 2019 | Chopra al. 2021 He et al.
2020 Particularly. in split learning (Vepakomma et al. 2018 Poirot et al. | Chopra et al. 2021
the model is partitioned into two parts, one (small) part is assigned to clients for local training, while the other (large) part is outsourced to the server. He et al. (2020) proposes FedGKT that combines the model splitting approach with a bi-directional knowledge transfer technique between server and clients achieve
FL with much fewer communications than split learning. However, works such as split learning and FedGKT require sharing of intermediate activations (and in many cases, logits well as labels) with the server, directly leaking input information and potentially compromising privacy promises of FL (Zhang et al. 2020).
The works closely related to ours are HeteroFL (Diao et al. 2021). FjORD (Horvath et al. 2021 and FedHM (Yao et al. 2021). that aim to enable participation of resource-constrained client by letting train smaller sub-model based on its capabilities. In particular, HeteroFL and FjORD create sub- models for clients by selecting certain fixed number of original kernels of the server model. On the other hand, FedHM creates sub-models using fixed subsets of factorized principal kernels. However, in these works, the size of the server model gets limited by the clients with maximum computation, memory, and communication capacities, sacrificing the model performance. particular, methods like FedHM ineur more memory footprint for intermediate representations, even for small sub-models. This becomes even more critical in the realistic, cross-device FL setting wherein no client has the capacity train a large model. While another work, FedPara (Hyeon-Wod al. (021), proposes a low-rank factorized model training to reduce communication costs, computational footprint still remains prohibitive every client required to perform full-model training. Therefore, further efforts are still needed to effecttvely address the computation, memory, and communication bottleneck the edge, while still preserving the privacy promises of FL..
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
In this section, we first motivate our proposal, Principal random Sub-Model training (PriSM), with an observation of orthogonality in convolution layers. Then, describe the details of PriSM.
Notations- Il
Frobenius norm. o;: i-th singular value in a matrix. @: convolution. :: matrix multiplication. ( ) of element-wise multiplication inner product. tr(A): trace of matrix.
3.1 MOTIVATION: AN OBSERVATION ORTHOGONALITY
We consider a convolution layer with kernels W Nxxxxxxx and input X RMxHx where N and M denote the number of output channels and input channels, is kernel size, and x W the size of the input image along each channel. Based on a common technique im2col (Chellapilla al. 2006), the convolution layer can be converted to matrix multiplication as Y W X, where W E x M&? and X E T MRA HWW For kernel decorrelation, we apply singular value decomposition (SVD) to map kernels into orthogonal space as: W = Σ u -u, where {ur) {v:) are sets orthogonal vector The convolution can be decomposed
03 u U. XX.
For Vi * j. is easy verify that (Y,, Yj) = -0. tr . ut u..00. 0o X) namely the output features Y, and Y, are orthogonal. Therefore, if we regard W; = o; 2a principal kernel, different principal kernels create orthogonal output features. To illustrate this, Figure shows an input image (left) and the outputs (right three) generated by principal kernels. We can observe that principal kernels captures different features and serve different purposes.
As revealed in (Xie al. 2017 Balestriero et al. 2018 Wang al. 2020), imposing orthogonality on kernels leads to better training performance. This motivates us to initiate the training with a set of orthogonal kernels. Furthermore, to preserve kernel orthogonality during training, it is critical to constantly refresh the orthogonal space through re-decomposition. The above intuitions based on the observation orthogonality play key role in PriSM, which is described in the following section.
Inputs
Kernel
Kernel 2
Kernel
Orthogonal features generated by principal kernels. Different principal kernels capture different features (Kernel I: outline of the cat, Kernel 2 and 3: textures but on distinct regions).
Figure 3:
Remark 3.1. The layer decomposition using SVD also applies general linear layers (e.g., MLP, LSTM), where weights W E R X We can directly apply SVD and obtain the principal components.
3.2 PRISM: PRINCIPAL RANDOM SUB-MODEL TRAINING
Motivated by the observation that output features are orthogonal given orthogonal kernels, we propose PriSM, new sub-man model training method that directly trains orthogonal kernels in clients. Particularly, PriSM introduces two key components to ensure training performance with sub-models under very constrained settings. First, considering different orthogonal kernels and their contributions to outputs are weighed by the corresponding singular values, PriSM devises an importance-aware sampling scheme to create client sub-models to achieve computation and communication efficiency. In particular, the sampling scheme brings two benefits: i) each sub-mut mob-model is low-rank approximation of the server model; ii) the conglomerate of the sampled sub-modek enables nearly full converge of
We assume w.l.o.g W tall matrix.
4

---

Under review as a conference paper at ICLR 2023
orthogonal kernels. Second, further improve the memory efficiency while maintaining training performance, PriSM exploits low-rank structure in activations and allows each client to learn only a subset of intermediate representations in each layer. We start with unfolding sub-model creation, and then describe the training procedure in PriSM.
3.2. SUB-MODEL CREATION
Computation and communication efficiency via sub-model sampling, For a convolution layer with principal kernels {Wa) i=1' and the corresponding singular values {a, {or))==> we randomly sample
principal kernels denoted by W with sampling probability for i-th kernel as follows:
Ps
(2)
Here, r in each layer is decided by c-th client's 's resource budget, k is smooth factor controlling the probability distribution in sampling. Indices of selected kernels are denoted Iz.
The convolution output is calculated as
Y Y= EY.
Σ σ. u. 00 .X X.
(3)
IET.
IET
By performing the sampling process for every convolution layer, we create random low-rank model for each client. Important kernels with large singular values are more likely to be chosen, and all sub-models can together provide nearly full coverage of principal kernels. The resulting sub-model hence has convolution layers with fewer kernels, reducing required computations and communication overhead (when aggregating models). Other element- wise layers, such as ReLU and batch normalization, remain the same.
Memory efficiency via learning a subset of features. We further exploit low-rank structure in intermediate representations. For sub-model above, each convolution layer is decomposed into two sublayers: Convy and Conay with kernels (ui)iez, and {w; {ui respectively. Without further optimization, output from Convr consumes the same memory as the original layer, causing high memory pressure at the edge. However, noting that inputs to Convy, X E RrxHW is low-rank, and kernels W RNNx xr, based the rank inequality of matrix multiplication 2014), we obtain
(Banerjee Roy
Rank(Y) S min (Rank(X), Rank WW) S
(4)
Therefore, output of Convy also exhibits a low-rank structure. Such low-rank structure reflects cor- relation among output channels. indicating channel redundancy in outputs when N. Based on the observation, we further allow Convu to compute only subset of output channels, therefore reducing memory footprints of intermediate representations while still preserving necessary information in activations. Typically, given input to Conwy with input channels, PriSM only computes output channels. During implementation, we replace the select u, with its subset u, (1 : r) when computing output features from Conwy. For simplicity, will still use u, in the rest of the paper to denote the subset u1 r). The unselected channels are cached in ú; and are used in model aggregation.
3.2.2 TRAINING
This section details the training procedure in PriSM. We describe each component below.
Local training, On each client, during local training, the sub-model with parameter {a,, u, v. hiee are updated. The sub-model also consists of trainable layers such as BatchNorm but with fewer channels denoted in I. PriSM allows to be trained in local training, thus ensuring changes regarding each principal kernel's 's importance are captured in local clients. addition, d; will be merged into u, V; as u, = Vo,u, v, Voru, to reduce memory footprints.
Sub-model aggregation. On the server side, with sub-models obtained from clients, we aggregate i-th principal kernel as follows:
W; Iaaffs à. mn)) (
T
Easoie) cee
(5)
LOEC
5

---

Under review as a conference paper at ICLR 2023
where C denotes the subset of active clients, ag is the aggregation coefficient for i-th kernel. We propose a weighted averaging scheme: if i-th kernel is selected and trained by C, clients, then af = 1/C,. Furthermore, the unselected output channels út concatenated with the aggregated u, before reconstructing the orthogonal kernel to preserve model capacity. If W, not selected by any client, it remained unchanged in the server. The full model in the original space is constructed by converting each 2-dimensional W; the original dimension xd x and combining them.
Orthogonal space refresh. After model aggregation, we perform SVD on the updated kernels W preserve orthogonality among the principal kernels. Thus, in the next communication round, the importance-aw -warr sampling can still create low-rank sub-models for different clients..
We further additional techniques improve learning efficiency in the orthogonal space: activation normalization, and regularization on orthogonal kernels.
Activation normalization. We apply batch normalization without tracking running statistics; namely, the normalization always uses current batch statistics in the training and evaluation phases. Each client applies normalization separately with no sharing of statistics during model aggregation. Such an adaptation is effective in ensuring consistent outputs between different sub-models and avoids potential privacy leakage through the running statisties (Andreux et al. 2020).
Regularization. When learning factorized model on a client, applying weight decay to and e; separately results in poor final accuracy. Inspired by (Khodak et al. 2020). for training on client c, we add regularization to the subset of kernels as follows:
reg
Σ BEZ.
u
T
(6)
2
where À is the regularization factor, Ie denotes the subset of principal kernels client c.
Algorithm D presents an overall description of PriSM. We only show the procedure on single convolution layer with kernels W for the sake of simplifying notations.
Algorithm PriSM: Principal Random Sub-Model Training
Input: layer parameters W, client capacities.
1: for communication round = 1, 1.... T do
2: Decompose W into orthogonal kernel using SVD {W.) A
3: Choose a subset of clients + C.
4: for each client E C do
5: Compute the sub-model size for client e - [Zal-
6: Obtain sub-model following the procedure in Sec 3.2.1 + Ic, W. // Sub-model
7: Perform Local + Ie, W" Local training
8: end for
9: Aggregate parameters based on Eq. 5 W + { [w") Joec' // Sub-model aggregation
10: Perform SVD on W. Orthogonal space refresh
11: end for
12: LocalTrain Ig, W
13: for local iteration k = 1, ...., K do Sample an input batch from the local dataset + Dp.
Perform the forward and backward pass + D4. W
Update the local sub-model using SGD + W
17: end for
In the following remarks, we differentiate PriSM from Dropout and Low-Ramd compression. Remark 3.2. PriSM vs Dropout. PriSM shares some computation similarity with model training using regular dropout in clients. However, regular dropout suffers convergence instability due to

---

Under review as a conference paper at ICLR 2023
inconsistent activations across iterations, esp. with a high dropout probability (Horvath et al. 2027). In contrast, PriSM performs importance-awan sampling in the orthogonal space. Each sub-model approximates to the full model, and different sub-models do not create significant inconsistency.
Remark 3.3. PriSM Low-Rank Compression. PriSM is not low-rank compression method. Low-rank compression methods such as FedHM (Yao et al. 2021] aim to construct a smaller server model by completely discarding some kernels even though they can still contribute training performance (See Figurelojn Section PriSM randomly select sub-models so that every kernel possible be learned. Furthermore, PriSM achieves memory efficiency by exploiting low-rank properties in intermediate representations, which is not seen in other low-rank methods.
4 EXPERIMENTS
We evaluate PriSM under resourced-constrraieed settings where clients can training the large full model. Furthermore, we consider both homogeneous and heterogencous client settings. Specifically, in homogeneous settings, all clients have the same limited computation, memory, and communication capacity, while in heterogeneous settings, clients' capacities might vary. We also compare PriSM with two other baselines: ordered dropout in orthogonal space (OrthDrop); and ordered dropout in original space (OrigDrop). At high level, our results demonstrate that PriSM achieves comparable server model accuracy even when only training very small sub-models on all clients. Additionally, we provide more insights into the superior performance of PriSM by analyzing server model's rank structure. Finally, we study the sampling process and cost breakdown in the FL system.
Baselines. Prior methods such as FjORD (Horvath et al.) 2021 and HeteroFL DDioo et al. 2021) select sub-models from the original kernel space, for which we denote as OrigDrop. In implementation, we follow the same procedure in HeteroFL. On the other hand, we use OrthDrop to denote selecting fixed top-k kernels from the orthogonal space such as in FedHM (Yao et al. 2021
Models and Datasets. We train ResNet- 18 on CIFAR- 10 (Krizhevsky al. 2009), CNN on FEM- NIST (Caldas et al. 2018) and LSTM model on IMDB MMas et al. 2011 Detailed architectures provided in Appendix A.I.I
Data Distribution. For CIFAR-10 10 and IMDB, we uniformly sample an equal number of training images for each client when creating i.idd datasets. For non-i.i.d datasets, we first use Dirichlet function Dir( (Reddi et al. 2020) to create sampling probability for each client and then sample an equal number of training images for clients. We create two different non-i.i.d datasets with 1 and a = 0.1, where a smaller indicates a higher degree of non-i.i.d. For FEMNIST, we directly use the dataset without any additional preprocessing.
FL Setting, We simulate an FL setting with 100 clients with 20 random clients active in each communication round. Each client trains its model for 2 local epochs in each round. We use SGD with momentum during training. The learning rate is initially 0.1 for ResNet-18 and 0.01 for CNN/FEMNIST, and decayed by a cosine annealing schedule. Further details are in Appendix A.1.2
4.1 PERFORMANCE ON CONSTRAINED HOMOGENEOUS CLIENTS
In this setting, we assume that all clients have the same limited capacity. We vary the client sub- model size from 0.2 to 0.8 of the full server model, where 0.r indicates only 0.r subset of the principal kernels are sampled in each convolution layer from the server model (denoted as keep ratio). Accordingly, r 0.z N, where N is the number of output channels in layer. In Table T list the computation, memory and communication footprints for different sub-models of ResNet- 18. CNN/FEMNIST and LSTM/IMDB also enjoy similar cost reductions. It is worth noting that PriSM incurs much smaller costs compared to FedHM (See Appendix A.4 further details).
Figure shows final validation accuracy drops of ResNet-18 with different sub-models on i.i.d and non-i.i.d datasets compare to full-model training. We note that PriSM constantly delivers better performance than the other two baselines. The performance gap is even more striking under very constrained settings. For instance, when only 0.2x sub-models are supported on clients, PriSM attains comparable accuracy as full-model training, and achieves up to 10% 14% performance improvement
Due the page limit, I.S results are deferred to Appendix A.2
7

---

Under review as a conference paper at ICLR 2023
Table 1: Model size, MACs, activation memory for different sub-models in PriSM (batch size: 32).
Model
Full
0.8
0.6
0.4
0.2
Params
11 M
7.9 M (72%)
4.5 M (41%)
M (18%)
.S M (4.5%)
ResNet-18
MACs
35 G
25.5 (73%)
14.7 (42%)
6.87 G (20%)
2 G (5.6%)
Mem
31.5
37 M (115%)
28.3 (90%)
18.9 M (60%)
9.5 M (30%)
compared to OrthDrop and OrigDrop on non-i.id dataset with a = 0.1. Furthermore, for 0.2x sub-models, if slightly relaxing memory constraints and allowing PriSM to learn 2x out channels in Convy (denoted as PriSM-O2), the accuracy can be further improved with only incurring additional 65% memory footprint. We also make two key observations. First, training with sub-models in the orthogonal space provides better performance than in the original space, which aligns with our intuition Section 3.1 Second, our importance-awware sampling strategy for creating sub-models is indispensable as demonstrated by the notable performance gap between PriSM and OrthDrop. Similar performance trends are also observed in FEMNIST, as shown in Figure
sec drop
buseline: 92.46
baseline: 92
hascling: $4.96
PrSM--2
OrhiDop
PriM
OreDimp
model
(a) Lid
(b) (an-
(c) non-i.id (ar 0.1)
Figure 4: Accuracy drops on CIFAR-10 10 on homogeneous clients. PriSM constantly delivers better performance compared to OrigDrop and OrthDrop. significantly outperforming them under very constrained realistic edge settings. (bar: mean; line: std)
4.2 PERFORMANCE ON CONSTRAINED HETEROGENEOUS CLIENTS
To simulate clients with varying limited capacity, we sim- ulate the following setting: 40% clients train 0.4x sub- models, and 60% clients train 0.2x sub-models. No par- ticipating client trains the full model. For baseline methods, we follow the same strategy in Section 4.1
drop
Iaseline 86
Table lists the final accuracy achieved by different methods under the heterogeneous setting. PriSM greatly outperforms the baseline methods even when 0.4x sub-models are sup- ported on small fraction of clients. Furthermore, similar to the results Section 4. the benefits of training the orthogonal space and importance-aware sampling strategy are also observed in heterogeneous client settings.
P=SM
Orlirip
model size
Figure 5: Accuracy drops on FEM- NIST homogeneous clients. (bar: mean; line: std)
Table 2: Training performance of ResNet- 18/CIFAR-10 on heterogeneous clients.
Distribution
Baseline
OrigDrop
OrthDrop
PriSM
Li.d non-i.id (1)
92.46
88.86 + 0.17
89.57 + 0.16
90.63 1 0.14
86.91 0.24
88.78 0.22
89.89 1 0.2
non-i.i.d (0.1)
84.96
76.38 + 0.92
78.37 1 0.99
82.58 1 0.51
4.3 INSIGHTS INTO PRISM
We now focus on providing further insights into PriSM by analyzing some its key aspects. To this end, first examine the low-rank structure of models during training, and pinpoint the cause behind the accuracy gap between fixed and random kernel dropout strategies in the orthogonal space. Thereafter, we study the sampling process and cost breakdown in the FL system
Model's rank during training, To analyze the server model's rank structure, we adopt a similar method in Alter et al. 2000) calculate the required number of principal kernels to accurately
8

---

Under review as a conference paper at ICLR 2023
approximate each layer as 2- log((o pi log Here, ps is calculated as in Eq. 2 with A 1. Figure 6 shows the number of required kernels for each layer in ResNet-18 18 during full-model FI. (Blocki-j: j-th convolution layer in i-th ResBlock). We first observe that while the server model attains a low-rank structure gradually, randomly initialized model does not. Therefore, sub-models with fixed top-k principal kernels inevitably causes reductions in the server model capacity. Furthermore, even at the end of the FL training, around half principal kernels are still required to approximate most layers. In fact, some layers require even more principal kernels. Therefore, our probabilistic sampling scheme is essential in preserving the server model capacity during FL training with sub-models,
128
241-
bskck>
Block52
443
Blvibi
Blolls 2
800
100
300
KeokT-3
BookT-2
250
Roddd-
Bodd-
Cormanicatin 1ramd
Conce meal
(a) ResBlock 3, 4 (128 kernels). (b) ResBlock 5, 6 (256 kemell) (c) ResBlock 7, 8 (512 kernels).
Figure 6: The number of principal kemelss required to accurately approximate each convolution layer in ResBlocks 3-8 in ResNet- 18 (Results of ResBlocks and 2 are discussed in Figure I
Kernel sampling profiling. Figure 7 shows the average number of clients assigned for each or- thogonal channel in one communication round. Each client trains 0.2x sub-moder of ResNet- 18 on CIFAR- 10 with i.idd distributions as in 1.c We observe that each kernel is selected by at least one client in each round, indicating every kernel will be activated and trained on lo- cal data in each round. Furthermore, orthogonal kernels with larger get more chances to be chosen, which ensures sub-models on all clients consistently approximate the full model.
Moock1-1
Bockt-1
Bocko
TBockk-2
Hassigned cleere
Biock24
BRock--1
Bock7-22
13
Block2 2
Blookd
Bookk
BBock3-1
BRocktad
BRock8-
Block3 2
kennel indes
Figure 7: The average number of clients assigned for each orthogonal kernel during training.
Runtime breakdown. In this section, we investigate the relative cost of SVD by breaking down the model training time into four stages: sub-model creation, local training, model aggregation, and obtaining orthogonal kernels (SVD). We choose a large model, ResNet- 18, as the target model. We adopt the same hyperparameters as in Table 7 We run the server and clients process on NVIDIA Quadro RTX 5000. Table 3 lists each stage's average time in one communication round. We observe that the time of SVD is nearly negligible compared to the time spent on the local training. Therefore, even for large models such as ResNet-18, 18, the relative overhead of SVD is still very small.
Table 3: Training time breakdown for ResNet- 18 on CIFAR- 10.
stage
sub-model creation
local training
aggregation
SVD
time
3.27
36.82
0.11s
0.96 s
5 CONCLUSION
We have considered the practical, yet under-explored, problem of federated learning in a resource- constrained edge setting, where no participating client has the capacity to train large model. As our main contribution, we propose the PriSM training methodology, that empowers the resource-limited clients by enabling them to train smaller sub-models. At the same time, PriSM utilizes a novel sampling approach to obtain sub-models for the clients, all of which together ensure that the server model achieves close to the full-model performance. PriSM further improves memory efficiency by exploiting low-rank structure in intermediate activations. Our extensive empirical results demonstrate that PriSM performs significantly better than the prior baselines, especially when each client can train only very small sub-model. We will consider providing theoretical analysis in the future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Orly Alter, Patrick O Brown, and David Botstein. Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences, 97
(18):10101-10106, 2000.
Mathieu Andreux, Jean Ogier du Terrail, Constance Beguier, and Eric Tramel. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 139. Springer, 2020.
Randall Balestriero et al. A spline theory of deep learning, In International Conference Machine Learning, pp. 374-383. PMLR, 2018.
Sudipto Banerjee and Anindya Roy. Linear algebra and matrix analysis for statistics, volume 181. Crc Press Boca Raton, 2014.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 2018.
Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers handwriting recognition. Suvisoft, 2006.
Ayush Chopra, Surya Kant Sahu, Abhishek Singh, Abhinav Java, Praneeth Vepakomma, Vivek Sharma, and Ramesh Raskar. Adasplit: Adaptive trade-offs for cocource-constrinne distributed deep learning. arXiv preprint arXiv:21 2021.
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient fed- crated learning for heterogeneous clients. International Conference Learning Representations, 2021.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large CNNs at the edge. Advances in Neural Information Processing Systems, 33: 14088- 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34, 2021.
Nam Hyeon- Hyeon-Woo, Moon Ye-Bin, and Tae- Hay- Oh. Fedpara: Low-rank hadamard product for communication-effiiine federated learning. arXiv preprint arXiv:2108. 06098, 2021.
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training CNNs with low-rank filters for efficient image classification. International Conference on Learning Representations (ICLR), 2016.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. British Machine Vision Conference (BMVC), 2014.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-200, 2021.
10

---

Under review as a conference paper at ICLR 2023
Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2020.
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. International Conference on Learning Representations (ICLR), 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effciiennn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Yue Niu, Ramy E Ali, and Salman Avestimehr. 3legrace: Privacy-preserving dnn training over tees and gpus. Proceedings on Privacy Enhancing Technologies, 4:183-203, 2022.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Sundar Pichai. Google's Sundar Pichai: Privacy Should Not Be Luxury Good. In New York Times, 2019.
Maarten G Poirot, Praneeth Vepakomma, Ken Chang, Jayashree Kalpathy-Cramn Crameer Rajiv Gupta, and Ramesh Raskar. Split learning for collaborative deep learning in healthcare. arXiv preprint arXiv: 2019.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint 00020 2020.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.000 00564,, 2018.
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pultern recognition, pp. 11505-11515, 2020.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormaltyy and modulation. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 6176-6185, 2017.
Dezhong Yao, Wanning Pan, Yao Wan, Hai Jin, and Lichao Sun. FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization. arXiv preprint arXiv:2111.14655 2021.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 253-261, 2020.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 MODELS AND HYPERPARAMETERS
In this section, provide detailed information about models and hyperparameter settings for the results presented in the paper. We will open our source code upon acceptance of the paper.
A.1.1 MODELS
ResNet-18/CIFAR-10, We ResNet-18 optimized for CIFAR-IO, in which kernel size in the first convolution layer is changed from 7 x to x 3. Details are shown in Table 4
Table 4: ResNet-18/CIFRR-U
Module
#kernels
size
stride
Batch Norm
ReL.U
Downsample
Conv1
64
Block1-1
64
ResBlock
x
Block1-2
64
3
Block2-1
64
ResBlock 2
X
Block2-2
64
Block3-1
128
ResBlock
Block3-2
128
Block4-1
128
ResBlock 4
X
Block4-2
128
3
Block5-1
256
3
ResBlock 5
Block5-2
256
Block6-1
256
ResBlock
Block6-2
256
Block7-1 Block7-2
512
ResBlock 7
512
Block8-1
512
ResBlock 8
X
Block8-2
512
Classification
10
X
CNNFEMNIST. We use a similar architecture as in FjORD Horvath et al. 202 model is shown in Table
The detailed
Table 5: CNN/FEMNIST
Module
#kernels
size
stride
ReLU
Conv1
64
5
1
Poolingl
2
2
Conv12 12
64
3
1
Pooling2
2
2
Classification
10
LSTM/MDB. We use common LSTM model as shown in Table 6
A.1.2 TRAINING HYPERPARAMETERS
ResNet-18/CIFAR-I on homogeneous clients. We simulate 100 clients during FL training, in which each client is assigned 500 training samples for both i.i.d and non-i.i.d datasets. In each
12

---

Under review as a conference paper at ICLR 2023
Table 6: LSTM/IMDB
Module
input size
output size
hidden size
#layers
Embedding
1001
64
LSTM
64
256
256
2
FC
256
communication round, each client performs local training for epochs using the local data, then uploads parameters to the server for aggregation. Table Flists detailed hyperparameters during FL training with ResNet-18.
Table 7: Hyperparameters for ResNet-18/CIFAR-I on homogeneous clients
#clients
#samples
distribution
augmentation
Datasets
100
500
i.i.d, non-i.id (a
1,0.1)
flip, random erop
#Rounds
#local epochs
batch size
#active clients
smooth factor k
Training
1000
2
32
20
2.5/4 (0.2 sub-model)
Optimizer
Momentum
wd
initial
scheduler
Opt
SGD
0.9
0.0002
0.1
cosine annealing
CNN/FEMNIST on homogeneous clients. We simulate 100 clients during FL training, in which each client is assigned users' data from the original training dataset. We use the whole validation dataset to compute the validation accuracy. Table 8 lists detailed hyperparameters during FL training with CNN.
Table 8: Hyperparameters for CNN/FEMNIST on homogeneous clients
#clients
#users/client
distribution
augmentation
Datasets
100
10
natural non-i.i.d
None
#Rounds
#local epochs
batch size
#active clients
smooth factor k
Training
100
2
32
20
2.5
Optimizer
Momentum
wd
initial Ir
scheduler cosine annealing
Opt
SGD
0.9
0.0002
0.01
on heterogeneous clients. We adopt the same setting as Table 7 except the fact that clients might vary in computation and communication capacity. Therefore different model might train sub-models with different sizes (See Sec 4.2 in the main paper).
A.2 EXPERIMENTS ON LSTM/IMBB
The LSTM model used in FL training is detailed in Table During training, we simulate 100 clients, in which each client is assigned 375 training samples. We create local datasets with two different distributions using the same method as in CIFAR-10: i.i.d and non-i.i.d (a = 0.1). Table list detailed hyperparameters for training LSTM/IMDB.
Figure 8 shows accuracy drops of sub-model training on IMDB in homogeneous client settings. While the task on IMDB is just a binary classification problem, PriSM still achieves the best final server model accuracy on both i.idd and non-i.i.d datasets. On i.d datasets, training using sub-models achieves comparable accuracy as full-model training, even only using very small mub-models such as 0.2x. On the other hand, on io dataset, OrigDrop suffers notable accuracy drops compared to PriSM.
13

---

Under review as a conference paper at ICLR 2023
Table 9: Hyperparameters for LSTM/IMDB on homogeneous clients
#clients
#samples
distribution
augmentation
Datasets
100
375
i.dd, non-i.id (a 0.1)
None
#Rounds
#local epochs
batch size
#active clients
smooth factor 8
Training
300
2
32
20
Optimizer
Momentum
wd
initial lr
scheduler
Opt
SGD
0.9
0.0002
0.1
cosine annealing
-325
465
000 do0g
-3.18
baseline: 84.99
355
basclinc: 83.23
OrtiDlop
Orig
model size
(a) Lid
(b) non-i.id (a 0.1)
Figure Accuracy drops on IMDB on homogeneous clients.
A.3 MORE ABLATION STUDY
In this section, we study the effects of two hyperparameters: the number of local epochs and active clients. Each of these two parameters affects the sampling process when creating sub-models for clients. Specifically, given a fixed number of total iterations, FI. training with small number of local epochs per communication round performs a more frequent sampling process, thus making more orthogonal kernels to be selected and trained. Similarly, the FL training with large number of active clients per round can also activate more orthogonal kernels.
Effects of number of local epochs. To investigate the effects of the number of local epochs on final server model accuracy, we train ResNet-18 in homogeneous client settings. The sub-model trained clients varies from 0.2x to 0.6 We also train a full model
baseline. The common training hyperparameters are the same as in Table 7 We fix the number of total iterations as 2000, namely, Round x Local epochs non-i.i.d datasets with a = 0.1. We observe that while final server model accuracy decreases as the number of local epochs increases, the accuracy gap between sub-model and full-model training also slightly increases. One potential reason is that the total number of sampling decreases in FL training with more local epochs. As a result, some orthogonal kernels are under-trained. Such observation also aligns with the intuition discussed in the main paper that all orthogonal kernels should be trained.
2000. Figure 9 shows final server model accuracy on i.i.d and
Fall
06
Accuracy (%))
Accuracy
#T.ocal epochs
#Local epochs
(a) i.i.d datasets.
(b) non-i.i.d datasets (a 0.1).
Figure 9: Effects of the number of local epochs training performance (ResNet-18/CIFAR-100
14

---

Under review as a conference paper at ICLR 2023
Effects of number of active clients. We follow the same settings in Table 7 except that the number of active clients varies from 10 to 30 in each communication round. Similar as above, we vary the sub-model trained clients from 0.2x to 0.6x. Figure shows the final server model accuracy on i.i.d and non-i.i.d datasets with a = 0.1. On i..d datasets, FL training with a different number of active clients does not significantly change the training performance. Furthermore, the accuracy gap between sub-model and full-model training is also almost the same, even though less frequent sampling process is performed in training with fewer clients. However, on non-i.i.d datasets, the finaly accuracy increases with an increasing number of participating clients. Further, the performance gap between sub-model and full-model training also shrinks more sampling processes are performed in each round. In practical edge settings, with plenty of devices such as smart-home devices connected, the performance gap can possibly further shrunk.
Accuracy (%)
TUI
82
PVR
A6
33
Accurazy
#Active clicuts
#Active clients
(a) i.i.d datasets.
(b) non datasets (a 0.1).
Figure 10: Effects the number of active clients on training performance (ResNet-88CIIFA--0))
A.4 COST COMPARISON BETWEEN PRISM AND FEDHM
In this section, we present comparison of PriSM and FedHM on sub-model size (#Params), computations (#MACs), and memory for intermediate features (Mem). For computations and memory, only consider the forward pass during training is sufficient to show the difference between these two methods. When measuring memory requirements, we ignore data representations (float/fixed point) and only count the number values in activations. Table [Olists the comparison. It is noted that FedHM still needs significant memory to store activations, even for very small sub- models. PriSM reduces both model size, computations, and activations when using small sub-models.
Table 10: Costs for different sub-models in PriSM and FedHM (batch size: 32).
Model
Full
0.8
0.6
0.4
0.2
PrISM
Params MACs
M
7.9 M (72%)
4.5 M (41%)
2 M (18%)
(4.5%)
ResNet-18
G G
25.5 G (73%)
14.7 G (12%)
6.87 G (20%)
2 G (5.6%) 9.5 M (30%)
Mem
31.5 M
37 M (115%)
28.3 M (90%)
18.9 M (60%)
FedHM
Params
11M
9.9 M (90%)
7.4 M (67%)
4,9 M (44%)
2.5 M (22%)
18
MACs
35
28.8 G (82%)
22.4 G (64%)
16 G (46%)
7.4 G (23%) 31.7 M (101%)
Mem
31.5 M
44.9 M (143%)
40.5 M (129%)
36.1 M (115%)
A.5
MORE ON-DEVICE EFFICIENCY ANALYSIS
Table 11: Per-round of training with different sub-models.
sub-model
full
0.8x
0.6x
0.4x
0.2x
time
31.41
28.39
23.53
20.21 s
17.55 s
We further looked into how computation and memory savings result in clock-time improvements. We use ResNet-18 18 and train it CIFAR-10. In each communication round, each client trains its
15

---

Under review as a conference paper at ICLR 2023
sub-model with epoch with batch size 32. Other hyperparameterss are the same in Table] 7 We use a single NVIDIA GTX 1080 GPU as the test platform. Table [T shows per-round time on local training with different sub-modek, We observe that per-round clock time decreases with the size of sub-models on clients. Therefore, this computation and memory saving indeed leads to clock-time improvements.
16


Reviews:
[{"review_id": "NP4nOimu9W", "reviewer": "Reviewer_v26F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of federated learning on edge devices with limited computation, memory, and communication capacities. Specifically, this paper proposed PriSM, a sub-model training approach for FL. PriSM is a low-rank approach that performs SVD on the kernels to build a sub-model on device during the training.  With empirical evaluation, the authors show that PriSM is able to train a model with only 20% of the principle kernels and has 10% performance improvements over existing solutions.\n", "summary_of_the_review": "\nThis paper proposes a powerful low-rank approach in FL for sub-model training on-device. I have already mentioned the strengths in the above section. Here I would like to raise a few questions for clarification.\n\n1. What is the on-device computation efficiency? How many FLOPs does the PriSM save when performing FL? Does this saving result in clock-time efficiency improvements?\n\n2.  Does PriSM generalize to MLP/Transformer models? The linear model can also be low-rank decomposed on-device for fast training.\n\n3. Is there any extra overhead in the aggregation phase? How does this overhead compare with the original aggregation procedure in FedAvg?", "strength_and_weaknesses": "Strength\n1. This paper is well-written with clear formulation and introduction of PriSM\n\n2. The empirical evaluation of ResNet models is solid across different vision FL datasets.\n\n3. In the appendix, the paper presents a detailed parameter selection, making PriSM reproducible\n\nWeakness\n1. A stronger analysis of the on-device efficiency should be introduced.\n\n2. It would be better for PriSM to be applied to general linear models instead of focusing on convolution blocks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KWP9pGCv95P", "reviewer": "Reviewer_vmY3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of learning a large model in a federated manner by clients with limited computational and memory capabilities. The paper is mainly about large neural networks. The proposed solution is based on SVD decomposition of each layer to provide clients with a low rank representation of each layer. This reduces the amount of computations required to update the models locally by clients.", "summary_of_the_review": "In summary, the paper studies an important problem and proposes a novel algorithm to solve this problem. However, there are important questions that should be answered and discussed by the paper. Also, writing and presentation of the paper need improvement.", "strength_and_weaknesses": "Strengths:\n\n1. The paper proposes a new method to enable clients with limited computational capability to train a large model.\n2. Using the proposed method, clients with limited memory can collaborate on training a large model that cannot be fit into their memory.\n\nWeaknesses:\n\n1. This works lacks theoretical analysis. For example, it would be interesting to see if there are any other sampling method that can provide better result. Also it is not clear that how $r$ affects the training accuracy.\n2. An important question that this paper does not respond is that why training a low rank representation of a large model is better than training of a original smaller model.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5hesGPcIW4R", "reviewer": "Reviewer_59uv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this paper is on federated learning in a setting where the edge clients do not have sufficient resources to train a large model and, additionally, the clients do not want to share any intermediate data and/or labels with the server. The main contribution in the paper is an algorithm, termed Principal Sub-Model (PriSM), in which the convolution kernels are mapped to the so-called *principal kernels* given by the singular value decomposition (SVD) at the server and then clients are asked to update low-rank approximations of the convolution kernels that are given by a subset of the principal kernels. While such an idea exists in the literature, as noted in the paper, the main idea the authors leverage in this paper is to probabilistically decide on the principal kernels that correspond to the submodels at each client.", "summary_of_the_review": "While the paper puts forth an effective idea, the contribution of the paper is limited and incremental. Such works are indeed valuable, but are perhaps better suited as workshop papers then as full conference papers.\n\n---\n\n**Response to the Authors**\n\nDear Authors,\n\nI greatly regret I was not able to engage with you during Phase I of the discussion period due to emergency of a personal nature. Please accept my sincerest apologies for this, as you put in significant efforts into the work and you deserved a timely response.\n\nI have gone through your response and have expanded / clarified parts of my review. I am however unable to raise my score because we have a disagreement on the novelty of the work. The ideas of low-rank matrix factorization and several other ad-hoc tweaks exist in the literature on compression of neural networks. While your work looks at the federated learning paradigm, and it utilizes an algorithm that helps learning take place in a federated setting, I maintain that these ideas by themselves are incremental in nature, when contrasted with the existing literature, and are not sufficient to warrant a publication in ICLR in the main conference. Perhaps this disagreement is because of I and you possibly coming from different communities, and I would let the AC and SAC resolve this disagreement.", "strength_and_weaknesses": "**Strengths**\n\n- The probabilistic selection of the principal kernels for the submodels using the strength of the singular values is an important insight that helps the authors with an improved algorithm.\n- The authors have carried out extensive numerical experiments to highlight the advantages of their approach.\n\n**Weaknesses**\n\n- The paper has an incremental nature, with the major difference from existing literature being a focus on probabilistic sampling of the principal kernels. Perhaps the authors can look into some theoretical aspects of PriSM to overcome the limited innovation in terms of the algorithm. (_Note added after reading authors' response:_ The literature being referred to here is the general literature on compression of neural networks. Low-rank matrix factorization and other structured factorizations, as well as many other ad-hoc tricks, are routinely utilized in works that study compression of neural networks.)\n- ~~The algorithm is very much limited to the case of convolutional neural networks and it is not clear how to adapt these ideas to the case of other architectures.~~ (_The reviewer agrees that some of the ideas in the paper are applicable to some other architectures._)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject