FedPara: Low-rank Hadamard Product for Communication-Efficient Federated Learning
Authors: Nam Hyeon-Woo
Moon Ye-Bin
Tae-Hyun Oh
Venue: ICLR 2022 Poster (2022)
Keywords: Federated learning
Parameterization
Communication efficiency

Abstract: In this work, we propose a communication-efficient parameterization, $\texttt{FedPara}$, for federated learning (FL) to overcome the burdens on frequent model uploads and downloads. Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conventional low-rank parameterization, our $\texttt{FedPara}$ method is not restricted to low-rank constraints, and thereby it has a far larger capacity. This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods. The efficiency of our method can be further improved by combining with other efficient FL optimizers. In addition, we extend our method to a personalized FL application, $\texttt{pFedPara}$, which separates parameters into global and local ones. We show that $\texttt{pFedPara}$ outperforms competing personalized FL methods with more than three times fewer parameters.

Published as a conference paper at ICLR 2022
FEDPARA: LOW-RANK HADAMARD PRODUCT FOR COMMUNICATION-ERRRIIENN FEDERATED LEARNING
Nam Hyeon-Woo', Moon Ye-Bin', Tae-Hyun Oh
Department of Flectrical Engineering, POSTECH 2Graduate School of AI, POSTECH
Yonsei University
Mtyeonw.nam, ybmoon, (aehyun)@postechh.
ABSTRACT
In this work, we propose a communication--ffiiinne parameterization, FedP ara, for federated learning (FL) overcome the burdens frequent model uploads and downloads. Our method re- oe-parametrrizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conven- tional low-rank parameterization, our FedPara method is not restricted to low- rank constraints, and thereby it has far larger capacity. This property enables achieve comparable performance while requiring 3 to 10 times lower communi- cation costs than the model with the original layers, which not achievable by the traditional low-rank methods. The efficiency of our method can be further im- proved by combining with other efficient FL. optimizers. In addition, we extend our method to personalized FL application, pFedP ara, which separates param- eters into global and local ones. We show that pF edPara outperforms competing personalized FL methods with more than three times fewer parameters. Project page: https: /tttpa::/gteee git..b..hhuugcgggg com/ South h-wuttthhwwwhhh hwtphh//FPPaaaaaa 2
ICLR22
INTRODUCTION
Federated learning (FL; McMahan et al., 2017) has been proposed as an efficient collaborative learn- ing strategy along with the advance and spread of mobile and IoT devices. FL allows leveraging local computing resources of edge devices and locally stored private data without data sharing for privacy. FL typically consists of the following steps: (1) clients download a globally shared model from central server, (2) the clients locally update each model using their own private data without accessing the others' data, (3) the clients upload their local models back to the server, and (4) the server consolidates the updated models and repeats these steps until the global model converges. FL has the key properties (McMahan al., 2017) that differentiate from distributed learning:
Heterogeneous data. Data is decentralized and non-IID as well as unbalanced in its amount due different characteristics of clients; thus, local data does not represent the population distribution.
Heterogeneous systems. Clients consist of heterogeneous setups of hardware and infrastructure; hence, those connections not guaranteed to be online, fast, cheap. Besides, massive client participation is expected through different communication paths, causing communication burdens.
These FL properties introduce challenges in the convergence stability with heterogeneous data and communication overheads. To improve the stability and reduce the communication rounds, the prior works in FL have proposed modified loss functions or model aggregation methods (Li et al., 2020; Karimireddy et al., 2020; Acar et al., 2021; Yu et al., 2020; Reddi al., 2021). However, the transferred data is still a lot for the edge devices with bandwidth constraints or countries having low-quality communication infrastructure. A large amount of transferred data introduces an energy consumption issue on edge devices because wireless communication is significantly more power- intensive than computation (Yadav& & Yadav, 2016; Yan et al., 2019).
'The gap hetween the fastest and the lowest communication speed across countries is significant; approximately 63 times different (Speedtest).

---

Published as a conference paper at ICLR 2022
In this work, we propose commumication-effciene re-parameterization for FL, FedP ara, which reduces the number bits transferred per round. FedPara directly re- re-parameterizes each fully- connected (FC) and convolutional layers of the model to have small and factorized form while preserving the model's capacity. Our key idea is to combine the Hadamard product with low-rank parameterization as W = (X, (Xx
E T= called low-rank Hadamard product. When then rank W) S r This outstanding property facilitates spanning full-rank matrix with much fewer parameters than the typical X n matrix. It signifi- cantly reduces the communication burdens during training. At the inference phase, we pre-compose and maintain W that boils down its original structure; thus, FedPas ara does not alter computa- tional complexity at inference time. Compared to the aforementioned prior works that tackle reduc- ing the required communication rounds for convergence, our rad a is an orthogonal approach in that F FedPa a r does not change the optimization part but re-defines each layer's internal structure.
rank(X, ) rank(X,) Y2
We demonstrate the effectiveness FrdPara with various network architectures, including VGG, ResNet, and LSTM, standard classification benchmark datasets for both IID and non-IID settings. The accuracy of our parameterization outperforms that of the traditional low-rank parameterization baseline given the same number of parameters. Besides, FedP ara has comparable accuracy to original counterpart models and even outperforms as the number of parameters increases at times. We also combine FedPa aa with other FL algorithms improve communication efficiency further.
We extend FedPara to the personalized FL. application, named pFedPara, which separates the roles of each sub-matrix into global and local inner matrices. The global and local inner matrices learn the globally shared common knowledge and client-specific knowledge, respectively. We de- vise three scenarios according the amount and heterogeneity of local data using the subset of FEMNIST and MNIST. We demonstrate performance improvement and robustness of pF eFedPa ara against competing algorithms. We summarize our main contributions follows:
We propose FedPara, low-rank Hadamard product parameterizatton for communication- efficient FL. Unlike traditional low-rank parameterization, we show that FedPaz ara can span a full-rank matrix and tensor with reduced parameters. We also show that FedPar requires up to ten times fewer total communication costs than the original model to achieve target accuracy. FedPa even outperforms the original model by adjusting ranks at times.
Our FedP ara takes novel approach: thereby, FedPas can be combined with other FL methods to get mutual benefits, which further increase accuracy and communication efficiency.
We propose pFedPara, personalization application of FedPara, which splits the layer weights into global and local parameters. pFedPara shows more robust results in challenging regimes than competing methods.
METHOD
In this section, we first provide the overview of three popular low-rank parameterizations in Sec- tion 2.1 and present our parameterization, FedP ara, with its algorithmic properties in Section 2.2. Then, we extend FedPara to the personalized FL application, pF redPaaa, in Section 2.3.
Notations. We denote the Hadamard product as C the Kronecker product as 69, n-mode tensor prod- uct X and the i-th unfolding of the tensor 7 E k xIxx given tensor T E R& xk,
2.1 OVERVIEW OF LOW-RANK PARAMETERIZATION
The low-rank decomposition in neural networks has been typically applied to pre-trained models for compression (Phan al., 2020), whereby the number parameters is reduced while minimizing the loss of encoded information. Given a learned parameter matrix W E m it is formulated finding the best rank-r approximation, arg min X Rm Y E R and r<C min (m, n)) It reduces the number of parameters from 0(m)) to O((m + n)r), and its closed-form optimal solution can be found by SVD.
||W W| such that W = XYT, where
This matrix decomposition is applicable to the FC layers and the reshaped kernels of the convolution layers. However, the natural shape of convolution kernel is a fourth-order tensor; thus, the low- rank tensor decomposition, such as Tucker and CP decomposition (Lebedev al., 2015; Phan al.,
2

---

Published as a conference paper at ICLR 2022
Yr (28xm)
Y1 (Rxn)
Y2
W (mice)
X (mx2R)
W (monn)
X1 (axX)
X2 (onxB))
W (mnNn)
W (mXoa)
rank =
man(WW = R²
(a) Conventional low-rank parameterization
(b) FedPara (Ours)
Figure I: Illustrations of low-rank matrix parameterization and FedPara with the same number of parameters 2R(m + n). (a) Low-rank parameterization is the summation of 2R number of rank- matrices, W = XY and rank W) < 2R. (b) FedPara is the Hadamard product of two
low-rank inner matrices, W W1 & W2 (X, Y ) (X2 2 and rank(W) RR²..
Layer
Parameterization
# Params.
Maximal Rank
Example [# Params. / Rank]
Original
min(m.n)
66 K
FC Layer
Low-rank
28(m + n)
2R
16 K/
PedPara
2f(m + m)
16 256
Original
OIK,K, 2R(O + RK:K) K2)
min OO K2,
590 K 256
Convolutional Layer
Low-rank
2R
21 K 32
FedE azs (Proposition 1)
2R(0 + IK,K,)
R²
82 K. 256
FedFara (Proposition 3)
2R(0 + RK;
256
Table 1: The number of parameters, maximal rank, and example. We assume that the weights of the FC and convolutional layers are in Rmxx and ROxIxK x.k, respectively, The rank of the convolutional layer is the rank of the 1st unfolding tensor. As a reference example, we set O 256, K₃ K₂ 3, and R 16.
2020), can be more suitable approach. Given a learned high-order tensor 7 R& x Tucker decomposition multiplies kernel tensor K. Th x- with matrices X, E Rrix., where rank(T(*) as T K. X X₁ X2 ... x Xn, and CP decomposition is the summation of rank-1 x x,". where x E A Likewise, it also reduces the number of model parameters. We refer these rank-constrained structure methods simply conventional low-rank constraints or low-rank parameterization methods.
tensors as T = Σ x x x
In the FL context, where the parameters frequently transferred between clients and the server during training, the reduced parameters lead to communication cost reduction, which is the main focus of this work. The nost-decomposition approaches (Lebedev al., 2015; Phan al., 2020) using SVD, Tucker, and CP decompositions do not reduce the communication costs because those are applied to the original parameterization after finishing training. That the original large-size parameters are transferred during training in FL, and the number of parameters is reduced after finishing training.
We take a different notion from the low-rank parameterizations. In the FL scenario, we train model from scratch with low-rank constraints, but specifically with low-rank Hadamard product parometerization. We re- ee-arameterize each learnable layer, including FC and convolutional layers, and train the surrogate model by FI.. Different from the existing low-rank method in FL (Konečný et al., 2016), parameterization can achieve comparable accuracy to the original counterpart.
2.2 FEDPARA: A COMMUNICATION-EFFEE
PARAMETERIZATION
As mentioned, the conventional low-rank parameterization has limited expressiveness due to its low- rank constraint. To overcome this while maintaining fewer parameters, present Hadamard product parameterization, called FedPara aaa, which has the favorable property as follows:
low-rank
Proposition 1 Let X, E Rmxri, X₂ E Rixxs Y, E n Y2 E Raxrs, r1,72 S min(m, n) and the constructed matrix be W = (X (XYY)) Then, Pank(W) I r-
All proofs can be found in the supplementary material including Proposition 1. Proposition 1 im- plics that, unlike the low-rank parameterization. higher-rande matrix can be constructed using the Hadamard product of two inner low-rank matrices, W1 and W2 (Refer to Figure 1). If we choose the
3

---

Published as a conference paper at ICLR 2022
inner ranks r1 and 122 such that r172 2 min(m, n)), the constructed matrix does not have low-rank restriction and is able to span a full-rank matrix with a high chance (See Figure 6); i.e., FedPa ara can control the number has the minimal parameter property achievable to full-rank. In addition, of parameters by changing the inner ranks rı and r2, respectively, but we have the following useful property set the ryper-parametery to be minimal number of parameters with maximal rank.
Proposition 2 Given R N. ri 12
R is the unique optimal choice of the following criteria,
arg min M.N
(rr (((++)) 8.1. rir2 R²
and its optimal value is 2R(m + n)).
Equation implies the criteria that minimize the number of weight parameters used in our parame- terization with the target rank constraint of the constructed matrix R². Proposition 2 provides an efficient way to set the hyper-parameters. It implies that, if we set 71 -==r== R and R 2 min(m, n), FedPara r is highly likely to have low-rank restriction even with much fewer parameters than that of a naïve weight, i... 2f(m + n) < mn. Moreover, given the same number of parameters, rank W) of FedParr ara higher than that of the naive low-rank parameterization by a square factor, as shown in Figure/ and Table 1.
To extend Proposition to the convolutional layers, we can simply reshape the fourth-order tensor kernel to the matrix as Ox
I (xx((I as naive way, where O, I, K1, and K₂ the output channels, the input channels, and the kernel sizes, respectively. That is, our parameteriza- tion spans convolution filters with few basis filters of size x Ki K2. However, we can derive more efficient parameterization of convolutional layers without reshaping as follows:
Proposition 3 Let Tı, To E Rxxxxx X1, X2 E RRIXR Y,, Y2 E RRaxR, R min(ki, ka) and the convolution kernel be W == (Ti x1 X1 X2 Y1) ② (T2 X1 X2 X2 22). Then, the rank of the kernel satisfies rank(WW()) = rank( S R
Proposition 3 is the extension of Proposition but can be applied to the convolutional layer without reshaping. In the convolutional layer of Table given the specific tensor size, Proposition requires 3.8 times fewer parameters than Proposition 1. Hence, we use Proposition 3 for the convo- lutional layer since the tensor method is more effective for common convolutional models.
Optionally, we employ non-linearity and the Jacobian correction regularization, of which details can be found in the supplementary material. These techniques improve the accuracy and convergence stability but not essential. Depending on the resources of devices, these techniques can be omitted.
2.3 PFEDPARA: PERSONALIZED FL APPLICATION
In practice, data are heterogeneous and personal due to different characteristies of each client, such as usage times and habits. FedPer (Arivazhagan al., 2019) has been proposed tackle this sce- nario by distinguishing global and local layers in the model. Clients only transfer global layers (the top layer) and keep local ones (the bottom layers) on each device. The global layers learn jointly to extract general features, while the local layers are biased towards each user.
Global
Local
Server
Server
t
Layer
Layer
Layer
Layer 1
Client
Client
(a) FedPer
(b) pFedaara (Ours)
With our FedP ara, we propose a personalization application, pFedPara, in which the Hadamard product is used as bridge between the global inner weight W1 and the local inner weight W2. Each layer of the personalized model is constructed by W = W, 3 (W2 + 1), where W, is transferred
Figure Diagrams of (a) FedPer and (b) pFedPare ara. The global part is transferred to the server and shared across clients, while the local part remains private in each client.
Its corollary and empirical evidence can be found in the supplementary material. Under Proposition 2, R² 2 min (m, n) is necessary and sufficient condition for achieving maximal rank.
4

---

Published as a conference paper at ICLR 2022
to the server while W2 is kept in local device during training. This induces W1 to learn glob- ally shared knowledge implicitly and acts as a switch of the term (W₂ + 1). Conceptually. we can interpret by rewriting W W,oWa W2+WW
W per. +W gio. where W per. Wi W2 and W glo = W. The construction of the final personalized parameter W in pFedPar arraaa can be viewed as an additive model of the global weight W glo and the personalizing residue W per.- PFedPar transfers only a half of the parameters compared to FedPara under the same rank condition; hence, the communication efficiency increased further.
Intuitively, FedPer and pFedPara are distinguished by their respective split directions, as illus- trated in Figure 2. We summarize our algorithms in the supplementary material. Although we only illustrate feed-forward network cases for convenience, it can be extended to general cases.
3 EXPERIMENTS
We evaluate our FedPara in terms of communication costs, the number of parameters, and com- patibility with other FL. methods. We also evaluate pFedPara in three different non-Ill) scenarios. We use the standard FL. algorithm, FedAvg, as a backhone optimizer in all experiments except for the compatibility experiments. More details and additional experiments can be found in the supplementary material.
3.1 SETUP
Datasets. In FedPara experiments, we use four popular FL datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), CINIC-10 (Darlow et al., 2018), and the subset of Shakespeare (Shake- speare, 1994). We split the datasets randomly into 100 partitions for the CIFAR- 10 and CINIC- 10 IID settings and 50 partitions for the CIFAR-100 IID setting. For the non-IlD settings, Dirichlet distribution for random partitioning and set the Dirichlet parameter a as 0.5 as suggested by He et al. (2020b). We assign one partition to each client and sample 16% of clients at each round during FL. In &FedPare experiments, we use the subset of handwritten datasets: MNIST (LeCun al., 1998) and FEMNIST (Caldas al., 2018). For the non-III setting with MNIST, we follow McMahan et al. (2017), where each of 100 clients has at most two classes
use the
Models. We experiment VGG and ResNet for the CNN architectures and LSTM for the RNN architecture as well as two FC layers for the multilayer perceptron. When using VGG, we use the VGG16 6 architecture (Simonyan & Zisserman, 2015) and replace the batch normalization with the group normalization as suggested by Hsieh et al. (2020). VGG1 6ori stands for the original VGG16, 6, VGG1 flo the one with the low-rank tensor parameterization in a Tucker form by following TKD (Phan et al., 2020), and VGG1 6redPara the one with our FedPara. In the pFedPara tests, we use two FC layers as suggested by McMahan et al. (2017).
Rank Hyper-parameter. We adjust the inner rank of W; and W₂ as (1-7)rmin + TT man.
where Tmin is the minimum rank allowing FedPara to achieve a full-rank by Proposition 2, Tmaz is the maximum rank such that the number of FedPara parameters do not exceed the number of original parameters, and y E [0, 1]. We fix the same y for all layers for simplicity. Note that ? determines the number of parameters.
3.2 QUANTITATIVE RESULTS
Capacity. In Table 2, we validate the propositions stating that our FedPara achieves a higher rank than the low-rank parameterization given the same number of parameters. We train VGG1 61ow and VGG1 GrGG6redPsre for the same target rounds T, and use 10.25% and 10%15% of the VGG1 6 param- eters, respectively, to be comparable. As shown in Table 2a, VGG1 6 surpasses VGG1 low on all the IID and non-IID settings benchmarks with noticeable margins. The same tendency is observed in the recurrent neural networks as shown Table 2b. We train LSTM on the subset of Shakespeare. Table 2b shows that LSTMeedFars has higher accuracy than LSTMoow where the num- ber of parameters is set to 19% and 16% of LSTM, respectively. This experiment evidences that our FedPara has better model expressiveness and accuracy than the low-rank parameterization.
'The parameter can he independently tuned for each layer in the model. Moreover, one may apply neural architecture search hapareparameter search algorithms for further improvement. We leave it for future work.
5

---

Published as a conference paper at ICLR 2022
CIFAR-10 (7"= 200)
CIFAR- 10AR-100 (7 400)
CINIC-10 (T = 300)
(a) CNN Models
IID
non-IID
IID
non-IID
IID
non-lID
VGGL Sjow
77.62
67.75
34.16 45.78
30.30
63.98
60.80
VGGL SpedPara (Ours)
82.88
71.35
43.94
70.35
64.95
Shakespeure (? 500)
RNN Model
ITI
non-Ill)
54.59
51.24
T.STTMeedPara (Ours)
63.65
51.56
Accuracy comparison between low-rank parameterization and FedPara, (a) The accuracy VGG1 6lo and VGG1 6radpara- We set the target rounds 200 CIFAR-IO, 400 for CIFAR- 100, and 300 for CINIC-10. (b) The accuracy of LSTMo and I.S ISTMFaaParr. rounds T 500 for the Shakespeare dataset.
Table 2:
We set the target
V6616
VG616 (Ours)
84.0
51.0
72.0
56.5
Acc. [%)
Acc. [%6
47.0
30.5
41.0
28.5
20.2
25.5
10.0
86
172
259
345
10.00
110
165
220
10.00
136
272
409
545
Communication Cost [GB)
Cammunication Cost
Communication Cost
CIFAR- IID (y=0.1)
(b) CIFAR 100 (y=0.4)
(c) CINIC- IID (y=0.3)
47.0-
67.0
58.0
37.8
52.8
Acc, [9]
Ncc
26.0
19.2
24.2
10.00
50
270 350
10.00
270 360
10.0
Communication Cost [GB]
Communication Cost (GB)
Communication Cost (G8)
(d) CIFAR-10 non-IID (7=0.1)
(e) CIFAR-100 non-IID (7=0.4)
(I) CINIC-10 non-lID (y=0.33
Communic adon Cost (GB]
VG616
VGG16resFars (Ours)
378.0/9.3
Energy Consumption Consummil [M]]
283.5/7.0
235.1/5.8
205.3/5.1
154.9/3.8
20.3/0.5
32.5/0.8
55.2/1.4
67.3/1.7
46.3/1.1
53.9/1.3
CIFARIO (80%)
CIFARIO non-IID (70%)
CIFAR100 IID (50%)
CIFAR100 non-IID (45%)
CINIC10 (70%)
CINIC10 nen-IID (65%)
(g) Communication Costs
Figure 3: (a-f): Accuracy [%] (y-axis) vs. communication costs [GBytes] (æ-axis) of VGG1 6uri. and VGG1 6redPara- Broken line and solid line represent VGG1 6ori. and VGG1 6reaparas respectively. (g): Size comparison of transferred parameters, which can be expressed [GBytes] (left y-axis) energy consumption [MJ] (right y-axis). for the same target accuracy. The white bars are the results of VGGl 6ori. and the black bars are the results of VGG1 6pedPars- The target accuracy is denoted in the parentheses under the a-axis of (g).
communication costs
Communication Cost. We compare VGG1 6rodPara and VGG1 6ari. in terms of accuracy and communication costs. FL evaluation typically measures the required rounds to achieve the tar- get accuracy communication costs, but we instead assess total transferred bit sizes, 2 x (#participants) x (model size) x (#rounds), which considers up-/down-link and a more prac- tical communication cost metric. Depending the difficulty of the datasets, set the model size of VGG1 6reasara as 10.1%, 29.4%. and 21.8% of VCG1 6 for CIFAR-10 10, CIFAR- 100, and CINIC-10, respectively.

---

Published as a conference paper at ICLR 2022
VGG16
VGG16rastara (Ours)
84.10
51.50
73.70
83.65
50.07
72.86
Acc [8]
NoC. [%]
82.76
47.21
71.19
82.310
45.7
70.340
67
Parameters [%]
Parameters (%)
Parameters [%]
(a) CIFAR-I0 IID
(b) CIFAR-100 IID
(c) CINIC-10 IID
74.72
47.36
67.24
[%]
Acc. [96]
72.47
45.08
65.71
71.346
43.94
64.95
Parameters 1%)
Parameters [%6
Parameters (%)
(d) CIFAR-I0 non-lID
(e) CIFAR-100 non-IID
(I) CINIC-10 non-IID
Test accuracy [%] (y-axis) vs. parameters ratio [%] (æ-axis) VGG1 6redvers the target rounds. The target rounds follow Table 2. The dotted line represents VGGl 6ori. with no parameter reduction, the solid line VGG1 6reapara adjusted by y 10.1, 0.9) in 0.1 increments.
Figure 4:
FedAvg
FedF
SCAFFOLD
FedDyn
FedAdam
Accuracy (T" = 200)
82,88
78.95
84.72
86.05
82.48
Round (80%)
110
92
80
117
Table 3: The compatibility of FedPara with other FL algorithms. The first row is the accuracy of FedP ara combined with other FL algorithms on the CIFAR-10 IID setting after 200 rounds, and the second row is the required rounds to achieve the target accuracy 80%.
In Figures 3a-3f, VGG1 6redF has comparable accuracy but requires much lower communica- tion costs than VGG1 6 Figure 3g shows communication costs and energy consumption required for model training to achieve the target accuracy; we compute the energy consumption by the en- ergy model of user-to-data center topology (Yan et al., 2019). VGG1 6pedpara needs 2.8 10.1 times fewer communication costs and energy consumption than VGG1 6 to achieve the same tar- get accuracy. Because of these properties, FedPara is suitable for edge devices suffering from communication and battery consumption constraints.
Model Parameter Ratio. We analyze how the number of parameters controlled by the rank ratio affects the accuracy of FedPara. As revealed in Figure 4, VGG1 SreaFara 's accuracy mostly increases the number of parameters increases. VGGl 6reaPaza can achieve even higher accuracy than VGG1 6 It is consistent with the reports from the prior works (Luo et al., 2017: Kim et al., 2019) on model compression, where reduced parameters often lead to accuracy improvement, iee. regularization effects.
Compatibility, We integrate the FedP CitPara-basee model with other FL optimizers show that our FedPara is compatible with them. We measure the accuracy during the target rounds and the required rounds achieve the target accuracy. Table 3 shows that VGG1 6reaFara combined with the current ofte--ofthheart method, FedDyn, the best among other combinations. Thereby, can further save the communication costs by combining FedPa ara with other efficient FL approaches.
Personalization. We evaluate pFedPara ara, assuming sub-sampling of clients for update. We train two FC layers on the FEMNIST or MNIST datasets using four algorithms, FedPAO, FedAvg, FedFer, and pF aFedPara, with ten clients. FedPAQ denotes the local models trained only using their own local data; FedAvg the global model trained by the FedAvg optimization; and FedPer and pFedPa the personalized models of which the first layer (FedPer) and half of the inner

---

Published as a conference paper at ICLR 2022
Local
FedAvg
FedPer
PFedPara (Ours)
75.0
59.0
99.5
(86)
[99
66.5
50.5
96.0
62.2
46.2
94.2
58.0
Local
FedAvg FeePer pFesPara
42.0
Local
Feoling
FedPer prodaars
92.5
Local
Festin
FedPer pFosPara
(a) Scenario
(b) Scenario 2
(c) Scenario
Figure 5: Average test accuracy over ten local models trained by each algorithm. (a) 100% of local training data on FEMNIST are used with the non-IID setting, which mimics enough local data to train and evaluates each local model their own data. (b) 20% of local training data FEMNIST are used with the non-llD setting, which mimics insufficient local data to train local models. (c) 100% of local training data MNIST are used with the highly-skew non-IlD setting, where each client has at most two classes. The error bars denote 95% confidence intervals obtained by 5 repeats.
matrices (pFedP ara) are trained by sharing with the server, respectively, while the other parts are locally updated. We validate these algorithms three scenarios in Figure 5.
In Scenario (Figure 5a), the FedPAQ accuracy is higher than those of FedAvg and FedPer because each client has sufficient data. Nevertheless, pFedPara surpasses the other methods. In Scenario (Figure 5b), the FedAvg accuracy is higher than that of FedPAQ because local data too scarce train the local models. The FedPez accuracy is also lower than that of FedAvg because the last layer of FedPer does not exploit other clients' data; thus, FedPer is susceptible lack of local data. The higher performance advantage of the wealth of distributed data in the personalized setup. In Scenario 3 (Figure 5c), the FedAvg accuracy is much lower than the other methods due to highly-skewed data distribution. In most scenarios, pF PFedPara performs better favorably against the others. It validates that PF reePaaa can train the personalized models collaboratively and robustly.
pFedPara shows that our pFedP ara can take
Additionally, both pFedPara and FedPer save the communication costs because they partially transfer parameters; pFedPara transfers 3.4 times fewer parameters, whereas FedPer transfers 1.07 times fewer than the original model in each round. The reduction of FedPer is negligible because it is designed to transfer all the layers except the last one. Contrarily, the reduction of factorized
PF edP ara is three times larger than FedPer because all the layers of pFedPaze by the Hadamard product during training, and only a half of each layer's parameters are transmitted. suitable in terms of both personalization performance and communi-
Thus, pFedPar ra is far cation efficiency in FL.
Additional Experiments. We conducted additional experiments of wall clock time simulation and other architectures including ResNet, LSTM, and Puf ferfist fisshhhh (Wang et al., 2021), but the results can be found in the supplementary material due the page limit.
RELATED WORK
Federated Learning. The most popular and de-facto algorithm, FedAvg (McMahan al., 2017), reduces communication costs by updating the global model using a simple model averaging once large number local SGD iterations per round. Variants of FedAvg (Li et al., 2020; Yu al., 2020: Diao et al., 2021) have been proposed to reduce the communication cost, to overcome data heterogeneity, to increase the convergence stability. Advanced FL. optimizers (Karimireddy al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) enhance the convergence behavior and improve communication efficiency by reducing the number necessary rounds until convergence. Federated quantization methods (Reisizadeh et al., 2020; Haddadpour et al., 2021) combine the quantization algorithms with FedAvg and reduce only upload costs to preserve the model accuracy. Our FedPare ara is drop-in replacement for layer parameterization, which means it is orthogonal and compatible approach to the aforementioned methods; thus, our method can integrated with other FL. optimizers and model quantization.
8

---

Published as a conference paper at ICLR 2022
Distributed Learning. In large-scale distributed learning of data centers, communication might be a bottleneck. Gradient compression approaches, including quantization (Alistarh et al., 2017: Bern- stein al., 2018; Wen et al., 2017; Reisizadeh et al., 2020; Haddadpour et al., 2021), sparsification (Alistarh et al., 2018; Lin et al., 2018), low-rank decomposition (Vogels et al., 2019; Wang et al., 2021), and adaptive compression (Agarwal al., 2021) have been developed cation traffic. These methods do not deal with FL properties such as data distribution and partial participants per round (Kairouz et al., 2019). Therefore, the extension of the distributed methods to approaches.
handle communi-
FL is non-trivial, especially for optimization-base
Low-rank Constraints. As described in section 2.1, low-rank decomposition methods (Lebedev et al., 2015; Tai et al., 2016; Phan et al., 2020) are inappropriate for FL due to additional steps; the post-decomposition after training and fine-tuning. (2021) have proposed low-rank approaches. Konečný et al. (2016) train the model from scratch with low-rank constraints, but the accuracy degraded when they set a high compression rate. To avoid such degradation, FedDLR (Qiao et al., 2021) uses an ad hoc adaptive rank selection and shows the improved performance. However, once deployed, those models are inherently restricted by lim- ited low-ranks. In particular, FedDLR requires matrix decomposition in every up/down transfer. In contrast, we show that FedP ara has no such low-rank constraints in theory. Empirically, the mod- els re-parameterized by method show comparable accuracy to the original counterparts when trained from scratch.
Fl., Konečný et al. (2016) and Qiao et al.
5 DISCUSSION AND CONCLUSION
To overcome the communication bottleneck in FL. propose new parameterization method, F FedPara, and its personalized version, pFedP ara. We demonstrate that both FedP ara and pFedPara can significantly reduce communication overheads with minimal performance degra- dation or better performance over the original counterpart at times. Even using a strong low-rank constraint, FedP ara has no low-rank limitation and can achieve full-rank matrix and tensor by virtue of our proposed low-rank Hadamard product parameterization. These favorable properties en- able communication-effe effficient FL. of which regimes have not been achieved by the previous low-rank parameterization and other FL approaches. We conclude our work with the following discussions.
Discussions. FedPara conducts multiplications many times during training, including the Hadamard product, to construct the weights of the layers. These multiplications may potentially be more susceptible to gradient exploding. vanishing, dead neurons, or numerical instability than the low-rank parameterization with an arbitrary initialization. In our experiments, we have not ob- served such issues when using He initialization (He et al., 2015) yet. Investigating initializations appropriate for our model might improve potential instability in our method.
Also, we have discussed the expressiveness of each layers in neural networks the view of rank. As another view of layer characteristics, statistical analysis of weights and activation also offers ways to initialize weights (He et al., 2015; Sitzmann et al., 2020) and understanding of neural networks (De & Smith, 2020), which is barely explored in this work. It would be promising future direction to analyze statistical properties composited weights from our parameterization and activations and may pave the way for FedP ara-sceiific initialization or optimization.
Through the extensive experiments, we show the superior performance improvement obtained by our method, and it appears to be with extra cost. However, the actual payoff exists in the additional computational cost when re-composing the original structure of W from our parameterization dur- ing training: thus, our method is slower than the original parameterization and low-rank approaches. However, the computation time is not a dominant factor in practical FL scenarios as shown in Ta- ble 7, but rather the communication cost takes a majority of the total training time. evidenced by the fact that FedPars offers better Pareto efficiency than all compared methods because our method has higher accuracy than low-rank approaches and much less training time than the original one. In contrast, the computation time might be non-negligible compared to the communication time in distributed learning regimes. While our method can be applied to distributed learning, the benefits of our method may be diminished there. Improving both computation and communication efficiency of FedP ara in large-scale distributed learning requires further research. It would a promising future direction.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We describe the ethical aspect in various fields, such as privacy, security, infrastructure level gap, and energy consumption.
Privacy and Security. Although FL is privacy-preserving distributed learning, personal informa- tion may be leaked due to the adversary who hijacks the model intentionally during FL. Like other FLs, this risk is also shared with FedPars araa due communication. Without care, the private data may be revealed by the membership inference or reconstruction attacks (Rigaki & Garcia, 2020). The local parameters in pFedPara could be used personal model, which would reduce the chance for the full model to be hijacked. It would be inter- esting investigate whether our pFedPara guarantees privacy preserving or the way improve robustness against those risks.
private key acquire the complete
Infrastructure Level Gap. Another concern introduced by FL is limited-service access to the people living in countries having inferior communication infrastructure, which may raise human rights concerns including discrimination, excluding, etc. is due to larger bandwidth requirement of FL to transmit larger models. Our work may broaden the countries capable of FL by reducing required bandwidths, whereby may contribute to addressing the technology gap between regions.
The communication efficiency of our FedP ara directly leads to noticeable energy-saving effects in the FL scenario. can contribute to reducing the battery consumption of loT devices and fossil fuels used to generate electricity. Moreover, compared to the optimization- based FL approaches that reduce necessary communication rounds, our method allows more clients to participate in each learning round under the fixed bandwidth, which would improve convergence speed and accuracy further.
Energy Consumption.
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No.2021-0-0200) Artificial In- telligence Innovation Hub), the National Research Foundation of Korea (NRF) grant funded by the 11-2002RRIICC10060
Korea government (MSIT) (No. ported by the 'Ministry of Science and ICT' and NIPA.
and the "HPC Support" Project sup-
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Con- ference on Learning Representations (ICIR), 2021.
Saurabh Agarwal, Hongyi Wang, Kangwook Lee, Shivaram Venkataraman, and Dimitris Papail- iopoulos. Adaptive gradient communication via critical learning regime identification. In Machine Learning and Systems (MLSys), 2021.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficeeet sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NeurIPS),2011
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cedric Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint 191.00111,.00818, 2019.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli. and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning (ICMI., 2018.
10

---

Published as a conference paper at ICLR 2022
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. Advances in Neural Information Processing Systems Workshops (NeurIPSW), 2018.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic- is not imagenet or cifar-10, arXiv preprint arXiv:1810.03500 2018.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In Advances Neural Information Processing Systems (NeurIPS), 2020.
Enmao Diao. Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient
federated learning for heterogeneous clients. In International Conference Learning Represen- fations (ICLR), 2021.
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Feder- ated learning with compression: Unified analysis and sharp guarantees. In Artificial Intelligence and Statistics (AISTATS), 2021.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Feder- ated learning of large cnns the edge. In Advances Neural Information Processing Systems (NeurIPS), 2020a.
Chaoyang He, Songze Li, Jinhyun So. Mi Zhang, Hongyi Wang, Xiaoyang Wang, Pranecth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, et al. Fedml: A research library and benchmark for federated machine learning. Advances in Neural Information Processing Systems Workshops (NeurIPSW), 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. International Conference on Machine Learning (ICML), 2020.
Yo-Seb Jeon, Mohammad Mohammadi Amiri, Jun Li, and H Vincent Poor. A compressive sensing approach for federated learning over massive mimo communication systems. IEEE Transactions on Wireless Communications, 1990):199,-2004 2020.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint 11Xiv:1912.00977, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning (ICML). 2020.
Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung. Efficient neural network com- pression. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up up convolutional neural networks using fine-tuned cp-decomposition. In International Conference Learning Representations (ICLR), 2015.

---

Published as a conference paper at ICLR 2022
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 2324,, 1998.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Machine Learning and Systems (MLSys), 2020.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations (ICLR), 2018.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision (ICCV), 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiee learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics (AISTATS), 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavský, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor de- composition for compression of convolutional neural network. In European Conference on Com- puter Vision (ECCV), 2020.
Zhefeng Qiao, Xianghao Yu, Jun Zhang, and Khaled B Letaief. Communication--fficieee federated learning with dual-side low-rank compression. arXiv preprint arXiv:2104. 12416, 2021.
Stephan Rabanser, Oleksandr Shchur, and Stephan Günnnemann. Introduction to tensor decomposi- tions and their applications in machine learning. arXiv preprint 17Xiv:/711..107) 2017.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations (ICLR), 2021.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaic, and Ramtin Pedarsani. Fedpaq: A communication-effiiee federated learning method with periodic averaging and quan- tization. In Artificial Intelligence and Statistics (AISTA7S), 2020.
Maria Rigaki and Sebastian Garcia. A survey of privacy attacks in machine learning. arXiv preprint arXiv:2007.07646. 2020.
William Shakespeare.
The complete works of william shakespeare. nttps://wwe gutenberg. www.or/oggg/////////ooooo/ 1994.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Speedtest. SPEEDTEST. httta://ww..seattst..ne/gggaaa index. Accessed: 2021-05-26.
Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low- rank regularization. In International Conference on Learning Representations (ICLR), 2016.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
12

---

Published as a conference paper at ICLR 2022
Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Commumication--fficee models at no extra cost. In Machine Learning and Systems (MLSys), 2021.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Sarika Yadav and Rama Shankar Yadav. A review on energy efficient protocols in wireless sensor networks. Wireless Networks, 22(1):335-335 2016.
Ming Yan, Chien Aun Chan, André F Gygax, Jinyao Yan, Leith Campbell, Ampalavanapillai Nir- malathas, and Christopher Leckie. Modeling the total energy consumption of mobile network services and applications. Energies, 12(1))184, 2019.
Felix Yu, Ankit Singh Rawat, Aditya Menon, and Sanjiv Kumar. Federated learning with only positive labels. In International Conference on Machine Learning (ICML), 2020.
Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. In Advances in
Neural Information Processing Systems (NeurIPS), 2020.
Guangxu Zhu, Yuqing Du, Deniz Gündüz, and Kaibin Huang. One-bit over-the-air aggregation for communication-effiiii federated edge learning: Design and convergence analysis. 1EEE Transactions Wireless Communications, 20(3):2120-213 2020.
13

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY
In this supplementary material, we present additional details, results, and experiments that are not included in the main paper due to the space limit. The contents of this supplementary material are listed as follows:
CONTENTS
Maximal Rank Property
Additional Techniques
Details of Experiment Setup
C.1 Datasets
C.2 Models
C33 FedPara & pFedPara
C.4 Hyper-parameters of Backbone Optimizer
C.5 Slyee--arameter of Other Optimizer
D Additional Experiments
D.1 Training Time
D.2 Other Models
A MAXIMAL RANK PROPERTY
This section presents the proofs of our propositions and an additional analysis our method's algorithmic behavior in terms of the rank property.
A.1 PROOFS
Proposition 1 Let X1 RMXX, X2 E Rmxrs Y1 paxri Y2 Rax r1,72 S min n) and the constructed matrix be W = (X (x,Y)) Then, rank(W) S 51r2
Proof. X, YI and X₂ Y.J can be expressed as the summation of rank-1 matrices such that X, Y.Y. === X,yYij, where X.j and yiz are the j-th column vectors of X, and Y,, and i [1,2]. Then,
j=n
j=ra
k= jmre
W = x O X.Y, = Σ x1,y1
Σ
X23 = Σ Σ
(XILYTa) @ (x2yy
(2)
j=1
J=1
Ø=1 1=1
W is the summation of r172 number of rank-1 matrices; thus, rank(W) is bounded above 7172- Proposition 2 Given R E N. ri =
R is the unique optimal choice of the following criteria,
arg min r.Nr
(r1 r2) +)) s.t. r1f2 2 R²
(3)
and its optimal value is 2R(m + n)).
Proof. We use crtthetii-geomeete mean inequality and the given constraint. We have (r1 + +2) + n) 2 2 n) 2 2R(m + n).
14

---

Published as a conference paper at ICLR 2022
The equality holds if and only r1 r2 = R. by the arithmetic-geenn geometric mean inequality.
Corollary Under Proposition 2, R > min(m, n) is necessary and sufficient condition for achieving the maximal rank of W (X Y o (X)Y2) E Rmxx, where X1 E Ruxr, X2 Rmxra YIER R Y2
Ruxx? and r1, 12 min(m,n).
Proof. We first prove the sufficient condition. Given ri = 72 R under Proposition 2 and R² > min(m, n),, rank(W) I min(r1f2, m, n) = min( m, n) = min(m, nt). The matrix W has no low-rank restriction; thus the condition, R² 2 min(m, n), is the sufficient condition.
The necessary condition proved by contraposition; if R < min(m, n). the matrix W cannot achieve the maximal rank. Since r1 = 72 R under Proposition 2and R < min(m, n2,, then rank(W) (W) S min(r172, m, n) min R² m. n)) = R² < min(m, n). That is, rank( is upper- bounded by R2, which is lower than the maximal achievable rank of W. Therefore, the condition, R2t 2 min(m, n), is the necessary condition because the contrapositive is true.
Corollary implies that, with n² 2 min(m, n). the constructed weight W does not have the low- rank limitation, and allows us to define the minimum inner rank as Yimn == min( ( m 1, [V 1)- we set T1 12 Tmins rank W) our FedPara can achieve the maximal rank because T172 rimin 2 min (m, n.) while minimizing the number of parameters.
Proposition 3 Let Tı, To E R Rx x X1, X2 E RR Y1, Y2 E RR R min(k1, k2) and the convolution kernel be W (Ti X X2 (T2x) Y2). Then, the rank the kernel satisfies rank(W(()) rank(W()) R.a.
Proof. According to Rabanser et al. (2017), the 1 and 2nd unfolding of tensors can be expressed as
)x(2) (x, T, (T(4)
(3)
Y1) (X272
Y2)T),
(S)
W (Y, ( (T
X1 (Y&T2² 22)
X₂ T)
where ((3 E x and (4) E & are identity matrices. Since W(1) and W(2) are matrices, we apply the same process used Eq. 2, then we obtain rank(W(())) = rank(Wv(2)) < R?.
A.2 ANALYSTS OF THE RANK PROPERTY
To demonstrate propositions empirically, sample the parameters randomly and count rank(W). When applying our parameterization to W E R 100x 100 we set T'mn = 10 by Corol- lary Proposition 2. We sample the entries of X1, X2, Y1, Y2 100010 from the standard Gaussian distribution and repeat this experiment 1, 000 times.
As shown in Figure 6, observe that our parameterization achieves the full rank with the probabil- ity of 100% but requires 2.5 times fewer entries than the original 100 x 100 matrix. This empirical result demonstrates that our parameterization can span the full-rank matrix with fewer parameters efficiently.
1000
750
Count
500
250
0
20
40
60
80
100
Rank
Figure 6: Histogram of rank rank(W). We apply FedPara to W E 10001100 and set 7 = =72 = 10. We repeat this experiment 1, 000 times.
15

---

Published as a conference paper at ICLR 2022
B ADDITIONAL TECHNIQUES
We devise additional techniques to improve the accuracy and stability of our parameterization. We consider injecting non-linearity in FedPara and Jacobian correction regularization to further squeeze out the performance and stability. However, as will be discussed later, these are optional.
Non-linear Function. The first is apply non-linear function before the Hadamard product. FedPara composites the weight as W W13We and we inject non-linearity as W o (WW, 1)@ (W2) as a practical design. This makes our algorithm departing from directly applying the proofs, le., empirical heuristics. However, the favorable performance of our FedP ara regardless of this non-linearity injection in practice suggests that our re-parameterization is a reasonable model to deploy.
The non-lineat function a--) can any non-linear function such as ReLU, Tanh, and Sigmoid, but ReLU and Sigmoid restrict the range to only positive, whereas Tanh has both negative and positive values of range [-1,1]. The filters may learn to extract the distinct features, such as edge, color, and blob, which require both negative and positive values of the filters. Furthermore, the reasonably bounded range can prevent overshooting a large value by multiplication. Therefore, Tanh is suitable, and we use Tanh through experiments except for those of pF FedPara.
Jacobian Correction. The second is the Jacobian correction regularization, which induces that the Jacobians of X1, X2, Y1, and Y2 follow the Jacobian of the constructed matrix W. Suppose that X1, X2 E Rmxr and Y1, Y2 E R are given. We construct the weight as W = W1 @ W2, where W = =,=x,Y;; Y and W₂ X2 Additionally, suppose that the Jacobian of W with respect aL
to the objective function is given: Jw 8w- We compute the Jacobians of X1 X2, Y1, and Y, with respect to the objective function and apply one-step optimization with SGD. For simplicity, set the momentum and the weight decay be zero. Then, we can compute the Jacobian of other variables using the chain rule:
ØL
OL
ØL
Jwi
Jw
W2,
Jx,
Jw, YT, JY,
X
J.
OW
0X,
UY,
(6)
ØL
OL
OL
Jw,
Jw
& W1, Jx, OX,
= Jw, Y,, JY, = UY,
x, Jwa
OW2
We update the parameters using SGD with the step size follows:
x = X1 nJ,,
Y = Y2 - nJy,,
X X2 nJxx
Y- = Y2 - nJy,
We can compute W which is the constructed weight after one-step optimization:
W' = =XxYYI) о (X)Y.)
=((X, (Y. nJ (X2 nJx,) v]]2)]
-XXYY (Jx, Y. x, + (X2Y, [Jx Y2 xxJT,) + +x
=(X,Y]) ( +
(Jx,Jy,)
-i2((xx, YI +X,
(Jx, Y. X,JY,) (Jx,J,,))
+I/[(Jx, Yi +X, [xx, Y x.Jy,) (Jx,JY,) (X,Y (Jx,JY,) (X,Y)))
-n((Jx, Y. x,JY, (X.Y)) ( X.JY.) (X,Y.)).
(8)
This shows that gradient descent and ascent are mixed, as shown in the signs of each term. We propose the Jacobian correction regularization to minimize the difference between W and W nJ which induces our parameterization to follows the direction of W -JJJ.w The total objective function consists of the target loss function and the Jacobian correction regularization as:
R L(X, X2, Y
W - (W - wjw]ll))ll.
2
Results. We evaluate the effects of each technique in Table 4. We train VGG16 6 with group normal- ization the CIFAR-10 IID setting during the same target rounds. We set 7 0.1 and À 10. As
16

---

Published as a conference paper at ICLR 2022
Models
Accuracy
F FedPara (base)
82,45 + 0.35
+ Tanh
82.42 0.33
+ Regularization
82.38 + 0.30
+ Both
82.52 + 0.26
Table 4: Accuracy of FedPaze ara with additional techniques. 95% confidence intervals are presented with eight repetitions.
shown, the model with both Tanh and regularization has higher accuracy and lower variation than the base model. There is gain in accuracy and variance with both techniques, whereas only variance with only one technique.
Again, note that these additional techniques not essential for FedPara work; therefore, we can optionally use these techniques depending on the situation where the device has enough computing power.
C DETAILS OF EXPERIMENT SETUP
In this section, we explain the details of the experiments, including datasets, models, and hyper- parameters. We also summarize our FedPara ara and pFedPara into the pseudo algorithm. For implementation, we use PyTorch Distributed library (Paszke et al., 2019) and 8 NVIDIA GeForce RTX 3090 GPUs
C.1 DATASETS
CIFAR-10. CIFAR-10 (Krizhevsky et al., 2009) is the popular classification benchmark dataset. CIFAR-10 10 consists of 32 x 32 resolution images in 10 classes, with 6, 000 images per class. We use 50, 000 images for training and 10, 000 images for testing. For federated learning, we split training images into 100 partitions and assign one partition to each client. For the IID setting, we split the dataset into 100 partitions randomly. For the non-lID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0.5 suggested by He et al. (2020b:a).
CIFAR-100. CIFAR-100 (Krizhevsky et al., 2009) is the popular classification benchmark dataset. CIFAR-100 consists of 32 32 resolution images in 100 classes, with 6, 000 images per class. We use 50, 000 images for training and 10, 000 images for testing. For federated learning, we split training images into 50 partitions. For the IID setting, we split the dataset into 50 partitions randomly. For the non-lID setting, we use the Dirichlet distribution and set the Dirichlet parameter 0.5 as suggested by He al. (2020b;a).
CINIC-10. CINIC-10 (Darlow et al., 2018) is a drop-in replacement for CIFAR-10 and also the popular classification benchmark dataset. CINIC-10 consists of 32 x 32 resolution images in 10 classes and three subsets: training, validation, and test. Each subset has 90, 000 images with 9, 000 per class, and we not use the validation subset for training. For federated learning, we split training images into 100 partitions. For the IID setting, we split the dataset into 100 partitions randomly. For the non-lID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0.5 as suggested by He et al. (2020b:a).
popular handwritten number image dataset. MNIST consists of 70, 000 number of 28 x 28 resolution images in 10 classes. We use 60, 000 images are for training and 10, 000 images for testing. We do not use MNIST IID-seting, and we split the dataset so that clients have at most two classes as suggested by McMahan et al. (2017) for highly-skew non-IID setting.
MNIST. MNIST (LeCun et al., 1998)
FEMNIST. FEMNIST (Caldas al., 2018) is a handwritten image dataset for federated settings. FEMNIST has 62 classes and 3, 550 clients, and each client has 226.83 data samples on average of 28 28 resolution images. FEMNIST is the non-IIID dataset labeled by writers.
17

---

Published as a conference paper at ICLR 2022
Shakespeare. Shakespeare (Shakespeare, 1994) is next word prediction dataset for federated learning settings. Shakespeare has 80 classes and 1, 129 clients, and each client has 3, 743.2 data samples average (Caldas et al., 2018).
C.2 MODELS
VGG16. Py PyTorh library (Paszke et al., 2019) provides VGG1 6 with batch normalization, but we replace the batch normalization layers with the group normalization layers suggested by Hsieh et al. (2020) for federated learning. We also modify the FC layers to comply with the number of classes. The dimensions of the output features in the last three FC layers are 512-512- (#laasee)) sequentially. We do not apply our parameterization to the last three FC layers and set the same all convolutional layers in the model for simplicity. For reference purpose, Table shows the number of parameters corresponding to each 7-
No. parameters
10-classes
100-classes
original
15.25M
15.30M
0.1
1.55 M
1.59 M
0.2
2.33 M
2.38 M
0.3
3.31 M
3.36 M
0.4
4.45 M
4.50 M
0.5
5,79 M
5.84 M
0.6
7.33 M
7.38 M
0.7
9.01 M
9.05 M
0.8
10.90 M
10.94 M
0.9
12.92 M
12.96 M
Table 7's and their corresponding numbers of parameters for VGG1 6ori and VGG1 6gedPare-
Two FC Layers. In personalization experiments, we use two FC layers as suggested by McMahan et al. (2017) but modify the size of the hidden features corresponding to the number of classes in the datasets. The dimensions of the output features in FC layers are 256 and the number of classes, respectively; i.e., 256((#classes) We do not use other layers, such normalization and dropout, and set 0.5 for pFedPara.
LSTM. For the Shakespeare dataset, we use two-layer LSTM as suggested by McMahan et al. (2017) and Acar et (2021). We set the hidden dimension as 256 and the number of classes as 80. We also apply the weight normalization technique on original parameterization, low-rank parameterization, and F redPara.
C.3 FEDPARA & PFEDPARA
Algorithm 1: FedPara
Input: rounds T. parameters (Xi, X21, Yu, Y2)EE where (X1r, X21, Yi,, Y23 is the layer of the model and L is the number of layers
for 1,2, T do
Sample the subset S of clients:
for each client E S do
Download { (Xat, Xat, Yıs, Y
i= from the server;
Optimize( ( X X2s Yu, Y2 1E
Upload {Xu, X2s, Yu, Yu 1e/ the server;
end
Aggregate (Xn, Xmi Yn, Velh
end
18

---

Published as a conference paper at ICLR 2022
Algorithm 2: pFedPara
Input:
rounds T, parameters {X X21, Yu, Yi1hii where (X1,, X21, Yu, Y2) is the 1th
layer of the model and I, is the number of layers
Transmit (Xu, X2t, it, Y=) [= clients train the same initial point start;
for ........ do
Sample the subset $ of clients:
for each client E S do
Download half parameters (Xur, Kf from the server;
Optimize({ X n1 X2,, Yu, Y2)i=D:
Upload {Xu,, Yu the server; end
Aggregate (Xu, Yuhii:
end
We summarize our two methods, FedPazz ara and 3FedPara, into Algorithms 1 and 2 for apparent comparison. In these algorithms, we mainly use the popular and standard algorithm, FedAvg, as backbone optimizer, but we can switch with other optimizer and aggregate methods. As mentioned in Section B, we can consider the additional techniques. In the FedP ara experiments, we use the regularization to Algorithm I, and set the regularization coefficient as 1.0. pFedPara experiments, we do not apply the additional techniques to Algorithm 2.
C.4 HYPER-PARAMETER OF BACKBONE OPTIMIZER
FedAvg (McMahan et al., 2017) is the most popular algorithm in federated learning. The server samples S number of clients subset in each round, each client of the subset trains the model locally by E number of SGD epochs, and the server aggregates the locally updated models and repeats these processes during the total rounds T. We use FedAvg algorithm, and its batch size B, and learning rate decay 7, are described in Table
backbone optimization of our experiments, such as the initial learning rate local
CIFAR-10
CIFAR-100
CINIC-10
LSTM
FEMNIST & MNIST
Models
IID
non-lll)
IID
non-lID
IID
non-lII
IID
non-lII)
K
16
16
8
8
16
16
16
16
10
T
200
200
400
400
300
300
500
500
100
E
10
10
5
10
5
B
64
64
64
64
64
64
64
64
10
0.1
0.1
0.1
0.1
0.1
0.1
1.0
1.0
0.1-0.01
T 0.992
0.992
0,992
0.992
0.992
0,992
0.992
0.992
0.999
1
Table 6: Hyper-parametenn of our FrdPara with FedAvg
C.5
OF OTHER OPTIMIZERS
For compatibility experiment, combine FedPara with other optimization-based FL algorithms: FedP rox (Li et al., 2020), SCAFFOLD (Karimireddy et al., 2020), FedDyn (Acar et al., 2021), and FedAdam (Reddi et al., 2021). FedProx (Li al., 2020) imposes a proximal term to the objective function to mitigate heterogeneity; SCAFFOLD (Karimireddy et al., 2020) allows clients reduce the variance of gradients by introducing auxiliary variables; FedDyn (Acar et al., 2021) introduces dynamic regularization to reduce the inconsistency between minima of the local device level empirical losses and the global one; FedAdam employs Adam (Kingma & Ba, 2015) at the server-side instead of the simple model average.
They need a local optimizer to update the model in each client, and we use the SGD optimizer for a fair comparison, and the SGD configuration is the same that of FedAvg. The four algo- rithms have additional
parameters. FedP rox has proximal coefficient μ., and we set μ as 0.1. SCAFFOLD has Options and II to update the control variate, and global learning rate ny (= 1.0). FedDyn has the hyper-paramett a (= 0.1) in the regularization. FedAdam Adam optimizer aggregate the updated models at the server-side, and the
Option II with
19

---

Published as a conference paper at ICLR 2022
Network speed
Model
fcomp.
feomm.
VGG1 bort.
1.64 sec.
470.2 sec.
471.84 sec.
Mhps
V3G1 6pedDaca (y=0.1)
2.34 sec.
47.2 sec.
49.54 sec. (x 9.52)
VGG1 6ori.
1.64 sec.
94.04 sec.
94,68 sec. 11.78 sec. (x 8.04)
Mbps
VGG1 6pedPara (y=0.1)
2.34 sec.
9.44 sec.
VGG1 oori.
1. 164 sec.
18.61 sec.
20.25 sec. 4.22 sec. (x 4.80)
50 Mbps
VGG1 Spodresa (y=0.1)
2.34 sec.
1.88 sec.
Table 7: The required time during one round. We denote the computation time, the communication time, and the total time during one round ceomp.r bcomm., and respectively, We the network speeds as 2, 10, and 50 Mbps.
Network speed
Model
Training time
VCG1 6ori.
880.77 min. 94.95 min. (x 9.28)
Mbps
VGG1 GzedPara (y=0.1)
VGG1 6ori.
176.74 min. 22.58 min. (x 7.83)
10 Mbps
VGG1 GredFara (y=0.1)
VGGl 6ori.
37.80 min. 8.09 min. (x 4.67)
Mbps
VGG1 6zadPasa (7=0.1)
Table 8: The real training time achieve the target accuracy. We set the network speeds 2, 10, and 50 Mbps, and the required rounds for VGGl 6ori. is 112 FedP ara (7 0.1) is 115 to achieve the same target accuracy in the CIFAR-10 10 IID setting.
parameters 81 0.9, 32 = 0.99, the global learning rate Mg 0.01, and the local learning rate n 10-1.5 for Adam optimizer.
D ADDITIONAL EXPERIMENTS
In this section, we simulate the computation time, the communication time, and total training time in Section D.1, experiment about other models in Section D.2, and compare our method and the quantization approach in Section D.3.
D. TRAINING TIME
We can compute the elapsed time during one round in FL by Leomp. + leomm.. where leomp. is the computation time of training the model on local data for several epochs and leomm. is the com- munication time of downloading and uploading the updated model. We estimate the computation time by measuring the elapsed time during local epochs. We compute the communication time by 2 2-dde size (Mbyte) network speed (Mbyte/s)+ considering upload and download of the model in one round.
Since challenging to experiment in real heterogeneous network environment, follow the simple standard network simulation setting widely used in the communication literature (Zhu et al., 2020; Jeon al., 2020). They assume the homogeneous link quality by the average quality to simplify complex network environments in FL communication simulation, i.e., the network speeds are identical for all clients.
We compare the elapsed time per round of VGG1 6 and VGG1 6rodPar: on different network speeds such as 2, 10, and 50 Mhps. As shown Table 7, the communication time is larger than the computation time indicating that the communication is a bottleneck in FL as expected. Although VGG1 6redPara takes more computation time than VGG1 6ori. due to the weight composition time, VGG1 GredPara decreases the communication time by about ten times and the total time by 4.80 to 9.52 times. Table 8 shows the total training time to achieve the same accuracy on the CIFAR- 10 IID setting. Although FedP ara a needs three rounds more than the original parameterization, VGG1 6redPara requires 4.67 to 9.68 times less training time than VCG1 6.i. because of communi- cation efficient parameterization.
20

---

Published as a conference paper at ICLR 2022
72.0
65.5
40.8
56.5
Acc. [86]
Acc. 1%4
47.0
30.5
41.0
VGG16
28.5
V0016
20.2
V0G16 (y-6
25.5
VG515 (y=8
www.
Ey Cyn8.9)
10.0
86
172
259
345
10.00
110
165
220
10.0
136
272
409
545
Communication Cost [GB]
Cammunication Cost (GB)
Communication Cost (GB)
(a) CIFAR-10 IID
(b) CIFAR-100 100 IID
(c) CINIC-10 IID
74.0
47.0
67.0
37.8
52.8
42.0
Acc.
28.5
38.5
10016 y-
V0116 woan
24.2
V6016 ay- 91
V0616 V6036 uy
10.00
86
172
259
345
10.00
90
180
270
360
10.0g
128
256
384
512
Communication Cost (GB)
Communication Cost (GB)
Communication Cost (GB)
(d) CIFAR-10 non-lID
(e) CIFAR-100 non-IID
(I) CINIC-10 non-IID
Figure 7: (a-f) Accuracy [%) (y-axis) vs. communication costs [GBytes] (æ-axis) of VGG1 6ori. and VGGl arad Broken black line represents VGGl 6ori. red solid line VGG1 6redPara with low y blue solid line VGGl GredFaza with mid y, and yellow solid line VGGl 6ped2 aza. with high 7
Model
Acc. of 200 rounds
Acc. of 1000 rounds (gain)
Original
83.68
84.1 (+ 0.42)
FedPara(y 0.1)
82.88
83.35 (+ 0.47)
FedPara(? 7 = 0.2)
82.53
83.34 (+ 0.81)
FedPara(7 0.3)
83.11
83.94 (+ 0.83)
FedPara(7 = 0.4)
84.05
84.59 (+ 0.54)
FedPara(7 0.5)
83.82
84.57 (+ 0.75)
FedPara(7 = 0.6)
83.63
84.16 (+ 0.53)
FedPara(7 7 0.7)
83.79
84.24 (+ 0.45)
FedPara(y = 0.8)
83.40
84.00 (+ 0.6)
Table 9: The accuracy VGG1 with original and our parameterization on the CIFAR-10 IID setting during 200 and 1000 rounds.
D.2 OTHER MODELS
VGG16 As shown in Figure 7, we compare VGG1 6 and VGGl GredPare with three dif- ferent y values. Note that a higher y uses more parameters than a lower 7. Compared to VGG1 6uri., VGG1 6redsern achieves comparable accuracy and even higher accuracy with high y. VGG1 GrecPaso also requires fewer communication costs than VGG16. 6ori.
We also investigate how much accuracy is increased for longer rounds. Table 9 shows that the accuracy is increased in the long round experiment, but the tendency of the 1000 round training is consistent with the 200 round training.
ResNet18 We demonstrate the consistent effectiveness of our method with another architecture, ResNet For ResNe:18, we train without replacing batch normalization layers, set 7 of the first layer, the second layer, and the x convolution layers as 1.0, and adjust 7 of remaining layers to control the ResNet Bradrara size; ? 0.1 for small model size, y 0.6 for mid model size, and 7 = 0.9 for large model size. To train the ResNet 1 Brearara in FL., we set the batch size to 10.
As revealed in Figure 8a, ResNet 1 Bredeara has comparable accuracy and uses fewer communica- tion costs than ResNet 1 Buri., of which the results are consistent with the VGG16 6 experiments. Figure 8h shows the communication costs required for model training achieve the same tar-
21

---

Published as a conference paper at ICLR 2022
Communicatin Cost (GB)
169.2
93.0
72.2
Acc. [%6]
87.3
51.5
Reshet M0r..
30.8
Brahe 8.
33.2
10.00
86
172
259
345
ResNet18gr1.
ResNe(1BreoPars R (y=0.1)
ResNet1Bragoor (y=0.9)
Communication Cost [GB]
(y=0.6)
(a) Accuracy
(h) Communication Costs
Figure 8: (a) Accuracy [%] (y-axis) vs. communication costs [GBytes] (a-axis) of ResNet 8 8ori. and ResNet BradPare with three y values. (b) Size comparison of transferred parameters, which expressed as communication costs [GBytes] (y-axis), for the same target accuracy 90%. (a, b): Black represents ResNet Boii. and red, blue, and yellow represents ResNet BredPaza with low, mid, and high 7, respectively.
Model
Acc.
# parameters (ratio)
VGG1 6rufferfish
82.07
0.33
VGG1 Grufferfish
82.42
0.44
VGG1 6rodPora (7 0.2)
82.53
0.15
VGG1 SrodFars (7 0.4)
84.05
0.29
Table 10: The accuracy of VGG1 6ourterrish and VGGl GradPara CIFAR-10 IID setting. # parameters the the ratio of each model parameter when the number parameters of VGG1 6ori is set 1.0.
get accuracy. Our ResNet1 1 needs 1.17 to 5.1 times fewer communication costs than ResNet18 1ori. and the results demonstrate that FedP ara a also applicable to the ResNet struc- ture.
Pufferfish Pufferfish (Wang al., 2021) similar to our method, where they use partially pre- factorized networks and employ the hybrid architecture by maintaining original size weights to minimize the accuracy loss due to the low-rank constraints. Since this work can be directly applied in the FL setup, we compare the parameterization of PufferFish and FedPara as follows.
We train VGG16 with PufferFish and FedPara on the CIFAR-10 IID dataset in FL, and evaluate the models according to varying the number of parameters. As shown in Table 10, our FedPara has higher accuracy with fewer parameters compared with PufferFish. Although PufferFish is superior to the naive low-rank pre-dccomposition method due to its hybrid architecture, think the hybrid architecture of PufferFish still suffers from the low-rank constraints on the top layers. However, our method is free from such limitations both theoretically and empirically.
LSTM
We train LS TS.r.... LS LSTM and LSTMyedrara on the Shakespeare dataset. The parameters ratio of LSTM and LSTMredFars about 16% and 19% of the LSTMon. parameters, respectively. Table 11 shows that LSTMredFara outperforms LS LSTM and LSTMori. on the IID setting. In the non- IID setting, LSTMredFara accuracy is higher than LSTM and slightly lower than L.T.ori. with only 19% of parameters. Therefore, our parameterization can be applied to general neural networks.
D.3 QUANTIZATION
FedPAo AQ (Reisizadeh et al., 2020) is a quantization approach to reduce communication costs for FL. To compare the accuracy and transferred size per round, train VGG1 6 the CIFAR-10 IID setting. consider both downlink and uplink to evaluate communication costs. We quantize the model from 32 bits floating-poin numbers 16 bits floating-point numbers.
22

---

122MB Published as a conference paper at ICLR 2022
Model
Acc. (IID)
Acc. (non-IID)
parameters (ratio)
LSTMori.
60.17
52.66
1.0
L.STMoww
54.59
51.24
0.16
LS TMPaaaaaa (7 = 0.0)
63.65
51.56
0.19
Table 11: The accuracy of LSTMs on IID and non-III setting. # parameters is the the ratio of each model parameter when the number of parameters of LSTMor. is set 1.0.
Model
Acc.
Transferred size per round
FedAvg
83.68
FedPAQ
82.67
91.4MB
FedPara (7 0.5)
83.82
46,4MB
+ FeedPAQ
83.58
34.74MB
Table 12: The accuracy of V on CIFAR-10 IID setting for original, quantization, and our parameterization models.
Table 12 shows the comparison between FedPara and FedPAt AQ as well as those integration. Com- pared to FedPAQ, FedPara transfers 1.96 times lower bits per round because FedPAQ only duces the uplink communication cost. Compared with FedAvg, FedPara achieves 0. 14% higher accuracy, but FedPAQ 1.01% lower accuracy. FedPara dated weights to the server, whereas FedPAQ loses the updated weight information due to the quantization. Furthermore, since the way FedPart reduces communication costs different from quantization, we integrate FedPaze ara with FedPAQ as an extension. Combining our method with FedPAQ reduces communication costs by 25% further from our FedPara without the integration while having minor accuracy drop, 0. %, from that of FedAvg.
transfer the full information of up-
23


Reviews:
[{"review_id": "fARo8P-xIYE", "reviewer": "Reviewer_5RoZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "======================= Post Rebuttal ======================\n\nAfter reading the authors' response. Part of my concerns are addressed. Thus\nI would like to increase my overall evaluation to 6. And I believe that this work\ncan make a decent contribution to the efficient FL training area.\n\n========================================================\n\n\nThe paper proposes to train low rank factorized model on the client side to improve the communication efficiency of federated learning while introducing a novel low rank factorization method. \n\nPros:\n1. The paper is well written, and the research direction of enhancing the communication efficiency of federated learning is potentially impactful. \n2. The proposed low rank factorization method is novel. \n3. Both theoretical analysis and empirical results are provided to justify the effectiveness of FedPara. \n\nCons: \n1. The main concern I have for the current version of the draft is that the proposed FedPara method is only compared against the traditional low rank based model factorization method, e.g., vanilla low rank model or Pufferfish. It is not clear if FedPara is also better compared against other communication efficient federated learning methods, e.g., [1, 2] or recently proposed low rank based communication efficient FL method [3]. \n2. By looking at Algorithms 1 and 2, it is not clear which weights are kept on the client side and which weights are kept on the server side. It says “Optimize(W)”, does it mean only $X_1, Y_1$ and $X_2, Y_2$ are optimized? Also for aggregate $X_1, Y_1$, and $X_2, Y_2$, does it require to transform them back to W before aggregation across clients’ updates? The authors are supposed to clarify this. \n3. How to choose $\\gamma$ is not clear, a fixed rank ratio seems to make sense. But it may also be possible that different layers can contain different redundancy. Is there a chance to adaptively choose $\\gamma$ for different layers? \n4. It is recently observed that the model weights of the Transformer are not low rank. Does it mean FedPara can not be applied over the Transformer architectures? \n\n[1] https://arxiv.org/pdf/1610.05492.pdf\n\n[2] http://proceedings.mlr.press/v108/reisizadeh20a/reisizadeh20a.pdf\n\n[3] https://arxiv.org/pdf/2104.12416.pdf\n\n[4] https://openreview.net/forum?id=_sSHg203jSu", "summary_of_the_paper": "This paper introduces FedPara, a low-rank based method for achieving communication efficient federated learning. On the client-side, low rank factorized models are trained. Different from the traditional approaches to finding low rank factorized networks, FedPara introduces a novel low rank factorization strategy, which attains a higher maximal rank for the weight matrices. The paper also identifies that FedPara can be used to enhance personalized federated learning (by introducing a variant of FedPara, i.e., pFedPara). Both theoretical analysis and experimental results are provided to show the effectiveness of FedPara. ", "summary_of_the_review": "Overall I think the paper introduces a valid research idea on improving the communication efficiency of federated learning. However, several concerns exist (please refer to “Cons” in “Main Review”). If my major concerns are addressed, I will consider increasing my overall evaluation score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cxs9Om3G9Qm", "reviewer": "Reviewer_M7KC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not very familiar with reparametrization, and I am not able to comment on the significance of this paper.\n\nI think that the idea of this paper is interesting, using the Hadamard product of two low-rank matrices may only increase the model size by a factor of 2, but may drastically increase the expressiveness of the parametrization.\n\nAs for the experiments, I think the experiment design is generally enough. The experiment compares FedPara with original training methods with no reparametrization, and low-rank reparametrization without the Hadamard product on different machine learning models, e.g. VGG and LSTM.\n\nBelow are some suggestions and questions:\n1. In the experiment, the authors compare FedPara with original low-rank reparametrization and without reparametrization, and the optimization algorithms are chosen to be the same, e.g. FedAvg, SCAFFOLD, etc. I wonder what are the results between FedPara and gradient compression algorithms, for example, quantization. Or if the result can become better if using gradient compression schemes in FedPara instead of local steps. It may also be good to add some experiment results with compression.\n\n2. FedPara converges much faster than the original models in terms of the communication bits. How about the final accuracy? What is the tradeoff between communication efficiency and the testing accuracy?\n\n3. Although the low-rank constraints reduce the number of communication bits during the training procedures, it may increase the local computational cost since the gradient of the new model may become more complicated. In your experiment, what is the local training cost, e.g. the local training time between two communications, or the number of flops for one backpropagation?", "summary_of_the_paper": "The paper introduces FedPara, which is a low-rank parametrization method for the neural networks aiming to reduce the total number of communication bits while preserving the model accuracy in the federated learning scenario. Besides FedPara, which is designed for federated learning applications, the paper also generalizes FedPara into pFedPara, which is designed for personalized federated learning.\n\nThe novelty of this paper includes using the Hadamard product in the low-rank approximation. Compared with directly using low-rank approximation, using Hadamard produce can increase the expressiveness of the parametrization.\n\nThe paper also includes empirical results to show the effectiveness of FedPara/pFedPara in the federated learning setting.", "summary_of_the_review": "I am not very familiar with reparametrization and its corresponding related works, and I am not able to comment on the significance. However, I think that the technique used in this paper is interesting and simple, which can be applied to real applications. The experiments shown in the paper are enough. Some questions and small suggestions are listed in the main review part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EMIaAJHju6", "reviewer": "Reviewer_yBc3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "======================= After Rebuttal ======================\n\nI have read the authors' responses. Part of my concerns is alleviated. I would like to increase my overall evaluation to 6. \n\n========================================================\n\nStrengths:\nThis paper is well organized and easy to follow. The authors provided comprehensive experiments.\n\nQuestions:\n1. As for Figure 1, does the low-rank parameterization $W=XY^{\\rm T}$ always leads to the Hadamard product of two low-rank inner matrices $W=W_1\\circ W_2=(X_1Y_1^{\\rm T})\\circ(X_2Y_2^{\\rm T})$? What if the low-rank weight matrix $W$ cannot be decomposed into the Hadamard product of two low-rank inner matrices?\n2. Proposition 1 only provides the upper bound for the rank(W). It does not necessarily mean that $rank(W)=r_1r_2$. How could the constructed matrix always span a full-rank matrix when $r_1r_2\\ge \\min(m,n)$?\n3. In the numerical results with varying $\\gamma\\in[0.1,0.9]$, what is the corresponding number of parameters?\n4. The accuracy differences between the low-rank method and FedPara in the non-IID case are smaller than the differences in the IID case. Is that any explanation for this phenomenon?\n5. It would be better to move the details of proof to the appendix and add more discussions in the main manuscript.", "summary_of_the_paper": "This paper proposes a communication-efficient parameterization for federated learning by using low-rank weights. Several numerical experiments are conducted to demonstrate the effectiveness of the proposed FedPara and pFedPara.", "summary_of_the_review": "The paper is mostly sound, while the contributions are marginally significant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AnbVdNFcRIs", "reviewer": "Reviewer_3Rte", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a low-rank parameterization method to help improve the communication efficiency of federated learning. The work provides theoretical justification of the effectiveness in terms of the maximal rank the parameterization can express. The work shows extensive empirical studies of the proposed method on multiple FL datasets. The results indicate the superiority of the proposed method in terms of model accuracy and communication cost.\n\nQuestions and comments:\n1. In terms of expressiveness, what is the key limitation of the Hadamard product parameterization compared with general unconstrained parameterization?\n2. The paper does not discuss any related work about the Hadamard product parameterization. Is this a new idea? Or what is the difference compared to other literature involving Hadamard product parameterization?\n3. In terms of computational cost, how does training the Hadamard product parameterization compare to previous low-rank approaches?", "summary_of_the_paper": "The paper proposes a communication-efficient parameterization methodology for federated learning tasks through the Hadamard product of low-rank weights. Such parameterization has better model expressiveness in terms of minimal parameters to achieve a maximal rank, compared with conventional low-rank approaches. Using such parameterization, the work designs the corresponding communication-efficient federated learning framework which only uploads a low-rank part of parameters of each level to the server. The work presents empirical studies of the proposed algorithm and validates its effectiveness in terms of accuracy vs communication costs.", "summary_of_the_review": "The work proposes an effective low-rank parameterization that shows the superiority of communication efficiency in federated learning tasks. The idea is relatively interesting to the reviewer. The overall presentation of the idea and empirical results is clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear Authors,

The response you have provided, based on the main concerns of reviewers, have answered most of the questions raised. 
As far as I understand from the added experiments you have provided, the proposed methodology shows resilience in being at least as good as state of the art approaches, while at the same time it is a mathematically interesting approach. 

Your response has covered concerns like comparison to other communication techniques (the comparison list is not complete, but yet your effort is appreciated), adding discussion on the rank parameter, add comments on expressiveness and the connection with low-rank parameterization, etc. 

These efforts cannot be overlooked, and for that reason I suggest acceptance (poster).

Best

AC
Decision: Accept (Poster)