FedPara: Low-rank Hadamard Product for Communication-Efficient Federated Learning

1 INTRODUCTION . Federated learning ( FL ; McMahan et al. , 2017 ) has been proposed as an efficient collaborative learning strategy along with the advance and spread of mobile and IoT devices . FL allows leveraging local computing resources of edge devices and locally stored private data without data sharing for privacy . FL typically consists of the following steps : ( 1 ) clients download a globally shared model from a central server , ( 2 ) the clients locally update each model using their own private data without accessing the others ’ data , ( 3 ) the clients upload their local models back to the server , and ( 4 ) the server consolidates the updated models and repeats these steps until the global model converges . FL has the key properties ( McMahan et al. , 2017 ) that differentiate it from distributed learning : • Heterogeneous data . Data is decentralized and non-IID as well as unbalanced in its amount due to different characteristics of clients ; thus , local data does not represent the population distribution . • Heterogeneous systems . Clients consist of heterogeneous setups of hardware and infrastructure ; hence , those connections are not guaranteed to be online , fast , or cheap . Besides , massive client participation is expected through different communication paths , causing communication burdens . These FL properties introduce challenges in the convergence stability with heterogeneous data and communication overheads . To improve the stability and reduce the communication rounds , the prior works in FL have proposed modified loss functions or model aggregation methods ( Li et al. , 2020 ; Karimireddy et al. , 2020 ; Acar et al. , 2021 ; Yu et al. , 2020 ; Reddi et al. , 2021 ) . However , the transferred data is still a lot for the edge devices with bandwidth constraints or countries having low-quality communication infrastructure.1 A large amount of transferred data introduces an energy consumption issue on edge devices because wireless communication is significantly more powerintensive than computation ( Yadav & Yadav , 2016 ; Yan et al. , 2019 ) . In this work , we propose a communication-efficient re-parameterization for FL , FedPara , which reduces the number of bits transferred per round . FedPara directly re-parameterizes each fully- 1The gap between the fastest and the lowest communication speed across countries is significant ; approximately 63 times different ( Speedtest ) . connected ( FC ) and convolutional layers of the model to have a small and factorized form while preserving the model ’ s capacity . Our key idea is to combine the Hadamard product with low-rank parameterization as W = ( X1Y > 1 ) ( X2Y > 2 ) ∈ Rm×n , called low-rank Hadamard product . When rank ( X1Y > 1 ) = rank ( X2Y > 2 ) = r , then rank ( W ) ≤ r2 . This outstanding property facilitates spanning a full-rank matrix with much fewer parameters than the typical m× n matrix . It significantly reduces the communication burdens during training . At the inference phase , we pre-compose and maintain W that boils down to its original structure ; thus , FedPara does not alter computational complexity at inference time . Compared to the aforementioned prior works that tackle reducing the required communication rounds for convergence , our FedPara is an orthogonal approach in that FedPara does not change the optimization part but re-defines each layer ’ s internal structure . We demonstrate the effectiveness of FedPara with various network architectures , including VGG , ResNet , and LSTM , on standard classification benchmark datasets for both IID and non-IID settings . The accuracy of our parameterization outperforms that of the traditional low-rank parameterization baseline given the same number of parameters . Besides , FedPara has comparable accuracy to original counterpart models and even outperforms as the number of parameters increases at times . We also combine FedPara with other FL algorithms to improve communication efficiency further . We extend FedPara to the personalized FL application , named pFedPara , which separates the roles of each sub-matrix into global and local inner matrices . The global and local inner matrices learn the globally shared common knowledge and client-specific knowledge , respectively . We devise three scenarios according to the amount and heterogeneity of local data using the subset of FEMNIST and MNIST . We demonstrate performance improvement and robustness of pFedPara against competing algorithms . We summarize our main contributions as follows : • We propose FedPara , a low-rank Hadamard product parameterization for communicationefficient FL . Unlike traditional low-rank parameterization , we show that FedPara can span a full-rank matrix and tensor with reduced parameters . We also show that FedPara requires up to ten times fewer total communication costs than the original model to achieve target accuracy . FedPara even outperforms the original model by adjusting ranks at times . • Our FedPara takes a novel approach ; thereby , our FedPara can be combined with other FL methods to get mutual benefits , which further increase accuracy and communication efficiency . • We propose pFedPara , a personalization application of FedPara , which splits the layer weights into global and local parameters . pFedPara shows more robust results in challenging regimes than competing methods . 2 METHOD . In this section , we first provide the overview of three popular low-rank parameterizations in Section 2.1 and present our parameterization , FedPara , with its algorithmic properties in Section 2.2 . Then , we extend FedPara to the personalized FL application , pFedPara , in Section 2.3 . Notations . We denote the Hadamard product as , the Kronecker product as⊗ , n-mode tensor product as ×n , and the i-th unfolding of the tensor T ( i ) ∈ Rki× ∏ j 6=i kj given a tensor T ∈ Rk1×···×kn . 2.1 OVERVIEW OF LOW-RANK PARAMETERIZATION . The low-rank decomposition in neural networks has been typically applied to pre-trained models for compression ( Phan et al. , 2020 ) , whereby the number of parameters is reduced while minimizing the loss of encoded information . Given a learned parameter matrix W ∈ Rm×n , it is formulated as finding the best rank-r approximation , as argmin W̃ ||W − W̃||F such that W̃ = XY > , where X ∈ Rm×r , Y ∈ Rn×r , and r min ( m , n ) . It reduces the number of parameters from O ( mn ) to O ( ( m+ n ) r ) , and its closed-form optimal solution can be found by SVD . This matrix decomposition is applicable to the FC layers and the reshaped kernels of the convolution layers . However , the natural shape of a convolution kernel is a fourth-order tensor ; thus , the lowrank tensor decomposition , such as Tucker and CP decomposition ( Lebedev et al. , 2015 ; Phan et al. , 2020 ) , can be a more suitable approach . Given a learned high-order tensor T ∈ Rk1×···×kn , Tucker decomposition multiplies a kernel tensor K ∈ Rr1×···×rn with matrices Xi ∈ Rri×n1 , where ri = rank ( T̃ ( i ) ) as T̃ = K ×1 X1 ×2 · · · ×n Xn , and CP decomposition is the summation of rank-1 tensors as T̃ = ∑i=r i=1 x 1 i × x2i × · · · × xni , where x j i ∈ Rkj . Likewise , it also reduces the number of model parameters . We refer to these rank-constrained structure methods simply as conventional low-rank constraints or low-rank parameterization methods . In the FL context , where the parameters are frequently transferred between clients and the server during training , the reduced parameters lead to communication cost reduction , which is the main focus of this work . The post-decomposition approaches ( Lebedev et al. , 2015 ; Phan et al. , 2020 ) using SVD , Tucker , and CP decompositions do not reduce the communication costs because those are applied to the original parameterization after finishing training . That is , the original large-size parameters are transferred during training in FL , and the number of parameters is reduced after finishing training . We take a different notion from the low-rank parameterizations . In the FL scenario , we train a model from scratch with low-rank constraints , but specifically with low-rank Hadamard product reparameterization . We re-parameterize each learnable layer , including FC and convolutional layers , and train the surrogate model by FL . Different from the existing low-rank method in FL ( Konečnỳ et al. , 2016 ) , our parameterization can achieve comparable accuracy to the original counterpart . 2.2 FEDPARA : A COMMUNICATION-EFFICIENT PARAMETERIZATION . As mentioned , the conventional low-rank parameterization has limited expressiveness due to its lowrank constraint . To overcome this while maintaining fewer parameters , we present our new low-rank Hadamard product parameterization , called FedPara , which has the favorable property as follows : Proposition 1 Let X1 ∈ Rm×r1 , X2 ∈ Rm×r2 , Y1 ∈ Rn×r1 , Y2 ∈ Rn×r2 , r1 , r2 ≤ min ( m , n ) and the constructed matrix be W : = ( X1Y > 1 ) ( X2Y > 2 ) . Then , rank ( W ) ≤ r1r2 . All proofs can be found in the supplementary material including Proposition 1 . Proposition 1 implies that , unlike the low-rank parameterization , a higher-rank matrix can be constructed using the Hadamard product of two inner low-rank matrices , W1 andW2 ( Refer to Figure 1 ) . If we choose the inner ranks r1 and r2 such that r1r2 ≥ min ( m , n ) , the constructed matrix does not have a low-rank restriction and is able to span a full-rank matrix with a high chance ( See Figure 6 ) ; i.e. , FedPara has the minimal parameter property achievable to full-rank . In addition , we can control the number of parameters by changing the inner ranks r1 and r2 , respectively , but we have the following useful property to set the hyper-parameters to be a minimal number of parameters with a maximal rank . Proposition 2 Given R ∈ N , r1 = r2 = R is the unique optimal choice of the following criteria , argmin r1 , r2∈N ( r1 + r2 ) ( m+ n ) s.t . r1r2 ≥ R 2 , ( 1 ) and its optimal value is 2R ( m+ n ) . Equation 3 implies the criteria that minimize the number of weight parameters used in our parameterization with the target rank constraint of the constructed matrix as R2 . Proposition 2 provides an efficient way to set the hyper-parameters . It implies that , if we set r1=r2=R and R2 ≥ min ( m , n ) , FedPara is highly likely to have no low-rank restriction2 even with much fewer parameters than that of a naı̈ve weight , i.e. , 2R ( m + n ) mn . Moreover , given the same number of parameters , rank ( W ) of FedPara is higher than that of the naı̈ve low-rank parameterization by a square factor , as shown in Figure 1 and Table 1 . To extend Proposition 1 to the convolutional layers , we can simply reshape the fourth-order tensor kernel to the matrix as RO×I×K1×K2 → RO× ( IK1K2 ) as a naı̈ve way , where O , I , K1 , and K2 are the output channels , the input channels , and the kernel sizes , respectively . That is , our parameterization spans convolution filters with a few basis filters of size I ×K1 ×K2 . However , we can derive more efficient parameterization of convolutional layers without reshaping as follows : Proposition 3 Let T1 , T2 ∈ RR×R×k3×k4 , X1 , X2 ∈ Rk1×R , Y1 , Y2 ∈ Rk2×R , R ≤ min ( k1 , k2 ) and the convolution kernel beW : = ( T1×1 X1×2 Y1 ) ( T2×1 X2×2 Y2 ) . Then , the rank of the kernel satisfies rank ( W ( 1 ) ) = rank ( W ( 2 ) ) ≤ R2 . Proposition 3 is the extension of Proposition 1 but can be applied to the convolutional layer without reshaping . In the convolutional layer case of Table 1 , given the specific tensor size , Proposition 3 requires 3.8 times fewer parameters than Proposition 1 . Hence , we use Proposition 3 for the convolutional layer since the tensor method is more effective for common convolutional models . Optionally , we employ non-linearity and the Jacobian correction regularization , of which details can be found in the supplementary material . These techniques improve the accuracy and convergence stability but not essential . Depending on the resources of devices , these techniques can be omitted . 2.3 PFEDPARA : PERSONALIZED FL APPLICATION In practice , data are heterogeneous and personal due to different characteristics of each client , such as usage times and habits . FedPer ( Arivazhagan et al. , 2019 ) has been proposed to tackle this scenario by distinguishing global and local layers in the model . Clients only transfer global layers ( the top layer ) and keep local ones ( the bottom layers ) on each device . The global layers learn jointly to extract general features , while the local layers are biased towards each user . With our FedPara , we propose a personalization application , pFedPara , in which the Hadamard product is used as a bridge between the global inner weight W1 and the local inner weight W2 . Each layer of the personalized model is constructed by W = W1 ( W2 + 1 ) , where W1 is transferred to the server while W2 is kept in a local device during training . This induces W1 to learn globally shared knowledge implicitly and acts as a switch of the term ( W2 + 1 ) . Conceptually , we 2Its corollary and empirical evidence can be found in the supplementary material . Under Proposition 2 , R2 ≥ min ( m , n ) is a necessary and sufficient condition for achieving a maximal rank . can interpret by rewriting W = W1 W2+W1 = Wper.+Wglo. , where Wper . = W1 W2 and Wglo . = W1 . The construction of the final personalized parameter W in pFedPara can be viewed as an additive model of the global weight Wglo . and the personalizing residue Wper .. pFedPara transfers only a half of the parameters compared to FedPara under the same rank condition ; hence , the communication efficiency is increased further . Intuitively , FedPer and pFedPara are distinguished by their respective split directions , as illustrated in Figure 2 . We summarize our algorithms in the supplementary material . Although we only illustrate feed-forward network cases for convenience , it can be extended to general cases .

Summary: This paper introduces FedPara, a low-rank based method for achieving communication efficient federated learning. On the client-side, low rank factorized models are trained. Different from the traditional approaches to finding low rank factorized networks, FedPara introduces a novel low rank factorization strategy, which attains a higher maximal rank for the weight matrices. The paper also identifies that FedPara can be used to enhance personalized federated learning (by introducing a variant of FedPara, i.e., pFedPara). Both theoretical analysis and experimental results are provided to show the effectiveness of FedPara.