FedPara: Low-rank Hadamard Product for Communication-Efficient Federated Learning

1 INTRODUCTION . Federated learning ( FL ; McMahan et al. , 2017 ) has been proposed as an efficient collaborative learning strategy along with the advance and spread of mobile and IoT devices . FL allows leveraging local computing resources of edge devices and locally stored private data without data sharing for privacy . FL typically consists of the following steps : ( 1 ) clients download a globally shared model from a central server , ( 2 ) the clients locally update each model using their own private data without accessing the others ’ data , ( 3 ) the clients upload their local models back to the server , and ( 4 ) the server consolidates the updated models and repeats these steps until the global model converges . FL has the key properties ( McMahan et al. , 2017 ) that differentiate it from distributed learning : • Heterogeneous data . Data is decentralized and non-IID as well as unbalanced in its amount due to different characteristics of clients ; thus , local data does not represent the population distribution . • Heterogeneous systems . Clients consist of heterogeneous setups of hardware and infrastructure ; hence , those connections are not guaranteed to be online , fast , or cheap . Besides , massive client participation is expected through different communication paths , causing communication burdens . These FL properties introduce challenges in the convergence stability with heterogeneous data and communication overheads . To improve the stability and reduce the communication rounds , the prior works in FL have proposed modified loss functions or model aggregation methods ( Li et al. , 2020 ; Karimireddy et al. , 2020 ; Acar et al. , 2021 ; Yu et al. , 2020 ; Reddi et al. , 2021 ) . However , the transferred data is still a lot for the edge devices with bandwidth constraints or countries having low-quality communication infrastructure.1 A large amount of transferred data introduces an energy consumption issue on edge devices because wireless communication is significantly more powerintensive than computation ( Yadav & Yadav , 2016 ; Yan et al. , 2019 ) . In this work , we propose a communication-efficient re-parameterization for FL , FedPara , which reduces the number of bits transferred per round . FedPara directly re-parameterizes each fully- 1The gap between the fastest and the lowest communication speed across countries is significant ; approximately 63 times different ( Speedtest ) . connected ( FC ) and convolutional layers of the model to have a small and factorized form while preserving the model ’ s capacity . Our key idea is to combine the Hadamard product with low-rank parameterization as W = ( X1Y > 1 ) ( X2Y > 2 ) ∈ Rm×n , called low-rank Hadamard product . When rank ( X1Y > 1 ) = rank ( X2Y > 2 ) = r , then rank ( W ) ≤ r2 . This outstanding property facilitates spanning a full-rank matrix with much fewer parameters than the typical m× n matrix . It significantly reduces the communication burdens during training . At the inference phase , we pre-compose and maintain W that boils down to its original structure ; thus , FedPara does not alter computational complexity at inference time . Compared to the aforementioned prior works that tackle reducing the required communication rounds for convergence , our FedPara is an orthogonal approach in that FedPara does not change the optimization part but re-defines each layer ’ s internal structure . We demonstrate the effectiveness of FedPara with various network architectures , including VGG , ResNet , and LSTM , on standard classification benchmark datasets for both IID and non-IID settings . The accuracy of our parameterization outperforms that of the traditional low-rank parameterization baseline given the same number of parameters . Besides , FedPara has comparable accuracy to original counterpart models and even outperforms as the number of parameters increases at times . We also combine FedPara with other FL algorithms to improve communication efficiency further . We extend FedPara to the personalized FL application , named pFedPara , which separates the roles of each sub-matrix into global and local inner matrices . The global and local inner matrices learn the globally shared common knowledge and client-specific knowledge , respectively . We devise three scenarios according to the amount and heterogeneity of local data using the subset of FEMNIST and MNIST . We demonstrate performance improvement and robustness of pFedPara against competing algorithms . We summarize our main contributions as follows : • We propose FedPara , a low-rank Hadamard product parameterization for communicationefficient FL . Unlike traditional low-rank parameterization , we show that FedPara can span a full-rank matrix and tensor with reduced parameters . We also show that FedPara requires up to ten times fewer total communication costs than the original model to achieve target accuracy . FedPara even outperforms the original model by adjusting ranks at times . • Our FedPara takes a novel approach ; thereby , our FedPara can be combined with other FL methods to get mutual benefits , which further increase accuracy and communication efficiency . • We propose pFedPara , a personalization application of FedPara , which splits the layer weights into global and local parameters . pFedPara shows more robust results in challenging regimes than competing methods . 2 METHOD . In this section , we first provide the overview of three popular low-rank parameterizations in Section 2.1 and present our parameterization , FedPara , with its algorithmic properties in Section 2.2 . Then , we extend FedPara to the personalized FL application , pFedPara , in Section 2.3 . Notations . We denote the Hadamard product as , the Kronecker product as⊗ , n-mode tensor product as ×n , and the i-th unfolding of the tensor T ( i ) ∈ Rki× ∏ j 6=i kj given a tensor T ∈ Rk1×···×kn . 2.1 OVERVIEW OF LOW-RANK PARAMETERIZATION . The low-rank decomposition in neural networks has been typically applied to pre-trained models for compression ( Phan et al. , 2020 ) , whereby the number of parameters is reduced while minimizing the loss of encoded information . Given a learned parameter matrix W ∈ Rm×n , it is formulated as finding the best rank-r approximation , as argmin W̃ ||W − W̃||F such that W̃ = XY > , where X ∈ Rm×r , Y ∈ Rn×r , and r min ( m , n ) . It reduces the number of parameters from O ( mn ) to O ( ( m+ n ) r ) , and its closed-form optimal solution can be found by SVD . This matrix decomposition is applicable to the FC layers and the reshaped kernels of the convolution layers . However , the natural shape of a convolution kernel is a fourth-order tensor ; thus , the lowrank tensor decomposition , such as Tucker and CP decomposition ( Lebedev et al. , 2015 ; Phan et al. , 2020 ) , can be a more suitable approach . Given a learned high-order tensor T ∈ Rk1×···×kn , Tucker decomposition multiplies a kernel tensor K ∈ Rr1×···×rn with matrices Xi ∈ Rri×n1 , where ri = rank ( T̃ ( i ) ) as T̃ = K ×1 X1 ×2 · · · ×n Xn , and CP decomposition is the summation of rank-1 tensors as T̃ = ∑i=r i=1 x 1 i × x2i × · · · × xni , where x j i ∈ Rkj . Likewise , it also reduces the number of model parameters . We refer to these rank-constrained structure methods simply as conventional low-rank constraints or low-rank parameterization methods . In the FL context , where the parameters are frequently transferred between clients and the server during training , the reduced parameters lead to communication cost reduction , which is the main focus of this work . The post-decomposition approaches ( Lebedev et al. , 2015 ; Phan et al. , 2020 ) using SVD , Tucker , and CP decompositions do not reduce the communication costs because those are applied to the original parameterization after finishing training . That is , the original large-size parameters are transferred during training in FL , and the number of parameters is reduced after finishing training . We take a different notion from the low-rank parameterizations . In the FL scenario , we train a model from scratch with low-rank constraints , but specifically with low-rank Hadamard product reparameterization . We re-parameterize each learnable layer , including FC and convolutional layers , and train the surrogate model by FL . Different from the existing low-rank method in FL ( Konečnỳ et al. , 2016 ) , our parameterization can achieve comparable accuracy to the original counterpart . 2.2 FEDPARA : A COMMUNICATION-EFFICIENT PARAMETERIZATION . As mentioned , the conventional low-rank parameterization has limited expressiveness due to its lowrank constraint . To overcome this while maintaining fewer parameters , we present our new low-rank Hadamard product parameterization , called FedPara , which has the favorable property as follows : Proposition 1 Let X1 ∈ Rm×r1 , X2 ∈ Rm×r2 , Y1 ∈ Rn×r1 , Y2 ∈ Rn×r2 , r1 , r2 ≤ min ( m , n ) and the constructed matrix be W : = ( X1Y > 1 ) ( X2Y > 2 ) . Then , rank ( W ) ≤ r1r2 . All proofs can be found in the supplementary material including Proposition 1 . Proposition 1 implies that , unlike the low-rank parameterization , a higher-rank matrix can be constructed using the Hadamard product of two inner low-rank matrices , W1 andW2 ( Refer to Figure 1 ) . If we choose the inner ranks r1 and r2 such that r1r2 ≥ min ( m , n ) , the constructed matrix does not have a low-rank restriction and is able to span a full-rank matrix with a high chance ( See Figure 6 ) ; i.e. , FedPara has the minimal parameter property achievable to full-rank . In addition , we can control the number of parameters by changing the inner ranks r1 and r2 , respectively , but we have the following useful property to set the hyper-parameters to be a minimal number of parameters with a maximal rank . Proposition 2 Given R ∈ N , r1 = r2 = R is the unique optimal choice of the following criteria , argmin r1 , r2∈N ( r1 + r2 ) ( m+ n ) s.t . r1r2 ≥ R 2 , ( 1 ) and its optimal value is 2R ( m+ n ) . Equation 3 implies the criteria that minimize the number of weight parameters used in our parameterization with the target rank constraint of the constructed matrix as R2 . Proposition 2 provides an efficient way to set the hyper-parameters . It implies that , if we set r1=r2=R and R2 ≥ min ( m , n ) , FedPara is highly likely to have no low-rank restriction2 even with much fewer parameters than that of a naı̈ve weight , i.e. , 2R ( m + n ) mn . Moreover , given the same number of parameters , rank ( W ) of FedPara is higher than that of the naı̈ve low-rank parameterization by a square factor , as shown in Figure 1 and Table 1 . To extend Proposition 1 to the convolutional layers , we can simply reshape the fourth-order tensor kernel to the matrix as RO×I×K1×K2 → RO× ( IK1K2 ) as a naı̈ve way , where O , I , K1 , and K2 are the output channels , the input channels , and the kernel sizes , respectively . That is , our parameterization spans convolution filters with a few basis filters of size I ×K1 ×K2 . However , we can derive more efficient parameterization of convolutional layers without reshaping as follows : Proposition 3 Let T1 , T2 ∈ RR×R×k3×k4 , X1 , X2 ∈ Rk1×R , Y1 , Y2 ∈ Rk2×R , R ≤ min ( k1 , k2 ) and the convolution kernel beW : = ( T1×1 X1×2 Y1 ) ( T2×1 X2×2 Y2 ) . Then , the rank of the kernel satisfies rank ( W ( 1 ) ) = rank ( W ( 2 ) ) ≤ R2 . Proposition 3 is the extension of Proposition 1 but can be applied to the convolutional layer without reshaping . In the convolutional layer case of Table 1 , given the specific tensor size , Proposition 3 requires 3.8 times fewer parameters than Proposition 1 . Hence , we use Proposition 3 for the convolutional layer since the tensor method is more effective for common convolutional models . Optionally , we employ non-linearity and the Jacobian correction regularization , of which details can be found in the supplementary material . These techniques improve the accuracy and convergence stability but not essential . Depending on the resources of devices , these techniques can be omitted . 2.3 PFEDPARA : PERSONALIZED FL APPLICATION In practice , data are heterogeneous and personal due to different characteristics of each client , such as usage times and habits . FedPer ( Arivazhagan et al. , 2019 ) has been proposed to tackle this scenario by distinguishing global and local layers in the model . Clients only transfer global layers ( the top layer ) and keep local ones ( the bottom layers ) on each device . The global layers learn jointly to extract general features , while the local layers are biased towards each user . With our FedPara , we propose a personalization application , pFedPara , in which the Hadamard product is used as a bridge between the global inner weight W1 and the local inner weight W2 . Each layer of the personalized model is constructed by W = W1 ( W2 + 1 ) , where W1 is transferred to the server while W2 is kept in a local device during training . This induces W1 to learn globally shared knowledge implicitly and acts as a switch of the term ( W2 + 1 ) . Conceptually , we 2Its corollary and empirical evidence can be found in the supplementary material . Under Proposition 2 , R2 ≥ min ( m , n ) is a necessary and sufficient condition for achieving a maximal rank . can interpret by rewriting W = W1 W2+W1 = Wper.+Wglo. , where Wper . = W1 W2 and Wglo . = W1 . The construction of the final personalized parameter W in pFedPara can be viewed as an additive model of the global weight Wglo . and the personalizing residue Wper .. pFedPara transfers only a half of the parameters compared to FedPara under the same rank condition ; hence , the communication efficiency is increased further . Intuitively , FedPer and pFedPara are distinguished by their respective split directions , as illustrated in Figure 2 . We summarize our algorithms in the supplementary material . Although we only illustrate feed-forward network cases for convenience , it can be extended to general cases .

Summary: The paper introduces FedPara, which is a low-rank parametrization method for the neural networks aiming to reduce the total number of communication bits while preserving the model accuracy in the federated learning scenario. Besides FedPara, which is designed for federated learning applications, the paper also generalizes FedPara into pFedPara, which is designed for personalized federated learning.  The novelty of this paper includes using the Hadamard product in the low-rank approximation. Compared with directly using low-rank approximation, using Hadamard produce can increase the expressiveness of the parametrization.  The paper also includes empirical results to show the effectiveness of FedPara/pFedPara in the federated learning setting.