FEDPARA: LOW-RANK HADAMARD PRODUCT FOR COMMUNICATION-EFFICIENT FEDERATED LEARNING
Year: 2022

Abstract:
In this work, we propose a communication-efficient parameterization, FedPara, for federated learning (FL) to overcome the burdens on frequent model uploads and downloads. Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conventional low-rank parameterization, our FedPara method is not restricted to lowrank constraints, and thereby it has a far larger capacity. This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods. The efficiency of our method can be further improved by combining with other efficient FL optimizers. In addition, we extend our method to a personalized FL application, pFedPara, which separates parameters into global and local ones. We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.

heading: 1 INTRODUCTION
text: Federated learning (FL; McMahan et al., 2017) has been proposed as an efficient collaborative learning strategy along with the advance and spread of mobile and IoT devices. FL allows leveraging local computing resources of edge devices and locally stored private data without data sharing for privacy. FL typically consists of the following steps: (1) clients download a globally shared model from a central server, (2) the clients locally update each model using their own private data without accessing the others’ data, (3) the clients upload their local models back to the server, and (4) the server consolidates the updated models and repeats these steps until the global model converges. FL has the key properties (McMahan et al., 2017) that differentiate it from distributed learning:
• Heterogeneous data. Data is decentralized and non-IID as well as unbalanced in its amount due to different characteristics of clients; thus, local data does not represent the population distribution.
• Heterogeneous systems. Clients consist of heterogeneous setups of hardware and infrastructure; hence, those connections are not guaranteed to be online, fast, or cheap. Besides, massive client participation is expected through different communication paths, causing communication burdens.
These FL properties introduce challenges in the convergence stability with heterogeneous data and communication overheads. To improve the stability and reduce the communication rounds, the prior works in FL have proposed modified loss functions or model aggregation methods (Li et al., 2020; Karimireddy et al., 2020; Acar et al., 2021; Yu et al., 2020; Reddi et al., 2021). However, the transferred data is still a lot for the edge devices with bandwidth constraints or countries having low-quality communication infrastructure.1 A large amount of transferred data introduces an energy consumption issue on edge devices because wireless communication is significantly more powerintensive than computation (Yadav & Yadav, 2016; Yan et al., 2019).
In this work, we propose a communication-efficient re-parameterization for FL, FedPara, which reduces the number of bits transferred per round. FedPara directly re-parameterizes each fully-
1The gap between the fastest and the lowest communication speed across countries is significant; approximately 63 times different (Speedtest).
connected (FC) and convolutional layers of the model to have a small and factorized form while preserving the model’s capacity. Our key idea is to combine the Hadamard product with low-rank parameterization as W = (X1Y>1 ) (X2Y>2 ) ∈ Rm×n, called low-rank Hadamard product. When rank(X1Y > 1 ) = rank(X2Y > 2 ) = r, then rank(W) ≤ r2. This outstanding property facilitates spanning a full-rank matrix with much fewer parameters than the typical m× n matrix. It significantly reduces the communication burdens during training. At the inference phase, we pre-compose and maintain W that boils down to its original structure; thus, FedPara does not alter computational complexity at inference time. Compared to the aforementioned prior works that tackle reducing the required communication rounds for convergence, our FedPara is an orthogonal approach in that FedPara does not change the optimization part but re-defines each layer’s internal structure.
We demonstrate the effectiveness of FedPara with various network architectures, including VGG, ResNet, and LSTM, on standard classification benchmark datasets for both IID and non-IID settings. The accuracy of our parameterization outperforms that of the traditional low-rank parameterization baseline given the same number of parameters. Besides, FedPara has comparable accuracy to original counterpart models and even outperforms as the number of parameters increases at times. We also combine FedPara with other FL algorithms to improve communication efficiency further.
We extend FedPara to the personalized FL application, named pFedPara, which separates the roles of each sub-matrix into global and local inner matrices. The global and local inner matrices learn the globally shared common knowledge and client-specific knowledge, respectively. We devise three scenarios according to the amount and heterogeneity of local data using the subset of FEMNIST and MNIST. We demonstrate performance improvement and robustness of pFedPara against competing algorithms. We summarize our main contributions as follows:
• We propose FedPara, a low-rank Hadamard product parameterization for communicationefficient FL. Unlike traditional low-rank parameterization, we show that FedPara can span a full-rank matrix and tensor with reduced parameters. We also show that FedPara requires up to ten times fewer total communication costs than the original model to achieve target accuracy. FedPara even outperforms the original model by adjusting ranks at times.
• Our FedPara takes a novel approach; thereby, our FedPara can be combined with other FL methods to get mutual benefits, which further increase accuracy and communication efficiency.
• We propose pFedPara, a personalization application of FedPara, which splits the layer weights into global and local parameters. pFedPara shows more robust results in challenging regimes than competing methods.

heading: 2 METHOD
text: In this section, we first provide the overview of three popular low-rank parameterizations in Section 2.1 and present our parameterization, FedPara, with its algorithmic properties in Section 2.2. Then, we extend FedPara to the personalized FL application, pFedPara, in Section 2.3.
Notations. We denote the Hadamard product as , the Kronecker product as⊗, n-mode tensor product as ×n, and the i-th unfolding of the tensor T (i) ∈ Rki× ∏ j 6=i kj given a tensor T ∈ Rk1×···×kn .

heading: 2.1 OVERVIEW OF LOW-RANK PARAMETERIZATION
text: The low-rank decomposition in neural networks has been typically applied to pre-trained models for compression (Phan et al., 2020), whereby the number of parameters is reduced while minimizing the loss of encoded information. Given a learned parameter matrix W ∈ Rm×n, it is formulated as finding the best rank-r approximation, as argmin
W̃ ||W − W̃||F such that W̃ = XY>, where
X ∈ Rm×r, Y ∈ Rn×r, and r min (m,n). It reduces the number of parameters from O(mn) to O((m+ n)r), and its closed-form optimal solution can be found by SVD.
This matrix decomposition is applicable to the FC layers and the reshaped kernels of the convolution layers. However, the natural shape of a convolution kernel is a fourth-order tensor; thus, the lowrank tensor decomposition, such as Tucker and CP decomposition (Lebedev et al., 2015; Phan et al., 2020), can be a more suitable approach. Given a learned high-order tensor T ∈ Rk1×···×kn , Tucker decomposition multiplies a kernel tensor K ∈ Rr1×···×rn with matrices Xi ∈ Rri×n1 , where ri =
rank(T̃ (i)) as T̃ = K ×1 X1 ×2 · · · ×n Xn, and CP decomposition is the summation of rank-1 tensors as T̃ = ∑i=r i=1 x 1 i × x2i × · · · × xni , where x j i ∈ Rkj . Likewise, it also reduces the number of model parameters. We refer to these rank-constrained structure methods simply as conventional low-rank constraints or low-rank parameterization methods.
In the FL context, where the parameters are frequently transferred between clients and the server during training, the reduced parameters lead to communication cost reduction, which is the main focus of this work. The post-decomposition approaches (Lebedev et al., 2015; Phan et al., 2020) using SVD, Tucker, and CP decompositions do not reduce the communication costs because those are applied to the original parameterization after finishing training. That is, the original large-size parameters are transferred during training in FL, and the number of parameters is reduced after finishing training.
We take a different notion from the low-rank parameterizations. In the FL scenario, we train a model from scratch with low-rank constraints, but specifically with low-rank Hadamard product reparameterization. We re-parameterize each learnable layer, including FC and convolutional layers, and train the surrogate model by FL. Different from the existing low-rank method in FL (Konečnỳ et al., 2016), our parameterization can achieve comparable accuracy to the original counterpart.

heading: 2.2 FEDPARA: A COMMUNICATION-EFFICIENT PARAMETERIZATION
text: As mentioned, the conventional low-rank parameterization has limited expressiveness due to its lowrank constraint. To overcome this while maintaining fewer parameters, we present our new low-rank Hadamard product parameterization, called FedPara, which has the favorable property as follows:
Proposition 1 Let X1 ∈ Rm×r1 ,X2 ∈ Rm×r2 ,Y1 ∈ Rn×r1 ,Y2 ∈ Rn×r2 , r1, r2 ≤ min(m,n) and the constructed matrix be W := (X1Y>1 ) (X2Y>2 ). Then, rank(W) ≤ r1r2.
All proofs can be found in the supplementary material including Proposition 1. Proposition 1 implies that, unlike the low-rank parameterization, a higher-rank matrix can be constructed using the Hadamard product of two inner low-rank matrices, W1 andW2 (Refer to Figure 1). If we choose the inner ranks r1 and r2 such that r1r2 ≥ min(m,n), the constructed matrix does not have a low-rank restriction and is able to span a full-rank matrix with a high chance (See Figure 6); i.e., FedPara
has the minimal parameter property achievable to full-rank. In addition, we can control the number of parameters by changing the inner ranks r1 and r2, respectively, but we have the following useful property to set the hyper-parameters to be a minimal number of parameters with a maximal rank.
Proposition 2 Given R ∈ N, r1 = r2 = R is the unique optimal choice of the following criteria,
argmin r1,r2∈N (r1 + r2)(m+ n) s.t. r1r2 ≥ R 2, (1)
and its optimal value is 2R(m+ n).
Equation 3 implies the criteria that minimize the number of weight parameters used in our parameterization with the target rank constraint of the constructed matrix as R2. Proposition 2 provides an efficient way to set the hyper-parameters. It implies that, if we set r1=r2=R and R2 ≥ min(m,n), FedPara is highly likely to have no low-rank restriction2 even with much fewer parameters than that of a naı̈ve weight, i.e., 2R(m + n) mn. Moreover, given the same number of parameters, rank(W) of FedPara is higher than that of the naı̈ve low-rank parameterization by a square factor, as shown in Figure 1 and Table 1.
To extend Proposition 1 to the convolutional layers, we can simply reshape the fourth-order tensor kernel to the matrix as RO×I×K1×K2 → RO×(IK1K2) as a naı̈ve way, where O, I,K1, and K2 are the output channels, the input channels, and the kernel sizes, respectively. That is, our parameterization spans convolution filters with a few basis filters of size I ×K1 ×K2. However, we can derive more efficient parameterization of convolutional layers without reshaping as follows:
Proposition 3 Let T1, T2 ∈ RR×R×k3×k4 , X1,X2 ∈ Rk1×R, Y1,Y2 ∈ Rk2×R, R ≤ min(k1, k2) and the convolution kernel beW := (T1×1 X1×2 Y1) (T2×1 X2×2 Y2). Then, the rank of the kernel satisfies rank(W(1)) = rank(W(2)) ≤ R2.
Proposition 3 is the extension of Proposition 1 but can be applied to the convolutional layer without reshaping. In the convolutional layer case of Table 1, given the specific tensor size, Proposition 3 requires 3.8 times fewer parameters than Proposition 1. Hence, we use Proposition 3 for the convolutional layer since the tensor method is more effective for common convolutional models.
Optionally, we employ non-linearity and the Jacobian correction regularization, of which details can be found in the supplementary material. These techniques improve the accuracy and convergence stability but not essential. Depending on the resources of devices, these techniques can be omitted.
2.3 PFEDPARA: PERSONALIZED FL APPLICATION
In practice, data are heterogeneous and personal due to different characteristics of each client, such as usage times and habits. FedPer (Arivazhagan et al., 2019) has been proposed to tackle this scenario by distinguishing global and local layers in the model. Clients only transfer global layers (the top layer) and keep local ones (the bottom layers) on each device. The global layers learn jointly to extract general features, while the local layers are biased towards each user.
With our FedPara, we propose a personalization application, pFedPara, in which the Hadamard product is used as a bridge between the global inner weight W1 and the local inner weight W2. Each layer of the personalized model is constructed by W = W1 (W2 + 1), where W1 is transferred
to the server while W2 is kept in a local device during training. This induces W1 to learn globally shared knowledge implicitly and acts as a switch of the term (W2 + 1). Conceptually, we
2Its corollary and empirical evidence can be found in the supplementary material. Under Proposition 2, R2 ≥ min(m,n) is a necessary and sufficient condition for achieving a maximal rank.
can interpret by rewriting W = W1 W2+W1 = Wper.+Wglo., where Wper. = W1 W2 and Wglo. = W1. The construction of the final personalized parameter W in pFedPara can be viewed as an additive model of the global weight Wglo. and the personalizing residue Wper.. pFedPara transfers only a half of the parameters compared to FedPara under the same rank condition; hence, the communication efficiency is increased further.
Intuitively, FedPer and pFedPara are distinguished by their respective split directions, as illustrated in Figure 2. We summarize our algorithms in the supplementary material. Although we only illustrate feed-forward network cases for convenience, it can be extended to general cases.

heading: 3 EXPERIMENTS
text: We evaluate our FedPara in terms of communication costs, the number of parameters, and compatibility with other FL methods. We also evaluate pFedPara in three different non-IID scenarios. We use the standard FL algorithm, FedAvg, as a backbone optimizer in all experiments except for the compatibility experiments. More details and additional experiments can be found in the supplementary material.

heading: 3.1 SETUP
text: Datasets. In FedPara experiments, we use four popular FL datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), CINIC-10 (Darlow et al., 2018), and the subset of Shakespeare (Shakespeare, 1994). We split the datasets randomly into 100 partitions for the CIFAR-10 and CINIC-10 IID settings and 50 partitions for the CIFAR-100 IID setting. For the non-IID settings, we use the Dirichlet distribution for random partitioning and set the Dirichlet parameter α as 0.5 as suggested by He et al. (2020b). We assign one partition to each client and sample 16% of clients at each round during FL. In pFedPara experiments, we use the subset of handwritten datasets: MNIST (LeCun et al., 1998) and FEMNIST (Caldas et al., 2018). For the non-IID setting with MNIST, we follow McMahan et al. (2017), where each of 100 clients has at most two classes.
Models. We experiment VGG and ResNet for the CNN architectures and LSTM for the RNN architecture as well as two FC layers for the multilayer perceptron. When using VGG, we use the VGG16 architecture (Simonyan & Zisserman, 2015) and replace the batch normalization with the group normalization as suggested by Hsieh et al. (2020). VGG16ori. stands for the original VGG16, VGG16low the one with the low-rank tensor parameterization in a Tucker form by following TKD (Phan et al., 2020), and VGG16FedPara the one with our FedPara. In the pFedPara tests, we use two FC layers as suggested by McMahan et al. (2017).
Rank Hyper-parameter. We adjust the inner rank of W1 and W2 as r = (1−γ)rmin + γrmax, where rmin is the minimum rank allowing FedPara to achieve a full-rank by Proposition 2, rmax is the maximum rank such that the number of FedPara’s parameters do not exceed the number of original parameters, and γ ∈ [0, 1]. We fix the same γ for all layers for simplicity.3 Note that γ determines the number of parameters.

heading: 3.2 QUANTITATIVE RESULTS
text: Capacity. In Table 2, we validate the propositions stating that our FedPara achieves a higher rank than the low-rank parameterization given the same number of parameters. We train VGG16low and VGG16FedPara for the same target rounds T , and use 10.25% and 10.15% of the VGG16ori. parameters, respectively, to be comparable. As shown in Table 2a, VGG16FedPara surpasses VGG16low on all the IID and non-IID settings benchmarks with noticeable margins. The same tendency is observed in the recurrent neural networks as shown in Table 2b. We train LSTM on the subset of Shakespeare. Table 2b shows that LSTMFedPara has higher accuracy than LSTMlow, where the number of parameters is set to 19% and 16% of LSTM, respectively. This experiment evidences that our FedPara has a better model expressiveness and accuracy than the low-rank parameterization.
3The parameter γ can be independently tuned for each layer in the model. Moreover, one may apply neural architecture search or hyper-parameter search algorithms for further improvement. We leave it for future work.
Communication Cost. We compare VGG16FedPara and VGG16ori. in terms of accuracy and communication costs. FL evaluation typically measures the required rounds to achieve the target accuracy as communication costs, but we instead assess total transferred bit sizes, 2 × (#participants)×(model size)×(#rounds), which considers up-/down-link and is a more practical communication cost metric. Depending on the difficulty of the datasets, we set the model size of VGG16FedPara as 10.1%, 29.4%, and 21.8% of VGG16ori. for CIFAR-10, CIFAR-100, and CINIC-10, respectively.
In Figures 3a-3f, VGG16FedPara has comparable accuracy but requires much lower communication costs than VGG16ori.. Figure 3g shows communication costs and energy consumption required for model training to achieve the target accuracy; we compute the energy consumption by the energy model of user-to-data center topology (Yan et al., 2019). VGG16FedPara needs 2.8 to 10.1 times fewer communication costs and energy consumption than VGG16ori. to achieve the same target accuracy. Because of these properties, FedPara is suitable for edge devices suffering from communication and battery consumption constraints.
Model Parameter Ratio. We analyze how the number of parameters controlled by the rank ratio γ affects the accuracy of FedPara. As revealed in Figure 4, VGG16FedPara’s accuracy mostly increases as the number of parameters increases. VGG16FedPara can achieve even higher accuracy than VGG16ori.. It is consistent with the reports from the prior works (Luo et al., 2017; Kim et al., 2019) on model compression, where reduced parameters often lead to accuracy improvement, i.e., regularization effects.
Compatibility. We integrate the FedPara-based model with other FL optimizers to show that our FedPara is compatible with them. We measure the accuracy during the target rounds and the required rounds to achieve the target accuracy. Table 3 shows that VGG16FedPara combined with the current state-of-the-art method, FedDyn, is the best among other combinations. Thereby, we can further save the communication costs by combining FedPara with other efficient FL approaches.
Personalization. We evaluate pFedPara, assuming no sub-sampling of clients for an update. We train two FC layers on the FEMNIST or MNIST datasets using four algorithms, FedPAQ, FedAvg, FedPer, and pFedPara, with ten clients. FedPAQ denotes the local models trained only using their own local data; FedAvg the global model trained by the FedAvg optimization; and FedPer and pFedPara the personalized models of which the first layer (FedPer) and half of the inner
matrices (pFedPara) are trained by sharing with the server, respectively, while the other parts are locally updated. We validate these algorithms on three scenarios in Figure 5.
In Scenario 1 (Figure 5a), the FedPAQ accuracy is higher than those of FedAvg and FedPer because each client has sufficient data. Nevertheless, pFedPara surpasses the other methods. In Scenario 2 (Figure 5b), the FedAvg accuracy is higher than that of FedPAQ because local data are too scarce to train the local models. The FedPer accuracy is also lower than that of FedAvg because the last layer of FedPer does not exploit other clients’ data; thus, FedPer is susceptible to a lack of local data. The higher performance of pFedPara shows that our pFedPara can take advantage of the wealth of distributed data in the personalized setup. In Scenario 3 (Figure 5c), the FedAvg accuracy is much lower than the other methods due to the highly-skewed data distribution. In most scenarios, pFedPara performs better or favorably against the others. It validates that pFedPara can train the personalized models collaboratively and robustly.
Additionally, both pFedPara and FedPer save the communication costs because they partially transfer parameters; pFedPara transfers 3.4 times fewer parameters, whereas FedPer transfers 1.07 times fewer than the original model in each round. The reduction of FedPer is negligible because it is designed to transfer all the layers except the last one. Contrarily, the reduction of pFedPara is three times larger than FedPer because all the layers of pFedPara are factorized by the Hadamard product during training, and only a half of each layer’s parameters are transmitted. Thus, pFedPara is far more suitable in terms of both personalization performance and communication efficiency in FL.
Additional Experiments. We conducted additional experiments of wall clock time simulation and other architectures including ResNet, LSTM, and Pufferfish (Wang et al., 2021), but the results can be found in the supplementary material due to the page limit.

heading: 4 RELATED WORK
text: Federated Learning. The most popular and de-facto algorithm, FedAvg (McMahan et al., 2017), reduces communication costs by updating the global model using a simple model averaging once a large number of local SGD iterations per round. Variants of FedAvg (Li et al., 2020; Yu et al., 2020; Diao et al., 2021) have been proposed to reduce the communication cost, to overcome data heterogeneity, or to increase the convergence stability. Advanced FL optimizers (Karimireddy et al., 2020; Acar et al., 2021; Reddi et al., 2021; Yuan & Ma, 2020) enhance the convergence behavior and improve communication efficiency by reducing the number of necessary rounds until convergence. Federated quantization methods (Reisizadeh et al., 2020; Haddadpour et al., 2021) combine the quantization algorithms with FedAvg and reduce only upload costs to preserve the model accuracy. Our FedPara is a drop-in replacement for layer parameterization, which means it is an orthogonal and compatible approach to the aforementioned methods; thus, our method can be integrated with other FL optimizers and model quantization.
Distributed Learning. In large-scale distributed learning of data centers, communication might be a bottleneck. Gradient compression approaches, including quantization (Alistarh et al., 2017; Bernstein et al., 2018; Wen et al., 2017; Reisizadeh et al., 2020; Haddadpour et al., 2021), sparsification (Alistarh et al., 2018; Lin et al., 2018), low-rank decomposition (Vogels et al., 2019; Wang et al., 2021), and adaptive compression (Agarwal et al., 2021) have been developed to handle communication traffic. These methods do not deal with FL properties such as data distribution and partial participants per round (Kairouz et al., 2019). Therefore, the extension of the distributed methods to FL is non-trivial, especially for optimization-based approaches.
Low-rank Constraints. As described in section 2.1, low-rank decomposition methods (Lebedev et al., 2015; Tai et al., 2016; Phan et al., 2020) are inappropriate for FL due to additional steps; the post-decomposition after training and fine-tuning. In FL, Konečnỳ et al. (2016) and Qiao et al. (2021) have proposed low-rank approaches. Konečnỳ et al. (2016) train the model from scratch with low-rank constraints, but the accuracy is degraded when they set a high compression rate. To avoid such degradation, FedDLR (Qiao et al., 2021) uses an ad hoc adaptive rank selection and shows the improved performance. However, once deployed, those models are inherently restricted by limited low-ranks. In particular, FedDLR requires matrix decomposition in every up/down transfer. In contrast, we show that FedPara has no such low-rank constraints in theory. Empirically, the models re-parameterized by our method show comparable accuracy to the original counterparts when trained from scratch.

heading: 5 DISCUSSION AND CONCLUSION
text: To overcome the communication bottleneck in FL, we propose a new parameterization method, FedPara, and its personalized version, pFedPara. We demonstrate that both FedPara and pFedPara can significantly reduce communication overheads with minimal performance degradation or better performance over the original counterpart at times. Even using a strong low-rank constraint, FedPara has no low-rank limitation and can achieve a full-rank matrix and tensor by virtue of our proposed low-rank Hadamard product parameterization. These favorable properties enable communication-efficient FL, of which regimes have not been achieved by the previous low-rank parameterization and other FL approaches. We conclude our work with the following discussions.
Discussions. FedPara conducts multiplications many times during training, including the Hadamard product, to construct the weights of the layers. These multiplications may potentially be more susceptible to gradient exploding, vanishing, dead neurons, or numerical instability than the low-rank parameterization with an arbitrary initialization. In our experiments, we have not observed such issues when using He initialization (He et al., 2015) yet. Investigating initializations appropriate for our model might improve potential instability in our method.
Also, we have discussed the expressiveness of each layers in neural networks in the view of rank. As another view of layer characteristics, statistical analysis of weights and activation also offers ways to initialize weights (He et al., 2015; Sitzmann et al., 2020) and understanding of neural networks (De & Smith, 2020), which is barely explored in this work. It would be a promising future direction to analyze statistical properties of composited weights from our parameterization and activations and may pave the way for FedPara-specific initialization or optimization.
Through the extensive experiments, we show the superior performance improvement obtained by our method, and it appears to be with no extra cost. However, the actual payoff exists in the additional computational cost when re-composing the original structure of W from our parameterization during training; thus, our method is slower than the original parameterization and low-rank approaches. However, the computation time is not a dominant factor in practical FL scenarios as shown in Table 7, but rather the communication cost takes a majority of the total training time. It is evidenced by the fact that FedPara offers better Pareto efficiency than all compared methods because our method has higher accuracy than low-rank approaches and much less training time than the original one. In contrast, the computation time might be non-negligible compared to the communication time in distributed learning regimes. While our method can be applied to distributed learning, the benefits of our method may be diminished there. Improving both computation and communication efficiency of FedPara in large-scale distributed learning requires further research. It would be a promising future direction.

heading: ETHICS STATEMENT
text: We describe the ethical aspect in various fields, such as privacy, security, infrastructure level gap, and energy consumption.
Privacy and Security. Although FL is privacy-preserving distributed learning, personal information may be leaked due to the adversary who hijacks the model intentionally during FL. Like other FLs, this risk is also shared with FedPara due to communication. Without care, the private data may be revealed by the membership inference or reconstruction attacks (Rigaki & Garcia, 2020). The local parameters in our pFedPara could be used as a private key to acquire the complete personal model, which would reduce the chance for the full model to be hijacked. It would be interesting to investigate whether our pFedPara guarantees privacy preserving or the way to improve robustness against those risks.
Infrastructure Level Gap. Another concern introduced by FL is limited-service access to the people living in countries having inferior communication infrastructure, which may raise human rights concerns including discrimination, excluding, etc. It is due to a larger bandwidth requirement of FL to transmit larger models. Our work may broaden the countries capable of FL by reducing required bandwidths, whereby it may contribute to addressing the technology gap between regions.
Energy Consumption. The communication efficiency of our FedPara directly leads to noticeable energy-saving effects in the FL scenario. It can contribute to reducing the battery consumption of IoT devices and fossil fuels used to generate electricity. Moreover, compared to the optimizationbased FL approaches that reduce necessary communication rounds, our method allows more clients to participate in each learning round under the fixed bandwidth, which would improve convergence speed and accuracy further.

heading: ACKNOWLEDGEMENT
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2021R1C1C1006799), and the ”HPC Support” Project supported by the ‘Ministry of Science and ICT’ and NIPA.

heading: SUPPLEMENTARY
text: In this supplementary material, we present additional details, results, and experiments that are not included in the main paper due to the space limit. The contents of this supplementary material are listed as follows:

heading: CONTENTS

heading: A Maximal Rank Property

heading: B Additional Techniques

heading: C Details of Experiment Setup
text: C.1 Datasets
C.2 Models
C.3 FedPara & pFedPara
C.4 Hyper-parameters of Backbone Optimizer
C.5 Hyper-parameters of Other Optimizer

heading: D Additional Experiments
text: D.1 Training Time
D.2 Other Models

heading: A MAXIMAL RANK PROPERTY
text: This section presents the proofs of our propositions and an additional analysis of our method’s algorithmic behavior in terms of the rank property.

heading: A.1 PROOFS
text: Proposition 1 Let X1 ∈ Rm×r1 ,X2 ∈ Rm×r2 ,Y1 ∈ Rn×r1 ,Y2 ∈ Rn×r2 , r1, r2 ≤ min(m,n) and the constructed matrix be W := (X1Y>1 ) (X2Y>2 ). Then, rank(W) ≤ r1r2. Proof. X1Y>1 and X2Y > 2 can be expressed as the summation of rank-1 matrices such that XiY
> i =∑j=ri
j=1 xijy > ij , where xij and yij are the j-th column vectors of Xi and Yi, and i ∈ {1, 2}. Then,
W = X1Y > 1 X2Y>2 = j=r1∑ j=1 x1jy > 1j j=r2∑ j=1 x2jy > 2j = k=r1∑ k=1 j=r2∑ j=1 (x1ky > 1k) (x2jy>2j). (2)
W is the summation of r1r2 number of rank-1 matrices; thus, rank(W) is bounded above r1r2.
Proposition 2 Given R ∈ N, r1 = r2 = R is the unique optimal choice of the following criteria, argmin r1,r2∈N (r1 + r2)(m+ n) s.t. r1r2 ≥ R 2, (3)
and its optimal value is 2R(m+ n).
Proof. We use arithmetic-geometric mean inequality and the given constraint. We have
(r1 + r2)(m+ n) ≥ 2 √ r1r2(m+ n) ≥ 2R(m+ n). (4)
The equality holds if and only if r1 = r2 = R by the arithmetic–geometric mean inequality.
Corollary 1 Under Proposition 2, R2 ≥ min(m,n) is a necessary and sufficient condition for achieving the maximal rank of W = (X1Y>1 ) (X2Y>2 ) ∈ Rm×n, where X1 ∈ Rm×r1 ,X2 ∈ Rm×r2 ,Y1 ∈ Rn×r1 ,Y2 ∈ Rn×r2, and r1, r2 ≤ min(m,n).
Proof. We first prove the sufficient condition. Given r1 = r2 = R under Proposition 2 and R2 ≥ min(m,n), rank(W) ≤ min(r1r2,m, n) = min(R2,m, n) = min(m,n). The matrix W has no low-rank restriction; thus the condition, R2 ≥ min(m,n), is the sufficient condition. The necessary condition is proved by contraposition; if R2 < min(m,n), the matrix W cannot achieve the maximal rank. Since r1 = r2 = R under Proposition 2and R2 < min(m,n), then rank(W) ≤ min(r1r2,m, n) = min(R2,m, n) = R2 < min(m,n). That is, rank(W) is upperbounded by R2, which is lower than the maximal achievable rank of W. Therefore, the condition, R2 ≥ min(m,n), is the necessary condition because the contrapositive is true. Corollary 1 implies that, with R2 ≥ min(m,n), the constructed weight W does not have the lowrank limitation, and allows us to define the minimum inner rank as rmin := min(d √ m e, d √ n e). If we set r1 = r2 = rmin, rank(W) of our FedPara can achieve the maximal rank because r1r2 = r 2 min ≥ min(m,n) while minimizing the number of parameters.
Proposition 3 Let T1, T2 ∈ RR×R×k3×k4 , X1,X2 ∈ Rk1×R, Y1,Y2 ∈ Rk2×R, R ≤ min(k1, k2) and the convolution kernel beW := (T1×1 X1×2 Y1) (T2×1 X2×2 Y2). Then, the rank of the kernel satisfies rank(W(1)) = rank(W(2)) ≤ R2. Proof. According to Rabanser et al. (2017), the 1st and 2nd unfolding of tensors can be expressed as
W(1) = (X1T (1)1 (I(4) ⊗ I(3) ⊗Y1)>) (X2T (1) 2 (I (4) ⊗ I(3) ⊗Y2)>),
W(2) = (Y1T (2)1 (I(4) ⊗ I(3) ⊗X1)>) (Y2T (2) 2 (I (4) ⊗ I(3) ⊗X2)>), (5)
where I(3) ∈ Rk3×k3 and I(4) ∈ Rk4×k4 are identity matrices. SinceW(1) andW(2) are matrices, we apply the same process used in Eq. 2, then we obtain rank(W(1)) = rank(W(2)) ≤ R2.

heading: A.2 ANALYSIS OF THE RANK PROPERTY
text: To demonstrate our propositions empirically, we sample the parameters randomly and count rank(W). When applying our parameterization to W ∈ R100×100, we set rmin = 10 by Corollary 1 to Proposition 2. We sample the entries of X1,X2,Y1,Y2 ∈ R100×10 from the standard Gaussian distribution and repeat this experiment 1, 000 times.
As shown in Figure 6, we observe that our parameterization achieves the full rank with the probability of 100% but requires 2.5 times fewer entries than the original 100× 100 matrix. This empirical result demonstrates that our parameterization can span the full-rank matrix with fewer parameters efficiently.

heading: B ADDITIONAL TECHNIQUES
text: We devise additional techniques to improve the accuracy and stability of our parameterization. We consider injecting a non-linearity in FedPara and Jacobian correction regularization to further squeeze out the performance and stability. However, as will be discussed later, these are optional.
Non-linear Function. The first is to apply a non-linear function before the Hadamard product. FedPara composites the weight as W = W1 W2, and we inject non-linearity as W = σ(W1) σ(W2) as a practical design. This makes our algorithm departing from directly applying the proofs, i.e., empirical heuristics. However, the favorable performance of our FedPara regardless of this non-linearity injection in practice suggests that our re-parameterization is a reasonable model to deploy.
The non-linear function σ(·) can be any non-linear function such as ReLU, Tanh, and Sigmoid, but ReLU and Sigmoid restrict the range to only positive, whereas Tanh has both negative and positive values of range [−1, 1]. The filters may learn to extract the distinct features, such as edge, color, and blob, which require both negative and positive values of the filters. Furthermore, the reasonably bounded range can prevent overshooting a large value by multiplication. Therefore, Tanh is suitable, and we use Tanh through experiments except for those of pFedPara.
Jacobian Correction. The second is the Jacobian correction regularization, which induces that the Jacobians of X1,X2,Y1, and Y2 follow the Jacobian of the constructed matrix W. Suppose that X1,X2 ∈ Rm×r and Y1,Y2 ∈ Rn×r are given. We construct the weight as W = W1 W2, where W1 = X1Y>1 and W2 = X2Y > 2 . Additionally, suppose that the Jacobian of W with respect to the objective function is given: JW = ∂L∂W . We can compute the Jacobians of X1,X2,Y1, and Y2 with respect to the objective function and apply one-step optimization with SGD. For simplicity, we set the momentum and the weight decay to be zero. Then, we can compute the Jacobian of other variables using the chain rule:
JW1 = ∂L
∂W1 = JW W2, JX1 =
∂L
∂X1 = JW1Y
> 1 , JY1 =
∂L
∂Y1 = X>1 JW1 ,
JW2 = ∂L
∂W2 = JW W1, JX2 =
∂L
∂X2 = JW2Y
> 2 , JY2 =
∂L
∂Y2 = X>2 JW2 .
(6)
We update the parameters using SGD with the step size η as follows:
X′1 = X1 − ηJX1 , Y′1 = Y1 − ηJY1 , X′2 = X2 − ηJX2 , Y′2 = Y2 − ηJY2 .
(7)
We can compute W′, which is the constructed weight after one-step optimization:
W′ = (X′1Y ′> 1 ) (X′2Y′>2 )
= {(X1 − ηJX1)(Y > 1 − ηJ>Y1)} {(X2 − ηJX2)(Y > 2 − ηJ>Y2)} = {X1Y>1 − η(JX1Y > 1 +X1J > Y1) + η 2JX1J > Y1} {X2Y > 2 − η(JX2Y > 2 +X2J > Y2) + η 2JX2J > Y2} = (X1Y > 1 ) (X2Y>2 ) + η4(JX1J > Y1) (JX2J > Y2)
− η3{(JX1Y > 1 +X1J > Y1) (JX2J > Y2) + (JX2Y > 2 +X2J > Y2) (JX1J > Y1)} + η2{(JX1Y > 1 +X1J > Y1) (JX2Y > 2 +X2J > Y2) + (JX2J > Y2) (X2Y > 2 ) + (JX2J > Y2) (X1Y > 1 )} − η{(JX1Y > 1 +X1J > Y1) (X2Y > 2 ) + (JX2Y > 2 +X2J > Y2) (X1Y > 1 )}.
(8)
This shows that gradient descent and ascent are mixed, as shown in the signs of each term. We propose the Jacobian correction regularization to minimize the difference between W′ and W − ηJW, which induces our parameterization to follows the direction of W−ηJW. The total objective function consists of the target loss function and the Jacobian correction regularization as:
R = L(X1,X2,Y1,Y2) + λ
2 ||W′ − (W − ηJW)||2. (9)
Results. We evaluate the effects of each technique in Table 4. We train VGG16 with group normalization on the CIFAR-10 IID setting during the same target rounds. We set γ = 0.1 and λ = 10. As
shown, the model with both Tanh and regularization has higher accuracy and lower variation than the base model. There is gain in accuracy and variance with both techniques, whereas only variance with only one technique.
Again, note that these additional techniques are not essential for FedPara to work; therefore, we can optionally use these techniques depending on the situation where the device has enough computing power.

heading: C DETAILS OF EXPERIMENT SETUP
text: In this section, we explain the details of the experiments, including datasets, models, and hyperparameters. We also summarize our FedPara and pFedPara into the pseudo algorithm. For implementation, we use PyTorch Distributed library (Paszke et al., 2019) and 8 NVIDIA GeForce RTX 3090 GPUs.

heading: C.1 DATASETS
text: CIFAR-10. CIFAR-10 (Krizhevsky et al., 2009) is the popular classification benchmark dataset. CIFAR-10 consists of 32× 32 resolution images in 10 classes, with 6, 000 images per class. We use 50, 000 images for training and 10, 000 images for testing. For federated learning, we split training images into 100 partitions and assign one partition to each client. For the IID setting, we split the dataset into 100 partitions randomly. For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0.5 as suggested by He et al. (2020b;a).
CIFAR-100. CIFAR-100 (Krizhevsky et al., 2009) is the popular classification benchmark dataset. CIFAR-100 consists of 32 × 32 resolution images in 100 classes, with 6, 000 images per class. We use 50, 000 images for training and 10, 000 images for testing. For federated learning, we split training images into 50 partitions. For the IID setting, we split the dataset into 50 partitions randomly. For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0.5 as suggested by He et al. (2020b;a).
CINIC-10. CINIC-10 (Darlow et al., 2018) is a drop-in replacement for CIFAR-10 and also the popular classification benchmark dataset. CINIC-10 consists of 32 × 32 resolution images in 10 classes and three subsets: training, validation, and test. Each subset has 90, 000 images with 9, 000 per class, and we do not use the validation subset for training. For federated learning, we split training images into 100 partitions. For the IID setting, we split the dataset into 100 partitions randomly. For the non-IID setting, we use the Dirichlet distribution and set the Dirichlet parameter as 0.5 as suggested by He et al. (2020b;a).
MNIST. MNIST (LeCun et al., 1998) is a popular handwritten number image dataset. MNIST consists of 70, 000 number of 28 × 28 resolution images in 10 classes. We use 60, 000 images are for training and 10, 000 images for testing. We do not use MNIST IID-setting, and we split the dataset so that clients have at most two classes as suggested by McMahan et al. (2017) for a highly-skew non-IID setting.
FEMNIST. FEMNIST (Caldas et al., 2018) is a handwritten image dataset for federated settings. FEMNIST has 62 classes and 3, 550 clients, and each client has 226.83 data samples on average of 28× 28 resolution images. FEMNIST is the non-IID dataset labeled by writers.
Shakespeare. Shakespeare (Shakespeare, 1994) is a next word prediction dataset for federated learning settings. Shakespeare has 80 classes and 1, 129 clients, and each client has 3, 743.2 data samples on average (Caldas et al., 2018).

heading: C.2 MODELS
text: VGG16. PyTorch library (Paszke et al., 2019) provides VGG16 with batch normalization, but we replace the batch normalization layers with the group normalization layers as suggested by Hsieh et al. (2020) for federated learning. We also modify the FC layers to comply with the number of classes. The dimensions of the output features in the last three FC layers are 512–512–〈#classes〉, sequentially. We do not apply our parameterization to the last three FC layers and set the same γ to all convolutional layers in the model for simplicity. For reference purpose, Table 5 shows the number of parameters corresponding to each γ.
Two FC Layers. In personalization experiments, we use two FC layers as suggested by McMahan et al. (2017) but modify the size of the hidden features corresponding to the number of classes in the datasets. The dimensions of the output features in two FC layers are 256 and the number of classes, respectively; i.e., 256–〈#classes〉. We do not use other layers, such as normalization and dropout, and set γ = 0.5 for pFedPara.
LSTM. For the Shakespeare dataset, we use two-layer LSTM as suggested by McMahan et al. (2017) and Acar et al. (2021). We set the hidden dimension as 256 and the number of classes as 80. We also apply the weight normalization technique on original parameterization, low-rank parameterization, and FedPara.

heading: C.3 FEDPARA & PFEDPARA
text: Algorithm 1: FedPara Input: rounds T , parameters {X1l,X2l,Y1l,Y2l}l=Ll=1 where {X1l,X2l,Y1l,Y2l} is the lth
layer of the model and L is the number of layers
for t = 1, 2, . . . , T do Sample the subset S of clients; for each client c ∈ S do
Download {X1l,X2l,Y1l,Y2l}l=Ll=1 from the server; Optimize({X1l,X2l,Y1l,Y2l}l=Ll=1 ); Upload {X1l,X2l,Y1l,Y2l}l=Ll=1 to the server;
end Aggregate {X1l,X2l,Y1l,Y2l}l=Ll=1 ;
end
Algorithm 2: pFedPara Input: rounds T , parameters {X1l,X2l,Y1l,Y2l}l=Ll=1 where {X1l,X2l,Y1l,Y2l} is the lth
layer of the model and L is the number of layers Transmit {X1l,X2l,Y1l,Y2l}l=Ll=1 to clients to train the same initial point at start; for t = 1, 2, . . . , T do
Sample the subset S of clients; for each client c ∈ S do
Download half of parameters {X1l,Y1l}l=Ll=1 from the server; Optimize({X1l,X2l,Y1l,Y2l}l=Ll=1 ); Upload {X1l,Y1l}l=Ll=1 to the server;
end Aggregate {X1l,Y1l}l=Ll=1 ;
end
We summarize our two methods, FedPara and pFedPara, into Algorithms 1 and 2 for apparent comparison. In these algorithms, we mainly use the popular and standard algorithm, FedAvg, as a backbone optimizer, but we can switch with other optimizer and aggregate methods. As mentioned in Section B, we can consider the additional techniques. In the FedPara experiments, we use the regularization to Algorithm 1, and set the regularization coefficient λ as 1.0. In pFedPara experiments, we do not apply the additional techniques to Algorithm 2.

heading: C.4 HYPER-PARAMETERS OF BACKBONE OPTIMIZER
text: FedAvg (McMahan et al., 2017) is the most popular algorithm in federated learning. The server samples S number of clients as a subset in each round, each client of the subset trains the model locally by E number of SGD epochs, and the server aggregates the locally updated models and repeats these processes during the total rounds T . We use FedAvg as a backbone optimization algorithm, and its hyper-parameters of our experiments, such as the initial learning rate η, local batch size B, and learning rate decay τ , are described in Table 6.

heading: C.5 HYPER-PARAMETERS OF OTHER OPTIMIZERS
text: For compatibility experiment, we combine FedPara with other optimization-based FL algorithms: FedProx (Li et al., 2020), SCAFFOLD (Karimireddy et al., 2020), FedDyn (Acar et al., 2021), and FedAdam (Reddi et al., 2021). FedProx (Li et al., 2020) imposes a proximal term to the objective function to mitigate heterogeneity; SCAFFOLD (Karimireddy et al., 2020) allows clients to reduce the variance of gradients by introducing auxiliary variables; FedDyn (Acar et al., 2021) introduces dynamic regularization to reduce the inconsistency between minima of the local device level empirical losses and the global one; FedAdam employs Adam (Kingma & Ba, 2015) at the server-side instead of the simple model average.
They need a local optimizer to update the model in each client, and we use the SGD optimizer for a fair comparison, and the SGD configuration is the same as that of FedAvg. The four algorithms have additional hyper-parameters. FedProx has a proximal coefficient µ, and we set µ as 0.1. SCAFFOLD has Options I and II to update the control variate, and we use Option II with global learning rate ηg (= 1.0). FedDyn has the hyper-parameter α (= 0.1) in the regularization. FedAdam uses Adam optimizer to aggregate the updated models at the server-side, and we use the
parameters β1 = 0.9, β2 = 0.99, the global learning rate ηg = 0.01, and the local learning rate η = 10−1.5 for Adam optimizer.

heading: D ADDITIONAL EXPERIMENTS
text: In this section, we simulate the computation time, the communication time, and total training time in Section D.1, experiment about other models in Section D.2, and compare our method and the quantization approach in Section D.3.

heading: D.1 TRAINING TIME
text: We can compute the elapsed time during one round in FL by t = tcomp. + tcomm., where tcomp. is the computation time of training the model on local data for several epochs and tcomm. is the communication time of downloading and uploading the updated model. We estimate the computation time by measuring the elapsed time during local epochs. We compute the communication time by
2·model size (Mbyte) network speed (Mbyte/s) , considering upload and download of the model in one round.
Since it is challenging to experiment in a real heterogeneous network environment, we follow the simple standard network simulation setting widely used in the communication literature (Zhu et al., 2020; Jeon et al., 2020). They assume the homogeneous link quality by the average quality to simplify complex network environments in FL communication simulation, i.e., the network speeds are identical for all clients.
We compare the elapsed time per round of VGG16ori. and VGG16FedPara on different network speeds such as 2, 10, and 50 Mbps. As shown in Table 7, the communication time is larger than the computation time indicating that the communication is a bottleneck in FL as expected. Although VGG16FedPara takes more computation time than VGG16ori. due to the weight composition time, VGG16FedPara decreases the communication time by about ten times and the total time by 4.80 to 9.52 times. Table 8 shows the total training time to achieve the same accuracy on the CIFAR10 IID setting. Although FedPara needs three rounds more than the original parameterization, VGG16FedPara requires 4.67 to 9.68 times less training time than VGG16ori. because of communication efficient parameterization.

heading: D.2 OTHER MODELS
text: VGG16 As shown in Figure 7, we compare VGG16ori. and VGG16FedPara with three different γ values. Note that a higher γ uses more parameters than a lower γ. Compared to VGG16ori., VGG16FedPara achieves comparable accuracy and even higher accuracy with a high γ. VGG16FedPara also requires fewer communication costs than VGG16ori..
We also investigate how much accuracy is increased for longer rounds. Table 9 shows that the accuracy is increased in the long round experiment, but the tendency of the 1000 round training is consistent with the 200 round training.
ResNet18 We demonstrate the consistent effectiveness of our method with another architecture, ResNet18. For ResNet18, we train without replacing batch normalization layers, set γ of the first layer, the second layer, and the 1×1 convolution layers as 1.0, and adjust γ of remaining layers to control the ResNet18FedPara size; γ = 0.1 for small model size, γ = 0.6 for mid model size, and γ = 0.9 for large model size. To train the ResNet18FedPara in FL, we set the batch size to 10.
As revealed in Figure 8a, ResNet18FedPara has comparable accuracy and uses fewer communication costs than ResNet18ori., of which the results are consistent with the VGG16 experiments. Figure 8b shows the communication costs required for model training to achieve the same tar-
get accuracy. Our ResNet18FedPara needs 1.17 to 5.1 times fewer communication costs than ResNet18ori., and the results demonstrate that FedPara is also applicable to the ResNet structure.
Pufferfish Pufferfish (Wang et al., 2021) is similar to our method, where they use partially prefactorized networks and employ the hybrid architecture by maintaining original size weights to minimize the accuracy loss due to the low-rank constraints. Since this work can be directly applied in the FL setup, we compare the parameterization of PufferFish and FedPara as follows.
We train VGG16 with PufferFish and FedPara on the CIFAR-10 IID dataset in FL, and evaluate the models according to varying the number of parameters. As shown in Table 10, our FedPara has higher accuracy with fewer parameters compared with PufferFish. Although PufferFish is superior to the naive low-rank pre-decomposition method due to its hybrid architecture, we think the hybrid architecture of PufferFish still suffers from the low-rank constraints on the top layers. However, our method is free from such limitations both theoretically and empirically.
LSTM We train LSTMori., LSTMlow, and LSTMFedPara on the Shakespeare dataset. The parameters ratio of LSTMlow and LSTMFedPara are about 16% and 19% of the LSTMori. parameters, respectively. Table 11 shows that LSTMFedPara outperforms LSTMlow and LSTMori. on the IID setting. In the nonIID setting, LSTMFedPara accuracy is higher than LSTMlow and slightly lower than LSTMori. with only 19% of parameters. Therefore, our parameterization can be applied to general neural networks.

heading: D.3 QUANTIZATION
text: FedPAQ (Reisizadeh et al., 2020) is a quantization approach to reduce communication costs for FL. To compare the accuracy and transferred size per round, we train VGG16 on the CIFAR-10 IID setting. We consider both downlink and uplink to evaluate communication costs. We quantize the model from 32 bits floating-point numbers to 16 bits floating-point numbers.
Table 12 shows the comparison between FedPara and FedPAQ as well as those integration. Compared to FedPAQ, FedPara transfers 1.96 times lower bits per round because FedPAQ only reduces the uplink communication cost. Compared with FedAvg, FedPara achieves 0.14% higher accuracy, but FedPAQ 1.01% lower accuracy. FedPara can transfer the full information of updated weights to the server, whereas FedPAQ loses the updated weight information due to the quantization. Furthermore, since the way FedPara reduces communication costs is different from quantization, we integrate FedPara with FedPAQ as an extension. Combining our method with FedPAQ reduces communication costs by 25% further from our FedPara without the integration while having a minor accuracy drop, 0.1%, from that of FedAvg.

Review summaries:
This paper introduces FedPara, a low-rank based method for achieving communication efficient federated learning. On the client-side, low rank factorized models are trained. Different from the traditional approaches to finding low rank factorized networks, FedPara introduces a novel low rank factorization strategy, which attains a higher maximal rank for the weight matrices. The paper also identifies that FedPara can be used to enhance personalized federated learning (by introducing a variant of FedPara, i.e., pFedPara). Both theoretical analysis and experimental results are provided to show the effectiveness of FedPara. 

The paper proposes a communication-efficient parameterization methodology for federated learning tasks through the Hadamard product of low-rank weights. Such parameterization has better model expressiveness in terms of minimal parameters to achieve a maximal rank, compared with conventional low-rank approaches. Using such parameterization, the work designs the corresponding communication-efficient federated learning framework which only uploads a low-rank part of parameters of each level to the server. The work presents empirical studies of the proposed algorithm and validates its effectiveness in terms of accuracy vs communication costs.

The paper introduces FedPara, which is a low-rank parametrization method for the neural networks aiming to reduce the total number of communication bits while preserving the model accuracy in the federated learning scenario. Besides FedPara, which is designed for federated learning applications, the paper also generalizes FedPara into pFedPara, which is designed for personalized federated learning.  The novelty of this paper includes using the Hadamard product in the low-rank approximation. Compared with directly using low-rank approximation, using Hadamard produce can increase the expressiveness of the parametrization.  The paper also includes empirical results to show the effectiveness of FedPara/pFedPara in the federated learning setting.