The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
Authors: Shiwei Li
Xiandi Luo
Haozhao Wang
Xing Tang
Shijie Xu
weihongluo
Yuhua Li
xiuqiang He
Ruixuan Li
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Communication Efficiency
Low-Rank Decomposition.
Primary area: optimization->large_scale_parallel_and_distributed
TL;DR: This paper introduces three novel techniques to improve low-rank decomposition in communication-efficient federated learning.

Abstract: To improve the training efficiency of federated learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. 
In this paper, we seek to enhance the performance of these low-rank decomposition methods. Specifically, we focus on three key issues related to decomposition in FL: what to decompose, how to decompose, and how to aggregate. Subsequently, we introduce three novel techniques: Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD), each targeting a specific issue. These techniques are complementary and can be applied simultaneously to achieve optimal performance. Additionally, we provide a rigorous theoretical analysis to ensure the convergence of the proposed MUD. Extensive experimental results show that our approach achieves faster convergence and superior accuracy compared to relevant baseline methods. The code is available at https://github.com/Leopold1423/fedmud-icml25.

The Panaceas for Improving Low-Rank Decomposition in Communication- Federated Learning
Shiwei Li 12* Xiandi Luo
Haozhao Wang' Xing Tang² Shijie Xu Weihong Luo Yuhua Li
Xiuqiang He Ruixuan Li
using its local data; (3) after local training, each client sends the model updates (i.e., changes to model parameters) back learning (FL), previous research has employed low-rank decomposition techniques to reduce communication overhead. In this paper, we seek to enhance the performance of these low-rank decomposition methods. Specifically, we focus three key issues related decomposition in FL: what to decompose, how to decompose, and how to aggregate. Subsequently, we introduce three novel techniques: Model Update Decom- position (MUD), Block-wise Kronecker Decom- position (BKD), and Aggregation-Aware Decom- position (AAD), each targeting a specific issue. These techniques are complementary and can be applied simultaneously to achieve optimal per- formance. Additionally, we provide a rigorous theoretical analysis to ensure the convergence of the proposed MUD. Extensive experimental re- sults show that our approach achieves faster con- vergence and superior accuracy compared rel- evant baseline methods. The code is available at https:: h/tp:::///gttttt http://tttb..cmmmeepo/ooooo/ www.coo////////e
Abstract
To improve the training efficiency of federated to the server; and (4) the server aggregates the model up- dates from clients to generate a new global model. These steps together constitute one training round in FL.
However, FL generally requires multiple training rounds to achieve convergence, leading to considerable commu- nication overhead. Specifically, communication occurs in two phases: the uplink, where clients transmit local model updates to the server, and the downlink, where the server sends global model parameters to the clients. Latency in both phases can significantly reduce training efficiency, es- pecially in scenarios with limited bandwidth large model sizes (Hönig et al., 2022). To mitigate this issue, various strategies have been proposed to compress either the uplink (Li et al., 2024a;b; Reisizadeh et al., 2020) or the bidirec- tional (Tang et al., 2019; Dorfman et al., 2023) communi- cation. This paper focuses on investigating bidirectional compression methods for better training efficiency.
Low-rank decomposition (Sainath al., 2013) is an effec- tive technique for parameter compression, which approxi- mates a matrix by the product of smaller sub-matrices. It has been widely employed in FL for bidirectional commu- nication compression (Yao et al., 2021; Hyeon-Wonn et al., 2022). these approaches, the server transmits low-rank model to the clients for training and subsequently receives the optimized models from them. For example, FedHM (Yao et al., 2021) generates a low-rank model by applying truncated singular value decomposition (SVD) to the global model. However, the SVD process introduces approxima- tion errors, causing the low-rank model to deviate from the global model. In contrast, FedLMT (Liu et al., 2024) di- rectly trains a pre-decomposed global model, eliminating the need for SVD. Similarly, FedPara (Hyeon-Woo -oo et al., 2022) trains pre-decomposed model and enhances the rank of the recovered matrices using the Hadamard product. (Mei et al., 2022) further improves compression by sharing low- rank matrices across multiple layers. These methods have laid the foundation for low-rank decomposition techniques in ceapuunccatioo-ffficieee FL (CEFL). Nevertheless, sev- eral challenges remain. In this paper, address three key
fedmud- im 2.5.
1. Introduction
Federated learning (FL) is a distributed training framework that preserves data privacy by exchanging model parameters instead of sharing decentralized data (McMahan et al., 2017). In FL system, central server manages the training pro- cess across multiple distributed clients, typically involving the following steps: (1) the server broadcasts the global model to the clients; (2) each client optimizes the model
Eqqual contribution. 'Huazhong University of Science and Technology, Wuhan, China "Shenzhen Technology University, Shenzhen, China FYT, Tencent, Shenzhen, China. Correspondence to: Ruixuan Li <rlli @ hus...ddu.nn........
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Panaccas for Improving Now-Rank Decomposition in Communicationn-Iliiccinn Federated Learning
Aggregate
Implacii
pf e arm
Wo
+
AW
Ru
# PCam
Ispliiii
Recoves
Aggregate
SVD
Modal Tpdate Desemppsitionn (MEIT)
Hlokkwiis Kronecker Decompostton (HKD)
Aggregation- Decompostion (AAD)
VT
P
B
Ria
yr
W
AW
Wo
B2
32
Bij
w
Z
Ban
(a) what to decompose
(b) how to decompose
(c) how to aggregate
Figure An illustration three key issues related to decomposition in FL and their corresponding solutions: What to decompose to minimize information loss? Existing methods decomposes the entire parameters, while MUD decomposes only the model update, effectively reducing information loss. (b) How to decompose achieve a higher rank? Standard low-rank decomposition achieves rank(W)) < min < min mm, n) with (m+n)r parameters, while BKD achieves higher rank upper bound, i.c., rank(W)) S min (m, n) with kv mn parameters. (c) How to aggregate to reduce compression errors? Directly aggregating sub-matrices results in implicit enrors, W (1(1)) +1((5)+++(Vaa)) +Va(VV)) //2 (V() (V2) +.,((1)) +U2(V2) /4- U(V) AAD can effectively avoid this deviation.
issues related to decomposition FL to further enhance communication efficiency, shown in Figure 1.
Consequently, Liu et al. (2024) suggest directly aggregating the sub-matrices avoid the SVD process. However, our analysis indicates that this aggregation introduces implicit errors. Specifically, we find that (VV) * W, where U, V and W are the aggregated results of the sub-matrices U, V and the recovered matrix W, respectively. To address this issue, we propose Aggregation-Aware Decomposition (AAD), which decouples the multiplication of trainable sub- matrices, as shown Figure (c)). Through AAD, there is discrepancy between recovering then aggregating and aggregating then recovering, i.c., W UV + UV, where U and V are fixed matrices shared among clients. AAD ap- plies to all forms of products involving trainable parameters that require aggregation, including the Kronecker product.
First, what decompose to minimize information loss? In FL., communication is driven by the model updates derived from local training or global aggregation. Both the server and clients can generate the latest model parameters by transmitting only the model updates. Therefore, we argue that decomposing model updates is more efficient than de- composing the entire model parameters, the latter may lead to greater information loss. Building this insight, we propose Model Update Decomposition (MUD), which freezes the original model parameters and then learns low- rank sub-matrices to serve as model updates during local training, as illustrated Figure ((a).
The contributions of this paper are summarized as follows:
Second, how to decompose to achieve higher rank? The rank of matrix indicates the amount of information it codes, maximizing the rank of the recovered matrix is crucial in matrix decomposition (Hyeon-Woo Woo al., 2022). To achieve this, we propose Block-wise Kronecker Decom- position (BKD), which partitions a matrix into blocks and then decomposes each block with the Kronecker product, as shown Figure (b). BKD offers a higher rank upper bound for the recovered matrix while requiring fewer parameters. Furthermore, the block structure of BKD enables dynamic compression by varying the number of blocks, k. similar to adjusting the rank, r, in traditional low-rank decomposition.
A comprehensive study on low-rank decomposition in CEFL is presented, along with three novel tech- niques to enhance performance: Model Update De- composition, Block-wise Kronecker Decomposition, and Aggregation Decomposition.
A rigorous theoretical analysis of the proposed method is provided, confirming its effectiveness and conver- gence. Specifically, it is demonstrated that the pro- posed method converges faster than existing low-rank decomposition approaches in FL, such as FedLMT.
Third, how to aggregate to reduce compression errors? As shown Figure 1(c), after receiving the optimized low-rank models. the server should aggregate the recovered matri- and then obtain a new low-rank model through SVD, which explicitly introduces errors into the model parameters.
Extensive experiments are conducted on four popular datasets to evaluate the superiority of the proposed method. Notably, our method achieves up to a 12% improvement in test accuracy compared to relevant baselines, with each technique contributing positively.
2

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Ffiiion Federated Learning
2. Preliminaries
model update AW?, as this avoids introducing errors into W To this end, we propose Federated Learning with Model Update Decomposition (FedMUD). As shown in Figure 1, FedMUD freezes the original model parameters and then learns low-rank matrices to serve as model updates. In round t and on client i, the parameter W is optimized as:
The goal of FL is to train a global model using decentralized datasets (D), D2...... ...D,, which can be formulated as:
min S(w)
N [(W),
W = = + AW? = W + UU()),
(3)
where w denotes the model parameters. fi(-) denotes the expected loss function of client i, defined as f((w) & (F, {)], where (w, is the loss value corre- sponding to the data sample ç. For simplicity, we will omit the symbol & in F.(.) in the following discussions.
where W represents the received global parameter, which remains frozen during local training. The trainable compo- nent the pre-decomposed model update, denoted as U. and V Prior to local training. U. is initialized randomly, while V is initialized to zero, ensuring that the model up- date starts at zero. Additionally, the server sends a random seed to the clients to ensure consistent initialization of U across all clients. After local training, the clients send their optimized U and V back to the server, which aggregates these aub-matrices as follows:
To solve (1) in a privacy-preserving manner, FL generally requires multiple rounds of training. In each round t, the server sends the global model parameters w' to clients. Fach client i then optimizes w' using local data through several steps of gradient descent, yielding w [+1 Subsequently, each client sends the local model update, Aw? = - w-
w', back to the server, which aggregates these updates to generate the global model update as follows:
N
N V N 211
U =
US
V'=
(4)
Aw'
Awf.
(2)
In the next round, the server sends Uf and Vi back to the clients. Upon receiving Uf and V/', the clients have two options: (1) incorporate these sub-matrices into the frozen parameters and then reinitialize the sub-matrices, or (2) con- tinue training with the received sub-matrices. Let s denote the number of rounds for resetting model updates, referred to the reset interval. Every s rounds, the recovered matrix U(V) is added to the frozen parameters. Assuming the total number of training rounds is R = ns, the final global model parameters can be expressed as follows:
Using Aw', the global model is updated as w w' + Aw". In the next round, the server can send either w11 or just Aw' to the clients. In the latter case, clients re- construct w+1 with their locally stored w'. Notably, even if a client does not participate in a specific round, it must still download the global model update to ensure correct restora- tion of the model parameters in future rounds. However, the communication overhead incurred by non-participating clients does not impact the overall training efficiency of FL.
R = W
Σ
(5)
3. Methodology
which implies that the initialized parameters are updated through the accumulation of low-rank updates. As 8 de- creases, the number of low-rank updates increases, thereby improving the information richness. To achieve optimal accuracy, 8 set to I by default. Notably, by setting w 0, s 2 R and initializing both U and V randomly, FedMUD reduces to FedLMT. In other words, the parame- ters of FedLMT correspond solely low-rank update of FedMUD, causing Fe to perform much worse than FedMUD. Theoretically, we show that increasing s can neg- atively affect the convergence rate of FedMUD in Section 4.
3.1. Model Update Decomposition
Recent approaches in FL reduce communication overhead by training low-rank models on the client side (Hyeon- Woo et al., 2022). For example, FedHM (Yao al., 2021) gen- erates low-rank models by applying truncated SVD to the global model, while FedLMT (Liu et al., 2024) directly trains a pre-decomposed global model.
However, we argue that these approaches result in substan- tial information loss, which consequently reduces accuracy. Taking the parameter W an example, the communication content in round t can be expressed as W WV'+AW" where AW denotes either a local or global model update. Decomposing the entire model parameter W++ can intro- duce significant errors in both W and AW Since both the server and clients typically have access to W*, it is suffi- cient and more efficient to decompose and transmit only the
3.2. Block-wise Kronecker Decomposition
In the previous section, we demonstrated the necessity of decomposing the model update. With standard low-rank de-
composition, the model update AW E Rmxa is represented as the product of two smaller matrices, U E Rxrr and
3

---

The Panaccas for Improving Low-Rank Decomposition in Commmunication-Efiiiie Federated Learning
Ve Raxr. The compression ratio is 2(m=t))r/mm, where < min(m,n). However, due to the property of matrix multiplication, the rank of the recovered model update is constrained as (ink(W) I min mnin(ramn (rank(U') rank r,, which limits the choice of smaller values for r.
averaging based on the data volume each client holds. How- ever, low-rank decomposition presents new challenges for parameter aggregation. Specifically, there are two ways to aggregate the sub-matrices: direct aggregation and aggrega- tion after recovery. In direct aggregation, the results are also sub-matrices, which can be directly sent to clients for the next training round. In contrast, aggregation after recovery requires applying truncated SVD on the aggregated results to obtain new sub-matrices before transmission to the clients. However, the SVD process introduces approximation errors, which can significantly affect convergence. Therefore, we primarily focus on the direct aggregation.
To address this issue, we utilize the Kronecker product to enhance the rank of the recovered matrix. Assume ym and vn are integers, the Kronecker decomposition can be for- mulated as AW = V, where U, V E R // v AW E Rmxx, and & denotes the Kronecker product. The re- sulting compression ratio is 2/vmi, and the rank satisfies rank(AA) = rU) x rvnk S min (,, n),, which matches the upper bound for a matrix of dimensions (m, n)).
However, directly aggregating the sub-matrices, U and V, introduces bias into the model parameters implicitly. This issue persists regardless of whether U and are used to represent the model parameters the model update. For simplicity, we will explain this bias in the context of the former scenario. Let Uo and Vo denote the initial parameters received by the clients, while U, = Un + AU; and V = Vo + AV; denote the optimized parameters on client i. The recovered matrix, which represents the parameters actually used during the client's forward propagation, is given by Wi U.(V) Their aggregated result is:
However, the compression ratio of the Kronecker decompo- sition, specifically 2//mn, is overly rigid and lacks dynamic introduce Block-
adaptability. To address this limitation, wise Kronecker Decomposition (BKD), which partitions the matrix into h blocks and applies the Kronecker decom- position to each block, as shown in Figure 1. Assume m = ka", .n = kb?, a matrix W E Rmxx will be represented by 1.2 pairs of sub-matrices U. V E Raxb Consequently, the compression ratio is given by 2//ab/mm = 2*/ /m/. Dy- namic compression can be achieved by adjusting k, like ad- justing r in the standard low-rank decomposition. Notably, when k= = 1, BKD reduces to the Kronecker decomposition. The rank upper bound of BKD is full rank, similar to that of the Kronecker product-basse based decomposition discussed above. A detailed analysis of the rank upper bound of BKD provided in Appendix B.
W = A(U.(V))]] U6(V)) (6)
UoA(AV) + (() + A(AU(AVVT),
where A(x) returns the aggregated result of variable x. In direct aggregation, the aggregated results are U = Uo + A(AU) and V = Vo + A(AV).. The recovered matrix of Ú and V is then given by:
Decomposition is commonly applied to linear and con- volutional layers. The parameter of a linear layer is a two-dimensional tensor, making low-rank decomposi- tion straightforward. In contrast, the parameter of a convolutional layer is a four-dimensional tensor W E which should be reshaped into a two- dimensional tensor W E Roourkeenkk before decompo- sition. This strategy has also been utilized in (Liu et al., 2024). In BKD, we let the sub-matrices be square for sim- plicity, i.c., a = b = so that the restored matrix becomes W E xx==" Here, 222 represents the number of blocks, and is determined by / and the size of the target tensor. For instance, in the case of a linear layer with dimensions (m, n), e is computed as = = / mn/ 1.. When k22²4 24 > mn, only the first mn parameters of W | x are reshaped into the target parameter matrix with dimensions (m, This process is applicable to tensors of any dimension.
W = TV(V = Uo(V)) (7) UoA(AV) + AA(U))(V)) A(AU))((VV)
R TooixCC Gr x
It evident that W in Eq. deviates from the desired result w in (6) The discrepancy arises from the last term, i.e., the second-order update induced by AU and AV. To ad- dress this issue, we propose decoupling the multiplication of the trainable matrices U and V to eliminate the second-orde update. Specifically, we introduce Aggregation-Awan De- composition (AAD). which reformulates the operation of UVV)" as follows:
W W- U(V) +Ú(V),
(8)
where U and V are randomly initialized and remain fixed during local training. Through AAD, the actual and desired recovered matrices are equal as follows:
3.3. -ggregation-Aware Decomposition
Aggregation in FL is to synthesize the knowledge learned by different clients. This is commonly achieved by averag- ing the parameters of different clients or through weighted
W = W = -U + (()
T + +Ü(V + A(AV))
When AAD is applied to model updates, both U and V shall be initialized zeros, ensuring that the model update

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Effiicoe Federated Learning
starts at zero. It is important to note that AAD is appli- cable to any product involving trainable parameters that require aggregation, including matrix multiplication and the Kronecker product. Therefore, AAD can also be used to enhance BKD. Integrating AAD with BKD requires only replacing the matrix multiplication in Eq.(8) with the pro- posed BKD operator.
which demonstrates the convergence of FedMUD with non- convex settings. The proof is provided in Appendix C.
Theorem 1. Under Assumptions 1, 2, and 4, let < be a constant and the learning rate satisfy 0 < 1j S
min 1 )- T.,
1}, with FedMUD, we have
1 T
1) 11125
n/T 37 - (((")) (10)
2
E
V
w
4. Convergence A Analysis
O 9 [1+ (L- 0)O - -
In this section, we present a theoretical analysis that demon- strates the convergence of FedMUD and highlights its ad- vantages over Fed Our objective is to compare the convergence rates of full-weight decomposition and model update decomposition (i.e., MM vs. FedMUD); there- fore, BKD and AAD are not considered in this analysis. The notations used are summarized below.
where w denotes the optimal parameters, r - min (2 2² G k2g, x(]] and 1 & min + =2n G² r
Remark 1. By setting p = L, Theorem I is equivalent to the convergence analysis of FedAvg. Additionally, by setting = र and letting c + 1, the convergence rates of both FedAvg and FedMUD are O ((7)).
Notations. For an I-layer neural network, the model pa- rameters are represented as w (WI.... Wa, W Wo+1 + Up+1 Up+i(V++3) (V++1) ...., W,, + Un (V.)) where MUD is ap- plied to the last (L - p) layers. R denotes the number of training rounds. E denotes the number of iterations per round, and T RE denotes the total number of iterations. Il-112 denotes the l norm of a vector and ||-1|p denotes the Frobenius norm of a matrix. In this section, & E [T] de- notes the current iteration number, rather than the training round. For the sake of analysis, we make the following assumptions, as used by FedLMT (Liu et al., 2024). Assumption 1. Each loss function f; (for is differentiable and La-smooth.
Remark 2. Compared to FedAvg, the convergence rate of FedMUD is further affected by the term (L - p) + 1+.. As S + T, Tu and Fv gradually increase, eventually reaching their respective upper bounds, fo and ke. Thus, as S increases, the convergence of FedMUD becomes slower.
Remark 3. By setting S 3 T. Wt- 0 and initializing both Ut and Vi randomly for all l > p. Theorem I reduces the convergence of FedLMT. As noted in Remark 2, FedLMT employs a much larger value for S, which results in a slower convergence rate compared to FedMUD.
.N)
5. Experiments
Assumption 2. The stochastic gradient FF((w) is un- biased, with bounded variance and norm. Specifically, S o²,
5.1. Experimental Setup
FVF.(W) = V.((W) (II - and E (W) S
Datasets and Models. In this section, we evaluate the pro- posed method on four widely used datasets: FMNIST (Xiao et al., 2017), SVHN (Netzer et al., 20,1), CIFAR- 10, and CIFAR- 100 (Krizhevsky & Hinton, 2009). For FMNIST and SVHN, we employ a convolutional neural network (CNN) with four convolutional layers and one fully connected layer. For CIFAR-10 and CIFAR-100, eight convolutional layers and one fully connected layer. ReLU (Glorot al., 2011) is used as the activation function, and batch normalization (BN) (Ioffe & Szegedy, 2015) is utilized to ensure stable training. For better performance, we do not compress the first and last layers of the model.
Assumption 3. The sub-matrices U and V are bounded during training, i.e., ||Uell=- S x2 and [IVil S niz Fur- thermore, the initialization values of U and V also satisfy [[ill S < x2g and [|Ueli== S < x²g
Assumption 4. At least one of the matrices Üi N &hmn U2 and Vi N E ViJ has a positive smallest singular value, i.e. (6min(Ui)] + [6min 2 vur > O, where 6min()) denotes the smallest singular value.
employ a CNN with
Assumptions and are commonly used in convergence analysis under distributed settings (Li et al., 2020; Yu et al., 2019; Li et al., 2024b). Assumption 3 is also commonplace in the theoretical analysis of machine learning models (Chen et al., 2020; Liu et al., 2024). In our case, this assumption holds exactly; see Appendix C.2 for details. Given that the model dimension is finite, it is reasonable to assume that the model weights are bounded. Assumption 4 is supported by the Marchenko-Pastee theory, as discussed by Liu et al. (2024). Based on these assumptions, we present Theorem 1,
Data Partitioning, Each dataset is partitioned into several subsets to serve as the local data of different clients. Based the data partitioning benchmark of FL (Li et al., 2022), we consider two kinds of nID- data distribution, termed Non-IID-1 and NOn-IID-2. In Non-IID-1, the proportion of the same label across clients follows the Dirichlet distri- bution (Yurochkin et al., 2019), while in Non-IID-2, each client only contains data of partial labels. For 100,
5

---

The Panaccas for Improving Low-Rank Decomposition in Communication Federated Learning
Table 1. Accuracy of different methods. Except for FedAvg, the best accuracy is in hold and the second hest accuracy is underlined.
FMNIST
SVHN
CIFAR-10
CIFAR-100
Non-IID-1
Non-IID-2
Non-IID-1
Non-IID-2
Non-IID-1
Non-IID-2
Non-IID-1
Non-IID-2
FodAvg (McMahan et al., 2017)
90.3(4 0.2)
88.6 (#: 0.3)
89.7(= 0.3)
88.3(# 0.3)
81.0(+ 0.7)
77.8(# 0.7)
49.8(t 0.6)
44.8(+ 0.5)
FedHM (Yao al., 2021)
87.2( 0.1)
84.5(# 0.4)
73.8(= 0.8)
7 0.9)
66.5 0.4)
63.3(# 0.7)
28.5(= 0.8)
23.3( 0.3)
FedLMT (Liu et al., 2024)
87.3(1 0.4)
84.4(# 0.5)
71.0= 0.6)
70.1(# 1.0)
64.7 0.8)
62.1(= 0.8)
28.4(# 0.3)
23.4(4 0.6)
FedPara Woo et al., 2022)
87.1 0.2)
84.4(2 0.3)
82.4(= 0.6)
79.3(# 0.5)
70.6( 0.6)
66.6( 0.7)
29.3(# 0.9)
25.3(1 0.6)
EF21-P (Gruntkowsku et al., 2023)
84.1(1: 0.8)
82.0(#: 0.8)
82.5(= 0.5)
76.7(# 0.9)
47.2(4 0.8)
46.8( 0.9)
15.2(4 0.7)
13.8(4 1.0)
FedBAT (Li et ul., 2024b)
88.4(1: 0.1)
86.7(: 0.1)
85.2( 0.4)
83.8(1: 0.3)
75.0(:: 0.1)
72.3(# 0.4)
37. 0.7)
34.2(: 0.6)
FedMUD
87.9(1 0.2)
85.8(: 0.4)
81.3( 0.8)
78.5(1: 0.7)
68.9(1: 0.9)
65.8( 0.4)
29.4(c 0.5)
28.9(:t 0.4)
FedMUD+BKI)
88.2 0.1)
86.7( 0.6)
84.8(+ 0.5)
83.5( 0.4)
70:2(+ 0.7)
67.5( 0.8)
35.7(+ 0.5)
30.7(+ 0.8)
FedMUD+AAD
88.5( 0.3)
87.0(+ 0.3)
84.7(# 0.2)
82.5(+ 0.6)
73.3(t 0.4)
72.3( 0.3)
37.4(+ 0.4)
35. 0.2)
FedMUDDABDD+AAD
89.0(+ 0.2)
87.6( 0.2)
86.6( 0.3)
84.9(+ 0.3)
75.9(++ 0.2)
73.9(+ 0.6)
41.2(+ 0.5)
36.1(+ 0.9)
we set the Dirichlet parameter to 0,1 in Non-IID-1 and as- sign 10 random labels to each client in Non-IID-2. For the other datasets, we set the Dirichlet parameter to 0.3 in Non-IID-1 and assign random labels to each client in Non- IID-2. The data heterogeneity of Non-IID-2 is generally higher than that of Non-IID-1. The experiments under the IID data distribution can be found in Appendix A.
cally, the sub-matrices are randomly initialized with values drawn from the uniform distribution U(-a, a), where a is selected from the set (0.01, 0.05, 0.1, 0.5, 1, 5, 10). In the main experiments, the compression ratio is set to 1/a2 for all methods. For FedMUD, we also report the performance of its variants, corresponding to whether BKD or AAD is applied on top of MUD. Each experiment is run five times, and the average results are reported. The code is provided in the supplementary materials.
Baselines. FedMUD is compared to several CEFL meth- ods, including FedHM (Yao et al., 2021), FedLMT (Liu et al., 2024), FedPara (Hyeon- Woo et al., 2022). EF21-P (Gruntkowska et al., 2023) and FedBAT (Li et al., 2024b). FedHM decomposes the global model on the server through truncated SVD and sends the decomposed model to clients for local training. FedLMT directly trains a pre-decomposed global model. FedPara enhances the rank of the recovered matrix by applying the Hadamard product. EF21-Pis is a state- of-the-ar algorithm for communication compression with the error feedback mechanism in FL. Following the original setting of EF21-P, the Rand-K K and Top-K compressors are used to compress the local and global model updates, respec- tively. FedBAT is an advanced communication quantization algorithm in FL. which learns binary model updates during local training. However, it only compresses uplink commu- nication. For a fair comparison, we also use its quantizer to compress the global model update.
5.2. Overall Performance
In this section, we compare the performance of our methods and the baselines in terms of test accuracy and convergence speed. All numerical results are reported in Table 1, and partial convergence curves are shown in Figure 2. More experimental results can be found in Appendix A.
First, we observe that FedHM and FedLMT exhibit similar performance. As discussed in Section 1, these two meth- ods introduce errors into model parameters in distinct man- ners. Specifically, the errors in FedHM stem from truncated SVD, while those in FedLMT arise from biases introduced by directly aggregating sub- nub-matrices. In contrast, FedPara achieves higher accuracy by increasing the rank of the recov- ered matrix, which highlights the crucial role of matrix rank in low-rank decomposition methods. Compared to these de- composition methods, FedMUD consistently demonstrates superior accuracy and faster convergence. Moreover, its per- formance is further enhanced when integrated with the BKD and AAD techniques. For EF21-P, we attribute its poor per- formance to the excessively high compression rate, with only 3% of the parameters being updated per round. Among the baselines, FedBAT performs the best, notably surpass- ing existing low-rank decomposition methods. However, our method still exceeds FedBAT by substantial margin, bridging the gap in low-rank decomposition for CEFL.
Hy dyperparameters. The number of clients is set to 100, with 10 clients randomly selected to participate in each round of training. The local epoch is set to 3, and the
batch size is 64. SGD (Bottou, 2010) is employed as the local optimizer, with the lcarning rate tuned from the set (1.0, 0.3, 0.1, 0.03, 0.01}. J. The number of training rounds is set to 100 for FMNIST and SVHN, and 200 for CIFAR- 10 and CIFAR-100 100. The initialization of sub-matrices in the decomposition methods has a significant impact on model performance. To ensure a fair comparison, we adjust the ini- tialization across methods to optimize performance. Specifi-
6

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Iffice Federated Learning
F-MNIST with Non-IID-1
SVHN with Non-IID-1
CIFAR-10 with Non-IID-1
CIFAR-100 with Non-IID-1
Accuracy
Accuracy
Accuracy
Accurrry
TodtM
Redeitt RedEMT
feSHH
Fedizty
fedLM!
FodFora
FodPara
RodPara
FosPara
IF21
1121-P
EF221 P
I121
12
RedRAT
MMD
TedBAT
PedRUID SKD-AAC
RedBOT
FeSM4LI BID
FestUTT
100
60
50
Communication volume (48:log1000)
Communication volume (4B:log10(x))
Communication volume (4B:log0000)
Communication volume (1B:tog10(x))
Figure 2. Convergence curves of different methads under the Non-IIl>-1 data distribution.
F- F-MNIST with Non III.1
SVHN with Non-IID. IID-1
CIFAR 10 with Non- con-IIf-1
CIFAR 100 with Non IID-1
Accuracy
Aecurary
Accurscy
184
Pechiin FechUD1 BCD
TedMiiD
RedRUDIBRR BKD
141
F
Fo:M J.H.DI BKD
(4.0
TeIVUD-010 HAD
Ted BKD
Techt D+ADD
Fack DDDKDD+AD
incervals
intervels
Figure 3. Accuracy of FedMUD under different reset intervals.
5.4. Ablation on Reset Intervals
Table Accuracy FedMUD when applied with freezing (+F)
and decoupling (+AAD) under the Non-IIID-1 data distribution.
By default, FedMUD reinitializes the sub-matrices before each round of local training. i.e., the reset interval 8 1. In this section, we examine the effect of the reset interval model performance. As illustrated in Figure 3, model accuracy generally decreases as the reset interval increases, consistent with our theoretical analysis. However, com- pared to s = 1, a slight improvement in model accuracy is occasionally observed for a = or = 4, which can be at- tributed to data heterogeneity. Specifically, a larger interval allows the low-rank matrices observe more diverse data before being added into the frozen parameters, thereby miti- gating the effects of data heterogeneity. Yet, as the interval continues to increase, previously learned knowledge may be forgotten or overwritten, leading to a decline in accuracy. Moreover, when the reset interval equals the total number of training rounds, FedMUD reduces to FedLMT, and its per- formance becomes comparable to that of FedLMT. Figure 3 also demonstrates the performance improvements achieved by AAD and BKD individually, with AAD showing more significant contribution. In conclusion, the reset interval should typically be set to for optimal performance.
FMNIST
SVHN
CIFAR-10
CIFAR-100
FedMUD+F
88,5
81.0
71.1
35.3
FodMUD+AAD
88.5
84.7
73.3
37.4
FedMUD+BKD+F
88.6
84.9
74.4
38.0
FeDMUD-BKDIAAD
89.0
86.6
75.9
41.2
5.3. Decoupling vs. Freezing
ADD decouples the multiplication of trainable matrices to avoid errors introduced by aggregating sub-matrices. In principle, a similar effect could also be achieved by freezing one of the sub-matrices (Sun et al., 2024). Therefore, we conduct an ablation experiment to compare the difference between decoupling and freezing, under the same commu- nication cost. Table 2 demonstrates that AAD consistently outperforms the method that freezes the matrix U, where FedMUD+AAD and FedMUD+F utilize different update matrix shapes: T + (VAA)) for the former and (V=) for the latter. Although UAAD and VAAD indi- vidually have significantly fewer parameters than VF, their sum is approximately equal. As discussed by Hayou al. (2024), the roles of U and V in low-rank training differ, and both of their updates significantly affect convergence. Therefore, freezing either U or V results in suboptimal per- formance. In contrast, AAD can be viewed as an additive combination of two freezing modules, which ensures the simultaneous trainability of U and V.
5.5. Ablation on Initialization Values
The initialization of sub-matrices plays critical role in the performance of low-rank decomposition methods. In our ex- periments, we employed a uniform distribution U(-a, a) to initialize the sub-matrices. This section examines how differ- ent initialization values affect the performance of FedMUD.
7

---

The Panaccas for Improving Low-Rank Decomposition in Communication-EEiiiioe Federated Learning
MNIST with Non IID
SVIIN with Non IID
1
CITAR 10 with Non IID
CIFAR 100 with Non IID 1
033
32.3
Accurety 2.
AccurcC)
Accursoy
Accurccy
23.
FodVUD
R
1..0
FosMUD
ReORLD
PeOMLDD-RK
FADNUD
FoSMUD1
FedRLL HAAD
05:0
0.1
35
18
30
20
13.0
X0
11
03
10
24
150
02
15
50
100
300
2.2
05 Lo
2.
200 20.8
intiaalizeioon Value
Inbializatin Value
Initis izstion Value
Initie lcaaiin Value
Figure 4. Accuracy of FedMUD under different initialization values.
MNIST with Non IID
SVHN with Non IID 1
CIFAR 10 with Non IID
CIFAR 100 with Non IID 1
Accurcy
423
Bocurecy
Aecurncy
AccursCy
TedMUDD TeSMUDD48D
TodMUDD
065
FedMUD
PodPLD
PidMUD+ 5KD
FedMUD BEDM4.D
FedRUDIAAD
FedPLD
Compression Ratio
Compression Radio
Compression Ratio
Compression Ratio
Figure 5. Accuracy of FedMUD under different compression ratios.
In Figure 4, the r-axis denotes the parameter which de- termines the magnitude of the initialization range. As illus- trated, we observe that inappropriate initialization values can lead to decrease in model accuracy, In contrast, the proposed BKD exhibits reduced sensitivity to variations in initialization values. Further, BKD typically requires larger initialization values compared to traditional low-rank de- composition. Overall, some adjustment of the initialization values is necessary for FedMUD to achieve optimal perfor- mance. Based on these observations, we believe that it is feasible to dynamically adjust the initialization values for FedMUD during training, and we leave this as future work.
2018) and Tensor-Trai Decomposition (Novikov et al.,
2018). Recently, such techniques have been widely adopted in model compression (Yao et al., 2021; Novikov et al., 2018), parameter-efficient fine-tuning (Hu et al., 2022; Mao et al., 2024), and related domains.
This paper investigates the application of low-rank decompo- sition in communication--ffiiieee federated learning (CEFL). The work in (Konečný et al., 2016) was the first to apply low-rank decomposition for reducing communication costs, although it only compresses the uplink communication. As previously discussed, subsequent studies (Yao et al., 2021; Liu al., 2024) face several challenges. To address these, we propose three novel techniques that significantly improve the performance of low-rank decomposition in federated learning. It is also noteworthy that low-rank decomposi- tion has seen broad application in personalized federated learning (FL) (Hyeon- Woo et al., 2022; Liu et al., 2024; Wu et al., 2024; Dadras et al., 2024), and our proposed methods can be effectively extended to this setting to further enhance performance.
5.6. A Ablation on Compression Ratios
The compression ratio is set to 1/32 for all methods in Sec- tion 5.2, where FedMUD significantly outperforms the base- lines in terms of accuracy, though it still lags behind FedAvg. This section examines the impact of varying compression ratios on model performance. As illustrated in Figure 5, the accuracy improves as the compression strength decreases. Notably, with 8-fold compression, FedMUDD-BID+AAD achieves accuracy comparable to that of FedAvg.
In addition. low-rank techniques have been extensively uti- lized for the efficient fine-tuning of large language models (LLMs). For example, Hu et al. (2022) introduced Low- Rank Adaptation (LoRA), which approximates updates to pretrained weights using products of low-rank matrices, sig- nificantly reducing memory consumption. However, the con- strained rank may limit model performance. To overcome this, Lialin et al. (2024) proposed superimposing multiple low-rank matrices to increase the effective rank. Further- more, Edalati et al. (2023) replaced matrix multiplication in LoRA with the Kronecker product, enabling theoreti-
6. Related Work
6.1. Low-Rank Decomposition
Low-rank decomposition is matrix compression technique that expresses matrix as the product of two sub-matrices (Sainath et al., 2013). To enhance performance, researchers have further explored decomposing matrices into more sub- matrices, as in Tucker Decomposition (Malik & Becker,
8

---

The Panaccas for Improving Low-Rank Decomposition in Communication-PPiiionn Federated Learning
cal reconstruction of full-rank matrices. Nevertheless, this approach is less flexible in controlling compression rates compared to the proposed BKD method. BoRA (Li et al., 2025b) employs a similar block-wise mechanism in LoRA, where block matrix multiplication with block-wise diagonal matrices is used to enhance the expressivity of LoRA. Li et al. (2025a) have also explored the initialization of low- rank matrices, and demonstrated the potential benefits of non-zero initialization.
Acknowledgements
This work is supported by the National Key Re- search and Development Program of China under grant 2024YFC3307900; the National Natural Science Founda- tion of China under grants 62376103, 62302184, 62436003, and 62206102; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Sci- ence and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation completed in the HPC Platform of Huazhong University of Science and Technology.
6.2. Communication Efficceet Federated Learning
Existing methods primarily reduce communication cost in FL from two aspects: model compression and gradient com- pression. Model compression typically involves training smaller model on the client side. For example, (Caldas et al., 2018; Bouacida et al., 2021) allow clients to train sub- models derived from a larger server model, while (Isik et al., 2023; Bibikar et al., 2022) train pruned models during local training. Furthermore, Yang et al. (2021) trains and commu- nicates a binary neural network. Low-rank decomposition methods, such as FedLMT and FedHM, also fall within the model compression category. However, model compres- sion often sacrifices the model's expressiveness, resulting in decreased accuracy, especially when higher compression strengths are applied. In contrast, gradient compression tar- gets model updates (i.e., accumulated gradients), minimiz- ing the negative impact on model's expressiveness. Existing methods typically apply pruning (Stripelis al., 2022; Qiu et al., 2022; Li et al., 2024a) or quantization (Karimireddy et al., 2019; Li et al., 2024b) techniques to the model up- dates to reduce communication overhead. Notably, MUD is a form of gradient compression. By representing the model update as sub-matrices, our approach achieves sig- nificantly higher accuracy compared to existing low-rank decomposition methods. Additionally, experimental results demonstrate that method outperforms the gradient com- pression methods based on pruning and quantization.
Impact Statement
In this paper, we improve low-rank decomposition tech- niques in communication-efficieet federated learning. The proposed method focuses solely on improving the commu- nication efficiency. Therefore, we believe that our method does not result in any adverse social impact.
References
Bibikar, S., Vikalo, H., Wang, Z., and Chen, X. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
Bottou, L. Large-scale machine learning with stochastic gradient descent. In International Conference on Compu- tational Statistics (COMPSTAT), pp. 177-186. Physica- Verlag, 2010.
Bouacida, N., Hou, J., Zang, H., and Liu, X. Adaptive federated dropout: Improving communication efficiency and generalization for federated learning. In 2021 IEEE Conference on Computer Communications Workshops, INFOCOM Workshops, pp. 1-6. IEEE, 2021.
Caldas, S., Konečný. J., McMahan, H. B., and Talwalkar, A. Expanding the reach of federated learning by reducing client resource requirements. CoRR, abs/18 12.0010..07.. 2018.
7. Conclusion
In this paper, we provide a comprehensive study on low-rank decomposition techniques in the context of communication- efficient federated learning (CEFL). Specifically, we focus on three key challenges related to decomposition in FL: what to decompose, how to decompose, and how to aggre- gate. In response to these challenges, we propose three novel techniques: Model Update Decomposition, Block- wise Kronecker Decomposition, and Aggregation-Aware Decomposition. These techniques have been theoretically analyzed and empirically validated, and the results demon- strate that our approach significantly improves the perfor- mance of low-rank decomposition in CEFL.
Chen, M., Li, X., and Zhao, T. On generalization bounds of family of recurrent neural networks. In The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Dadras, A., Stich, S. U., and Yurtsever, A. Personal- ized federated learning via low-rank matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS) Workshop OPT, 2024.
Dorfman, R., Vargaftik, S., Ben-Ieenakk Y., and Levy, K. Y. Docofl: Downlink compression for cross-device feder-
9

---

The Panaccas for Improving Low-Rank Decomposition in Communication- Federated Learning
ated learning. In International Conference on Machine Learning (ICML), 2023.
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. TT, and Bacon, D. Federated learning: Strate- gies for improving communication efficiency. CoRR, 2016.
Edalati, A., Tahaei, M. S., Kobyzev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. Krona: Parameter efficient tuning with kronecker adapter. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2023.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 315-323, 2011.
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge, 2015. Stanford CS231n Course Project.
Li, Q. Diao, Y., Chen, Q., and He, B. Federated learning non-iid data silos: An experimental study. In Interna- tional Conference on Data Engineering (ICDE), 2022.
Gruntkowska, K., Tyurin, A., and Richtárik, P. EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compres- sion. In International Conference on Machine Learning (ICML), 2023.
Li, S.,, Cheng, Y., Wang, H., Tang, X., Xu, S., Luo, W., Li, Y., Liu, D., He, X., and Li, R. Masked random noise for communication efficient federated learning. In ACM International Conference on Multimedia (ACM MM), 2024a.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024.
Li, S., Xu, W. Wang, H., Tang, X., Qi. Y., Xu, S., Luo, W., Li, Y., He, X., and Li, R. Fedbat: Communication- efficient federated learning via learnable binarization. In International Conference on Machine Learning (ICML.) 2024b.
He, K., Zhang. X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 770-778. IEEE Computer Society, 2016.
Li, S., Luo, X., Tang, X., Wang, H., Chen, H., Luo, W., Li, Y., He, X., and Li, R. Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics. In International Conference on Machine Learning (ICML), 2025a.
Hönig, R., Zhao, Y., and Mullins, R. D. Dadaquant: Doubly- feder-
adaptive quantization for communication-efiin ated learning. In International Conference on Machine Learning (ICMI., 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Li, S.,, Luo, X., Wang, H., Tang, X., Cui, Z.,, Liu, D., Li, Y., xiuqiang He, and Li, R. Bora: Towards more expressive low-rank adaptation with block diversity. CoRR, 2025b.
Li, X., Huang, K., Yang. W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (ICLR), 2020.
Hyeon-Wooe N., Ye-Bin, M., and Oh, T. Fedpara: Low-rank federated
hadamard product for communication-efficin learning. In International Conference on Learning Rep- resentations (ICLR), 2022.
Lialin, V., Muckatira, S., Shivagunde, N., and Rumshisky, A. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations (ICLR), 2024.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerat- ing deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Liu, J., Zhou, Y., WU, D., Hu, M., Guizani, M., and Sheng, Q. Z. Fedlmt: Tackling system heterogeneity of federated learning via low-rank model training with theoretical guarantees. In International Conference on Machine Learning (ICML). 2024.
Isik, B., Pase, F., Gündüi, D., Weissman, T., and Zorzi, M. Sparse random networks for communicatoon-ffiiienn fed- erated learning. In International Conference on Learning Representations (ICLR), 2023.
Malik, O. A. and Becker, S. Low-rank tucker decomposition of large tensors using tensorsketch. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Karimireddy, S. P., Rebjock, Q. Stich, S. U., and Jaggi, M. Error feedback fixes signsgd and other gradient compres- sion schemes. In International Conference on Machine Learning (ICMI), 2019.
Mao, Y., Ge, Y., Fan, Y. Xu, W., Mi, Y., Hu, Z., and Gao, Y. A survey on lora of large language models. CoRR, 2024.
10

---

The Panaccas for Improving Low-Rank Decomposition in Communication- Federated Learning
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-effiieent learning of deep networks from decentralized data. In International Con- ference on Artificial Intelligence and Statistics (AISTATS), 2017.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, 2017.
Yang, Y., Zhang, Z., and Yang. Q. Communication-effiiennt federated learning with binary neural networks. IEEE J. Sel. Areas Commun., 39 (9 2):3336--8888 2021.
Mei, Y., Guo, P., Zhou, M., and Patel, V. Resource-adaptive federated learning with all-in-one neural composition. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yao, D., Pan, W., Wan, Y., Jin, H., and Sun, L. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization. CoRR, 2021.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2011.
Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystify- ing why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence (AAAI), 2019.
Novikov, A., Izmailov, P., Khrulkov, V., Figurnov, M., and Oseledets, V. Tensor train decomposition on tensorflow (T3F). CORR, 2018.
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K. H., Hoang, N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning (ICML), 2019.
Qiu, X., Fernández- Fernández-Marqués, J., de Gusmao, P. P. B., Gao, Y., Parcollet, T., and Lane, N. D. Zerofl: Efficient on- device training for federated learning with local sparsity. In International Conference on Learning Representations (ICLR), 2022.
Zou, D., Long, P. M., and Gu, Q. On the global conver- gence of training deep linear resnets. In International Conference on Learning Representations (ICLR), 2020.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, Fedpaq: A communication-effichte federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Sainath, N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP), 2013.
Stripelis, D., Gupta, U., Ver Steeg, G., and Ambite, J. L. Federated progressive sparsification (purge-merge merge-tune)+. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2022.
Sun, Y., Li, Z., Li, Y., and Ding, B. Improving lora in federated learning. In International Conference on Learning Representations (ICLR), 2024.
Tang, H., Yu, C., Lian, X., Zhang. T., and Liu, J. Dou- blesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In Interna- tional Conference on Machine Learning (ICML), 2019. 2019.
Wu, X., Liu, X., Niu, J., Wang, H., Tang, S., Zhu, G., and Su, H. Decoupling general and personalized knowledge in federated learning via additive and low-rank decompo- sition. In ACM International Conference on Multimedia (ACM MM), 2024.

---

The Panaccas for Improving Low-Rank Decomposition in Communication Federated Learning
A. Additional Experimental Results
In this section, we present convergence curves of different methods under various data distributions, as illustrated in Figure 6. Then, we evaluate the performance of FedMUD under the Non-IID-2 data distribution, considering various reset intervals, initialization values, and compression ratios. The experimental results, presented in Figure align with the findings in Section 5. Table 4 summarizes the results of ablation experiments decoupling and freezing, discussed in Section 5.3, under Non-IID-2 and IID data distributions. Additionally, provide the experimental results for all methods under the IID data distribution, as shown in Table 3. Moreover, we evaluate our method against FedAvg and FedLMT using ResNet18 (He et al., 2016) on the CIFAR-I0 and TinylmageNet (Le & Yang, 2015) datasets under the Non-IID-1 setting, The compression ratio is set to 16x or 32x, and the learning rate is tuned over {0.001, 0.003, 0.01, 0.03, 0.1}. All other experimental settings follow those described Section 5.1. The results, summarized in Table show that our method retains superior performance even with 32x communication compression, achieving accuracy closest to FedAvg.
F-MNIST with Non-lID-1
SVHN Non-IID-1
CIFAR-10 Non-IID-1
CIFAR-100 with Non-IID-1
Accuracy
Accuracy
Accurscy
Accuracy
330
181
15
100
1ta
3/5
206
19
300
125
20
Communication
Communication
Communication rounds
reddwg
TeoMUD+AAD
F-MNIST with Non-lID-2
SVHN Non-IID-2
CIFAR-10 Non-IID-2
CIFAR-100 with Non-IID-2
Accuracy
Accuracy
Accuracy
Sccuracy
12h
Communication
Communication
Communication
F-MNIST IID
SVHN with
CIFAR-10 with
CIFAR-100 with
Accuros
Accuracy
Accuracy
Accuracy
730
90
189
23
222
125
136
206
19
30
200
125
200
Communication rounds
Cammunication rounds
Communicction rounds
Communication rounds
TedAwg
TedMUD-AAD
TedMUD+6<C
DeSMUD-3KD-AAD
Figure 6. Convergence of different methods under Non-IID-1, Non-IID-2 IID-2 and IID data distribution.
12

---

The Panaccas for Improving Low-Rank Decomposition in Communication--Piiiice Federated Learning
F-MNIST with Non-IID-2
SVHN with Non-IID-2
CIFAR-10 with Non-lID-2
CIFAR-100 with Non-lID-2
116
12
Accursty
rccurery
721
24.
04.27
PedMUD
PedMUE+3KD
--- PEMMUC+AAD
FedMUD+ BKD
MNIST with Non IID
SVHN with Non IID 2
CIFAR 10 with Non IID
CIFAR 100 with Non IID
00
Accurpe
15.3
Actureey
Accuracy
Accuracy
Intiaaization
Inttalization
alizatztnnn
FodMUC
PedMUIC-500
--- FAdMUECAAAD
RedMUDD BKD BKD
F-MNIST with Non-IID-2
SVHN with Non-IID-2
CIFAR-10 with Non-lID-2
CIFAR-100 with Non-lID-2
Accuraty
Accurary
11.
16.
A/u
Compression Retio
Compression Reblic
Componesion Ratio
Ratio
FedMUc
FedMUc++30D
PEAMUC+AAD
BKDMUDDBKD-A4D
Figure Ablation on hyperparameters for the proposed methods under Non-IID-2 and data distribution.
Table 3. Accuracy of different methods under the IID data distribution.
FMNIST (IID)
SVHN (IID)
CIFAR-10 (IID)
CIFAR-100 (IID)
FedAvg (McMahan et al., 2017)
91.9(1 0.1)
92.3(1 0.1)
87.4(= 0.2)
59.1(= 0.4)
FedHM (Yao et al., 2021)
89.1(t 0.3)
79.9( 0.9)
76.4(= 0.7)
35.9(= 0.8)
FedLMT (Liu et al., 2024)
89.1(E 0.1)
79.6(1 0.7)
75.9(= 0.7)
34.6(= 0.3)
FedPara (Hyeon- Woo al., 2022)
89.2(1 0.2)
86.1 0.2)
76.7(= 0.6)
37.8(= 0.5)
EF21-P (Gruntkowska al., 2023)
87.9(t: 0.3)
87.9( 0.2)
64.4(= 0.4)
28.1(= 04.)
FedBAT (Li et al., 2024b)
90.5(1 0.1)
89.3(1 0.0)
82.8(= 0.1)
47.3(= 0.5)
FedMUD
90.0( 0.2)
87.3(+ 0.1)
79.2(= 0.3)
43.5(= 0.3)
FedMUD+AAD
90.9(:t 0.1)
89.1 0.2)
83.0(= 0.1)
50.5(= 0.1)
FedMUD+BKD
90.5(t 0.2)
88,5(t 0.1)
79.0( - 0.2)
45.7(= 0.4)
FedMUD+BKD+AAD
91.0(ct: 0.3)
89.5(:t: 0.1)
83.2(= 0.3)
51.4( 0.3)
13

---

The Panaccas for Improving Low-Rank Decomposition in Communication- Federated Learning
Table Accuracy of FedMUD when applied with freezing (+F) and decoupling (+ADID) under the Non-IID-2 and ITD data distribution.
FMNIST
SVHN
CIFAR-10
CIFAR- 100
Non-IID-2
Non-III--2
Non-III>-2
Non-III>-2
III
FedMUD+F
87.1
90.9
80.2
87.4
67.5
82.2
33.0
48.4
AeA
87.0
90.9
82.5
89.1
72.3
83.0
35.1
50.5
FedMUD-BKD4F
87.4
90.9
83.3
88.5
71.2
81.9
33.3
48.7
FodMUD-BDD-AAD
87.6
91.0
84,9
89.5
73,9
83.2
36.1
51.4
Table 5. Accuracy of R on CIFAR-10 and TinylmageNet.
16x Compression
32x Compression
Model Dataset
FedAvg
FedLMT
FedMUD-BKD-AAD
FodLMT
FdMMUD+BDD+AAD
ResNet18 CIFAR-10
84.1
76.4
83.2
73.9
79.6
R on TinyImageNet
40.0
31.3
36.9
13.8
33.4
B. Discussion about the Upper Rank Bound of BKD
In this section, we briefly examine the upper bound on the rank of BKD by leveraging properties of the Kronecker product and matrix concatenation. Specifically, we use the identity rank(A & B) = runk(A) rank(B) and the inequality B)) I rank(A) + rank(B), Let A, B E Raxb, then their Kronecker product yields a matrix W E _ xt² According to the rank property of the Kronecker product, we have rank(W) = rank(A) rrak( which can be as large as min a², 1 (i.e., the maximum possible rank for matrix of size a² 1P). Since BKD can be interpreted as the concatenation of several such Kronecker product matrices, and each sub-matrix can individually attain full rank, the overall recovery matrix of BKD is capable of achieving full rank.
C. Proof of Theorem
In this section, we first present the problem formulation, then outline the assumptions and lemmas necessary for the proof, and conclude with a detailed proof process. Our theoretical analysis relies mainly on FedLMT (Liu et al., 2024).
C.1. Problem Formulation
Let us consider N clients with local datasets D = =(D,, D2...., DN The goal of FL is to solve the following problem
N
min f(w) A f((w).
(11)
where w = (W1, W } denotes the parameters of L layers. fi(w) A Eg,en, (F(W,.)) is the expected loss function of client and & is a random data sample of client i. In this paper, we propose freezing the original model parameters and learning low-rank matrices the model updates. Assuming that the first p layers remain unchanged and low-rank model updates only applied to the subsequent layers, the entire set of model parameters can be expressed
x = (WI...., Wp W Wpt1, Up+i, Vot1...., WL, Uz, Vz):
(12)
where (W,-1, Wp+2;-.. .,., Wr remains frozen during gradient descent and will only be updated when the model updates U(V) are manually added. Each pair of U/ Rxr and Vi Raxr denotes the low-rank updates of the corresponding frozen parameters W, Rmxxn Throughout the entire training process, the recovered parameters can be expressed as
W = (Wi... WG, W + Up- UV W. + T..
(13)
Let h(-) denote the local objective function for training with low-rank model updates, the goal of FL with FedMUD becomes
N
min h(x)
A
hx(x).
(14)
N
14

---

The Panaccas for Improving Low-Rank Decomposition in Ceamuunication-Efiiiiee Federated Learning
where (x) Eg.eD; (x, [r) is the excepted loss function of client with low-rank model updates.
Let x denote the model parameters of client at iteration & and E < 'T' denote the number of iterations between successive aggregations, we have
x x - - VH, .5 - V .f)
if mod E 0, otherwise,
(15)
Next, we define S A nE' to be the number of iterations between consecutive resets of low-rank updates. Specifically, for every rounds S iterations, low-rank model updates U T will be manually added to frozen parameters. Subsequently, U is reinitialized randomly and V is reinitialized to zero, ensuring that the values of the recovered model parameters w remain unchanged before and after the reset. In particular, the initialized value of U must remain consistent between different clients, which can be ensured by passing a random seed from the server to clients. This reset process is defined as
reset (pdates(x)) if mod 5 = 0, otherwise.
x
(16)
Table 6. Commonly used notations and descriptions.
Notation
Description
7
The learning rate.
i
N
The index and total number of clients.
The index and total number of training iterations.
S
The number of iterations for model aggregation and resetting model updates. The iteration indices of last model aggregation and resetting model updates. The index and total number of the layers within neural network.
Le
P
The first p layers of the network remain the same without applying low-rank model updates. The model parameters and recovered model parameters defined in (q. and (q.() The average results of x and w of all clients as defined in (1.( and (1.)
X
X
W
|1-11m
11-112
Frobenius norm of a matrix and la norm of a vector.
h4
H,
The objective function and its expectation of client i, defined in Eq. (14), taking x as their input. The objective function and its expectation of client i, defined in Eq. (11), taking w as their input.
fa
F.
For the sake of convenience in subsequent analysis, we give the following definitions.
N
N
N V
W
W
0
U
N i=1
N
N
i=1
t=1
(17)
x
( ..... Wt. Wt
Dt
V
Wi, U, Vi).
p+ly p+l
p-l.....
Further, can get the recovered parameters as follows.
w' = (W...., W W p-1 + D IT
Wt + +
(18)
p
According to the definition of x, we have the following property for x.
1 I1 NN VH1((
x x'=1
(19)
Since w is recovered from x and w recovered from x, we have Fr(w, E) = H2 E) and F,(wg)) = H2 E). Next, we discuss the relationship between the derivatives of F(.) and H-)). We use the subseript to indicate the trainable parameters and corresponding gradients of layer I. For each layer S p. we have (w {f) (x,, {t). since
= =:= wir = Wit,- For each layer > p. V-(-) and F.(-) satisfy
HH,( d+1) =
(20)
(F.( st UTU
15

---

The Panaccas for Improving Low-Rank Decomposition in Communication- Federated Learning
Note that we will omit in F(-) and H()) for ease of writing in the following analysis. Considering that h2(x) [E[H((x)]
and Vh(x) = N Σ 1= (x,(x), we have
V fu (w)(w (w))
Vii (w Vi (F U.
W
Vh V ft, (w)) 1.
VHI((*
Vhi(x*)
(21)
V.((ww)) [TO
The Frobenius norm of (-) and (.))) has the following relationship
I }| = [F. > I| (W UU 1
(22)
[x }1 = V ft. VV F Vf..( UU 11
Finally, the whole gradient of x and w can be represented as
+) = (V (x
Vh (x = (VH, VF((W--))=
VH 1x;
(23)
V/((w!-)) = (V (w: -1) Vl })
- (VP. (w(-t), ).... VF "};
where (h, (H(( Vf(Ww and (w()) are flattened vectors, and we have
(x 112 (hu |r
< Σ
1=1
(24)
Σ
F (w: 111 Σ
In the following. we demonstrate that the sequence of recovered model parameters {w', ww? w w obtained by FedMUD, converges to a local stationary point in the optimization space of under non-convex and smooth assumptions. The commonly used notations are summarized in Table where fe and t, are used to represent the iteration indices of last aggregation and last resetting model updates. Thus, have 0 s - te E and 0 S t S.
C.2. Assumptions
Assumption 1. The loss functions f((-) and h4(-) are differentiable and L-smooth, with a constant L..
L.g 2
f((w)) S f:(22) + (W1 - W2, V f((w)))
+
|[1 -22l2
Vi, Vw1, VW2,
(25)
+ L
h₂(x1)= S (x22 + (x1-XX - X2. (x2())))
|x| x2 Vi, Vxi, VX2.
2
Assumption 2. The stochastic gradient FF: and (Hi(( Et) are unbiased, with bounded variance and norm.
(ww, + = Vi((w)) EEED, (H, { (x,(,,) Vi,
(26)
B [V V/(( 12 1 o Eg,D, [ , VI,
Assumption 3. The Frobenius norm of matrices U and V is bounded during both initialization and training.
|| < 2
I[ and ||0% 11 s E
VL
(27)
Assumption 4. At least one of the matrices in (Ūi, Vi) has smallest singular value greater than zero.
[6min (isinn(U)))) (U1))² + [6min (V,))2 2 v/2 O, v1,4
(28)
where - - I Vi, and (.) return the smallest singular value of a matrix.
Note that Assumption 3 is reasonable and commonly adopted in convergence analysis, discussed in Section 4. In FedMUD, the low-rank matrices U and V undergo local gradient descent updates for finite number of steps (r). Following this, they are merged into the base model parameters and subsequently reinitialized. This periodic reinitialization mechanism prevents the unbounded growth of U and V. To illustrate, for any matrix U, its value after t steps (t S 7) is U, = Un + 17 D 9.. Given |[ S G, its squared norm is bounded by ||U,|| I (r + 1) [| + G Therefore. the combination of finite local training and periodic reinitialization ensures that and V remain bounded, validating our assumption.
16

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Ifiiion Federated Learning
C.3. Lemmas
Before presenting the lemmas, we first introduce an inequality as follows
Σ
S
(29)
which holds for any vector and any positive integer This inequality also applies to the Frobenius norm of matrices and can be seen as special case of the Minkowski inequality. It will be used frequently in the subsequent analysis. For convenience, we will refer to it as the Minkowski inequality in the following discussion, without further elaboration.
Lemma 1. Under Assumption 4, at each iteration and for each layer p. it follows that
Vht(x')) I1 2 the [V
(30)
Proof of Lemma 1.
According to (2)( for each layer > we have
Vh, || = [f((w')))))))) V + | ((f((w")) TU
((()) Vf((w') ) 11 + [[())) VVi(("))l'
W9 N9 (6 (V/)) + min (Ut)) /f((w))
(31)
Vfi(w") F
where (a) follows from the inequality Smin (U)
F S |UVIl for any matrix U E Raxr and matrix E Rx*x, The proof of this inequality can be found in Lemma B.3 in (Zou et al., 2020). (b) follows from Eq. in Assumption
|
Lemma 2. Under Assumptions 2 and 3, each iteration t and every client and each layer > p. it follows that
E [| - r A (2e 2n S G2 ki: nal.
E [[Ival
< <
min(2e? + 2n G2 kgy meg)
(32)
Proof of Lemma 2. Based on the relationship between t and f., we consider two cases. First, when t = t., the matrix U | S = s Next, for the case where
reinitialized.. In this case, by Assumption 3, we have | t ta. the update of U after initialization can be divided into aggregated gradients and unaggregated gradients as follows
te Σ T=t,
U U3 -1 1
[Eyi(W V n Σ VFr(W
(33)
Thus, we have
E
||
(a)
1.
S 2E
U 1,1
+ 172
Σ
V
F2 -
Σ
(fu Va
F
r=t++1
j=l
r=++1
(8)
5 23
2
-2n2(t
-t,DE
Σ
(F
V.,((W] "
F
r=
7=t++1
(34)
(0)
22
2
2n72 -t,)E
+ Σ
VF,(WW -
+=
71
(2)
2
52E
21 G22
F
(c)
22
+ 2 52
17

---

The Panaccas for Improving Low-Rank Decomposition in Communication--Piiceenn Federttt Learning
where (a), (b) and (c) result from applying the Minkowski inequality in Eq. with 2, la and N, respectively, (d) follows from the basic inequality || PQl < ||P|I ||Q1 Eq.(26) in Assumption and Eq.(27) in Assumption 3. (e) follows from the inequality (t - (e) S and (27) in Assumption 3.
Furthermore, by (2,) we have E
U
S min + 2r m r Considering the symmetry of the matrices
| S min + 21 Kur )-)
U and V. we can repeat the above process and have IE
Lemma 3. Under Assumptions I and 2, each iteration t and every client i and each layer > p. it that
(35)
Proof of Lemma 3. Similar to Lemma 2, we analyze two cases based the relationship between and le. First, for the case x
1 = 0 < 4n 4T
I t= fe. all model parameters are aggregated. Thus, we have x = x and E
|
Next, for the case > t, both x and x' can be represented using x as follows
N VHa.(xI-1)
x
VH.((x]"))
= T=
(36)
N
+=
Thus, we have
N N
E
||x
x2,111
= E
( - Σ
T
i=1
rmtut1
(a)
S 21 tz)
(x
I
r=
(37)
(6)
N
S 21 L.,
Σ
i=1 r=
r=l++1
(c) - 4t7(1 11
||
1 + VRu((
Fr
@ 4n/ 4
where (a) and (b) result from applying the Minkowski inequality in (29) with 2(t -.)) and N, respectively. (c) follows from (22) (d) follows from inequality |PQII S I| |1Q11 Assumption and in Lemma 2. Lemma 4. Under Assumptions 1, 2 and 3, at each iteration é and for every client i, it follows that
|| w:
< e 4pm G2 + 12(L - PO(L-p)n E2 (T + +2
(38)
Proof of Lemma 4. Similar to Lemma 3, we analyze two cases based on the relationship between i and f.. First, for the case = fe, w = w and thus E ||w w(ll = 0 < A. Next, for the case t > le, we can divide the model parameters
into full and low-rank components and compute E I
will -w:l-wlll as follows
|| w
will
= E
Wi Will + Σ
-
(39)
1=+11
W 1 have
For the first consider the term Ei
Wi
Wh, = W/- 7 Σ VF.(WT-1)
W/
W 7
Σ
VF
(W 1)
(40)
N
+=1++1
+=
j=l
18

---

The Panaccas for Improving Low-Rank Decomposition in Communication--ficcioe Federated Learning
Thus, have
P
N
E
WE
= E
(w
N
[=1
+=
j=1
T=
2
N
21
(t te)
Σ
E
VF J. (wJ 1)
Fi W
1=1
+=
(41)
(6) P
Σ
(t- - -(e) Σ E
(W
N
1=1
1=++1
1=1
(t-(te))C
- G22 S 4mm
where (a) and (b) follow from the Minkowski inequality. (c) follows from Assumption 2. (d) follows from - te < E.
For the second term E
Σ El=p+1
T UV, T we have
Ui11 = 01:- -
VP. - 1
[W -1)
T=te+1
T= +1
j=1
(42)
V V/ 1
VF.(wW-1) UFT
V 1
7=te+1
j=1
Thus, we have
1 N
(((P)) UL.VE)
[W T- V
Σ VF,(W, -V
(V/T)T
=-
N
+=
n
Σ (
Σ
r=le+1
r=
B1
(43)
N
N
+r²²
V [W V
(Wz
N
r=
gml
jml
VPu(W -\
Σ
I V
TV. (..(w?-))
+1
T +1
Next, we try to find the upper bound of E
||Ai E
[|B
[3]
and E [.,] 2.]. respectively. E [|c
19

---

The Panaccas for Improving Low-Rank Decomposition in Communication--Pficcinn Federated Learning
E
|
EE
Σ
VF (F
VFi W
(V/t0)
N
r=
+=4+1
(4)
N
S B
VFza(WJ
r=t++1
==1
T=
(6)
- 21) 2 - te)
Σ
E
N
VF 1))
+
VFLl(W; ") 2
(44)
N
s=be11
j=1
F
(0)
I 21 - te)
Σ E
+ (W
VV 2
+1
(4) Σ 4n - t.)) t.. 47²² E
01
where (a) and (d) follow from the inequality |PQII S IIPI 11Q1 (b) and (c) result from applying the Minkowski
inequality in (29) with n 2(t te) and N, respectively. Note that A and Bj are symmetric, and similarly, we can deduce E
|B
2 W 4n E G Next, for Ct, we have
N VF (w
N
E
= ' [|
Σ
Σ r=ta+1
1
Σ
N
N
1=1
(a)
N
"
Fiz(
j.1
(U)H
V F3J
N j=1
N
1=1
11 to)2 N2
E
N
V Fy 1リ
(U]) )) T V FF
1
(45)
r=le+1ja 1=
T=t++1
(1 - 1o)²
N
N Σ Σ
E
Σ
(w 112
||
, 1)
(w5
N2
r-let 1=1
+
(8) (1) t2)²
N
N Σ
E
GT
Σ
G u
= -
N2
j=1
T=
j=1
where (a) and (c) follow from the inequality |PQ]| < |||| 1|Q1 (b) results from applying the Minkowski inequality
with n N(L - Le). (d) follows from Assumption 2 and in Lemma 2. Similarly to E [|0|]]]]] we can obtain the same
bound for Dj, which is E [ S Combining the result of A,, Bi, Ct and D,, we have
E Σ
( - -Z, 1' E
|| + B + C. DI12
1=pt1
(=p+1
E
| +
||B +1C Dil
33 Σ
E |1 + ||B.11 + 2 +2||D
(46)
1mp+1
1=p+1
3
G
+ A = -20 -2L + +
1=p+1
Finally, adding (41 and (46 to (40) yields the result of Lemma
20

---

The Panaccas for Improving Low-Rank Decomposition in Communication-IFiiionnn Federated Learning
Lemma S. Under Assumptions 1, 2 and 3, each iteration it follows that
27
1/02
E W/ 1
+3(I-
+ ( (47)
Proof of Lemma 5.
Similar to Lemma 4, we divide the model parameters into full and low-rank components as follows
w' =E
Σ wi
-
11
+
-
||
(48)
[-p+1
For the first term, we have
P
N Σ ==1
W
w--112
-En 1 2
V
(w
N
(a)
B
- - 1)
Sur(w! (-1)
1=1
1=1
(49)
E
+
2
Σ
fü.
N2
N
1200 (8)
N
N
where both (a) and (c) follow from Assumption 2. (a) and (c) leverage the unbiased and bounded variance properties of the gradient, respectively.
For the second term, we have
1 N N
N of (U
-f =
V (w 11
(V)
1
(VF,((
11 N
A
(50)
TVR.
N
N
Next, we try to find the upper bound of
|
E.
[1 and [Ic.12]]
respectively.
E
(V/-1))
N
N
$ 1
VF,(( | (V. (VT-l)
(51)
N
- 11
| 1V || S
E
N
21

---

The Panaccas for Improving Low-Rank Decomposition in Communication-PPiiccnnn Federated Learning
Note that A and Bj are symmetric, and similarly, we can deduce E
[
G Then, for Ct, we have
(w. 2 F.
E
|
'
1
VF.( W
'
V
(w
EF
-
T
(52)
2
(w -
Sn
S
N
Combining the result of A,, Bj and C₁, have
E
Σ
-0
$3
Σ 1=p+1
E | + I + |IG||
1=p+1
(53)
33
Σ
+ = 3(L G2 + (
Finally, adding (4.) and Eq.(53) to (48) yields the result of Lemma
Lemma 6. Under Assumptions 1, 2, 3 and at each iteration follows that
E
[V](W
w² w²
N fi.(w [1)
||
+ Dー
+ LLA
2
2
(54)
1=1
1=1
1=1
1/2
E
+ - CC
1 [ GT
2
1=p+1
Proof of Lemma 6. Similarly, we divide the model parameters into full and low-rank parameters as follows
E [VS(WW w ww
E
V(V vWW - -1--1
(55)
+ E
V(Vfi(W'-))
-
1=p+1
22

---

The Panaccas for Improving Low-Rank Decomposition in Communication-EPiiiennn Federated Learning
where V(-) denotes the operator of converting a matrix to column vector. Next, for the first term, we have
E
V(V
(V[i(("'')))
v(Wi
w N 1(
P
Σ
E
V(V w
N Σ
(W (w
1)
V(Vfi(
1) v( N
V (w 11
1=1
E
V - V
2
(56)
E
V(V - v
1=1
E
| ||| +
v
2
N
1=1
N
+
E
V
(w
N
1=1
27
(d)
P
+
(w
+ L.A
2
2
1=1
1=1
where (a) follows from f((w) A E [F (w, {. (b) follows from -a T b = a - 1 - ||a||
-|bb1/ 2] for two
column vectors a and (c) follows from [V( 2 || for any matrix X. (d) follows from (5.)
N (-1
Vfi(w
1=1
E
(w
N
[Va ( p
E
(57)
E
V - (
Σ
-
(4)
$
L.A.
where (a) follows from the Minkowski inequality. (b) follows from p S L. (c) follows from the smoothness of f. (d) follows from the result of Lemma
23

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Ifiiion Federated Learning
For the second term, have
E
V(VSi(W!-1))
-( - (V,-()))))
(=pt1
N 1)) V( 1
Σ
B
V(V
VF,, (V
N
1=p+1
L
+
(-n)
V(Vf
1=p+1
V(VVI(
1=p+1
(58)
(-n) E
V(VSi(W 1!
1=pt1
(-n)
V(
Imp+1
B;
}
n E V(VII( V
V (w Wi
N
1=p+1
where (a) follows from fi(w) A EGGED. (F.w, Ex)] and a T(cc) = (abT) To for column vectors a, b and c.
N Vinl(
- < E \/
E
VVi(
N
(59)
F
Av.1
Ar,
-B < E
(1(0
V 11
T
Σ
N
(60)
1
B
Further, we have
(a)
Ar1 + B1,1
-
Vhi
V
2
2
Ar.2 + B1.2
2
1
1
Σ
V
E
h (x* (x'-1)
Vhe (x} -1]]
N
(61)
(8)
1
||
111
2
N
E
x
x
2N
2N
i=1
<
where (a) follows from Lemma (b) follows from the Minkowski inequality. (c) follows from the smoothness of (d)
24

---

The Panaccas for Improving Low-Rank Decomposition in Communication-Elicion Federated Learning
follows from Lemma 3. Next, for Cr, we have
N (
Ci $ R
+
U
VF
N
N
F.
(62)
|| + G
E
Combining the results of A,, Bj and Ct. we have
E
V(V -VVVV" - (V(-))))
1=++1
H (~Ar Bi) + Σ r²Ci
1mpt1
1mp+1
(63)
L
I.
TA1,11 + Bi,11 + A1,2 B1,2) +
Σ
12C
1=p+1
1=p+1
n²² -
V ||
+1))+++)))0 1 (b (L--pn
E
1=p+1
Finally, adding (56( and (63) (55) yields the result of Lemma
C.4. Proof of Theorem
Theorem
Under Assumptions 1, 3 and 4, let < < be constant and the learning rate satisfy < n S
2
min
1}, have
T
2 [1]] S TT
(/(( - (
- (L- + (L- p) + rt] (64)
where w is the optimal parameters, 1 è (in + 20 neg x²) and 1 min + 20 x3)
Proof of Theorem According to Assumption 1, we have
+ Lg E || will
E VV((") < E [Vf((' = (VF((( w w'11]] -
(65)
2
Using the results of Lemmas 5 and 6, we have
[V[(")) < E [V((w=1)) ')]
7 E
Σ
n
up
E
(
2
t=1
[mp+1
A
- E
fun(w!-1)
+
0
Lan?
1
(1)
N
1=1
B.
11 LE + 2(L (1
+ (
Lonooz
+
3L
(L
+
+ ,
2N
2
(66)
25

---

The Panaccas for Improving Low-Rank Decomposition in Communicationn-Pficienn Federated Learning
For As, Be, we have
(a)
P
-If
1f
A
E
V
E
fi
2
!mpt1
Σ
of
E
1|
2n
E
f
V
2
1=1
1=p+1
(67)
(8)
Σ
R
||
||
V E
V
1(
w
1| ||
2
2
1=1
B₁
L
-
E
V
fi.((
0,
1=1
where (a) follows from 0 < S and c thus -1 -1f and 1/2 rf. (b) follows from 17 ( (2
and thus
2n mp S O. (c) follows from 17 S I. and thus 2 - 17 S 0. Therefore, we have
} + " 2
|/( +C
(68)
Rearranging the above inequality, summing over E (1, T} and then dividing both sides by 2// yields
[[f]] S
+ -
2 TpT - (((w))
20
op S
2 (S(ww) - ((ww)) + np
2C
(69)
T
nPT
Next, rearranging the terms in C' yields
C 2LL G2 1.,02
2N mp
+ (L 2)21 E ( +12)
(70)
+ (L - ()6, 2 Lon + + i
3 I. G +
(L P)L,
2 Multiplying C by ofr and analyzing its complexity with respect to 1), we have
20
LO02 17 -²- (I=
-A
tfc
N
+ + - (12 + 3L, + +1)"" -1 (71)
0 (1 -
Finally, adding Eq.( to (69) yields the result in Theorem 1.
26


Reviews:
[{"review_id": "ckCCm3fcTU", "reviewer": "Reviewer_CShD", "summary": "This paper introduces three novel techniques for Communication Efficient Federated Learning (CEFL) based on low-rank matrix decomposition: Model Updates Decomposition, Block-wise Kronecker Decomposition, and Aggregation-Aware Decomposition, each of which are targetting a specific issue. First, to reduce information loss, a Model Updates Decomposition (FedMUD) approach is developed that  involves factorization of model updates rather than entire model parameters. Second, to maximize the rank of the recovered matrix, a Block-wise Kronecker Decomposition (BKD) approach is proposed that partitions the matrix into blocks and uses Kronecker product decomposition and the block structure enables dynamic compression. Finally, to minimize the effect of compression errors during model aggregation stage, the authors propose an Aggregation-Aware Decomposition (AAD) approach that works by decoupling the multiplication of trainable submatrices. These techniques are complementary and can be applied simultaneously to\nachieve optimal performance. The authors provide rigorous theoretical analysis of these methods and provide a comprehensive set of experiments to show that the combinations of the above approaches achieve superior accuracy and faster convergence than baseline methods like FedHM (Yao et al., 2021), FedLMT (Liu et al., 2024), FedPara (Hyeon-Woo et al., 2022), EF21-P (Gruntkowska et al., 2023) and FedBAT (Li et al., 2024b).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer CShD:**\n\nWe sincerely appreciate your valuable feedback. Below, we address each of your comments in detail. For additional experimental results, please refer to the anonymous link: ​**https://anonymous.4open.science/r/fedmud_rebuttal-962F**.\n\n***Q1: \"experimental evidence of the functionality of the proposed modules\"***\n\n**R1:** \nBased on your suggestion, we present experimental evidence supporting the functionality of MUD, BKD, and AAD below:\n\n1. **MUD Reduces Information Loss:** We apply post-training 'model update decomposition' and 'full-weight decomposition' to the locally trained model (of FedAvG) and compare their validation accuracy. Information loss is measured by the loss in validation accuracy. The experimental setup and results are presented in **Figure 9** (see link). As compression strength increases, the accuracy of the full parameter decomposition drops significantly, even falling below the accuracy achieved before local training, which indicates substantial information loss. In contrast, MUD exhibits minimal accuracy loss.\n\n2. **BKD Maximizes the Rank of the Recovered Matrix:** We compare the ranks of the model updates obtained by FedMUD and FedMUD+BKD on FMNIST, at the same communication cost. The results, shown in **Figure 10** (see link), indicate that the rank of FedMUD is significantly smaller than the full-rank level, and is constrained by the parameter $r$ derived from the compression ratio. In contrast, FedMUD+BKD, utilizing the Kronecker product, maintains nearly full-rank performance.\n\n3. **AAD Avoids Aggregation Errors:** We report the errors introduced by aggregating low-rank matrices using methods other than AAD (note that AAD introduces no aggregation error). The specific calculation method and settings are shown in **Figure 11** (see link). The results demonstrate that FedLMT, FedHM, FedMUD, and FedMUD+BKD all exhibit varying degrees of aggregation error. Moreover, it is evident that the aggregation error in the MUD and BKD is considerably smaller than that in the full-weight decomposition model.\n\n\n***Q2: \"which experiments do you vary $a\\in\\\\{0.01,0.05,0.1,0.5,1,5,10\\\\}$\"***\n\n**R2:** \nIn our experiments, we initialize the submatrix of low-rank methods using a uniform distribution, $U(-a,a)$, where $a$ controls the initialization size. In Section 5.1, we tune $a$ over the values ${0.01, 0.05, 0.1, 0.5, 1, 5, 10}$ for FedLMT, FedPara, and our method. Specifically, FedLMT uses $a=0.05$, FedDMU and FedPara use $a=0.5$, and FedDMU+BKD uses $a=5$. All other experiments follow this configuration, except for the ablation study of $a$ in Section 5.5 (Figure 5).\n\n\n***Q3: \"the curves in Fig. 4 are not lined up and the meaning of the x-axis is unclear\"***\n\n**R3:** \nIn our experiments, we initialized the submatrices of the low-rank methods using a uniform distribution, $U(-a,a)$. The x-axis in Figure 4 represents the parameter $a$, which controls the initialization amplitude. \n\nTo investigate the impact of initialization, we conducted detailed ablation studies around the optimal value of $a$ for each method. For FedMUD, the optimal value is $a=0.5$, and the ablation interval is $\\\\{0.1, 0.3, 0.5, 1.0, 3.0\\\\}$, and for FedMUD+BKD, the optimal value is $a=5$, and the ablation interval is $\\\\{1, 3, 5, 10, 30\\\\}$. \n\nAs suggested, we further aligned the curves in Figure 4 (refer to **Figure 12** in the above link). The experimental results still show that BKD requires larger values of $a$, which can be attributed to the differing variance effects of Kronecker products and matrix multiplications. Specifically, for $A \\in \\mathbb{R}^{m \\times r}$ and $B \\in \\mathbb{R}^{r \\times n}$, assuming both are i.i.d., we have $Var(AB) = r \\cdot Var(A) \\cdot Var(B)$ and $Var(A \\otimes B) = Var(A) \\cdot Var(B)$. Since $r$ affects and amplifies $Var(AB)$, BKD requires a larger initialization range to match the magnitudes. \n\nAdditionally, for the same reason, BKD is more robust to changes in $a$, as the variance is no longer influenced by $r$ ($r$ is determined by the compression ratio).\n\n\n***Q4: \"code reproducibility issues\"***\n\n**R4:** \nSorry for the reproducibility issues with our code. Our implementation builds upon the open-source work \"FedBAT\" (ICML 2024), ensuring replicability. We will update the codebase to include the necessary dependencies and conduct thorough reviews to ensure compatibility across different operating systems and devices.\n\n***Q5: \"typo in Eq.(43)\"***\n\n**R5:** \nThank you for your meticulous review. We will address the typos in the upcoming version.\n\n***Q6: \"One related reference not discussed.\"***\n\n**R6:** \nAlthough we covered low-rank decomposition in personalized federated learning in the related work section, we acknowledge the oversight of this valuable work. We will include a discussion of it in the revised version.", "claims_and_evidence": "The authors do a very comprehensive set of experiments that show speed of convergence with communication volume and training rounds of different combinations of FedMUD + BKD + AAD vs baselines as well as quantify the effect of compression ratios, reset intervals and initialization values on the performance of various combinations of FedMUD, BKD and AAD. The main claim in their paper is that their algorithms do a better job of handling information loss, maximizing the rank of the recovered matrices and reducing the effect of compression errors in the model aggregation stage. What the first set of experiments show is that the speed of convergence (Figure 1) is better with FedMUD + BKD + AAD and that the achieved test accuracy is better than the baselines (Table 1). They do not show that their proposed algorithms reduce information loss or maximize the rank of the recovered matrix or reduce compression errors at the aggregation stage *better than the competitors like FedLMT, FedHM and so on*. What the second set of experiments with compression ratios, reset intervals and initialization values show is that combination of FedMUD + BKD + AAD does best in achieving these goals.", "methods_and_evaluation_criteria": "The choice of benchmark datasets is good and make sense for the problem at hand. Their experiments are comprehensive. It is good to show that speed of convergence of different combinations of FedMUD + BKD + AAD is better than the baselines (Figure 1) and that the achieved test accuracy is better than the baselines (Table 1) across 4 benchmark datasets FMNIST, SVHN, CIFAR-10 and CIFAR-100. They also look at different data distributions (IID, non-IID-1 and non-IID-2) and different compressors: the Rand-K and Top-K compressors. These are the most important experiments. The comprehensive experimental evaluation is why I am inclined towards acceptance of the paper despite the shortcomings I present below.\n\nHowever, what is missing and really needed are experiments that quantify the effect of compression ratios, reset intervals (where applicable) and initialization values on the performance of various combinations of FedMUD, BKD and AAD *as well as baselines*. In general, we need more additional experiments that backup the claims of the paper: the proposed algorithms FedMUD, BKD and AAD together do a better job of handling information loss, maximizing the rank of the recovered matrices and reducing the effect of compression errors in the model aggregation stage *compared to the baseline approaches*. For instance, what is the experimental evidence that the proposed algorithms maximize the rank of the recovered matrices?", "theoretical_claims": "I tried to check the correctness of the proofs up to Eq. 43 in the appendix. I don't see any issues so far.", "experimental_designs_or_analyses": "I went through the description of the experimental setup and analysis of the experiments in the paper and examined the plots and tables to judge soundness/validity of the experiment. As the authors state, the initialization of sub-matrices in the decomposition methods can have a significant effect on the model performance and the submatrices were initialized with values drawn from a uniform distribution $U(-a, a)$ with $a \\in \\{0.01, 0.05, 0.1, 0.5, 1, 5, 10\\}$. What would be interesting is to see how varying this initialization value for elements of $U_l$ and $V_l$ affect the performance of FedMUD + BKD + AAD and other baselines. This is key because they are claiming that their methods are more resilient to performance differences due to differences in initialization. Figure 4 shows an ablation study based on this but the curves are not lined up so it is difficult to make a fair comparison between the different methods and the range of values on the x-axis are different from $a \\in \\{0.01, 0.05, 0.1, 0.5, 1, 5, 10\\}$. So it would be helpful to clarify what the numbers on the x-axis of Figure 4 mean.", "relation_to_broader_scientific_literature": "Low-rank decomposition (Sainath et al. 2013) is a method for parameter compression in federated learning and works by approximating a  matrix by the product of smaller sub-matrices. More specifically, the server sends a low-rank model tot he clients for training and subsequently receives the optimized low-rank model from them. FedHM (Yao et al. 2021) generate a low-rank model by using truncated SVD to the global model but the SVD algorithm introduces approximation errors causing deviations of the client models from the global model over time. To avoid this issue, FedLMT (Liu et al., 2024) directly trains a pre-decomposed global model, eliminating the need for SVD. FedPara (Hyeon-Woo et al., 2022) uses the Hadamard product to increase the rank of the recovered matrices. (Mei et al., 2022) further improves compression by sharing low-rank matrices across multiple layers. The FedMUD approach is an extension of FedLMT in the sense that it also trains a pre-decomposed global model and FedMUD reduces to FedLMT in the case where we have simply low rank matrix factor updates. The BKD approach uses the same concept as FedPara but with block structures to promote dynamic compression.", "essential_references_not_discussed": "One reference that has not been mentioned is this. This work also employs low-rank matrix factorization to do federated learning. It achieves a $\\mathcal{O}(1/T)$ rate for the full-gradient and $\\mathcal{O}(1/\\sqrt{T})$ rate for stochastic gradients. \n\nDadras, Ali, Sebastian U. Stich, and Alp Yurtsever. \"Personalized Federated Learning via Low-Rank Matrix Factorization.\" OPT 2024: Optimization for Machine Learning. 2024.", "other_strengths_and_weaknesses": "The main strength of the paper is the extensive experimental evaluation and three different proposed approaches to mitigate various issues associated with employing matrix factorization to train ML models in a federated setting.", "other_comments_or_suggestions": "1. For Equation 43, you have a missing ^ before the \"T\" so the subscript doesn't render.", "questions_for_authors": "1. In which experiments do you vary $a \\in \\{0.01, 0.05, 0.1, 0.5, 1, 5, 10\\}$?\n2. Can you clarify what the numbers on the x-axis of Figure 4 mean. Are they fixed values used to initialize the matrix elements or are they used as hyperparameters for the distribution from which initial values of the elements of $U_l$ and $V_l$ are set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "BBgLpc6sKw", "reviewer": "Reviewer_W27g", "summary": "This paper introduces three techniques to enhance low-rank decomposition for communication-efficient federated learning (CEFL): \n\n**Model Updates Decomposition** (MUD), **Block-wise Kronecker Decomposition** (BKD), and **Aggregate-Aware Decomposition** (AAD). Each method addresses specific challenges—what to decompose, how to decompose, and how to aggregate. Theoretical analysis is provided to prove convergence, and experiments across multiple datasets (FMNIST, SVHN, CIFAR-10, and CIFAR-100) are conducted, demonstrating superior accuracy compared to baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer W27g:**\n\nWe sincerely appreciate your valuable feedback. Below, we address each of your comments in detail. For additional experimental results, please refer to the anonymous link: ​**https://anonymous.4open.science/r/fedmud_rebuttal-962F**.\n\n***Q1: \"novelty of MUD and difference with LoRA\"***\n\n**R1:** \nPlease kindly refer to **R3** from reviewer **u9mP** for our response.\n\n***Q2: \"novelty of BKD and difference with EvoFed\"***\n\n**R2:** \nThere may be some misunderstandings, and we offer the following clarifications:\n\n1. After carefully studying Evofed, we found that it does not involve reshaping or partitioning vectors, nor does it use low-rank decomposition. In essence, Evofed substitutes the transmission of model parameters with the use of distance similarity between the trained model parameters and a noise-perturbed model population.\n\n2. The primary focus of BKD is not on reshaping or partitioning vectors, but rather on the innovative application of Kronecker products to enhance the rank of model updates. While the block structure in BKD serves as a partitioning mechanism, its main purpose is to enable dynamic compression strength, rather than being the key factor in improving efficiency.\n\nTherefore, we assert that the innovation of BKD is worthy of recognition.\n\n***Q3: \"AAD's doubles communication costs compared to freezing one matrix.\"***\n\n**R3:** \nAAD does not double communication costs. The following explanations clarify this:\n\n1. Compared to standard decomposition ($UV^\\top$), AAD ($U\\tilde{V}^\\top + \\tilde{U}V^\\top$) does not increase the communication volume. This is because $\\tilde{V}$ and $\\tilde{U}$ are noise terms determined by random seeds and are not updated. Thus, the communication volume remains #$param(U)$ + #$param(V)$.\n\n2. The comparison in Table 2 is made under identical communication conditions, meaning that with the same communication volume, AAD outperforms the approach of freezing a matrix.\n\nThese points will be emphasized in the revised version of the paper.\n\n\n***Q4: \"additional computational costs.\"***\n\n**R4:** \nAs you noted, while our focus is communication compression, our approach incurs additional computational and memory overhead. We reported the local training time of different methods. Results (**Table 5** in the avove link) show that our method does not excessively increase local training time. On the contrary, the disadvantages of additional computing time overhead can be offset by the benefits of efficient communication and will not affect the acceleration of federated training.\n\n***Q5: \"concern about scalability\"***\n\n**R5:** \nPlease kindly refer to **R3** from reviewer **Gugc** for our response.\n\n***Q6: \"hyperparameter sensitivity:\"***\n\n**R6:** \nOur hyperparameters typically do not require manual tuning. The following clarifications are provided:\n\n1. Our method introduces three additional hyperparameters: compression strength ($r$ or $k$), reset interval ($s$), and initialization size ($a$).\n\n2. Compression strength: The values of $r$ and $k$ are determined by the desired compression strength and do not require further adjustment.\n\n3. Reset interval: Our experiments and analysis indicate that larger values of $s$ lead to lower accuracy. As such, $s$ should be set to 1 by default without modification. To better demonstrate the mechanism of MUD, we include experiments showing the accuracy reduction associated with larger values of $s$. Therefore, $s$ does not need to be treated as a tunable hyperparameter, and there is no sensitivity issue.\n\n4. Initialization size: While the initialization of sub-matrices affects model accuracy, this issue is inherent in all low-rank methods. As shown in Fig. 4 of our paper, BKD effectively reduces the sensitivity of the model to initialization size. For further details, please refer to **R2** from reviewer **u9mP**.\n\n\n***Q7: \"choice of parameter s and question about FedMUD vs. FedLMT Performance:\"***\n\n**R7:** \nFedMUD reduces to FedLMT when $s\\geq T$, where $T$ is the number of training rounds, and $s$ denotes the number of rounds for resetting model updates (i.e., adding a submatrix to the base parameters and reinitializing the submatrix). In MUD, the final model parameters are expressed as: $W=W_0+ \\sum_i^{T/s} UV^\\top$. \nAs shown in Figure 3 in our paper, model accuracy generally decreases as $s$ increases, which is consistent with Theorem 1. However, compared to $s=1$, slight improvements in accuracy are occasionally observed for $s=2$ or $s=4$, which can be attributed to data heterogeneity. Specifically, a larger interval allows the low-rank matrices to observe more diverse data before being added to the frozen parameters, thereby mitigating the effects of data heterogeneity. However, as the interval increases further, previously learned knowledge may be forgotten or overwritten, leading to a decline in accuracy.", "claims_and_evidence": "- Claim 1: MUD reduces information loss by decomposing only model updates rather than full parameters.\n  - **Evidence**: Theoretical discussion and experimental results show reduced compression errors and better accuracy.\n  - **Concern**: MUD is not novel, as similar ideas exist in LoRA[1] techniques, EvoFed[2], SA-LoRA[3], FFA-LoRA[4], and MAPA[5], which also freeze model parameters and only compress updates. The authors fail to highlight these connections.\n\n- Claim 2: BKD enhances the rank of decomposed matrices, improving information preservation during compression.\n  - **Evidence**: Theoretical rank bounds and experimental validation are provided.\n  - **Concern**: BKD novelty is limited, as improved rank and parameter efficiency are explored in EvoFed through *reshaping* into single vector and *partitioning*. There is little justification for why this particular approach is better than existing alternatives.\n\n- Claim 3: AAD mitigates implicit aggregation errors introduced by low-rank approximations.\n  - **Evidence**: Theoretical formulation and ablation studies validate reduced bias in aggregation.\n  - **Concern**: AAD is novel, but it doubles communication costs compared to freezing one matrix. There is no comparison of whether this additional cost is justified by improved performance. (Accuracy per Communication analysis)\n\n\n[1] LoRA:  Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models. arXiv 2021.\" arXiv preprint arXiv:2106.09685 (2021).\n\n[2] EvoFed: Rahimi, Mohammad Mahdi, et al. \"EvoFed: leveraging evolutionary strategies for communication-efficient federated learning.\" Advances in Neural Information Processing Systems 36 (2023): 62428-62441.\n\n[3] SA-LoRA: Guo, Pengxin, et al. \"Selective Aggregation for Low-Rank Adaptation in Federated Learning.\" arXiv preprint arXiv:2410.01463 (2024).\n\n[4] FFA-LoRA: Sun, Youbang, et al. \"Improving loRA in privacy-preserving federated learning.\" arXiv preprint arXiv:2403.12313 (2024).\n\n[5] MAPA: Rahimi, Mohammad Mahdi, et al. \"Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation.\" https://openreview.net/forum?id=rhfOzJzsKN", "methods_and_evaluation_criteria": "- **Methods**: Introduction of MUD, BKD, and AAD techniques. Theoretical proofs are provided for convergence and decomposition efficiency.\n- **Evaluation Criteria**: Accuracy on multiple datasets under non-IID and IID settings. Baselines include FedHM, FedLMT, FedPara, EF21-P, and FedBAT.", "theoretical_claims": "1. **Convergence Guarantees**: Provided under certain and solid assumptions (e.g., L-smoothness, bounded gradients).\n2. **Improved Rank and Compression Efficiency**: BKD offers higher rank upper bounds and flexible compression ratios.\n3. **Error Reduction**: AAD minimizes bias during aggregation, validated by rigorous mathematical derivation.\n\n**Concern**: While the convergence analysis is provided, additional computational costs are not analyzed, which is critical for understanding scalability.", "experimental_designs_or_analyses": "**Datasets:** FMNIST, SVHN, CIFAR-10, CIFAR-100.  \n**Data Distribution:** Non-IID and IID settings.  \n**Metrics:** Test accuracy and convergence speed.  \n**Comparisons:** Against multiple baseline algorithms under controlled hyperparameters.  \n**Ablation Studies:** Performed to isolate the effects of MUD, BKD, and AAD, as well as the impacts of reset intervals and initialization values.", "relation_to_broader_scientific_literature": "This work fits within the broader theme of communication-efficient FL. Particularly methods that address reducing communication load through the low-rank representation of the model updates, which can extend to LoRA architecture in FL.", "essential_references_not_discussed": "[1] LoRA:  Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models. arXiv 2021.\" arXiv preprint arXiv:2106.09685 (2021).\n\n[2] EvoFed: Rahimi, Mohammad Mahdi, et al. \"EvoFed: leveraging evolutionary strategies for communication-efficient federated learning.\" Advances in Neural Information Processing Systems 36 (2023): 62428-62441.", "other_strengths_and_weaknesses": "**Strengths:**  \n- Introduction of three complementary techniques that tackle distinct challenges in low-rank decomposition.  \n- Strong theoretical backing with convergence proofs.  \n- Comprehensive and well-controlled experimental evaluation.  \n- Detailed ablation studies enhancing the robustness of the claims.  \n\n**Weaknesses:**  \n- **Lack of Novelty in MUD and BKD:** The core concepts behind MUD and BKD closely align with prior work (e.g., LoRA, EvoFed, MAPA). However, the paper does not sufficiently acknowledge these connections or differentiate its contributions.  \n- **Insufficient Justification for AAD's Overhead:** AAD effectively doubles communication costs, yet the paper lacks a thorough trade-off analysis to justify this overhead.  \n- **Scalability Not Demonstrated:** Experiments are confined to relatively small models, with no evaluation on larger-scale tasks (e.g., ImageNet) or with more complex FL architectures.  \n- **Absence of Computational Cost Analysis:** While communication efficiency is discussed, the computational overhead introduced by the proposed methods on both client and server sides remains unaddressed. This is critical for assessing feasibility in large-scale FL deployments.  \n- **Hyperparameter Sensitivity:** The methods exhibit high sensitivity to initialization values and reset intervals, necessitating extensive fine-tuning. Moreover, the impact of parameters such as *rr* or *kk* (rank or BKD constant) on training dynamics and accuracy-communication trade-offs is not clearly analyzed.", "other_comments_or_suggestions": "There are no additional comments or suggestions at this time.", "questions_for_authors": "**Overhead Analysis:**  \n- What are the memory and computational overheads of the proposed methods compared to baseline approaches?  \n\n**Communication Costs in AAD:**  \n- Is the additional communication cost of AAD over Freezing justified across all scenarios, or is it contingent on the dataset and FL setting?  \n\n**Scalability Considerations:**  \n- Can the approach scale to larger datasets and model architectures, such as ImageNet with Transformers?  \n- How does low-rank factorization perform with architectures beyond MLP and CNN?  \n\n**Choice of Parameter _s_:**  \n- The text suggests that FedMUD reduces to FedLMT when _s = 1_, and also implies that _s = 1_ is often the optimal choice.  \n- What is the rationale for selecting _s > 1_ in certain scenarios?  \n- Are there specific advantages to higher _s_ values?  \n\n**FedMUD vs. FedLMT Performance:**  \n- If FedMUD reduces to FedLMT with _s = 1_, why does Table 1 indicate that FedMUD outperforms FedLMT?  \n- Shouldn’t their accuracies converge in this case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zfx9avwdzq", "reviewer": "Reviewer_fwVG", "summary": "The authors introduce a novel communication-efficient federated learning algorithm that integrates Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD). This approach is particularly well-suited for training large neural networks, which commonly involve high-dimensional tensors. They provide a theoretical analysis of the local method applied to a restricted function $h_i(x)$, where $x$ represents the trainable low-rank updates to the neural network parameters. The proposed algorithm is further validated through extensive computational experiments.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer fwVG:**\n\nWe sincerely appreciate your valuable feedback. Below, we address each of your comments in detail. For additional experimental results, please refer to the anonymous link: ​**https://anonymous.4open.science/r/fedmud_rebuttal-962F**.\n\n***Q1: \"The target object of Theorem 1\"***\n\n**R1:** Sorry for the confusion caused and hope to clarify this misunderstanding with the following points:  \n\n1. In this paper, we propose three modules to enhance low-rank decomposition performance: MUD, BKD, and AAD. MUD serves as the foundational framework, and FedMUD refers to the training scheme using only MUD. BKD and AAD are plugins that further improve the rank of model updates and avoid aggregation errors. When using BKD and AAD, we will add corresponding labels as shown in Table 1 in our paper (e.g., +BKD or +AAD).\n2. Theorem 1 analyzes the convergence of FedMUD (without considering BKD and AAD). Its purpose is to highlight the difference between MUD and full-weight decomposition (i.e., FedLMT vs. FedMUD). By introducing the reset interval $S$ (the number of rounds before adding model update to base weights and then reinitializing low-rank matrices), we establish the relationship between FedLMT and FedMUD to compare their convergence rates. Note that FedLMT is equivalent to FedMUD with $S=\\infty$.\n3. We did not conduct a theoretical analysis of BKD and AAD. Instead, we further added experiments (**Figures 10-11** in the above link) demonstrating their effectiveness in increasing the rank of recovered model updates and avoiding aggregation errors. We will revise the paper to clarify these points and consider integrating Section 4 into Section 3.1 to avoid ambiguity.\n   \n\n***Q2: \"concern about reshaping convolutional kernels\"***\n\n**R2:** Reshaping the convolutional kernel into a 2D matrix for low-rank decomposition is a method employed by FedLMT and [1], representing a common approach to decomposing convolutional layers. This technique aims to  reduce computational complexity from $\\mathcal{O}(k^2 c_{in} c_{out})$ to $\\mathcal{O}(kr(c_{in} + c_{out}))$. Therefor, for a fair comparison, we follow their processing approach of convolutional kernels.\n\n[1] Initialization and Regularization of Factorized Neural Layers, ICLR 2021.\n\n***Q3: \"definition of abbreviation AAD\"***\n\n**R3:** \nThank you for raising this point. The definition of AAD is provided in Figure 1(c), though it may not be obvious. We will include a clearer, formal definition of AAD at an appropriate location in Section 1.\n\n***Q4: \"how BKD is applied in Eq.(8)\"***\n\n**R4:** \nEqs (8) and (9) describe the process of recovering weights and aggregating submatrices using only AAD. When combined with MUD, the local weights of client $i$ are given by:\n\n$W_i = W_g + [U_i(\\widetilde{V})^\\top + \\widetilde{U}(V_i)^\\top]$\n\nwhere $i$ is the client index, $W_g$ is the latest global model parameter, and $U_i(\\widetilde{V})^\\top + \\widetilde{U}(V_i)^\\top$ represents the local model update. $\\widetilde{V}$ and $\\widetilde{U}$ are random noise terms, determined by a unified random seed, which are untrainable and identical across all clients.\nWhen incorporating BKD, the matrix multiplication of $UV^\\top$ is replaced by the BKD operator ($\\otimes_B$), defined as follows:\n\n$W_i = W_g + [U_i\\otimes_B\\widetilde{V}  + \\widetilde{U}\\otimes_BV_i]$\n\nAfter local training, client $i$ only needs to send the trained $U_i$ and $V_i$ to the server for aggregation, regardless of whether standard matrix multiplication or BKD is used.\n\n***Q5: \"algorithm description\"***\n\n**R5:** \nThank you for your suggestion. We will add a detailed description of our algorithm in the main text, as well as an overall algorithm table for a clearer presentation.\n\n***Q6: \"meaning of $l$ in $\\nabla F_{i,l}$\"***\n\n**R6:** \n$l$ denotes the network layer, and $i$ is the client index. We will include a detailed and prominent explanation in the paper.\n\n***Q7: \"the expression of $L$-smooth\"***\n\n**R7:** \nThank you for the correction. We will revise the description about $L_s$-smooth in the paper accordingly.", "claims_and_evidence": "The authors effectively support their claims with extensive computational experiments. However, I find Theorem 1 somewhat detached from the main flow of the paper. Although the caption of Figure 1, the discussion on discrepancies between equations (6) and (7), and the introduction of AAD collectively refute the direct aggregation of updates, the proof of Theorem 1 appears to analyze the opposite. If my understanding is correct, this discrepancy is particularly evident in equation (18), which forms the basis of the reviewer's concern. Moreover, the theoretical analysis seems to focus on MUD with truncated SVD rather than with BKD.", "methods_and_evaluation_criteria": "Proposed methods and evalution criteteria seem adequate.", "theoretical_claims": "No, I did not.", "experimental_designs_or_analyses": "The experiments appear to be valid.\n\nThere is a minor concern regarding the validity of linearizing the stack of convolutional layers in the CNN model used in the experiments. Intuitively, flattening all convolutional kernels and then reshaping them back may introduce significant approximation errors. However, I do not consider this to be a major issue.", "relation_to_broader_scientific_literature": "The paper falls within the broad area of communication-efficient federated learning (CEFL) algorithms. Unlike approaches that employ well-defined compression operators—whether unbiased or biased—which typically first compute an intermediate update based on the iterate and loss before applying the compression operator, this work belongs to a subfield of CEFL where optimization and compression occur simultaneously. A notable example from this subfield is FedLMT, which directly decomposes the neural network layer matrix rather than the update to it.", "essential_references_not_discussed": "Related work is adequately described.", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "1. The abbreviation AAD is introduced on line 74 but is only defined later in the text.  \n2. It is unclear how BKD is applied in equation (8). Specifically, how is the model update $\\Delta W$ distributed between the two summations when combining MUD+BKD+AAD?  \n3. In line 230, the phrase *\"L-smooth with a constant $L_s$\"* should be revised to *\"$L_s$-smooth.\"*  \n4. In line 235, the notation $l$ in $\\nabla F_{i, l}$ is ambiguous. Does it refer to the layer number?  \n5. The algorithm description is missing from the main text and appears only in the appendix, in equation (15).", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4A2E4R84gU", "reviewer": "Reviewer_Gugc", "summary": "The paper focuses on enhancing communication efficiency in FL by improving low-rank decomposition techniques.  The authors identify three key issues: what to decompose, how to decompose, and how to aggregate.  To address these, they propose three novel techniques: decompose only model updates, block-wise Kronecker decomposition, and ways the matrices are assembled by the server.  These techniques improve convergence speed and accuracy compared to existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer Gugc:**\n\nWe sincerely appreciate your valuable feedback. Below, we address each of your comments in detail. For additional experimental results, please refer to the anonymous link: **https://anonymous.4open.science/r/fedmud_rebuttal-962F**.\n\n***Q1: \"datasets and models\"***\n\n**R1:**\nIn response to your question, we have added experiments with other model architectures (ResNet18, ViT) and datasets (TinyImageNet). Specifically, we test ResNet18 on CIFAR-10 and TinyImageNet, and ViT (pretrained on ImageNet) on CIFAR-10. We compared only FedAvg, FedLMT/FedHM, and FedMUD+BKD+AAD. The results (**Figure 8** in the above link) further validate the superiority of our method. By the way, due to limited time, we did not test on LLM. But we conducted experiments on a transformer-based model (ViT) and proved that our method should be applicable to LLM. We will include the complete experimental results in the revised paper.\n\n***Q2: \"FedMUD's formal statement\"***\n\n**R2:**\nWe have formally defines FedMUD in Section 3.1, line 159: *Federated Learning with Model Update Decomposition (FedMUD)*. Notably, FedMUD refers to using only MUD, excluding the BKD and AAD designs. For better presentation, we will add its formal definition in Section 1 of the revised paper.\n\n***Q3: \"difference between model update decomposition (MUD) and LoRA\"***\n\n**R3:**\nPlease kindly refer to **R3** from reviewer **u9mP** for our response.\n\n***Q4: \"BKD is obfuscated with other aspects such as MUD\"***\n\n**R4:**\nTable 1 presents the accuracy results using FedMUD as the base framework, with BKD and AAD as plugins. The role of BKD is indeed obfuscated with that of MUD or AAD. However, Figure 3 provides an ablation study on the reset interval. Notably, when the reset interval equals the total number of rounds, MUD no longer contributes. In this case, FedMUD+BKD reflects the performance of BKD alone. For further details, please refer to Figure 3 in our paper, which intuitively demonstrates the superiority of BKD over the standard decomposition.\n\n***Q5: \"convergence rates of BKD\"***\n\n**R5:**\nThe convergence of BKD is indeed a noteworthy problem. Theorem 1 focuses on the convergence of FedMUD (excluding BKD and AAD), demonstrating that FedMUD converges faster than FedLMT. However, comparing the convergence of BKD with methods using matrix multiplication is challenging. In convergence analysis, unifying the number of parameters across different decomposition operators is difficult, making a direct comparison of convergence rates infeasible. We will continue to explore solutions to this problem to provide stronger theoretical support for BKD.\n\n***Q6: \"whether low-rank is enough in practice\"***\n\n**R6:**\nIn few cases, low-rank approximations are sufficient, such as in LoRA fine-tuning of LLM on small datasets. However, in most scenarios, low-rank methods are primarily used to reduce computational costs, model size, and communication overhead rather than because they are inherently sufficient. Conversely, many approaches aim to achieve higher matrix ranks using a limited number of parameters. Such as FedPara [1] uses Hadamard product to increase matrix rank, whereas MeLoRA [2] enhances the rank of LoRA by arranging smaller matrices along the diagonal.\nBesides, Figure 5 in our paper reveals that as the rank, controlled by the compression ratio, increases, model accuracy improves. This observation confirms that higher ranks contribute to better accuracy. Thus, low-rank approximations are generally insufficient, particularly for communication compression.\n\n[1]FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning, ICLR 2022\n[2]MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning, ACL 2024\n\n***Q7: \"bounded weights assumption\"***\n\n**R7:**\nYour concern regarding the bounded weights assumption is valid. However, FedMUD ensures its reasonableness by periodically reinitializing the low-rank matrices:\n\n1. First, we assume that the gradients are bounded, which is a reasonable and commonly adopted assumption in convergence analysis, as discussed in lines 245–247 of the paper.\n2. The bounded weights assumption applies to the low-rank matrices $U$ and $V$ during local training. In FedMUD, $U$ and $V$ are updated via gradient descent for a finite number of steps, after which they are manually added into the base model parameters and reinitialized. This process prevents the unbounded growth of $U$ and $V$.\n3. To further substantiate the feasibility of this assumption, we provide a theoretical justification. Let $U$ represent a matrix with an initial value $U_0$ of dimension $n$, an upper gradient bound $G$, and $\\tau$ gradient descent steps before reinitialization. Then, $U_t = U_0 +\\eta\\sum_{i=1}^t g_i$, where $t\\leq \\tau$. Thus, $\\Vert U_t\\Vert^2\\leq (t+1) \\left[\\Vert U_0 \\Vert^2+\\eta^2 \\sum_{i=1}^t\\Vert g_i\\Vert^2 \\right] \\leq (\\tau+1)\\left[\\Vert U_0\\Vert^2+\\eta^2\\tau^2 G^2\\right]$.", "claims_and_evidence": "They are clear but some of the claims are questionable. I disagree with all of the claims regarding the model update decomposition. The original LoRA paper proposes to min f(W+AB) which means that only model updates get a low rank decomposition. Other FL low-rank algorithms also assume the same setting. Unless I'm missing something, the first contribution is not a contribution. To the contrary, it's common practice.", "methods_and_evaluation_criteria": "They are okay. Very standard small datasets. It would be great to apply it for find tuning of an LLM. \nI wonder why a standard CNN wasn't used (ResNet18 for example) instead of a 'out-of-nowhere' architecture.", "theoretical_claims": "I didn't check the proofs. The statement is a typical convergent statement and thus I does I have fairly high confidence in the proof. \n\nThe analysis of the algorithm with Kronecker and aggregation would be much more interesting. \nIn general, convergence analyses are fairly standard. A more interesting question: with the same number of trainable parameters, compare convergence rates of Kronecker vs standard AB decomposition.", "experimental_designs_or_analyses": "I did. See Methods And Evaluation Criteria for comments.", "relation_to_broader_scientific_literature": "See my other remarks regarding LoRA and other FL low-rank algorithms.", "essential_references_not_discussed": "Nothing to report.", "other_strengths_and_weaknesses": "FedMUD is never formally stated. Based on theory it seems not to include Section 3.2. It's unclear if it includes Section 3.3. \nIf it includes only Section 3.1, based on my LoRA comment, it questions the contribution of the analytical convergence proof. \n\nI think the idea of using the Kronecker product is a great one. Unfortunately the authors don't do enough with it; it is obfuscated with other aspects such as model update decomposition whose novelty is questionable.", "other_comments_or_suggestions": "Lines 84 to 88, left column: maximizing the rank is crucial: I'm not sure where this is coming from. In practice and as demonstrated in many paper, low-rank is enough.", "questions_for_authors": "1. how is model decomposition different from LoRA and many offsprings?\n2. Why not fine tuning of an LLM in experiments?\n3. Why not ResNet18 (or a different standard model)?\n4. The assumption of bounded weights is questionable. In practice gradient clipping is used because the weights are not bounded (unless being forced to be bounded).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cA3Qoi82bw", "reviewer": "Reviewer_u9mP", "summary": "This paper reduces communication overhead in FL by enhancing low-rank decomposition techniques. The authors focus on three key issues: what to decompose, how to decompose, and how to aggregate. They propose three techniques respectively: Model Update Decomposition (MUD), Block-wise Kronecker Decomposition (BKD), and Aggregation-Aware Decomposition (AAD). The paper provides both theoretical analysis and empirical studies on benchmark datasets.\n\n## update after rebuttal\nMy concerns are mainly addressed during rebuttal. Thus, I will keep my positive rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hi Reviewer u9mP:**\n\nWe sincerely appreciate your valuable feedback. Below, we address each of your comments in detail. For additional experimental results, please refer to the anonymous link: ​**https://anonymous.4open.science/r/fedmud_rebuttal-962F**.\n\n***Q1: \"a more detailed discussion on the upper rank bound of BKD\"***\n\n**R1:** \nAccording to your suggestion, we provide a detailed discussion on the upper rank bound of BKD.\n\n1. Theoretical Analysis: We leverage the properties of the Kronecker product ($rank(A \\otimes B) = rank(A) \\cdot rank(B)$) and matrix concatenation ($rank(A|B) \\leq rank(A) + rank(B)$). Suppose $A, B \\in \\mathbb{R}^{a \\times b}$, then $W = A \\otimes B \\in \\mathbb{R}^{a^2 \\times b^2}$. By the nature of the Kronecker product, $rank(A \\otimes B) = rank(A) \\cdot rank(B) \\leq \\min\\\\{a^2, b^2\\\\}$, which corresponds to the full rank of a matrix with dimensions $(a^2, b^2)$. Furthermore, BKD can be viewed as the concatenation of multiple Kronecker product matrices. Given that each submatrix retains full rank, the concatenated BKD recovery matrix also achieves full rank.\n\n2. Experimental Validation: We compare the rank of model updates obtained via matrix multiplication and BKD. The results, presented in **Figure 10** in the above link, demonstrate that BKD produces near full-rank model updates, significantly outperforming standard matrix multiplication.\n\nWe will incorporate above disscussion into the paper.\n\n***Q2: \"how do other low-rank decomposition methods handle the initialization of low-rank matrices\"***\n\n**R2:** \nOther low-rank decomposition methods, such as FedLMT and FedPara, also suffer from initialization variance.\n\nHyeon-Woo et al., in FedPara, employ the default He initialization [1] but acknowledge that \"investigating initializations appropriate for our model might improve potential instability in our method.\"\n\nSimilarly, Liu et al., in FedLMT, state that \"the performance of low-rank models can be boosted through a customized initialization called spectral initialization [2],\" which \"uses SVD to initialize low-rank model parameters.\"\n\nIn our experiments, to ensure a fair comparison, we adjust the initialization size across different methods. As shown in Figure 4 in our paper, BKD mitigates the sensitivity of the model to initialization variance to some extent. We attribute this to the fact that $Var(AB)=r\\cdot Var(A)\\cdot Var(B)$ (where $A\\in\\mathbb{R}^{a\\times r}, B\\in\\mathbb{R}^{r\\times b}$, assuming both are i.i.d.), while $Var(A\\otimes B)=Var(A)\\cdot Var(B)$. Clearly, the variance of the reconstructed matrix using matrix multiplication is also influenced by $r$, making it more unstable. \n\nIn this paper, we did not delve into the impact of low-rank matrix initialization on model performance. This is an interesting and meaningful direction that we will continue to pay attention to and conduct possible explorations.\n\n[1] FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning. ICLR 2022.\n\n[2] FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees. ICML 2024.\n\nBelow, we address a general issue regarding novelty raised by other reviewers, and we hope this will provide you with new insights.\n\n\n***Q3: \"novelty of MUD and difference with LoRA\"***\n\n**R3:** \nThe core difference between MUD and other model update decompositions is that our model update is for several training rounds, not the entire training process.\nLet's take LoRA as an example to explain the difference and relationship between FedMUD and LoRA in the federated fine-tuning scenario. This also applies to other technologies with similar ideas.\n\n1. Assuming the pre-trained weight is $W_p$, LoRA will learn $W_p+AB$ during the entire FL training process, where $W_p$ is frozen. $A$ and $B$ are trainable parameters, which represent the model update during the entire FL training process.\n2. In FedMUD, we learn the model update every $S$ rounds. Reset Interval $s$ is a hyperparameter of fedMUD (the default value is 1), which means that every $S$ rounds, we manually add the model update $AB$ to the pre-trained parameter $W_p$ and reinitialize the submatrix. In this way, the parameters of the final model can be expressed as $W=W_p+\\sum_i^{T/s}A_SB_S$, where $T$ is the number of training rounds. By training different low-rank increments in different rounds, we can achieve better accuracy with the same amount of communication.\n3. When we set $S$ of FedMUD to $\\infty$ (or any value larger than the number of communication rounds $T$), which means there is one low-rank increment during the entire training process, FedMUD and FL-LoRA are equivalent. In fact, FL-LoRA is equivalent to FedLMT.\n\nHopefully the above explanation can clarify for you the difference between how MUD and LoRA and other methods handle model updates.", "claims_and_evidence": "The claims made in this paper are well-supported by both theoretical proofs and experimental validation.", "methods_and_evaluation_criteria": "This paper focuses on improving communication efficiency in FL by proposing three techniques that comprehensively improve low-rank decomposition. The proposed methods have been validated on multiple benchmark datasets.", "theoretical_claims": "The claims are well-supported by the proofs in general.", "experimental_designs_or_analyses": "The authors conduct experiments on four datasets, comparing the proposed methods against traditional low-rank decomposition techniques and sota pruning and quantization methods. Ablation studies and hyperparameter analysis are also conducted.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "This paper adequately discusses essential literature related to low-rank decomposition in FL.", "other_strengths_and_weaknesses": "Strengths:\n1. Communication overhead in FL is a critical problem. This paper identifies three key challenges in low-rank decomposition for FL and proposes three novel techniques to address them.\n2. The paper provides a rigorous theoretical analysis, proving that FedMUD achieves better convergence than the traditional method.\n\nWeaknesses:\n1. The authors claim that BKD achieves $rank(W)\\leq\\min\\{m,n\\}$. I suggest including a more detailed explanation.", "other_comments_or_suggestions": "I suggest the authors provide a more detailed discussion on the rank upper bound of BKD.", "questions_for_authors": "In Section 5.5, the authors discuss the significant impact of initialization on low-rank decomposition performance. I am curious whether traditional low-rank decomposition methods in FL also face this issue. If so, how do they typically determine the initialization size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three low-rank decomposition methods are suggested to reduce communication overhead in FL: model update decomposition, block-wise Kronecker decomposition, and aggregation-aware decomposition. These methods address different issues and can be used together. Model updates decomposition factorizes model updates rather than entire model parameters to minimize information loss. Block-wise decomposition enables high-rank recovery with a relatively small number of parameters, promoting dynamic compression. Aggregation-aware decomposition reduces compression errors during model aggregation by decoupling multiplication using a trainable matrix. Experiments are extensive, and theoretical supports including convergence analysis are convincing. On the down side, even as the core concepts behind MUD and BKD closely align with prior work (e.g., LoRA, EvoFed, MAPA), the paper does not sufficiently acknowledge the connections or differentiate its contributions. Overall, the strengths and merits outweigh the shortcomings.
Decision: Accept (poster)