ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
Authors: Yujia Wang
Yuanpu Cao
Jinghui Chen
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.

Published in Transactions on Machine Learning Research (03/2026)
ParaBlock: Communication-Compuaaaioo Parallel Block Coordinate Federated Learning for Large Language Models
Yujia Wang
yuu54270psu.edu
College of Information Sciences and Technology Peonsyloania State Universiby
Yuanpu Cao
ymc55330psu.co edu
College of Information Sciences and Technology Pennsyloania State University
Jinghui Chen
jzc5917@pes edu
College of Information Sciences and Technology Pennsgloonia State University
Reviewed on OpenReview: https: opwnreviee..........mmmmm net/ forun? id- Hn. HeCBBee
Abstract
Federated learning (FL) has been extensively studied as privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource- clients. To address this challenge in federated wraining/fine-tunne LLMs, propose ParaBlock, novel approach that establishes two parallel threads for communication and compulation to enhance communication efficiency. We thelretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.
1 Introduction
Federated learning (FL) (McMahan et al. 2017) is known promising privacy-proserving paradigm, it keeps data local clients without exposing sensitive information. A typical FL framework consists of central server and multiple local clients operating in single-threaded design. In this setup, clients repeatedly train local models, transmit the updates to the server for aggregation, and wait for the updated model from the server to continue training in the next round. While standard FedAvg type frameworks (McMahan et al, 2017: Karimireddy et al., 2020 Li et al., 2020 Wang Ji. 2023 Wang et al.
2024bb have demonstrated federated block coordinato 2024a have recently gained increasing attention. Such schemes integrate local block coordinate descent (BCD) update strategies into FL, allowing clients to train only subset of the model (commonly block) locally instead of the entire model. Although federated BCD still operates in a single-thread manner, where clients need to wait for the model transmission to complete before the next computation step, such communication time is often negligible when models are smaller and the block size limited.
success across various applications (Koutsoubis al. 2024 Beutel et al. 2022))
descent schemes (Liu al.] 2019 Wu et al. 2022 Liu et al. 2024 Wang al.
However, with the recent trend of billion-scale large language models (LLMs), such as GPT-3 Brown al. 2020), Llama 3 (AT@Meta 2024) and Gemma (Team et al. 2024 even single layer can encompass

---

Published in Transactions on Machine Learning Research (03/2026)
tens to hundreds of millions of parameters. This massive scale significantly increases the communication time ween the server and clients. While such communication time was considered negligible in traditional federated block coordinate methods, it has now become a notable factor when fine-tuning LLMs on edge clients. This raises a critical bottleneck for fine-tuning LLMs with federated block coordinate descent methods: for clients with limited network bandwidth or those requiring long-distance transmission, the communication latency significantly impacts the overall efficiency of FL deployment. This inefficiency of such a single-thread approach hinders the scalability of federated block coordinate descent methods for LLMs, underscoring the need for solutions to reduce commmnication latency for clients.
To address the aforementioned challenge, we propose ParaBlock, a federated commuicctionn-cmpputaaii Parallel Block coordinate descent method that provides a simple yet effective way to improve the communi- cation efficiency of BCD updates for fine-tuning large-scale models on resource-constrained clients. The key contributions of this paper are summarized as follows:
We propose ParaBlock, a novel method designed to address communication latency during the fine-tuning of LLMs using federated block coordinate descent. ParaBlock replaces the traditional single-threaded design with two parallel threads for communication and computation, effectively reducing communication delays and improving efficiency.
We rigorously show that the proposed ParaBlock algorithm achieves convergence rate of O(1/VT) for non-convex objectives. This rate is consistent with that of standard federated block coordinate descent methods, ensuring that the improvements of ParaBlock in communication efficiency do not compromise its convergence performance.
We conduct extensive experiments across various models and tasks empirically validate the effectiveness of ParaBlock. Specifically, we evaluate the downstream performance of ParaBlock on general instruction following and mathematical reasoning tasks. Our results demonstrate that ParaBlock significantly reduces wall-clock runtime while maintaining performance on par with standard federated block coordinate descent, baselines, highlighting its ability to enhance efficiency without compromising utility.
2
Related Works
BCD for FL Block Coordinate Descent (BCD) has been extensively studied for optimization problems Tseng
2001
Nesterov 2012 Beck & Tetruashvili, 2013), particularly due to its elficiency in solving large-scale
optimization tasks. Recent advancements in BCD variants, such as those proposed by [uo et al. (024 Pan
ct al. (2024), further highlight its adaptability and effectiveness, especially in the context of large language
Wu et al., 2021) enhance communication efficiency by assigning each client the responsibility for a partition of the model in a collaborative traïning framework.
models. Federated BCD schemes (Liu et al. 2019: 2024
FedCyBGD (Wang et al. (224aa) focuses on fine-tuning large language models by cyclically electing one client at time to train its assigned block of the model, enabling efficient fine-tuning with limited computational resources.
Communication-fflcieenen for FL Various methods have been proposed to improve communication effi- ciency in federated learning, focusing on techniques such as quantization and update compression (Reisizadeh
le al.]
2020;
Haddadpour et al., 2019 Jin et al.] 2020; Jhunjhunwala et al. 2021 Wang et al.) [022 [i
ctt al.
2024), model pruning (Li et al. 2021 Jiang et al. 2022; Isik et al. 2022). and model distillation (Wu
ee al. 2022). Several existing works have been shown to enhance communication efficiency specifically in
the context of federated fine-tuning of LLMs. For example, many applications of PEFT in FL help reduce
communication costs (Liu et al.) 2019 2024). Additionally, FedMKT (Fan et al. 2024 leverages a mutual knowledge transfer framework, where clients and the server exchange knowledge datasets instead of model updates. Similarly, CG-FedLLM (Wu et al.) 2024) introduces an encoder on clients to compress gradient. features, with a corresponding decoder on the server to reconstruct the transmitted features.
FL for LLMs In the era of LLMs, FL has evolved to develop more complex fine-tuning and deployment of large language models (Zhang et al. 2024 Ye et al. 2024). Several existing approaches explore the use
2

---

Published in Transactions on Machine Learning Research (03/2026)
of Parameter Efficient Fine-Tuning (PEFT) techniques for federated fine-tuning of LLMs. These include leveraging low-rank adapters (LoRA) in methods such as FedIT (Zhang et al. 2024 p, the OpenFedL.LM framework Ye et al. 2024), HetLoRA (Cho et al.) 2024), SLoRA (Babakniya et al. 2023), FLoRA (Wang FI. (Bai et al. 2024) as well as employing adapters and bias-tuning in FedPEFT (Sun
ict al.
2024d). eet al.
In addition to the widely adopted PEFT methods in FL,, recent work, such as FedCyBGD [Wann et al.
2022j.
2024a), also highlights the potential of BCD in improving federated fine-tuning performance
while managing resource limitations.
3 Preliminaries and Motivations
We start with a standard federated learning objective with N total workers:
N
min f(0) agdd
5i(0) N Be,-,,(((0EE.).
(1)
N
where 0 E Rd indicates the model parameter with d dimensions, f:) represents the loss function associated with client i, and Dr denotes the local data distribution for client i. FedAvg (McMahan et al. 2017) is widely used for solving ( In t-th global round, cach participating client performs local training using standard SGD optimizers. The server periodically collects and aggregates the local models or updates to obtain a new global model.
BCD and Block-wise update for FL BCD usually has the same optimization objective as (stochastic) gradient descent. It. iteratively optimizes over a small subset of parameters, while keeping the remaining parameters fixed. This approach makes BCD practical for large-scale model training. Federated BCD is then applicd in FL by substituting BCD for the SGD optimizer in FedAvg. Consider a block partitioning of the model parameters / into B blocks, i.e., = [[0]1 [0]2...... [0]a]. In federated BCD schemes, the goal is to approximately optimize the objective in 1 by updating only one block during local training, while keeping the remaining blocks unchanged across clients. For example, if block b assigned for training at global round t, the blsck-wise optimization for (0)b, is defined as:
1 Σ _( (0).. (O........ (0))) N
argmin f(0) 10),
where [0), E Rdd,, and do, denotes the dimension for block by and there is Dg key component of the local BCD update for federated BCD in Algorithm t, client approximately optimizes local objective fa with the stochastic gradient of f::
== dg = d. We summarize the
[ Specifically, during global round
Off 000
Of:
0f: 000BB
(..:
(2)
where the partial stochastic gradient for block b, is denoted as gg').. (e; The local model training could be summarized in Line 4 in Algorithm [ After completing K local steps of training, the client calculates the difference in local updates as shown in Line 6. The client then sends A to the server to update the global model, and the server sends back the global model to clients for next round of local training.
The communication bottleneck for federated BCD While federated BCD has been applied to large-scale FL training, still faces subsequent communication inefficiency the era of LLMSs. With the significantly larger model size and the accompanied large communication time, the current single-thread design in the federated BCD method increased overall runtime due to the outstanding communication latency. This prolonged communication time forces clients to wait extensively before receiving the latest global information and proceeding to the next round of local computation. Such communication latency poses a critical bottleneck during the deployment of federated BCD, motivating us to explore new methods for improving communication efficiency and the sillt of federated BCD. Drawing inspiration from distributed training paradigms, if clients can change from the single thread design to two parallel threads that overlap the local computation with the model communication, then the communication latency can be ideally eliminated.
as shown in Figure D has led to
3

---

Published in Transactions on Machine Learning Research (03/2026)
4 ParaBlock: A new federated fine-tuning method
To achieve this communication computation overlapping in federated BCD methods, we in- troduce ParaBlock, Parallel Block coordi- nate descent method designed for fine-tuning LLMs in federated learning. ParaBlock aims to parallelize the communication thread and com- putation thread on local clients (as shown in Figure D. enabling clients perform local up- dates while concurrently communicating with the server. This design ensures that clients no longer need to wait for communication to fin- ish, siguificantly reducing clients' idle time and accelerating the federated fine-tuning process. We summarize the proposed ParaBlock in A1- gorithm 2 In nutshell, the communication thread and computation thread proceed in porallel. Specifically, for each client i, the computation thread fine-tunes block b, using the LocalBlockTraining summarized in Algorithm [ During this computation, except for the first round when t = 0, client i proceeds with the communication thread to synchronize with the server regarding the model update variables from the previous round (one round behind). This involves sending A to the server and receiving the aggregated A from the server. Once both threads are completed, client i immediately updates its local model for the next round of training, as illustrated in Line 9. Note that the new local model 0[ inherits most of the parameters from the current model 0 but incorporates two key update steps:
A Algorithm LocalBlockTraining
local learning rate m, global model 0 with B blocks, number of local steps K, assigned block b
Input:
Initialize model 0- = on client i for == 0 to K - do
3: Compute local partial stochastic gradient lgkb [V El
4: Local update: 0/+1 + 0% 10 +- [0]] - vn Lg
5: end for
Client gets =' 10 [0',
Output: A
1. The latest fine-tuned block b, of the lo- cal model is updated with A;, ensuring that 1l. is updated with the latest local infor- mation before starting the next computation thread.
Downlink
Co mputsttoon
Uplink
Model block
1
2
Time
Single-Threar
D2
C2
Ug
D2
C2
Ug
Dy
C2
Ug
latency
2. The client. uses the received global vari- able At- to correct the previous update on the local block b by applying correction 10ttlle-, + [0 +n( A 1i-1). This is because the previous update on block b only used the local update م (since the global update A, is one round behind and has not arrived yet that time), and now we want it to be consistent with the global model by replacing the local update with the global one.
Orignnll
Time
Comm. Thread Comp. Thread
D2
Dg
D
C
C2
G
ParaBlock
Figure Comparison between the original federated BCD and the proposed ParaBlock. The original BCD's single- thread approach leads to higher runtime due to commu- nication latency, while ParaBlock improves efficiency by overlapping communication and computation.
After two blocks are updated in 01 client i begins the new communication and computa- tion thread for global round & ++7 1. Therefore, unlike standard foderated BCD methods and most traditional single-thread FL paradigns, the two-thread parallel design of ParaBlock allows clients to continue local fine-tuning while conducting communication. Although the global information is updated to the local model with one-round delay, the impact caused by this delay can be effectively mitigated by correction of ParaBlock.
Moreover, while clients update the local model, the server simultaneously updates the global model using the aggregated A,-I (Line 10 in Algorithm D. Compared with standard foderated BCD mothods, here the global model 0₂ round t exhibits one-round staleness because the most recently trained block b, has not yet been incorporated into the global model. Due to this staleness, extra communication and aggregation
4

---

Published in Transactions on Machine Learning Research (03/2026)
step is required at the end of the fine-tuning (Lines 12-14) to ensure that the final computation results are aggregated and updated to the global model, then the server obtains the final model Or.
Algorithm 2 ParaBlock
Input: local learning rale m global learning rale 7, number of block partition B
1: Initialize global model 0g and generate a block partition = 1.2,..., B
2: for t = to - do
Each client runs a compute thread and communication thread parallel:
4: Compute thread:
A + LocalBlockTraining (6;, m. K by)
Communication thread:
if >> 0 then
7: Client sends - to the server and waits for the aggregated A,-1
8: end if
9:
// When both threads finish, client i processes:
updating local model: 0141 + 0j [0it1l, + [0]],, + n if then
10 [0%),,,, + + 7) A
10:
Server maintains the global model
0, + 0.-1, Or)mm-1 10,-1]a_- + -
11: end for
12: Client send A to the server and wait for the aggregated Ar-1
13: Server updates the global model
+ 0r-1. [0r].r.: (1r-1 + n
14: Output 0r
In the following, we will mathematically demonstrate how the correction in Line 9 addresses the inconsistency in local models and we provide an analysis of the relationship between the global and local models.
Recursive derivation of local and global models We begin with round 1, where client continues fine-tuning using the local model 0j where [0]),.., [0i-11b.-1+ of block b1-1 differ across all clients, as they have not yet. finished the synchronizing the latest updates with the server. Then by global round as the synchronization between the server and clients regarding block b is completed, clients can correct block b in 0:41 using A,- A
+ +n At this stage, the model parameters
0 1
[0/b... 7 A 1 = 10 Ib.., + +
10%1o-,, t Σ I. (bbb1)) 10ulo_., + Σ 11.61144
(3)
==0
==0
where (b:-1) is an indicator function that equals to block b,-1 assigned for round s, and 0 otherwise. This recursive derivation shows that the local model for block 6;-1 can be expressed as the initial global model Gg combined with all relevant global updates. Thus, the onc-round delayed A,-1 corrects the inconsistent parameters in block b of local models. This ensures that the inconsistent parameters in block b, are effectively corrected after one global round, making sure that most of the parameters across local models align in consistent direction for subsequent local training setg///fnetttttug
Similarly, for the global model 0,, there is
0 b (0-1)6, n = [00]a, +
Σ T (,
(4)
This indicates that the global model's block b,-1 can also be recursively expressed as the initial global model with relevant updates. Based upon the result in Eq. (3) and Eq. 4 the block b, of local model 0 exactly matches the block b in global model 0,. This demonstrates the effectiveness of the model correction mechanism in ParaBlock.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Discussion about privacy protection While FL ensures data ownership protection (raw data stays local), it does not automatically prevent the model from learning and outputting sensitive patterns. This privacy concern is a known challenge in generative language models. ParaBlock primarily addresses the efficiency and communication bottlenecks of training LLMs in this distributed setting. Thus, ParaBlock would be orthogonal to advanced privacy protection techniques like data anonymization or differential privacy minimize the risk of data privacy leakage.
5 Theoretical Analysis
In this section, delve into the convergence guarantee of the proposed ParaBlock algorithm. We begin by outlining the key assumptions necessary for the analysis. Following this, we present the convergence rate and provide a detailed discussion of the results.
Assumption 5.1 (Smoothncss). The local objective function f:(0) L-smooth, i.., Y0,, 02 E R.
(Vi) f.22)) - L 02ll.
Assumption 5.2 (Bounded Variance). The stochastic gradient computed on local client is unbiased and has a bounded local variance, i.c., for all and E [V], we have [ v (0)) I o², and the loss functions has global variance bound,
Σ N [ V (o)|| ogg
Assumption 5.1 and 5.2 are common assumptions in analyzing federated non-convex optimization methods (Li
eet al. 2019 Yang el al. 2021 Reddi et al., 2021 Wang el al. 2022 Wang & Ji 2023 Wang et al., 2024ch.
The global variance upper bound of o in Assumption 5.2 measures the data heterogeneity across clients, where of = indicates i.i.d. data distribution across clients.
In the following, we present the theoretical convergence analysis of ParaBlock. For clarity and fair comparison with existing analyses of FedBCD methods, conduct the analysis under the local SGD optimizer. Extensions to local adaptive optimizers, along with additional discussions on the connection between general and block-wise properties are provided in Appendix
B
5. 5.1. let represent the total number of global rounds, K be the number of local SGD bruining steps and N be the wumber of the clients. If the learning rale n and np satisfy m 22RL and mp S aKL then the global iterates {0,)T- of Algorithm satisfy
Thcorem 5.3. Under Assumptions
1
8F
T
MiTK
S +
TT
0² m
+ n
10m L2 k2² 6Ko,)),
(5)
N
where F = f(0o) fo and f. ming f()) -c. Corollary 5.4. If we choose the global learning rate 17 = ((VKN and m = O 11) for sufficiently large T, the global iterates T-1 of Algorithm satisfy
VTK in Theorem 5.3 then
T-1 M
E o F+ VTKN
(F
No² + +o2
(6)
T
Remark 5.5. Compared with the existing federated BCD methods such as FedBCD (Liu al. 2019 Wu et al. 2021) and FedBCGD (Liu al. 2024). our proposed method obtains the same O(IVTT) convergence rate under general non-convex settings. This demonstrates that the one round staleness introduced by the commuication-copptaann parallel thread in ParaBlock does not compromisc its overall convergence guarantee.
6 Experiments
In general, the proposed method leverages local block updates combined with wwwpmmiicaiiooooomaaattttt parallel scheme during fine-tuning. This section presents series of experiments to evaluate both the

---

Published in Transactions on Machine Learning Research (03/2026)
performance and efficiency of our approach. First, we assess the performance of ParaBlock and compare it against several existing lederated fine-tuning algorithms. Next, we analyse the time efficiency of ParaBlock under varying network conditions and computational demands. Finally, we perform ablation studies to gain deeper insights into the key factors inftuencing the effectiveness of ParaBlock.
6.1 Experimental settings
We summarize some crucial implementation details in the following, and we leave some additional results and experiment details to Appendix A
Datasets, models, and evaluations We utilize the Alpaca-GPTT dataset (Peng et al.) 2023 for general instruction following tasks, and we use 50000 data samples sampled from the MathInstruct dataset Yuu
et al. 2023) for mathematical reasoning task. We fine-tune two models, Llama 3-8B (Dubey et al.
2024
AT@Meta, 2024) and lightweight Llama 3.2-3B MAt&Meta| 2024) designed for on-device use. To assess
the performance when fine-tuning on instruction following task, we utilize MT-Bench (Zheng et al. 2023 with GPT-4o as a judge model. For evaluating the performance on mathematical reasoning, we employ the widely used OpenI.LM Leaderboard (Beeching al.) 2023) as the evaluation benchmark and we report the evaluation score on GSMSK (Cobbe et al.) 2021 to show the math problem solving capability.
Table 1: Finc-tune Llama 3-8B and Llama 3.2-3B
AloMctt
2024 on Alpaca-GPT4 dataset (Peng et al. In our results, we highlight the best score in bold and
2023) and MathInstruct dataset (Yue al. 2023 the second-best score with an underline.
Alpaca- GPT4
Math Instruct
Method
Llama 3-8B
Llama 3.2-3B
Llama 3-BB
Llama 3.2-3B
MT-Benchf RT(m) MT-Benchh RT(m) 4
GSMSKt RT(m)
GSMSKt RT(m)
Base
4.72
4.18
51.55
27.98
Fed full FT
5.33
16.0
4.36
12.5
55.80
15.5
32.22
12.3
FedIT
5.11
34.5
4.31
23.8
54.60
23.1
29.87
15.4
FEA-LORA
5.08
30.2
4.25
23.3
54.59
21.2
30.55
14.9
FLoRA
4.95
65.4
4.23
34.0
52.54
64.9
28.51
29.3
FedCyBGD
4.87
59.9
4.29
57.8
51.40
63.2
27.60
53.3
FedBCD
5.14
30.2
4.33
19.1
54.74
24.9
31.84
17.3
ParaBlock
5.14
21.1
4.40
11.9
55.88
15.8
31.77
10.1
Baselines We compare the ParaBlock with several federated fine-tuning baselines including 1) Federated full model fine-tuning (Fed full FTT), 2) FedIT (Zhang et al.] 2024). which is one of the most commonly used federated finc-tuning methods that integrates LoRA (Hu et al. 2021 into standard FedAvg (McMahan et al, 2017) method, 3) FFA-LoRA (Sun et al. 2024)), which fixes the LoRA matrix B and fine-tunes the LoRA matrix A reduce server aggregation bias, 4) FLORA (Wang et al. (2024d), a recent method for federated finc-tuning with LoRA, 5) FedCyBGD (Wang al.) 2024a), which employs the cyclic update for block coordinate federated fine-tuning, and 6) FedBCD, standard federated BCD scheme that directly integrates local BCD updates into the original FedAvg aggregation schemes. We provide the GPU consumption of all baselines in Table in Appendix A
Implementation details We implement federated fine-tuning of LLMs by setting up an FL framework with 10 clients, each assigned a local dataset. Notably, we perform heterogencous data partitioning for both datasets. For the Alpaca-GPT> dataset, we adopt a text clustering method similar to the one used in Lin et al. [021]] to get cluster label. For the Mathinstruct dataset, use the "source" from the original dataset as label and follow traditional data partitioning using Dirichlet distribution Dirichlet(O.66 Wang et al. (2020b/a). Specifically, we adopt Dirichlet(0.1) for the Alpaca-GPT4 dataset and for the Mnthlnstruct datasel. Regarding LoRA-related methods, the LoRA rank is set to 32 for FedlT, FFA-LoRA and FLoRA. For FedCyBGD, the model is partitioned into 10 blocks, with each block assigned to corresponding client, and clients perform fine-tuning soquentially. For FedBCD and the proposed ParaBlock, the model is partitioned into 16 blocks for Llama 3-8B and 14 blocks for Llama 3.2-3B. The block partition is based the default layer of the language model. During each global round, the server randomly selects
described in
7

---

Published in Transactions on Machine Learning Research (03/2026)
block for fine-tuning, We conduct 32 global rounds for fino-tuning Llama 3-8B and 28 rounds for fine-tuning Llama 3.2-3B for all BCD-based and LoRA-based baselines, and we conduct 3 global rounds for Fed full FT. We adopt AdamW as the local optimizer, i.e, conducting local block training via AdamW, as it is the default optimizer for most of LLMs training and fine-tuning. The default effective batch size in our experiment is set to 4. The random seeds for all libraries is 42. All experiments are conducted on NVIDIA A100 GPUs. To fully support reproducibility, our code will be released later.
6.2 Main results
Results on general instruction following task We begin by evaluating the performance of the proposed ParaBlock on the general cnttructionfolllwwiin dataset, Alpaca GPT-4 (Peng et al., 2023). for language model fine-tuning. We report the MT-bench score for evaluation. As shown in the results for Alpaca-GPT4 in Table D ParaBlock achieves lower score than Fed full FT but consistently out putporfssms most LoRA-based and BCD-based PEFT methods across two models. For the Llama 3-8B model, ParaBlock achieves an MTTbbench score of 5.14, which is the same score as the FedBCD, and ParaBlock shows substantial improvement over FedCyBGD, another federated block-coordinate method. Additionally, ParaBlock significantly outperforms LoRA-based FL, methods such as FedIT, FFA-LoRA and FLoRA. A key highlight of ParaBlock its superior runtime efficiency, as it requires outstanding less runtime compared to other baselines, particularly LoRA- based FL methods, while achieving notable performance improvements. Similarly, when fine-tuning the lightweight Llama 3.2-3B model, ParaBlock achieves the best performance among all baselines. Its ability to surpass competing methods in both performance and time efficiency highlights the effectiveness of ParaBlock in fine-tuning LLMs for general instruction-following tasks.
Results on mathematical reasoning task We evaluate ParaBlock on mathematical reasoning tasks using the Mathinstruct (Yue et al. 2023) dataset, with GSM8K (Cobbe et al. 2021) as the benchmark for evaluation. As shown in the right main columns of Table [ ParaBlock outperforms all bascline methods and achieves the second-best runtime on the 8B model, and outperforms most baselines with the less runtime on the 3B model. For 8B model, ParaBlock demonstrates improvements over FedBCD, while for 3B model, it is slightly less accurate than FedBCD, This indicates that while the communication thread of ParaBlock exists one round behind, it does not compromise overall performance. ParaBlock also consistently exhibits strong performance compared to other baselines, with a notable 11.7% improvement over FedCyBGD when tun-tunnng the lightweight 3B model. Furthermore, ParaBlock keeps its advantage of runtime saving among all basclines, highlighting its ability to roduce communication latency for federated BCD schemes and achieve communication efficiency in fine-tuning LLMs.
Time efficiency We conduct a detailed comparison of the runtime and emphasize the time efficiency benefits achieved by leveraging commuiication-compttaaaii parallelism, as illustrated Figure 2 To evaluate the necessity and benefits of this scheme, we measure wall-clock time across different network bandwidth conditions (50M/s, 100M/s, and 150M/s) and varying effective batch sizes (2, 4, and 8) on each client. D Among the LoRA-based federated fine-tuning methods, FFA-LoRA requires slightly less runtime than FedIT, while FLoRA incurs significantly more runtime than FedIT. Due to space limitations, we include only the detailed runtime comparison results for FedIT here, with a comprehensive discussion of all methods provided in Appendix A
Figure 2 aa illustrates the runtime when fine-tuning Llama 3-8B model using Alpaca GPT-4 dataset under various conditions. In scenarios with low network bandwidth and minimal client-side compulation, such as a bandwidth of 50M/s and effective batch size of 2, the communication time cannot be fully overlapped by client computation. This results significantly extra communication overhead for ParaBlock, represented by the dark yellow portion of the histogram in Figure aa.. As the effective batch size grows, requiring more client-side computation, a greater portion of the computation time can be overlapped with communication. We notice that for faster communication networks (100M/s and 150M/s), the computation time for ParaBlock does not introduce extra communication during fine-tuning. The negligible dark yellow portion in Figure represents the final communicatin process, as described in Line 12 of Algorithm 2
Due deployment constraints, we can only simulate the commmication bandwidth with three network bandwidth conditions, which were frequent settings in FL and decentralized learning.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Training Loss
18-73 2014
2011
PALIIC 241611
010000 D4000
(a) Llama 3-8B
(b) Llama 3.2-3B
(c) Training loss comparison
Figure 2: Time efficiency: wall-clock runtime for various network communication bandwidths and effective batch sizes.
Figure Z(a) also demonstrates that ParaBlock significantly reduces runtime compared to other basclines. Notably, we emphasize the time savings achieved over block-coordinate baselines, including vanilla FedBCD and FedCyBGD. The overall runtime reduction can exceed 30% for network bandwidths of 50M/s and 100M/s with an effective batch size (ebs) of 2, with efficiency gains over FedCyBGD being particularly pronounced. It is important to note that the higher runtime costs for FedCyBGD can be attributed several factors. First, FedCyBGD employs cyclic update approach that selects only one client at time, inherently prolonging the fine-tuning process. Additionally, FedCyBGD introduces extra communication overhead, all clients must periodically synchronize to receive model updates. Thus FedCyBGD's design results in increased latency and contributes to the overall longer runtime observed in our evaluations.
The runtime savings over the LoRA-based FedIT are also notable. Figure 2 aa indicates that ParaBlock achieves comparable, and slightly better, time efficiency than FedIT under conditions of low computationa ional cost and high communication overhead (e.g., 50M/s network and ebs=2). This is primarily due to the additional communication time associated with ParaBlock. Morcover, under most other settings, ParaBlock consistently reduces runtime by approximately 40% compared to FedIT. This improvement is attributed to the communicttion-compuaaa coordinate method.
parallelism of ParaBlock and the computational efficiency of the block
Figure b) illustrates the fine-tuning runtime of the Llama 3.2-3B model under conditions similar to those previously described. In addition to highlighting the superior time efficiency of ParaBlock, we observe that parameter communication constitutes larger proportion of the total runtime during the fine-tuning of the Llama 3.2-3B model. This is reflected in the increased size of the dark yellow portion in Figure 2 (), which represents greater share of the overall runtime. These underscore the critical need to reduce communication latency for clients. By mitigating the impact of increased communication overhead, ParaBlock achieves even greater time savings. These results further demonstrate the effectiveness of ParaBlock in enhancing fine-tuning efficiency by minimizing the influence of communication delays.
We verify the convergence of the proposed ParaBlock through the training loss as shown in Figure 2 c)- From an optimization perspective, ParaBlock and FedBCD exhibit very similar convergence behavior, with ParaBlock showing only marginally slower convergence than FedBCD in the initial stages, while this gap diminishes significantly in later rounds. Note that the primary distinction between FedBCD and ParaBlock lies in the global model of ParaBlock, which introduces onc-step staleness. Nevertheless, the results from Figure indicate that this staleness has a minimal impact on the fine-tuning convergence of ParaBlock.
6.3 Ablation studies
We analyze several aspects of the proposed ParaBlock, including: 1) How many blocks should ParaBlock use to balance utility and time efficiency? 2) Is there any block scheduling strategy can achieve better fine-tuning results? 3) How does the data distribution among clients impact the fine-tuning performance? We further study staleness beyond one round, extend the analysis beyond LLaMA architectures, and evaluate under cross-silo partial participation. Additional results are provided in Appendix A
9

---

Published in Transactions on Machine Learning Research (03/2026)
Ablation for block partition In our experiments, as pre- viously mentioned, we partition blocks based on the default layers in language models. To investigate the impact of block partitioning, we explore different configurations by varying the number of layers within each block in the proposed ParaBlock approach. As shown in Table 2 we evaluate configurations where each block contains partial, 1, 2, or layers during fine- tuning of the Llama 3-8B model both general and math tasks. The results in Table 2 indicate that the runtime gets longer as there are more layers assigned to one block. Note that the increase in runtime encompasses the growth of both computation and communication runtime, and we notice that assigning 2 layers per block yields slightly better performance compared to other configurations in both tasks.
Table Ablation for the block assignment. the number of layers.
Models
MT-Bt RT (T(m)) GSM8K1 RT (m)
Partial layer
5.12
20.3
53.22
15.0
layer
5.08
20.4
53.90
15.1
2 layers
5.14
21.1
55.88
15.8
layers
5.13
22.7
55.04
19.4
Ablation for block scheduling We investigate how various block partitions would impact on the overall performance. We compare the default random scheduling which randomly selects two layers based on the default layers in Llama 3-8B, the sequentially, reverse sequentially layer scheduling and gradient-guided scheduling based on the original gradient. As shown in Table 3 we find that Ran- dom scheduling can achieve the best result in math reasoning but performs slightly worse than gradient- based in general instruction tuning. However, the gradient-based approach incurs extra computational cost, as it requires a full-model backward pass prior to fine-tuning in order to determine the scheduling.
Ablation for heterogeneous distribution on clients We compare the i.i.d. and non-i.idd. data partitions when fine-
Table 3: Ablation for block schoduling
Models
MT-Bt
GSM8K1
tuning the general instruction dataset Alpaca-GPT4 (Peng
et al. 2023) and math reasoning dalaset Math Instruct Yuue et al. 2023), as shown in Table 4 We further conduct exper-
Random
5.14
55.88
Seq.
5.03
54.21
Rev. seq.
5.06
54.59
iments under an extreme non-i.i.d. sctting with Dirichlet(0.01) applied to both datasets. Our observations indicate that i.i.d. data sampling consistently results in higher evaluation scores for both general instruction tuning and mathematical reasoning tasks.
Gradient based
5.19
53.60
Ablation for the number of staleness rounds The one- round staleness in the original ParaBlock can be readily ex- tended to multi-round staleness settings. For example, if we assume that all clients in the network experience lower com- munication bandwidth, leading to parallelism with two rounds of staleness, we obtain the following results as shown in Table 5 It shows slight performance degradation when all clients ineur two-round staleness; however, the performance remains comparable to the base model score.
Table 4: The results for fine-tuning LLaMA 3-8B two datasets, considering ii.d., non- i.i.d. and extreme non-i.i.d. partitions.
Data distribution i..d. Non-ii.d.
MT-Bt
GSM8K1
5.21
56.14
5.14
55.88
Extreme non-i.i.d.
5.13
54.66
Experiments beyond Llama model architectures We conduct mathematical reasoning experiments using the same MathInstruct dataset as in the main experiments with the Qwen-2.5-1.55 I.nssut model Team 20044), and evaluate the finc-tuned models on GSMSK. The results Table d demonstrate that ParaBlock consistently outperforms other basclines in terms of accuracy, while also achieving the second-best runtime, highlighting the effectiveness of our design.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Fine tune 2.ven-2.5-1.5B-Insttie model MathInstruct dataset. We high- light the best score in bold and the second- best score with an underline.
Table Fine tune Llama 3-8B model on MathInstruct dataset with 50 clients cross-silo settings.
Method
GSM8Kt
RT(m)
Base Fed full FT
54.28
Method MT-B f
RT(m)4
GSM8K1
RT(m)4
60.58
9.3
FedIT
5.06
49.31
53.75
44.15
FedIT
62.32
19.76
FedBCD
5.08
55.36
53.60
52.16
FFA-LORA
61.56
17.12
ParaBlock
5.10
28.45
53.90
28.44
FLoRA
57.77
52.42
FedCyBGD
54.51
42.01
FedBCD
61.79
18.48
ParaBlock 62.70
10.56
Extension to cross-silo partial participation settings ParaBlock is fully compatible with partial participation, and we have conducted experiments demonstrating its effectiveness under such scttings. Specifically, we consider a cross-silo sctup where 20% of 50 clients are randomly selected each round, and we fine-tune the Llama-3 8B model on Alpaca-GPy) 4 and Math Instruct dataset. We compare this cross-silo ParaBlook with FedBCD and FedIT. As shown in Table 7 ParaBlock outperforms FedIT and FedBCD while retaining its computation time-saving advantage.
Table 5: Ablation for the number stalc- ness rounds.
Method
GSM8K1
RT(m)4
Base
51.55
one-round staleness
55.88
15.8
two-round staleness
54.74
16.1
Extension to heterogeneous communication bandwidth The proposed ParaBlock is natural to extend to heterogencous communication bandwidth. For example, assume one client was significantly slower than the others. This extremely slow client incurs three-round staleness, while other clients maintain the original one round of staleness. The results in Table illustrate this scenario. Note that in this heterogeneous bandwidth sctting, slow clients with signíficant latency may degrade overall performance. We believe it requires non-trivial design for better performance and conclude this future work.
Table Fine tune Llama 3-8B model on MathInstruct dataset with heterogeneous communication bandwidth.
Method
GSMSK1
RT(m)4
homogeneous bandwidth (original)
55.88
38.72
heterogencous bandwidth
52.46
20.16
Discussion on the comparison with asynchronous FL baselines ParaBlock is significantly different from asynchronous FL, although both share the motivation addressing communication delays and efficiency. The efficiency asynchronous FL comes from letting clients update the server at their own pace, so it may happen that one client is conducting communication while another is computing gradients. ParaBlock, however, sets up two-thread parallel scheme for every single client. Thus, it may not be fair to directly compare asynchronous FL approaches with ParaBlock, they designed for full-parameter training, Nevertheless, we compare one representative asynchronous FL bascline, FedBuff (Nguyen et al. 2022), with ParaBlock for fine-tuning the Llama 3-8B model on mathematica reasoning tasks. In a setting where out of 10 clients experience extreme delays, and the FedBuff algorithm is applied, the GSMSK score drops to 52.24, which is significantly lower than ParaBlock's score of 55.88. We believe this degradation due the impact slow clients in FedBuff. While asynchronous training remains promising direction, would require non-trivial redesign to be effective.
11

---

Published in Transactions on Machine Learning Research (03/2026)
7 Conclusion
In this paper, propose ParaBlock, communicationn-comptaaabb parallel block coordinate method for federated BCD to enhance communication efficiency. To better support this two-thread parallel method, we adjust the local model initialization and global model update based on the FL-BCD schemes. We theoretically show the convergence rate for the proposed ParaBlock under general non-couvex settings, which indicates our design does not sacrifice the convergence of the standard FL-BCD. We perform extensive experiments on diverse models and tasks to empirically assess the effectiveness of ParaBlock. The results show that ParaBlock significantly improves communication efficiency while achieving performance comparable to standard FL-BCDD baselines, highlighting its effectiveness in enhancing efficiency in FL deployments.
Acknowledgments
We thank the anonymous reviewers for their helpful comments. This work is partially supported by the National Science Foundation under Grant No. 2348541. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
References
AI@Meta. Llama 3 model card. 2024. URL https///s/tttthccc//////////////
CARD.m md]
Sara Babakniya, Aluned Roushdy Elkordy, Yahya H Ezzeldin, Qingleng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. Xiv preprint crXi::2308.033 2023.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogencous language tasks and client resources. arXiv e-prints, pp. arXiv-2402, 2024.
Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM journaal on Optimization, (T4):2037-2066) 2013.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open 11m leaderboard, 2023.
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXio preprint ar arXiv:2007.1439, 2020.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ar 2rXic:2005.14166 2020.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models. arXiv preprint arXiv:2401.06432 2024.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXio preprint Xrxi::21 10. 168, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint ar axXiv:2407.211888 2024.
Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen, and Qiang Yang. Fodmkt: Federated mutual knowlegge transfer for large and small language models. orXiv preprint arXiv:2406.0222 2024.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe. Local SGD with periodic averaging: Tighter analysis and adaptive synchronization. Advances in Neural Infor Information Processing Systems, 32, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ar preprint arXiv:2106. 096655 2021.
Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, and Michele Zorzi. Sparse random networks for communictioon-effiiant federated learning, arXiv preprint arXiv:2209. 15328, 2022.
Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, and Yonina C Eldar. Adaptive quantization of model updates for commmicationn-fffcinn federated learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3110-3114. TEEE, 2021.
Yuang Jiang, Shiqiang Wang. Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning edge devices. IEEE Transactions Newral Networks and Leurning Systems, 34(12):10374 10386, 2022.
Richeng Jin, Yufan Huang, Xiaofan He, Huaiyu Dai, and Tianfu Wu. Stochastic-sign SGD for federated learning with theoretical guarantees. arXie preprint Xix: 10940, 2020.
Sai Praneeth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Nikolas Koutsoubis, Yasin Yilmaz, Ravi P Ramachandran, Matthew Schabath, and Ghulam Rasool. Pri- vacy preserving federated learning in medical imaging with uncertainty estimation. arXiv preprint ar.iv:2406. 12815, 2024.
Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-effiienn personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 42-56 12-55, 2021.
Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi, Shijie Xu, Weihong Luo, Yubua Li, Xiuqiang He, and Ruixuan Li. Fedbat: Commuicction--ffiiian federated learning via learnable binarization. arXiv preprinl ar Xrv::2408..000 03215, 2024.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 4400,,45,440 2020.
Xiang Li, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Commuicatioon-ffiienn local decentralized SGD 2019.
methods. Xiv preprint ar
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. Fednlp: Benchmarking federated learning methods for natural language processing tasks. arXiv preprint arXiv:2104.08811 2021.
Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong. Fedbegd: Commuicctiion-effiienn accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 2955-2963, 2024.
Yang Liu, Yan Kang, Xinwei Zhang, Liping Li, Yong Cheng, Tianjian Chen, Mingyi Hong, and Qiang Yang. A communication efficient collaborative learning framework for distributed features. arXiv preprint xrXiv:1912.11187, 2019.
Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter training method for large language models. arXiv preprint 2ixx:2444.0282) 2024.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communictiion efficient, learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp- 1273 1282. PMLR, 2017.
Yu Nesterov. Efficiency of coordinate descent methods on hugo-scale optimization problems. SIAM Journal on Oplimization, 22(2):341 362, 2012.
John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference Artificial Intelligence and Statistics, pp. 3581 3607. PMLR, 2022.
Rui Pan, Xiang Liu, Shizhe Diao, Renjic Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficiet large language model fine-tuning, arXiv preprint ar Kiv:222003 17919, 2024.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint orXiv:2304.022 2023.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization, In International Conference Learning Representations, 2021.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fed- paq: A communicttion-effiiaen federated learning method with periodic averaging and quantization, International Conference on Artificial Intelligence and Stalistics, pp. 2021 2031. PMTRR 2020.
Guangyu Sun, Umar Khalid, Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, and Chen Chen. Conquering the commuiictiio constraints to enable large pre-trained models in federated learning. arXiv preprint arXte:2210001700 2022.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in arXiv preprint arXio:2403.1220 2024.
federated learning.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models a practical size. arXiv preprint arXiv:2408.0011 00118, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlnn github. io/ blog/quen2.5/
Paul Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of oplimization theory and applications, 109:475 494, 2001.
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020a. URL
https::/openrevieww http://wwww.....e.ooooo.a/aaa/a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020b.
Lin Wang, Zhichao Wang, and Xiaoying Tang. Save it all: Enabling full parameter tuning for federated large language models via cycle black gradient descent. arXiv preprint 3rXiv:2406.11189 2024a.
Shiqiang Wang and Mingyue Ji. A lightweight method for tackling unknown participation probabilities in federated averaging. Xixiv preprint ar KVv:2306.03401, 2023.
Yujia Wang, Lu Lin, and Jinghui Chen. Commuicatiion-ffiiinn adaptive federated learning. In Proceedings of the 39th International Conference on Machine Learning, pp. 22802 22838.. PMLR, 2022.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Yujia Wang, Yuanpu Cao, Jingcheng Wu, Ruoyu Chen, and Jinghui Chen. Tackling the data heterogencity in asynchronous federated learning with cached update calibration. In The Twelfth Internalional Conference on Learning Representations, 2024b. URL https:// http:://paaneeeeeenetftuuuoooood http://oooou//////aaaaararbr1
Yujia Wang, Shiqiang Wang, Songtao Lu, and Jinghui Chen. Fadas: Towards federated adaptive asynchronous optimization. arXiv preprint (47::2407.18663 2024c.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine- ane-tuning large language models with heterogencous low-rank adaptations. arXiv preprint arXiu:2409.05976 2024d.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongleng Huang, and Xing Xie. Communicaiion-ellciieenl federated learning via knowledge distillation. Nature communications, 13(1) 2022.
Huiwen Wu, Xiaohan Li, Deyi Zhang, Xiaogang Xu, Jiafei Wu, Puning Zhao, and Zhe Liu. Cg-fedllm: How to compress gradients in federated fune-tuning for large language models. arXiv preprint arX1v:240511374 2024.
Ruiyuan Wu, Anna Scaglione, Hoi-To Wai, Nurullah Karakoc, Kari Hreinsson, and Wing-Kin Ma. Federated block coordinate descent scheme for learning global and personalized models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10355-103622 2021.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-III) federated learning. In International Conference on Learning Representations, 2021.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 1137-6147, 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist. models through hybrid instruction tuning. orXin preprint ar XrXxx2300..0 X6552309.05653, 2023.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1015-6919, IEEE, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wi, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-ajuude with mt-bench and chatbot arena, 2023.
15

---

Published in Transactions on Machine Learning Research (03/2026)
A Additional Experiments
A.1 Additional Results
Additional experiments on multilingual settings We conducted multilingual experiments by fine- tuning both the Spanish and the original English Alpaca-GPT4 models, and evaluated them on the ARC challenge. Due time and space constraints, only compare ParaBlock with FedBCD here to demonstrate that Para Block can still achieve performance comparable to FedBCD.
GPU memory consumption We first present the peak GPU memory consumption for all baselines Table when fine-tuning on the Alpaca-GPT4 dataset (Peng et al. 2023). The results indicate that ParaBlock and FedBCD incur the lowest GPU memory costs among the fine-tuning methods. FedCyBGD exhibits relatively higher memory cost compared to FedBCD and ParaBlock due to differences in block assignment. Furthermore, the proposed ParaBlock consistently consumes less memory than LoRA-based methods.
Table 9: GPU peak consumption when fine-tuming the Alpaca-GPT4 dataset.
Methods
Llama 3-8B
Llama 3.2-3B
FedIT
27.0G
14.3G
FFA-LoRA
26.8G
14.2G
FLoRA
27.0G
14.3G
FedCyBGD
29.6G
13.9G
FedBCD
23.3G
10.0G
ParaBlock
23.3G
10.0G
Orthogonal to existing communication efficient FL methods Previous studies in FI. have explored various communccttion-cffiienn techniques, such as model/update compression, quantization, and prun- ing (Reisizadel et al. 2020 Haddadpour el al. 2019 Wang el al. 2022 Jiang et al. 2022). These approaches primarily aim reduce the number of communication bits, thereby improving the communication efficiency in FL systems. In contrast, our proposed PlocBlock targets the latency inherent in the communication process, making ParaBlock orthogonal to existing commuicatioo--ecluttie methods. In Table 10 1)we compare ParaBlock with existing communication--eedeeee for cases where the computation time cannot overlap the communication time, we can further apply existing communication-rcauutii methods to improve the communication efficiency.
method applied to FedBCD, and 2) we show that
The top part of Table 10 shows that ParaBlock achieves superior performance compared to directly applying top-k compression (as utilized in (Wang et al., 22022] Li et al. (2024)) with top 20% ratio to the standard FedBCD bascline, while also requiring less runtime. This is because top-k compression, despite reducing communication bits, still necessitates transmitting compressed model at each global round. In contrast, ParaBlock directly reduces the communication latency, resulting in better overall performance than applying compression to the vanilla FedBCD baseline. Moreover, ParaBlock is compatible with existing communica cinunicaaion reduction techniques, as shown in the bottom lines in Table IO By further integrating top-k compression, ParaBlock can effectively reduce the extra communication time with still achieving reasonable performance in both tasks.
Hyper-parameter details We conduct learning rate searches to find the best learning rate for each baseline. We perform a grid search over the learning rate 77 from (3e-7, 1e-6, 3e-6, 1e-5, 3e-5 }, and the global learning rate 7) for all experiments. The extra hyper-parameters for AdamW optimizer is following the default parameter Trainer, i.c., ,1 0.9, B2 0.999, e 10-6, Table summarizes the learning rates in experiments.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Table 10: Comparison with Top-k compression methods, where MT-B is the abbreviation for MT-Bench.
Method
MT-B Runtime(m) GSM8K1 Runtime(m) J
100M/s, ebs=4
FedBCD
5.14
30.2
54.74
24.9
+Tpp-20%
5.00
26.4
54.12
21.1
ParaBlock
5.14
21.1
55.88
15.8
+ +Top-20%
5.11
21.0
55.27
15.6
50M/s, ebs-2
FodBCD
5.03
30.0
54.66
+Top-20%
4.99
22.5
53.90
19.2
ParaBlock
4.99
19.4
53.53
19.4
+Top-20% 20%
4.98
11.6
54.14
11.6
Table 11: Learning rates in our experiments
Alpaca-GPTI GPT4
Math Instruct
Llama 3-8B
Llama 3.2-3B
Llama 3-8B
Llama 3.2-3B
FFT
3e-7
1e-7
1e-7
1e-6
FedIT
3e-6
3e-7
3e-6
3e-5
FFA-LoRA
3e-6
3e-7
3e-6
3e-5
FLoRA
3e-6
3e-7
3e-6
3e-5
FedCyBGD
1e-5
1e-6
3e-6
3e-5
FedBCD
1e-5
1e-6
3e-6
3e-5
ParaBlock
1e-5
1e-6
3e-6
3e-5
B Theoretical Analysis
B.1 Additional Discussions about Assumptions
The block-wise smoothness property could be naturally implied by the general smoothness of objective function. Given the non-negativity of the norm operation, there is [V,)(0,) - Vef(@2)l < ([v) - vf(e2) = L 0zll. We adopt the general smoothness for convenience and notational clarity. Alternatively, if we assume block-wise smoot hiees:: for each block b, there is (+) v.f(0,) & Lo|l0 - 02]1, and with L = max, Ly, the convergence analysis can be modified accordingly. The convergence rate will maintain O(1/VT) but depend on instead of L.
Discussion about Assumption 5.1
The block-wise heterogeneity naturally follows from the original bounded heterogencity in Assumption 5.2 as well. Using the property of partial derivatives, Vof(0.) (O,)),)), and following the argument in Lemma. C.3, have
Discussion about Assumption 5.2
R
N Σ (V)(0) V/((0) < oo²
[V), -
(7)
N
Therefore, for simplicity, we adopt general bounded variance assumption on the full gradient. Moreover, if instead assume bounded variances for each block, the convergence rate of 0(1 ((1/VT) remains valid.
17

---

Published in Transactions on Machine Learning Research (03/2026)
B.2 Convergence Analysis
For the global model of two consecutive sleps, there
12+1|6, [0s]b, n-
(8)
For Å, 00, 00,...,, A.; 0, :00, where |Ar, A.. Given the fact that [VS(0))] VoS(O:), for each time step
-
= -]]
-
L 11| 0.1121
<E(V) 0ut1 -0,) +
2
L B|lln
= =V](Vv,, n1:)]
1 L
TB((V..(0..) A.)
B
17
(9)
where the first inequality follows Assumption 5.1 For the first term I, there is
and the second equation holds by [V)()))] Vof(0.) and
Eq. 8
1 = (V (O.). A.
ITEE ((.)) A. + (,V.
= 10 A. +
(10)
Then
(v, A. + kK bf ((.)
mK Σ
Vo, ((.)) ]
(O.), Σ
A
N
K-1
N
V (0.), -N
1/
Σ
Σ
gi.k
M N
Σ v(.)
1=1
N K-1
K N
n7
1 (0.)
1/I N
fi
N
1
K=
(Vnk (0),)
V
N K-1
VnK
ΣV v vo,
NK
N K-1
Elll E. (0.)11
+
mA
Σ
2
2N
N K-1 Σ
V v. f: 17.
m
E
(11)
2
K=
18

---

Published in Transactions on Machine Learning Research (03/2026)
where the third equation holds by the unbiased-ness of stochastic gradient, and the last one holds by the fact of (a, b) 2||aa|| + ||b]2 + laa - blla]. For the second term in Eq. 11 there is
K-1
Σ
-
-v,,((0.))
2N2
N K-1
V. - Vo.f(00)111)
2N ==1 km0
Σ
Bll 0.11
(12)
2N
k=D
Therefore, for the whole 1 term, we have
Wik
L
I < V
1(0))
E f(0.)
0.11
2N
k=0
N K-1
TM7
V
b. f((
2N2K
k K
mpth 1,2
N K-1
N K-1
|| (0,)
0.11
E
V
fi ( (0
(13)
2N
2N2K
A==
where the equation holds by Lemma
C.3
Note that the model 0it the k-th local step model for update block by at time step t, thus the previous b block has not been updated yet, i.e.,
0 LocalBlockTraininng/ (i.- 7h k),
0.0
0 0 + A 1-1 + A 2-2 1 1-2
0, + A 1
and for the global model,
0, = 0-=0--1 + 7
(14)
then
0 - 0.11]= = E|| - 0 + 0 - 0.1
< 2
0 2 2 0.11
(15)
19

---

Published in Transactions on Machine Learning Research (03/2026)
where the first term consists of & steps of local updates, while the second term includes the updates difference when updating the 1 block. For & = 0,.... K - 1, we obtain
E E[|00, 00:01 [ 0i 0l, []²]
B[ 07,0 mgi [1
= B[ - -0, - m (9 - Vo.Ji(0) + - + Vo,
Vo.f(Of,o)) + Vo. (0",)))
< + 2K
E [ - 10 [1 + [1 - (b.
+ 6K m V 6, (:(0
Vo.
- V vor
+ 6KIE|| m bb, f / (0
1 2k + 1Z
E Bill0 1 10 [1 + 11/0²
+ 6k m b.. (((0 - Vo. (or
1 + 2k + 6kn L2 B110 00,0111 11 + nff0
1
+ (.T.BI//(V......... ((00 - Vo (00 6KE V (()
(16)
then by taking average over all clients i E [N],
N N
1
B B 1.k
01,01
1+
+ 6K
- 0%,01111+ + n/20
N
2K -
1=1
6km
+ 6Knjo
Σ E[| b,, f(t
(17)
N
Since 1y S SKL.
unrolling the recursion, then we have
N
1
El - 0%
N
1 )"( 11002 + 6K
6K
Σ
1 + K -
Σ B / (0
N
p=0
<(K (K-1) 11 11(111 |+
K
6K N
-1 1
17 + 6K o
N Σ
K -
30K N
< +30X2
Ell V ((0
(18)
N
for the last item, we have
N
1 N
E
E[Jl b, (( - ((.) + Vb.
N
BIIV - V., 110))1 + Σ , (0.)
N
2T
Bi|lo 0.11 + V ((0,)l
(19)
N
20

---

Published in Transactions on Machine Learning Research (03/2026)
where there is
0 0c11² -E11 n (-1 (0 + n7:-1)111
= -1 f-1 - 1-1
S 2 -
B11 - - A |]]
(20)
Merging itemss together, then we obtain
N
2 N
2 N
1I - 0.1195
Σ
Bi]00 - 01,01
Σ
Bi|10,, - 0,11
N 1=1
N
N
1=1
60K m N
2
$ 110K) 60K2"103
E|ll V (0
(00
Σ Bll00 0.1131
N
N
i=1
< 10K1f00
60K 1 9
120K n/ F|l V / (0:)1
12 L2
N
N
0+1
N Σ
Bill 0 0.
N
4 +
<
120K E v., (0)))
E 0
N
2=1
817² Σ
o0
8
A 2-
BIIA (21)
Therefore, reorganizing the / term, we obtain
K
E E b, (0,,)
L2
N
K
El 0,119
TP F
Σ
N K-1
27
2N
2N²
k=
mp
E E by ((0))
+ mpा 2 K
5K 5Kni02 + 30K22 + 60K E bb, 11
2
4n² N + 1n72E| 1
K
Σ
E||| E|llAi1111))
MM 2N2K
N
1=1
k=0
(22)
21

---

Published in Transactions on Machine Learning Research (03/2026)
Summing up Eq. (9)
[[S(Or)] ((0)) 1n T-1
T/L T
EV, ((Or). A.)]+
El|| 11
t=
mm K
B[[V b. (0.)) + mp L2 K
5TKKn K 02 33TK3" oo
T-1
41² T-I N
600KK n
B. [(,)] + 41/
Σ
B|| +
ΣΣ
E (
N
t=0
N K-1
17]
1 2 T-1
R
+
[B[| |
2N2 K
2
K=0
t=
mm T-1
T-1
E
Vl.. 19 + 60K mm L
E
V f..
1=0
T
+mmLL " 3OTK 1203)) + mnn'm
Σ B م 1
1,2K T-1 N
T-1
N K-1 ΣI
17.
Bi| 1-111
mp
Σ
V (B
(23)
N
2N2
0=
By Lemma C.2
the inequality becomes
Ki T-1 1
T-1 B[ 5, (0.)]1
B(((Or)] S(00) <
60 mn ([
2
1=
1=
T
1.2K + +p 1.2 1 + 30TK331003
EE| |P
2
EI EI MAK
+ 4nji' L
Bll E(I
+ 4np211 2 (2T 2 0² + 20T ml L2 12
ΣE
K-1
17.
mp
f.O
(24)
2N
k=D
22

---

Published in Transactions on Machine Learning Research (03/2026)
By condition on learning rates, i.e. m 22KL and m I 4K
E[f(Or)] - f(0b) - mm
V
+ On I2 + T²L)
Σ
E |19]
t=0
+ mn L2 K (2T K of 20T K 1/ L3 + 120T
K-1
V by f((O mn) 1'1
mp
E
2N2K
MA
ElI f (0. 1| + m. mm +30TK3
+ 8n m + L TKm 0²
2 N
+ Inn'm o2
20TK 1f + 120TK>1(I'03)
(25)
Then,
Bll f(0))11
m [f(00) [[]((())]]]]]]]] + 10 L2
Snn'n nT Tm 0²
2
+ L2 + 20T + Loo
(26)
Dividing by T, there is
Σ
Bil
V b. (0,))1
mmTK [f(()) [[f(Or)]] + 40K 240K3 +2
T
+ (512 + n T N
2
+ 320 L22 L2 + 20 L + (f
(27)
With F f(0o) - f- and f. = ming 1(0) > -0, and there is f(0)) B[ S s(00)) - f. = F, then
T-1
Σ
B b. (0)1) < TK [S(On) - [B] + 40 220K2
T
1-0
+ m.2 + 7 " n²2 2
+ 32m L (2K2 + +0K 1// L2 + 120K 1 L'og)
mTTK F+ + 1/ L'003
S K nT. m
2 No²0
+ +32 1,2 (2K + 20K 11 11'022 + 22
8F
+ L' + +KKO))
"mTK
+ *(Si'n
nb
" _ + K + + +2 (28)
N/=
2
23

---

Published in Transactions on Machine Learning Research (03/2026)
This concludes the proof of Theorem 5.3
B.3 Extension to Local Adaptive Optimizer
The theoretical analysis of the proposed ParaBlock is not limited to the local SGD setting. Essentially, the main differences between the convergence analysis under SGD and adaptive optimizers can be summarized follows:
The local updates A are aggregated to A the server. Hence, the most crucial part of modifying to AdamW is to deal with these A terms.
For A in Adam, there is A 0i 07 Σ =k=I (0
Thus there is
N
N
N
K
4
Σ
00,
07,00
=
0 - 0 x11
N
N
N
i=1
k=1
N
m
T
N
+
N
K
mi,
m
1i +
N
K
mi,k
K
3:)0
gi.s
1=1
N
K
-
gg.
(29)
therefore, by similar theoretical analysis in Lemma C.1 we have
N
K
r
E
A
1
(1 BK
k+ N Vo.f(((),)) + V.
N
0
N
K
E
(1-37 -
Miin
Vo, Vo.fi(Oi.N)
K-
8k 3K- Vo. Vo.
Ki
N K
J. 17
d² +
V V
(30)
Nc
N26
i=1 k=0
The properties about bounding Σr- N Lemma
EN
B||| [|| would be also similar to the analysis in
In nutshell, adopting local Adam achieves the same convergence rate of 0(1/VT) SGD.
24

---

Published in Transactions on Machine Learning Research (03/2026)
C
Supporting Lemmas
Lemma C.1. The global update parameter A
1 N
Ei=
A
satisfies
Km
V E
K -1 Σ
E1| A 11
S
0
V
b. (( k
(31)
N
N2
-=
k=
Proof. By definition,
A
E
M
Σ
Pi,k
K
M
igL. - Vo..((t) + F.
0=0
K
N K-1
|
27
v..f((00
+
V Ny (
k=0
Km
Vb, J((i 1
(32)
No
where the third equation holds by the unbiased-ness of the sLochastic gradient, and the inequality holds by Assumption 5.2
Lemma C.2. The global update parameter A
satisfies
Σ Σ
E 11
B[| V b, ((:))1
2,
1=0
+ o2
20 |/
120T K3 n 1,200
(33)
Proof. By definition,
--1
E
A
E
k=0
E
7
[g
K 1
igi - vo.f((0(.))) >//
+E mm
k Σ
E
V f((i
km0
K
17.
K Knj 1000 + BE
Σ
V 1
(34)
R==
25

---

Published in Transactions on Machine Learning Research (03/2026)
where the third equation holds by the unbiased-ness of the stochastic gradient, and the inequality holds by
Assumption 5.2
K=
]
E
Vb (((0,,))
k=0
Vb, Vo,))( |
E
Vo, (:(0 Vo,l((0 + Vb,Ji(0f,o)
+ Vo.1((0)) ((O,)
11]
121
K-1
V ( 1'1
2E
Vo (100k Vo.r((0;.)) 0000
+ 2E
k=0
k=0
K-1
|
< 2K
L
1 0
+ +
Vo.
(35)
k=
where
30 N N
1
Bl|l0
01 <5Kn/0² + 30
Ell V f( 10
(36)
N
and
N
2L N N
1
f(0
E 0.11 + 2E| Vb, v.,
(37)
N
where there is
E
0
2 1-111
11
27/ 1111/
(38)
Then
N
N Σ
K-1
(.(. 11
13] = Knod 02
M N
E
V
2
N
K-1
2k N
S <n +
Σ
E 1.k
01.0111
V
F. f(
N
N
1=1
k=0
(00K") L2
2k
<Kn002 + 10 60K 12
V ((0 1(0,,)))1
N
12
4K I,2
S <KnR0² + 10 60K 4 11 L'
Ell0i, - 0.112]
N
+ +(20K 1/2 43
[0
< Kn + 10K m| L - 60 L30
L2
EI Aill 1-11
(240K*
8K 13 A || (120K 1/ LL +4K vI,
(39)
First we previously assume that 1/ S SKL: and also (1) for simplicity, if we have a sequence x, S af + 0/1-1 B2, C, then we have
S S a0 0y:-1 3zz C
$ Salazi-+ a 3 + C) + ath-1 + B2g + C
< <
+
a y1 Σ
3z, +C Σ
1=1
26

---

Published in Transactions on Machine Learning Research (03/2026)
(2) for simplicity, if we have sequence Fı I a21-1 av/11 B2g + C, then we have
It S 01
+ 03/11
+ B2g + C
T-1
T-1
T-1
T-1
It 5 a
11-1 +
1/1 + 3
4
+++
1=0
t=0
T-1
T-1
T-1
T-1
Σ
31 2
Σ
31 + 0
Σ
11-1 +
Σ
4 CCT
1=
1=
t=0
T-1
T-1
T-1
(1 0)
Σ
Σ
1/-
+ 3
21 +T
t=0
1=
t=0
Σ 34 S a(1 - a) Σ
1t-++ +1-1+
Σ cx + C(1 -a)- +T,
we want that 2 < a <a 1, which means < a S 2 therefore, we have 1< <1 - a1 S Morcover, since
a = 24 1 S 2 have 240K 240K 8K2 S 2,
Σ
T-1
Σ
U1 + 23
Σ t=0
24 + 2C T
t=
1=0
(40)
N
Σ
Σ
E 111
E|| || + (240K 8K2 Σ
E (.
2T + 20T m 02 + 120T tp L'o²
E||l
B ((.))
2,
2T 2T 0² 20T n 02 + 120T n
(41)
Lemma C.3. For 0 = |0 02 00....,0 i.e., there is a block partition b = 1,2,.... B partitioned 0 into B
blocks, then we have ||0| Σ |
Proof.
| 1102112 + +..++1|081
Σ (2l1112) + Σ (22.92)) +.. ( Σ (z B.g2
de
(x=)²= = |1012
(42)
Lemma C.4. For 0 = |0 ........................ 8 and = v2 ...,,,.,, i.e., there is a block partition b = 1,2, ..2,..., partitioned 0 and y into B blocks, then we have (0. 1 yy²)
B
27

---

Published in Transactions on Machine Learning Research (03/2026)
Proof.
0 4²) + (02, y²)
+... 108 y
d1
dg
dB
y
2 y
2. 2.
+
r
B B,
y
1=1
i=1
[=1
d
Σ
r'y' (0, v)
(43)
28


Reviews:
[{"review_id": "GgnQTuOJ7r", "reviewer": "Reviewer_TsJE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes ParaBlock, a parallel client-side execution framework for federated block coordinate descent in LLM fine-tuning. By overlapping local computation and communication via dual threads and introducing a staleness-aware correction mechanism, the method reduces idle waiting without altering the core optimization structure. The authors provide convergence guarantees comparable to standard federated BCD and demonstrate that ParaBlock achieves lower wall-clock training time while maintaining competitive performance across multiple LLM fine-tuning tasks under varying communication conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claims are generally supported by clear experimental results showing reduced wall-clock time and comparable task performance to federated BCD baselines. The algorithm and correction mechanism are described clearly, and the efficiency gains are consistently demonstrated under different bandwidth settings.", "audience_yn": "Yes", "audience_explanation": "The paper would interest parts of the TMLR audience working on federated learning, LLM fine-tuning, and communication-efficient optimization, as it addresses a practical and timely systems bottleneck.", "requested_changes": "- All experiments use only 10 clients, which is far from realistic federated learning settings with hundreds or thousands of clients. Consequently, the claimed benefits of ParaBlock in large-scale FL are not fully supported, and the effect of one-round staleness at scale remains unclear. Moreover, the evaluation assumes fixed, homogeneous bandwidths (50/100/150 Mbps) and does not consider network heterogeneity or stragglers. Additional clarification would strengthen this part.\n- The convergence analysis assumes local SGD, while all experiments use AdamW, resulting in a theory–practice mismatch that should be clarified or addressed.\n- The baselines mainly include LoRA-based methods and FedCyBGD, but recent asynchronous federated learning approaches also address communication waiting. Including such baselines would provide a more comprehensive evaluation.\n- The authors should consider improving the reproducibility of the work by providing additional experimental details, such as bandwidth simulation, client configuration, and random seeds.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gx19SG0y1b", "reviewer": "Reviewer_WqtU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present ParaBlock, a federated supervised learning method particularly useful for very large models such as LLMs. For such large models, communication of the parameters can cause a significant slowdown for existing federated learning methods.   ParaBlock uses concurrent communication and computation threads to speed up the overall learning process so that communication and computation overlap better. Appropriate algorithms are designed for updating the current block as well as \"catching up\" the previously updated block at each round. Theoretical asymptotic convergence results in terms of the number of global rounds are obtained. Largely superior performance results are obtained in terms of speed and accuracy in comparison with competing methods on some LLM benchmarks. Ablation experiments are also presented.\n\nThe paper is certainly relevant to today's cutting edge AI concerns, although the utility of federated learning for LLMs could be better explained (I will elaborate in a later section). Industrial-strength LLM models are now enormous and any speedup algorithms should be of widespread interest. \n\nThe method makes a lot of intuitive sense and the experimental results are reasonably thorough and convincing, although the details of the 'full fine tuning' benchmark and the interpretation of the comparison in speed and performance between 'full fine tuning' and ParaBlock needs more explanation.\n\nThe asymptotic theoretical convergence results are of debatable significance, in my opinion, since it's not clear to me whether there are enough global rounds T to make asymptotics relevant.", "claims_and_evidence_yn": "Yes", "claims_explanation": "It's honestly borderline for me between 'yes' and 'no'. Some clarifications are needed before publication, as I will explain in the next section, but on the balance, I would say yes.", "audience_yn": "Yes", "audience_explanation": "There should be plenty of interest in this paper. Very large scale LLMs are of widespread interest and speedup is of their training is of great interest.", "requested_changes": "The 'full fine-tuning' benchmark and its performance and speed as compared to ParaBlock needs to be better explained.\n\nIs 'full fine-tuning' no federation at all, i.e., the entire model being trained with full, granular access to all the training data? If there's no federation at all, why would it ever lose to ParaBlock, as it does in terms of both accuracy and speed on Llama3.2-3B in Table 1?\n\nSomewhat relatedly, at the start of page 8, the paper says ParaBlock achieves a lower score in the 'main middle columns' of Table 1. However, it's actually the first 2 columns on the left, for Llama3-8B,  not the middle ones, where ParaBlock gets a lower score. ParaBlock gets a higher score in the middle column, for Llama3.2-3B. This seems to be a typo that needs fixing, unless I've misunderstood something. \n\nPage 7 says that 32 and 28 global rounds T of fine-tuning were used. Theorem 5.3 and Corollary 5.4 have bounds in terms of T. Is T ~30 large enough to appeal to the asymptotic convergence behavior? The constants up front in equation 5 are pretty big (64, 40, etc). Sometimes, a bound that appears useful at first glance may not be that useful when you plug in realistic numbers. I would like more reassurance that the bound is useful given real-life values for the variables in the bound.", "broader_impact_concerns": "I'm not very worried here but I am slightly worried. I'm neither an expert on LLMs nor federated learning, but it's not intuitively obvious to me that federated learning can preserve privacy in an LLM in the desired way. There's clearly been a fair amount of work on federated learning for LLMs, judging from the paper references, so I guess I will trust in the research community here.  Still, it's not obvious to me that specific facts from text within one silo of the federation, where privacy is a concern, won't end up being regurgitated by the final LLM in a response to a user query. I don't see how you could guarantee that.", "additional_comments": "Some typos and suggested rewrites:\n\nP2 by cyclically elects -> by cyclically electing\nP2 schemes enhances -> schemes enhance\nP 2 have demonstrated to enhance -> have been shown to enhance\nP3 descent, it. -> descent. It (new sentence needed)\nP3 substituting the SGD optimizer to BCD -> substituting BCD for the SGD optimizer\nP4 are proceed -> proceed\nP5 Alg 2 step 7 client I sends, waits (not send, wait)\nP5 This demonstrated -> This demonstrates \nP7 and an lightweight -> and a lightweight\nP3 Approximates to optimize -> approximately optimizes\nP10 various block partition-> various block partitions", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ca22P23CFB", "reviewer": "Reviewer_gZ8W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to parallelize training and parameter exchange in federated learning to reduce latency caused by exchanging large parameter vectors over a network. The parallelization is enabled by a block gradient descent scheme, where each training run between parameter exchange only trains on a subset of the parameters.", "claims_and_evidence_yn": "No", "claims_explanation": "The main theoretical result, Theorem 5.3, differs from the result that is proven in the Appendix contained in the supplementary material. The result in the Appendix does not contain the constant $\\mathcal{F}$ but instead depends on the difference between the parameter initialization and the expected value of the final parameters. Furthermore, the constants in the Appendix and in Theorem 5.3 do not match. \n\nThe explanation of the Algorithm is also flawed, in particular, Figure 1.", "audience_yn": "No", "audience_explanation": "Federated learning with large parameter models has become an increasingly relevant field since the advent of large transformer models.", "requested_changes": "- Fix Theorem 5.3 or the respective proof. \n- Figure 1 displays that the clients receive parameter updates downstream before having sent any parameter updates. That does not make sense in this setup.\n- The recursive step in Equation (3) is unclear. It is also unclear to me what this discussion is trying to show. If the goal is only to show that the local parameters in block $b_{t-1}$ are equal to the global parameters after two iterations, the discussion is unnecessarily convoluted.  \n- The presentation needs to be improved. Examples from the first page only:\n  * \"... the proposed ParaBlock achieves ...\" What part of ParaBlock are you referring to?\n  * \"Federated learning is known as a promising privacy-preserving solution\" ... solution to what?\n  * Gemma 2 was not published by an individual with last name \"Team\" and their collaborators, but the Qwen Team, without further contributors.", "broader_impact_concerns": "None at this point in time.", "additional_comments": "I recommend that the authors thoroughly revise the paper and resubmit.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)