Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
Authors: Yide Ran
Wentao Guo
Jingwei Sun
Yanzhou Pan
Xiaodong Yu
Hao Wang
Jianwen Xie
Yiran Chen
Denghui Zhang
Zhaozhuo Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Sparse Zeroth-Order Optimization
Federate Learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Federated Learning enables collaborative fine-tuning of Large Language Models (LLMs) across decentralized Non-Independent and Identically Distributed (Non-IID) clients, but such models' massive parameter sizes lead to significant memory and communication challenges. This work introduces Meerkat, a sparse zeroth-order optimization (ZO) method designed for federated LLM fine-tuning. By limiting fine-tuning to a transferable, static, extremely sparse subset of parameters, Meerkat achieves remarkable communication efficiency, enabling cost-effective high-frequency synchronization. With theoretical analysis and experiments, we show that this high-frequency communication effectively mitigates Non-IID data challenges and leads to superior performance compared to full-parameter ZO. Furthermore, experimental results show that Meerkat outperforms existing sparsity baselines with better performance at the same communication frequency. To further handle Non-IID drift, Meerkat leverages traceable local updates and forms a virtual path for each client. This virtual path mechanism reveals the GradIP phenomenon: the inner products between LLM pre-training gradients maintained by server and client gradients estimated via ZO converge for extreme Non-IID clients but oscillate for IID ones. This distinct behavior provides a signal for identifying clients with extreme data heterogeneity.
Using this signal, Meerkat-vp is proposed to analyze GradIP trajectories to identify extreme Non-IID clients and applies early stopping to enhance aggregated model quality. Experiments confirm that Meerkat and Meerkat-vp significantly improve the efficiency and effectiveness of ZO federated LLM fine-tuning.

Published as a conference paper at ICLR 2026
MITIGATING NON-IID DRIFT IN ZEROTH-ORDER FED- ERATED LLM FINE-TUNING WITH TRANSFERABLE SPARSITY
Sun', Yanzhou Pan', Xiaodong Yu', Hao Wang', Jianwen Xie Yiran Chen', Denghui Zhang', Zhaozhuo Xul
Yide Ran', Wentao
Steeenns Institute of Technology. Prrineeton University, UUiiversity of Florida, *Google LLC, LLmmdd, DDkee University
(yranl, xyu38, hwang9, dzhang42, zxu79) (steeessttteeeeeeeeeeeeeee
1990220@princet.om edu, sun. wiingeeeuull... edu, yanzpegoogle.com jianwen, xie0 ambdal. com, yiran. chen@duke.
ABSTRACT
Federated Learning enables collaborative fine-tuning of Large Language Models (LLMs) across decentralized Non- non-Independent and Identically Distributed (Non- IID) clients, but such models' massive parameter sizes lead to significant memory and communication challenges. This work introduces MEERKAT, sparse zeroth- order optimization (2O) method designed for federated LLM fine-tuning. By limiting fine-tuning to transferable, static, extremely sparse subset of parameters, MEERKAT achieves remarkable communication efficiency, enabling cost-effective high-frequency synchronization. With theoretical analysis and experiments, we show that this high-frequency communication effectively mitigates Non-IID data challenges and leads to superior performance compared to full- parameter ZO. Fur- thermore, experimental results show that MEERKAT outperforms existing sparsity baselines with better performance at the same communication frequency. To fur- ther handle Non-IID drift, MEERKAN AT leverages traceable local updates and forms a virtual path for each client. This virtual path mechanism reveals the GradIP phenomenon: the inner products between LLM pre-training gradients maintained by server and client gradients estimated via ZO converge for extreme Non-IID clients but oscillate for IID ones. This distinct behavior provides signal for iden- tifying clients with extreme data heterogeneity. Using this signal, MEERKAT-VP is proposed analyze GradIP trajectories identify extreme Non-IID clients and applies early stopping to enhance aggregated model quality. Experiments confirm that MEERKAT and MEERKAT-VE significantly improve the efficiency and effectiveness of ZO federated LLM fine-tuning.
INTRODUCTION
Federated Learning (FL) (McMahan et al. 2017) has emerged as a powerful paradigm for enabling decentralized collaboration, particularly relevant for fine-tuning Large Language Models (LLMs) across numerous client devices (Dubey et al. 2024 Brown et al. 2020). Unlike centralized training, FL allows clients to train models locally and share only model updates with central server. However, fine-tuning LLMs in a FL setting faces two major challenges: the massive model parameter size and the Indeeedddenndtt and Identically Distributed (Non-IID) data distribution across clients. The former leads to high computation demands on clients and significant communication overhead, while the latter causes client drift and hinders global convergence. These challenges make LLM fine-tuning impractical on resource- resource-constraingg constrained clients and hinder the effective use of decentralized data.
Zeroth- Optimization (ZO) provides a promising avenue for addressing some of these challenges in federated LLM fine-tuning, By estimating gradients through model perturbations and forward passes, ZO bypasses the need for backpropagation and the storage of intermediate activations, leading
"Corresponding author; 7 www.tteeeeaaa....mm.oommmmm..... edu

---

Published as a conference paper at ICLR 2026
to more memory- efficre-effiieent learning client devices (Zhang et al. 2021 Fang al., 2022 Ling
jet al., 2024; Liu et al. 2024] Malladi et al. 2023). However, applying standard Zo directly to the massive parameter space of LLMs can still be computationally inefficient and the optimization process unstable (Malladi et al. 2023). Moreover, adapting Z0 for federated LLM fine-tuning remains challenging, particularly in balancing computational efficiency, communication overhead, and model performance under Non-IID data heterogeneity.
In order to address the above challenges, we propose MEERKAT, a sparse ZO method designed for efficient federated LLM fine-tuning. MEERKAT addresses the computational and communication burdens by focusing ZO updates on static, extremely sparse (less than 0.1%). and transferable subset of LLM parameters. This subset is strategically identified using gradients derived from pre- pre-training data, ensuring that updates target parameters most sensitive to the loss function. This selective approach dramatically reduces communication overhead and supports cost-effective synchronization. As we will demonstrate through theoretical analysis and extensive experiments, the combination of high communication frequency and sparsity in MEERKAT enables frequent yet lightweight synchronization. This effectively reduces the convergence error floor in theory and practice, leading consistently superior performance compared to full-parameter ZO fine-tuning and other sparsity methods under the same communication frequency.
Leveraging MEERKAT'S efficient high-frequee frequency synchronization to effectively mitigate Non-IID data challenges, we further enhance its adaptability to weak network conditions. By employing virtual path mechanism track client updates, enable the server analyze client training dynamics without accessing raw data, thus facilitating robust operation even when frequent direct communication is constrained. Within this virtual path, we observe the GradIP phenomenon, pattern revealed by the GradIP score, which computes the inner product between local client gradients estimated via ZO and server pre-training gradients. GradIP scores converge for Non-IID clients while oscillating for IID clients, serving as a clear indicator of data heterogeneity. Leveraging this insight, we propose MEERK AT- VP that introduces a virtual path client selection method to identify clients with significant Non-IID characteristics and apply early stopping, thereby reducing their adverse impact on the aggregated model and enhancing its quality.
In summary, this paper makes the following contributions:
Performance Improvement with Sparsity. Meerkat consistently outperforms full-parameter ZO optimization in both IID and NID-IID settings, demonstrating the effectiveness of our sparse update strategy. Extensive experiments show that Meerkat surpasses not only full aall-arameter ZO but also other sparse methods, such as LoRA and magnitua, achieving superior performance.
High Frequency Communication with Sparsity Can Lower the Error Floor. MEERKAT leverages extreme model sparsity reduce local computational memory. Exchanging scalar gradients drastically decreases communication costs, enabling high-frequency communication.
Traceable Local Updates and GradIP Phenomenon: MEERKAT leverages traceable sparse local updates and forms a viriual path. The virtual paths reveal the GradIP phenomenon: the inner product between LLM pre-training gradients maintained by server and client gradients estimated via ZO converges for extreme Non-IID clients but oscillates for IID ones. This distinct behavior serves as a signal for detecting clients with extreme data heterogeneity.
MEERKAT-VP: Early Stopping for Extreme Non-IID Clients. Leveraging the GradIP phe- nomenon via virtual path client selection, MEERKAT-VV VP effectively manages extreme Non-IID clients, by early stopping these clients to improve global model quality.
Theoretical and Experimental Validation. We present theoretical analysis and extensive ex- periments across diverse FL settings, validating the scalability and performance benefits of both MEERKAT and MEERKAT-VP.
2 SPARSE ZEROTH-ORDER OPTIMIZATION FOR FEDERATED LLM FINE-TUNING
This section introduces MEERKAT, sparse ZO method for federated LLM fine-tuning, and its upgraded version, MEERKAT-VE VP, which incorporates Virtual Path Client Selection (VPCS) strategy. This strategy leverages the traceable virtual path of client local updates to identify clients with extremely Non-IID data and applies early stopping to mitigate their adverse impact on global model convergence. We first introduce the technical details of MEERKAT, as illustrated in Figure] | and
2

---

Published as a conference paper at ICLR 2026
Clienda Sparss Modals
Device Local Environment
Clients
Sparso Mudel!
Stop Aggregate sperse models
(Devires)
Device Dataset
Global Sparse
Step Celculate
sparse model
updade path
cllnnt
Sparse Model m
Ganerate Boed List
Cilents
Device
Projected Gradient
Projected Gradient
Initial Step
Sparso
Projected Gradient
Server
Model
Device
Proystted Gradient
Generate Sparse Mask
Mask
Step 1.2: Send To Server
Figure 1: MEERKAT: Sparse zeroth-order optimization for federated LLM fine-tuning workflow.
We then present theoretical convergence analysis for both methods and discuss their strengths in terms of cost- cost-effeeciveeess, traceability, and the use of early stopping to mitigate client drift caused by Non-IID data.
subsequently describe MEERKAT- vP, shown in Figure 5
2.1 MEERKAT: EXTREME SPARSE ZEROTH- FEDERATED LLM FINE-TUNING
Sparse ZO On-Device LLM Fine-Tuning, MEERKAT performs sparse ZO for LLM fine-tuning on the client device. Let D denote the client dataset we would like an LLM to fine-tune with loss function f. Given the LLM weight w E R2d we perform an iterative optimization by randomly sampling a batch B D for each step and performing the local update step as
f(w + e(z & m); B) - f(w - e(z @ m):B)
- V5=g(zōm)
(I)
2e
where z E Rdd is a random vector sampled from a Gaussian distribution (O, Ia). c E R is the perturbation magnitude, and m E {0, 13 is binary sparse mask with density ratio u that selects subset of parameters for updates.
Extremely Sparse Parameters Obtained from Pre-Training. According the formulation in equa- tion I we focus the perturbation of the LLM on a subset of parameters determined by binary mask m. The mask m derived from the pre-training process of the LLM. We compute the average squared gradients of each parameter over subset of the C4 dataset (Raffel et al. (020).. Then, we select the top u parameters with the highest average squared gradient values and mark them as in m. In practice, set to 0.1%, resulting in extremely sparse updates.
FL with MEERKAT. The workflow of MEERKAT illustrated in Figure and Algorithm 2 MEERKAT first loads each client with the pre-trained weight wo and the sparse mask m. Next, MEERKAT initializes a random seed list {st...., of at the server to generate the random Gaussian vector for each local step in the first round. Next, MEERKAT performs iterative federated optimization with rounds of client-server synchronization with each round as follows.
(1) Local ZO update at each client. Upon receiving global model weights w,-1 and seed list (s......s? sT} from the server, each client performs local iteration steps. In each local step f, the client perturbs the model parameters selected by with the random vector z generated by the random seed sy. Each client then computes projected gradient o (a scalar) according to equation
Using of. each client calculates the local gradient V fí and updates the local model w2 with learning rate After T local steps, each client uploads list of projected gradients server. (2) Server reconstructs client update with virtual puth. Since the server shares the same random seed list with clients for the round, it can reconstruct each client's 's local model update path upon receiving their projected gradicnts. We term this server-side reconstruction process the virtual path, as it allows the server to follow the client's local steps without accessing raw data. As shown in Step 2 of Algorithm? 2 the server uses the preserved random seed and receives project gradients of each local step from each client to recover the local model update path for each client. (3) Server aggregates and initiates the next round: After virtual path reconstruction, the server aggregates the reconstructed client model weights w sparsely update the global model to w,, Subsequently, the server sends w, and new seed list (8 (81415..-,54 .411 to clients and initializes next round.
g2...... ge } to the
3

---

Published as a conference paper at ICLR 2026
MEERKAT-VP: Virtual Path Client Selection and Early Stopping. MEERKAT-VP extends MEERKAT by incorporating a VPCS strategy designed for heterogeneous environments. Lever- aging the virtual path reconstruction capability, the server analyzes client update trajectories to identify those with extremely Non-III data distributions. MEERKAT-VE then applies an early stop- ping mechanism these identified clients, restricting them to single local step mitigate the negative impact of their skewed updates on global model convergence and performance.
2.2 THFORETICAL CONVERGENCE ANALYSIS
We theoretically analyze the convergence of MEERKAT and MEERKAT-VP under the Polyak- Eojaskee (PL)-type non-convex condition. All technical assumptions and the corre- sponding proof are presented in Appendix C
Theorem 2.1 (Convergence rate of MEERKAT). Under Assumptions C.1 C.6 if the learning rate
satisfies 7) = min L(u+2)' 227(2+ S. then the global model {w") generated by the MEERKAT
algorithm satisfies the following convergence bound:
1 - R p=0
(2 u)² TR
T
(B((wW)) - [(w + 0
01)
(2)
2 2
Theorem 2.2 (Convergence rate of MEERKAT-VP). Under Assumptions C.1 C.6.C.
if the learning rale (2+1)2 12277 and each client Kτ performs T 1 local step
satisfies 1) min ((w42) 1 a (K, T+K,)
while the remaining Kg clients perform T local steps, then the global model {w"} generated by the MEERKAT-VP algorithm satisfies the following convergence bound:
Ex [f(w") - 1 5 (Kg + Kb)² (2 (2+
c(KTT + +Ko)² k.)
O 1+
K, Σ
Kg + Kb
cT cTKg
+ O (Kg K₂)(1 + u) T7y
c cKn o
+
+ O1).
(Kg + 1,)(1+ +u)
(3)
The detailed theoretical analysis and proof's for Theorem
2.1 (MEERKAT) can be found in Ap-
pendix C.4 and for Theorem 2.2 (MEERKAT-VP) in Appendix
Insights of MEERKAT. MEERKAT'S convergence reveals the intricate interplay of local steps T and density u on performance. (1) MEERKAT'S sparsity can theoretically improve performance. Lower u (higher sparsity) quadratically benefits the rate-dependent term (xc (2 + u)²), favoring faster initial convergence. However, it inflates the steady-state error (ox 2 Comparing to the full-parameter case (u 1), sparsity (u < 1) can reduce the overall bound by decreasing the rate-dependent term, offering communication and computational benefits. Yet. excessive sparsity can increase the steady-state error, suggesting optimal density level (0, 1]. (2) High frequency communication with sparsity can lower the error floor. Increasing T improves the transient term
scaling with O (2+w)²² ). potentially accelerating convergence towards the steady state; however, expands the steady-state term OX 2.))) thereby increasing the error floor. Conversely, decreasing T reduces the steady-state term, leading to tighter final accuracy. Although smaller T can lead to larger rate-dependent term. It's impact diminishes the number of rounds f increases. This analysis suggests that operating with frequent communication can thcoretically reduce the steady-state error.
TH
Ad Advantaees of MEERKAT-VP. We compare each component of the error bound under the same and R. First, the transient term ratio between MEERKAT-VP and MEERKAT is approximately 7 > O, the product 7(1+ ) - 0, causing the transient error /2
y11+ ) < 1, and as с₁
=(1+v=x)² which remains below whenever u(1 + 2 2. Since μ < 1 empirically, this condition typically holds. cKio, Moreover, MEERKAT-VP introduces additional variance term 2k(2-u]/T= that decays as O(ITT), making it negligible for large local steps. Lastly, in terms of heterogeneity, the coefficient of the
to vanish. Second, the noise term ratio is given by o3/(m([+vv))))
4

---

Published as a conference paper at ICLR 2026
heterogeneity term Eg AS in MEERKAT-VP is smaller: scales inversely with K, thus diminishing in larger systems. Therefore, EMEERKAI-VP < EMBERKAT and this gap widens data heterogeneity en increases. The detailed mathematical derivations and analysis, please refer to the Appendix
(2+u)L < K. and the extra variance term 4k
2.3 CLAIM 1: MEERKAT CAN OUTPERFORM FULL--PARAMTTE FEDERATED ZO UNDER SAME SYNCHRONIZATION FREQUENCY
We claim that with fixed and extreme sparsity, MEERKAT outperforms full-parameter ZO in federated LLM fine-tuning under the same synchronization frequency and effectively mitigates the Non-IID client data problem through frequent synchronization and sparsity.
Ad Advantages of Sparsity in Federated ZO. ZO has intrinsic need for sparsity due to its reliance on nearly uniform perturbations across dimensions. Research on ZO shows that selecting sensitive parameters using gradient-hasee -haseddt--aedddd--e methods consistently outperforms alternative strategies such as weight magnitude or random parameter selection (Giuo et al.] 2024). Following this idea, MEERKAT produces LLM-sensitive parameters with gradient-based sparsification on pre-training data such as C4 (Raffel et al. 2020). Moreover, MEERKAT fine-tunes LLMs by estimating gradients through forward passes, completely bypassing backpropagation. This approach minimizes the need to cache gradients and activations, leading to significant memory savings. Focusing on sensitive parameters, MEERKAT ensures efficient and effective fine-tuning even under extreme sparsity levels (e.g., updating only 0.1% of the parameters). Furthermore, these sensitive parameters exhibit transferability across downstream tasks. Theoretical analysis (Appendix C.4) also confirms that lower density u leads to faster convergence via improved rate-dependent terms ( + u)² (TTR) while excessive sparsity increases the steady-state error O(T/(2 + u)), suggesting an optimal sparsity trade-off.
Performance Under High Synchronization Frequency. The lightweight communication of MEERKAT enables frequent client- server synchronization at a low cost, which is crucial for ad- dressing data heterogeneity (Yang et al.) (2024] Mendieta et al.) 2022) in FL. In high-frequency communication scenarios, both the clients and the server only exchange a list of scalars (projected gradients) whereas in lower-frequency synchronization, clients have upload projected gradients but still download sparse model parameters. By eliminating the need to download sparse model parameters in high-frequency synchronization, this approach is significantly more handwidth-efffcieet further minimizing communication overhead. We present the high-frequency synchronization algo- rithm of MEERKAT in Appendix ( Algorithm 3 By facilitating frequent synchronization, training can better prevent clients from drifting. Our previous theoretical analysis also demonstrates that smaller might influence the rate-dependent term, its beneficial impact on reducing the steady-state error is significant for achieving tighter final accuracy over many rounds R.
2.4 CLAIM 2: EMPIRICAL GRADIP PHENOMENON REVEALS DATA HETEROGENBITY
MEERKAT'S traceable virtual path allows us analyze client local training dynamies, revealing an empirical phenomenon related to data heterogeneity via metric we call GradIP.
Definition 2.3. Gradient Inner Product (GradIP) score: Let V VÍ (see Algorithm 2) denote the ZO gradient of LLM with equation I on client & at local step t. Let V denote the gradient of LLM
computed by backpropagation on pre-training data. We define the GradIP score SV V
GradIP As Indicator for Data Heterogeneity. Leveraging the virtual path reconstruction capability of MEERKAT, the server can trace each client's 's local training trajectory. This process uses the uploaded projected gradients o along with the shared random seeds (which regenerate zi)) and the sparse mask m to reconstruct the local gradient V.-. To understand the impact of client's local data distribution on its training process, we introduce the GradIP metric. Inspired by the use of pre-training data gradients to identify sensitive parameters, GradIP quantifics the inner product between the local gradient computed during client training and the LLM pre-training gradient.
Empirical GradIP Phenomenon. Through the traceable virtual path provided by MEERKAT, we empirically investigated the behavior of the GradIP score among clients with different data distri- butions (IID and Non-IID) over their local training steps. Our analysis, presented in Appendix C.66 demonstrates distinct patterns in the dynamics of gradient norms based on data heterogeneity. While
5

---

Published as a conference paper at ICLR 2026
IID client gradient norms exhibit fluctuations, those of extremely Non-IID clients decay and converge towards zero. The GradIP definition depends on the fixed pre- pre-traiming gradient norm, local client gra- dient norm, and the angle 0 between them. We hypothesize that between these two gradient vectors nearly orthogonal. This leads us to expect a different manifestation of the GradIP Phenomenon when comparing IID and extremely Non-III) clients, primarily influenced by their differing local gradient norm trajectories.
2.5 CLAIM 3: VIRTUAL PATH CLIENT SELECTION VIA GRADIP ANALYSIS
Building upon the traceable virtual path capa- bility introduced in MEERKAT, we claim that
Algorithm 1 MEERKAT-VP
VPCS, by leveraging GradIP analysis, effec- tively identifies and manages clients with ex- tremely Non-IID data distribution, thereby im- proving global model performance and conver- gence. As established in Section 2.4 the GradIP score, computable by the server through vir-
I: Input:
calibration step Teal, pre-training gradients fes, projected gradients .... Ds sparse mask m, initial phase steps Tint, later phase steps Tiste. convergence threshold a, Initial to later
Tot seed st.
ratio Plater. quiescent step ratio Pasie
2: Step 1: Virtual Path Reconstruction & GradIP Calculation
tual path reconstruction, provides an effective signal identify such clients. VPCS utilizes
3: Generate z using sy.
Compute V = gi (z) ② m)
this GradIP signal to detect extremely Non-III) clients. By analyzing the GradIP score trajec- tory and its behavior over local steps during
5: Compute Gradip V fca (Definition 2.3
6: Step 2: Identify Extremely Non-IID Clients
7: Compute the average value of Gradip over the initial-phase steps.
calibration phase, using metries defined in Ap- pendix ta the server empirically identifies clients exhibiting the characteristic diminishing GradIP behavior associated with extremely Non- IID data distribution. Upon identification via GradIP analysis, VPCS applies early stopping: these clients perform only one local training step per communication round. To ensure full data utilization over training, a data pointer tracks the batch processed, allowing clients to resume from that point in subsequent rounds. This strat- egy mitigates client drift from skewed data while ensuring their entire dataset is eventually pro- cessed. Algorithm outlines the detailed proce- dure, and Figure illustrates the workflow. Our previous theoretical analysis of MEFRKAT-VE AT-VP suggests that early stopping extremely Non- IID clients can lead to improved global model performance.
The
Gradip
Σ
Gradip,
Timit 1=1
8: Compute the average value of Gradip over the later- phase steps.
Gradip inter__vg
Gradip,
Tlater
9:
Compute the client's Initial to later ratio Plater client and quieseent step ratio Pqúe dient
{ f1, 2, ..... Taser }
Gradip, < <o}
Pauie ciient
Tiater
Gradip avg
Prater_ diiet
Gradipmner_owg
10: Record client IDs whose Plster cient or Pauie cient CX- ceed Plater Paula-
11: Step 3: Early Stopping Require these identified clients to only perform one local training step.
EXPERIMENT
In this section, we aim to validate the effectiveness of MEERKAT and MEERKAT-VP, We aim to address the following research questions in response to claims in Section 2 (1) RQ 1 for Claim 1 (2.3): MEERKAT more effective than full parameter federated ZO under the same synchronization frequency, especially in heterogeneous environments? (2) RQ 2 for Claim 2 (2.4): Can the empirical GradIP phenomenon, observed via the virtual path, effectively reveal data heterogeneity by showing distinct behaviors for IID and Non-IID data distribution clients?
(3) RQ 3 for Claim 3 (25): Can
MEERKAT-VP. leveraging GradIP analysis, mitigate the impact of extreme Non-IID data compared to MEERKAT?
We focus on models Gemma-2-22 Team 2024), Qwen2- 1.5B (Qwen Team 2024
Llama-3.
IB (Dubey et al. 2024). We conduct experiments SST2 (Socher et al. 2013). AG's News (Zhang ct al. 2015), Yelp polarity (yelp) (Zhang et al. 2013). RTE [Wang]] [ang]|2018 2018), BoolQ (Clark et al. 2019),
WSC (Levesque et al. 2012), WiC (Pilehvar & Camacho-Collados 2018 datasets. The datasets

---

Published as a conference paper at ICLR 2026
partitioned across clients following Dirichlet distribution to simulate clients with Non-IID data. For more experimental settings, we refer the readers to Appendix D.I
Table I: Performance comparison of MEERKAT and Full-FedZO on multiple non-IID data distribution settings. "Acc" is the average test accuracy across tasks. Bold numbers indicate the highest value in each row.
Methods
Local Step
SST-2
AgNews
Yelp
BoolQ
RTE
WSC
WIC
Acc
Full FedZO
0.909
0.940
0.641
0.542
0.634
0.523
0.699
Weight Magnitude
10
0.902
0.857
0.951
0.696
0.551
0,519
0.546
0.717
Lora-Feed2O
0.901
0.749
0.96
0.649
0.634
0.59
0.715
LLaMA-3.2-1B
MEERKAT
10
0.916
0.872
0.964
0.695
0.600
0.653
0.614
0.759
Fell-Fed70
0.904
0.706
0.935
0.636
0,533
0.634
0.539
0.698
Weight Magnitude
30
0.902
0,84
0.946
0.674
0,542
0,556
0.550
0,716
Lon--ed70
0.904
0.556
0.964
0.652
0,533
0.634
0.545
0.684
MEERKAT
30
0.897
0.862
0.965
0.646
0,577
0.644
0.583
0,739
Full-FedZO
0.889
0.696
0.935
0.633
0,542
0.634
0.529
0.694
Weight Magnitude
50
0.897
0.838
0.948
0.662
0,551
0,562
0.554
0.716
Lona-FeZZO
0.876
0.447
0.967
0.639
0,541
0.634
0.562
0.667
AT
50
0.909
0.827
0.965
0.647
0.595
0.634
0.567
0.734
Full-FodZO
100
0.901
0.705
0.939
0.632
0.533
0.634
0.525
0.695
Weight Magnitude
100
0.885
0.83
0.946
0.66
0.56
0.534
0.548
0,709
Lora-FodZO
100
0.868
0.247
0.953
0.642
0,521
0.634
0.529
0.628
MEERKAT
100
0.896
0.777
0.961
0.658
0.577
0.644
0.573
0.726
Full-FedZOO
10
0.888
0.700
0.928
0.694
0.808
0.673
0.639
0.761
Weight Magnituie
10
0
0.84
0.939
0.681
0.795
0.672
0.623
0.776
Lora-FodZO
10
0,939
0.847
0.944
0.667
0.795
0.663
0.521
0.768
Qweu2-1.5b
MCCRKAT
10
0,949
0,881
0.934
0.752
0.813
0.682
0.628
0,805
Full FedZO
30
0.892
0.699
0.926
0.708
0.791
0.663
0.594
0.753
Weight Magnitude
30
0.88
0.843
0.939
0.681
0.786
0.673
0.594
0.771
Lora-FedZZ0
30
0,923
0.843
0.948
0.666
0.777
0.673
0.519
0.764
MEERKAT
30
0,944
0,878
0.928
0.734
0.800
0.663
0.624
0.795
Full-Fed70
50
0.868
0.696
0.922
0.707
0.773
0.663
0.594
0.746
Weight Magnitude
0.883
0.855
0.938
0.703
0,768
0.673
0.595
0.774
Foma-Fed700
50
0.934
0.834
0.941
0.679
0.76
0.653
0.510
0.759
MEERKAT
50
0,948
0.872
0.926
0.746
0.795
0.663
0.594
0.792
Full-PedZO
100
0.864
0.691
0.917
0.675
0.777
0.653
0.620
0.742
Weight Magnitude
100
0,888
0.842
0.934
0.695
0,768
0.656
0.579
0.766
Lora-FedZO
100
0.934
0.785
0.937
0.664
0.786
0.653
0,512
0.753
MEERKAT
100
0.936
0.878
0.925
0.741
0.795
0.663
0.610
0.792
Full-FedZ0
10
0.928
0.721
0.943
0.731
0,564
0.644
0.595
0.732
Weight Magnitude
0.931
0.955
0.778
0,711
0.634
0.595
0.779
Lora-FodZO
10
0.936
0.853
0. 0.966
0.763
0.568
0.663
0.605
0.765
Gemmaa-2b
MEERKAT
10
0.939
0.869
0.96
0.804
0.591
0.634
0.609
0.772
Full-FedZO
30
0.927
0.802
0.932
0.725
0.568
0.634
0,581
0.738
Weight Magnitude
0.935
0.951
0.771
0.598
0.770
Lora-FodZO
30
0.952
0.804
0.966
0.671
0.551
0.634
0,589
0.735
MEERKAT
0.94
0.855
0.947
0.734
0.568
0.644
0.601
0.756
Full Fedl-FedZO
50
0.932
0.791
0.943
0.712
0.582
0.634
0.567
0.737
Weight Magnitude
0.936
0.941
0.745
0.591
0.628
0.597
0.756
Lora- FedZO
50
0.91
0.779
0.942
0.664
0.557
0.634
0.597
0,726
MEERKAT
0.945
0.857
0.767
0.634
0.623
0.772
Full FedZO
100
0.925
0.818
0.933
0.672
0.533
0.615
0.567
0.723
Weight Magnitude
0.922
0.942
0.723
0,568
0.644
0.592
0.747
Lora. FedZO
100
0.922
0.247
0.942
0.62
0.541
0.634
0.573
0.640
MEERKAT
0.94
0.951
0.745
0.551
0.634
0.574
0.749
3.1 ANSWER TO RQ1: SUPERIORITY OF MEERKAT COMPARED TO FULL-FEDZO FL
This section experimentally validates Claim 1 (Section 2.3
demonstrating MEERKAT'S superiority over fall-parameter Federated ZO under the same synchronization frequency and its effectiveness in mitigating Non-IID challenges via high-frequency synchronization.
First, to assess sparsity's 's benefits, we compare MEERKAT to Full-FedZO and other sparse methods (Weight Magnitude, LORA-FedZO), Random-Select) with equivalent synchronization frequencies (local steps T E {10, 30, 50, 100)). With a fixed 0.1% mask MEERKAT reduces communication budget by over 1000 compared to Full-FedZO and achieves a strong computational and communi- cation efficiency (Table Using C4 as a calibration dataset, our analysis shows that the sensitivity of the gradient is highly concentrated: the top 0.1% 1% of the parameters have 52x larger average square gradients than the next 0.1-1% bucket (Table 9. which motivates extreme sparsity. The mask is transferred across domain-shifted calibration datasets, and client-aggregated UnionMask
7

---

Published as a conference paper at ICLR 2026
performs comparably (Table Across IID and Non-IID data distributions, MEERKAT outperforms
10 13p. Under the same settings, MEERKAT also outperforms DeComFL (Li et al. 2024 (Table 19).
Full-FedZO and other sparsity methods on many tasks (Tables
Next, we evaluate performance under an extreme communication regime with a single local step 1T= We compare MEERKAT with Full-FedZO and LoRA-FedZO in the IID and Non-III data distributions (Dirichlet e (0.5, 0.3, 0.1)). Figure 2 presents the results for a = 0.5, the results for = 0.3 and 0.1 are available in Appendix D.2 figure 6 Specifically, Figure 2 reveals remarkable finding: on the Qwen2-1.50 model, MEERK K AT's average test accuracy over seven tasks under Non-IID data distribution matches that under IID data distribution. Beyond this exact match, results show that at a local step of T = L, MEERKAT effectively bridges the performance gap between IID and Non-IID data distribution settings, achieving nearly comparable test accuracy across both data distributions, and consistently outperforms baselines. Varying sparsity under T= 1 (Table 15) confirms strong aceuracy even 30-3-10-4, substantially reducing client memory demands and making it ideal for resource-constrained FL. These results support Claim 1: high-frequency communication combined with extreme sparsity mitigates Non-IID drift. We also explored sensitive parameter selection using downstream task data. Since performance remained comparable under identical communication frequencies and sparsity levels, we prioritized pre-training data to better preserve client privacy
(Appendix D.2 Tables 21 20 22
LLaMA-3.2-1B
Qwen2-1.50
Gemma2-20
0.62
0.8056.805
0.18
0.80
Non-HD
0.009
0.7856.763
Accuracy
0.70
0.767
0.763
0.76
0.756
0.749
0.74
0.741
0.737
0.726
0718
0.70
Ful-Fed20
LoRA-FedZO
Meerkat
FillFee22
LoRA-FedZO
Meerkat
Pul-Fed20
LoRA-Fad20
Meeraal
Figure 2: This figure compares three methods Full-FedZo, LoRA-FedZO, and MEERKAT on three ILMss: LLLMA-3.2-11 IB, Qwen2-1.55 1.5b, and Gemma2-20 -hbb. The x-axis shows the different methods, and each method has two bars indicating performance under IID and Non-IIID settings. The Non-III results are obtained under Dirichlet distribution with 0.5 ,he y-axis represents the average test accuracy across multiple downstream tasks-SST2, AgNews, Yelp, BoolQ, RTE, WSC, and WiC. All detailed results for these tasks are provided in Appendix D.2] Table 15
3.2 ANSWER TO RQ2: GRADIP TRAJECTORIES AS EFFECTIVE INDICATORS OF DATA HETEROGENEITY
This section experimentally validates Claim 2 (Sec- tion 2.4]. investigating GradIP trajectories as indicators of data heterogeneity. Based on our theoretical analysis as- suming single-label Non-IID data (Section C.6). study the dynamics of gradient-related metrics during local train- ing. We first compare two extremes: IID clients vs. clients with single-label (extreme Non-IID) data. We track three metrics: GradIP score, local gradient norm, and cosine value between the local and pre-training gradients. As shown in Figures 3 and 7 GradIP for extreme NID clients steadily decays to zero over 100 steps, while for II clients fluctuates persistently. To understand this, we analyze its components: Figure aa)) shows cosine value stays near zero (i.e., gradients are nearly orthogonal) for both settings, suggesting the gradient norm is the key fac- tor. Indeed, Figure B)) shows that the gradient norm mirrors GradIP's behavior across the two settings. More- over, in later stages, GradIP declines more sharply for Non-IID clients than for IID ones, making this stage-wise mean difference additional criterion for identifying Non-III clients. We further extend
Gradip Step
Gradie (ID)
Grade (Non-ID)
Gredip
400
0
100
Step
Figure 3: Under density ratio of 5 x 10 we track the GradIP (see Defini- tion 2 over 100 local training steps on the SST-2 dataset using LLaMA-3.2-1 model, comparing a client with IID data to client with Non-III data.
8

---

Published as a conference paper at ICLR 2026
analysis to more general Non-IID scenarios (Figure Figure 10 exhibits similar dynamics that correlate with the degree of heterogeneity.
Figure 11
where GradIP
3.3
ANSWER TO RQ3: VPCS EARLY STOPPING FOR CLIENTS WITH EXTREMELY NON-IID DATA DISTRIBUTION
This section experimentally validates Claim 3 (Section 2.5). As established in Section 3.2 GradIP trajectories provide an effective signal for identifying clients with extremely Non-IID data, exhibiting distinct behaviors. Leveraging this signal, VPCS detects extremely Non-III clients during calibra- tion phase and applies early stopping, limiting them to one local training step per communication round (Algorithm D. To validate the effectiveness of this VPCS strategy improving performance, we compared MEERKAT-VP with MEERKAT and Random Client Selection, which randomly selects the same number of clients for early stopping VPCS, under Non-IID data distributions Dirichlet = 0.5 and the same communication frequencies. Crucially, for the same model, dataset, and communication frequency, the three methods employed the same sparsity level. Figure + illus- trates the average test accuracy across multiple downstream tasks for MEERK AT-VP compared to MEERKAT and RANDOM CLIENT SELECTION. Detailed results for individual tasks are presented in Appendix D.2 Table 14
As shown in Figure 4 MEERKAT-VE consistently outperforms both MEERKAT and RANDOM CLIENT SELECTION in different communication frequencies. Furthermore, Table 25 shows that MEERKAT-VE VP achieves performance competitive with back-propagation up- per bound and significantly outperforms an adapted FedDYN (Acar et al., (021) baseline. These experimental results strongly validate Claim 3, confirming that VPCS effectively leverages GradIP analysis to manage extremely Non-IID data distribution clients, leading to improved performance for ZO federated LLM fine-tuning.
LLaMA-3.2-1B
Qwen2-1.5
Gemma2-20
Meerlat--p
Random Meerlal
8.50
0.90
79
0.78
Acturacy
0.76
6.37
0.74
0.23
0.72
0.70
Local Step
Figure 4: This figure compares three methods MEERKAT-VP. MEERKAT and Random Client Selection- across three LLMs: LLaMA-3.2-IB. Qwen2-1.5b and Gemma2-2b. The x-axis shows the local step values (10, 30, 50, 100), while the y-axis indicates the average test accuracy over multiple downstream tasks SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WiC-in Non-III data distribution setting. All detailed results for these tasks presented in Appendix D.2
Table 14
4 RELATED WORK
Our research leverages advances in ZO federated optimization, sparsity techniques for LLMs, and communication frequency adjustments strategies for addressing data heterogeneity. 70 methods significantly reduce computational and communication overhead. Integrating sparsity into L.M fine-tuning amplifies these benefits, substantially decreasing resource demands during training and inference. Concurrently, communication frequency adjustments mitigate performance degradation induced by Non-IID data, emphasizing a crucial trade-off between communication budget and global model performance. detailed discussion is provided in Appendix B
5 CONCLUSION
In this paper, we introduce MEERKAT, a sparse zeroth-order federated fine-tuning methodology. Experiments show MEERKAT outperforms Full-FedZO and other sparsity methods most tasks at

---

Published as a conference paper at ICLR 2026
equivalent communication frequencies. MEERKAT'S efficiency enables high-frequence communica- tion, effectively mitigating Non-IID drift. Moreover, we propose MEERKAT-VP. This methodology utilizes VPCS, which analyzes GradIP via virtual paths enable the selective early stopping of extreme Non-IID clients. This approach is shown to improve model performance. Our work thus offers effective methods for efficient 7O federated LLM fine-tuning under varying network conditions and data heterogeneity. Given the technical focus of this work on algorithmic contributions, there are no direct negative societal consequences inherent to it that need be emphasized; potential negative impacts would arise from the specific applications where these methods are deployed.
ACKNOWLEDGMENTS
We gratefully acknowledge the support of Lambda, Inc, and Stevens Institute for Artificial Intelligence, for providing compute resources for this project. This research also used resources of the Argonne Leadership Computing Facility. which is U.S. Department of Energy Office of Science User Facility operated under contract DE-ACC02-0CCH112 was supported by National Science Foundation awards 2451398 and 2450524. The work of Hao Wang was supported by National Science Foundation awards 2523997, 2534286, and 2315612. The work of Xiaodong Yu was supported by National Science Foundation award 2348465, This work was also supported in part by National Science Foundation awards 21 22562, 2328805, and ARO W911NF-23-2-0224.
The work of Yide Ran and Zhaozhuo Xu
ETHICS STATEMENT
This work follows the ICLR Code of Ethics. This paper aims to advance zeroth-order optimization for federated LLM fine-tuning by addressing key challenges related to efficiency and data heterogeneity. All datasets used are publicly available for academic evaluation. Our method is designed to protect user privacy by operating within the Federated Learning framework, where raw data remains local devices. Respecting the broader research community, we acknowledge prior work appropriately and ensure our contributions are situated within ongoing academic efforts. We declare no conflicts of interest or external sponsorships associated with this work.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work. The datasets and models used in this study are detailed in the experiments section (Section 3 The workflow for MEERKAY is presented in Figure [] with further details in Section 2 and Algorithm 2 The workflow for MEERKAT-VE is
demonstrated in Figure Algorithm I All experimental parameters are listed in Appendix D.]
Tables 4 g
The complete theoretical analysis for methods can be found in Appendix C
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization, 2021. URL
https: ottp/bs////lllaaaaaeb//o////// 04263
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning, 2018. URL BRL/https://araat http:://wwwwwwwwaa/////////s//// wwtp:///////p/bb/oooobaaa/// 04838
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901. 2020.
Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. MultiEURLEX multi-lingual and multi-label legal document classification dataset for zero-shot cross- lingual transfer. In Marie- Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6974-6996. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational
10

---

Published as a conference paper at ICLR 2026
Linguistics. doi: 10.188633/1122222.cmmm 2021. www.p-main..nn.... 5597
URL httpa//s....ttttp///////atttttt.
Jun Chen, Hong Chen, Bin Gu, and Hao Deng. Fine-graine theoretical analysis of federated zeroth-order optimization. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:/905, 10044, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint vrXiv2247..21783 2024.
Wenzhi Fang, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N Jones, and Yong Zhou. Communication- efficient stochastic zeroth-order optimization for federated learning. IEEE Transactions on Signal Processing, 70:5058-500 2022.
Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation federated learning, 2025. URL https::/atxxvv.og// abs 21100.410.114
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu. Xinyu Yang, Yide Ran, Jacob Gardner, Osbert Bastani, Christopher De Sa. Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-order fine-tuning of Ilms with extreme sparsity, 2024. URL URL https: http//aavvvv..e///e//eee/ea.a. org. abs
hbspg//ab//244666 2406. 02913
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification, 2019. URL URRptttss://axiiit http::::aaaxvvv..oo..
06335
Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, and Jianfei Chen, Pruning large language models with semi-structural adaptive sparse training, 2024a. URL htttts://arxiv..rtt www./orgggbbba.eeeee/... 2407 20584
Xinmeng Huang, Ping Li, and Xiaoyun Li. Stochastic controlled averagigg for federated learning with communication compression, 2024b. URL/tpps://axxxl.oo///////////
081
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2021.
URL httph:tts/sce
/tatp:://axxivv /ttx//axvvvoo 0ttp://bs/1911.06311
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference the principles of knowledge representation and reasoning, 2012.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning non-iid data silos: An experimental study, 2021. URL https:// http:::://xxixvv.rr.. wttp://wwwr//bb///222000000
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.
Federated optimization in heterogeneous networks, 2020a. BtRltttps///axiiv /atttpaa//axx.v.orr/bbbbss org/abs..r/b......
1812.0612
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg non-iid data, 2020b. URL http:::/aat.i...oooo
Zhe Li, Bicheng Ying, Zidong Liu, Chaosheng Dong, and Haibo Yang. Achieving dimension- free communication in federated learning via zeroth-order optimization, 2024. URL https
Tlaaxiv.orrg.. abs/////b///// 2405. 15861
Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the convergence of
zeroth-order federated tuning for large language models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1827-1833 2024.
11

---

Published as a conference paper at ICLR 2026
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order Ilm fine-tuning, 2024. URL https
7/arxiv. wwt.abs///aaooooooooe //axxi.org/abs// 2402. 15751
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidiss, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz, Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023a.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023b.
Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. SPP: Sparsity- preserved parameter-efficient fine-tuning for large language models. In Forty-first International Conference on Machine Learning. 2024. URL https: http:://opppeevveeeeee net woremt/fooummeeeeeoo ?idd
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53075. 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficieett learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Matias Mendieta, Taojiannan Yang, Pu Wang. Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808.09121, 2018.
Alibaba Group Qwen Team. Qwen2 technical report. 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11296-11300, IEEE, 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Junda Su, Zirui Liu, Zeju Qiu, Weiyang Liu, and Zhaozhuo Xu. In defense of structural sparse adapters for concurrent Ilm serving. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9948-4953, 2024.
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang. Daguang Xu, Yiran Chen, and Holger R. Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models, 2023. URL
hhttps://arxin> org/ htrgg//b/////
Gemma Team. Gemma. 2024. doi: 10.34740KK URL wtt.tttss//ww.keeeeeeee
com/m/ m/ com/m/3301 3301
Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.0804441111 2018.
12

---

Published as a conference paper at ICLR 2026
Jianyu Wang. Qinghua Liu, Hao Liang. Gauri Joshi, and H. Vincent Poor. A novel framework for the analysis and design of heterogeneous federated learning. IEEE Transactions on Signal Processing, 69:5234- 2021. doi: 10.1 10.10099TSP2222111100000
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:19/0.04732 2019.
Yuheng Wu, Wentao Guo, Zirui Liu, Heng Ji, Zhaozhuo Xu, and Denghui Zhang. How large language models encode theory-of-mind: study sparse parameter patterns. upj Artificial Intelligence, 1 (1):20, 2025.
Zhaozhuo Xu, Zirui Liu, Beidi Chen, Shaochen Zhong, Yuxin Tang, Jue Wang. Kaixiong Zhou, Xia Hu, and Anshumali Shrivastava. Soft prompt recovers compressed Ilms, transferably. In Forty-forst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http:::tt////t http://ppppeeee.eeeeeeeeeeeeeeep
/ net wttp://ett//oommm/m BBB.PIIIidm
Zhiqin Yang, Yonggang Zhang, Yu Zheng. Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Qingsong Zhang. Bin Gu, Zhiyuan Dang, Cheng Deng, and Heng Huang. vertical federated learning: New zeroth-order gradient based algorithm. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pp.
Desirable companion for
2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenging Zheng, Pin- Yu Chen, Jason Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang. Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. ]Mr
OpenReview.net, 2024a. / www.:/wwwweeeeeeeeeeeeeeeeeeeeee wereet/oouumicomm
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse Ilms, 2024b. URL https: /ttp///avvvv///////a// org abs/ 08915
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. 2018. doi: 1044850/ARRXI..186.0055 URL https://aaxii:
erg/abs. 1806. 00582
Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. arXiv preprint 20Xi:22402.061266 2024.
Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Xi V Lin, and Beidi Chen. Sirius: Contextual sparsity with correction for efficient llms. Advances in Neural Information Processing Systems, 37:24046- 24080, 2024.
APPENDIX
In Section A we discuss the usage of large language models in this work. In Section B we present the related work relevant to this study. Section C we present the theoretical convergence analysis of MEERKAT, including its high-frequency communication method. Additionally, we analyze the convergence of MEERKAT- VP and demonstrate its superior performance compared to MEER KT. We further prove that under extreme Non-IID settings, the gradient norm gradually vanishes during convergence, whereas in IID settings, it tends to oscillate. In Section D we provide details on experimental hyperparameters and report supplementary results.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used an LLM-based writing assistant solely for grammar and typographical corrections to improve the clarity of this paper. All outputs were carefully reviewed and revised by the authors to ensure technical accuracy and consistency with the intended scientific meaning. The intellectual contributions, methodological advances, and scientific insights are entirely original and author-driven.
B RELATED WORK
Federated Zeroth-Order Optimization. Zeroth-order optimization (Malladi et al. 2023 Zhang ct al. 2024a) has gained increasing attention in federated learning (Fang et al. 2022 Zhang et al. 2021) D. particularly for addressing challenges in training costs, privacy, and communication overhead. Fine-Grained (Chen et al. 2024) demonstrates how clients can reduce upload overhead by sending estimated gradients rather than full model parameters to the server, though download costs remain significant due to complete model weight transfers. DeComFL (Li et al., 2024) further advances this approach by using gradient scalars for both uploads and downloads, substantially reducing bidirectional communication costs. However, it does not address the challenges posed by data heterogeneity (Non-IID) in federated learning. The integration of AirComp wireless technology enables direct over-the-air aggregation of model updates (Fang et al. 20022.. In black-box settings where pre-trained language model parameters are inaccessible, FedBPT (Sun al. 2023) employs ZO to optimize prompt vectors, achieving efficient distributed optimization with reduced computational and communication overhead. FedMeZO (Ling et al. 2024) analyzes the convergence properties of ZO for federated LLM fine-tuning.
Sparsity in LLM. Current research on sparsity in LLMs explores techniques such as pruning,
contextual sparsity prediction, and structured sparsity (Wang et al. 2019 Liu et al. 2023b a. Huang
ct al. Zhang al. 2024b Zheng et al. 2024| Zhou et al. 2024). These methods enhance both training and inference by improving computational efficiency, reducing memory usage, and enabling deployment
2024a Lu et al. 2024 Shao et al. 2024, Wu ct al. 2025 Xu ct al.) 2024; Su et al. 2024
in cesource-constrainge environments. Sparsity has also proven particularly effective in zeroth-order (ZO) optimization (Guo et al., 2024, Liu et al., (0024) especially when combined with weight quantization for fine-tuning LLMss Building on this, our work investigates the role of sparsity in resource-frugal federated fine-tuning of LLMs.
figh-Frequency Communication for Non-IID Federated Learning. Data heterogeneity across clients is major challenge in Federated Learning, significantly degrading performance compared IID settings. Increasing communication frequency, by reducing local training steps per round, is explored as a strategy to mitigate this issue. Early work showed that merely reducing local steps had limited improvements in extreme non-IID scenarios (Zhao et al. 2018). Theoretical analysis later confirmed that smaller local training steps can improve convergence speed under Non-IID conditions, but at the cost of increased communication budget, highlighting a critical trade-off (Li et al.] 2020b). To effectively handle challenges arising from non-llI data that often necessitate higher communication, various algorithms have been proposed: SCAFFOLD (Karimiredyy et al.) 2021) highlights the 'client-drift" problem in FedAvg, noting it's exacerbated by increased local training steps (reduced communication frequency). and proposes using control variates to mitigate this drift, enabling improved convergence; FedDyn (Acar al., 2021) guarantees consistent convergence the global optimum even with a larger number of local training steps (lower communication frequency). This overcomes the limitation of traditional methods where high communication frequency needed to compensate for local-global optimum inconsistency. Empirical studies further demonstrate that performance is highly sensitive to the number of local training steps under different non-IID distri- butions, and the optimal communication frequency depends on the specific data heterogeneity (Li al. (02T). FedSA-LORA (Guo et al., 2025) tackles Non-III heterogeneity showing that the two low-rank matrices play asymmetric roles-A A learns global shared knowledge while captures client-specific patterns- and then aggregating only the matrices on the server while keeping the B matrices local, which reduces aggregation distortion and cross-client knowledge contamination under skewed client distributions. FedAvgM (Hsu et al. 2019) directly studies FedAvg under synthetic label-skewed Dirichlet partitions and proposes adding server-side momentum to miti- gate Non-IID elient drift: by accumulating past aggregated updates, the server smooths noisy, biased client gradients and recovers much higher test accuracy and more stable training even when client
federated LoRA by
14

---

Published as a conference paper at ICLR 2026
label distributions are highly skewed and only small fraction of clients participate in each round. Stochastic controlled averaging with compression (Huang et al. 2024b) addresses Non-IID data in compressed FL by building SCALLION and SCAFCOM control variates and local momentum keep local updates aligned with the global direction, and the authors prove (and verify empirically) that these algorithms remain robust to arbitrary client hetero- geneity and partial participation even under aggressive communication compression. Theesssoo.TTsse works underscore the complex interplay between data heterogeneity, local computation, and communication frequency. This complexity motivates the development of algorithmic solutions to improve efficiency and robustness in FL under Non-IID settings.
simplified SCAFFOLD backbone:
C THEORETICAL AND ALGORITHM ANALYSIS
C.I NOTATIONS AND DEFINITIONS
In this subsection, we formally define the assumptions, notations and concepts used in the convergence analysis of MEERKAT and MEERKAT-VP. Table summarizes
summaries the key symbols.
Table 2: Notations used in our theoretical analysis.
Notation
Meaning
w
global model parameter
K
total number of clients in the federated system
Pk
probability weight assigned to client k
fa (w)
total loss computed over all data samples of the client k.
[((w)
global loss function evaluated by the global model over all data
T
number of local update steps per communication round
r
communication round
t
local update time step
7
local learning rate
perturbation magnitude in ZO estimation
z
standard Gaussian vector for client at local step t from N (0.
binary sparse mask vector (m {0, 1)
m
d
model dimension
R
federated learning training round
u
sparsity ratio
gradient coverage
9k
projected gradient estimate for client & at local step
Vff ff
zeroth-order gradient of client is local step f
L
Lipschitz smoothness (Assumption I)
μ
PL inequality (Assumption 2)
f
minimal global loss achieved by optimizing the global model
ft
minimal client loss achieved by optimizing the local model on client A:
Ch and o
heterogeneity-indeee induced variance (Assumption 4)
op
operator norm of matrix
0²
variance of the sparse ZO gradient estimatoor/Assupple 6)
The clients with balanced data distributions contribute to the global model during training.
C.2 ASSUMPTIONS
We introduce the assumptions used in the convergence analysis of MEERKAT and MEERKAT-VP. Assumption C.1 ( (Lipschitz smoothness). We assume that each client k's local objective function f& (w) is differentiable and has T.-Lipschitz continuous gradients:
[v((w)) - V.(( < T.|[ wells Vw1, W2 E IRd.
Consequently, the global loss f(w) ΣK Ph. (w)) is also T--mooth.
15

---

Published as a conference paper at ICLR 2026
Assumption C.2 (PL inequality). We assume that f(w) satisfies the PojyseeLojasiewicz (PL) condition:
f(w) l 24
/V[()])? Vw E Rd
1 >> 0 is the PL constant. This condition holds for broad class of non-convex objectives and is commonly used in analyzing convergence of gradient-base based and zeroth-order methods.
Assumption C.3 (Global-Locce Disparities in Non-i.i.d Setting). For any 0 E R the discrepancy between the local and global gradient bounded by
V ff(o))l S ch | of
where с > and o 2 are constants, and 0 the global model parameter broadcast to clients at the start of each round. We further assume E (0. 1). In particular,
A smaller ca corresponds to lower data heterogeneity: local gradient deviations from the global gradient are small, indicating that client data distributions are nearly i.i.d.
A larger signals stronger non-ii.d data distribution effects, with greater variation between each client's gradient and the global gradient.
Assumption C.4 (Bounded stochastic gradient variance). For any sample (x. y) and any w Rdd denote (w; (zr,y) y)) the loss that single data point, and let f(w) (x,,)) [[w; ; (3r, v))] be the average full-batch loss. We assume
VV[(; v)) - ff(w)) - o².
Assumption C.5 Glocll-Global Optimality Gap). For each client k define the local-glos optimality gap as
A w
where w is the local optimal model on client i and w the global optimal model.
Assumption C.6 (Sensitive parameters are sparse). At each local step (and for every client k), there exists binary mask m {0, 1jd with exactly non-zero entries and constant E [0. 1] such that
Vik (w1: yx) = Ji (wi: (x)).
We further assume e > z meaning this small subset "sensitive" parameters captures a dispropor- tionately large fraction of the gradient norm.
These assumptions standard and foundational in optimization and FL literature (Bottou et al. 2018 Li al. 2020a b. Wang et al. 2021 Guo et al. 2024)
We start by formulating the expectation of the sensitive sparse ZO surrogate gradient norm square in terms of its corresponding stochastic gradient norm square.
Lemma C.7 (Sensitive sparse ZO surrogate gradient norm square).
Eg f (Ie, (Ir.
a)|l x.) = (2+ + u)e (w (Ie, k))
Proof. Our masked perturbation ğ is sampled as ž ~ N(0, i fa,,), where iam equals the identity matrix La with its main diagonal masked by m.
We expand the sensitive sparse ZO surrogate surogate-gradient covariance matrix:
E, f (z,v), 2) f E)T
= [z=T] ((m @ V (f (a, (m) Vs(w; (z, T
2m C (f (z, (m) c ff (z) ) m c V (x, i
16

---

Published as a conference paper at ICLR 2026
The above expected squared norm is obtained by summing the diagonal elements of this covariance matrix:
21, 20) I|| = (d f(W, f(w, (a, y). 2) f( w. (x, v). 1)²
V (wr (wr
2c Vf (Ie. vn)) uc V Wi) (Iz. (x)) 1|
= (2 + u)c (wr (re, B()
Lemma C.8 (Unbiasedness of Masked Sparse 70 Surrogate Gradient).
E (wE,, ]] = m (w where 2 = 20m.
(4)
Proof. First, consider the estimator defined as:
fu (wi, 2) fu( c(2 m)) fa(w.. c(2 m)) (z m).
2e
To proceed, we apply first-order Taylor expansion of Sa around w for small e: f& (w + c(z O m)) = fa (w() + (e(V//(()))))) m) + O(e)).
Substitute these expansions into the numerator the estimator:
S& (w (= m)) - [u(w: - e
= [sa(w)) + ( : m)]
[ 20 m)] 0(2).
Simplify the expression:
fu(w] + ((== - (0)
Thus, the estimator becomes:
fu 2) = Vz 2 m) + 0(e))
(:0 m) =
+ O(e] (20m).
2e
As c - 0, the O(c) term disappears, yielding the approximation:
fic( 2) (V.f.(w)),= m) (2
Next, compute the expectation E, f 2) Since the estimator is vector, consider its j-th component:
Vil(wWie)) ( m) (zym,).
Express the inner product explicitly:
V (wi),, = Σ ((Vf(wW)))a
Thus, the j-th component is:
d
(w ~
V (V z,m.. Σ
17

---

Published as a conference paper at ICLR 2026
Now, take the expectation (O), where 2, are independent standard normal variables:
E= [ (VVI
VVf (m
Since B[2/2]] = 6ig i = j. 0 otherwise), the sum reduces to:
(V (W
Given m3 m, (as my 1) and E[2]] 1, this becomes:
(v(a(w())))):
Thus, for each component j:
Eg V ( [w( ~ my((Su(w()))
This implics:
12
(w
2)]
m VSa(w)).
Finally, as - 0, the higher-order terms in the Taylor expansion vanish, making the approximation exact:
E3 Vis((h
C.3 MEERKAT CONVERGENCE ANALYSIS
We consider the federated zeroth-order optimization problem, where the objective is to minimize the global loss function(Ling al. 2024):
K
min f(w) Σ Paf((w) k=1
w
Each client performs T local steps:
w ++1
w (E(W), 1=
starting from the global model w w After clients finish local updates, the server performs weighted aggregation of their model updates.
K
Pk P&
Theorem C.9. [Client Local 70 Update Convergence] fk be T--moooth and & ft. be an unbiased sparse zeroth-order gradient estimator with variance bounded by o², Then have
If we set constant learning rate 1) L[u+2] and T local steps, the output of client satisfies:
V so 20(2) +0(02)
(5)
18

---

Published as a conference paper at ICLR 2026
Proof. We start by proving Theorem C.9 euqation 5 that each client achieves local convergence during training with sparse zeroth-ordes finetuning. Next, we demonstrate that server-side aggregation also converge. Finally, by leveraging the PL inequality, prove that MEERKAT exhibits linear convergence to global minimum.
Part 1: Client Local ZO Update Convergence
We analyze the effect of one local step of MEERKAT under sparse zeroth-order updates. Let client perform the local update:
w V
where the stochastic sparse zeroth-order gradient estimator is defined as:
Jk(w', <(2) o m))
f'((W c(z) @ m))
gh
2c
A g (2) m)
Descent via Lipschitz smoothness. Since fú (w) is Lipschitz smoothness:
fe(wktt) S Saa sw() + (V f1 w - ww) + 1w -
Substituting the update w - w/ = -n we obtain:
fw & ) S fx(w)) fw (l( + Ln² 1 f (w, 2:)13.
2
Taking expectation, we have:
LI²² Eg V fa (w1. 21)1
Ealfa(W( (xff((w!) E V(((w())
2
{ + < (eff B. - Ln² (2 + u)cB2 f (w{) ||a
Lif c(u +2) // + Lib
< (Baf(w)))))) (on-
c(u + 2)0².
2
2
Denote a = Lo(u + 2), we can rewrite as:
Bafa(w & s Ex [a(W)) 1/1 (m- IV Sa [1 =000
From the above inequality, we get 17 25, Suppose a constant learning rate ?/ 11
M(422) we get:
Egfa ( ) < Ez fw(w))
C7 V (wff((W))) 2
a oo
(6)
2
Accumulating over steps.
Summing equation 6 over t = to T 1, we get:
2 (fu(w)) ft) + 2011
2
Eg fW < cni
272
2L(u + 2)
(7)
(fa(w2) - fk) +
cl
= -o (((((( -st) + +0().
19

---

Published as a conference paper at ICLR 2026
C.4 MEERKAT CONVERGENCE ANALYSIS
We now proceed to analyze the convergence of the global model in our federated learning framework. Having established the convergence properties of local client updates, demonstrate how these results extend to guarantee the convergence of the server-aggregated global model.
Proof. We approach this proof systematically by analyzing how the local convergence properties of clients extend to the global model through the aggregation process.
Global Model Update Representation. First, the global model update can be represented as:
W
W
pa (& -w")
where each client starts from the global model w" and performs 7 local updates to reach w Client Local Update Accumulation For any client ki the accumulated local updates can be expressed as:
V
t=
Global Loss Descent Analysis By the L-smoothness property (Assumption C.I
we have:
L
f(w" f(w - f(w") + (Vf(w'), w ++
w")
w
w
(8)
2
For the inner product we can get:
K
(Vf(w"), wr+1 w") Σ
(VW(") w w")
Pk
Accoding to the client local update process, we have:
Pho VVf((f'Ww)) wh -w") Σ Pk (Vf((")
fa('"" 2:))
K
T-1
( Vfa(w'd, 2.))
(V((w')
= -
k=1
t=0
We assume that each client's weight is equal pı = 1/K, by substituting it into the above inequality, we have:
K
Pk (Va("")) w T w²)
" K
(f(("), Vfa("" z.))
(9)
t=
Based on the equation and V.f. is unbiased, we have:
T-1
E. E 2:)))
We substitute the equation and get:
K T-1
K T-1 ΣΣ
Vf(w") Eg "r (.)))
Vf(w") m G ()
Am t=
20

---

Published as a conference paper at ICLR 2026
Under the Cauchy-Schhe Schwarz inequality, have:
Vfa(w")) |[V](w")|
m fi W
We substitute Assumption
C.6 get:
[Vf(w")
m
(wl VellVf(w
Thus we get:
Vf(("), m O f
& VallVf( V
By the triangle inequality, we have
We substitute Assumption C.3 and use the properties of square roots we get:
[V[(w']| + f (w vs(w")l
+ V en [VW((1) of.
$ (1+ ( //(w"]|+
Using the bound (V((w") m @ f (w S ve Vf I| V from Cauchy-Schward and Assumption C.6 and then plugging in the above, we obtain
(Vf(w"), @ () Vell
vch) fV((w")]| tnn
S vc ch [V](w")|| Vean IVf(w')I
Recall that the server update inner product is
K T-1 K
(V(("") w wr1 w²)
VJ, m mOVA).
Substituting the bound equation We have:
(Vl(""), w w²) TT val [Vf(')) nT Voon IVf(w")]l (10)
Substituting this inequality to equation g have:
)1x+t) S ((w'') TT Ve + Voh) V (w])2
- - [ +
| - w²
(11)
Applying Jensen's inequality, the last term of the equation
II
will be:
K
T-1
11
Pkl
1=1
t=0
And then we apply Sauchy-Schwane inequality, the last term of the equation will be:
T
|| -
w/1 <
1
Pk
1=1
Substitute this inequaltiy equation 11 We get:
f(w < sf(w") - T Vo(1+ ON
2 T Voon [f((")]
K
Ps
t=l 1=0
21

---

Published as a conference paper at ICLR 2026
Taking Expectation and lemma C.7
Eg f(w s Es f(w") 17 T (e Voa) [V
V[V'os, T u)
K T-1
(w
2K
According to the equation 7 we know that the client-average squared gradient has upper bound. We to the above inequality last term we get:
substitute the equation 7
Eg $
Ez - T1T VC(1+ ( (o.) [[ nI (VF(w")
Lnj (2 + (2+ u)
22 (u + 2) (fk(w") Sb) +To2
2K
c
I Es f(w") TT Vc(1 + Von) [V TT VCo, [V
1,2 122 T ( + u) (u + 2)
[(((w))) -ix= + Lup T (2 + w) 0² (12)
K
K
2
Accumulating Over R Rounds. Summing equation 12 over r = to 1, we get:
R-1
Eg [[]]]]] Eg - + VCh)
Vc
Σ
R-1 1
TT VCol
V fw("
(13)
L 12 T (2 + u) (u 2)
R-1 K
Σ (f((w))- -f(w"))fi)) -ft)
K
m= k=1
L T (2 + u) c
0² R.
From the accumulated global descent inequality over R rounds: First we set
R-1 1
S= Σ |f
r=0
This represents the sum of squared gradient norms over R rounds. The second term in the inequality involves RR-1 [V(("") IlI and we apply the Cauchy-Schward inequality to it. For the sequence
a, = [IVS(ww) [I (with r = =0, l,..,. 1), we consider it as a vector in R along with vector of
ones:
R-1 [VS(w))| |||
R-1 1
R
R Σ
Σ
Vf(w")] .15
Σ V [f
12
0=
Since ΣR=1 12 = R, we obtain:
R
R-1
Σ
Vf(w)))
-RVS= =
22

---

Published as a conference paper at ICLR 2026
Substituting this into the second term, we have:
Vean
VS(w")| S SiT VRS.
Thus, the inequality becomes:
[g[f(w[)] Es(f(w))] - + veh 1/I VOO RS
L22 u)(u + 2)
R-1 K
ΣΣ (f((w)) -fk fE) k=
+
K
T2 + ulc o²R.
2
Second, we focus on the term r/ VCa, v RS and apply Young's Inequality with d > 0 and non- negative real numbers and y,
ry S x²
y²8
+
28
2
We identify x = VS and y = nIT Vea R, since:
1/ VoV RS (1/l Veon VS.
Applying Young's Inequality:
(v5)²
(n VoOA v 28
VS. (TTV <
28
2
Therefore:
11T Voon VRS < T Rd
S
28
RS -If yoo VooN
S
1
co
2
2
Finally we replace the second term the inequality with the above result:
- nTTV + VorJ5+ + ( S
T2 RD
28
2
122 + u)²
R-1
Σ
> (falw")- -
K
+ (2 u)c R2R.
2
This inequality now depends on 5.
1 R-1
(ITVVLL++ Vor)
Σ
[Vf(w')1 s -fwR)) + ,0,12003,R82
28
L2 (2 u22 R-1 K
Σ
(fuu(w") (t) (14)
K
P= 0=
L 22 u
+
2
According to Assumption C.I we have:
L fa(w") s fa (w + (W). w - wi) +
|| w
2
23

---

Published as a conference paper at ICLR 2026
Since wi is the minimizer of (w), the gradient at the local optimum must be zero:
Vs((W)) = O.
Substituting this into the inner product term:
(w w' wx) (0, w' wx) 0.
Thus, the inner product term disappears because the gradient at w is zero, making the inner product with any vector (including w w equal zero.
With the inner product term vanishing, the inequality simplifies to:
L
fx(w") S (wi)
2 A
This provides an upper bound on fw("") in terms of the local optimal loss ft and the optimality gap Ag.
The global optimal loss is defined as:
K
= f(w") = Σ Paff(w").
1=
Using the bound derived for each local loss:
Su(w*) S 1 + 2 Dk.
substitute this into the expression for f
K
L
Ax)
-
Σ Pk
fh +
2
k=1
Expanding the right-hand side:
L Pk
Σ PrAk.
2
From the above equation, we have:
L K
K
Pk
S
Paff-
2
k=1
k=1
1
Σ
L
A
K
2K
From the equation 14 we have the term:
L2r2T(22 + u)² R-1 K
(fa(w)) - fft.
K
First, we express the double sum as:
ΣΣ (fe(ww) - fit Σ Σ fa(w") Σ fk
Since Pk t. have:
K
Σ fa(w") = Kf(w"),
24

---

Published as a conference paper at ICLR 2026
where f(w") Ex= Pafa(w")) k Ik=1fa(u>>
fa(ww). Therefore:
ΣΣ (Ja(w") 12) Σ Kf(w0")
SK
From the earlier derivation, we have the inequality:
L
K
--
A
K
2K
h=1
Substituting this into the expression above:
n-1 K
R-1 S
Kf -
K
))
Σ
Σ
(fa(w))
- f2)
Kf(w")
P= R=1
r=
k=
Thus:
R-1 K ΣΣ
R-1
L K KS
(1)
12)
Σ
Ks(w")
Since A& is constant across iterations, can factor it out:
R-1
L R-1 X
R-1
LR K
--)
2 Σ A
r=0
r=0 Rm
T=U
Now, multiply by the coefficient:
1. T22 + 3
R R-1
K (fa(w") - f%)
T22 + u)²
R-1
- _ン
K
Σ r=
(f(w") -
K
K
k=1
Simplifying:
R-1 (f(w") -
TT u)
K
12 + u)²
Ak.
2K
Substituting this result into the original target inequality, we get:
Ve vch)
1
R-1
Vf(w")) 1E [ - -[(wR]]
1222c00 Rd
28
2
R-1
+ +2' + u)² Σ (f(w") - -f¹)
r=0
+ (2
-R)RR
K
2K
km
T + u)co² R
2
According to the Assumption C.2 we have:
2p(f(w") - 1r) = [VV((")) Vw" e Rd.
R-1
R
2μ (f(ww) - <
//(w") Vw ER,
25

---

Published as a conference paper at ICLR 2026
We let nT VC + vCh)
2/8
0 and substitute the above inequality, we have:
R-1
1
2/(1rTve(1++ vch)
Σ (J(W") - M s Eg [S(wW) - [(w0
28
2
R-1
+ 1221TT(2 + u)² Σ (f(w") - 5*)
r=0
1 12 + +u)2R
k
A
2K
h=1
(2)00000R + (2
u)
2
Ex [( f(w
Σ (f(w))- fr) x
2n vcl1+ + von) 28 - (2
T² coa Rd
222u(nTVc(1 vox) 2/1
1,2 u)²
K
(15)
(2
u)² R
Σ
DE
k=1
2K [2 + /ch) 28
- (2
L1211² (2 + uu) R
2 22 + Voh) '28
-L2 u)?
R-1
E Bz[f(w0) - [[("
R (f(w)) - f²) s
1 21
(onnnTvell (ch)
T.3 u)²
22 co 6
2 2u(TTV + voh)
L 1) T2 + w)²]
u)
28
K
(16)
L (2 + =u²
Σ
A
k=
2K 2ulTTTVal + /Ch) 28
T.2 [1)]
m)²]
1 (2 + u) co2
2 (CT voh) -/2
L (2
We select 5 nT (TV(( which leads to:
1
1
1/Tl vo(1+ + vCr)
28
Substituting into the denominator:
T/T vo(1 + vON)
21 nT Vc(1 + (hh
= - Ve(1 + Voh)
2
26

---

Published as a conference paper at ICLR 2026
With the chosen 8, we have:
R Σ
E: [f(wO) - {(
(f(w")-f)
R
R
+ vch) (2 u3²
Voo
2(1 + voh) (NVOLL + ( L2 u) K
(17)
L2 + u)² 2k=1 A
2K [uve(1 + vch) 12 (2 + u)²
L2 + 2. cO2
+ 2 pVvel V Ch (2 u)²
where the step-size must satisfy:
(V 1.(24a))
to ensure denominator positivity.
a (vd(++ve)) [(u=2)) 212
). We substitute this n to
Plugging in a constant learning rate = min
equation 17 and get:
1 s -*)
-(sww))-
22 + u)²
[=[s(w0) s
R
12c(1 + vca)
'TR
o
L
Tco²
+ m(1 + voh)
+
D +
K
2L (2 +
k=1
((ww))
s s0
(2 (2+ u)
[B([(W)) s(w -[([))))
+0
T
+ +01).
(18)
R
TR
2+
C.5 MEERKAT-VP CONVERGENCE ANALYSIS
We propose Virtual Path Client Selection (MEERKAT-VP) mechanism that identifies clients with highly heterogeneous data distributions based on their optimization trajectories. Instead of excluding them, MEERKAT-VP applies early stopping these clients limit their adverse influence on global model updates while still preserving their participation.
Proof. Motivation for Early Stopping: In federated learning, clients perform local updates starting from the global model w". For T > 1, clients may drift towards their local optima, introducing bias into the global update due to data heterogeneity. By identifying "bad" clients and limiting them update step, we reduce their drift and align their contributions more closely with the global gradient.
We divide the K clients into two groups:
Balanned-dissrribation clients (K₃): Perform T local step updates. deewed-distribuution clients (K,): Perform only 1 local step update.
The global model update becomes:
r+1 w. + K
Σ KEK,
(w
w²) +
Σ
(w w')
K keKa
where:
T
T.T
w
V (u(
w r.1
t=0
27

---

Published as a conference paper at ICLR 2026
Loss Descent Analysis
Using the L-smoothness property:
f(w & f(w") + (Vf(w"), wn - w") + 2 11 -w'1
w=1
We analyze the inner product term:
(V(""), w - w²) Σ Pk (Vf(w"), w. w/"
w()
Pk (Vf((ww) w TT w") "1
Pk (Vf(w))
fi'' 22))
k=1
= =- Pk
(Vf(w"). fft .2))
Since we have balanced-disttrbution clients and skewed-dissributiinn clients:
(VI(w"), wr+1 - w²)= -w"- 1
Σ
(VI(w"),
T.F ww" Σ "F -w")
K
K
KEK,
keKb
(Vf(w"), w - w²)
11
Vf(w"), ((
kEK 1=0
(19)
Σ (V(("),, Via(w)))
K
kEK
Since AE is unbiased, we have:
K TT
K T-1 ΣΣ
ΣΣ (V[(w") V'e(w"' 2.))
Vs(w)) Ea 2x)])
k= 1-
k= 0
We substitute the equation and
get:
K T-1
K T-1
Vf(w"), (
Vf(w") m 3 V)((wt))).
1=
Thus taking expectation equation 19
we can get:
Eg (Vf(ww) +1 -w)
7
Σ
[V((w), m o V(((w"+))
K
kEK
(20)
(VS(w"), m Vf((w"))
KE
Under the Cauchy-Schwer inequality, have:
(V((W),
V[a(w>))) ||/[(w"]| |m Vie(w**)]l
28

---

Published as a conference paper at ICLR 2026
We substitute Assumption C.6 get:
[Vf(w")] m (f vollVf(w")] (W'
Thus we get:
(Ww"", Vsa(w"))) s vellV/(
I1-
By the triangle inequality, we have
(V=(\l))) V [VS((") + [V/h(w*>)) v/(w")ll
We substitute Assumption C.3 and use the properties of square roots we get:
[V + + (w"r)) - V/(w")| = [V(("\(())))))))) ca I [[") + o
s (1+ [w")|] a on
Using the bound (VS(w"), ) m @ (w(.":)))) v Ve | /xll from Cauchy-Schward and Assumption C.6. and then plugging in the above, we obtain
(Vf(w"), m O V.((w"))) & Vc|lV[(>'|[[ + ( Vf(w")] + tn
S Vc(+ Voh) | [V )12 + Vcan [Vf(w))l I|--
Since this bound holds uniformly for all k and t, and based on the equation 20 we get:
Σ T
(V((w') ) m o V(a(w"))) +
Σ (VS(w')
m V/((w"))
kEK, 1=0
kEKs
$ K T + K&l) [Ve(1 + (h) Vf(w") Voa, vf(w")]]
We get:
Es Es]f(w]+)]] =B(((""") - noa0
K (1 + Van)ll/(("")
(21)
1V00 K
T. 2
on onllVf(w"\]l +
EE -
where a = KKyIT + [K.).
Since the global model update is given by:
w++1= w + K
Σ
(m) IT (w
w²)
1
Σ
(w"'-=") w')
K
KEK,
keRb
We substitute the local updates and the squared norm is:
2
2²2
||00 r+1
Vellwii)) + Σ (lo(ww)
kEK,
AEKs
Define the update contribution per client:
71 IT- fit
if E Kg. if E Ko.
1/ fi(w")
Then:
K
1 K
n+1 w"
k
K
I r+
wr1
K2
Using the Cauchy-Schwer inequality:
K
K
Σ
K
12
where Å denotes the actual model update on client k.
k=
29

---

Published as a conference paper at ICLR 2026
So:
1 K
11
w01 -
| 1
K
Now compute ||
T-
| 12 = 112
Σ
V La (w)
for k Kg
| 12 = 1? [w |
for k E Kn-
Thus:
1??
[
1 w² 10 S
Σ K
Σ
+ Σ
K
t=
REKb
We take the expectation:
n
Ex|lw +1 - -w1
Σ Ex kg
Σ 1=
A (w
+ +
Ez
K
AERb
For E Kg:
E
= (2 - -we [V
For k E Kg:
E&
fi(
Using the Cauchy-Schwarc incquality:
T-1
T-1
Σ
<T
B2
0=
t=0
According to the lemma
C.7
E;
= =(2 + u)c [
So:
2
T-1
T-1
Eg
Σ (u
< + u)c
V (w
0=
Combine the terms we get:
Eg E1 10 <
1) + u)c
uue
T E Σ keK, 1=
T-1
Σ [Vfa(w"))?
K
keK.
We substitute this inequality to the equation
2T
T/oo
Eg [z 1)] SiSSEE [=[[(w")]
(1 (V)
K
Wea
K oh [[V((())I
(22)
22 + u)cL
T T T-1 kek,
Σ [
2K
t=0
kek,
30

---

Published as a conference paper at ICLR 2026
Wea K
aVea
Eg [[w s [x[['[[]]
(1 (Ch) \V(x'\l
K on
/ [/(''|
12 + u.) CLT
1/2 + u) CL
2K
2K
kEK,
kEKn
According to the equation
we know that the client-average squared gradient has upper bound.
VVea
Vea
E3 [f
Es [=[[(w")]
(1+ (c,) [VF((")
K on [V(w''I|
K
12 + ILT
21 (fa(u - fk) + T TTo²
2K
c
(23)
kEK
(2 + u) CL
2K
kEKb
Using Assumption C.3 which states that for any E Rdd
[V
|)( S c [V]] + to
we can bound the squared norm of the local gradient V fu(w")] )1² Specifically, by the incquality (a y)² < 2x² + 2y²², we have:
f (w)"| [V" + (Vfk(w")
Then, applying Assumption C.3 with 0 = w" and = k:
[v [=("") < cn [V(w")| + +
Therefore,
[v' < 2 - 2 (c. [V + oî = 2cx) [s(w")| + 20
Thus, we obtain the bound:
I fa (w 212 (2+ 2cx)) [V(('') 20%
We substitute the bound to the inequality 23 according to the Assumption C.3 we substitute the last term:
Veea
Veea (1+ /Cl)) (1))('))1
ES [[( S EE [f('']
K on [VS("")I
K
17(22 + L.T
2T + w) (fe(w - f() + +To2
2K
KEK,
12 + u) eL
(2 + 2cr) [V[[2)]] σî
2K
kcKs
IVea
(1 + vCh) [/ Veca K on [VS('')]
E [ S E. [s(w')]
K
12 + u) LT
2L(2+u) (fe(we)) fk) T oz]
2K
kEK
12(2 + u) c L Kı| (1+ CA)
[ + 77(2 + u) c L Kbl o7
K
K
31

---

Published as a conference paper at ICLR 2026
Eg [[( S Ex [f(w")]
1/ + uj c Kb + cn) nVea
[
VVeaσ
[ل[(w'|||
K
(24)
1/ + u)² L2T
(Julwk - Sk)
K
REKg
12 + c (T2 Kg + 2 Kooh)
2K
Accumulating Over R Rounds. Summing equation 24 over 010 1,
(2 Ko (1 + c4) nVea R-1
E3 [[] E3 [[(w]]]] s
V vf(w")
R-1 weaan
Vf(w")||
K
r=0
(25)
1/ + 2²
L R-1
Σ
(Ja(uf)) - 12)
+
K
m= kEK,
17 (2 u) cL
+
2K
According to our previous derivation, we know that:
R 1 Vf(w")|| S
R-1 VR
Σ
Vf(w")
(26)
Apply Young's inequality with d >> 0 and nonnegative real numbers x and y,
xy S x² y28
28
2
Vycona R K
[V[w")]| Weaon R
Σ (/
K
vp cooof R8 2K²
[V[w")||
28
TVCona
1
R
[f(("']|1 + 1/ of R
Σ
[Vf(w") II
Σ
K
28
2
We substitute this to the equation 25
32

---

Published as a conference paper at ICLR 2026
1/ 1/(2- u) K2 (1 cz)
Weca
R-1
E3 [[ - Ex[s(w0)
+
Σ
[V[(w")|
K
2
1f ca'o Rd
1 (2 + u)² L
R
(fa(w fi.
2 2
K
P= 0 KE
17 (2 + +2+ L
(2 +2Kooo)
2K
(27)
Given that ux 0r w", this term is equivalent Ex= EKEK, (Ja(w") - SE). From our previous discussion, we have the inequality for a single round
K
Σ (fa(w)) fk)
Σ (falw") fk)
kEK
k=1
and the inequality used in Part 2 of the proof:
R-1 K
R-1
L K
ΣΣ (faww)) f2)
Σ
(f((w)-Kf
+
Σ Ak
T=0
Combining these two inequalities, we obtain bound for the sum over the set Kg
We set y S which means that the subset clients the effect to the global:
R-1 1
R-1 1
K
Σ Σ (fx(w") - f2) S
Y
K (f(w") f*) +
A4
2
T=U KEK,
We substitute this to the above inequality get:
1/ + u) c) KG (1 (1+ ch)
nyea
1
R-1
Eg [f("(]]]]] B2 [f((]]]]] s
[f(w")
K
K
28
1/ ca of Rd
R-1
T
Σ (f(w²) -8")
2
(2 u)2 3 T R
K
A
2K
2/ + u) cLR R
(T2 (T"0² Kg +
22,g2K200 2K.o
+
2
We substitute at
17(2+u)cle u1)cKKKKKK CN)
ToCKKTT + Ki))
1
Eg [f([]]] - EB [E([f(w]]] <
+
K
K
28
r=0
n²c(KTT² + 2K,TK, + K2)0
R-1
+ 1 + w) Σ (f(w") - -f¹)
2K²
+ (2 + 222++)2TT39 1,3 Ry
K
12 + u)eLR
A +
(T² + 2Kko
2K
2K
33

---

Published as a conference paper at ICLR 2026
To simplify the inequality, we solve for d:
1
VVCKKTT + Kb) ./2(2 + (U)+tu)CK((LL000 cb) + NVCKKT+ + Kb)
26
2K
K
K
WVC(KTT +
T Kb)
12 +
w)L.KK(1 (1 cn)
28
2K
K
K
8
TVcKK + Kb) 21 (2 + w)eLKK(l cc)
For 8 > 0, the denominator must be positive:
WcCKTT + Kb) 2n + u)cLKs(1 + +x) > 0,
yielding the condition:
< Vo(KTT + Kb) 2(2 + u)cLKs(
Substitute 6:
B3 [f - E=[s(w[]] S VVCKKTT {b)) R-1
V [V(("))
2K
r=0
1) TT + K2)
+ 2K (NVC(KKT + Kb) - 2172 (2 + (1 cn))
R-1
Σ (f(w)) - -5)
+
p=0
12(2 + TT
K
A
2K
k=1
22 (2 + w)LLR (T + 2k
2K
According to the Assumption
we have:
2p( (²) < [V][(')) Vw" E Rd.
R-1 Σ (f(w")
R
2/
< [/(w') Vw ERC,
Combine the PL inequality to the above function we get:
Ve(KKT + Kb)
R-1
V {
2K
1
+ Ku)
2K (nVeCKKT + Kb)
21 + u)e..K(11 cc))
R-1
+ 2
(f(w") --)
r=0
1/ + 133T.TR R2y
K
AB
2K
(2 + u)cLR
+ (T
2K
34

---

Published as a conference paper at ICLR 2026
Let SE = - TV(K,TT Kb) - 2n + w) ch). We require D& > O. Substituting this back into the original inequality:
Do
[l ([]((f"))) navo(K,T TTT + Kb)
SF +
K rfo(KTT + Ko)
2KDs
12 + U) Ry
K
D
2K
12 + u.cLR
OT Kg + 2Kooa)
2K
Collecting terms involving Sg:
[] [[f(w")) - 1) nuyo(KgT + Kib)
Sp + other terms.
K
Moving Sg to the left side:
nuveeKKTT (K₃T + Ko).
12222 + w2)++ Sp S [[f(w]]] - [l)(
K
1 + L) Ry r2c + oo)
K
+
Ak
2K Do
2K
(2 u.eL. R
(T² Kg +2Koo)) (28)
2K
Blf( Since E[f 2 fo (typically f* is the minimum), we have -
(lf
J Let fi = [[ ] (the initial expected suboptimality). Let the coefficient of Sg be
C²3 nHVAKK (T.k) 12 + LT2 To ensure C's > O, we need v) sufficiently small such that
K KVRK, T+Ks)
n K(2+u)=LPT, Then:
S6 Pe[KKTT + K2) R +
17 u)2 R
2 D5
2K
12 + u) c R
2 02 Kg +2K,
+
2
Our goal is S 1 -R-1 [lf(w") - -. Dividing both sides by R:
R-1
fö
PeCKKT + Kb) of
C's
Σ
[l]( - -5 -
R
R
2K D6
r=
1 (2 + v)2 L3 T
K
A
2K
k=1
1/ (2 + uL (T2o² + 2Kbo))
2K
Finally, dividing both sides by O's (assuming C' > 0):
R-1 5
1 fö
17 n2cc K.)
Σ
B+ll(w")
+
C²
R
2K
(nVe(KTT + Kb) - -2n2 + w)cLKi(1 (1(110+000 tch))
2 + u) T3
K
1 + u)cL. (T + 2K
+
AB +
2K
2K
k=l
35

---

Published as a conference paper at ICLR 2026
where
C's nuVA T + Ko)
-1²(2tu)²
D3 - NVC(K,TKK) - (2+ (b(1+cn).
K
To ensure both C's
0 and Ds > 0, we require
v IT + Ki)
PV BVCK Kb)
1 < 2(2 + u IL. Ko + cn)
((2 + u) 222.TTT77
Let
Mmax = min (78, nsh:
0 E (0, 2
Choosing = gives
PV + Kb)
1 Mmax
17
2K u
We select
PVCKKT+ Kb)
7 2K + 1.)
And from previous client convergence conclusion, we pick constant local learning rate
1
2c
Melient
a ( 2)
a
pV (K, T+K.)
Substituting the learning rate = mink I(u+2j:
since rj is a small value, we
2K (24u) DT9
neglect ?
R Ex [[]
R
42 2 E[f(w) - fr1
Th
(2+ L
Σ
K
Ak
1² c (K₂ 2
4K
(T"0² + 22,oh
4K(2+ LT
k² (2+ TT TT c &(KyT f Ko)² R
レ - s of
K₂
1 +
A kg
Abb)
K k=1
(29)
cT Kg
+ 0
K(1 + u)
+ o
c CK
k(1+u)T7) +
Define the error upper-bounds for MEERKAT-VP and the baseline MEERKAT follows:
EMBERKAT-VV-
4K2 + LTT Lf [f 122 c cK + K6)
- f*]
o
(2 +
L
Σ A
K
o(T'O' + 2Kbo7)
R
2
4K
4K(2 - T.
k=
(1) Transient term
(II) Steady-state term
u)2 [l - fr1 2² c (1 + T
of
K M A +
Teo² 22
EMERRKA
c 2 +
R
P
+
K
k=1
(1') Transsent
(II) Steady-state
36

---

Published as a conference paper at ICLR 2026
Transient term ratio:
(
71+ vor)² 1, and as ch +,, 7(1 ven)2 0. (1)
Noise term ratio:
22 f1 (h)
(1 + von)² which is < when j (1 vch) <2
Empirically < 1, thus u (1 + vca)² < 2 is True. Additionally, VPCS includes extra term cK (2 which decays + and becomes negligible for large T.
Heterogeneity and variance terms:
+ u Ak
K
K
Σ
DBs
and the extra variance term decays as 1/K.
4K
K
Therefore, under the same T and R, EMGERKAT-V < geneity C₁ increases.
EMBERKAT and this gap widens as data hetero-
REMARKS
The analysis the upper bound in Equation 17 reveals how the local training step T, density level u. and communication rounds R collectively influence the optimization dynamics through balance of convergence rate, vari-nar trade-offs, and steady-state error control:
Impact of Local Update Steps T: A smaller T amplifies the term (2+w)² [Ef((w - f(w increasing the average optimality gap after R communi- TR
cation rounds when R is fixed. However, this effect can be mitigated by increasing R, as the scaling factor Ω reduces the term's 's impact. Conversely, reducing 7 diminishes the variance term
leading to smaller steady-state error. Thus, smaller T may prolong the transient
phase but ultimately achieves tighter optimality gap relative to f after sufficient rounds.
Density Level Reducing (i.e., increasing sparsity) quadratically benefits the transient term, yet it also inflates the steady-state term through the denominator + u. Choosing u therefore amounts balancing communication savings against the plateau error; aggressive sparsification should be coupled with smaller 7' to avoid performance degradation.
MEERKAT-VP Client Selection Strategy: By early-stopping extreme iuta-imbalance clients with single local training step. MEERKAT-VE effectively reduces Non-IIID drift in zeroth-orde federated IIm fine-tuning. This strategy lowers the coefficient of the transient term and further reduces heterogeneity- and variance-induced steady-state error. Under fixed T and R, these effects yield strictly faster convergence and a tighter optimality gap Non-IIID settings.
These conclusions illustrate how tuning T, R. and the MEERKAT- vP client selection strategy can optimize performance in federated, sparse, and Non-IID learning scenarios.
C.6 EMPIRICAL ANALYSIS OF THE GRADIP PHENOMENON
By Lemma C.8. the masked sparse zeroth-order (ZO) surrogate gradient is an unbiased estimator of the masked lirst-order gradient. Building on this fact, we define the vector ge (w; æ; y) is obtained by computing the gradient of the cross-entropy loss for single sample with respect to small subset of parameters selected by a mask.
From logits to Softmax Probabilities we have:
The model's final layer outputs a logit for each class:
h(x;; w) (hi...., .c)) E C.C.
The softmax probabilities are given by:
c/
P5 (2; w)
Σ ehhn"
37

---

Published as a conference paper at ICLR 2026
The enosspyntropy loss for single sample is:
lw;; v. y) log (og where E (I....,C}, C}.
For each logit hy, the partial derivative
aE 0h = P4 1 (y=}} = (eg)i
where Ey is the one-hot vector with in the y-th component.
Since we are only interested the sensitive parameters selected by the mask m, the gradient with respect to the parameters can be written as:
9e x;, V wm ((u; v.
Ol
Σ j=1
V Vwn (m)
=p(a;; w) ex) )² Vun k(;
Here:
um hy (I; w) is the gradient//acobian of the logit h2 with respect to the masked parameter Wm
By collecting the coefficients Pj 1y=j into vector, we obtain the compact form:
Ie 00; x. (p kwy)
mm w))
In our existing local client convergence inequality and from the assumption C.4 write the key constant estimator variance:
we can empirically
of Da v)].
We write go in matrix form: Define:
J J(z; w) = V wm h(:; w) E R DC
a;x,y; = p(x;w) C/ E CRC.
Thus:
ge(w;x, J (x; w a w: E Rd...
We substitute this equation to the above estimator variance:
1 tr dm
o d
Eia.) [ow x. v) f
[1)
J Cov [a us (a
u; J
(1)
variance
Note:
E. E CxC is determined solely by the label distribution and prediction probabilities.
J reflects the network structure and influences only similarity coefficient.
Analysis of Extreme Non-IID (Single Label y'):
The label is fixed, so yy=j is constant.
the model is mostly correct: p as eyt, then a(x, y w) 0, yielding:
0
o
dm tr( = 0.
non
Analysis of Approximate IID (Balanced Multi-Label)
The label y varies across (1,..., C).
38

---

Published as a conference paper at ICLR 2026
Even as the loss decreases, Pj differs across classes. The covariance is: (Ea)r= = E [(p, (ps - ly=s)] - (Elp, y (Elps 1 y=s))
This matrix has diagonal elements E [(p, - 1y=r) > 0, making Ea positive definite or semi- definite but non-zero. Thus:
1
ir (r(J" (T"EEn End) > O. dm
Our local convergence bound is:
( (+)
Σ
EIV
fa
+ o
T
which indicates that in the steady state, the upper bound of the gradient norm is determined by of Therefore,
IID clients: Gradient Norm oscillates significantly;
Gid o
non iid
Non-IID clients: Gradient Norm decreases monotonically and approaches 0.
REMARKS
In summary, by substituting the explicit form of the cross-entropy gradient into our sparse 70 convergence formula, we can empirically explain that due to the variance differences caused by label distributions, the Gradient Norms of IID clients maintain significant fluctuations, while that of extremely Non-IID clients rapidly decays and converges to zero.
Step Vintual Path Reconstruction & Gradip Calculation
Training With Callbration Siep
Step Identity Clients with Extrumaly Non-ID Data
Propected Gradintt
Local Step Gradip
Iritial phase steps
Step 3.2: Clients with ectremely Mon-lD dhes use step
B.DO
Calculate Initial later ratio
Projected De Gradient fot
training
Local Step Gradip
ratio
Client2 traning path
Calculate Inttial ratlo
Cliet
O
sampled from disribution En
quieseent Stop raño
Inbalanced Data Distribttion
1151 1352
Server
Callbrattoon Stop Gradip
Pretraninng Datanee Denerate Gradient
training path
Clent
Virtull
Client training pam
Cllent n with Data Dieributtion
with extremaly Non-ID dats.
Step Early Stopping
Figure 5: MEERKAT-VE. VP: Each client locally trains with a prescribed number of logging steps, yielding sequence of projected gradients. The server leverages randomly sampled vector af from the Gaussian distribution (00,, Ia) to reconstruct Vf and then computes GradIP (see Definition 2.3 every local training step. By analyzing the resulting GradlP values across all clients, the system distinguishes those clients with extremely Non-IID data from those that are relatively balanced. For the definition of parameters later phase steps, initial phase steps, quiescent step ratio, and initial to later ratio, please refer to Table 3 Appendix D.I
D MORE EXPERIMENTAL DETAILS
D.1 ADDITIONAL EXPERIMENTAL SETTINGS
Testbed. All experiments are run on servers with the following configurations: RTX A6000 Setup: Ubuntu 18.04.6 LTS with 2 NVIDIA RTX A6000 GPUs (each with 48GB GPU memory). GH200
39

---

Published as a conference paper at ICLR 2026
Algorithm 2 MEERKAT: Sparse Zeroth-Order Optimization for Federated LLM Fine-Tuning
pre-trained weight sparse mask m, learning rate y,, perturbation scale number of rounds R, total number of clients K number of local steps Server initiates seed list ..t.....s.)
Input:
for Round r = to R do
Step 1. Local ZO update.
for each client in parallel do Download model from server: Wa f- w,-1
Download seed list {s,,.... . from server for local step
to T' do Initialize z with seed sç.
Sample batch B on client dataset.
W < w/. + E- (z) ④ m)
Compute loss f+ <- f(wt; B3)
w w 2x (2) ④ m)
Compute loss: f- + f(w B)
Compute projected gradient:
9 + (f+ - -f-)/2c
Update client model:
4 gi (z) ① m)
w w n n ft
end for
Send projected gradients (9), ........ end for
g to server.
Step 2. Server recovers each client's update with virtual path. for = 1 to K do for local step t= to T do Generate z with seed s". Perform virtual path:
V 9 (z) m) w w/ 17 T/ f
Store recovered client model parameters w end for end for Step 3. Server aggregates reconstructed sparse model update.
K
W,
W
K
Generate new seed list (8/415" end for Output: wR
,8441
40

---

Published as a conference paper at ICLR 2026
Algorithm 3 MEERKAT with high frequency server-client synchronization
Seed and projected gradients 9% from all clients, learning rate 7, number of clients K, sparse mask m
Input:
Aggregate projected gradients from all clients with same seed:
9 f
gk
Calculate Zeroth-Order Gradients:
V g. (z m)
Update global model parameters:
Wr+1 Wr 77 (V/ m)
Generate new seed s_new
Output: Send aggregated global projected gradients g and seed s_new to all clients.
Sctup: Ubuntu 20.04 with NVIDIA GH200 GPU (96GB GPU memory). A100 Setup: Ubuntu 22.04 with NVIDIA A100 GPU (40GB GPU memory).
Dataset. We conducted experiments using datasets from the GLUE and SuperGLUE benchmarks, including SST2, AgNews, Yelp, BoolQ, RTE, WSC, and WiC. To create IID client data, we shuffle entire dataset and evenly divide among the clients. create Non-IID client data, we split the data using a Dirichlet distribution. For all tasks, the Dirichlet parameter is set to 0.5 to control the degree of data heterogeneity.
Evaluation metric. In our experiments, test accuracy is used as the primary evaluation metric. Accuracy is computed as the proportion correctly predicted labels across all evaluation samples. Additionally, incorporate the GradIP score (see Definition 2.3) to analyze further the dynamics of local model training under IID and Non-IID client data settings. GradIP provides a metric to measure the quality of client training trajectories, particularly in heterogeneous data distributions.
Notations. We present the definitions of parameters used in MEERKAT-VP Table
Table 3: MEERKAT-VP Parameters Notation
Term
Explanation
calibration steps Teali initial phase steps Tini later phase steps Tlate convergence threshold quiescent step ratio Pquie Initial to later ratio Plater
Number of steps each client performs to measure GradIP.
Number of carliest local steps used to measure the early-phase GradIP.
Number of latest local steps used observe the late-phase GradIP.
Threshold indicating when GradIP is effectively zero.
Fraction of later phase where GradIP stays below threshold
Ratio of average GradIP in the initial phase to average GradIP in the later phase.
Hyper-parameters. We use the following ppprr-parameters in experiments; see Table
MEERKAT-VP Hyperparameter Selection. Below, we present the default hyperparameter values for MEERKAT-VP, task-specific adjustments, and the results of our hyperparameter sensitivity analysis to demonstrate the robustness the method.
These default values work well for most tasks. However, some tasks benefit from task-specifi tuning, particularly the RTE task which shows sensitivity to these parameters:
Sensitivity Analysis. We conducted sensitivity analysis understand the robustness of our method to hyperparameter variations. Tables and show the performance stability across different parameter settings.
41

---

Published as a conference paper at ICLR 2026
Table 4: Hyper-parameters used in our experiments.
Parameter
Value
MEERKAT learning rate
[2e-4, 2c-8]
MEERKAT-VP learning rate
[2e-4, 2c-8]
LoRA-FedZO learning rate
[2e-4, 2c-8]
Full-FedZO learning rate
[2e-4, 2e-8]
Batch size
16
Dirichlet alpha
0.5, 0.3, 0.1
LoRA rank
16
LoRA alpha
16
initial phase steps
20
later phase steps convergence threshold quiescent step ratio Initial to later ratio calibration steps Total clients
20
[0.4, 0.5, 0.7]
[1.5, 2, 5, 10, 15]
100
10
Table 5: Default MEERKAT-VP Hyperparameter Values
initial phase steps
later phase steps
convergence threshold
quiescent step ratio
Initial to later ratio
20
20
0.5
Table 6: Task-Specific VPCS Hyperparameters for RTE Task
Model
initial phuse steps
later phase steps
convergence threshold
quiescent step rntio
Initial to later ratio
Gemma2-2B
LLaMA-3.2-1BB
Qwen2-1.58
0.5
0.5
Table 7: Parameter Sensitivity Analysis for LLaMA-3.2-1BB on SST-2 Task
initial phuse steps
phase steps
convergence threshold
quiescent step
Initial
Performance
20
20
0.5
3
0.922
20
20
0.5
5
0.922
20
7
0.922
20
20
0.5
10
0.922
0.922
Table 8: Parameter Sensitivity Analysis for RTE Task
Model
Initial phase steps
later phuse stegs
com ernergence threshold
quiescent step ratlo
Initial later ratio
Performance
LLaMA-3.2-18
20
0.8
0.7
0.5
0.5
7
0.617
0.617
Gemmu2-20 2-2B
20
20
20
0.5
0.5
5
0.657
0.657
D.2 ADDITIONAL EXPERIMENT RESULTS
In this section, we present additional experimental results to compare MEERKAT, MEERKAT-VE Full-FedZO, and LoRA-FedZZ under various settings. The results include multiple tables and figures, providing a detailed evaluation of performance across different models, datasets and experiment settings. Table 3 provides provids description of the parameters used in MEERKAT-VP, and Table lists the experiment parameters used in this experiment. Tables [ and 6 list the hyperparameter values for MEERKAT-VP. Tables 7 and 8 demonstrate the robustness of the MEERKAT-VE parameter selec- tion. Table provides quantitative analysis that demonstrates the significant disparity in gradient sensitivity across different parameter groups, thereby justifying shows that a domain-shiin shiftee calibration dataset can be used effectively to select sensitive model parameters. Furthermore, designed experiment where each client builds local parameter
selection criteria. Table []]
42

---

Published as a conference paper at ICLR 2026
mask from its own dataset. The results demonstrate that aggregating these local masks into union mask does not achieve better performance than using a single, globally unified mask. Table [3] compares MEERKAT and Full-FedZO multiple tasks at the same communication frequency for Llama-3.2-1B, Jowen2-1,5B,, and Gemma-2-26 models. Table [4 results in a Non-IID client data scenario, comparing MEERKAT-VP and MEERKAT under the same communication frequency and sparsity density, and demonstrating MEERKAT-VP improved performance. Table 13 investigates the robustness of MEERKAT by evaluating test accuracy with local step across different sparsity densities. Table 16 compares MEERKAT, Full-FedZO and LoRA-FedZO under high communication frequency across IID and Non-IID client data settings. Table 23 details the number of training rounds required for convergence across different models and tasks. Table 24 benchmarks computational and communication efficiency, demonstrating that MEERKAT significantly reduces peak RAM usage and client download bandwidth compared to the Full-FedZO and LoRA-FedZO baselines. Table 25 shows that our MEERKAT-N VP method achieves competitive performance against the back prooapaopagttion upper bound and substantially outperforms FedDYN (Acar et al. 2021). Figure7and Figure 9 further illustrate the phenomenon of GradIP under IID and Non-IID client data settings.
Table 9: Gradient Sensitivity Analysis for Qwen2-1.5B Model on C4 Dataset (Top 0.1% Parameters). To quantitatively analyze gradient sensitivity, ranked all parameters by their average squared gradients from pre-training and divided them into four disjoint (non-overlapping) buckets: 0-0.1 %,
0.1-1% 1-10% and 10%-100%
Bucket Metric
Top 0.1 %
0.1 11-1% %
1 1%-10%
10%-100%
Avg Gradient Square
4.403 10-3
8.536 10-5 5
1.075
10 6
1.764
10-6
Std Gradient Square
8.094 x 10-2
5.858
10-5
6.255
10-6
1.099
10-6
Max Grudient Square
1.413 x 10'
3.147
10
3.505
10-5
5.245
10-6
Min Gradient Square
3.166 x 10
3.529 x 10-5
5.245 x 10-6
1.025 x 10 10
Table 10: Accuracy of MEERKAT vs. Random-Seledt (Qwen2-1.5B, 0.1% mask). Directly
addressing the comparison with random selection, we ran control experiment that shows our method is significantly better across tasks. The local step is 10.
Method
SST-2
AGNews
Yelp
BoolQ
RTE
WSC
WIC
Avg
MEERKAT
0.949
0.881
0,934
0.752
0.813
0.682
0.628
0.806
Random Seleet
0.821
0.543
0.852
0.667
0.711
0.663
0.539
0.685
Improvement
+12.8%
+33.8%
+8.2%
+8.5%
+10.2%
+1.9%
+8.9%
+12.1%
43

---

Published as a conference paper at ICLR 2026
Table 11: Performance Comparison with Different Calibration Datasets and Methods. Our method does not require the original pre-training data. uses small sample (128 sequences) from any public, high-quality text corpus to create a transferable parameter mask. This table confirms MEERKAT'S flexibility and transferability across different domains, including web-text, code, and medical data, consistently outperforming the Full-FedZO baseline. We also explore UnionMask, a client-specific mask aggregation approach: (1) Each client computes its own mask based on local data distribution; (2) Clients send masks to the server for aggregation into union mask; (3) All clients use this union mask for ZO training; (4) The server uses the union mask for parameter updates. Results show that the specialized UnionMask performs similarly to our transferable mask, validating our universality approach. The local step is 10. Code data: microsoft/rStar-Cooe Medical data:
Method
SST-2
AGNews
Yelp
BoolQ
RTE
WSC
WIC
Avg
Full-FedZO
0.909
0.705
0.940
0.641
0.542
0.634
0.523
0.699
Web-Texi Domain Colibration Data
MEERKAT (C4, 0.1%)
0.916
0.872
0,964
0.695
0.600
0,653
0.614
0.759
MEERKAT (Wiki, 0.1%)
0.913
0.855
0.952
0.646
0.582
0.634
0.567
0.736
MEERKAT (ArXiv, 0,1%)
0,901
0.851
0.949
0.714
0.573
0.644
0.562
0.742
MEERKAT Web, 0.1%)
0.902
0.846
0.958
0.695
0.584
0.634
0.561
0.740
Domain-Shiffed Calibration Data
MEERKAT (Code, 0.1%)
0.915
0.843
0.956
0.695
0.551
0.612
0.602
0.739
MEERKAT (Bio. 0.1%)
0.912
0,850
0.956
0.694
0.560
0.625
0.595
0.742
Plent-Speeiffc Mask Aggregation
UnionMask (per-client, C4, 0.1%)
0,902
0.845
0.950
0.669
0.582
0.634
0.569
0.736
Table 12: Transferability of the sparse mask between legal-domain (MultiBURLEX) 2021 calibration datasets on LLaMA-3.2-IB.
Chalkidis et al.
Mask domain
SST2
AgNews
Yelp
BoolQ
Legal-domain (MultiUUREEX)
0.912
0.845
0.948
0.703
Table 13: Performance comparison of MEERKAT and Full-FedZO tasks SST-2, AgNews, Yelp, BoolQ, RTE, WSC, WIC under an IID client data setting. "Acc" is the average test accuracy across tasks. Bold numbers indicate the highest value in each row.
Methods
Local Step
SST-2
AgNews
Yelp
Bool()
RTE
WSC
WIC
Ace
Full-FedZ0
0.938
MEERKAT
10
0.925
0.881
0.964
0.751
0.684
0.634
0.648
0.784
FelZ-
30
0.913
0.700
0.935
0.645
0.542
0.634
0.538
0.099
LLaMA-32-1IB
MEERKAT
30
0.919
0.865
0.967
0.729
0.644
0.663
0.617
0.772
Ril-FedZO
0913
0.939
0.520
MEERKAT
50
0.920
0.871
0.966
0.734
0.648
0.633
0.614
0.772
Holl-FoiZo
100
03903
0.705
0.934
0.656
0.537
0.634
0.537
0.701
MEERKAT
100
0.913
0.945
0.573
0.634
0.746
Full-FedZO
10
0.891
0.70]
0.931
0.695
0.800
0.682
0.579
0.754
MEERKAT
10
0,944
0.889
0.942
0.788
0.817
0.700
0.656
0.819
Full-FedZ2
30
0.902
0.702
0.930
0.709
0.817
0.663
0.583
0.758
Qwen2-1.5h
Full-FeaZO
0.902
0.705
0.940
0.929
0.701
0,840
0.808
0.663
0.590
0.757
MEERKAT
50
0.942
0.885
0.934
0.784
0.840
0.634
0.637
0.508
Fal-FaZ0
0.811
0.682
01766
MEERKAT
100
0.946
0.386
0.930
0.776
0.804
0.653
0.653
0.807
Fell-Fed20
10
0.87
0.752
0.944
0.717
0.564
0.634
0.592
0.723
MEERKAT
10
0.943
0.892
0.97
0.817
0.724
0.653
0.636
0.805
Fol-Feazo
0.912
0.641
Gemma2-2h
Fell-
MEERKAT
50
30
0.911
0.943
0.887
0.812
0.912
0.973
0.735
0.812
0.551
0.617
0.663
0.634
0.608
0,572
0.757
0.786
MEERKAT
50
0.94
0.873
0.964
0.812
0,604
0.234
0.778
Fell-Fe220
0.917
0.916
0.56
0.644
MEERKAT
100
0.949
0.87
0.954
0.815
0.568
0.034
(L592
0.769
44

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of MEERKAT-VP and MEERKAT under Non-IID client data setting, with the same local step and sparsity. Tasks include SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC. "Acc" indicates the average test accuracy across all tasks. Bold numbers highlight the best result in each row.
Methods
Local Step
SST-2
AgNems
Yelp
BoolQ
RTE
WSC
WIC
Acc
MEERKAT-VV
0.922
0.962
0.713
MEERKAT
10
0.916
0.872
0.964
0.695
0.600
0.633
0.614
0.759
MIERKAT-V
30
0.919
0.325
0.961
0.685
0.595
0.634
[.631
0.750
LLaMA-32-IB
MEERKAT
30
0.897
0.862
0.965
0.646
0.577
0.644
0.583
0.739
MEERKAT-VP
0.909
0.959
0.691
0.615
0,615
MEERXAT
0.909
0.827
0,965
0.617
0.595
0.64
0.567
0.734
MEERKAT-VP
T00
0.904
0.824
0.962
0.684
0.371
0.533
0.630
0.747
MIERRAT
100
0.896
0.777
0.961
0.658
0.577
0.644
0,573
0.726
MEERKAT-VP
0.941
0.947
0.76
MEERXAT
0.949
0.881
0.934
0.252
0.813
0.682
0.628
0.805
MEERKAT-VP
30
0.935
0.876
0.953
0.759
0.822
0.633
0.626
0.803
Owen2-1.5b
MEERKAL-V
MIERRAT
30
0.944
0.381
0.878
0.928
0,944
0.734
0.754
0.800
0.663
0.624
0.77%
0.795
MEERKAT
0.948
0.926
0.246
0,504
0.792
MECRKAIVE
100
0.335
0.874
0.947
0.751
0.817
0.553
0.644
0.803
MOERKAT
100
0.936
0.878
0,925
0.741
0.795
0.663
0.51
0.792
MEERKAT-VP
0.948
0.873
0.971
0.802
0.657
0.663
0.609
0.789
MEERKAT
0.939
0.96
0.804
0.772
MEERKAT-VP
30
0.948
0.86
0.974
0.799
0.6
0.634
0.619
0.776
2emma2-2h
MEERKAT-VP
MEERXAT
30
0,349
0.91
0.851
0.855
0,969
0.917
0.731
0.782
0.568
0.551
0.615
0.644
0.620
0.601
0.762
0.755
MEERKAT
0.945
0,966
0.767
0.623
0.772
MEERKAT-VP
100
0.944
0.12
0.97
0.233
0.551
0.631
0.634
0.754
MEERKAT
100
0.94
0.851
0.951
0.745
0.551
0.634
0.574
0.749
Table 15: MEERKAT performance at local step- = with varying outlier percentages across the LLaMA-3.2-1B, Qwen2-1 .5b, and Gemma2-2b models. We report test accuracy on SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC under both IID and Non-IID client data settings. Bold numbers indicate the highest value in each row.
IID
Non-1ID
Model
Outlier Percestage
SST-2
AgWews
Yelp
Bool))
RIE
WSC
WIC
SST-2
AgNews
Yelp
Bool()
RIE
WSC
WIC
5e
0.917
0.72
0.965
0.725
0.653
0.646
0.634
0.395
0.669
0,964
0.684
0.644
0.653
0.594
1.M1-3.2-1B
3e-3
5e-2
0.913
0.500
0.585
0.361
0.966
0.971
0.769
0.249
0.653
0.7U2
0.653
0.646
0.614
0.633
0.915
0,930
0.874
0.87
0.963
0.97
0.722
0.753
0.620
0.653
0.664
0.66
0.619
0.62
3c-4
0.910
0.777
0.954
0.773
0.720
0,663
0.641
0.11
0.888
0.956
0,700
0.693
0.633
0,628
3e
0.922
0.879
0.964
0.724
0.631
0.625
0.648
0.92
0.876
0,940
0.725
0.613
0.663
0.626
5e:1
0.351
0.355
0.947
0.766
0.82
0.663
0.644
0.865
0.854
0.946
0.753
0,826
0.82
0.631
Owen2-1..5
36
5p-3
0.925
0.926
0.368
0351
0.945
0.939
0.778
0.763
0.825
0.133
0,692
0,692
0,658
0.647
0.24
0.93
0.866
0.853
0,943
0.94
0.759
0.75)
0.822
0,822
0.63
BA92
0.661
0,665
5e-4
1192
0.764
0.943
0.774
0.813
0.682
0.645
0915
0.84
0.943
0.762
0.813
0682
0.647
36-5
0.505
0.78
0.941
0.745
0,80
0.673
0.625
0.996
0.799
0.937
0.739
0.30
0.63
0,633
56-1
0.342
0.867
0.963
0.55
0,673
0.626
0871
0.55
0,952
0.695
0.653
8663
0.619
Gonmal-22 2bb
36:3
56-2
0.952
0.932
0.571
0.578
0.91
0.977
0,09
0.37
(1791
0.500
0.663
0.663
0,639
0.623
0.942
0.2
0.833
0,863
0.968
0.97
0,807
0.766
0.751
0.706
0.633
0653
0.634
0.645
564
0.941
0.244
0.967
0.83
0.764
0.663
0.612
0.41
0.83
0.962
0,831
0.746
0634
0.63
36-5
0192
0.528
0.952
0.797
0.6
0.644
0.606
0.22
0.764
0.949
0.774
0.56
0634
0.601
Table 16: Performance comparison of Full-FedZO, FedZO, and MEERKAT under synchronous updates with localstep = 1, evaluated on both IID and Non-IID client data settings(Dirichhe a = 0.5) across LLaMA-3.2-IBB Qwen2-1.5h .bb, and Gemma2-2b. We report test accuracy on SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC, Bold numbers indicate the highest value in each row.
Model
Method
S81-2
AgNews
Yelp
HoolQ
RTE
WSC
WIC
Ace
Full FediOFeeZO
0.918
0.801
0.937
0.686
0.54
0.625
0.58
0.726
LLaMA-32-IB (ID)
LoRA-FedZO
0.855
0.914
0.599
0.599
MEERKAT
0.900
0.885
0.971
0.773
0.702
0.653
0.614
0.785
Full-FedZ0
0.911
0.831
0.937
0.672
0.528
0.587
0.567
0.719
1.LMAA332-18 (Non-IID)
LoRA-Fed7O
0.6669
01842
0.53
MERRKAT
0.93
0.858
0.963
0.67
0.66
Full-FedZO
0.9013
0.726
0.918
0,700
0.797
0.710
0.579
0.761
Qwen2-1.5h (IID)
LoRA-FelZO
0,935
0.752
0.925
0.686
0.794
0.673
0.600
0.767
MEERKAT
0.945
0.813
0.692
Full-FedZO
0.844
0.725
0.937
0.688
0.769
0.663
0.565
0.741
Qwen2-1.Sh (Non-IID)
LoRA-FedZO
0.932
0.76
0.944
0.682
0.773
0.682
0.565
0.763
MEERKAT
0.924
0.866
0.94
0.762
0.822
0.692
0.661
0.509
Fall-FodZ00
0.934
0.84
0.953
0.774
0.542
0.644
0.6066
0.756
Gemma2-2h (IID)
LRA--Fel7
0.942
0.856
0.94
0.735
0.52
0.644
(1.006
0.749
MEERKAT
0,952
0.871
0.971
0.837
0.8
0.663
0.639
0,819
FedZ0
0.56
0.575
Gemma2-2h 2h (Non-IID)
LoRA-FedZO
0.9415
0.825
0.954
0.711
0.528
0.625
0.578
0.737
MEBRKAT
0.942
0.853
0.97
0.307
0.751
0.653
0.645
0,803
45

---

Published as a conference paper at ICLR 2026
Table 17: Performance comparison of Full-FedZO,, LoRA-FedZO, and MEERKAT AT under synchronous updates with localstep = 1, evaluated on Non-IID client data settings (Dirichlet a = 0.3) across LLaMA-3.2-IB, Qwen2-1.5b, and Gemma2-2b. We report test accuracy on SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC. Bold numbers indicate the highest value in each row.
Model
Method
SST-2
AgNews
Yelp
BoolQ
RTE
WSC
WIC
Acc
Full-FedZO
0.891
0.759
0.94
0.623
0.528
0.644
0,551
0.705
1.MMA-32-1B (Non-IID)
LoRA-FadZD
0.955
03866
0.952
0.646
0.586
0.653
MEERKAT
0
0.843
0.97
0.761
0.626
0.653
0.609
0.769
Fell-FeuZZ
0.52
0.347
0.45
0.62
0.532
0.632
0.51
0,516
Qwen2-1.5h (Non-1II)
LoRA-FalZO
0.732
0.72
0.634
0.603
MEERKAT
0.91
0.809
0.954
0.772
0.822
0.682
0.661
0.801
Fell-FelZ0
0.881
0.761
0.94
0.688
0.552
0.613
0.603
0.720
Gemma2-2b (Non-IID)
LaRA-FadZO
0.922
0.826
0.921
0.681
0.52
0.625
0.606
0.729
MEERKAT
0.942
0.873
0.97
0.806
0.688
0.634
0.615
0.79
Table 18: Performance comparison of Full-FedZO, LoRA-FedZO, and MEERKAT under synchronous updates with localstep = 1. evaluated on Non-IID client data settings (Dirichlet a = 0.1) across LLaMA-3.2 1BB Qwen2-1.55 11.5b, and Gemma2-2b. We report test accuracy on SST-2,
AgNews, Yelp, BoolQ, RTE, WSC, and WIC. Bold numbers indicate the highest value in each row.
Model
Method
SST-2
AgNews
Yelp
BoolQ
RTE
WSC
WIC
Acc
Full-FedZO
0
0.754
0.933
0.626
0.522
0.365
0.512
1.MMA-32-1B (Non-IID)
LoRA-FadZD
0.845
0.942
0.643
0.533
0.365
0.559
MEERKAT
0.92
0,794
0.965
0.745
0,582
0.644
0
0.750
Full-FedZ2
0.49
0.247
0.44
0.62
0.528
0.634
0.5
0.494
Owen2-15h (Non-IID)
LeRA-FedZO
0.746
0.548
0.601
MIERKAT
0L889
0.78
0.944
0.732
0.822
0.634
0.637
0.777
Full Fuul-F
0.879
0.741
0.937
0.681
0.48
0.634
0.601
0,708
Gemma2-2b (Non-IID)
LoRA-FalZO
0.567
MEERKAT
0.944
0.866
0.971
0.805
0.728
0.605
0.628
0.792
Table 19: Test accuracy of MEERKAT versus DecomFL. on Qwen2-1.5b with a single local step under Non-IID data settings (Dirichlet a 1). Results are shown for SST-2, BoolQ, RTE, and WSC; bold indicates the best score in each row. Experiments use 8 clients in total, with 2 clients participating in each round, following the DecomFL configuration.
Model
Method
SST-2
BoolQ
RTE
WSC
DecomFL
0.868
0.674
0.773
0.653
Qwen2-1.5b
MEERKAT
0.918
0.734
0.817
0.682
Table 20: Performance comparison of Mask, and MEERKAT under synchronous updates with loealstep = 1, evaluated on IID client data settings across LLaMA-3.2-11 BB, Qwen2-1.55 and Gemma2-2b. We report test accuracy on SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC. Bold numbers indicate the highest value in each row.
Model
Method
SST-2
AgNews
Yelp
BoolQ
RTE
WSC
WIC
Task
0.910
0.847
0.957
0.718
0.661
0.644
0.661
LLaMAA-32-IB (IID)
MEERKAT
0.90
0.885
0.971
0.773
0.702
0.653
0.614
Task
0.936
0.827
0.954
0.765
0.83
0.711
0.664
Qwen2-1.5b (IID)
MEERKAT
0.926
0.851
0.945
0.778
0.813
0.692
0.658
Gemma2-2h (IID)
Task
0.942
0.868
0.972
0.78
0.728
0.644
0.6
MEERKAT
0.952
0.871
0.971
0.837
0.8
0.663
0.639
46

---

Published as a conference paper at ICLR 2026
LLaMA-3.2-1B
Qwen2-1.50
Gemma2-2b
IID
0.8030.801
0.819
0.8
Non:IID
0.739
0.785
0.769
0.761
0.767
0.758
0.749
0.790
Accuracy
0.726
0.705
0.720
0.7
0.6
0.516
0.5
Full-FeIZO
LoRA-FedZO
Meerkat
Full-FedZO
LoRA-FedZO
Meerkat
Full-FezZo
LoRA-FedZO
Meerkat
(a) This figure compares three methods Full-FedZO, LedZo,
MEERKAT on three LLMs: LLaMA-3.2-1B IB, Qwen2-1.5b, and Gemma2-20 2b. The x-axis shows the different methods, and each method has two bars indicating performance under IID and Non-IID settings. The Non- IID results are obtained under Dirichlet 0.3 The y-axis represents the average test accuracy across multiple downstream tasks SST2, AgNews, Yelp, BoolQ, RTE, WSC, and WiC.
LLaMA-3.2-1B
Qwen2-1.5b
Gemma2-2b
0.9
0.8
Non-llD
0.749
0.750
0.761
0.767
0,809
0.777
0.756
0.749
0.792
Accuracy
0.726
0.708
0.716
0.7
0.684
0.5
Full-FedZ0
LoRA-FedZO
Meerkat
Full-FedZO
LoRA-FedZO
Meerkat
Full-FedZO
LoRA-FedZO
Meerkat
(b) This figure compares three Srathodd-FFll-FedZOOO LoRA-FedZO, and MEERKAT-On three LLMs: LLaMA-3.2-IB, Qwen2-1.5h, and Gemma2-2b. The x-axis shows the different methods, and each method has two bars indicating performance under results are obtained under Dirichlet a 0.1 .hh y-axis represents the average test accuracy across multiple downstream tasks SST2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC.
Non- IID settings. The Non-IID
Figure 6: Comparison of Full-FedZO, LoRA-FedZO, and MEERKAT on LLaMA-3.2-1B, Qwen2- 1.5b, and Gemma2-22 under IID and Non-IID settings with varying Dirichlet a. Subfigure (a) presents results for Non-IID data generated with 0.3, while Subfigure (b) shows results for Non-III data with a = 0.1.
Table 21: Performance comparison of Task-Mask, which uses downstream task data to select
sensitive model parameters, and MEERKAT under synchronous updates with localstep = evaluated on Non-IID client data settings (Dirichlet and Gemma2-2b. We report test accuracy on SST-2, AgNews, Yelp, BoolQ, RTE, WSC, and WIC. Bold numbers indicate the highest value in each row.
0.5) across ALaMA-3.2-IB Qwen2-1.5b
Model
Method
SST-2
AgNews
Yelp
BoolQ
RTE
WSC
WIC
LLaMA-3.2-1B (Non-IID)
Task
0.904
0.874
0.956
0.744
0.591
0.615
0.622
MEERKAT
0.93
0.888
0.963
0.753
0.62
0.66
0.62
Qwen2-1.55 1.5b (Non-IID)
Task
0.938
0.863
0.956
0.779
0.817
0.692
0.65
MEERKAT
0.924
0.866
0.94
0.762
0.822
0.692
0.661
Gemma2-2b (Non-IID)
0.91
0.966
0.72
0.644
0,578
MEERKAT
0.942
0,853
0.97
0,807
0.751
0.653
0.645
47

---

Published as a conference paper at ICLR 2026
Table 22: Test accuracy of MEERKAT versus Task-Mask on Qwen2-1.5b with a local step under Non-IID data settings (Dirichlet bold indicates the best score in each row. Experiments use clients in total, with 2 clients participating in each round, following the DecomFL configuration.
0.5). Results are shown for SST-2, BoolQ, RTE, and WSC;
Model
Method
SST-2
BoolQ
RTE
WSC
Task
0.932
0.784
0.823
0.681
Qwen2-1.5b
MEERKAT
0.944
0.752
0.813
0.682
Table 23: MEERKAT Convergence Rounds for the L.aMA-3.2-II Gemma2-2B, and Qwen2-1.5B models on the SST-2, AgNews, Yelp, and BoolQ tasks, with 10 local steps.
Model
SST-2
AgNews
Yelp
BoolQ
Gemma2-2B
39
61
29
43
Qwen2-1 1.5B
51
75
36
70
LLaMA-3.2-IB
85
77
52
Table 24: Computation and Communication Efficiency Benchmark Shows MEERKAT'S Superior
Resource Usage over Baselines. We benchmarked resource usage on Qwen2-1.5B with 10 clients (FP16). Setting: Full-FedZO Meerkat vs LoRA-FedZO, where LoRA is configured with rank
16, a 16-the same setting used in Table I
Method/Metries
RAM (Peak)
Upload/Client
Downlosd/Client
Full-FedZO
12,600 MiB
0.078 KB
2,875 GB
LoRA- LoRA-FedZO
10,741 MiB
0.078 KB
35.22 MB
MEERKAT (0.1%) 11 mask)
7,850 MiB
0.078 KB
2.50 MB
Table 25: Performance comparison on LLaMA-3.2-11 1B under Non-IID Dirichlet partition (a 0.5) with T = 10 local steps. While ZO methods cannot match proppaaopgttioon due to gradient noise from limited sampling, MEERK AT-VP achieves competitive accuracy (0.764 avg) with significantly lower memory consumption, and outperforms several Non-IID FL. baselines (FedDYN, FedAvgM, FedSA-LoRA, and stochastic controlled) under the same training setup. We adapt FedDYN
following the original paper with = 0.01.
Method
SST-2
AGNews
Yelp
BoolQ
RTE
WSC
WIC
Avg
Back- packkpropaggtion
0.925
0.893
0.968
0.751
0.644
0.660
0.630
0.782
Stochastic Controlled
0.880
0.720
0.901
0.612
0.523
0.612
0.580
0.690
FedAvgM
0.901
0.821
0.941
0.629
0.580
0.613
0.600
0.726
FedS LORA
0.905
0.832
0.920
0.630
0.570
0.622
0.570
0.721
MEERK ATEERKAT+FeDDYN
0.917
0.841
0.954
0.638
0.564
0.615
0.570
0.728
MEERKAT-VE
0.922
0.864
0.962
0.713
0.617
0.644
0.625
0.764
Table 26: Effect of different sparsity density ratios on LLaMA-3-8B under Non-IID Dirichlet
a 0.5. We compare density ratios 10-3 and 10-4 using the same transferable mask construction pipeline. Both settings achieve strong performance.
Density ratio
SST-2
AGNews
Yelp
BoolQ
RTE
WSC
WIC
Avg
10-3 0.950
0.851
0.954
0.831
0.755
0.674
0.660
0.811
1 10
0.941
0.862
0,956
0.861
0.783
0.664
0.640
0.815
Table 27: Scalability of MEERKAT and MEERKAT-VP with respect to the number of clients on Qwen2-1.5B. Increasing the number of clients from 10 to 20 does not degrade performance; MEERKAT-VP-20 even slightly improves the average accuracy.
Model
SST2
AgNews
Yelp
BoolQ
MEERKAT-VP (20 clients)
0.951
0.885
0.936
0.756
MEERKAT (20 clients)
0.929
0.869
0.922
0.719
MEERKAT (10 clients)
0.949
0.881
0.934
0.752
48

---

Published as a conference paper at ICLR 2026
Gradip vs. Step
Gradip Stop
Gradtp Gradtp (D)
100
150
Gradip
60
Gradlp
100
40
20
20
40
60
80
100
0
20
40
60
60
100
Step
Step
(a) The GradIP measured for IID and Non- IID client data under the WIC task using the 11lma-3.2-111 BB model.
(b) The GradIP measured for IID and Non- IID client data under the AgNews task using
the Llama-3.2- 1B model.
Gradip VS. Step
Gradip vs. Step
150
Gredte: (ID)
125
Gractp (II)
125
Greate (Non-lD))
Gractp thon-1D)
100
100
Gradio
Gradio
75
25-
0
0
20
40
60
80
100
20
40
60
80
100
Stop
Step
(c) The GradIP measured for IID and Non-
(d) The GradIP measured for IID and Non- IID client data under the BoolQ task using the 3lama-3.2211 2-B model.
IID client data under the Yelp task using the Llama-3.2 3.2- IB model.
Gradip vs. Step
Gradip vs. Step
Gracip (ID)
100
Gradp (12)
100
Gracip (Kan-4D)
Gradp (len-1D)
Gradip
60
20
20
60
20
60
Stop
Step
(c) The GradIP measured for IID and Non- IID client data under the RTE task using the Llama- 3.2- model.
(f) The GradIP measured for IID and Non- IID client data under the WSC task using the Llama-3.2-11 2- model.
Gradip Step
Grodip (IB: Grodp (ND
1000
800
600
400
Stop
(g) The GradIP measured for III and Non- IID client data under the BoolQ task using the Gemma-2-2h model.
Figure 7: These figures show GradIP (Definition 2.3b curves under IID and Non-IID settings, computed over 100 local training steps on six datasets (WSC, BoolQ, RTE, WIC, AgNews, Yelp) using the Llama-3.2-11 model with density level 5 x 10-3, An extra BoolQ result is shown for the Gemma-2-2B model.
49

---

Published as a conference paper at ICLR 2026
Cosine Similarity vs. Step
Grad Norm Step
0.0012
Cosine Smilanty
Norm (Non-IID)
Casine Similarity
200000
150000
0.0000
0.0004
0.0002
0.0000
80
Step
Step
(a) Cosine similarity between local 70 gradients and C4 pre-trained gradients.
(h) Gradient norm from local 70 training under Non-IID and IID data distribution.
The left panel shows the cosine similarity between locally computed ZO gradients and gradients from the C4-pre-trained data, illustrating that the two gradient vectors remain nearly orthogonal throughout training. The right panel presents the norm of local 70 gradients over training steps, showing consistent decay and convergence in magnitude under Non-IID and III) data distribution. These observations are obtained under density level of x 10-3
Figure 8:
Gradip Step
Gradip Step
150
Gradio (Non-IID
125
100
Gredip
Gradip
75
50
25
25
20
40
60
80
100
0
20
40
60
80
100
Step
Step
(a) GradIP for Non-IID elients on the AgNews task, where the two classes have a highly imbal- anced ratio (5 vs. 89 samples).
(b) GradIP for Non-IID client data on the BoolQ task, where the two classes have highly imbal- anced ratio (6 vs. 190 samples).
Figure 9: These subfigures show GradIP (see Definition 2.3 for LLaMA-3.2- IB under Non-IID client data with 100 local training steps. Subfigure (a) uses AgNews (5 vs. 89), while Subfigure (b) uses BoolQ (6 vs. 190).
50

---

Published as a conference paper at ICLR 2026
GradIP vs. Step BoolQ Qwen
Client 0 (ratio 1:1)
12.
Client 1 (ratio 1:10)
Client 2 (ratio 1:20)
10
6-
Gradip
2
0-
30
40
50
60
70
80
90
100
Step
(a) The experiments, conducted using the Qwen2-1 model on the BoolQ dataset, reveal that under with a 1:20 class imbalance there a pronounced decline in GradIP between the early and later stages of training. In the extreme Non-IID case, the GradIP values in the later stages lend approach zero.
Non-IID
GradIP Step BoolQ Llama
Client 0 (ratlo 1:1)
Client 1 (ratio 1:10)
20
Client (ratio 1:20)
GradIP
10
70
80
90
100
Step
(b) The experiments, conducted using the Llama-3.2- 3.2- model on the BoolQ dataset, reveal that under Non-IID eettins--especilly with a 1:20 class imbalance there is pronounced decline in GradIP between the early and later stages of training. In the extreme Non-IID case, the GradIP values in the later stages tend approach
GradIP vs. Step BoolQ Gemma
250
Client 0 (ratio 1:1)
Client (ratio 1:10)
200
Client 2 (ratio 1:20)
150
Gradip
100
0-
40
50
60
70
90
100
Step
(c) The experiments, conducted using the Gemma-2-2B model on the BoolQ dataset, reveal that under Non-IID setting--especially with 1:20 class imbalance-there is pronounced decline in GradIP between early later stages training. In the extreme Non-IID case, GradIP values in the later stages tend to approach zero.
Figure 10: GradIP analysis for different models on the BoolQ dataset under Non-IID and IID conditions: As the class imbalance ratio increases, GradIP in the later training stages tends to approach zero. This decline more pronounced under Non-IID settings, where the gap between initial and final GradIP values is larger than in the IID case. All trends are visualized using moving average for clarity.
51

---

Published as a conference paper at ICLR 2026
GradiP vs. Step AGnows Owen
Client 0 (ratio 1:1:1:1)
Client (ratio 5:1:1:1)
Client (ratio 1:15:1:1)
15
Gradip
10
5-
60
Step
(a) The experiments, conducted using the Qwen2-1.5B model on the AGNews dataset, reveal that under Non-IID settings- settings-espccially with 1:15:1:1 class imbalance there a pronounced decline in GradIP between early and later stages of training. In the extreme Non-IID case, the GradIP values in the later stages tend to approach zero.
GradiP vs. Step AGnews Llama
Client (ratio 1:1:1:1)
Client (ratio 5:1:1:1)
50
Client (ratio 1:15:1:1)
Gradip
10
60
Step
(b) The experiments, conducted using the Llama-3.2-18 model on the AGNews dataset, reveal that under Non-IID settings especially with ::15:1:1 class imbalance teeeencee--rrreeeeeeeee is a pronounced decline in GradIP between the carly and later stages of training. In the extreme Non-IID IID case, the GradIP values in the later stages tend approach
Figure 11: GradIP analysis for different models on the AGNews dataset under Non-IID and IID
conditions: As the class imbalance ratio increases, GradII in the later training stages tends to approach zero. This decline is more pronounced under Non-IID settings, where the gap between initial and final GradIP values larger than the case. All trends are visualized using a moving average for clarity; consequently, the plotted lines do not begin at step zero, as the initial data points are used to compute the first averaged value. This is an intentional effect the visualization, not an error or result of missing data.
52

---

Published as a conference paper at ICLR 2026
Before MEERKAT-VE
Client
70
02
GradlP
20
10
20
40
60
80
100
Local Training Step
(a) GradIP trajectories for the two extreme Non-IID clients the SST2 task with Qwen2-1.555 1.5B, before MEERKAT-VP training.
MEERKAT-VP Managed
Client Client
100
100
GoodP
60
20
80
100
Local Training Step
(b) GradIP trajectories for the extreme Non-IID clients SST2 after MEERKAT-VE VP training has converged (global validation accuracy z 90%).
Figure 12: GradIP trajectories (Definition 2.3 for Qwen2-1.55 1.5BB SST2 with clients (2 extreme Non-III,, 4 IID. Subfigure (a) shows the GradIP trajectories of the two extreme Non-III clients at initialization, while Subfigure (b) shows the trajectories for the same clients after MEERKAT-VP training has converged. The shape of the trajectories remains similar before and after training, supporting claim that GradIP is primarily data dataadistribution-die signal rather than direct reflection of the global model state.
53


Reviews:
[{"review_id": "hcmEXDvqra", "reviewer": "Reviewer_bNcr", "summary": "This paper proposes a sparse zeroth order federated fine tuning framework for LLMs that is both theoretically grounded and highly practical. MEERKAT uses a transferable 0.1 percent sensitivity mask and frequent synchronization to tame memory and bandwidth while reducing Non IID drift. Virtual path reconstruction and the GradIP score then allow the server to detect extreme Non IID clients and actively limit their damage through MEERKAT VP. Experiments across multiple open LLMs and multiple classification and reasoning benchmarks show consistent gains over strong baselines like Full FedZO and LoRA FedZO at matched communication frequency, along with dramatic savings in memory and bandwidth.", "questions": "see weakness 1, 2, 3", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Virtual path and GradIP are novel ideas. By reconstructing each client’s local update trajectory, the server gains visibility into how that client is moving in parameter space without seeing private data. GradIP then becomes a quantitative signal that reveals which clients are extreme Non IID.\n\n2. The paper provides convergence analyses for both MEERKAT and MEERKAT VP.\n\n2. Strong empirical validation. The experiments cover multiple open LLMs (Llama 3.2 1B, Qwen2 1.5B, Gemma2 2B), multiple tasks (SST2, AgNews, Yelp polarity, BoolQ, RTE, WSC, WiC), both IID and strongly Non IID Dirichlet splits, and multiple baselines", "weaknesses": "1. hyperparameters in MEERKAT VP : The early stopping rule uses thresholds on GradIP phase ratios and quiescent duration. Although a small sensitivity study is reported, it is still unclear how practitioners should pick these thresholds for new tasks with no oracle labels. \n\n\n2. Baselines in experiments:  There is active work on federated LoRA under heterogeneous clients, which explicitly tackles aggregation noise, knowledge contamination, and aggregation distortion by separating global and client specific structure or by using rank adaptive aggregation. The paper mentions LoRA FedZO but does not deeply compare against these newer structured aggregation approaches, so it is hard to see whether MEERKAT VP replaces them, complements them, or could be combined with them. Existing works are listed below: \n\n[1] Zhe Li, Bicheng Ying, Zidong Liu, Chaosheng Dong, and Haibo Yang. Achieving dimension-free\ncommunication in federated learning via zeroth-order optimization. In The Thirteenth International\nConference on Learning Representations, 2025.\n\n[2] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated\nlearning. In The Twelfth International Conference on Learning Representations, 2024.\n\n[3] Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective\naggregation for low-rank adaptation in federated learning. In The Thirteenth International\nConference on Learning Representations, 2025.\n\n3. Lack of explicit adversarial evaluation: MEERKAT VP is essentially throttling clients whose updates are harmful. This is similar to defending against malicious or poisoned clients. However, the experiments do not include adversarial threat models like label flipping or Byzantine behavior. Showing that GradIP based early stopping also mitigates such attacks would significantly strengthen the robustness story.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNb9VBNcCX", "reviewer": "Reviewer_TB4g", "summary": "This paper introduces MEERKAT and its extension MEERKAT-VP, a novel methodology to tackle the twin challenges of high communication overhead and Non-IID data drift when fine-tuning LLMs in a Federated Learning setting. MEERKAT addresses efficiency by using a sparse Zeroth-Order Optimization method that limits fine-tuning to an extremely sparse (e.g., <0.1%), static subset of parameters, drastically reducing communication costs and enabling high-frequency server synchronization to mitigate drift. Building on this, MEERKAT-VP uses the concept of a virtual path based on local updates to detect the GradIP phenomenon in extreme Non-IID clients, subsequently applying early stopping to restrict these clients' local steps, thereby improving global model quality. The experiments confirm that MEERKAT, through its combined approach of extreme sparsity and strategic client management, effectively mitigates Non-IID data challenges and achieves superior performance and efficiency compared to existing baselines.", "questions": "i.Would the performance of MEERKAT/MEERKAT-VP significantly decline as the number of clients increases to a larger magnitude? For example, would the server-side virtual path reconstruction incur a higher computational overhead with an increased number of clients, and has the potential reduction in the efficiency of MEERKAT-VP's early stopping strategy been considered if a large number of clients are categorized as \"extreme Non-IID\"?\n\nii.The early stopping strategy of  MEERKAT-VP relies on a threshold for identifying the GradIP phenomenon. How is this threshold determined? Should the paper analyze the model performance's sensitivity to this threshold choice, and potentially propose an automatic or adaptive mechanism for threshold selection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "i.MEERKAT employs an extremely sparse and static subset of parameters for fine-tuning. This drastically reduces the communication load and memory consumption on client devices, addressing the primary scalability bottleneck of LLMs in FL.\n\nii.The extreme sparsity enables cost-effective high-frequency client-server synchronization. This high synchronization rate is key to effectively suppressing the client drift caused by Non-IID data distributions across decentralized clients.\n\niii.The paper proposes using a static sparsity mask identified through gradients from pre-training data. This transferable sparsity ensures that the chosen small subset of parameters is highly effective for downstream tasks, allowing for consistent performance throughout the FL process.\n\niv.MEERKAT-VP introduces the Virtual Path mechanism, which allows the server to diagnose the severity of Non-IID data without accessing the raw client data.It utilizes the discovered GradIP phenomenon to strategically identify and apply early stopping to severely Non-IID clients. This ensures robustness and convergence quality while maintaining data privacy.\n\nv.The work provides strong experimental evidence demonstrating that MEERKAT consistently outperforms full-parameter Zeroth-Order Optimization and other SOTA sparse fine-tuning methods under various Non-IID settings.\n\nvi.The use of Zeroth-Order Optimization , combined with extreme sparsity, makes the approach highly suitable for clients with limited computational resources, as it avoids the need to calculate full, complex second-order gradient information.", "weaknesses": "i.In the description of the steps in Figure 1 of the paper, the word \"Aggregrate\" appears, and the correct spelling should be \"Aggregate\".In the phrase on lines 153–154, \"(3) Sever aggregates and initiate the next round...\", shouldn't the word \"Sever\" actually be spelled \"Server\"?Line 165 contains a redundant repetition of the article \"a\" in the phrase \"and a a new seed list.\"\n\nii.Although the paper validates the effectiveness of the extreme sparsity level of 0.1%, it fails to deeply analyze the performance impact of varying sparsity levels (such as 0.01%, 0.5%, 1%) across different scales of LLMs (e.g., larger models like 7B/13B) and different task types (e.g., complex reasoning tasks versus simple classification tasks). Furthermore, the transferability of the sparsity mask was only tested on a few domain shift datasets (C4, Wiki, Code datasets) and was not verified for its adaptability in highly specialized domains (such as legal), thus making it difficult to support a conclusion of broader applicability.\n\niii.The paper compares MEERKAT/MEERKAT-VP against Full-FedZO, LoRA-FedZO, and the baseline FedDYN improvement method. However, it fails to include state-of-the-art Federated LLM fine-tuning methods specifically designed for Non-IID scenarios. These missing baselines include established methods adaptable for LLMs, such as FedAvgM and SCAFFOLD, or FedSparse. This omission makes it difficult to ascertain whether the performance advantages of the proposed method truly surpass the latest research achievements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QOVLWJLch", "reviewer": "Reviewer_Gtsq", "summary": "This paper proposes MEERKAT, a sparse zeroth-order optimization method for federated LLM fine-tuning. By updating a static, highly sparse subset of parameters, MEERKAT reduces communication costs and enables frequent synchronization to alleviate Non-IID drift. It also introduces a “virtual path” mechanism that uncovers the GradIP phenomenon—the inner product between server and client gradients varies across data heterogeneity. Building on this, MEERKAT-VP detects extreme Non-IID clients and mitigates their impact through early stopping. The paper provides theoretical convergence results and validates the method with extensive experiments.", "questions": "* The client identification mechanism in MEERKAT-VP uses an \"initial phase\" and a \"later phase\" of local training to analyze GradIP scores. What is the motivation for this two-phase design?\n* Figure 3 shows the GradIP trajectories. To make the effect of the proposed solution more intuitive, could the authors add a plot showing the trajectory for a client being managed by MEERKAT-VP (i.e., after mitigating the global weight changes after the design)?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly presented, systematically proposing three distinct claims and validating them one by one.\n* It includes a detailed convergence analysis for the proposed methods.\n* The experimental evaluation is comprehensive, covering various benchmarks and recent LLMs.", "weaknesses": "* The use of sparse updates for communication efficiency in FL is a well-explored area, often framed as model compression. The paper should provide a more thorough comparison with this body of work to clarify the unique contributions of applying sparsity specifically to ZO-based FL fine-tuning. Applying sparsity to new FL models (here is federated fine-tuning) seems not a strong contribution.\n* Methods for tackling the Non-IID issue in compressed FL have also been developed, such as the one in [a]. The paper would be stronger if it compared its GradIP-aware solution not just with other non-sparse fine-tuning methods, but also with compression-aware FL baselines that explicitly address Non-IID data, even if they are not designed for LLMs.\n* The rationale for setting the number of local training steps to one for identified Non-IID clients is not fully convincing. These clients may hold important minority data that is crucial for preventing the global model from becoming biased. You should not move it only because it is minority and hard to converge into the global model. The paper should better justify why this aggressive early stopping is preferable to other strategies that might still leverage the clients' unique data more fully. \n\n[a] Huang, Xinmeng, Ping Li, and Xiaoyun Li. \"Stochastic Controlled Averaging for Federated Learning with Communication Compression.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrdDmQh0tY", "reviewer": "Reviewer_46ze", "summary": "This paper proposes MEERKAT, a novel framework for federated fine-tuning of LLMs that combines Zeroth-Order optimization with extreme sparsity. The core idea is that this combination is extremely memory-efficient and communication-efficient. The authors claim this efficiency enables high-frequency client-server synchronization, which in turn is the key to mitigating Non-IID client drift. Furthermore, the paper introduces a \"virtual path\" mechanism, where the server can reconstruct client update trajectories, and use\"GradIP\", a metric based on the inner product of client and pre-training gradients, that can identify clients with extreme Non-IID data. The authors then propose MEERKAT-VP, an extension that uses this signal to apply early stopping to these problematic clients, further improving global model performance.", "questions": "The core optimization (sparse ZO on a static mask) seems to be a general technique, not specialized for FL. Are there any other properties of this sparse ZO method that make it uniquely suited for the federated setting, beyond the communication efficiency gains and the virtual path mechanisms they enable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Strong Empirical Performance:** The method is well-validated empirically. Empirical results clearly show that MEERKAT consistently and significantly outperforms full-parameter ZO and LoRA-based ZO baselines, especially in Non-IID settings. \n2. **Comprehensive Analysis:** The paper provides strong support for its claims through both theoretical convergence analysis and extensive empirical ablations. The validation is structured clearly around three main claims, and each is convincingly argued with evidence.\n3. **Novel Mechanism for Heterogeneity:** The discovery of the \"GradIP\" phenomenon is a novel contribution. Using this signal, which is enabled by the virtual path, to identify and manage extreme Non-IID clients (MEERKAT-VP) is a clever and effective FL-native strategy for handling heterogeneity.", "weaknesses": "The paper's primary strategy for handling extreme Non-IID clients (MEERKAT-VP) is purely defensive. It aims to \"mitigate their adverse impact\" by identifying them via GradIP and applying early stopping. This implies that the base MEERKAT method is still vulnerable to significant client drift from these clients. This strategy is inherently limited: it focuses on reducing the participation of \"bad\" clients rather than developing a mechanism to effectively learn from their (potentially valuable) information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)