Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning
Authors: Zhiyi Wan
Yijia Chi
Liang Li
Wanrou Du
Miao Pan
Xiaoqi Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning; Communication Compression
Primary area: optimization

Abstract: Federated Learning (FL) offers a privacy-preserving paradigm for distributed model training by enabling clients to collaboratively learn a shared model without exchanging their raw data. However, the communication overhead associated with exchanging model updates remains a critical challenge, particularly for devices with limited bandwidth and battery resources. 
Existing communication compression methods largely rely on simple heuristics based on magnitude or randomness. 
For example, Top-k drops the elements with small magnitude, while low-rank methods such as ATOMO and PowerSGD truncate singular values with small magnitude. 
However, these rules do not account for the discrepancy between the compressed and the original outputs, which can lead to the loss of important information. 
To address this issue, we propose a novel discrepancy-aware communication compression method that enhances performance under severely constrained communication conditions.
Each client uses a small subset of its local data as calibration data to directly measure the output discrepancy induced by dropping candidate compression units and uses it as a compression metric to guide the selection. 
By integrating this strategy, we can enhance existing mainstream compression schemes, enabling more efficient communication.
Empirical results across multiple datasets and models show that our method achieves a significant improvement in accuracy under stringent communication constraints, notably an $18.9\\%$ relative accuracy improvement at a compression ratio of $0.1$, validating its efficacy for scalable and communication-efficient FL.
Our code is available at https://github.com/wzy1026wzy/Discrepancy-aware-Compression-for-FL.

Published as a conference paper at ICLR 2026
ENHANCING COMMUNICATION COMPRESSION VIA DISCREPANCY-AWAR CALIBRATION FOR FEDERATED LEARNING
Zhiyi Wan Yijia Chi Liang Li,, Wanrou Du', Miao Pan Xiaoqi Qin'
Beijing University of Posts and Telecommunications
Peng Cheng Laboratory
University of Houston
ABSTRACT
Federated Learning (FL) offers a privacy-preserving paradigm for distributed model training by enabling clients to collaboratively learn shared model with- out exchanging their raw data. However, the communication overhead associ- ated with exchanging model updates remains a critical challenge, particularly for devices with limited bandwidth and battery resources. Existing communication compression methods largely rely on selection rules based on magnitude or ran- domness. For example, Top-k drops the elements with small magnitude, while low-rank methods such as ATOMO and PowerSGD truncate singular values with small magnitude. However, these rules do not account for the discrepancy be- tween the compressed and the original outputs, which can lead to the loss of im- portant information. To address this issue, we propose novel discrepancy-awe communication compression method that enhances performance under severely constrained communication conditions. Each client uses a small subset of its lo- cal data as calibration data to directly measure the output discrepancy induced by dropping candidate compression units and uses it compression metric to guide the selection. By integrating this strategy, we can enhance existing mainstream compression schemes, enabling more efficient communication. Empirical results across multiple datasets and models show that our method achieves significant improvement in accuracy under stringent communication constraints, notably an 18.9% relative accuracy improvement at compression ratio of 0.1, validating its efficacy for scalable and communication--fticiai FL. Our code is available at https://ithub..oom/wzy 1026wzy/Discrepanny wwwreaeeecy--------aaaaaaa wwwaaa--Compressoon-ffo for-FF................
INTRODUCTION
Federated Learning (FL) has emerged as paradigm for distributed learning, enabling models to be trained across decentralized devices without requiring data leave the local device. This decentral- ized approach reduces data privacy concerns by transmitting parameter updates instead of raw data during training. However, FL. presents unique challenges due the limited communication capacity of client devices, which result in high communication burden during the training process. In this context, a key challenge in FL is optimizing the trade-off between model accuracy and com- munication efficiency. Consequently, most FL studies have reduced server-client synchronization frequency since the inception of FL (McMahan et al. 2017). Clients often train for multiple steps epochs locally before transmitting and aggregating their updates.
One of the key strategies to alleviate the communication burden in FL is the use of communication compression techniques, many of which adapted from distributed learning's communication optimization methods. These methods, such as Top-k (Lin et [ Alistarh et al., 2018 Renggli
2019), Random-k, Random-Blood et al. 2019), aim to reduce the volume of data exchanged between clients and the server in FL. Top-
ATOMO (Wang et al.] (018)) and PowerSGD (Vogels
k, Random-k, and Random Block are element-wise sparsification methods that focus on selecting
These authors contribute equally this work.

---

Published as a conference paper at ICLR 2026
elements to drop based on magnitude random selection. ATOMO and PowerSGD are low-rank methods that transmit the low-rank decomposition of the updates, retaining only the singular values with larger magnitudes and their corresponding singular vectors.
However, all of these communication compression methods in FL follow selection rules based on magnitude randomness, determine which compression units retain for further upload and aggregation. In the case of element-wise sparsification methods like Top-k, the compression units correspond to individual elements. And for low-rank methods like ATOMO, the compression units consist of singular triplets, each of which includes a singular value and its corresponding singular vectors. While these selection rules are simple and efficient, they ignore the impact of dropping specific compression units on the output discrepancy. As result, important units may be discarded, leading to significant compression losses and harming the model's performance. This observation consistent with the well-known gap between Top-k and Random-k in practical behavior, even though they share the almost same worst-case bounds and convergence guarantees (Beznosikov al. 2023]
Condat et al. 2022 Li & Li 2023 Qian et al. 2021 Yi et al. 2025 Karimireddy et al. 2019) This is because the coordinates with the largest magnitudes always captures more informative updates. Our method pushes this intuition further, as detailed in Section 3.2
In FL, there have been many studies that tailor compression methods to the server-client train- ing loop. For example, FedFQ (Li et al., 2024) introduces parameter-level adaptive quantization, selecting per-parameter bit widths via constraint-guide 2024) jointly optimizes uplink and downlink bit widths under energy constraints, derives conver- gence guarantees, and adopts a schedule with decreasing uplink and increasing downlink precision across rounds; FedMPQ (Chen & Vikalo (024) brings mixed-precision quantization to heteroge- neous FL, assigning layer-wise bit widths while the server allocates per-client bit budgets across rounds; AdapComFL (Zhuansun al.) (024) employs compression based on bandwidth prediction; and HGC (Hu et al. (2024) unifies sparsification, quantization, and entropy coding within hybrid framework. These approaches address several meaningful challenges in FI.. However, they still rely on selection rules based on magnitude or randomness, without leveraging Fl Fl.-specific characteristics such as low-frequency communication to address the limitations of existing compression methods.
simulated annealing: FedAQ (Qu et al.)
This limitation partly stems from not fully leveraging a key characteristic that distinguishes FL from traditional distributed learning-its infrequent communication. Unlike data-center training, FL al- lows clients to perform multiple local updates before global aggregation. There is fundamental difference in the type of communicated information: distributed learning typically transmits gradi- multiple local steps (McMa- han et al. 2017 Li et al. 2020 Dandi et al., 2022). As a result, the content to be compressed in HI. carries little direct loss-related information, further reducing the correlation between magnitude and importance as discussed in A.2 Consequently, employing selection rules that rely on magnitude or randomness to determine which compression units drop fundamentally inadequate. Further- more, the infrequent communication in FL makes additional local computation a practical way to maximize the value of each transmitted bit, which would be too costly in distributed learning.
ents, while FL. typically transmits parameter updates accumulated
Building on this observation, we propose a novel discrepancy- aware compression strategy for communication-effici FL. that enhances performance under severely constrained communication conditions. Each client uses small subset of its own local data as a calibration dataset to directly measure the output discrepancy of dropping different compression units, guiding the compression selection process. This discrepancy- randomness with decisions grounded in output sensitivity, leading to more efficient communication and improved model accuracy under constrained communication budgets.
selection replaces selection rules based on magnitude or
Our key contributions are as follows:
We propose a novel discrepancy-awart compression strategy for communication-efficii FL. Each client uses a small local calibration dataset to directly measure and minimize the impact of compression on output discrepancy. This approach addresses the drawback of existing FL compression methods that use magnitude as the importance evaluation.
Our proposed strategy scamlessly enhance existing compression schemes plug- in module. We demonstrate its versatility by improving two representative methods: the sparsification method Top-k and the low-rank decomposition method ATOMO.
2

---

Published as a conference paper at ICLR 2026
We conduct extensive experiments diverse models and datasets. The results show that our approach achieves superior accuracy under stringent communication budgets. For in- stance, on CIFAR-10 with the ViT-tiny model, it yields 18.9% relative accuracy im- provement at a compression ratio of 0.1 compared to the baseline.
2 RELATED WORK
Federated Learning (FL). A classic way to reduce the communication frequency in distributed learning is to allow clients to perform multiple local Stochastic Gradient Descent (SGD) steps before synchronization. The seminal work FL was introduced by Konečný al. 2016), who presented a novel paradigm for distributed optimization in machine learning. Building on this framework, FedAvg (McMahan al.] (017)) introduced the idea of reducing communication overhead by al- lowing clients to perform multiple local iterations between synchronizations, maintaining accuracy while improving wall-clock efficiency. Subsequent theory on local/periodic averaging sharpened this picture: local SGD can match mini-batch SGD in gradient complexity and dramatically reduce communication rounds (Stich 2019). with tighter nonconvex analyses and adaptive schedules in periodic averaging (Lin et al. 2020 Recent works, including AdaComm (Wang & Joshi 2019).
FedPAQ (Reisizadeh et al. 2020p. FedProx et. 2020). SCAFFOLD (Karimireddy al., 2020),
FedBuff (Nguyen et al.) 2022), and CTUS (Wang et al. (024), use drift correction, buffering, and adaptive strategies to improve performance under partial participation and heterogeneous clients. Overall, communication-delal methods aim to reduce uplink and downlink traffic. Our method aligns with the general trend in FL reducing communication cost and improving the trade-off between communication frequency and model accuracy.
Communication-eficient FL. Limited uplink bandwidth and device energy constraints make communication a critical bottleneck FL. To mitigate this, most approaches focus on traditional communication optimization techniques, such as quantization and compression methods. Quanti- zation methods aim to reduce the encoding bit-width of parameter updates. Representative works include QSGD (Alistarh et al., 2017) and SignSGD (Bernstein et al. [2018), while recent FL-specific approaches (Li et al.) 2024] Cao et al. 2024 2025 Qu et al. 2024; Chen & Vikalo 2024) per- form fine-grained adaptive quantization actoss parameters to optimize communication efficiency. Compression methods, including sparsification and low-rank decomposition, have also been widely adopted in FL, originally developed for distributed training. These methods perform compression operations on the matrix itself and its transformed form, respectively. Sparsification methods re- duce communication by transmitting only small subset of updates. The most common approaches are Random-k and magnitude-drive 2019). Beyond these general methods, FL-tailored variants such as STC (Sattler al. 2019)] Fed- SPA (uu et al. 2021a), A AdapComFL. (Zhuansun et al. 2024). and HGC (Hu et al. 2024 have explored adaptive compression ratio, bidirectional compression, secure aggregation, and differential privacy mechanisms. Low-rank methods exploit the low-rank property of weight update matrices, reducing communication load proportional the rank. ATOMO (Wang et al., 2018) was the first to propose this sparsification framework, and PowerSGD (Vogels et al. 2019) quickly approximates low rank using power iteration, widely applied in industrial and open-source distributed training. However, these compression schemes generally rely selection rules based based magnitude or randomness, often resulting in higher compression loss.
Top-k (Lin et al. 2018 Alistarh et al.) 2018)] Renggli et al.)
3 DISCREPANCY-AWAR COMPRESSION SCHEMES FOR FL
In this section, we first present the objectives and motivation behind our approach. We then intro- duce a compression method that estimates the induced output discrepancy from dropping candidate compression units-either element or singular triplet-on a small local calibration dataset, and selectively drops candidates. Subsequently, we derive a compression metric tailored for dif- ferent network architectures and compression techniques. Building this framework, enhance commonly used element- -iise sparsification and low-rank compression methods, while ensuring compatibility with efficient low-rank approximations such PowerSGD.
3

---

Published as a conference paper at ICLR 2026
3.1 OBJECTIVE: MINIMIZE THE OUTPUT DISCREPANCY
For a Transformer layer with a matrix Wo. parameter update matrix W and the corresponding input activations X, the output will be Y = (Wo + W)X. For CNN with kernel size F, stride s, and padding size p. we model the layer as cross-correlation. Let Wk.c.0, WK E R x denote the original kernel and its update from input channel e to output channel & and X be X zero-padded by pixels on each border. Then the output at channel k is Ys(u, e] = E Ei= o (Wk,c,o + Wi. j] Xa|us +i, us + j]. Given compressed update W producing Y', our goal in both cases minimize the output discrepancy on calibration set, measured by the Frobenius norm:
min Ccomp (W- - W) W
Σ X
| =
3.2 MOTIVATION: Low MAGNITUDE F Low IMPORTANCE
Existing compression methods decide which compression units to drop using selection rules that do not incorporate loss- or discrepancy-related information. For example, Top-k drops the small- magnitude elements, while low-rank methods such as ATOMO and PowerSGD truncate small- magnitude singular values. Without considering the output discrepancy induced by compression, these rules can drop compression units whose removal causes large change in the layer output, leading to inefficient utilization of communication resources.
In FL. this challenge is exacerbated by a fundamental difference from traditional distributed learning (DL), where communication is frequent and gradients directly dictate the importance of each coor- dinate. In contrast, FL uses parameter updates across multiple local steps, resulting in compressed content that does not directly correlate with the gradient magnitudes, as detailed in A.2] Therefore, magnitude-basee based selection rules, such those employed in the Top-k and other methods, fail to appropriately capture the true importance of the components being compressed, further limiting the accuracy and communication efficiency trade-off in Fl..
A Motivating Example of Element-wise Sparsification. Selecting the largest elements of a ma- trix W E Romxd does not guarantee minimal distortion, because the actual impact on outputs Y = W depends on the input activations X E Rdx", Write X by rows as X = [f ...........
The induced output discrepancy in the form of Frobenius loss from removing an element wi, equals
Ccomp(w:.). = II(W - W ( | = || = w2, I1
Consider two candidate elements wA1 0.1 and u12 = 10 with their corresponding input activation feature ||fil = 10x and [| P = 10-3 If we drop wi1, the induced output discrepancy would be calculated as Ceomp
= (0.1 10 10 On the other hand, we drop W42 the output discrepancy would be Leomp ( = (10- 10-10033::
10-4 The magnitude-based Top-k rule would keep 222 and drop W.1 However, this choice induces an output discrepancy 10 times larger than the alternative, revealing a clear gap between selecting by element magnitude and selecting by output discrepancy induced. The argument holds whether the Top-k is applied per or globally over all entries of W. Random-k can fare worse by discarding high-impact entries purely by chance.
Motivating Example of Low-rank Decomposition. Truncating smaller singular values is not necessarily safe if the associated singular vectors align with high-energy input directions. Let the SVD of W E R be W Σ ".u.y.T _ A standard low-rank truncation would remove the smaller singular value 01. However, the induced output discrepancy the form of Frobenius loss from dropping component equals
Ccompla; (a:) (I(W- Wx ||o, uo, uo,, of [1u X)II (o,
1|, X
Consider two candidate singular values og = with |oT X||r = 103 and 02 = 100 with |uu X|| = 10-3 If we drop di, the induced output discrepancy would be calculated as foomp (T1) 1 103 103. On the other hand, if we drop d2 the output discrepancy would be Lcomp ( 100 10-3 10 Therefore, removing the smaller singular value incurs an output discrepancy 10 times larger than removing the larger one.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: Communication-efiinn efficiient FL. with discrepancy-aware compression
Input :
Number of communication rounds l'; Client set C: Global model w Learning rate f)
Local training epochs E; Communication budget B;
Output: Final global model W
for f = l....,,, do
Server samples clients C, S C and broadcasts global model W for each Client & C2 in parallel do
Local training based on W using private training data for epochs, obtain AW,-
Discrepancy-awee compression based on local calibration dataset, obtain AWR: k-
Transmit AW Server.
Server aggregates clients' updates W + W Σkec, AW k, my Ekec, nk.
The me where n& is the number of samples on client & in round t.
These motivating examples indicate that existing compression strategies remove compression units without regard their actual contribution the output, leading to disproportionate output discrep- ancy for given communication budget. To address this limitation, perspective that explicitly links the selection of compression units to the output discrepancy. This perspective unifies disparate compressors and highlights why selection rules based solely magni- tudes or randomness are fundamentally insufficient.
adopt calibration-based
3.3
KEY DESIGN
seek unified, compression-awant principle for communi- cation compression that directly scores each candidate compression unit by the output discrepancy it would induce small calibration dataset. Let W be layer update be compressed, and X denote the calibration activations. We recommend that each client randomly select small subset of its local training dataset in each round to compute calibration activations, which are then used for scoring compression units. This choice is empirically evaluated in Section [.4]. For each candidate compression unit u (whether an element calibration set as compression metric to estimate its removal cost:
As outlined in Algorithm and 2
singular triplet), we use the output discrepancy on
Lcomp (u) = Σ ||AY = --Y-Y'R
cal X
x
Then we rank these units by their compression metric (u)mp(u), and select the subset that approx- imately minimizes Leomp(u) under the given budget. This leads to a simple drop rule: keep units with large Lcomp(u) and drop those with small Ccomp Below we derive Lcomp (u) for different granularities that encompass common compressors. Notably, like most compression methods, our approach can operate under either global or layer communication budget.
In addition, Our method is fully compatible with classic error feedback scheme. Concretely, each client maintains residual vector that accumulates the compression error from previous rounds. At each communication round, each client first forms compensated update by adding the residual to its current local update. The compensated update then passed through the proposed compression op- erator, yielding the transmitted message. And the residual is updated by subtracting the transmitted compressed update from the compensated update.
3.4 COMPRESSION METRIC
For Element-wise Sparsification in Transformer Layers. places W by W Wiw.,, W Wi.eeee. Then
Dropping single element wa
AYw,, (W- W W.,, ) Wi. eie, X wiy eif where 1J is row of X. Hence the compression metric for element-wise compressors in Transformer layers is
Lcomp (w:)) = ||wis eiJ, 11 = w 11
5

---

Published as a conference paper at ICLR 2026
Algorithm 2: Discrepancy-awwre compression
Input Layer update W; Compression granularity g E {ELEMENT, SINGULAR TRIPLET): Communication budget B (compression ratio retained rank);
Output: compressed layer update W under communication budget B.
Define compression unit set 2 (whether elements or singular triplets).
for each u E li do
Randomly select small sample size of calibration data to obtain calibration activation X,
and compute compression metric (u) Eeal x |Y I'
Select a subset C 2 that satisfies the budget B and maximizes Eves Cromp(u)
Form W by retaining units in S and dropping the rest.
For Element-wise Sparsification in Convolution Layers. Consider an individual kernel with input channels Can. output channels Cout. stride s, and padding Dropping single kernel element wk.eli,5] with 0 S j.j F yields an output perturbation
[Y&[u, vj = wp, X,lus + i, us +]-
Hence the compression metric for element-wise compressors in Convolution layers is
11 W
Loomp (i
||aY|||E wi.li,,
j1
(Xalus + i, + i1)²
10
0=0
Here H'
2 1 and W' W+2p-P
+
For Low-rank Decomposition in Transformer Layers. Let the SVD be W = USVT
Eiu ouunn" Dropping single singular value ag causes
Y - y' = (W- - W = un X,
Hence the compression metric for low-rank compressors in Transformer layers is
Leomp (σt) = ||o, (1 x) of 1ll1 1
For Low-rank Decomposition in Convolution Layers. Fix (k: c) and take the SVD of its kernel Wk.c Ex=1 og "ayn with < and unit singular vectors ue, Ut E Rf Dropping component removes its rank-l contribution, and the resulting output perturbation with stride and padding size p is given explicitly by
F-15-1
AY
m, n) Tz Σ
Σ
( veli) Xa(ms + +1, ns as++) j).
To evaluate efficiently, we stack {v, YE= as horizontal 1 F filters and compute Z, H (X.) in a single pass; then apply the r vertical F filters (w,) in grouped fashion obtain V,
(s. (Z,) for all singular values. A naive implementation would perform separate F x F 2D correlations with cost O(r p2 H'W"), whereas this two-pass scheme costs O(r H'W'). Define
F-1
F-1
(H( (X.) = Σ ulil (m ns + +), (V(++) [n,
Σ 1=0
ulil (ms +i, n),
1=0
so that AY (t.o) compressors Convolution layers is
[m, n] of (v((a.) (H( (Xx)) [m,n]. n².. Then the compression metric for low-rank
Ccomp (comp(arkh (a = (y of ||( (H(*P) (Xx)) -

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1
SETUP
We evaluate on CIFAR-10, CIFAR-100 (Krizhessky et al. 2009), and Pashion-MNIST (Xiao et al. 2017) datasets under non-IlI client partitions, using three Transformers (ViT-tiny, ViT-small, and ViT-base (Dosovitskiy et al. (02I)) and two CNNs (AlexNet (Krizhevsky et al. 2012) and ResNet- 18 (He al. 2016)). We simulate the non-IlD scenario by considering heterogeneous partition, where the number of data points and class proportions are unbalanced. Specifically, we simulate a partition into N clients following a Dirichlet allocation with a = 0.2. The level of heterogeneity among local datasets across different clients decreases as a increases. We set the total number of clients to be N = 100, and randomly sample 10 clients per round for training and aggregation. The training process consists of 200 communication rounds. Each participating client performs E = 2 local training epochs per communication round with batch size bs = 16. Training employs a cosine annealing learning rate schedule (Loshchilov & Hutter (017) and linear warm-up. For ViTs, use the Adam W optimizer (Loshchilov & Hutter 2019) with an initial learning rate of 10 decayed by a cosine annealing factor and weight decay of 0.05, while for ResNet- 18 use the SGD optimizer with an initial learning rate of 0.01 decayed by a cosine annealing factor and a weight decay of 10-1. Unless stated otherwise, each client uses 64 randomly sampled local examples per round for calibration. All clients initialize from the same global model and employ identical within-round learning-rate schedules. We evaluate two basclines: the element-wiss sparsification method Top-k and the low-rank decomposition method ATOMO. For each, we compare the original magnitude- based method with our discrepancy-awes For element-wise sparsification, we use the global budget configuration, as it typically performs better and is more widely adopted than the layer budget. For low-rank decomposition, we follow ATOMO's original setup with the layer budget configuration (Wang et al. 2018). We conduct each experiment with three independent trials and report the average results. all experiments, we employ the classic error feedback scheme to mitigate the bias introduced by lossy compression.
re-ranking variant under the same communication budget.
4.2 MAIN RESULTS
Element-wise Sparsification. To evaluate the effectiveness of our discrepancy-aware compres- sion in enhancing existing element-level sparsification methods, we compare the final test accuracy of the original magnitude-based Top-k method and its discrepancy-awan different compression ratios, datasets, and models. As shown in Table [ the discrepancy-awen compression leads consistent improvements over the baseline. Notably, the advantage becomes more pronounced as the compression ratio decreases. This is because under stronger compression, the overlap between compression units selected by different methods reduces, as detailed in Sec- tion Thus, the ability to identify important information becomes more critical to performance.
augmented variant across
Table 1: Final test accuracy of the element-wise sparsification method Top-k and the corresponding diserepancy-awan augmented method across different compression ratios, datasets, and models.
Dataset (Model)
Compression Ratio
Method
0.01
0.1
0.2
0.4
0.6
1.0
CIFAR-I0
-aase
21.03
34,93+
37.69=az
38.25
38.57=
51,561 03
(ViT-tiny)
Discrepancy-awer
29.62 0.1
41.52
43.11
42.81 0.1
41.21.1
CIFAR-100
Magnitude-baseed
9.29 0.4
26.91 0.3
28.51= 05
29,974 04
30.23 0.
34.13 02
(ViT-small)
Discrepancy-awar
13.29 02
28.62 0.3
31.27=01 0.1
32.87 0.4
33,32:
CIFAR-100
Magnitude-base
27.28 10.76=
30.71= 1.0
32.34 n9
32.82= 066
35.33 01
(ResNet-18)
Discrepancy-award
15.58 03
29,71 0.4
32.54 0.
33.65 0.7
33.71 0.4
Fashion-MNIST
Magnitude-bascd
63.31 a1
70.32+ 09
71.50=a5 as
71.59 u.7
71.98= 07
78.96
(AlexNet)
Discrepancy-ee aware
67.55 0.3
1.7
73.61 0.4
73.70 0.4
74.01 0.5
Low-rank Decomposition. Similarly, evaluate discrepancy- compression applied to low-rank decomposition methods by comparing the original ATOMO method and its discrepancy- aware augmented variant across various retained ranks, datasets, and models. As shown in Table
7

---

Published as a conference paper at ICLR 2026
Table 2: Final test accuracy of the low-rank decomposition method ATOMO and the corresponding discrepancy-awer augmented method across different retained ranks, datasets, and models.
Dataset
Retained Rank
Method
(Model)
CIFAR-I0
Magnituee-aseed
26.46#
28.33= 0.3
36.14 0.4
42,44 0.3
(ViT-tiny)
Discrepancy-ann
28.03:
30.74
36.40
42.68
CIFAR-10
Magnitude-basee
33.41 0.4
37.01 0.4
41.32 0.2
44.01 0.4
(ViT-small)
Discrepancy-awant
34.61 0.4
40.10=03
43.17: 0.3
45.29 0.3
CIFAR-I00
Magnitude based
17.08
19.07= 0.5
24.62::
29.01: 0.6
(ViT-base)
Discrepancy-aware
20.17
23.99 0.5
26.13 0.4
30.62: 0.4
the discrepancy-avve augmented approach again leads to consistent and significant improvements. As with glement-wiss sparsification, the benefit increases under higher compression, due to reduced overlap and increased differentiation among selected candidate compression units making accurate identification of important features more critical, as further analyzed Section
Overall Communication Rounds to Achieve Target Accuracy. To demonstrate that the discrepancy-awary method converges in fewer communication rounds, we further compare the number of communication rounds required reach target accuracy between magnitude-based and discrepancy-avee aware compression methods. Specifically, we conduct experiments on two representa- tive configurations various compression ratios. The target accuracy defined as 60% or 80% of method for each specific configuration. The results are summarized in Tables 3 and 4 Across all configurations and compression ratios, the discrepancy-awarc method consistently reaches the same target accuracy in fewer rounds than the magnitude-basee compression, with speedup of up to 1.56x in reaching the target accuracy at compression tio 0.01 on the CIFAR-10 dataset using the ViT-tiny model. This analysis demonstrates that discrepancy-avve compression not only enhances final accuracy, but also accelerates convergence in terms of wall-clock time under the same communication budget.
the final test accuracy achieved by the discrepancy-awent
baseline. The performance gains are particularly pronounced under strong
Table 3: Communication rounds required to reach target accuracy on FMNIST (AlexNet) of the element-wise sparsification method Top-k and the corresponding discrepancy- awware variant. Values in parentheses indicate the speedup of discrepancy-aware methods over magnitude-based baselines.
Compression Ratio
Targel Accuracy
Method
0.01
0.05
0.1
0.2
0.4
0.6
60% of the final test accuracy
Magnitude-based
73
54
55
54
56
56
Disrrepancy-waare
62(1.18x)
46(1.17x)
50(1.10x)
54(1.00x)
56(1.00x)
53(.05x)
80% of the final test uccuracy
Magnitude-based
105
85
83
84
81
Discrepancy-wware
82(1.28x)
72(1.18x)
71(111x))
83(100xx)
81(1.03))
82(0.98x)
Table 4: Communication rounds required to reach target accuracy CIFAR-10 (ViT-tiny) of the element-wise sparsífication method Top-k and the corresponding discrepancy-awwr variant. Values in parentheses indicate the speedup of discrepancy- methods magnitude-basee baselines.
Compression Ratio
Target Accuracy
Method
0.01
0.05
0.1
0.2
0.4
0.6
60% the final accuracy
Magnitude-
45
Discrepancy-wware
44(1.20x)
31(L.45x)
26(1.77x)
34(1.06x)
37(105x)
39(1033x)
80% of the final
Magaitude-based
128
117
120
99
116
89
test accuracy
Diserepancy-aware
82(1.56x)
81(1.44x)
83(145x)
79(1.25x)
x9(1.47))
86(1.03x)
4.3 OVERLAP ANALYSIS OF SELECTED TRANSMITTED CONTENT
To demonstrate the differences between discrepancy-aware and magnitude- based selections across varying compression levels and data heterogeneity, report the overlap rate of the selected com- pression units between the Top-k k method and its discrepancy-awart augmented variant. This is shown in the 2D heatmaps of Figure] i with the compression ratio (x-axis) and Dirichlet parameter
8

---

Published as a conference paper at ICLR 2026
(y-axis). Across datasets and models, the overlap increases with both the compression ratio and a. When compression is aggressive and data are highly non-lID, the overlap is minimal, whereas milder compression and more IID data yield substantially higher overlap. These trends reinforce our central conclusion: discrepancy-awar bandwidth with low compression ratios and strong heterogeneity with small When resources are constrained, discrepancy-avee Additionally, the uneven distribution of local data characteristies actoss clients increases the impact of these characteristics on the importance of compression units. However, as compression becomes milder and data approach IID, the two criteria partially align and the overlap increases. This suggests that while arge-magnitude units are no longer the most critical, they still regain a relative advantage. We also observe higher overlaps on Fashion-MNIST (AlexNet) compared to CIFAR-IO (ViT-tiny), indicating that simpler datasets and architectures lead to more agreement between the two strategies.
compression is most beneficial under conditions of tight
compression selectively transmits the most important content.
9.484
0.519
0.233
8.
8.--
0.191
0.264
0.345
8.411
8.473
0.587
0.549
0.872
0.4
0.230
0.322
6.380
0.518
8.722
0
0.844
0.596
Alpha (8)
0.4
9.17%
6,383
8.441
IU412
0.511
0.214
0.399
0.490
8.541
11668
6.794
0.1EM
0.220
0
0.342
0.394
0.422
0.46
0.55M
81
0.177
0.254
8.309
0.433
0.475
0.583
0.705
8.746
Compression Ralie
Ratio PMNIST(AAesne))
CIFAR (ly
Figure I: Overlap rate heatmaps for different Non-IID degree (alpha) and compression ratios on (a) CIFAR-10 (ViT-tiny) and (b) FMNIST (AlexNet).
4.4 ANALYSIS AND ABLATION
Selection Strategies and Sample Size of Calibration Data. We investigate the influence of cal- ibration data selection strategies and the sample size of calibration data on the performance of discrepancy-awar compression. As shown in Table[5] randomly sampling calibration data at each round consistently outperforms strategies that always choose either the earliest or the latest samples, without incurring additional cost. Therefore, we adopt random sampling the default strategy in experiments. Moreover, Table demonstrates that number of calibration samples has only marginal effeet performance, which remains stable across different quantities. These findings highlight the robustness of our method in different calibration sample sizes.
Table 5: Impact of calibration data selection strategies and sample size.
Selection Strategies First
Calibration Samples
Dataset (Model)
Random
Last
32
64
128
256
512
Fashion-MNISTT (AlexNet)
73.95
72.49
73.14
72.89
72.41
72.86
72.76
72.91
CIFAR-10 (ViT-tiny)
43.23
43.14
42.91
42,97
43.11
43.52
43.22
42.96
Additional computational cost. We evaluated the additional computational cost introduced by aware compression method by measuring the average time spent at different stages. As shown in Table 6 while the compression time with the discrepancy-awan up to 18.2% 2%, the overall computation time per client per round showed only a slight increase. Given the low communication frequency and communication bottlenecks in FL, an acceptable trade- off. Our method significantly enhances communication resouree utilization under tight communi- cation budgets and thereby improves performance. This makes viable strategy for enhancing communication efficiency in FL under constrained communication resources.
our discrepancy- aware
method increased by

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of per-round time consumption per client (calibration sample size 64).
Dataset (Model)
Avg Time Consumption (Top-k)
Avg Time Consumption (ATOMO)
Method
Training
Compression
Total
Training
Compression
Total
CIFAR-I0
Magnitude-based
0.5681s
0.0013s
0,5694s
0.7033s
0.1006s
0.8039s
(ViT-tiny)
Discrepancy-aware
0.5662s
0.0568s
0.6380s
0.7055s
0.2232s
0.9287s
CIFAR-100
Magnitude-bane bassd
0.5789s
0.0014s
0.5803s
0.7480s
0.3258s
1.0738s
(ViT-small)
Diserepancy-awee -ware
0.5799s
0.0586s
0.6385s
0.7423s
0,5217s
1.2694s
Non-IID Degree. We simulate a heterogeneous data partition into N clients using the Dirichlet distribution with a. Here, we conduct an ablation study investigate the impacts of Non-IID degree, while keeping all other settings identical to Section 4.1 As shown in Figure 2 across all heterogeneity settings, discrepancy-awes
compression consistently improves the final test accuracy over magnitude-based baselines. Notably, the gains are most pronounced under highly non-III) splits with smaller a, suggesting that selecting candidate compression units according to client- specific calibration data, rather than pure magnitude, can better capture what matters for each client. candidate compression units exert little influence on the local ontut-output mapping and are therefore deprioritized by discrepancy-awart compression. As the non-IID degree decreases as & increases, the final test accuracy generally improves as expected.
Some
FUB-IT Diprepancy
CIFA + Doorsparyy ovirt CIAR-1D 4 Meggr Ruderbed
FUNSTT+ Bacaanno
CIFAR1D Decrpersyyиаи PUNEET Vege beadl
T18.1T Moppitstetoee
CPWN-11 Naptaad-bnse
0.5
0.7
Final Accuracy
2.7
0.8
34-
04
34
Alpha
Epochs
Figure 2: Impact of non-IlD degree.
Figure 3: Impact of local training epochs.
Local Training Epochs. We conduct ablation study investigate the impact of local training epochs, while keeping all other settings identical to Section As shown in Figure 3 increasing the number of local training epochs consistently improves the final test accuracy, reflecting the benefit of more thorough local updates. Across all settings, our discrepancy- higher accuracy than magnitude-based baselines. These results highlight that selecting compression units based on calibration data consistently provides robust gains.
compression achieves
5 CONCLUSION
In this work, we introduce discrepancy-aware communication compression strategy for FL that addresses the challenge under tight communication budgets by minimizing the output discrepancy through calibration data. This approach can be seamlessly integrated with existing FL compression techniques, offering enhanced performance under limited communication resources. Moreover, is compatible with wide range of other FL methods, including those focused on security, privacy, and heterogeneity. Empirical results across various datasets and models demonstrate that the proposed method significantly outperforms baseline compression strategies. Our work provides fresh per- spective on communication compression process in FL, and we believe this work can significantly contribute the development of communication-efffcient FL..
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This work was supported by the National Key Research and Development Program of China under Grant 2023YFB4301900 and Beijing Nova Program 2024102
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD:
coomunication-efffcinn SGD via gradient quantization and encoding. In Isabelle Guyon, UI- rike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gameet (eds.), Advances in Neural Information Processing Systems 30: Aunual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1709-1720, 2017. URL http:tttts::pppeeedeeeee ngs neurips ccc cccppaper/2017 hash/
httmlsttaatthttmlll httll
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. pp. 5977-5987, 2018. URL https: https::/proceeddnggg mew.eeiiiiinnn....s........... html
cc/ppper/2018/nast/
011450663369e0ee 7aca7
Jeremy Bernstein, Yu-Xiang Wang. Kamyar Azizzadenesheli, and Animashree Anandkumar. SIGNSGD: compressed optimisation for non-convex problems. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 559-568. P0..5R.....RR PMLR, 2018. URL RtLLhtttpp//prcceedine .mlr. >ress//800bernstees 8a. html
Alcksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased com- pression for distributed learning. Mach. Learn. Res., 14:27:11-276:55 2023. URL http:
7/
/20 html
Mei Cao, Tingting Zhang. Bowen Zhang. Jianbo Lu, Zhaoyan Shen, and Mengying Zhao. Fedmq: Multi-grained quantization for heterogeneous federated learning. In Zhipeng Cai, Daniel Tak- abi, Shaoyong Guo, and Yifei Zou (eds.), Wireless Artificial Intelligent Computing Systems and Applications 18th International Conference, WASA 2024, Qindao, China, June 21-23, 2024, Proceedings, Part II, volume 14998 of Lecture Notes in Computer Science, pp. 1-14. Springer, 2024. doi: 10. 0011007797883300311146--1111 URL https: / htt///ic.oorgg// 1007/ 978-3-031. 7111-124672_11111
Mei Cao, Huiyu Wang. Yuan Yuan, Jianbo Lu, Xiaojun Cai, Dongxiao Yu, and Mengying Zhao. Fedmq+: Towards efficient heterogeneous federated learning with multi-grained quantization. J. Syst. Archit 107:10 2025. doi: 10.
103460. (tRts:/ts:
brg/100 1 101 6/j sysare........ 2025. 103460
Huancheng Chen and Haris Vikalo. Mixed eeeed-precisionee quantization for federated learning resource- constrained heterogeneous devices. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 6138-6148. IEEE, 2024. doi: 10. 11w.////PP2733..2220000025 2024. 00587
Laurent Condat, Kai Yi, and Peter Richt'arik. and variance reduction mechanisms for biased and unbiased compression in distributed opti- mization. ArXiv, abs/2205.041 2022. URL :/atps://ap..... https///ap.eemaattseeeee.eee sewanticschoollaa. /org
Ef-bv: A unified theory of error feedback
Corpus ID: 2D:24857191
Yatin Dandi, Luis Barba, and Martin Jaggi. Implicit gradient alignment in distributed and federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6454- 6462, 2022.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbor,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. helLhttps://ppmeeee idd dNTTy
net/ forum?
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: Utthttps:: http:///ooooogggg//// http:///........................
10.1109/CVPR.2016
www.:/11........ 2016.
Rui Hu, Yanmin Gong, and Yuanxiong Guo. Federated learning with sparsification -ampliiie pri- vacy and adaptive optimization. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth Interna-
fional Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada,
19-27 August 2021, pp. 1463-1446400 1469. ijcai.org, 2021a. doi: 10.24966/IICAA122222222 URL.
https: //doi org. / /ttp:/1.24444 20w.00229633jjaa.... 2021/2222
Sixu Hu, Linshan Jiang, and Bingsheng He. Practical hybrid gradient compression for federated learning systems. In Proceedings of the Thirty-Third International Join! Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 4147-4155. ijcai.org, 2024. wtt.:ooggpppccceooo ngs / //////224445555
htl.https://wvii
Yifan Hu, Xin Chen, and Niao He. On the bias-variance-cos cost tradeoff of stochastic optimization. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 22119-22131, 2021b. URL https: htttpss//proceediggss.oo neurips......pp....... cC/ ppccppppp//221///// wwt.::22-bbbttt.....mm html
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich, and Martin Jaggi. Error feed- back fixes signsgd and other gradient compression schemes. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Ma- chine Learning Research, pp. -252-3261 PMLR, 2019. URL http:::/pprcceecaggs..... press/vs' ht////9///kkk www.//aaiiiiii wtdd////r//aimmredyy............ 9a. html
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 2020. URL
ngs ml wtt.a//.....ers//11111///////// wwtpppes////ll//aammimmimmmmmaa reddy
html
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Haowei Li, Weiying Xie, Hangyu Ye, Jitao Ma. Shuran Ma, and Yunsong Li. Fedfq: Federated learning with fine-graing grained quantization. CoRR, abs/2408.089777, 2024. doi: 10.48550/ARXIV. 2408.08977. https:
wwt.ooggg...... 48550//rxxv.24008833 08977
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dim- itris S. Papailiopoulos, and Vivienne Sze (eds.), Proceedings of the Third Conference on Machine Learning and Systems, MLSys 2020. Austin, TX, USA, March 2-4 2020. ml- sys.org, 2020. URL httpt:://// httpss://prooeeeennssss www.s.
m arq/ wtwp://www....ol...ooop//pap.r files/paper. Abst htm.
12

---

Published as a conference paper at ICLR 2026
Xiaoyun Li and Ping Li. Analysis of enror feedback in federated non-convex optimization with biased compression: Fast convergence and partial participation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 19638 19688. PMLR, 23-29 Jul 2023. URL
mlr www.mmt....epsee.mmmm press /202////222//2 1tt.:///123.....i. html
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini batches, use local SGD. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. htt.:ttts:// htap:s:///ppeee //tp:///eee/eeeeeeeeeeeee
met/forun saaimnidddBlyyle Bley01BFFr
Yujun Lin, Song Han, Huizi Mao, Yu Wang. and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URLhttps: httphttts///opeevesstttee /ttp///peeee/e/eeeeeeeeeeeeeeeee net/ / forum?id-
SKhOHMWOW
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICI. 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. htt.:ttts://ppeereeeet net/ (roorummid=Sk@89Sce
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
net /wt//ferruooreeeo umr wttpammmiiiiieeeeeeeemee d=BkggRiCqY)
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--fficieee learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 2017. URL http::/proceeddnng
wal..press... v54 www.mmmmmmancccmm.cmmmmmmm.mm..a 17 hhmll
John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In Gus- tau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, vol- ume 151 of Proceedings of Machine Learning Research, pp. 3581-3607, PMLR, 2022. URL https::/proceeeie wtt.es.m...pp www.mll..rrss//ee https///w.1.11ggge22mmm... html
Xun Qian, Peter Richtárik, and Tong Zhang. Error compensated distributed SGD can be accelerated. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 30401-30415 2021. URL
neur www.wew.......ee....e........... ps. cttpppp.rrpp -bw.aaaddaaaccaaaaaaaaaaae html
Linping Qu, Shenghui Song, and Chi- Ying Tsui. Fedaq: coammuiication-efficieeeee federated edge learning via joint uplink and downlink adaptive quantization. CoRR, abs/2406 18156, 2024. doi: 10.48550/ARiV.24466 18156. URL /tttt::t////et //op::://dd.... 4at55//0.4555aaa/ 2tt.5500/aaviv.v. w.w..200w........... 2406.
18156
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A ciammuniattion-fffiien federated learning method with periodic averaging and quan- tization. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Pulermo, Sicily Italy), volume 108 of Proceedings of Machine Learning Research, pp. 2021 12031, 1.0........ 2020. URL httt::://roceeeaii
mir. wtt..pppeee/////reeeeeesaaaaaaaa
html
Cédric Renggli, Salch Ashkboos, Mehdi Aghagolzadeh, Dan Alistarh, and Torsten Hoefler. Spareml: hirf-performance sparse communication for machine learning. In Michela Taufer, Pavan Balaji,
13

---

Published as a conference paper at ICLR 2026
and Antonio J. Peña (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, Colorado, USA, November 17- 19, 2019, pp. 11:1-11:15, ACM, 2019. doi: 10.1145/29950003350222 URL httpp:///oo. org/ 1tg/10.10 10. /4/3325550 03155500.3336620 3356222
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Robust and communication-efficiin federated learning from non-iid data. CoRR, abs/1903.02891, 2019.
URL http: org/ abs/1903..... 02891
Sebastian U. Stich. Local SGD converges fast and communicates little. In 7th International Con- ference Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019, Open- Review.net, 2019. htttttttss://oeeneeeee ew. net net/forum? forum? d=$1 d251g23nRcFX
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gra- dient compression for distributed optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' d'Alché-Bue Buch Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14236-14245 2019. URL https: htttps::/proocediggg wwwpeeeeiinnnn.euuiigee www.ips....... wwwperripssccppppee//22ea cc/paper/2019/ -bteeee6cccccc---taaaateee html
Bin Wang, Jun Fang, Hongbin Li, and Yonina Eldar. Communication efficient confederated learn- ing: An event-triggered SAGA approach. IEEE Trans. Signal Process. 72:2054-2072, 2024, doi: 10.1109/TSPP2004333387511 URL
1101 2024. 3368775.333711
Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris S. Papailiopoolos,, and Stephen Wright. ATOMO: communication-effiiiaii learning via atomic sparsification. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9872 9883, 2018. URL https: http:///cocceeeeeeseeee neurips.co. html
Jiany Wang and Gauri Joshi. Adaptive communication strategies achieve the best error-runtime trade-off in local-update SGD. In Ameet Talwalkar, Virginia Smith, and Matei Zaharia (eds.). Proceedings of the Second Conference on Machine Learn- ing and Systems, SysML 2019, Stanford, CA, USA, March 31 April 2. 2019. ml- sys.org, 2019. URL
ml sys http:///gr..msssssoorara..r.rrra www.f/fapeerrrre htm
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 1708.0777 2017.
Kai Yi, Georg Meinhardt, Laurent Condat, and Peter Richtárik. Fedcomloc: Communication efficient distributed training of sparse and quantized models. Trans. Mach. Learn. Res., 2025, 2025. URL https::::: : / wwp./ppeeevveeeeeeeeeeeeee net. www.fooddmwmoo......eoo.
Ying Zhuansun, Dandan Li, Xiaohong Huang, and Caijun Sun. Communication-effiiieenn federated learning with adaptive compression under dynamic bandwidth. CoRR, abs/2405.032488, 2024. doi: 1044550/ARXXI..445..222 URL petps://do/: org/10, 48550/arx..550/
03248
A APPENDIX
A.1 1 LLM USAGE DECLARATION
Large language models (e.g., ChatGPT) were used solely for language editing and formatting. They did not contribute to the conception, design. implementation, analysis, data generation labeling, or evaluation of the methods and results. All technical content and claims were authored and verified by the authors, and personal, proprietary, sensitive data were shared with L.LM services.
14

---

Published as a conference paper at ICLR 2026
A.2 COMPARISON OF CONTENT TO COMPRESSED IN FEDERATED LEARNING AND DISTRIBUTED LEARNING
Let the global objective be
F(W) Σ Pk BE [((W:))]
and g(W = VF(W) represents the gradient, and H(W) = FWW is the Hessian.
In data-center distributed learning (DL), every iteration synchronizes across workers, transmitting gradients, and the compression step involves reducing the gradient g(W) using compressor C(-), such Top-k. For given parameter update W, the compressed update AW C(AW) leads to compression loss:
F(W - AW) - F(W + AW) nng' (g - 0(??)
where C(gg) zeroes out some components of the gradient. In this case, the compression loss is directly related to the gradient magnitude, specifically to oz. because the object being compressed and transmitted is the gradient itself.
However, in FL, the situation is different. Instead of transmitting single gradient, each client performs multiple local SGD steps, accumulating updates over several epochs. The update for client k is the difference:
AW₂ WE.: WR
gh (W
7=0
which is series of gradients evaluated at different local iterates. Expanding around the global model W. we approximate:
9&. (War) = gw( + H& (W), - W.)
Since We,7 W a summation of previous gradients, the update becomes:
1(1 H&
AWA (w) + 2
where the first term is the gradient-based update and the second term introduces curvature effects due to the Hessian. This shows that the client updates are influenced by both the gradient and the model curvature H2 (W), making the relationship between the update and the gradient more complex than in DL.
Now consider the server-side aggregation of updates from multiple clients:
AW Σ AW
The compressed update is AW C(AW), and the compression error is given by the residual R AW ÁW. The resulting compression loss is:
F(W AW) F(W + AW) TW O([AW]]I [[]]2).
Crucially, this residual R is a difference in parameter updates, not gradients. The importance of coordinate j is determined by the product 9j Ry, not just by the magnitude [Ry alone. This reflects fundamental difference from DL, where magnitude-based compression decisions align with gradient importance.
This analysis confirms the core observation: in FL, the content being compressed (parameter up- dates) carries little gradient information, which weakens the relationship between magnitude and importance. This motivates discrepancy- compression strategy, which ranks compression units based on their effect on layer output using the Loomp rather than relying on magnitude or randomness.
15

---

Published as a conference paper at ICLR 2026
A.3 COMPATIBILITY WITH POWERSGD
We have already shown in the main text that our discrepancy-aware selection principle can be in- stantiated top of ATOMO, where the layer update is factorized via exact SVD. Importantly, the same principle is not restricted to this setting. Our discrepancy-aware selection mechanism only requires that a layer update matrix can be written as sum of rank-1 components and does not rely on these components being exact singular vectors. Concretely, suppose a layer update matrix W E Re da admits factorization the form
W
abb
where ar Rda and Rdd define rank-1 components azb; Let X Rd denote small local calibration set of layer inputs (activations). The corresponding layer outputs are
Y (Wo W)X,
where Wo is the current model parameter of the layer. If we remove a single component a,6, the modified update becomes W arb; and the new outputs are
Y' (Wo + a.b,) = a,b, X.
(3)
Hence, the discrepancy incurred on the calibration set by dropping component is
A = - Y' = a,b, X = ab x).
(4)
Our per-component discrepancy metric is precisely the Frobenius norm this difference:
Cemmp (L)
|| = ||ap T xI| = a|a (b, {
(5)
In the ATOMO instantiation, W is factorized using the SVD W E u, In this special case, (a,, by) = (a,4e, 1t), and Leom (t) reduces to a function of the singular value and the right singular vector evaluated on the calibration inputs.
Specialization to PowerSGD. PowerSGD produces low-rank approximation of the form
W RJ PQT,
(6)
where P E Robax and R are obtained via iterative power iterations. Let pe and qe denote the f-th columns of and Q, respectively. Then
PQ
Σ pig T
t=l
so (ar, (pt, qi) in Equation 5 The discrepancy metric for component under PowerSGD becomes
(e (4) = | X|| = Il:(( ||
1= 1=
(8)
Thus, our method applies directly to the rank-1 I components produced by PowerSGD; does not depend on how P and Q are computed, only on the resulting factorization W = PQT.
Discrepancy-avee aware selection on top of PowerSGD. Given a standard PowerSGD implementa- tion for layer, our discrepancy-aware selection be implemented in each communication round as follows.
1. PowerSGD factorization with an oversized candidate rank. Run PowerSGD to obtain rank-reand approximation W PQ for each compressed layer, where Teand pre- specified candidate rank that is intentionally larger than the effective rank allowed by the communication budget, i.e., Tbudget Yrand-
2. Compute discrepancy scores compute 93 X for each column de of Q, and evaluate
calibration set. On small local calibration set X,
T Tecomp (1) ||p( x>Ip t= 1,..., Trand- Teand-
(9)
16

---

Published as a conference paper at ICLR 2026
3. Select components under the communication budget. Rank the r'ann components {peq.' } by (t) in descending order, and select the top Ybudget components that fit within the desired communication budget.
4. Form truncated factors and communicate them. Form truncated factors P IR X Khudge and Q' E X by keeping only the selected columns of P and Q. and communicate (P Q') instead of the full (P. Q).
This procedure does not modify the internal power-iteration scheme or the encoding format of Pow- erSGD; it only changes which rank-1 components are ultimately transmitted, based their data- driven impact on the layer output as measured by Leamp
A.4 FUTURE WORK: EXTENSION TO QUANTIZATION BASED METHODS
Our discrepancy- aware selection principle is designed to be compressor-annostic it only requires the ability simulate the effect of given compression operation on the layer outputs over a small calibration set. In the main text, this principle is instantiated for element- wise sparsification (e-g., тор- and low-rank decomposition (c.g., ATOMO) by treating either an individual element or a singular triplet the compression unit and scoring it via the induced output discrepancy.
The same idea naturally extends to quantization- U. For each unit U, we allow finite set of candidate bit-widths B. For given pair (u, b) with b E B, we construct a compression configuration ((u, b) where unit u is quantized with bits, and all other units u' bit-width). We then measure the corresponding discrepancy
methods. Consider a set quantization units
kept fixed reference precision (e.g., full precision baseline
L(u, b) = (((u,, b)) = (Wo + W)X - (Wo + Ca(u.b) (W))
(10)
Given a fixed total bit budget for each layer, bits can be allocated by prioritizing units for which increasing the bit-width yields the largest reduction in discrepancy per additional bit. A simple strategy is to start from a baseline bit-width besse for all units and then iteratively consider raising the bit-width of some units to higher candidate values. For each unit u and a feasible higher bit-width b' > beur((u) (u), we estimate the decrease in the discrepancy measure L(u, b) when changing from beur(u) to b', and normalize this decrease by the corresponding increase in the number of bits. At each step. we choose the unit and bit-width change that provides the largest discrepancy reduction per additional bit, update (u) accordingly, and repeat this greedy procedure until the total bit budget for the layer is fully used. This yields discrepancy-aa units that are more critical for the layer outputs (high digh-diicceppacyy units) are assigned more bits, while less critical ones can be more aggressively quantized.
quantization scheme in which
This construction complementary to existing federated learning quantization frameworks: the discrepancy-avee aware principle can be used as drop-in mechanism for deciding bit allocation within such frameworks, without modifying their overall communication or system design. We expect that specific instantiations of this idea, tailored particular quantization schemes and hardware constraints, will be promising directions for future work in communication--ffiiinn quantization.
A.5 CONVERGENCE OF DISCREPANCY- COMPRESSOR WITH BEEEORRFE
In this section, following previous works on compressor convergence in distributed learn- ing (Beznosikov et al. 2023 Karimireddy al. 2019 Condat et al. 2022 Hu et al., 202Tb Qian
et al. [202]] & Li 2023), we adopted similar settings and approaches to discuss the convergence of our discrepancy-avee aware compressor.
We consider minimizing possibly non-convex objective
min f(x) IE ()(; #e
(II)
A.5.1 ERROR-FEEDBACK SCHEME
We study the following generic error-feedhad scheme, detailed in Algorithm 3
17

---

Published as a conference paper at ICLR 2026
Algorithm 3: feroorfeedbbck SGD with Compressor
Input:
stepsize 7 > 0, compressor C')), initial weight matrix w.;
Set en 0. for = 0, i.... I' do
Draw stochastic gradient Ir at wt--
Form corrected direction g = 79t + Ct.
Compress: C(g()-
Update iterate: We++ wt C(g.).
Update error: c++1 = l/ C(s').
A.5.2 ASSUMPTIONS
Assumption (Smoothness).
The function f is L-smooth, i.e. for all x, E RA.
fly) < f(x) + (Vf()), 2) + 2012
(12)
Assumption 2 (Unbiased stochastic gradients with bounded second moment). At iteration the algorithm queries a stochastic gradient 9t such that
ge
wi] = V (wi),
E 11 we S so²;
(13)
for some a > O.
Assumption 3 (8-approximate compressor). A (possibly randomized) operator C Rd Rd is called a d-approximate compressor with E (0, 1] for all e E R2
E[I[C(u) u|| = S (1 8) ||ul
(14)
The expectation is taken over the internal randomness of C if any.
A.5.3 CONVERGENCE ANALYSIS
Lemma (discrepancy-awve) -wwre top-k compressor is a ò-approximate compressor). Let R& denote the fixed calibration activations obtained from the calibration dataset, and let my ma(w, a.) be a scalar discrepancy score for coordinate which depends on both the weight entry w, and the calibration activation ag. We assume that ma(:, az) is continuous and strictly increasing in wel for every fixed a, The discrepancy-awe awaare aware top-k compressor Cdis-k is defined by
Wis iE kutsk-(ww),
Sais = indices of the k largest mz oz), (Cais-k =
0,
otherwise.
(15)
Denote by Ssop-((w) the index set of the k largest entries of w magnitude, and define
((w)= = LieSan-nalw) w
w + O.
(16)
EieSrop w?
Assume that the feasible set Q we have uniform lower bound
δ(x)
0,
Vx E Q {0),
(17)
i.e., the subset selected by the discrepancy-aware rule always retains at least a fixed fraction c of the squared l f2-mass that would be retained by plain top-k.
This assumption is mild for our discrepancy-awarc compressor. During training, both the model parameters and the calibration activations are constrained to stay in a bounded region, so it is natural to restrict attention formly bounded. Moreover, for our Leomp the discrepancy score m2 (W., a,) depends not only on the calibration activation 04, In particular, |w:|| enters score. Hence the discrepancy-awan selection rule is hard to systematically assign large scores to coordinates with vanishingly small [wm] while ignoring coordinates with moderate or large (wy).
compact feasible set Q Rd on which all coordinates are uni-
important multiplicative factor in the
Then Cais is a 6-approximate compressor on Q with
k
c
(18)
d'
18

---

Published as a conference paper at ICLR 2026
Proof. Fix any Q. Since Cdis Cdws- is obtained by zeroing out a subset of coordinates of w, have
IC)ts-
Σ
w
Σ
wi +
Σ
w/:
(19)
iE k(c)
SE k(t))
iq
((0)
and therefore
(Ciis-
-w12
" = 1110112 - ||Cais-
(20)
i
On the other hand, the standard top-k compressor satisfies (see Appendix B.2 in Karimireddy al. 2019
k d
|
(21)
#EStop. (10)
Lee, the top-k operator retains least & this implies
à fraction of the total energy. Together with Equation 17
u? = ó(w)
Σ
W
Σ
Wf 2
100 12.
(22)
SE (w))
SESrop- (w)
iESrop-e(w)
Substituting Equation 22 into Equation 20
we obtain
[[Catsk(w)) w112 |
Σ
wf 1 (1-c
(23)
ie k(vn
Therefore, if we define
8 = c d'
(24)
then
(w) 31 - (1 8) 1w
E
(25)
Note that our compressor is deterministic, the expectation in Assumption 3 over the internal
randomness of C is not needed in this case. Hence Cais- 6-approximate compressor on Q which completes the proof.
Lemma 2 (Bounded enror). Based on Assumptions and Lemma
7 let {ea)e20 be generated by
Algorithm 3 Then for any 2
4(1
[2
200
(26)
82
Proof. By construction et+1 and Assumption
3 we have
| = |l C( < (1 5) || = (1- 117 + e212
(27)
Taking expectation and expanding the square, for any parameter > 0,
B[[|e+1|| < (1 8) E [l + e
(28)
+ (1 6) 6) 1 r E 11a [2]
< (1 () + 7) E |el
(29)
7
S (1 (1 + 7) E || ||
+ (1 -0)()-11)) + 7200
1
(30)
where we used the inequality || + b11 < (1 + || + (1 + 1/12
Unrolling this recursion and using E| 1 0 gives
B[llet]l - 61 [1+ + 1 1)
2220 02
[1 61 + n))*
(31)
1=0
19

---

Published as a conference paper at ICLR 2026
For any 17 E (0,6/(2(1 -))) we have (1 - (1 + 7) < 1, and thus
1
Σ
(1 -1 5)+++) - (1 5) + n)
(32)
8 3-n(11 8)
1=0
Choosing 2[1-6] yields
8
1
2(1 6)
8
1
8 - 8-n(1 8) = 2
8
and therefore
4(1 8)
E [lle
P]]
7202
(33)
8²
Since the bound does not depend on t, holds for all t 2 0.
To relate Algorithm to standard analysis in Karimireddy et al. (2019), it is convenient to introduce the virtual iterate
00 Wt Ct-
(34)
Using the update rules for wi and e, one checks that
11+1 = W++1 E Wt C(s)) - (gf C(g.)) = we - -i = w. 79t
(35)
Lee We follows the trajectory of vanilla SGD with stepsize y broad sense.
Theorem 1 (Non-convex convergence of error-feedback SGD). Let Assumptions 7 2 and 3 hold,
and let { (w 1T be generated by Algorithm with constant stepsize y > 0. Denote J. int (w)
and fo = f (wo) fa. Then
2f0 7( +1
to + 4(1 - 5)
E [[]Vs(w>)111
82 7 22
(36)
T+1
Consequently, choosing
@((T 1)-11/2 yields
E [V( 0
min
(37)
VT+1
which matches the rate of vanilla SGI) up constants and higher-order O(1/T) term depending on the compression quality 8.
Proof. By Lmsoothnssss (Assumption applied at W. we have
E [[f(w++1) | ma] & f BB [
0x+1 w.t we]
+ E
[1 - 10x11
wx
(38)
2
LT2
f - - (V.((())) Elge mel)
IE. 119 112 we]]
(39)
2
$ I(i)) - - + LA²
0²
(40)
2
Taking full expectation and rearranging gives
L22
E F[(IV[(()) < [[S(m)] - [[](((+1)] +
(41)
2
Next we relate (zx)) to Vf Using Lipschitz continuity of the gradient,
< = 2 22] ff 2 2L
2 |2,
-3.12
(42)
Taking expectations and recalling H, 0/ ct2 we obtain
E
V (w, (w: )| < 2 R | (1)
21. E [1e..1]]]
(43)
Combining Equation 41 and Equation 43 and using Lemma 2 gives
[[IVS(()))) < [[f())) - B[((@++))) + Inyo + 2T. E [[e
(44)
8(1
$ [l.(()))] -B[(((())) Iyo2
52 72 22202
(45)
20

---

Published as a conference paper at ICLR 2026
Summing this inequality over t O..... and dividing by T + yields
B [[[[[(
2 - 7(T + +1]
(B[s(ūo)] - -B([([+()()))))))))
+ Ina
8(1 8(1-8) 6)
1² 0²
82
(46)
(1) Lyo + 2 (T +1)
8(1 8) L2
(47)
82
Absorbing the factor 8 into the constant or tightening the intermediate bounds gives the advertised inequality equation 36 which completes the proof.
Based on the above analysis, our discrepancy-awart compression with error feedback in FL fits directly into the theoretical framework of Fed-EF (Li Li 2023). In particular, Fed-EF assumes biased compression operator whose relative-norm of compression error bounded. Lemma I shows that our discrepancy-awarn top-k compressor Cais is d-approximate compressor, and thus also w/ 2 < - =)]l=] for all Together with
satisfies a relative error bound of the form |[Cdis-k(()) comppeeso with error- arror-feebbcck setting error-feetee crror-feedback mechanism, our method matches the biased studied in Li & Li (202)) and therefore enjoys the same type of non-convex convergence guarantees federated learning, while additionally providing a more informative compression rule.
21


Reviews:
[{"review_id": "eaB8L0bqvr", "reviewer": "Reviewer_qMu3", "summary": "Federated Learning (FL) enables privacy-preserving collaborative training by allowing clients to jointly learn a shared model without exchanging raw data. However, the communication overhead from model updates remains a key bottleneck, especially for resource-constrained devices. Existing compression methods largely rely on magnitude-based (e.g., Top-k) or randomness-based heuristics (e.g., ATOMO, PowerSGD), which ignore the discrepancy between compressed and original outputs, often leading to critical information loss. They propose a discrepancy-aware compression method that significantly boosts performance under extreme communication constraints. Each client uses a small subset of its local data for calibration to directly measure the output discrepancy caused by dropping candidate compression units and uses this as a selection criterion. This strategy can be integrated into mainstream compression schemes to enhance communication efficiency. Experiments show a 18.9% relative accuracy improvement at a compression ratio of 0.1, demonstrating its effectiveness for scalable and communication-efficient FL.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important problem with insightful observation.\n2. The paper is well-written and clearly organized.", "weaknesses": "1.\tTraditional compression algorithms come with theoretical convergence guarantees. Does this algorithm also ensure convergence? \n2.\tThe experiments suffer from significant shortcomings. In fact, the efficiency of conventional distributed communication compression methods largely stems from their use of error compensation mechanisms, yet this paper does not compare against such approaches in its baselines. \n3.\tIn traditional distributed communication settings, compression ratios are typically above 1%; in contrast, the highest compression ratio evaluated in this paper is only 10%.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkjxjxLYTD", "reviewer": "Reviewer_atF7", "summary": "This paper proposes a discrepancy-aware compression method for Federated Learning. It uses a small local calibration dataset to measure the output impact of dropping compression units, replacing conventional magnitude-based rules. This approach serves as a plug-in to enhance existing methods like Top-k and ATOMO, significantly boosting accuracy under tight communication budgets.", "questions": "Please see weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Results showing significant accuracy gains under high compression ratios.\t\n2. The method is designed as a plug-in, making it compatible with existing compression schemes.\n3. Provides theoretical motivation and intuitive examples to illustrate the limitations of magnitude-based heuristics.", "weaknesses": "1.While the proposed method introduces non-negligible computational overhead due to the calibration process—a significant concern for resource-constrained devices—the empirical validation of its performance gain is not sufficiently comprehensive. The scope of tested compression methods is limited to Top-k and ATOMO. To fully justify the incurred overhead, it is crucial to demonstrate the method's effectiveness across a broader spectrum of techniques, such as SignSGD/z-SignSGD and other quantization-based methods.\n\n2. Lack of comparison with FedAvg or other uncompressed baselines to illustrate the absolute performance gap.\n\n3. No discussion on whether the method can be combined with compensation mechanisms (e.g., error feedback) to further reduce compression loss.\n\n4.The experiments are confined to computer vision tasks. The effectiveness on NLP datasets and models remains unverified, limiting the generalizability of the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tzn5464zKJ", "reviewer": "Reviewer_yzYJ", "summary": "This paper tackles the key issue of communication costs in federated learning. Existing methods like Top-k sparsification and low-rank decomposition such as ATOMO depend on magnitude heuristics. These approaches fail in federated learning. Clients send accumulated parameter updates over local steps, not single gradients. This breaks the link between magnitude and real importance. The paper introduces a discrepancy-aware compression strategy. It avoids magnitude as an importance measure. Clients use a small local data subset for calibration. This directly assesses output changes from dropping each compression unit, like an element or singular triplet. The method keeps units with the largest output impact. It uses the limited bandwidth more wisely. The experiments demonstrate its effectiveness by augmenting both Top-k and ATOMO , achieving significant accuracy improvements, especially under high compression ratios and non-IID data distributions.", "questions": "1.The core idea is to find a better importance metric than magnitude. Could this discrepancy-aware metric $L_{comp}(u)$ also be used to guide other compression types, such as quantization?  Could it be used to allocate more bits to elements or singular triplets that have a high output discrepancy score?\n\n2.The paper claims compatibility with PowerSGD. Were there any specific technical challenges that prevented its inclusion in the experiments? Would the application of the discrepancy metric be as straightforward as it is for ATOMO, given that PowerSGD approximates the SVD using iterative methods?\n\n3.There is a subject-verb agreement error in lines 723–724: \"the content being compressed (parameter updates) carry little gradient.\" The subject \"the content\" is singular, but the verb \"carry\" is plural. It should be corrected to \"carries little gradient\" to agree with the singular noun \"content. You may also  check similar constructions elsewhere in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The paper's primary strength is its core insight. It correctly diagnoses why existing heuristics fail in the FL paradigm (parameter updates vs. gradients) and proposes a solution that directly exploits a key characteristic of FL (infrequent communication allows for more local computation). This is a fundamental contribution.\n\n2.The experimental results are comprehensive and convincing. The method shows consistent, significant gains over baselines across multiple models, datasets, and compression types (element-wise and low-rank). The gains are most pronounced in the most challenging regimes: high compression and high data heterogeneity.\n\n3.The paper is written with clarity. The motivation, method, and results are all easy to follow.", "weaknesses": "1.The overhead analysis in Table 4 is transparent about the per-round computational cost, showing a non-trivial increase (e.g., ~12-18% total time increase per round in some cases). The authors argue this is an acceptable trade-off. This argument would be much stronger if supported by a \"time-to-accuracy\" plot. Given the large accuracy gains, it is highly likely that the discrepancy-aware method converges in fewer rounds, potentially leading to a faster total wall-clock time to reach a target accuracy, even with the higher per-round cost. This analysis is currently missing.\n\n2.The paper mentions PowerSGD in the introduction and states the framework is compatible with it. However, the low-rank experiments are conducted only with ATOMO. Given that PowerSGD is a more practical and widely used low-rank approximation than ATOMO (which requires a full SVD), demonstrating the \"plug-in\" capability and gains on PowerSGD would have significantly strengthened the paper's practical claims.\n\n3.The paper's experiments are rightly focused on comparing the augmented methods to their original magnitude-based versions. However, the related work section mentions other advanced compression techniques, such as adaptive quantization (e.g., FedFQ, FedAQ). While the proposed method addresses selection and not quantization, a comparison or discussion of how it performs against other SOTA communication-efficiency methods (not just magnitude-based ones) would help situate its contribution in the broader landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fieeNq3md8", "reviewer": "Reviewer_qW7h", "summary": "This work proposes a new way to do compression in distributed optimization, especially for federated learning. To that aim, the clients use a small subset of their local data as calibration data to measure the output discrepancy induced by compression.", "questions": "No", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Reducing the communication cost in distributed optimization, in particular for federated learning, is crucial. The approach of taking into account the discrepancy induced by compression is interesting. The experiments are convincing.", "weaknesses": "My main concern is that you use the word \"heuristic\" multiple times to characterize existing compression methods, in a dismissive and incorrect way. In fact, this is quite the opposite: existing approaches come with thorough theoretical guarantees, which is the contrary of being heuristic, but these guarantees are based on worst-case analysis, which is often too conservative. For instance, top-k and rand-k have the same guarantees, because the worst case vector to which they are applied is a vector will all elements equal, and in that case top-k and rand-k are equivalent. But top-k performs better in practice, and this is intuitive: it is better to keep the coordinates with largest amplitude, which capture the most information, than random coordinates chosen blindly. You should rephrase your statements and cite some papers, such as Beznosikov et al. \"On biased compression for distributed learning,\" 2023; Condat et al. \"EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization,\" 2022; Yi et al. \"FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models,\" 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)