Bi-Directional Communication-Efficient Stochastic FL via Remote Source Generation
Authors: Maximilian Egger
Rawad Bitar
Antonia Wachter-Zeh
Nir Weinberger
Deniz Gunduz
Venue: NeurIPS 2025 poster ()
Keywords: Communication-efficiency
Importance sampling
Minimal Random Coding
Stochastic federated learning
Primary area: probabilistic_methods
TL;DR: We propose a stochastic federated learning framework with inherent communication regularization and principled compression via remote source generation. It achieves 5–32× communication savings underlined by theoretical guarantees.

Abstract: Federated Learning (FL) incurs high communication costs in both uplink and downlink. The literature largely focuses on lossy compression of model updates in deterministic FL. In contrast, stochastic (Bayesian) FL considers distributions over parameters, enabling uncertainty quantification, better generalization, and, crucially, inherent communication-regularized training through a mirror-descent structure. 
In this paper, we consider both uplink and downlink communication in stochastic FL, and propose a communication framework based on remote source generation. Employing Minimal Random Coding (MRC) for remote generation, we allow the server and the clients to sample from local and global posteriors (sources), respectively, rather than transmitting locally sampled updates. The framework encompasses communication-regularized local optimization and principled compression of model updates, leveraging gradually updated prior distributions as side information. 
Through extensive simulations, we show that our method achieves $5-32\times$ reduction in total communication cost while preserving accuracy. We further analyze the communication cost, refining existing MRC bounds and enabling precise quantification of uplink and downlink trade-offs. We also extend our method to conventional FL via stochastic quantization and prove a contraction property for the biased MRC compressor to facilitate convergence analysis.

Bi-Directional Communication--fficceli Stochastic FL via Remote Source Generation
Maximilian Egger', Rawad Bitar, Antonia Wachter-Zeh Technical University of Munich
Munich, Germany {maximilian, egger, rawad hatwaddbitar... antonia. wawiiaawwhhteeezeaataaeee
Nir Weinberger
Deniz Güiddüz
Israel Institute Technology Haifa, Israel nirwein@technion.c ac. i1l
Imperial College London
London, United Kingdom
d. gunduz@imeeriia. ial. ac. uk
Abstract
Federated Learning (FL.) incurs high communication costs in both uplink and downlink. The literature largely focuses on lossy compression of model updates deterministic FL. In contrast, stochastic (Bayesian) FL considers distributions over parameters, enabling uncertainty quantification, better generalization, and, crucially, inherent cummunication-regulattt In this paper, we consider both uplink and downlink communication in stochastic FL, and propose communication framework based remote source generation. Employing Minimal Random Coding (MRC) for remote generation, we allow the server and the clients sample from local and global posteriors (sources), respectively, rather than transmitting locally sampled updates. The framework encompasses
training through mirror-descent structure.
wttpuuniiiaiooongaaaiiaiaaaaa local optimization and principled com- pression of model updates, leveraging gradually updated prior distributions as side information. Through extensive simulations, show that our method achieves 5 - 32x reduction in total communication cost while preserving accuracy. We further analyze the communication cost, refining existing MRC bounds and en- abling precise quantification of uplink and downlink trade-offs. We also extend our method conventional FL via stochastic quantization and prove a contraction property for the biased MRC compressor to facilitate convergence analysis.
Introduction
Federated learning (FL) enables collaborative machine learning (ML) across multiple clients orches- trated by central federator [McMahan et al.) 2017). Communication efficiency, privacy, security, and data heterogeneity are well-established chall challenges in FL [Zhang et al.) 2021]] Wen et al.] [2023] As bi-directional process, FL requires substantial uplink and downlink communication, posing increasing pressure on communication networks as MI. models grow larger. To address this, lossy compression techniques have been widely adopted to reduce uplink gradient transmissions and downlink model
broadcasts (Seide et al. 2014 Alistarh et al. 2017 PPhilippenko and Dieuleveut 2020 Giruntkowska al. 2023). However, these methods almost exclusively focus on conventional (non-stochastic) FL,
where clients train deterministic models and transmit fixed updates.
Alternatively, stochastic (Bayesian) FL offers improved generalization, robustness, and inherent uncertainty estimation [Zhang et al. 2022 Milasheuski et al. 2025) Rather than training deterministic
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

models, clients learn local posterior distributions. aggregated by the federator to obtain global posterior. Recently. [[sik al. 2024] empirically demonstrated state-of-the art performance under limited uplink bandwidth using stochastic compression methods, outperforming classical approaches. This framework can be applied to a variety of Bayesian FL. solutions such as QSGD Alistarh et al. [2177] QLSD Vono et al. (2022. dithered quantization (Abii and Fekri (2019) and FedPM[sst et al. 2023 as well as to conventional FL settings augmented with stochastic compression.
A key technique enabling stochastic FL is remote source generation, which allows the federator to sample from the clients' local posterior, rather than obtaining samples locally generated by the clients. This avoids redundant transmission and enables tight, stochastic control over communication. Such remote generation requires common randomness shared between the transmitter and receiver in the form of a common prior Li [2024), which we also refer to as side information. If the downlink unlimited, this allows the server to broadcast the global posterior to all the clients, and this posterior serves as natural common prior. However, when both uplink and downlink are limited, the possibility for remote source generation is restricted, which challenges the application of efficient stochastic FL. Thus, in this paper, we explore and analyze communication-effin The rigorous treatment of stochastic FL in this scenario is further reinforced by its two fundamental advantages: (i) Communication cattruunicaiion-reallaiieeddd training: we show that stochastic FL, opposed to conventional solutions, inherently integrates communication constraints into the training process, effectively treating communication
stochastic FL.
integral part of the optimization objective; (ii) Priors as side information: the probabilistic structure allows principled integration of common priors as side information, reducing communication costs to the update from prior to posterior.
Concretely, we address the following question: Can joint uplink and downlink compression signifi- cantly reduce communication costs in stochastic FL without compromising accuracy? We answer this question affirmatively, tightening the communication-accuras trade-off in ways that deterministic methods cannot, and achieving communication reductions of up to 32x without performance loss. Below, we summarize our key contributions.
We propose two novel bi-directional compression algorithms for stochastic FL with Minimal Random Coding (MRC): one leveraging globally shared randomness, and one requiring private shared randomness between each client and the federator. Both enable efficient sampling-base communication by exploiting carefully selected side information.
We demonstrate substantial communication savings, reducing total cost by factors of 5 - 322 while maintaining competitive accuracy across standard benchmarks. Our ablation studies analyze the effects of shared randomness and the choice of side information.
We extend our approach to conventional FL with stochastic quantization, proving a contraction property of the resulting (biased) compression operator. This enables convergence guarantees in both uni- and bi-directional settings.
We develop a theoretical framework for communication analysis stochastic FL., quantifying uplink and downlink costs under MRC. Our results refine and generalize bounds from Chatterjee and Diaconis [0088), including tight analyses for Bernoulli distributions and tools applicable broader distribution classes.
2 Preliminaries: Stochastic FL with Bi-Directional Compression
In this section, we shortly review the concepts of stochastic FL and compression based on MRC, which are employed in our proposed stochastic bi-directional algorithm.
Stochastic FL. A set of n clients collaboratively and iteratively train a model, e.g., a neural network, under the orchestration of a federator. We primarily consider cross-silo federated learning with moderate number of reliable clients [Kairouz and McMahan 2021] Client E [M] == (1....,m)) possesses a dataset D,, We differentiate between homogeneous data where D, drawn independently from the same distribution for all clients (i.i.d),, and heterogeneous data, where each D, may come from different distribution (non i.i.d.). At each training iteration the federator holds model 0, described by probability distribution. After downlink transmission, each client i has an estimate ò., of 0., and locally optimizes òz, to obtain a local probabilistic model, called the posterior df- Compressed clients' posteriors of are transmitted back to the federator on the uplink to obtain an estimate jf. The federator aggregates the received posteriors using aggregation rule R (-) obtain

---

refined global distribution B.+1 = R (f heem)) A simple aggregation rule R (-) is the average over all clients' posteriors. This process repeated until a certain convergence criterion is met. In many stochastic FL settings, the sent client updates of are just samples from the posterior q!-
In fact, conventional FL with stochastic quantization though with the following differences: (i) the federator holds model 0r with deterministic parameters; (ii) each client locally optimizes f,, to obtain a local gradient gf. A stochastic compression (-) applied on the client's gradient to obtain a posterior distribution qf from Qx (iii) samples of q. are transmitted to the federator on the uplink to obtain an estimate of the gradient, which we still denote by q and (iv) the federator updates the global model as 0.+1 = 02 nit (f helm)) with learning rate 7). In this paper, we will investigate both settings, with a prominent focus on the former.
also be described by the procedure above,
Stochastic Compression by MRC. To efficiently transmit samples from the posterior qfs we employ MRC (Havasi et al., 2019]. which allows to leverage shared randomness and side information common to the federator and the clients. MRC is a stochastic compressor (.) whose input is a posterior distribution Q and a prior distribution P. and its output is sample from a distribution Q close to Q. It operates as follows: The encoder and decoder generate 745 samples {X; from P. The encoder computes a categorical distribution W, with (i)) x ((X.))P(X.)) and transmits an index i i W with log, (ms) bits. To obtain high accuracy, it is required that ris = O(exp((xx (Q) P))). where DKL (QIIP) is the KL-divergence between Q and P [Chatterjee and Diaconis, 2018 For brevity, in what follows, for two Bernoulli distributions with parameters and p. we will use the shorthands dkL (qllp) and (g.re(g.2)
The choice of MRC for remote generation stems from a practicality aspect and ease of exposition. Ordered random coding (ORC) (Theis and Ahmed [022] is natural advancement, reducing the entropy of the indices selected for transmission using the Gumbel-Max trick. The extension to ORC can be incorporated with our methods using a minor adaption that we omit here for clarity. [hheis) and Ahmed [2022] further expose an interesting connection to the Poisson functional representa- tion, allowing exact remote generation. Such lossless sampling schemes, however, usually incur substantially larger communication costs and sampling complexities, often rendering those methods impractical. Thus, we herein focus on lossy remote generation schemes for efficiency.
3 BICOMPFL
We next introduce our method BICOMPFL,, bi-directional stochastic compression strategy, which uses MRC to reduce both uplink and downlink communication costs. The scheme assumes that shared randomness between each of the clients and the federator exists, which can be implemented using pseudo-random sequences generated from a common seed. We distinguish two types of shared randomness: private shared randomness (between individual clients and the federator) and global shared common randomness (among all parties), with the latter being more challenging achieve in practice. We assume all clients and the federator share the same global model & at initialization. This does not incur any communication when global shared randomness is available, but necessitates an initial model transmission from the federator to clients when only private shared randomness exists.
Bi BICOMPFL: The General Algorithm. Our method serves as a general framework for stochastic optimization procedures. We explain BICOMPFL for Bayesian FL and show in the sequel how it can be used for conventional FL with stochastic quantization. Consider probabilistic mask training (similar to FedPM, [Isik et al. 2023) )) dimension d are parameters of Bernouli distributions. Those parameters determine for each weight of randomly initialized network with fixed weights whether it is activated or not. During inference, the weights w are masked with samples x' E (0, 1] ~ 0&, i.e., the network weights are w O x². We start with a general description, which is valid for the cases of global and private shared randomness.
example of Bayesian FL. The models 0, [o, 1]d of
At iteration t = O, each client E [n] shares with the federator the same global model, i.e., 0 do. for all i E [n). At iteration t, each client i locally trains model 0,, in L local iterations. In our previous example, when training Bernoulli distributions to mask a random network, the parameters are mapped to scores dual space, which are then trained for L local iterations m E [L] using stochastic gradient descent. Mapping the trained scores back to the primal space, each client i obtains a model update in terms of a posterior of- We refer to Appendix F for details. This optimization principle is a special instance of mirror descent, which, in the special case of optimizing over Bernoulli
3

---

Algorithm BICOMPFL-GG with Global Randomness
Require: Both clients and federator initialize the same global model 0g using shared seed Clients set prior p² = 0,,0 = do vi [n]
Ensure:
repeat
2: for Client i E [n]] do
3: of - Local training of 0,,
4: Employ Coro p²) sample indices 1i.0 E [mLL], 6 e [B] with prior p², transmitted to
federator reconstruct of
5: end for
6: Federator updates global model 0,+1 Ei= di
7: Federator relays to client j the other clients' indices (1 be
8: for Clients E [nx] do E of from {19,e)
9: Reconstruct 01.441
10: end for
11: Clients and federator set prior p² 0.+1
12: I f- ++1
until Convergence
distributions, leads a point-wise minimization with respect to a KI.-proximity term (as opposed to the Euclidean distance in standard SGD, cf. Appendix[[]oor details). The KL-divergence between the updated local model and the global model directly determines the communication cost. Hence, we regularize the minimization of the loss function by the communication cost, thereby treating communication as an inherent optimization objective.
To convey the model update of to the federator, each client employs C-re(+) (-) in B blocks of size d/B B each (assuming for simplicity that B divides d) with a prior distribution Pl,.> which set to pL,, 8.D0 iteration
0. The choice of pi,,, for > will be clarified later. For each block [J/ client conveys nun samples { of of the federator by transmitting for each block b an index Iř, with log2(nss bits, where nys is the number of samples per block, generated from the prior distribution p¹... at both the client and the federator using the available shared randomness. The samples of all blocks are concatenated for each f. Hence, the federator obtains an estimate of client i's posterior distribution using the empirical average of
E
RuL
vf,e-
RUL
By averaging the estimates of for all the clients' models, the federator updates the global model as 0.+1 1 E.= T. To transmit the new model to each client i, we assume the existence of a common prior Pi.a shared by the federator and the clients. With p²,a- the federator performs MRC in B blocks of size d/B B/B to make client sample from, and thereby estimate, the latest global model 0,+1- The client samples raDL masks (af,e) J |e each incurring a communication cost of logz (mis) bits. An estimate of the updated global model is obtained by concatenating the reconstructed samples for all the blocks b E [B). and averaging over masks 0,,
Σ Σe= xh.
TDL
Since the number of clients is typically large, ful 1 often suffices. The clients' contributions averaged at the federator, effectively reducing the noise due to the MRC step. This allowed Isik et al.) 2024) to theoretically analyze the uplink communication for importance sampling-base communication of model updates. We will follow a similar approach for downlink communication; however, since downlink communication cannot benefit from the averaging effect of multiple clients, we reduce the variance of the model estimate in the downlink by setting RDL = n AULL-
stochastic
The choice of the priors pi,, and p²., for MRC in the uplink and downlink channels, respectively, crucially affects the performance and the communication cost of the algorithm. As first-order characterization, the communication cost of MRC is determined by D (qf Ilp(..) in the uplink and by DxL (0.+ _.pp..... in the downlink. We continue the description with the easier setting in which global shared randomness available, before turning to the more challenging setting of private randomness.
Global Randomness. When global shared randomness is available, all clients can maintain the same priors at each iteration and, thereby, obtain the same global model estimates 0,,- The global model known the clients and the federator from initialization, and synchronization among all clients

---

Algorithm 2 BICOMPFL-PR with Private Randomness
Require: Both clients and federator initialize the same global model 0g using shared seed Clients set prior Pi.. Pi. 0,,0 to vi E [n]
Ensure:
repeat
2: for Client i E [n]] do
3: of Local training of 0z.,
4: Federator employs Cmro (9) pi..)) to draw Rul samples sh.e ~ of using prior pi...
5: Federator estimates client's posterior of mun Σ vt,
6: end for
7: Federator updates global model 0s+1 = N/4 Σ of
8: for Clients E [n]] do
9: Client employs (0r4100 to draw uDL samples oi. ~ oc+1 using prior pl.
Client est. global model: 1,1 n.m 1e=1 die
11
11: Clients set prior pl,u = pl,a 01,41
12:
end for
13:
t 1+1
14: until Convergence
ensured by choosing as prior ph., pL.a the latest estimate of the global model dz.- The clients utilize the globally shared randomness sample the exact same samples from the same prior for uplink transmission at all iterations. Selected indices of such samples are transmitted to the federator to convey an estimate q of the posterior q' who reconstructs the global model 0,+1- Using the same prior in the downlink, i.c.. the global model from the previous iteration, the updated model can be transmitted the clients through MRC. Leveraging the shared randomness, all clients E [n] sample from the same prior, and thus obtain the exact same estimate of the global model 0. = 02+1, for all i E [r]]. Hence, we have that pi., = pl.a = O, for all i E [r]].
In this version. the federator reconstructs the global model from estimates of the client posteriors o However, in the uplink, all clients sample from the same prior, which enables further improvements. Naively, the federator will reconstruct the global model using the indices 4 for b E [B], l E [Mui.] received by the clients i E [ri] through MRC, followed by additional MRC round for downlink transmission. Instead, and more efficiently, the federator can simply relay the indices to the respective other clients (i.e., client j receives 1he for b E [B]. E [n/] (i}. & [mu..]). which reconstruct the updated global model individually. This avoids additional noise by second compression round and allows better convergence without additional communication facilitated by global randomness. We term this approach BICOMPFL--GE summarized in Algorithm |
Private Randomness. Without global randomness, maintaining the same prior among all clients is impossible without additional communication. Instead, an additional round of MRC is needed for the downlink transmission, and cach client obtains diffcrent estimate of the global model 0.,, each iteration. Hence, the clients' local trainings start from different estimates of the global model. In non-stochastic setting, this has only been considered by PPillipenko and Dieuleveut 2021], Gruntkowska et al. [2024]. Understanding the additional cost incurred due to lack of shared randomness in terms of both the convergence speed, communication load, and the choice of the priors pi._ and Pi..- is then of interest.
For the uplink transmission of client i, any convex combination of Ô.,, and of can be used prior, i... pf.. X0 + (1 A)q for some 0 < A < 1 (cf. Appendix 1.2 for details). This due the availability of both quantities at the federator and client i, However, small À values are not expected to reduce the cost of communication reflected by dxr. (g/llpt.. since the previous global model estimate is likely to be similarly different from the posterior (in terms of the KL-divergence) than the previous posterior estimate of the federator. Indeed, our numerical experiments have shown that the savings from choosing X + 1, i.e., priors other than 0,., are not significant. For simplicity, we thus propose to use Pi... = pla = θ, We term this approach BICOMPFL-PR and summarize the procedure in Algorithm 2 Choosing different priors is possible and only affects line 11 in Algorithm 2 We mention in passing that BICOMPFL--PE allows partial client participation, which is incompatible
5

---

with shared randomness and the method BICOMPFL-GR, We further note that our methods are readily compatible with sparsification and pruning techniques (e.g.. [Wnggi et al. (2018). , Shi et al.) (019 by excluding pruned parameters from the block selection.
Block Allocation. We consider three different block allocation strategies: 1) fixed block size (referred to as "Fixed" in the experiments), where each block 6 E [B] is of the same size and constant across all adaptive block allocation (Adaptive) as proposed by Isik al. [0024, where each block size is separately optimized each iteration E; and 3) adaptive average allocation (Adaptive-Agg), where the block sizes are equal but optimized each iteration according to the average KL-divergence per block. We refer the reader to Appendix for a detailed discussion on this.
Partial Client Participation. We note that BICOMPFL-PR supports partial client participation without additional communication overhead, as it does not rely on full prior synchronization among the clients. BICOMPFL-GR the other hand, requires that all clients' priors are synchronized at each iteration. To allow partial client participation, the federator is required to transmit the previous global model the clients that were absent in the previous iteration. This ensures full synchronization of the global model estimate used for efficient uplink communication. Hence, BICoMPFL-GR's compatibility with partial client participation comes with a potentially increased downlink communication cost for previously passive clients.
4 Experiments
We next evaluate the performance of our proposed BICOMPFL-GR and BICOMPFL-PR schemes in experiments, and compare against baseline FL strategies without compression (FedAvg or PSGD) [McMahan et al. 2017) and several non-stochastic bi-directional compression schemes that employ different combinations of compression, reeob-fee and momentum. particular, we compare
[Tang et al.) 2019), SGD (Stich et al. [0018), NEOLITHIC (Huang) jet afl.f2022 2022], CSER [Xie et al.] 2020), and the recently proposed LIEC [Cheng al. [024). SignSGD
against DOUBLES
[Seide et al., 2014) serves to compress the transmitted gradients for all the schemes. We further
compare with M3 Gruntkowska et al. 2024], which partitions the model into disjoint parts for downlink transmission and transmits to each client a different part of the model. While M3 is focused on RandK compression for the uplink (i.c., transmitting random K entries of the gradient), use TopK (Wangni et al. 2018, Shi al.) 2019]. which achieved much more stable results. As mentioned above, the mirror descent approach outlined in Section 3 inherently minimizes the communication cost by-product. Hence, it is strong candidate for stochastic FL. Nonetheless, we show how our method can also be used to improve the communication efficiency in conventional FL, by using the uplink and downlink compression Care(-) combined with stochastic quantizers, e.g.. (Alistarh et al. 2017). In Section [3] we pave the way to convergence guarantees by proving contraction property of C-re(-) composed with stochastic quantization Q.(+) of gradients of- To compare our method to the baselines that use SignSGD as compressor, we evaluate BICOMPFLL-GR in conventional federated learning (CFL.) task with stochastic variant of SignSGD. We replace mirror descent over Bernoulli masks by standard learning procedure over a deterministic model, which takes input the global model estimate Q.,,, computes a gradient of (over L local epochs, using SGD and cross-entropy losses), and outputs a distribution Q²(g)) In stochastic SignSGD. (-) transforms each gradient entry ghee to a Bernoulli random variable with parameter qiee = 1/(1 + pp((-f(e/K), for some K > O, where the random variable takes value +1 with probabil- ity qi,., and -1 otherwise. We then employ Care (g), P(... to obtain samples vi,e. where the compression is performed element-wise. We apply this method to BICOMPFL-GEE where Step 6 is replaced by 0(+1 = 8, 7e n Σm 9 where of _ >mm vi and n/ is the feder- ator's learning rate. Step 9 is modified accord- ingly. The priors p' are chosen as Bernoulli random variables with parameter 0.5. We re-
1.0
Tell Accomryy
0.7
Foddug
BiComoFL Adoptive
-boblessusse
BiCompFL GR Assptivv Aug
Memoggi
BiCompFL-GE Fixed
Liec
BiCompFL -itodFLLGG---ttdttctt RakoRRccsttt-od
0.5
BiComoFL Food
BIComoFL
SpinDL
N3
BiCom G.----------LL-- Fined
0.4
Run
10r
1014
10
Commarication
Figure 1: Test accuracy for BICOMPFL and base- lines on Fashion MNIST 4CNN on i.i.d. data.

---

mark that while MRC samples are biased towards p² (as we discuss in Section 5) this particular prior choice avoids imbalance in stochastic SignSGD, and rather acts as a regularizer, pulling the clients' posteriors closer to maximum entropy distributions. Consequently, convergence is achieved under bi-directional compression even without error feedback. For general prior choices, error feedback may be needed, see Algorithm g
and Appendix B
We will refer to this method as CCCOMPFL--GR-FII
We study n 10 clients (see Appendix H for additional experiments with more clients) collab- oratively training a convolutional neural network (CNN)-based classifier for the datasets MNIST, Mashion-MNIST and CIFAR-10 under the orchestration of a federator. For MNIST, we use two different models. LeNet-5 Lecun et al. 1998] and 4-layer convolutional neural network (4CNN) proposed by Raamamujan et al. [2020]. The latter is also used to train on Fashion MNIST. For CIFAR-10, 10, we use larger neural network with 6 convolutional layers (6CNN). We train MNIST and Fashion-MNIST for 200 global iterations and CIFAR-10 for 400 global iterations. Through all experiments and datasets, we carry I. = 3 local iterations per client. The learning rates are carefully selected to ensure convergence and comparability across all methods. Particularly, we tune the hyperparameters so that all algorithms achieve similar accuracies, allowing fair comparison of their communication costs (see Appendix [.6] for details). Our main claims are the communication reduction of the bitrates per parameter per epoch, which are orthogonal to the choice of the learning rates of the algorithms. The code to reproduce our experiments is included in the supplementary material.
We evaluate the schemes in two settings: with a uniform data allocation (i.i.d)), to model homogeneous systems, and with a non-i.i.d. allocation, to model heterogeneous systems, where data allocation for each client is drawn from a Dirichlet distribution with parameter a = 0.1. This regime is challenging due to extreme class imbalance. Each result shows the average across three simulation runs with different seeds. Further details on the simulation setup and the network architectures are deferred to Appendix E Consistently throughout all experiments, our proposed methods provide order-wise improvements in the communication cost, while achieving state-of-the art accuracies.
We plot in Fig. D the test accuracies for all the schemes as a function of the total communication cost in bits per parameter and per global iteration. While all the schemes achieve approximately the same maximum test accuracy, BICoMPFL-GR and BICOMPFL-PR require substantially less communication. Hence, when the bandwidths of uplink and downlink transmissions are limited, both variations of the proposed method achieve better test accuracies. Turning our focus to the different variations of our scheme, it can be observed that, without partitioning the model for downlink compression, BICOMPFL-PR converges significantly slower than BICOMPFL-GR for any block allocation method. This highlights the intuition above that the additional MRC step in downlink incurs further noise, which reduces the convergence speed. However, when we partition the model in the downlink and only send disjoint parts to each client through MRC (BICOMPFL-PR-Fixce) SplitDL), the downlink communication cost reduces by factor of n. In the regime of Fashion MNIST with a uniform data allocation, this comes without performance degradation, and is hence the method of choice in this regime. We additionally simulated BICOMPFL- GR with the suboptimal implementation (BICOMPFL-GR- -eeGGRReees---ieeeeeeee Fixed), in which the federator first reconstructs the global model, and then performs additional MRC step for downlink transmission. This naturally reduces the convergence speed per iteration without gains in the communication cost. Hence, justifying the substantially
choice of BICOMPFL-GE. reduces the communication cost without loss in performance. In some cases, especially for non-i.i.d. data, we even observe improved performance, which we attribute to implicit regularization. Note that BICOMPFL-GR--O
We show that, in conventional FL, BICOMPFL-GR-CTI
provides improvements even without error-feedback or momentum. However, our method is fully compatible with such techniques and can be used as a plug-in approach to further minimize the communication cost in many existing schemes. We study the convergence in Section 5
We plot in Fig. the)the the schemes' average bitrates over the maximum test accuracy for MNIST and 4CNN. The average bitrate is reduced by more than a factor of 1000 compared to FedAvg, and more than factor of 32 compared to DOUBLESQUEEZE, NEOLITHIC and LIEC, which perform best among the conventional bi-directional compression methods. We repeat the study for non-i.i.d.dal allocation according
Dirichlet distribution with parameter = 0.1, and show maximum test accuracies over average bitrates in Fig. 2(b) Partitioning the model in BICOMPFL-PP worsens the final accuracy of the model. While the model converges faster, it does not achieve the same accuracies as BICOMPFL-GR and BICOMPFL-PP without partitioning. This hints towards hybrid schemes for BICOMPFL-PR. where the training begins with partitioning on the downlink, which is later switched
7

---

0.86
638
1275
E27
S Preull ca fond
Dur
X
6AR
104
(a) MNIST 4CNN ii.dd
(b) MNIST 4CNN non-i.i.d
(c) CIFAR- 6CNN i.i.d.
Figure 2: Maximum test accuracy over total communication cost measured by bitrate per parameter.
to full transmission. In Fig. 2(c) we provide the results for CIFAR-10 10 and uniform data allocation. BICoMPFL-GR and BICOMPFL-PP both achieve better results with bitrate smaller by factor of 5 than the best baselines. More detailed numerical results can be found in Appendices H and !
The adaptive block allocation (Adaptive) of Isik et al. 2024 saves communication costs in many settings and provides better performance than the fixed block allocation (Fixed), due to more accurate MRC tailored to the exact divergences. The proposed low complexity adaptive strategy based on the average KL-divergence (Adaptive- Avg) per block can additionally save in communication (and computation) with no or little performance degradation. We refer the reader to Appendix for further extensive experiments, graphs for accuracies over epochs, separate studies of uplink and downlink costs, and comparisons for the case of available broadcast channel from federator to the clients. Finally, we refer to Appendix I for various ablation studies analyzing the sensitivity of BICOMPFL with respect to the choices of the priors, n, nDLL nys, the block size d/ B, and the learning rate n).
5 Theoretical Results
Convergence. In stochastic FL, the exact time dynamics of the system are challenging to analyze due to the round-dependent interplay of the learning procedure with the transmission noise. However, when using BICOMPFL for conventional FL with stochastic quantization (cf. BICOMPFL-GR- CFL), convergence guarantees can be given. We prove the convergence for general and widely used class of stochastic quantizers Q.(-), which are natively unbiased.
Qa(-) takes as input the entry 9e of a gradient vector g E Rd and operates as follows. Let be the number of quantization intervals, and let 0 S < s be an integer such that < lsel ||||
< re+1 then (g(e) outputs (ign + 1)/8 with probability l/gel/1lg) - Te. and || sign(ge)re// otherwise. Q.(-) is unbiased, i.e., Q0Q,000 (x)] x, and its variance satisfies ExIlQ, - x min(d/s² vd/s)|x
Alistarh et al. 2017). See Remark in Appendix B for a discussion on the choice of s.
Replacing stochastic SignSGD by Q-(+) in BICOMPFL-GR-CHT the posterior given by a Bernoulli distribution with parameter ví.e = $ l/looll Te- The values |igll. sign(g). and Tr can be encoded
independently, e.g., using Elias coding. With a slight abuse of notation, let (mnre(Q(.)). the composition of Q.(-) and MRC with rys samples per entry. The compression Ciore Q2 (gf). takes a gradient of and outputs samples from a distribution close to Q*(gf). ). and falls in the class of biased compressors. We can prove the following contraction property for (Q,r((Q.)),) facilitate convergence analysis for uni- and bi-directional compression. This constitutes substantial improvement over [Isik et al.) 2024), where such guarantees were missing, and hence no convergence guarantees were given. A prominent biased contractive compressor is TopK.
denote
which will
Lemma 1. For any Rd and corresponding posterior q following (x)... and prior p [0, 1]d 1- A' = max.eld] de (%= + + and p == max Pe- The compres-
let Â maxee(d] Pe
sor Cmre (Qa(-) satisfies the following contraction property for nis = O (/2 log(6pAA A 6p A + { and s 2 22d
S[]lCinr((Q..es - 1 (1- 8)11211
6 = 1- - (1+
A ng
+0 ( A3) /
Gpi (2nus)
8

---

The underlying core result is a refinement of the MRC analysis, cf. Lemma 2 Hence, for sufficiently large nis. the compressor Cmre(Q, ().)) can be used as an alternative to common compressors such as (.). The use of MRC introduces a bias into the otherwise unbiased stochastic quantization. Based on the contraction property in Lemma standard convergence results (cf. Theorem 2 follow easily by a straightforward extension of our conventional FL algorithm
( (Appendix A
A.CoMPFL-GR-CEE to error feedback (cf. Algorithm
3 as detailed in Appendix B
Communication Cost. We analyze the communication cost in a specific iteration t and comment on the inter-round dependency later. When the latest global model estimate d.,, is chosen as a prior in MRC, the uplink cost is determined by how far the model evolves during the client's training, i.e., dkL (a' ((..) = dkı (u'll0,.)) After communicating samples of the posteriors, the federator obtains an estimate of for all E [n]. The cost of communication on the downlink to client i is then determined by dRL(,, >mm g (0s...) While dka (gf depends on the progress during client training, the core challenge is to bound the expected divediveeeeeee of each model estimate dgi. ( (Q..)) in the presence of potentially different priors, i.e., 0z. # 0z,,, i 4 j. For each client i, the overall communication cost is in the order of riDexpp (dKL. ( (+ g [Ipp ) + ruuL exp (dKL (d'llp(...)) We will next quantify (0..)) ) for the case pf... = pi,d however, the analysis can be extended to pi. # P by an additional assumption on the divergence between the two priors.
( ET_ of
For the theoretical analysis, we focus on the scalar case for a single iteration t, where client E [n] has posterior Q; and the federator and client share common prior P., both are Bernoulli distributions with parameters Ci and pi. respectively. In the context of FL, the client locally trains P. and results with Qi. According to Chatterjee and Diaconis (2088) and the multi-client extension of Isik et al.) 2024]. the communication cost in the uplink is determined by exp(dkı. (QillP:)). After uplink transmission, the federator obtains an estimate q: of q:: and hence, the updated global model is given by 1. - q: The downlink cost for client i is determined by dxL ( -.=1 lil...).
We derive new high probability upper bound on this quantity, refining previous MRC analysis for the special case of Bernoulli distributions. While a more general analysis can be conducted for other classes of distributions (cf. Remark iI Appendix (A), the Bernoulli-batt based optimization method described earlier proves particularly efficient by enabling conmunication-regulaaae train- ing, a unique property that renders our methods substantially more communication--fficies while preserving state-of-the-art performance.
Let X be Bernoulli sample obtained through MRC. As an initial step, we bound the difference between qi and the probability Pr(X- = 1) that the samples are drawn from, which vanishes when
pi q; (and hence dKL (g: [p:lpp:) 0). We note that the bound of Chatterree and Diaconis (2018)
Theorem 1.1] does not satisfy this natural property. We formally state the result in Proposition in Appendix A which, however, does not yet capture the dependency
the number of samples may used
in MRC to sample an index. We refine Proposition with Lemma (cf. Appendix (A). which addi-
tionally captures this dependency, and will allow to derive an upper bound on dxı ( +>>3 4:1 (p.).
Lemma 2 is of independent interest and can be seen as a refinement of [Chatterjee and Diaconis 2018 for Bernoulli distributions. It is required prove Theorem 1
For the statement of the following theorem, we assume that the progress by one local client training is bounded by bg pş| S p for all j E [n]]. Using Pinsker's inequality to bound [g; pyl S / dki. (gy [p;) /2, this is natural assumption given from the KL-proximity term of mirror descent (for one local iteration), and can be strictly enforced through the projection of qj onto a KL ball around p, of fixed divergence. We assume that the difference between the clients" priors, i.e., their global model estimates in our algorithms, are bounded as lp. P5 S ( for all i, E [m].
gi Theorem 1. Assume Pj > for all j E [m]. A;:= = T pc and D/ : (Pets + 1-P.ts)
with probability 1 8', the global model divergence ( ( Σソ= a [l::) is upper bounded by
2 minn 1 }
A
In(2/6")
6(p. + log (2nys)
+0+ + (A, + +
T 7
Tus
By Chatterjee and Diaconis 2018 J. this provides an immediate bound on the cost of downlink transmission. The bound applies to both algorithms BICOMPFI--PE and BICOMPFL-GR. However, Reconst), i.e., Ç = 0, the bound simplifies
when all priors p, are the same (such in BICOMPFL-GG -eR-eoos),

---

accordingly. The explicit dependency on the factor 1/vTLL reflects the interplay between uplink and downlink cost. The parameter Ç gives rise an inter-round dependency of the communication cost. The more accurate the estimation of the global model in the previous iteration (given the priors are chosen as O,,)), the smaller C. and hence the lower the transmission cost in the subsequent iteration. The proofs of Proposition T
Lemma 2 and Theorem can be found in Appendix A
Related Work
Following the introduction of FL [MMMahan et al., 2017). lossy compression of gradients or model
updates has been a long-studied narrative in FL, with prominent representatives such as SignSGD, also known as 1-bit Stochastic Gradient Descent (SGD) [Seide et al. [0144), QSGD
AAlistarh et al.
2017]. TernGrad [Wen et al.] 2017). SignSGD with error feedback [Karimireddy al.
2019
vector-
quantized SGD [Gandikota et al. 2021] and natural compression [Horvóth et al.
2022 Such
methods retain satisfactory final model accuracy even with aggressive quantization. Sparsification-
based methods have also been considered as alternatives, e.g., Topk (Wangni et al. 2018 Shi et al.
2019]. The importance of bi-directional gradient compression in many settings was outlined by
Philippenko and Dieuleveut (2020). Many schemes were proposed that leverage combinations of gradient compression in the uplink and downlink, error-feedback, and momentum, e.g., Mem-SGD (Stich et al. 2018 DoubleSqueeze (Tang et al.) 2019), block-wise SignSGD with momentum SGD with error reset (Cser) [ie et al. 2020], Artemis [Pilippenko and Dieuleveui [020]] Neolithic [Huang et al. 2022]. DoCoFL [Dorfman et al.) EF2I-P and friends (Gruntkowska et al. [023]] J. 2Direction Tyurrn and Richtárik [2023], MB Gruntkowska et al. 22224. and LIEC [Cheng al. [202]]. With the exception of the methods MCM Philippenko and Diculeveut 2021] and M3 Gruntkowska et al. 2024], each client receives the same broadcast, potentially compressed, global gradient or model update. Isik et al. [024] studied uplink compression for stochastic FL and showed significant communication reduction with competitive performance. Their framework, termed KLMS, applies to variety of stochastic compressors and to Bayesian FL settings, e.g., QLSD Vono et al. [2022]. The compression is based on importance sampling and MRC, thoroughly studied by Chatterjee and Diaconis (2018) and Havasi al. [2019). Such methods, known as relative entropy coding, have been used in FL in conjunction with differential privacy, cf. DP-REC (Triastcyn et al. 2022).
Zheng et al., 2019], communication--ffien
Since the lottery ticket hypothesis [Frankle and Carbin 2019]. finding sparse subnetworks of neural
networks that achieve satisfactory accuracy was investigated. Ramanujan al. (2020) showed that randomly weighted networks contain suitable subnetworks of large neural networks capable of achieving competitive performance. [Isik et al. (2023) formulated a probabilistic method of training neural network masks collaboratively in an FL. context.
7 Conclusion
We illuminated bi-directional compression in stochastic FL via federated probabilistic mask training, which we showed to inherently optimize both the learning objective and the communication costs. By leveraging side information through carefully chosen prior distributions, the total communication costs are reduced by factors between 5 32 compared to stochasccctiic FL baselines, while achieving state-of-the-art accuracies on classification tasks, for both homogeneous and heterogeneous data. We thus close the gap of downlink compression for stochastic FL and complement the existing literature on bi-directional compression for standard FI.. Applying our methods to stochastic quantization in conventional FL, we paved the way to convergence analysis for MRC-based compression. Allowing different priors among all clients, this work opens the door to studying compression under side- information in decentralized stochastic FL, where a central coordinator is missing. Our theoretical results are of independent interest and may be applied in various scenarios where MRC is used.
Client privacy and fairness are important directions beyond the primary focus of this work. Com- pression methods commonly strengthen the clients' privacy in FL. Particularly, the noise introduced through our sampling methods dilute the individual contributions of the clients observed by the federator, naturally enhancing the clients" privacy and reducing the risk of privacy breaches through, e.g., membership inference attacks. Quantifying the privacy-utility trade-off that arises in the presence of bi-directional compression is left an interesting direction for future work.
10

---

Acknowledgments
The research of R. Bitar is supported by the German Research Foundation (DFG) under grant agreement number BI 2492/1-1. The research of N. Weinberger was partially supported by the Israel Science Foundation (ISF), grant no. 1782/22. D. Gündüz acknowledges funding from UKRI under the ERC-Consolidttor project AI-R (EP/X030806/1) and INFORMED-AI Hub (EP/Y028732/1).
References
A. Abdi and F. Fekri. Nested dithered quantization for communication reduction in distributed training, arXiv preprint 1rXhv:190..01197, 2019.
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: coammumication-cfficceee SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems, volume 30, 2017.
S. Chatterjee and P. Diaconis. The sample size required in importance sampling. The Annals of Applied Probability, 28(2):1099-1135, 2018.
Y. Cheng, L. Shen, L. Xu. X. Qian, S. Wu, Y. Zhou, T. Zhang, D. Tao, and E. Chen. Communication--ffcciienn distributed learning with local immediate error compensation. arXiv preprint arXiv:2402. 1857, 2024.
R. Dorfman, S. Vargaftik, Y. Ben-ltzhak, and K. Y. Levy. DoCoFL: Downlink compression for cross- device federated learning. In International Conference on Machine Learning, pages 3356-8388, 2023.
Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
V. Gandikota, D. Kane, Kumar Maity, and A. Mazumdar. vqSGD: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, volume 130, pages 2197-2205, 2021.
K. Gruntkowska, A. Tyurin, and P. Richtárik. EF21-P and friends: Improved theoretical communi- cation complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning, pages 11761-11807, 2023.
K. Gruntkowska, A. Tyurin, and P. Richtárik. Improving the worst-case bidirectional communication complexity for nonconvex distributed optimization under function similarity. arXiv preprint arXiv:2402.06412, 2024.
M. Havasi, R. Peharz, and M. Hernández-Lobato, Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations, 2019.
S. Horvóth, C.-Y. Ho, L. Horvath, A. N. Sahu, M. Canini, and P. Richtarik. Natural compression for distributed deep learning. In Proceedings of Mathematical and Scientific Machine Learning. volume 190, pages 129-141, 2022.
X. Huang, Y. Chen, W. Yin, and K. Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. Advances in Neural Information Processing Systems, 35:18 35:18955-18969 2022.
B. Isik, F. Pase, D. Gunduz, T. Weissman, and Z. Michele. Sparse random networks for communication-efficcen federated learning. In International Conference on Learning Repre- sentations, 2023.
B. Isik, F. Pase, D. Gunduz, S. Koyejo, T. Weissman, and M. Zorzi Adaptive compression in federated learning via side information. International Conference on Artificial Intelligence and Statistics, pages 487-495, 2024.

---

P. Kairouz and H. B. McMahan. Advances and open problems in federated learning. Foundations and trends in machine learning. 14(1-2):1-210, 2021.
S. P. Karimireddy, Q Rebjock, S. Stich, and M. Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. International Conference on Machine Learning, volume 97, pages 2252-3261, 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
C. Li. Channel simulation: Theory and applications to lossy compression and differential privacy. Foundations and Trends@ in Communications and Information Theory, 21 2024.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas. Communication-Efion Efficieet Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics, volume 54, pages 1273-1282, 2017.
U. Milasheuski, L. Barbieri, S. Kianoush, M. Nicoli, and Savazzi. Bayesian federated learning for continual training. arXiv preprint arXiv:2504. 15328, 2025.
C. Philippenko and A. Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. arXiv preprint arXiv:2006. 14591, 2020.
C. Philippenko and A. Dieuleveut. Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems, 34:2387-2399, 2021.
V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in randomly weighted neural network? IEEE/CVE conference on computer vision and pattern recognition, pages 11893-11902, 2020.
P. Richtárik, I. Sokolov, and I. Fatkhullin. E121: A new, simpler. theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems, 34:4384-4396, 2021.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. Interspeech, pages
2014.
S. Shi, Q Wang, K. Zhao, Z. Tang, Y. Wang, X. Huang, and X. Chu. A distributed synchronous SGD algorithm with global top-k sparsification for low bandwidth networks. International Conference Distributed Computing Systems (ICDCS), pages 2238-2243 2019.
R. Srinivasan. Importance sampling: Applications in communications and detection. Springer Science & Business Media, 2002.
U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified SGD with memory. Advances in Neural Information Processing Systems, 31, 2018.
H. Tang, C. Yu, X. Lian, T. Zhang, and J. Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In International Conference on Machine Learning, pages 6155-6165, 2019.
L. Theis and N. Y. Ahmed. Algorithms for the communication of samples. In International Conference on Machine Learning, pages 21308-21328, 2022.
A. Triastcyn, M. Reisser, and C. Louizos. DP-REC: Private & communication-efficient federated learning, 2022.
A. Tyurin and P. Richtárik. 2Direction: theoretically faster distributed training with bidirectional communication compression. In Conference on Neural Information Processing Systems, 2023.
12

---

M. Vono, V. Plassier, A. Durmus, A. Dieuleveut, and E. Moulines. QLSD: quantised langevin stochastic dynamics for bayesian federated learning, In International Conference on Artificial Intelligence and Statistics, pages 6459-6500, 2022.
Wangni, J. Wang. J. Liu, and T. Zhang. Gradient sparsification for communication-effictie distributed optimization. Advances in Neural Information Processing Systems, 31, 2018.
N. Weinberger and M. Yemini. Multi-armed bandits with self-information rewards. IEEE Tronsactions on Information Theory, 2023.
Wen, Z. Zhang. Lan, Cui, J. Cai, and W. Zhang. A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics, 14(2)513-535, 2023.
W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. TernGrad: Ternary gradients to reduce communication distributed deep learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
C. Xie, S. Zheng, S. Koyejo, I. Gupta, M. Li, and H. Lin. Cser: Communication--ffiiicnnn sgd with error reset. Advances in Neural Information Processing Systems, 33: 18:12593-16610 2020.
C. Zhang, Y. Xie, Bai, B. Yu, W. Li, and Y. Gao. A survey on federated learning. Knowledge-Band Systems, 216: 106775, 2021.
X. Zhang, Y. Li, W. Li, K. Guo, and Y. Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pages 26293-26310, 2022.
S. Zheng, Z. Huang, and J. Kwok. Communication-efficeenn distributed blockwise momentum SGD with frror-feedback. Advances in Neural Information Processing Systems, 32, 2019.
A Proofs and Intermediate Results
In the following, we provide the formal statements Proposition aadd Lemma including their proofs. Parts of the proof of Proposition T will be used prove Lemma] 2 afterward.
We prove Theorem
Proposition 1. For sample X5 transmitted by MRC with posterior and prior Bernoulli distributions with parameters q and p. we have
max p 1-p 11-pp---- 1111-9
Pr(X, = 1) gl 5
9 1 111pp -1-11-1----1
Proof of Proposition 7 Assume party wants to sample from Bernoulli distribution Q with pa- rameter 9. which is held by another party. Both parties share common prior in the form of Bemouuli distribution with parameter and have access to shared randomness. Fix any sample index for the moment (this index will be needed for the proof of Theorem [). Both parties sample Knss ii.d. samples Xt,; ~ P for i E [nis] independently and identically from P. The party holding Q constructs an auxiliary distribution
We(1) Q(XE))/((E.)) EE Q(Xe.)/P(Xe))
from which it samples to obtain an index Ir. The index transmitted to the other party, which reconstructs the corresponding sample Xt,re-
To bound the difference Pr(X,, = 1) g) i.e., the target Bernoulli parameter compared to the parameter which the sample is drawn from, by the independence of the samples Xe,,, for different f,
13

---

we focus on single sample & E [K], for which it holds that
Pr(Xe, =
R35
Pr(Xe,1 21. Xe,us Ings) Pr(Ie = Xe,1 21...... Xt,ss (gg
Σ
Inns
2/1
(c)
shps
Σ
Pr(X111 = 1, Xx,2 = .2,...,, ..... =Fnss)
Pr((Te = 1 Xx,1 = 11, Xe,2 I2, ......., N,m, = Engs
(b)
THS
Σ
Pr = X,2 22...., Xemm = Inss)
L=0
P 2122 =L}
Pr(Ie = Xe,1 1, X,,2 I2....... Xe, Enss
where (a) follows from symmetry. (b) follows since by permutation invariance, the inner probability only depends the number of ones in 1
The inner probability given by the distribution Wi).). Given that Xt,1 and that Ens Xe,e I., it holds that
Σ Q(x.))/((x.))
2 + (ns - L p
1-
1 1-
Hence,
Pr(l = Xt,1 1, Xe, T2.
Xemm 2nns)
(L 1) " (nus - I. - 1) 1-p
1-9
which is independent of the exact choice of (I2,... Fmis given their sum im Xe, = L. Since
Pr(Xe.1 1, 1,, .2....., Xinss Ins) = p1+ (1 p)nss-L- by the Bernoulli distribution assumption, we have
ni -1
9 (L + 1). + (nus L 1)
Pr(Xe,, = 1) n
mis
p 1 (1pp)mmsss -)s -is---
1-q
T-p
Defining binary random variable M with sample space {g. 1- for a Bernoulli distribution
with success probability parameter L+1 where success refers to the outcome M 9 we can write that
Ber
Mis
1) p
Pr(Xe, 1) =
I
L+1 9
+
1is I-
L=0
1
q E
qI
L+1 I +
L- 1-4
[
Bor
L11
(0)
qE
l+t1 "
M
where the outer expectation is over the binomial distribution with nis trials and success probability i.c., L ~ Binomial( - 1,p), and where (a) follows from Jensen's 's inequality over the inner expectation. Hence,
Pr(Xe. =
Pr(Xe, = 1) =-
q
E
-1)
Ber
L+11 m1
M
14

---

Since
1
22
[M], it also follows from T
that
Bor
L+1
Ber
Lil
Pr(Xe,, = 1) = q IE
qi
L+11
[
11g P
Ber
2 q/
2
L+1
[
from which we have
(M)
Pr(Xe,,, 1) - 9 2 9
E Ber
Combining the upper and lower bound in and respectively, we derive
Pr((ee, = 1) d S
max
E
IE
Ber
L-1 mi
M
E
E
Ber
L+1
M
E
max
E
Ber
L+1 195
[M]
E
Ber
L+t mis
M
-1
1 M,
E
Ber
L+ 445
max
M
- q 9 q p 1 p ]
-
-1)
5 q
E
max
p 1-p 1-9
max
q- 1 1'' p-1-pp
This concludes the proof.
Lemma 2. For sample X₃ transmitted via MRC with posterior and prior being Bernoulli distribu- tions with parameters q and p. A
1-9 and A' == 1 ( + + we have
1-p
N'
6p log (2mıs)
Pr( 1) o S
+0
(A (A+
TUS
Proof of Lemma we follow until get
The proof starts with the same derivations as for the proof of Proposition
which
1
Pr(Xe, =) qE
M
Ber
Since L is a random quantity that follows Binomial distribution, we bound Pr(Xe,,, = 1) - using a concentration bound on L. The relative (multiplicative) Chernoff bound states that
Pr([L <(nss)) 2 Emssp) Pr(L- <(nIs) 2 Enss) Pr(I =(nss) S -EMsPP)
n
exp
3
for any e E [0, 1]. Setting e
3 (2/88 map
implies that
L nis
33nsp log(2/8)
with probability most d. Setting (2nrs) that
obtain for concentration parameter f/8
E = { mispl 2
Note that assume + nJ 1 and p n) 2 0, otherwise the concentration can be trivially bounded.
15

---

with probability Pr(E)= _ "B
Then, write
Pr(X,, =) q.E
M
"Ber ( 1-1
q.
1(83)
+ q
118}
Ber 1
M
Ber
[M]
Assume for now that q < p (we will later proof the opposite event), then
is strictly
M
non-increasing in L since < average that
1-g and hence, when &" holds and hence L concentration around the
E
M
((-s)) M
Ber
L-1 113
Ber
[nys
1) no)
1 (ngs 1)(p na)
705
I-P
p
P
176
2 +
p
+
1
+ 176
12
I-p
1-2
1
1 +
q
1-g
T-p
8
1)a
i
1
(1 (1-D 1/6 11188
1
74i
where the last step by Taylor expansion. Using (4) and the monotonicity of
we write
L11 15
[M]
Pr(Xe. = 1) = EiE
Ber
L++1 P955
I
"1-P+ 1--p+1 178 1176 )r)
1+ Σ 1) 111-) p
98"
x=1
and hence
q 1- p + V/6
Pr(Xe,, = 1) - 5 8p + (1 5) Σ
1)
p
k=1
Similarly, we get by bounding
[ and using
that
Bar
L-11 mjs
M
Ber
((++))()-+a))) 215
Pr(Xe, = 1) - 2 89
1
(1
8)
20
11) x
q
1-9
p
176 + 1878
1-p
x=1
1 1 p q- C-PP(XX,,e -
1)*+11 (g 1 1- " (1-
176 + 72716
+ 8)
744
When p q, then
is strictly non-decreasing, hence, under E, we have
M
u+1
1 1-9)
111-1 ((1 77 + 12176
1+
1 p)
M
M
Ber
Lt1
Ber
mi
16

---

and thus from ④ that
1 p
2 1) (1
- 1-
9
(1
(1-p- -1-P-10 78 + 12776
Pr(Xe,, = - 9 q0
+
(1 8)
1-
1
745
==1
Similarly, we bound
obtain
M
(++1)- "18
M
00
9 12-1-9 1-9
p + 176 12776
= - + (1 8)
1-1)
p 1-p
7HS
k=1
8
2 1-9 q " (1 (1- p 775
8
q q- -r =1 -00 (11 (1-0)
(11)
71376
p
ris
Since 0 < 118 < and 12 p-p 2 0 by appropriate choice of the concentration intervals, we have by approximations up to second order terms that
Pri(X., = =) - gl S gō max
p 1-p
+ 176
1-1 -
+ 9
q
+
T
q
q
p 1
1-9
q
1 1-1 9
27
6p log (2mRs)
+0
+
2 sis
q
1
-
1- p)
This concludes the proof.
Proof of Lemma 7 Using Lemma we can show the following. Recall the following probability law of the stochastic quantizer (.(.) Alistarh et al. 2017] using > 0 quantization intervals, which
takes as input the entry I, of a gradient E. vector. Let S Te < be integer such that Is S
[xal [x]
I=+1 then (x) is defined as Ber [sr.
-eT) which outputs ||x|| (iig((x)((T 1)/s [x
in case of success, and ||||| (e//
otherwise.
Focusing on an entry Te. we prove a contraction property for MRC with stochastic quantization with [|] Te. and an arbitrary prior Pe. In fact, the MRC methodology Cure leads to
posterior 4e = | sampling from approximate distribution with parameter ğe- To be more specific, Care (Ix) outputs
||x|| (ign(2))( (T= + 1)//s with probability ãe. and |x|| sign(a,, /re/ with probability Te- We established in Lemma an upper bound on 19c qel. which will be useful in the following.
To prove a contraction property of the kind
Bi][Curc(x)
x| S (1 x)11
we can write
[|[ Cmre (x) x||
Σ
(Cmre (Ix) xx0)
d =||x||
Cmre Care((x)) (xe)
27
Ze |x||
=
E
|x|
e=1
= ||x Σ
(ggn(xe)(Te
+
Dee
+ (1
)c))
(sign(z))
Ze |x||
||||
= =||||| Σ
(g. le (e)
Te 1
8
|xe|]
||x]
2
+
(1
Ge
go)
Te
8
|xee|
|
27
= |||| 2
To
xel
2
1
1
Te
x
(go + ge (e)
+
|
32
8
8
|
17

---

Te
[xel
2
+
(1
qe
+
90
|x||
8
d
27
|
(e
q)
+
1
+
Te
+
Te
82
8
8
|
2s
8
|
|
e=1
(5)
where
1
Te
9e
32
8
[3re]
X
$2
2
Te
xe
+
X
s
||
To
|atel
||[]
|x[|
Substituting the result in 5
obtain
d
[[]]Cmre(X - xX||3] = E
CCrr - xx0)²
|x
(īe qe)
1
+
1
Te
[
x
82
8
8
||x
|
e=1
|x
(āe 4e)
To
1
Te
|
8
|
8
||
< |x|| Σ
1ge gel
To +
+
Te
|x|||
|||||
s
e=1
d
d
S (1
Ze
+
52),
$2
where, by Lemma we have for
A. =
4e
1-9c 1-pe
and D' =
Pe
+ 1-pe
that
1-9.
Pe
Gpe log (2n1s)
Iğe del
5
+ O
(A. +2)g
nis
Let A = max,e(d]
Is 1-9e
A max idf ge
1-ps and p max. Pe- We will
Pe
I-Pe"
9e
1-4c
ensure that
A' ng
+ ((A
A2)
Gp long 1aas
by making each of the individual terms 1
By choosing n4s
2 V2A we have S 2.
To ensure that (Á + A?)
66 log (2ris)
< 2> we
require
log(2nnx
By Weinberger and Yemini 2023 Lemma 15], this holds when
Ras
V (
vis = (log(6pp( + A)) 6p(A A + A²)). Hence, choosing viss = (max 22 log(6p(A A
A 6p( + A we have A + O (A+
6p 6plog(2mss)
1. Thus, we have 0 S < 1
2d .
< 1, and hence 2 v 2d. This concludes the proof.
18

---

Proof of Theorem 7 Assume a party estimates the Bernoulli distributions Q2 with parameters g held by parties m). The estimating party shares with each of the other parties a common prior Pj in the form of Bernoulli distribution with parameter Pj and access to unlimited shared randomness. To help estimate Qj. the j-th party sends K samples the estimator through MRC. Therefore, both parties sample Kng i.i.d. samples Xe, ~ P3 for e E [K] i E [mss] independently and identically from P.. The party holding 2 constructs for each & E [K] an auxiliary distribution
(X)))))
We(1) Σ (,(Xe.)/P/((X..))
from which it samples to obtain an index It- The index is transmitted to the estimating party, which reconstructs the corresponding sample Xe.,-- Averaging the samples for all & [K] gives estimate ă of qj, i.c. dz = K Σ Xt,,,- This process is repeated for all j E [n]]
We assume that 1g, Pyl S for all i [r), and that the difference between the priors, is bounded
as lp: pyl S Ç for all j E [n]] The goal is to bound dkL ( Σÿ= d lp:)) from above for any i E [n).
By the convexity of Kl.-divergence, we have
dgi.
1 72 j=1
Σ
nt
' lp
n dxr (â>lls)).
n
To bound dkL (0il12) (p:) for any i,j E [n²]. by the triangle inequality, we can write
14 pıl S 14, Pr(Xx 1)] + Pr(X, 1) +1 + 1g p,l lp, - psl,
where 16, Pr(X; = 1)] is bounded by Lemma 2
By Hoeffding's 's inequality, we have with
probability least 1 5' that
l Pr(Xe =1)1 - ln(
2nps
Thus, with probability at least 1 - 8', since Py S P + C. we have with 4, = 41
1-91
and
Ps-3
I-py
AJ = 4i
B 4
+
1-pot( that T-9
A
(A, + + 6(pt + () log (2nss)
In((8/2)
Pil -
+0
+
+
+ +
nis
2
nas
2m1s
This holds under the assumption that pj S for all j E [m]. By the reversed Pinsker's inequality, obtain
A nis
6(p. log (2nys)
DKL (âgllps) S min (pr, 1 - -pp
+
(A,+ 13)
n4s
Im(81/2)
+ p p+s 5
2n1s
The statement of the theorem follows by the convexity of KL-divergence.
Remark 1. Note that our analysis can be extended to other parametric distributions, such as multi-variate Gaussians. The key ingredient to replace the specific upper bound on the bias in Lemma 2 with the generic results from Chatterjee and Diaconis [2018], which holds for all classes of distributions. Using this upper bound, one can follow our derivations to prove the communication costs in Theorem [ i.e., using the convexity of KL divergence and decomposing the error in parameter estimation into a bias term and a concentration term to bound the sampling error. Standard concentration results be utilized to bound the latter from above, e.g., Hoeffding's inequality for sub- Gaussian random variables. The remaining steps follow analogously to our proof. Similar adaptations apply to the contraction property in Lemma I necessary establish convergence guarantees for conventional FL.
19

---

B Convergence Analysis
Using the contraction property derived in Lemma] we can show that a straightforward extension
of BiCoMPFL-GR- CFL to error-feddbac as used in (Richtárik al. [021] leads to the following convergence guarantee. The algorithmic details of the extension can be found in Algorithm
Therefore, assume that for all for x, y E IR and [n]. the following Lipschitz property holds:
[F D.) - VF D.)Il S L -yll
Let F(O) = - D VF D,) be the global loss function and L' Di=1 L,
Theorem 2. If F = inf (F()) -0 and Bilg' VF S o², then with n S
(L+
1-8
L 1, 2d, and rus satisfying Lemmal I in every iteration t, we have
[1 11-
for Algorithm that
T EB(IFON)) 2(F( F* n/T
σ²
(1 v1- 8)T'
Similarly, guarantees be derived for other algorithms, such modified versions of BICOMPFL.- PR with error-feedback and momentum, using Lemma[] However, we emphasize the generality of BICOMPFL, reaching beyond conventional FL with stochastic compression pure stochastic narratives.
The choice of depends on the model architecture and dataset complexities, affecting the number of iterations required until convergence. The simpler the learning task, the fewer quantization intervals are sufficient for convergence. Beyond generic arguments. we particularly highlight that should be carefully selected, respecting the expected variance of the gradients. If gradients exhibit inherently large variance, e.g., through small mini batch sizes, large variance of the local datasets, or substantially pver-parumeterized networks, fewer quantization intervals might be efficient, and increasing that number would only negligibly improve the performance. We also note that one-bit quantization has been shown to be often remarkably effective despite its simplicity /eide et al.) 2014 Karimireddy et al. 20199.
Remark 2.
C Gradient Descent with a KL-Proximity
Mirror descent employs point-wise optimization in the form of a first-order approximation of F(O,, D:) with proximity term (p g), where Dr is the Bregman divergence associated with function F(-). When F(x)= = ||=|| and hence the Bregman divergence the Fuclidean distance, this is known gradient descent. Let now p and be vectors with the entries corresponding to independent Bernoulli parameters. When we choose F(x) = x log(x) + (1 - x) log(1 x), the Bregman divergence becomes (p. = Ex= DKL (pkl (p)l(q)).
gx).. Hence, we are optimizing with respect KL-proximity constraint. The mapping between dual and primal spaces then given by F=(x) = log(x) log(1 2) and (VF(x)) c-1 respectively; also known as the inverse sigmoid and the sigmoid functions.
D Block Allocation
The simplest yet effective strategy for block allocation is to partition the model into equally-sized blocks of size d/ B for MRC (Fixed). The partitioning into blocks is required to make MRC practically feasible this setting. is known that for vanishing MRC error, the number of samples nis from block vf,.. of the prior is supposed to be in the order of exp DKI.. (1.s.lle(...)))) where of & is the
b-th block of posterior yf- It was observed by Isik et al. 2024 that the KL--divergence decreases as the training progresses with the global model used as prior, which is intuitive since the local training will change the posterior less and less training converges. To adapt the block size according the
20

---

Algorithm 3 BICOMPFL-GR-CTH with stochastic quantization Q+(-) and EF21 from Richtárik et al. [002T]
Require: Both clients and federator initialize the same global model 8g using shared seed
Ensure: Set & clients set prior p' 6 = Ou, Vi E [n], clients compute and broadcast
v.. (Q p²), with of the local gradient for 06; hence, v = : - v? public
Update Vi 0.+1 0. syvt+1
2: repeat
3: for Client E [n] do
4: Compute gradient of by local training over L local iterations
5: Stochastic compression of 4 Qg v()
6: Sample indices /eee& [mo..] E [B] from of with prior p' and transmit federator to reconstruct of (m)e(gg.p')
7: Update v/ = = +
8: end for
9: Federator reconstructs and computes v v'1 + of from (R,e)
10: Federator updates 01+1 02 m/y+t
11: Federator relays to client j the other clients' indices {I,, (el
12: for Clients E [nx] do
13: Reconstruct and compute v = v + + 2 of from {1/
Update 02+1 0, myv*t1 from {
end for
16: Clients and federator set prior p² = 0.+1
+ [+
18: until Convergence
divergence from the posterior with respect to the prior, Isik et al. [024] proposed an adaptive block allocation strategy (Adaptive), where upon realizing large deviation from the target KL-divergence per block, clients partition their model into blocks with equal sums of parameter-wis KL-divergences and transmit the block intervals to the federator. The federator aggregates the indices of all the clients, and broadcasts the updated block allocation.
We propose in this work low complexity solution that adapts the block size according to the average KL-divergence per block (Adaptive-Avgs This alleviates the cost of computing and transmitting the exact block partitions, where the transmission of each block size requires loga (fmax) bits, with bmax the maximum pre-defined block size. Instead, the transmission of one size is enough in our solution. If the average KL per block DRI. (4..lllpp...s) deviates than given factor, the clients request to update the blocks. In the next iteration, each client proposes a block size, and the federator averages and broadcasts an updated size.
E
Additional Experimental Details
We use the cross-entropy loss and a batch size of 128 in all our experiments. We use Adam [Kingma and Ba 2015] an optimizer with learning rate 77 0.0003 for all non-stochastic methods, and 0. for probabilistic mask training. For non-stochastic FL. we use a federator (server) learning rate of 0.1, i.e., the clients' gradients are averaged, and the federator updates the global model with learning rate 0.1, and with learning rate of 0.005 for BICOMPFL-CC GR with SignSGD. For M3, we use a federator learning rate of 0.02 to obtain reliable results. For LIEC and CSER, we use an average period of 50 global iterations (cf. [Cheng et al. 2024 Xie et al. 2020 For M3, we use TopK with K = [d/n]. To run the simulations, we use cluster of different architectures, which we list in the following table.
21

---

Table System specifications of our simulation cluster.
(PU(s)
RAM
GPU(s)
VRAM
2x Intel Xeon Platinum 8176 (56 cores)
256 GB
2x NVIDIA GeForce GTX 1080
11 GB
2x AMD EPYC 7282 (32 cores)
512 GB
NVIDIA GeForce RTX 4090
24 GB
2x AMD EPYC 7282 (32 cores)
640 GB
NVIDIA GeForce RTX 4090
24 GB
2x AMD EPYC 7282 (32 cores)
448 GB
NVIDIA GeForce RTX 4080
16 GB
2x AMD EPYC 7282 (32 cores)
256 GB
NVIDIA GeForce RTX 4080
16 GB
HGX-AI00 (96 cores)
1 TB
4x NVIDIA A100
80 GB
DGX-A100 (252 cores)
2 TB
8x NVIDIA Tesla A100
80 GB
DGX-1-VINE (76 cores)
512 GB
8x NVIDIA Tesla V100
16 GB
DGX- (76 cores)
512 GB
8x NVIDIA Tesla P100
16 GB
HPE--P100 (28 cores)
256 GB
4x NVIDIA Tesla P100
GB
The details of the CNN architectures used in our experiments are summarized in the following. The parameter count is 706 for LeNet5, 1933258 for 4CNN, and 2262602 for 6CNN.
Table 2: LeNet5 Architecture Overview
Layer
Specification
Activation
5x5 Conv
6 filters, stride
ReLU, AvgPool (2x2)
5x5 Conv
16 filters, stride I
ReLU, AvgPool (2x2)
Linear
120 units
ReLU
Linear
84 units
ReLU
Linear
10 units
Softmax
Table 3:
4-layer CNN (4CNN) Architecture Overview
Layer
Specification
Activation
3x3 Conv
64 filters, stride
ReLU
3x3 Conv
64 filters, stride
ReLU, MaxPool (2x2)
3x3 Conv
128 filters, stride
ReLU
3x3 Conv
128 filters, stride
ReLU, MaxPool (2x2)
Linear
256 units
ReLU
Linear
256 units
ReLU
Linear
10 units
Softmax
Table 4: 6-layer CNN (6CNN) Architecture Overview
Layer
Specification
Activation
3x3 Conv
64 filters, stride
ReLU
3x3 Conv
64 filters, stride
ReLU, MaxPool (2x2)
3x3 Conv
128 filters, stride
ReLU
3x3 Conv
128 filters, stride
ReLU, MaxPool (2x2)
3x3 Conv
256 filters, stride
ReLU
3x3 Cony
256 filters, stride
ReLU, MaxPool (2x2)
Linear
256 units
ReLU
Linear
256 units
ReLU
Linear 10 units
Softmax
For the sake of clarity, in the paper restrict the analysis to a fixed number of importance samples Tis, block sizes B, and choice of priors pf... Pi.-- Our experiments have shown that, while increasing vis beyond the ones used our algorithms slightly improves the convergence over the number of epochs, the convergence with respect to the communication cost did not significantly improve. The block size is mainly limited by the system resources at hand, and one would choose the largest possible for best efficiency while complying with memory resources. We investigated many different
22

---

Algorithm 4 Local Training at Client
Require: Model d.e
1: Map model to scores in the dual space: $i, = o (0((),,)) = =log ( 6,,e
(0)
(1-12.1)
2: for Local iterations m E [L] do
(m 3: $ie S
(m)
V F F(O mm) Di), where 0!::- o(s (m--1))
4: end for
5: Map back to primal space: of = (s
prior choices and found the former global model to be reasonably good in almost all cases. With high heterogeneity, it might be beneficial to use different convex combinations as priors, which mix the former global model with the latest posterior estimate of a certain client, but the gains we experienced were minor. Hence, we settled on the former global estimate for simplicity in presenting the algorithm.
F Federated Probabilistic Mask Training
The idea in federated probabilistic mask training (FedPM) Isik al. 2023 is to collaboratively train a prohabilistic mask that determines which weights to maintain from a randomly initialized network. The motivation stems from the lottery-ficket hypothesis [Frankle and Carbin 2019), which claims that randomly initialized networks contain sub-networks capable of reaching accuracy comparable that of the full network. The weights of the network are randomly initialized at the start of training, and remain fixed. The federator and clients only train a mask, which determines for each parameter whether it is activated or not, i.e., identifying an efficient subnetwork within the given fixed network. The probabilistic masks 0, are described by Bernoulli distributions, i.e., Og [0, 1]ad contains a Bernoulli parameter to be trained for each weight of the network. These parameters determine the probability of retaining the corresponding weights. During inference, the weights are masked with samples x' {0. 1)dd Og from the distribution 0r, i.e., the inference conducted on a network with weights w C r². In FedPM, clients sample from their locally trained models, and send these samples to the federator, which, in turn, updates the global model by averaging these samples. The communication cost of this scheme is fixed for all iterations, even though the communication cost can be reduced since the KL-divergence between the global model and the locally trained models diminishes the training progresses.
We adopt the following federated learning procedure for collaboratively learning network masks, and highlight in the following the parallels to mirror descent by referring to primal and dual spaces. Starting from a common model Og. at iteration t, each client i locally trains the model Oi,, in L local iterations. To enable gradient descent, the model d,,, is mapped to scores S in a dual space by the inverse Sigmoid function 5 trained for L local iterations m E [L] by computing the gradient V V F.u-,, 1,t -)) the straight-through estimator used to compute the gradient of the non-differentiable Bernoulli sampling operation based on the distribution d = as ((-1)), i.e., the gradient equals the Bernoulli parameter. By mapping the model back the primal space, each client obtains a model update in terms of posterior of = =(s(())
(0,,c) = log((..)) log(1 0,,c). The scores are then Dr), where
(L)), The client training process is summarized in Algorithm
G Minimal Random Coding (MRC)
Isik et al. [2024] proposed a method, called KL minimization with side information (KLMS), to reduce the cost transmitting the local models of to the federator. Consequently, the communication cost depends on the KL-divergence between the desired distribution and the common prior. This method utilizes the common side information available both the clients and the federator, as well as shared randomness. The idea is that instead of sampling locally and sending the samples to the
23

---

federator, the federator in the KLMS method samples from the desired distribution through MRC. In a nutshell, MRC (Havasi et al. 2 based on importance sampling [Srinivasan] 2002] and makes use of a common prior to sample from desired distribution. Consider two distributions P and Q, where P is known to both parties, and Q is only known to the client. To make the federator sample from Q, both parties sample riss samples (Eems)) from P. The client forms
an auxiliary distribution (i) _ Q(X.)/P(X.) capturing the importance of the samples. A
Q(X.)/((X.)
sample from W is fully described by its index i, which can be transmitted with logz (nIs) bits, and approximates sample from Q2 Chatterjee and Diaconis [2018) shown that importance sampling with posterior Q and prior P requires 7is to be in the order of ((exp(DKu (Q)) where DxL (Q) P) denotes the KL-divergence between distributions Q and P. In what follows, we will also denote the KL-divergence between two Bernoulli distributions Q and P with parameters q and p by dkL (g) lp).
H
Additional Experiments
We provide in the following experiments for both uniform (i.id.) and heterogeneous (non-i.i.d.) data distributions for training LeNet5 and 4-layer CNN on MNIST, a 4-layer CNN on Fashion MNIST, and a 6-layer CNN on CIFAR-10. The details of the neural networks can be found in Tables to 4 For each setting and method depicted, we show the average of three simulation runs with different seeds. We plot for each setting the test accuracies over the communication cost in bits, and the maximum test aceuracy over the bitrate. We provide tables summarizing the maximum test accuracies with their standard deviation over multiple runs, the total bitrates and the bitrates split into uplink and downlink. The overall bitrates per parameter (bpp) are computed assuming point-to-poing -oopoonnt links between all participants, i.e., uplink and downlink costs have equal weight. For the case when a broadcast (BC) link between the federator and the clients is available, the bitrate per parameter for all baseline schemes reduces by factor of n. BICoMPFL-GRprofis similarly from the broadcast link, but BICOMPFL-PR giving the same overall bitrate compared to the point-to-point link scenario. We highlight for each of the measures the scheme with the best result. Consistently throughout all experiments, BICOMPFL. achieves order-wise savings in the bitrates per parameter while reaching state-of-the-art accuracies in the classification task. While the sampling can introduce an additional computational overhead depending the implementation, the storage cost is similar to the baselines. Since we leverage as priors the former global model, the additional storage cost incurred is limited to storing until the next iteration the estimate of the former global model at each client, i.e., where the training started, which is usually not a bottleneck. This can be cheaper than some baselines, which require storing data for momentum and error- fredbbkk
profit due to the absence of shared randomness,
1.0
0.99
0.8
0.95
FesArg
BiCompiL- Gh-Adaatiiee
Aocunny
Doublesaueeze
BiCompFL Adagtive
0.6
Memagd
BiCompFL
0.95
Lisc
#ComptL-an-Peooonsiie
0.4
Mamagd
Fechuy
BiCaipp L-Bl-find
BiGompFI-CR Ad-ppiiiiieeee
BiGmmpfL-CB-AAeee -cR-ddspliieee
Paammm
0.55
Neolthic
Cser
BiCompFL Foee
BiCompFL PR Ries SpiRDL
Uier
BiCompfL -GR Reconst- fleed
RiCampll
0.2
Haulkhic
0.33
M3
10'
10*
10'
10
10a
Communicatinn Cost Bits
Bitrotc P.
(a) Test Accuracy over Communication
(b) Test Accuracy over Bitrate
Figure 3: MNIST LeNet i.d.d
For LeNet5 on MNIST, it can be observed that all our proposed methods converge significantly faster to satisfying accuracies with respect to the communication cost, while achieving higher maximum accuracies after 200 epochs than the non-stochastic baselines. Partitioning the model on the downlink can help to further reduce the communication cost with only a minor loss in performance, especially in the i.i.d. setting. For non-i.i.d. data distribution, the loss in performance is larger than for i.i.d.
24

---

distribution. However, at the beginning of the training, the model improves faster with respect to the communication cost than all other schemes. The bitrates are comparable for all our methods, with the exception of suffer notable performance degradation from employing an additional MRC step (especially for i.i.d. data allocation).
Further, CEAMPFI--GRRRcoomee
does not
Table 5: MNIST LeNet i..d.
Method
Acc (mean std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.978 + 0.1
64.0
35.0
32.0
32.0
Doublesqueeze
0.981 0.1
2.0
1.1
1.0
1.0
Memsgd
0.977 = 0.1
33.0
4.2
1.0
32.0
Liec
0.983 0.1
4.5
2.5
2.3
2.3
Cser
0,982 + 0.09
34.0
4.3
1.0
33.0
Neolithic
0.982
4.0
2.2
2.0
2.0
M3
0.925 + 0.2
15.0
2.2
8.0
7.1
BiCompFL-GRR-Adappae
0.992 0.0006
0.36
0.068
0.036
0.32
BiCompEL-GRR-Alapiiv-AN
0.992 = 0.0003
0.29
0.055
0.029
0.26
BiCompFL-GR-Fixed
0.992 = 0.0002
0.31
0.059
0.031
0.28
BiCompPL-GR-RReonss-iiss
0.99 = 0.0002
0.34
0.063
0.031
0.31
BiCompFL-PR--Fin
0.99 0.0004
0.34
0.34
0.031
0.31
0.988 + 0.0009
0.063
0.063
0.031
0.031
10-
0.85
0.8
Maximum Text
FooAug
BiCempFL-GRR GR.A-ii
X
Fedking
BiCongF Adapiive
0.80
Dochlequeere
BiCompF LiN-Addpiii-t
Memaag
BiCampFLLCC-FFi
Memagd
Like
Cure
BcCompll -R--Feedd
BiCampfL LR Raconst- Foed
0.75
Luc
Cur
BiCongFL Raw.aaatttttttttttttco Reconst-Fixed
BiCompFl.PR-fee
Necitiic
0.70
Communication
p.
(a) Test Accuracy Communication
(b) Test Accuracy over Bitrate
Figure 4: MNIST LeNet non-i.i.d
Table 6: MNIST LeNet non-i.i.d.
Method
Acc (mean std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.911 + 0.2
64.0
35,0
32.0
32.0
Doublesqueeze
0.899 + 0.2
2.0
1.1
1.0
1.0
Memsgd
0.906 + 0.2
33.0
4.2
1.0
32.0
Liec
0.866 = 0.2
4.5
2.5
2.3
2.3
Cser
0.744 0.2
34.0
4.3
1.0
33.0
Ncolithic
0.904 - 0.2
4.0
2.2
2.0
2.0
M3
0.697 0.2
15.0
2.2
7.3
7.2
BiCoopFF--GRRAAappive
0.965 + 0.02
0.42
0.079
0.042
0.37
BiCompEL-GRR-AAupiiee--P
0.966 0.02
0.29
0.056
0.029
0.26
BiCompFL-GGR-Fied
0.96 0.03
0.31
0.059
0.031
0.28
BiCompEL-GR-RRccons-Fiiiee
0.949 0.03
0.34
0,063
0.031
0.31
BiComppEL-PR-Fixed
0.966 + 0.02
0.34
0.34
0.031
0.31
BiCompEL-PR-Fiixe-SppiiDL
0.926 + 0.04
0.063
0.063
0.031
0.031
25

---

For 4CNN trained on MNIST, the differences between the proposed approaches become more visible. In the i..d. setting, we can observe that the adaptive block allocations (both Adaptive and Adaptive- Avg) can drastically reduce the average bitrate in BICOMPFI.-GH Partitioning the model in the downlink (BICOMPFL.PPR-HHI PR-F----P--xeedppliii -ixed-SplitLL.) improves the accuracy over bitrate significantly compared to BICOMPFL- CEOOMPFL--R-Fix--- PR-Fixed.
10-
0.925
0.
Mazimum Twe Acruwry
0.854
X
2.6
0.853
X
Foddug
BiCompFL GR Adsptive
X
Festig
BiCompFL: Gd-sstee
0.4
Memagd
Do.blesqueege
BiCompFL GR Fixed
BiCompiL-GR-Adppe GR-Adspitxe-fug
Drublesquezee
Nemopa
BiCompFL-GMMtsppeer Gi---pprreereeeey
BiCompFL GR Fixed
BICompFL Reconst
BiCompFL GR Reconst Ried
BiCompfiL
(iCompFL-PR-nen BiCompFL SpiiDL
0.2
NestRhic
BiCompFL SpiIDL
M3
Communication Cost Bis
Bitrate p.
(a) Test Accuracy Communication
(b) Test Accuracy Bitrate
Figure 5: MNIST 4CNN i.dd
Table 7: MNIST 4CNN i..d.
Method
Acc (mean std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.994 + 0.06
64.0
35.0
32.0
32.0
Doublesqueeze
0.994 = 0.1
2.0
1.1
1.0
1.0
Memsgd
0.994 + 0.08
33.0
4.2
1.0
32.0
Liec
0.993 + 0.07
3.7
2.0
1.8
1.8
Cser
0.993 + 0.06
33.0
4.3
1.0
32.0
Neolithic
0.994 + 0.08
4.0
2.2
2.0
2.0
M3
0.989 0.2
16.0
2.2
8.4
7.4
BiCompEl-GGR-Adppiie
0.996 + 0.0001
0.18
0.034
0.018
0.16
0.995 0.0001
0.15
0.029
0.015
0.14
BiCompFI-GGR-Fixed
0.995 + 0.0002
0.31
0.059
0.031
0.28
BiCompEL-GRR-RconsstF
0.995 0.0001
0.34
0.062
0.031
0.31
BiCompFL-PR-Fixed
0.995 = 0.0002
0.34
0.34
0.031
0.31
BCCompFL-PR-FFiee-Sppttt
0.995 0.0002
0.062
0.062
0.031
0.031
X
x
0.97
X
0.95
BiCompFL Adaptive
Masimmm
Redling
BiConeFL Adaptive
Db.tlesquesee
BaommoFL-G-AAsstiiie Avg
Doublesqueze Memagi
Memsggi
RiCompFl-CR--iuee
BiCompFL-GR--Fee
Liec
BiComiff Lolt-l-eettteett-iiieee
0.93
Lwc
BiCompfL floed
Cser
BiCompFL
Spian-
BiCompFL SpinOl
Commurication Bks
P.P.
(a) Test Accuracy Communication
(b) Test Accuracy Bitrate
Figure 6: MNIST 4CNN non-i.i.d
26

---

In the non-i..d. case of 4CNN on MNIST, the adaptive average allocation strategy provides significant reduction in the bitrate for BICOMPFL-GR. with similar loss in the accuracy as SplitDL for BICOMPFL-PPE In this setting, it is also apparent that the reconstruction in BICOMPFL-GR degrades the performance without gains in the bitrate compared the proposed Algorithm
Table 8: MNIST 4CNN non-i..d.
Method
Acc (mean + std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.983 = 0.1
64.0
35.0
32.0
32.0
Doublesqueeze
0.982 = 0.2
2.0
1.1
1.0
1.0
Memsgd
0.982 = 0.2
33.0
4.2
1.0
32.0
Liec
0,963 + 0.2
4.5
2.5
2.3
2.3
Cser
0.915 0.1
34.0
4.3
1.0
33.0
Neolithic
0,983 = 0.2
4.0
2.2
2.0
2.0
M3
0.929 0.3
15.0
2.2
7.8
7.1
BiCompFL-GRR-Adpiive
0.984 0.009
0.27
0.051
0.026
0.24
BiCoopFE-GRR-AAptive--
0.974 0.02
0.067
0.013
0.0068
0.061
BiCompFL-GR-Fixx
0.985 + 0.008
0.31
0.059
0.031
0.28
BiCompFL-GRR-Rcomee
0.977 0.01
0.34
0.062
0.031
0.31
BiCompFL-PR-Fixee
0.984 + 0.009
0.34
0.34
0.031
0.31
BBCompELPPR-Fixc--ppitLL
0.971 + 0.02
0.062
0.062
0.031
0.031
X
X
0.925
0.920
Feetheg
RCCopptl-ll-Meptiie
0.6
0.935
Doublesquaaee
Memagd
BiCompfL-GR -R Fixed
Maximus Tead
Line
NiCempfi--Cl-Adppppn
BiCampll--n-tice
BiCompFL SpikoL.
Memoga
BiCompfi -GB-BBcooo-----eee
0.500
0.2
Nealthic
Cuer
BiCompli BiCompilL-BB--lee L-R--Fized
BICompFL FR Fleed Sp RDL
0.86
M3
X
10
107
10"
10a
10
10
10-3
10a
16
Communication
Biss
Bitrote
(a) Test Accuracy over Communication
(b) Test Accuracy over Bitrate
Figure 7: Fashion MNIST 4CNN i.i.d.
Table Fashion MNIST 4CNN i.i.d.
Method
Acc (mean std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.927 + 0.07
64.0
35,0
32.0
32.0
Doublesqueeze
0.928 + 0.1
2.0
1.1
1.0
1.0
Memsgd
0.928 + 0.09
33.0
4.2
1.0
32.0
Liec
0.923 + 0.08
4.5
2.5
2.3
2.3
Cser
0.92 0.08
34.0
4.3
1.0
33.0
Ncolithic
0.928 1 0.09
4.0
2.2
2.0
2.0
M3
0.892 + 0.2
16.0
2.2
8.3
7.6
BiCooppF--GRRAAaptive
0.925 + 0.001
0.31
0.059
0.031
0.28
BiCompEL-GR--Ataiiie--Ng
0.927 0.0007
0.31
0.059
0.031
0.28
BiCompFL-GGR-Fixed
0.925 0.0007
0.31
0.059
0.031
0.28
BiCompEL-GR-RRcooss-Fiiee
0.922 + 0.001
0.34
0,062
0.031
0.31
BiCooppEL-PR-Fixed
0.924 + 0.002
0.34
0.34
0.031
0.31
BiCompEL-PR&Fixxe-SppitDL
0.921 + 0.002
0.062
0.062
0.031
0.031
27

---

X
0.86
X
X
Veaime Tea Accunry
0.84
Beattesq.
Momsad
BiCompfi GR Fixed
Liec
FooAwg
BiGompFL GR Adsptive
0.82
Cour
BiCompfL-
Doublescueee
BiCompfL-GRR SAAiifLL-GR-Adgeee -R-bp-itveeee
Neolthic
BiCompfL SpikDL
Merogd
BiCompFL
0.00
Cosr
BiCompfi Fleed
Nealthir
BiCompll Pi-iioo----i
M3
104
107
10"
10a
10
10'
Communication Cost Bik
Bitrate p.
(a) Test Accuracy over Communication
(b) Test Accuracy over Bitrate
Figure 8: Fashion MNIST 4CNN non-i.id.
The results for Fashion MNIST are similar compared to the MNIST case. However, it becomes clear that BICOMPFL-PR can significantly suffer from the unavailability of shared randomness in terms of the achieved accuracy when data is highly heterogeneous.
Table 10: Fashion MNIST 4CNN non-i.i.d
Method
Acc (mean + std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.867 = 0.1
64.0
35.0
32.0
32.0
Doublesqueeze
0.861 = 0.2
2.0
1.1
1.0
Memsgd
0.863 = 0.2
33.0
4.2
1.0
32.0
Liec
0.853 = 0.1
4.5
2.5
2.3
2.3
Cser
0.781 = 0.1
34.0
4.3
1.0
33,0
Neolithic
0.864 = 0.2
4.0
2.2
2.0
2.0
M3
0.782 + 0.2
15.0
2.2
8.0
6.9
BiCompFI-GGR-Adapivv
0,866 + 0.03
0.21
0.04
0.021
0.19
BiComoeI-GRR-Audpiive.Agg
0.853 0.04
0.11
0.021
0.011
0.1
BiCompEL-GRR-ixed
0.868 0.03
0.31
0.059
0.031
0,28
BCCompEL-GR--Rcons-Finnn
0.86 0.02
0.34
0.062
0.031
0.31
BiCompFL-PR-Fixed
0,869 + 0.03
0.34
0.34
0.031
0.31
BCCompEL-PR-FiiedSppittDL
0.831 + 0.03
0.062
0.062
0.031
0.031
For 6CNN trained CIFAR-10, 10, the negative effects of missing global shared randomness and reconstructing in the case of BICOMPFL-GR are prominent. For non-i.i.d. data distributions, the adaptive average allocation shows improvements over the fixed or the average block allocation. Partitioning the model is not viable option in this setting, especially under non-i.i.d. data.
0.800
0.7
0.750
X
0.5
0.25
Maximm Ros
RodRug
BICompEL Adaptive
Festig
BICompFL Adapthve
0.3
Doublesqueeze
Merragd
BKompPL-GGR L.R.-gggtgeeeeeee
AComaFL-cR-Rinn
B.755
Doutesqueeze
Nemsgg
BiCompFL-Ge Asspptive-igg
BiCompFL-cR- sed
Lie
CommPL FR Foed
0.650
Liee
BiCompfi- Cittoppli-CAAteeonitt Codeeoonttiiio
BiCompFIi Aved
0.625
0.1
Commurication Cost
Bitrate p.
(a) Test Accuracy Communication
(b) Test Accuracy Bitrate
Figure 9; CIFAR-10 6CNN i.i.d.
28

---

Table 11: CIFAR-I0 6CNN i.id.
Method
Acc (mean std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.742 0.1
64.0
35.0
32.0
32.0
Doublesqueeze
0.723 = 0.1
2.0
1.1
1.0
1.0
Memsgd
0.727 0.1
33.0
4.2
1.0
32.0
Liec
0.684 + 0.09
4.5
2.5
2.3
2.3
Cser
0.663 + 0.08
34.0
4.3
1.0
33.0
Neolithic
0.73 +
4.0
2.2
2.0
2.0
M3
0.614
16.0
2.2
8.3
7.5
BiCompEL-GR--Aappivee
0.793 + 0.002
0.3
0.057
0.03
0.27
RiCompFE-GRR-AAapiive--e
0.793 + 0.002
0.32
0.061
0.032
0.29
BiCompEL-GR-Fiixd
0.793 + 0.004
0.31
0.059
0.031
0.28
BiCooppF-GGRRReconse-
0.777 0.002
0.34
0.062
0.031
0.31
BiCompFL-PRR-ixce
0.751 + 0.003
0.34
0.34
0.031
0.31
BiCompFL-PR-Fiie Diii-----iidd
0.732 + 0.02
0.062
0.062
0.031
0.031
0.65
0.5
0.55
Redling
BICOmpFL AdaptVe
Maximum
X
Redlig
BiCongFL Adsptive
0.3
Doublesq nece
Doubiesquezze
BiCompFL-GR. Sooompl--RRAdditiivee Aug
Merragd
BiCommall L-ci-fizeed
Memagd
hiCompFi-Gilfiioe
BICompFL Recerst Fixed
Uet
BiCongFL Reconst Fixed
#CompFLPP
BiComgFLPPR
Nealitic
Mealiric
BiCompF Htt
0.1
MS
0.40
M3
10'
10'
10
10
10
10
10
10a
Commurication Cost in Bits
Bitrate P.P.
(a) Test Accuracy over Communication
(b) Test Accuracy over Bitrate
Figure 10: CIFAR-10 6CNN non-i.idd
Table 12: CIFAR-10 6CNN non-i.idd
Method
Ace (mean + std)
bpp
bpp (BC)
Uplink
Downlink
FedAvg
0.599 + 0.1
64.0
35.0
32.0
32.0
Doublesqueeze
0.575 0.1
2.0
1.1
1.0
1.0
Memsgd
0.589 = 0.1
33.0
4.2
1.0
32.0
Liec
0.589 0.2
4.5
2.5
2.3
2.3
Cser
0.419 + 0.09
34.0
4.3
1.0
33.0
Neolithic
0.587 0.1
4.0
2.2
2.0
2.0
M3
0.385 0.1
15.0
2.2
8.3
6.7
HiCompFL-GR-Adapiinn
0.655 + 0.04
0.18
0.034
0.018
0.16
BiComppEL-GR-AAapiive-AA
0.636 + 0.05
0.15
0.028
0.015
0.13
BiCompFL-GRR-ixed
0.665 + 0.03
0.31
0.059
0.031
0.28
BiCooppFLGGR-ReconssFiiin
0.606 + 0.05
0.34
0.062
0.031
0.31
BiCompFL-PR-Fiixe
0.626 + 0.03
0.34
0.34
0.031
0.31
C.CompFL-P--Fiee--ppiini
0.47 + 0.07
0.062
0.062
0.031
0.031
For completeness, we present in Fig. [1 the test accuracies over the number of trained epochs for all scenarios considered above. The setting of interest to this work is that of limited communication cost, and in particular, which performance is achievable given fixed communication budget. Nonetheless, we can find that our proposed methods are not inferior convergence speed over epochs compared to the baselines.
29

---

1.0
2.8
Text Accuncy
0.6
0.6
Fodlig
MNCompFL-GR. Adsptive
Festig
BiCompfL GR Maptive
04
Memoge
Douélescueeee
0.4
Memagii
Dounlesgjeere
BCCampit-Gn--ood
BiCompFL Reconst
Lbc
BiCompFL Reconst Flees
Cser
BiCompFl Flied
Cser
BiComptL-PR Ficed
0.2
Neolthir i
BiCompll sppl
0.2
Nealthir
M3
M3
100
10-
107
107
10
10P
Epocts
Epochs
(a) MNIST LeNet i.i.d.
(b) MNIST LeNet non-i.idd
0.5
0.8
Taxt Accuncy
0.6
0.5
Foddig
BIGompFL Adsptive
Festirg
BiCompfL Adsetive
0.4
Beislesqueecee
Momsga
BiCompFL Fixe
Momagd
BiCompFL
Liec
BICompFL Reconst Fired
Liec
BiCompFL GR Reconot Flves
Coer
BiCompilLPP LPR Fleed
Cser
DiComptL-PR -Reeee BiCompFL PR Ries SpIRDL
0.2
Nealthic
BiCompFL PR Floed Sp RDL
Ncolthic
M3
10P
10
10P
10P
10
Epocho
Epochs
(c) MNIST 4CNN i.i.d.
(d) MNIST 4CNN non-i.i.d.
0.8
Test-ccouar
0.4
Feddig
Deublessueeee
BiCompFL-C Adaptive
BiCompll-GB-Ade -CB-Adppiiiieee
Test
Dniblesspeecee
Festwg
BiCompFL-GR -RrpppLL-----ggieee Adggtive
DCComptt-a--Mtettieeee
Hcmsgg
BiCompFL Fixed
Memsad
BiCompFL GR Fixed
Liec
BiCompFL LR Reconst Fired
Liec
BiCompfL GR Reconot Flees niCamptL-PR -----uudd
BiCompiiL.PR Par-ppl------eeee Fi--f
Cser
Nholthhic
BiCompFL Foce IDL
Nealthic
BiCompFL PR Roes SpiRoL
M3
10.
100
101
10
Epocts
Epecto
(e) Fashion MNIST 4CNN i.idd
(1 Fashion MNIST 4CNN non-i.i.d.
0.8
0.7
0.6
0.5
Accurryy
0.5
0.4
Fedhug
BiCompFL-GR Adsptive
Festkg
BiComprL--GR Adspbive
Doublsouesze
BiCompFL-Gk Adaptive Avg
0.3
Doutlesqueeze
BicompFL AtteooooL----dagiiiieeeeeo Avg
0.3
Homoga
BiCompFL GR Fixed
Momagd
BiCompfL GR Fived
BiCompfL G- Reconst Fleed
Lec
BiCompFL-GGR Reconst flves
0.2
BiCompfi
D.CompfI-
Nestihic
BiCompFL Facco SpinDL
Neolthic
BiCompFL PR Rios SpiRoL
0.1
MS
0.1
M3
10'
Epocts
Epects
(g) Cifar-10 10 6CNN i.i.d.
(h) Cifar- 10 6CNN non-i.i.d.
Figure 11: Test Accuracy over Epochs
30

---

I Ablation Studies
L11 Number of Clients
We study in what follows the sensitivity various hyperparameters of our algorithms. For compara- bility, we conduct all experiments on the model 4CNN, Fashion MNIST, and i... We plot for all experiments the accuracies over the number of epochs, and over the communication cost in bits. We first evaluate in Fig. 12 the effectiveness of BICOMPFL-PP and BICOMPFL-PR for different numbers of clients. can be found that both algorithms exhibit satisfying performance even for = 50, given that the same data is now distributed on more clients. The overall communication cost increases by roughly the factor the increase in the number of n. To illustrate this further, we additionally plot in Fig. 13 the bitrates per parameter.
0.7
0.7
0.6
0.6
0.5
0.5
Teat
0.4
0.4
0.3
BiCompFL-GB-CI
0.3
LiCumpFi Glo-11
BCompFL-GR.CS
BiCompFi c50
BiCampit--h--11
LiCompFl-P8--1
BiCompFLLitt-CS0
BiCumpf PM-c50
10'
16
10a
10'
10'
10'
10*
Epochs
Communication Cost in Bits
(a) Test Accuracy over Epochs
(b) Test Accuracy over Communication Cost
Figure 12: BICOMPFL-GR and BICOMPFL-GR With Different Number of Clients
0.925
Bicompft-Givcce BicompEL-GR cso RiCompfL-m--C1
0.915
0.910
0.800
16
10*
Bitrate p.
Figure 13: Bitrates for BICOMPFL-GR and BICOMPFL-GR With Different Number of Clients
1.2 Optimization of the Prior
As described in the main body of the paper. BICOMPFL-PR allows for optimizing the choice of the prior the clients by optimizing the convexity parameter À that mixes the global model estimate with the posterior transmitted by the client an iteration ahead, i.c., pt,, 10,, (1 \)oy to reduce the communication cost. To evaluate the potential of this method, we optimize À so that minimized the KL-divergence between the current posterior of (to be transmitted) and the prior pf.... representative for the uplink communication cost. The KL-minimizing À is transmitted to the federator, which is necessary for the federator to reconstruct the importance samples. This optimization is conducted at each iteration individually at the clients. We present in Fig. 14 the performance of this method compared with the algorithms that use as priors exclusively the global model estimates of the clients. Note that optimizing the prior individually at the clients is only possible for BICOMPFL-PRY plot
31

---

the performance of BICOMPFL-GR for reference only. To assess the potential, we ignore for the moment the cost of transmitting X, which could be reduced by further compression techniques and leveraging the inter-round dependencies of the choice of X.
0.9
0.9
0.5
Teat
0.4
0.4
0.3
0.3
BiCompFL-GR
BiCompFL-GR
02
BiCampFl-PR
0.2
107
10a
10a
20a
20
10'
10*
Epochs
(a) Test Accuracy over Epochs
(b) Test Accuracy over Communication Cost
Figure 14: BICOMPFL-PR With and Without Optimization over the Prior. Optimization over the Priors is denoted by OP.
can be found that, while optimizing the prior improves the accuracy over epochs and with respect to the communication cost compared BICoMPFL-PR the improvements are rather insignificant. We therefore present for clarity the algorithm with a fixed choice of the prior as the former global model estimate, which additionally reduce the computation overhead at the clients by avoiding the optimization over X. Nonetheless, we note that in certain edge cases, there can be merit in the optimization approach, for instance when the number rip of samples the downlink is very small, and hence the global model estimate is inaccurate.
I.3 Number of Samples
We continue assess the impact of the number 7pr of samples on the downlink. We therefore evaluate the performance of BICOMPFL-PE for rpI.. E (5, 10, 20). We evaluate the differences on BICOMPFL-PRTTI results in Fig. reflect the obvious: the larger ripr, the better the accuracy when plotted over the number of epochs. On the contrary, the larger rIDL,, the larger the communication cost per epoch. The final accuracies do not show substantial differences, and hence, NDL = 5 is sufficient in this setting. To avoid assessing methods, we choose nDD = 10 in all our experiments, noting that the communication can further be reduced in certain scenarios by lowering TDL without notable performance loss.
method overly optimistic and provide fair comparison to other
Test ArcunnC
0.6
0.6
0.4
0.4
BiCampfl- -R-011-4111
BiCampELLP.DLLS.L.
BlComoFL HR.DL-UL.
Epochs
Commuricalion Cosl in
(a) Test Accuracy over Epochs
(b) Test Accuracy over Communication Cost
Figure 15: BICOMPFL.PPR for Different Number of Downlink Samples and Single Uplink Sample.
32

---

1.4 Block Size
We compare in Fig. 16 the performance of BICOMPFL-GR for different block sizes BS d/B {128, 256, 512). As expected, fixing Tis, larger block sizes worsen the performance of the algorithm when evaluated over the number of epochs. However, larger block sizes simultaneously reduce the communication cost, and can hence be beneficial in many scenarios. However, we also note that larger block sizes comes at the expense of increases sampling complexities, and hence, the maximum block sizes are also dominated by the resources of the clients and the federator.
Accurayy
Teat Accurecy
0.6
0.6
BiCompFL-GR 85256
BiComp GR 85256
0.
3KompFL GR 85128
0.2
BicompFL GR 65128
3kamptL-GB-855522
10'
16a
10a
10a
30*
10'
12P
Epochs
Communic Cost
(a) Test Accuracy over Epochs
(b) Test Accuracy over Communication Cost
Figure 16: BICOMPFL-GR With Fixed Block Allocation for Varying Block Sizes (BS) d/ B.
1.5 Number of Importance Samples
In Fig. 17 we study the sensitivity of our algorithms with respect to the number importance samples Mys the example of BICOMPFL-GR. While larger number of nıs slightly improves the performance as of the epoch number, the improvements do not outweigh the additional communication costs. Overall, our algorithm proves rather stable within reasonable ranges for mss- We fix in all our experiments vys 256, presenting good trade-off between performance and efficiency.
0.8
0.8
Accurnoy
Recurery
0.6
0.6
BrGmmf
BCcomfL-Gn-5255
02
BiCompFL-GR-S 51-28
0.2
BiCampFL-GR-51
BiCompli PioomppL--G-511222
10'
10
10
10'
10'
20
Epechs
Communication Cost in Bits
(a) Test Accuracy over Epochs
(b) Test Accuracy over Communication Cost
Figure 17: BICOMPFL-GR with Varying Number of Importance Samples ryss per Block.
I.6
Learning Rate
Our main claims are centered around the per-client bitrates per parameter, rendering the choices of learning rate secondary to our reasoning. Nonetheless, we tune the learning rates of all methods so that the baselines and BICOMPFI. achieve roughly the same final accuracies, allowing a fair comparison of resulting communication costs. We analyze the impact of the learning rate choice
33

---

0.6
0.4
#iCompFL-Ge GR. g= 0.1
#CompFi-GG,
SiComsFL-GR, 02 BlCompFL
100
Figure 18: BICOMPFL-GR with Varying Number of Importance Samples nys per Block.
on BICOMPFL in Fig. 18 for E {0.01, 0.05, 0.2, It is particularly noteworthy that most learning rates we study, which attribute
BICOMPFL exhibits stable performance the regularization effects that occur in stochastic FL, detailed in the main body of the paper. Only for 7 = 0.01, the final performance is decreased, indicating that BICOMPFL
able escape local optima in this setting. Although 0.05 provides the best communication efficiency, we choose moderate learning rate of 0.1 not overestimate our method compared other approaches.
34

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All claims made in the abstract and the introduction are later justified in detail by the introduction of the schemes in Section 4 together with additional details in Appendices C and D the various experiments in Section 4 and Appendix HH the formal theoretical guarantees (cf. Section 5). and the proofs provided in Appendix A
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The theoretical and experimental limitations are discussed in the paper. For instance, we discuss in Sections |and how the availability of shared randomness affects our protocols. Further, we diseuss in Section 5 the assumptions required for main results to hold. We provide in Appendix [ details on the impact of various hyperparameters. including the limitations of optimizing the prior distributions, and how the block size is affected by the computational resources. The space constraints didn't 't allow for adding separate section.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover
35

---

limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are provided along with the statements of our theoretical results. We provide the full set of proofs in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides all experimental details to reproduce the results in Sec- tions 3 and | and the corresponding Appendices D to The code to reproduce our ex- periments is included in the supplementary material. This includes parameter files for the various experiments and recommended environments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
36

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The source code required to reproduce our experiments is included in the sup- plementary material, including parameter files for the various experiments and instructions on how to use the provided code and how to prepare the publicly available datasets. We also provide the Python environment used to generate the results. Guidelines:
Justification:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nips.cem cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips /niips.cc/pulll.. wttpb////g..eee//oeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper specifies all training and test details necessary to understand the results in Section and Appendix
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
37

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each experiment is repeated for multiple provide statistical confidence. The depicted results are averaged performance measures, consistently shown together with their respective standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The exact details of our simulation clusters are provided in Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'ttntt make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::////aarrpp.cccp//////hhhee
Answer: [Yes]
Justification: The paper conforms with the NeurIPS Code of Ethics. The research does not involve any human subjects or participants, and all datasets used are standard and available to the public.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
38

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents a method for bi-directional compression in stochastic (Bayesian) FL, with the aim of reducing communication overhead and improving scalability. The contribution methodological and does not target a specific downstream application. While the work does not explicitly discuss societal impacts, such efficiency improvements may enable more practical and energy-efficient deployment of machine learning systems in cnoource-constrannce environments, such edge devices, indirectly support privacy- preserving machine learning in settings such as medical healthcare. We do not foresee any immediate negative societal impacts arising from the work presented.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
39

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets properly credited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Alongside the paper, we introduce new source code that we provide as supple- mentary material. This material is well documented with particular instructions on how to use the provided framework.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:////eeripppscc/oooeeeeeeeee for what should or should be described.
41


Reviews:
[{"review_id": "BXjc4WwgK2", "reviewer": "Reviewer_EDwN", "summary": "The paper proposes BICOMPFL, a family of federated learning (FL) algorithms that cut both uplink and downlink communication by exploiting the stochastic/Bayesian view of model parameters and using Minimal Random Coding (MRC) with shared priors (“side information”) to remotely generate samples at the receiver instead of transmitting full local samples or model updates. The paper also provides experimental results to show the approach cuts total communication by about 5–32× while maintaining competitive accuracy.", "questions": "Questions:\n1. Can you provide evaluations results on large-scale benchmarks and models?\n2. On-device, how does BICOMPFL’s runtime and energy compare to vanilla FedAvg?\n3. When client data are extremely heterogeneous, does the KL gap explode for some clients?\n4. Could BICOMPFL be stacked with other sparsification or structured pruning-based compression methods?", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper aims to reduce the communication overhead in FL training and proposes a probabilistic compression method, which is novel to the my knowledge.\n2. The paper also provides the convergence rate to show the proposed algorithm leads to similar convergence rate as FedAvg.\n3. The paper includes some empirical evaluations and ablation studies to show the effectiveness of the algorithm.\n\nWeakness:\n1. Empirical validation limited to small vision benchmarks.\n2. Minimal Random Coding needs n_{IS} samples which scales exponentially w.r.t. the KL divergence between the prior and posterior, which makes the high-dimensional models hard to reach the condition.\n3. The most communication-efficient variant (BICOMPFL-GR) assumes all clients share global pseudo-randomness, but this shared randomness is hard to maintain in real world FL systems, establishing and maintaining perfectly synchronized randomness can be difficult. (But this is a minor point compared to 1 and 2)", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EDwN for their thorough review and helpful suggestions. We address their concerns as follows.\n\n- **W1/Q1** (Empirical validation limited to small vision benchmarks): Our work proposes a novel method within the scope of academic research. Given standard resources, we conducted experiments with complexities that are comparable to all related works in this line of research. Moreover, through our extensive numerical and theoretical results, we did not identify any particular negative effects as the task complexities scale; on the contrary, the positive effects of substantially reduced communication complexities gain increasing importance with larger models. % Nonetheless, we are currently running larger-scale experiments for follow-up works.\n\n- **W2** (MRC sampling complexity scales exponentially in the KL-divergence): As the theory suggests, the sampling complexity of MRC scales exponentially in the KL-divergence between prior and posterior. To overcome this limitation, we employ the method of block allocation. Therefore, we fix a per-block target KL-divergence and partition the model parameters into blocks to match the desired target divergence. We further highlight that the KL-divergence is not static and decreases as the training progresses, often leading to negligible divergences when advancing in the training. As a result, higher-dimensional models actually support meeting the KL-conditions, as larger dimensionality increases the degrees of freedom for block partitioning. With regard to the scaling properties, partitioning the models into blocks of equal KL-divergences leads to a linear dependence of the sampling complexity on the model size, alleviating concerns regarding exploding sampling costs. \n\n- **W3** (Shared randomness difficult to maintain in real-world FL): We clearly articulate the assumption that common randomness is required, and specifically propose a variant for cases where global randomness among all clients might be difficult to maintain. Nonetheless, the fact that our methods are fully synchronous facilitate easy piggy-backing of reasonably large seeds onto global model updates sent to the clients. This relative cost of this overhead diminishes, and becomes negligible, as the model size increases. In our experiments, the one-time transmission of few-bit seeds is sufficient to achieve state-of-the-art performance with substantially reduced communication costs.\n\n- **Q2** (Runtime and energy costs): Similar to other papers in this line of research, we propose a conceptual framework that uses standard models and building blocks, such as sampling from parametric distributions. Many works are particularly concerned with resource-efficient implementations of such generic building blocks, e.g., the sampling operations. To complement our experiments, we theoretically analyze the fundamental sampling complexity of our compression methods. However, naturally, implementing and deploying any method  on hardware in practice requires additional engineering efforts to make it efficient. Thus, measuring the energy or runtime of our methods on our servers would not yield meaningful comparisons, unless tailoring each method individually to the specific hardware at hand. We, hence, refrain from showing such plots, noting that such statements would be orthogonal to the message of our paper. \n\n- **Q3** (Exploding KL gap for extreme heterogeneity): This is a very good point brought up by the reviewer, which we carefully investigated (although not discussed much in the paper). Firstly, we do conduct our experiments in regimes considered extremely heterogeneous in the literature (Dirichlet distribution with $\\alpha=0.1$), e.g., [R1], and did not observe exploding KL gaps in such settings. We initially assumed such an effect would be present and studied whether it can actually be leveraged for improved system design. Assuming the clients' posteriors are moving substantially away from the global model when data is very heterogeneous, communication costs with private randomness would benefit substantially from using the previous posterior of the clients as prior instead of the previous global model. However, we couldn't observe notable positive effects from such individual prior selection. The key reason is the implicit regularization of our optimization methods. Specifically, the mirror descent approach studied in our paper naturally regularizes the posterior optimization w.r.t. the KL-divergence between prior and posterior, avoiding KL blow-ups in the optimization. This positively affects both the variance in local optimization in the clients and the incurred communication costs.\n\n- **Q4** (Compatibility with sparsification or structured pruning-based compression): Yes, our methods are entirely compatible with other compression methods that sparsify or prune the models/gradients before transmission. Applied prior to our compression methods, sparsified models can be readily fed into our compression methods without further adaptations. The pruned parameters/distribution are simply not included in the block selection of model parameters, thereby further reducing the incurred communication costs. We are happy to add a respective remark in the paper.\n\n[R1] Y. Allouah, S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan, “Fixing by\nmixing: A recipe for optimal Byzantine ml under heterogeneity,” in International Conference\non Artificial Intelligence and Statistics, 2023, pp. 1232–1300."}, {"review_id": "QRGqOM64pL", "reviewer": "Reviewer_oHgr", "summary": "This paper addresses the significant communication overhead in both uplink (client-to-server) and downlink (server-to-client) directions for ​**stochastic (Bayesian) Federated Learning (FL)​**, where probabilistic models are trained to capture parameter uncertainty. Unlike prior work focused on deterministic FL or unidirectional compression, the authors propose ​**BICOMPFL**, a novel framework leveraging ​**remote source generation**​ via ​**Minimal Random Coding (MRC)​**​ to enable bidirectional communication efficiency. \n\nThe core insight exploits shared randomness and ​**side information**​ (in the form of prior distributions) to allow clients and the server to sample from each other's local/global posteriors rather than transmitting explicit updates. This transforms communication into a sampling problem, where transmission costs scale with the KL-divergence between updated distributions and their priors. Two algorithm variants are introduced: ​**BICOMPFL-GR**​ (using global shared randomness) enables efficient relay-based compression by synchronizing priors across all parties, while ​**BICOMPFL-PR**​ (using private randomness per client-server pair) handles settings without globally synchronized randomness, albeit with slightly higher overhead. \n\nKey contributions include:  \n1) ​**Inherent communication regularization**​ through a mirror-descent optimization structure, treating communication costs as part of the learning objective.  \n2) ​**Substantial empirical gains**: Experiments on MNIST, Fashion MNIST, and CIFAR-10 demonstrate ​**5–32× reduction in total communication costs**​ (bits per parameter) while maintaining competitive accuracy against strong baselines (e.g., FedAvg, Doublesqueeze, M3). Adaptive block allocation strategies further optimize costs.  \n[](@replace=1)\n  \n3) ​**Theoretical innovations**: Refined bounds for MRC compression (for Bernoulli distributions) quantify uplink/downlink trade-offs, and a new contraction property is proven for biased MRC operators, enabling convergence guarantees in conventional FL settings.  \n4) ​**Methodological generality**: Extension to conventional FL via stochastic quantization (e.g., SignSGD) shows broader applicability without error feedback mechanisms.  \n\nThe work bridges stochastic FL with practical communication constraints, demonstrating that principled probabilistic modeling and remote sampling can dramatically reduce bidirectional bandwidth while preserving model performance.", "questions": "​**1. Scalability Validation to Modern Architectures**​  \nThe experimental validation relies exclusively on small CNNs (LeNet, 4-6 layer CNNs). To strengthen claims of broader applicability, please evaluate BICOMPFL on larger models (e.g., ResNet-18/50, ViT-Tiny) and complex datasets (e.g., ImageNet subset or CIFAR-100). Specifically, quantify how communication costs scale with model size and heterogeneity under your framework. \n\n​**2. Generalization of Theoretical Guarantees**​  \nThe theoretical analysis (Section 5) focuses entirely on Bernoulli distributions for posterior/prior modeling. Clarify whether Lemma 1’s contraction property and Theorem 1’s communication bounds extend to other distributions common in Bayesian FL (e.g., Gaussian posteriors). Provide either a proof sketch for multivariate Gaussians or empirical validation via Gaussian-based stochastic FL experiments. Should these extensions hold, emphasize their implications in the text; if not, explicitly state the limitations of the current theory and propose mitigation strategies (e.g., parameter transformation techniques).\n\n​**3. Algorithm Workflow Clarity**​  \nThe distinction between BICOMPFL-GR and BICOMPFL-PR is not intuitively conveyed in the pseudocode (Algorithms 1-2). Consider adding a schematic diagram illustrating the bidirectional remote sampling workflow, highlighting critical differences in prior synchronization, index relaying, and noise propagation. [](@replace=1)\n demonstrates performance gaps but lacks visual workflow context. A figure showing how shared randomness reduces downlink variance would clarify why BICOMPFL-GR outperforms BICOMPFL-PR.\n\n​**4. Privacy and Fairness Implications**​  \nThe paper does not address how compressed bidirectional updates affect privacy vulnerabilities (e.g., membership inference/gradient inversion attacks) or fairness across heterogeneous clients. Analyze:  \n- How MRC-based sampling compares to DP-SGD in privacy-utility tradeoffs  \n- Whether adaptive block allocation biases aggregation against clients with high KL-divergence  \n- Test accuracy variance across demographic groups in non-i.i.d. settings  \nIncluding a \"Broader Impacts\" section addressing these points would align with NeurIPS ethics guidelines and strengthen societal relevance.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nThe paper demonstrates ​**exceptional technical quality**​ through rigorous theoretical development and comprehensive experimental validation. The introduction of ​**BICOMPFL**​ represents a significant advancement in stochastic federated learning, with novel bidirectional compression mechanisms based on Minimal Random Coding (MRC) and remote source generation. The theoretical contributions are particularly strong, including refined MRC bounds for Bernoulli distributions and a new contraction property for biased compressors (Lemma 1), which enables convergence guarantees in conventional FL settings. Experimentally, the work is thorough, validating claims across three datasets (MNIST, Fashion MNIST, CIFAR-10) and both i.i.d. and non-i.i.d. settings, consistently demonstrating ​**5–32× communication reduction**​ without accuracy degradation. The adaptive block allocation strategies and extension to conventional FL showcase methodological versatility. The significance is high, addressing a critical bottleneck in FL deployment—communication overhead—through a principled probabilistic framework that inherently treats communication as an optimization objective via mirror descent. The approach is highly original, being the first to solve bidirectional compression in stochastic FL through remote sampling and prior-based side information. [](@replace=1)\n visually encapsulates this breakthrough by showing order-of-magnitude bitrate improvements over baselines.\n\n### Weaknesses\nWhile technically solid, the paper has ​**clarity issues**​ in theoretical sections where derivations become overly dense (e.g., Lemma 2 proof in Appendix A). The distinction between BICOMPFL-GR and BICOMPFL-PR could be better motivated visually—a conceptual diagram would help readers grasp the remote generation workflow. Experimentally, the ​**quality is slightly compromised by limited scope**: Evaluations use only small CNNs rather than modern architectures (e.g., ResNets), and comparisons exclude recent Bayesian FL methods like FedBE. The significance claims about scalability lack validation beyond 50 clients, and energy/runtime metrics—critical for edge deployment—are absent. The theoretical ​**originality is constrained to Bernoulli distributions**, with no discussion of extending MRC analysis to Gaussian posteriors common in Bayesian FL. The ​**significance of broader impacts**​ is underdeveloped: While reduced communication benefits sustainability, potential privacy risks from compressed updates (e.g., membership inference) aren't addressed. [](@replace=2)\n reveals another weakness: convergence instability in BICOMPFL-PR under high heterogeneity, which isn't sufficiently analyzed. Finally, adaptive block allocation's overhead isn't quantified, leaving practical trade-offs unclear.", "was_revised": true, "final_justification": "Most of our concerns are resolved during the rebuttal and discussion phase. The paper's core innovations are now rigorously supported and accessible. With these enhancements implemented in the camera-ready version, we will increase our score.", "rebuttal": "We thank Reviewer oHgr for their comments. We address the reviewer's concerns as follows.\n\n- **W1** (Clarity issues): We are unsure which parts are referred to as overly dense. If the reviewer has particular pointers, we are happy to modify the passages accordingly.\n\n- **W2/Q3** (Algorithm workflow clarity): We are happy to add such a conceptual diagram to the paper highlighting the differences between the two proposed methods, which lie mostly in the way shared randomness is leveraged.\n\n- **W3/Q1** (Scalability validation to model architectures): \nOur work proposes a novel method within the scope of academic research. Given standard resources, we conducted experiments with complexities that are comparable to all related works in this line of research.Given the numerical and theoretical evidence, we could not identify any particular negative effects as the task complexities scale; on the contrary, the positive effects of substantially reduced communication complexity gain increasing importance.\nThe communication costs are quantified both in theory and experiments for varying model sizes and levels of heterogeneity. We kindly refer the reviewer to the respective sections and various experiments, also in the appendix.\n\n- **W4** (Runtime and energy costs): Like other papers in this line of work, we propose a conceptual framework that uses standard models and building blocks, such as sampling from parametric distributions. Many works are particularly concerned with resource-efficient implementations of such generic building blocks, e.g., the sampling operations. To complement our experiments, we theoretically analyze the fundamental sampling complexity of our compression methods. However, implementing and deploying any method on hardware in practice requires additional engineering efforts to make it efficient. Thus, measuring the energy or runtime of our methods on our servers would not yield meaningful comparisons, unless tailoring each method individually to the specific hardware at hand. We, hence, refrain from showing such plots, noting that the results would be orthogonal to the message of our paper.\nOur experiments, which include 50 clientsm align well with the settings considered in the literature. Importantly, we substantially benefit from increased number of clients on the uplink through positive averaging effects of the noise introduced in the sampling operations. Hence, the averaged posterior estimates observed by the federator improve as the number of clients increases. We thus believe that our ablation studies provide sufficient evidence of the positive scaling properties of our method.\n\n- **W5/Q2** (Generalization of theoretical guarantees): We deliberately choose to conduct a tailored analysis for Bernoulli distributions. While it is possible to run a more general analysis for other classes of distributions (as detailed in the sequel), we found that the specific optimization method proposed in the paper is particularly communication-efficient and allows for inherent communication-regularized training, a unique property in this line of work that renders our methods substantially more communication-efficient while preserving state-of-the-art performance. \n\nNonetheless, our analysis can indeed be extended to other parametric distributions, such as multi-variate Gaussians. The key ingredient is to replace the bias bound in Lemma 2 with the generic results for MRC known from [R1]. Although weaker than our result, their result holds for all classes of distributions. Having such an upper bound on the bias, one can follow our proof strategy to prove the communication costs in Theorem 1, i.e., using the convexity of KL divergence and decomposing the error in parameter estimation into a bias term (where the result from [R1] generically applies) and employing concentration bounds, such as the Hoeffding inequality for subGaussian random variables, to bound the sampling error. The remaining steps follow analogously to our proof. As for the contraction property in Lemma 1 used to establish convergence guarantees for the case of conventional FL, similar adaptations can be made. However, we note that for the composition of stochastic quantization with MRC to turn conventional FL into a stochastic setting, the incentive to use distributions other than Bernoulli might be weaker than for the natively stochastic settings mentioned by the reviewer. We are happy to include a discussion related to this in the paper.\n\n- **W6/Q4** (Privacy and fairness implications): Although privacy and fairness are issues  widely studied in the literature, this work does not primarily focus on those aspects. For instance, privacy in FL with relative entropy coding was considered in [R2]. On the other hand, as a general statement, compression methods positively affect the clients' privacy in FL, which mainly stems from the fundamental data processing inequality [R3]. In our case, the noise introduced throughout the sampling operations used for compression dilutes the individual contributions of the clients observed by the federator, thus naturally enhancing the clients' privacy and reducing the risk of privacy breaches through, e.g., membership inference attacks. Concerning fairness, we note that adaptive block allocation does not have implications on potential bias w.r.t. particular clients in the aggregation process. The same amount of samples are observed for each client, ensuring balance in the aggregation process. Further testing the particularities of variance with regard to demographic groups is beyond the scope of this work. However, we are happy to add a respective discussion in our paper.\n\n- **W7** (Convergence instability under high heterogeneity): We kindly point the reviewer to our extensive experiments in the appendix, where we examine the stability of our methods with high data heterogeneity modeled by a Dirichlet with parameter $\\alpha=0.1$, which is considered an extremely heterogeneous regime [R4].\n\n- **W8** (Overhead of adaptive block allocation): All our experiments consider the quantitative cost of adaptive block allocation incurred to transmit the block locations. We find that the overhead for transmitting block indices is negligible compared to the positive effects of block size adaptation on the communication cost. As the training progresses, adaptive partitioning successively increases the block sizes, thus drastically reducing the communication costs over the runtime of the algorithm.\n\n[R1] S. Chatterjee and P. Diaconis, “The sample size required in importance sampling,” The\nAnnals of Applied Probability, vol. 28, no. 2, pp. 1099–1135, 2018.\n\n[R2] A. Triastcyn, M. Reisser, and C. Louizos, DP-REC: Private and communication-efficient federated learning, 2022.\n\n[R3] T. Cover and J. A. Thomas, Elements of information theory. Wiley-Interscience, 2006.\n\n[R4] Y. Allouah, S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan, “Fixing by\nmixing: A recipe for optimal Byzantine ML under heterogeneity,” in International Conference\non Artificial Intelligence and Statistics, 2023, pp. 1232–1300."}, {"review_id": "ArTBXTmDpb", "reviewer": "Reviewer_GJUK", "summary": "The authors proposed a bidirectional compression method for a wild federated learning setting in which uplink and downlink communications are both constrained. The underlying motivation is connected to the Bayesian federated learning mechanism, which employs an alternative view that the masks of weights are learned.", "questions": "Please refer to the *weaknesses* subsection above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n---\n*  The paper is well-written and the motivation and algorithm design are both convincing. There are tons of additional analyses that thankfully address most of my preliminary concerns raised during reading the manuscript.\n* The consideration of uplink bottlenecks and global/private randomness, both of which are overlooked in academic papers, is impressive.\n* The distributional convergence of a global model is sound and the analysis of communication cost related to the divergence is also intriguing.\n* The empirical validation is conducted extensively on both IID and non-IID scenarios using widely-used benchmark datasets with moderate number of clients.\n\n### Weaknesses\n---\n* Based on Algorithm 1, the proposed method seems not support partial client pariticpation, which is often required in low-bandwith settings.\n* To better illustrate the practical strength and justification of the small number of clients in the experiments, the authors could narrow down the target federation setting to a cross-silo federated learning setting (see Table 1 of Kairouz et al., 2018), or specify/exemplify plausible scenarios in which the proposed methods are most applicable.\n* (minor question) Is there any (theoretical) tips to set the number of quantization intervals, $s$?\n\n- (Kairouz et al., 2018) Advances and Open Problems in Federated Learning", "was_revised": true, "final_justification": "The authors clearly addressed concerns I raised and no further clarification is required.", "rebuttal": "We thank Reviewer GJUK for their positive evaluation of our work. We address their concerns as follows.\n\n- **W1** (Partial client participation) Although we do not consider partial client participation in the paper, our methods are compatible with such extensions. For BiCompFL-PR, partial client participation can readily be implemented on top of the methods we propose, as they do not rely on full prior synchronization among the clients. BiCompFL-GR on the other hand requires that all clients' priors are synchronized at each iteration. To allow partial client participation, the federator is required to transmit the previous global model to the clients that were absent in the previous iteration. This ensures full synchronization of the global model estimate used for communication-efficient uplink communication. Hence, while partial client participation is supported by BiCompFL-PR without additional communication overhead, BiCompFL-GR's compatibility with partial client participation comes with a potentially increased downlink communication cost for clients that were previously passive. If the reviewer deems it helpful, we will be happy to add a discussion on this in the paper.\n- **W2** (Practical strengths and justification of small number of clients): The number of clients is not an inherent limitation in our methods. While we experimentally examined our methods with at most 50 clients, our methods exhibit favorable scaling properties in the number of clients. The more clients, the better the averaging effects of the noise introduced by the sampling operations for uplink compression, effectively reducing the error made in estimating the clients' average posterior. We selected the number of clients used in the experiments in alignment with common studies in the literature and our available resources.\n- **W3** (Practical tips for the choice of quantization intervals): In general, the choice of the number of quantization intervals depends on the model architecture and dataset complexities, affecting the number of iterations required until convergence. The simpler the learning task, the fewer quantization intervals are sufficient for convergence. In addition to generic arguments, we particularly want to highlight that the number of quantization levels should be carefully selected respecting the expected variance of the gradients. If gradients exhibit inherently large variance, e.g., through small mini-batch sizes, large variance of the local datasets, or substantially over-parameterized networks, small number of quantization intervals might be particularly efficient, and increasing that number would only negligibly improve the performance. We also want to remark that one-bit quantization was often shown to be trivially effective [R1, R2]. We hope these explanations provide a glimpse on the problem of quantization interval selection. \n- **Formatting**: We thank the reviewer for this comment and will unify the citation style accordingly.\n\n[R1] F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” in Interspeech, 2014, pp. 1058–1062.\n\n[R2] S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi, “Error feedback fixes SignSGD and other gradient compression schemes,” in International Conference on Machine Learning, vol. 97, 2019, pp. 3252–3261."}, {"review_id": "rGyzbwdouK", "reviewer": "Reviewer_Bbin", "summary": "The paper considers stochastic federated learning to reduce communication costs in both uplink and downlink communication. The techniques are based on Minicam random coding, an existing remote source generation tool. Both theoretical guarantees and empirical validations are provided.", "questions": "I have a question about the choice of the remote source generation scheme. The authors used Minicam random coding, which is non-exact: the output distribution is distorted in the sense that the remotely generated sample only follows $\\hat{Q}$, rather than $Q$ exactly.I believe the authors are aware of this (as noted in lines 96–97). My question is whether the authors have considered exact schemes, such as the Poisson functional representation by Li and El Gamal (2018) or dithered-quantization-based schemes. It is fine if these were not considered, I'm just curious whether exactness might make a difference in FL tasks, for example in terms of accuracy.", "limitations": "There are no new limitations from my side. The authors have already discussed some limitations, e.g.,  the availability of shared randomness and some required assumptions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "** Strengths**:\n\n- The paper is well written and well structured.\n- The contribution is clear, and the literature review is thorough.\n- The work makes a solid contribution, both from theoretical and experimental perspectives.\n\n** Weaknesses**:\n\nThere are no obvious major weaknesses. I would suggest the author consider also trying other remote source generation schemes to see if any of them achieve better results in FL tasks.", "was_revised": true, "final_justification": "I will keep my score.", "rebuttal": "We thank Reviewer Bbin for their positive evaluation of our work and for their thoughtful comments. We answer their question as follows.\n\n- **Q1/W1** (Other remote generation schemes) The choice of MRC is motivated mainly by practicality and ease of exposition. A natural improvement is ordered random coding (ORC) [R1], which reduces the entropy of the indices selected for transmission using the Gumbel-Max trick. This extension can directly be applied in our method with a minor adaption that we refrained from including here for clarity. On a related note, the authors of ORC [R1] further expose an interesting connection to the Possion functional representation mentioned by the reviewer. While such proposals allow exact remote generation, lossless sampling schemes usually incur substantially larger communication costs and sampling complexities, often rendering those methods impractical. We believe that the overhead required for lossless compression compared to the lossy variants used in our methods are insufficiently motivated in a federated learning context, since the variance of the training process likely outweighs the negative effects of the noise of the sampling process conducted for compression. If the reviewer deems it useful, we are happy to add a respective discussion in the paper.\n\n[R1] L. Theis and N. Yosri, “Algorithms for the communication of samples,” in International Conference on Machine Learning, 2022, pp. 21 308–21 328."}]

Author rebuttal:

Meta-review: The paper investigates the stochastic federated learning problem, with the aim of reducing the communication costs in both uplink and downlink. The main technical approach is minimal random coding, an existing remote source generation tool. Both theoretical guarantees and empirical validations are provided. Overall, this paper is well-written. The main concerns raised by the reviewers are: (i) the numerical experiments are limited to small-scale vision tasks and the extensions to large-scale models seem non-trivial; (ii) the proposed compressed updates may incur privacy risks in practice. Nevertheless, the reviewers agree that this paper has its own merits and the contributions are sufficient. Therefore, I recommend Accept.

The authors are encouraged to incorporate the reviewers’ comments and polish the presentation during preparing the final version.
Decision: Accept (poster)