Title: Bi-Directional Communication-Efficient Stochastic FL via Remote Source Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Communication-efficiency

Importance sampling

Minimal Random Coding

Stochastic federated learning
Abstract: Federated Learning (FL) incurs high communication costs in both uplink and downlink. The literature largely focuses on lossy compression of model updates in deterministic FL. In contrast, stochastic (Bayesian) FL considers distributions over parameters, enabling uncertainty quantification, better generalization, and, crucially, inherent communication-regularized training through a mirror-descent structure. 
In this paper, we consider both uplink and downlink communication in stochastic FL, and propose a communication framework based on remote source generation. Employing Minimal Random Coding (MRC) for remote generation, we allow the server and the clients to sample from local and global posteriors (sources), respectively, rather than transmitting locally sampled updates. The framework encompasses communication-regularized local optimization and principled compression of model updates, leveraging gradually updated prior distributions as side information. 
Through extensive simulations, we show that our method achieves $5-32\times$ reduction in total communication cost while preserving accuracy. We further analyze the communication cost, refining existing MRC bounds and enabling precise quantification of uplink and downlink trade-offs. We also extend our method to conventional FL via stochastic quantization and prove a contraction property for the biased MRC compressor to facilitate convergence analysis.

Bi-Directional Communication-Efficient Stochastic FL via Remote Source Generation

Abstract

Federated Learning (FL) incurs high communication costs in both uplink and downlink. The literature largely focuses on lossy compression of model updates in deterministic FL. In contrast, stochastic (Bayesian) FL considers distributions over parameters, enabling uncertainty quantification, better generalization, and, crucially, inherent communication-regularized training through a mirror-descent structure. In this paper, we consider both uplink and downlink communication in stochastic FL, and propose a communication framework based on remote source generation. Employing Minimal Random Coding (MRC) for remote generation, we allow the server and the clients to sample from local and global posteriors (sources), respectively, rather than transmitting locally sampled updates. The framework encompasses communication-regularized local optimization and principled compression of model updates, leveraging gradually updated prior distributions as side information. Through extensive simulations, we show that our method achieves 5 -32 × reduction in total communication cost while preserving accuracy. We further analyze the communication cost, refining existing MRC bounds and enabling precise quantification of uplink and downlink trade-offs. We also extend our method to conventional FL via stochastic quantization and prove a contraction property for the biased MRC compressor to facilitate convergence analysis.

1 Introduction

Federated learning (FL) enables collaborative machine learning (ML) across multiple clients orchestrated by a central federator [McMahan et al., 2017]. Communication efficiency, privacy, security, and data heterogeneity are well-established challenges in FL [Zhang et al., 2021, Wen et al., 2023]. As a bi-directional process, FL requires substantial uplink and downlink communication, posing increasing pressure on communication networks as ML models grow larger. To address this, lossy compression techniques have been widely adopted to reduce uplink gradient transmissions and downlink model broadcasts [Seide et al., 2014, Alistarh et al., 2017, Philippenko and Dieuleveut, 2020, Gruntkowska et al., 2023]. However, these methods almost exclusively focus on conventional (non-stochastic) FL, where clients train deterministic models and transmit fixed updates.

Alternatively, stochastic (Bayesian) FL offers improved generalization, robustness, and inherent uncertainty estimation [Zhang et al., 2022, Milasheuski et al., 2025]. Rather than training deterministic

models, clients learn local posterior distributions, aggregated by the federator to obtain a global posterior. Recently, [Isik et al., 2024] empirically demonstrated state-of-the-art performance under limited uplink bandwidth using stochastic compression methods, outperforming classical approaches. This framework can be applied to a variety of Bayesian FL solutions such as QSGD Alistarh et al. [2017], QLSD Vono et al. [2022], dithered quantization Abdi and Fekri [2019] and FedPM Isik et al. [2023], as well as to conventional FL settings augmented with stochastic compression.

A key technique enabling stochastic FL is remote source generation , which allows the federator to sample from the clients' local posterior, rather than obtaining samples locally generated by the clients. This avoids redundant transmission and enables tight, stochastic control over communication. Such remote generation requires common randomness shared between the transmitter and receiver in the form of a common prior Li [2024], which we also refer to as side information . If the downlink is unlimited, this allows the server to broadcast the global posterior to all the clients, and this posterior serves as a natural common prior. However, when both uplink and downlink are limited, the possibility for remote source generation is restricted, which challenges the application of efficient stochastic FL. Thus, in this paper, we explore and analyze communication-efficient stochastic FL. The rigorous treatment of stochastic FL in this scenario is further reinforced by its two fundamental advantages: (i) Communication-regularized training : we show that stochastic FL, as opposed to conventional solutions, inherently integrates communication constraints into the training process, effectively treating communication as an integral part of the optimization objective; (ii) Priors as side information : the probabilistic structure allows principled integration of common priors as side information, reducing communication costs to the update from prior to posterior.

Concretely, we address the following question: Can joint uplink and downlink compression significantly reduce communication costs in stochastic FL without compromising accuracy? We answer this question affirmatively, tightening the communication-accuracy trade-off in ways that deterministic methods cannot, and achieving communication reductions of up to 32 × without performance loss. Below, we summarize our key contributions.

We propose two novel bi-directional compression algorithms for stochastic FL with Minimal Random Coding (MRC): one leveraging globally shared randomness, and one requiring private shared randomness between each client and the federator. Both enable efficient sampling-based communication by exploiting carefully selected side information.

We demonstrate substantial communication savings, reducing total cost by factors of 5 -32 while maintaining competitive accuracy across standard benchmarks. Our ablation studies analyze the effects of shared randomness and the choice of side information.

We extend our approach to conventional FL with stochastic quantization, proving a contraction property of the resulting (biased) compression operator. This enables convergence guarantees in both uni- and bi-directional settings.

We develop a theoretical framework for communication analysis in stochastic FL, quantifying uplink and downlink costs under MRC. Our results refine and generalize bounds from Chatterjee and Diaconis [2018], including tight analyses for Bernoulli distributions and tools applicable to broader distribution classes.

2 Preliminaries: Stochastic FL with Bi-Directional Compression

In this section, we shortly review the concepts of stochastic FL and compression based on MRC, which are employed in our proposed stochastic bi-directional algorithm.

Stochastic FL. A set of n clients collaboratively and iteratively train a model, e.g., a neural network, under the orchestration of a federator. We primarily consider cross-silo federated learning with a moderate number of reliable clients [Kairouz and McMahan, 2021]. Client i ∈ [ n ] := { 1 , . . . , n } possesses a dataset D i . Wedifferentiate between homogeneous data, where D i is drawn independently from the same distribution for all clients (i.i.d.), and heterogeneous data, where each D i may come from a different distribution (non i.i.d.). At each training iteration t , the federator holds a model θ t described by a probability distribution. After downlink transmission, each client i has an estimate ˆ θ i,t of θ t , and locally optimizes ˆ θ i,t to obtain a local probabilistic model, called the posterior q t i . Compressed clients' posteriors q t i are transmitted back to the federator on the uplink to obtain an estimate ˆ q t i . The federator aggregates the received posteriors using an aggregation rule R ( · ) to obtain

a refined global distribution θ t +1 = R ( { ˆ q t i } i ∈ [ n ] ) . A simple aggregation rule R ( · ) is the average over all clients' posteriors. This process is repeated until a certain convergence criterion is met. In many stochastic FL settings, the sent client updates ˆ q t i are just samples from the posterior q t i .

In fact, conventional FL with stochastic quantization can also be described by the procedure above, though with the following differences: (i) the federator holds a model θ t with deterministic parameters; (ii) each client i locally optimizes ˆ θ i,t to obtain a local gradient g t i . A stochastic compression Q s ( · ) is applied on the client's gradient to obtain a posterior distribution q t i from Q s ( g t i ) ; (iii) samples of q t i are transmitted to the federator on the uplink to obtain an estimate of the gradient, which we still denote by ˆ q t i ; and (iv) the federator updates the global model as θ t +1 = θ t -ηR ( { ˆ q t i } i ∈ [ n ] ) , with learning rate η . In this paper, we will investigate both settings, with a prominent focus on the former.

Stochastic Compression by MRC. To efficiently transmit samples from the posterior q t i , we employ MRC[Havasi et al., 2019], which allows to leverage shared randomness and side information common to the federator and the clients. MRC is a stochastic compressor C mrc ( · ) , whose input is a posterior distribution Q and a prior distribution P , and its output is a sample from a distribution ˆ Q close to Q . It operates as follows: The encoder and decoder generate n IS samples { X i } i ∈ [ n IS ] from P . The encoder computes a categorical distribution W , with W ( i ) ∝ Q ( X i ) /P ( X i ) , and transmits an index i ∼ W with log 2 ( n IS ) bits. To obtain high accuracy, it is required that n IS = Θ(exp(D KL ( Q ∥ P ))) , where D KL ( Q ∥ P ) is the KL-divergence between Q and P [Chatterjee and Diaconis, 2018]. For brevity, in what follows, for two Bernoulli distributions with parameters q and p , we will use the shorthands dKL ( q || p ) and C mrc ( q, p ) .

The choice of MRC for remote generation stems from a practicality aspect and ease of exposition. Ordered random coding (ORC) [Theis and Ahmed, 2022] is a natural advancement, reducing the entropy of the indices selected for transmission using the Gumbel-Max trick. The extension to ORC can be incorporated with our methods using a minor adaption that we omit here for clarity. Theis and Ahmed [2022] further expose an interesting connection to the Poisson functional representation, allowing exact remote generation. Such lossless sampling schemes, however, usually incur substantially larger communication costs and sampling complexities, often rendering those methods impractical. Thus, we herein focus on lossy remote generation schemes for efficiency.

3 BICOMPFL

We next introduce our method BICOMPFL, a bi-directional stochastic compression strategy, which uses MRC to reduce both uplink and downlink communication costs. The scheme assumes that shared randomness between each of the clients and the federator exists, which can be implemented using pseudo-random sequences generated from a common seed. We distinguish two types of shared randomness: private shared randomness (between individual clients and the federator) and global shared common randomness (among all parties), with the latter being more challenging to achieve in practice. We assume all clients and the federator share the same global model ˆ θ 0 at initialization. This does not incur any communication when global shared randomness is available, but necessitates an initial model transmission from the federator to clients when only private shared randomness exists.

BICOMPFL: The General Algorithm. Our method serves as a general framework for stochastic optimization procedures. We explain BICOMPFL for Bayesian FL and show in the sequel how it can be used for conventional FL with stochastic quantization. Consider probabilistic mask training (similar to FedPM, [Isik et al., 2023]) as an example of Bayesian FL. The models θ t ∈ [0 , 1] d of dimension d are parameters of Bernoulli distributions. Those parameters determine for each weight of a randomly initialized network with fixed weights w whether it is activated or not. During inference, the weights w are masked with samples x t ∈ { 0 , 1 } d ∼ θ t , i.e., the network weights are w ⊙ x t . We start with a general description, which is valid for the cases of global and private shared randomness.

At iteration t = 0 , each client i ∈ [ n ] shares with the federator the same global model, i.e., ˆ θ i, 0 = θ 0 , for all i ∈ [ n ] . At iteration t , each client i locally trains model ˆ θ i,t in L local iterations. In our previous example, when training Bernoulli distributions to mask a random network, the parameters are mapped to scores in a dual space, which are then trained for L local iterations m ∈ [ L ] using stochastic gradient descent. Mapping the trained scores back to the primal space, each client i obtains a model update in terms of a posterior q t i . We refer to Appendix F for details. This optimization principle is a special instance of mirror descent, which, in the special case of optimizing over Bernoulli

Algorithm 1 BICOMPFL-GR with Global Randomness

Require: Both clients and federator initialize the same global model θ 0 using a shared seed Ensure: Clients set prior p t = ˆ θ i, 0 = θ 0 , ∀ i ∈ [ n ]

1: repeat

2: for Client i ∈ [ n ] do

3: q t i ← Local training of ˆ θ i,t

4: Employ C mrc ( q t i , p t ) to sample indices I b i,ℓ , ℓ ∈ [ n UL ] , b ∈ [ B ] with prior p t , transmitted to federator to reconstruct ˆ q t i

5: end for

7: Federator relays to client j the other clients' indices { I b i,ℓ } ℓ ∈ [ n UL ] ,b ∈ [ B ] ,i ∈ [ n ] \{ j }

6: Federator updates global model θ t +1 = 1 n ∑ n i =1 ˆ q t i

8: for Clients i ∈ [ n ] do

9: Reconstruct ˆ θ i,t +1 = 1 n ∑ n i =1 ˆ q t i from { I b i,ℓ }

10: end for

11: Clients and federator set prior p t = ˆ θ t +1

12: t ← t +1

13: until Convergence

distributions, leads to a point-wise minimization with respect to a KL-proximity term (as opposed to the Euclidean distance in standard SGD, cf. Appendix C for details). The KL-divergence between the updated local model and the global model directly determines the communication cost. Hence, we regularize the minimization of the loss function by the communication cost, thereby treating communication as an inherent optimization objective.

To convey the model update q t i to the federator, each client employs C mrc ( · ) in B blocks of size d/B each (assuming for simplicity that B divides d ) with a prior distribution p t i,u , which is set to p 0 i,u = ˆ θ i, 0 at iteration t = 0 . The choice of p t i,u for t > 0 will be clarified later. For each block b ∈ [ d/B ] , client i conveys n UL samples { y t i,ℓ } ℓ ∈ [ n UL ] of q t i to the federator by transmitting for each block b an index I b i,ℓ with log 2 ( n IS ) bits, where n IS is the number of samples per block, generated from the prior distribution p t i,u at both the client and the federator using the available shared randomness. The samples of all blocks are concatenated for each ℓ . Hence, the federator obtains an estimate of client i 's posterior distribution using the empirical average ˆ q t i = 1 n UL ∑ n UL ℓ =1 y t i,ℓ .

By averaging the estimates ˆ q t i for all the clients' models, the federator updates the global model as θ t +1 = 1 n ∑ n i =1 ˆ q t i . To transmit the new model to each client i , we assume the existence of a common prior p t i,d shared by the federator and the clients. With p t i,d , the federator performs MRC in B blocks of size d/B to make client i sample from, and thereby estimate, the latest global model θ t +1 . The client samples n DL masks { x t i,ℓ } ℓ ∈ [ n DL ] , each incurring a communication cost of B log 2 ( n IS ) bits. An estimate of the updated global model is obtained by concatenating the reconstructed samples for all the blocks b ∈ [ B ] , and averaging over all masks ˆ θ i,t +1 = 1 n DL ∑ n DL ℓ =1 x t i,ℓ .

Since the number of clients is typically large, n UL = 1 often suffices. The clients' contributions are averaged at the federator, effectively reducing the noise due to the MRC step. This allowed Isik et al. [2024] to theoretically analyze the uplink communication for importance sampling-based stochastic communication of model updates. We will follow a similar approach for downlink communication; however, since downlink communication cannot benefit from the averaging effect of multiple clients, we reduce the variance of the model estimate in the downlink by setting n DL = n · n UL.

The choice of the priors p t i,u and p t i,d for MRC in the uplink and downlink channels, respectively, crucially affects the performance and the communication cost of the algorithm. As a first-order characterization, the communication cost of MRC is determined by D KL ( q t i ∥ p t i,u ) in the uplink and by D KL ( θ t +1 ∥ p t i,d ) in the downlink. We continue the description with the easier setting in which global shared randomness is available, before turning to the more challenging setting of private randomness.

Global Randomness. When global shared randomness is available, all clients can maintain the same priors at each iteration t , and, thereby, obtain the same global model estimates ˆ θ i,t . The global model is known to the clients and the federator from initialization, and synchronization among all clients

Algorithm 2 BICOMPFL-PR with Private Randomness

Require:

Both clients and federator initialize the same global model θ 0 using a shared seed

Ensure:

Clients set prior p t i,u = p t i,d = ˆ θ i, 0 = θ 0 , ∀ i ∈ [ n ]

1: repeat

2: for Client i ∈ [ n ] do

3: q t i ← Local training of ˆ θ i,t

5: Federator estimates client's posterior ˆ q t i = 1 n UL ∑ n UL ℓ =1 y t i,ℓ

4: Federator employs C mrc ( q t i , p t i,u ) to draw n UL samples y t i,ℓ ∼ q t i using prior p t i,u

6: end for

8: for Clients i ∈ [ n ] do

7: Federator updates global model θ t +1 = 1 n ∑ n i =1 ˆ q t i

9: Client employs C mrc ( θ t +1 , p t i,d ) to draw n DL samples x t i,ℓ ∼ θ t +1 using prior p t i,d

10: Client est. global model: ˆ θ i,t +1 = 1 n DL ∑ n DL ℓ =1 x t i,ℓ

11: Clients set prior p t i,u = p t i,d = ˆ θ i,t +1

12: end for

13: t ← t +1

14: until Convergence

is ensured by choosing as prior p t i,u = p t i,d the latest estimate of the global model ˆ θ i,t . The clients utilize the globally shared randomness to sample the exact same samples from the same prior for uplink transmission at all iterations. Selected indices of such samples are transmitted to the federator to convey an estimate ˆ q t i of the posterior q t i , who reconstructs the global model θ t +1 . Using the same prior in the downlink, i.e., the global model from the previous iteration, the updated model can be transmitted to the clients through MRC. Leveraging the shared randomness, all clients i ∈ [ n ] sample from the same prior, and thus obtain the exact same estimate of the global model ˆ θ i,t +1 = ˆ θ t +1 , for all i ∈ [ n ] . Hence, we have that p t i,u = p t i,d = ˆ θ t for all i ∈ [ n ] .

In this version, the federator reconstructs the global model from estimates of the client posteriors ˆ q t i . However, in the uplink, all clients sample from the same prior, which enables further improvements. Naively, the federator will reconstruct the global model using the indices I b i,ℓ for b ∈ [ B ] , ℓ ∈ [ n UL ] received by the clients i ∈ [ n ] through MRC, followed by an additional MRC round for downlink transmission. Instead, and more efficiently, the federator can simply relay the indices to the respective other clients (i.e., client j receives I b i,ℓ for b ∈ [ B ] , i ∈ [ n ] \ { j } , ℓ ∈ [ n UL ] ), which reconstruct the updated global model individually. This avoids additional noise by a second compression round and allows better convergence without additional communication facilitated by global randomness. We term this approach BICOMPFL-GR summarized in Algorithm 1.

Private Randomness. Without global randomness, maintaining the same prior among all clients is impossible without additional communication. Instead, an additional round of MRC is needed for the downlink transmission, and each client obtains a different estimate of the global model ˆ θ i,t at each iteration. Hence, the clients' local trainings start from different estimates of the global model. In a non-stochastic setting, this has only been considered by Philippenko and Dieuleveut [2021], Gruntkowska et al. [2024]. Understanding the additional cost incurred due to lack of shared randomness in terms of both the convergence speed, communication load, and the choice of the priors p t i,u and p t i,d , is then of interest.

ˆ t

For the uplink transmission of client i , any convex combination of θ i,t and ˆ q i can be used as prior, i.e., p t i,u = λ ˆ θ i,t +(1 -λ )ˆ q t i , for some 0 ≤ λ ≤ 1 (cf. Appendix I.2 for details). This is due to the availability of both quantities at the federator and client i . However, small λ values are not expected to reduce the cost of communication reflected by dKL ( q t i || p t i,u ) since the previous global model estimate is likely to be similarly different from the posterior (in terms of the KL-divergence) than the previous posterior estimate of the federator. Indeed, our numerical experiments have shown that the savings from choosing λ = 1 , i.e., priors other than ˆ θ i,t , are not significant. For simplicity, we thus propose to use p t i,u = p t i,d = ˆ θ i,t . We term this approach BICOMPFL-PR and summarize the procedure in Algorithm 2. Choosing different priors is possible and only affects line 11 in Algorithm 2. We mention in passing that BICOMPFL-PR allows partial client participation, which is incompatible

̸

with shared randomness and the method BICOMPFL-GR. We further note that our methods are readily compatible with sparsification and pruning techniques (e.g., Wangni et al. [2018], Shi et al. [2019]) by excluding pruned parameters from the block selection.

Block Allocation. We consider three different block allocation strategies: 1) fixed block size (referred to as 'Fixed' in the experiments), where each block b ∈ [ B ] is of the same size and constant across all t ; 2) adaptive block allocation (Adaptive) as proposed by Isik et al. [2024], where each block size is separately optimized each iteration t ; and 3) adaptive average allocation (Adaptive-Avg), where the block sizes are equal but optimized at each iteration t according to the average KL-divergence per block. We refer the reader to Appendix D for a detailed discussion on this.

Partial Client Participation. We note that BICOMPFL-PR supports partial client participation without additional communication overhead, as it does not rely on full prior synchronization among the clients. BICOMPFL-GR on the other hand, requires that all clients' priors are synchronized at each iteration. To allow partial client participation, the federator is required to transmit the previous global model to the clients that were absent in the previous iteration. This ensures full synchronization of the global model estimate used for efficient uplink communication. Hence, BICOMPFL-GR's compatibility with partial client participation comes with a potentially increased downlink communication cost for previously passive clients.

4 Experiments

We next evaluate the performance of our proposed BICOMPFL-GR and BICOMPFL-PR schemes in experiments, and compare against baseline FL strategies without compression (FedAvg or PSGD) [McMahan et al., 2017] and several non-stochastic bi-directional compression schemes that employ different combinations of compression, error-feedback, and momentum. In particular, we compare against DOUBLESQUEEZE [Tang et al., 2019], MEM-SGD [Stich et al., 2018], NEOLITHIC [Huang et al., 2022], CSER [Xie et al., 2020], and the recently proposed LIEC [Cheng et al., 2024]. SignSGD [Seide et al., 2014] serves to compress the transmitted gradients for all the schemes. We further compare with M3 [Gruntkowska et al., 2024], which partitions the model into disjoint parts for downlink transmission and transmits to each client a different part of the model. While M3 is focused on RandK compression for the uplink (i.e., transmitting random K entries of the gradient), we use TopK [Wangni et al., 2018, Shi et al., 2019], which achieved much more stable results. As mentioned above, the mirror descent approach outlined in Section 3 inherently minimizes the communication cost as a by-product. Hence, it is a strong candidate for communication-efficient stochastic FL. Nonetheless, we show how our method can also be used to improve the communication efficiency in conventional FL, by using the uplink and downlink compression C mrc ( · ) combined with stochastic quantizers, e.g., [Alistarh et al., 2017]. In Section 5, we pave the way to convergence guarantees by proving a contraction property of C mrc ( · ) composed with a stochastic quantization Q s ( · ) of gradients g t i . To compare our method to the baselines that use SignSGD as compressor, we evaluate BICOMPFL-GR in a conventional federated learning (CFL) task with a stochastic variant of SignSGD. We replace mirror descent over Bernoulli masks by a standard learning procedure over a deterministic model, which takes as input the global model estimate ˆ θ i,t , computes a gradient g t i (over L local epochs, using SGD and cross-entropy losses), and outputs a distribution Q s ( g t i ) .

In stochastic SignSGD, Q s ( · ) transforms each gradient entry g t i,e to a Bernoulli random variable with parameter q t i,e = 1 / (1 + exp( -g t i,e /K )) for some K > 0 , where the random variable takes value +1 with probability q t i,e , and -1 otherwise. We then employ C mrc ( q t i , p t i,u ) to obtain samples y t i,ℓ , where the compression is performed element-wise. We apply this method to BICOMPFL-GR where Step 6 is replaced by θ t +1 = θ t -η s 1 n ∑ n i =1 ˆ q t i , where ˆ q t i = 1 n UL ∑ n UL ℓ =1 y t i,ℓ and η s is the federator's learning rate. Step 9 is modified accordingly. The priors p t are chosen as Bernoulli random variables with parameter 0 . 5 . We re-

10 7

10 8

10 9

10 10

10 11

Communication Cost in Bits

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

BiCompFL-GR-CFL-Fixed

mark that while MRC samples are biased towards p t (as we discuss in Section 5), this particular prior choice avoids imbalance in stochastic SignSGD, and rather acts as a regularizer, pulling the clients' posteriors closer to maximum entropy distributions. Consequently, convergence is achieved under bi-directional compression even without error feedback. For general prior choices, error feedback may be needed, see Algorithm 3 and Appendix B. We will refer to this method as BICOMPFL-GR-CFL.

We study n = 10 clients (see Appendix H for additional experiments with more clients) collaboratively training a convolutional neural network (CNN)-based classifier for the datasets MNIST, Fashion-MNIST and CIFAR-10 under the orchestration of a federator. For MNIST, we use two different models, LeNet-5 [Lecun et al., 1998] and a 4 -layer convolutional neural network (4CNN) proposed by Ramanujan et al. [2020]. The latter is also used to train on Fashion MNIST. For CIFAR-10, we use a larger neural network with 6 convolutional layers (6CNN). We train MNIST and Fashion-MNIST for 200 global iterations and CIFAR-10 for 400 global iterations. Through all experiments and datasets, we carry L = 3 local iterations per client. The learning rates are carefully selected to ensure convergence and comparability across all methods. Particularly, we tune the hyperparameters so that all algorithms achieve similar accuracies, allowing a fair comparison of their communication costs (see Appendix I.6 for details). Our main claims are the communication reduction of the bitrates per parameter per epoch, which are orthogonal to the choice of the learning rates of the algorithms. The code to reproduce our experiments is included in the supplementary material.

Weevaluate the schemes in two settings: with a uniform data allocation (i.i.d.), to model homogeneous systems, and with a non-i.i.d. allocation, to model heterogeneous systems, where data allocation for each client is drawn from a Dirichlet distribution with parameter α = 0 . 1 . This regime is challenging due to extreme class imbalance. Each result shows the average across three simulation runs with different seeds. Further details on the simulation setup and the network architectures are deferred to Appendix E. Consistently throughout all experiments, our proposed methods provide order-wise improvements in the communication cost, while achieving state-of-the art accuracies.

We plot in Fig. 1 the test accuracies for all the schemes as a function of the total communication cost in bits per parameter and per global iteration. While all the schemes achieve approximately the same maximum test accuracy, BICOMPFL-GR and BICOMPFL-PR require substantially less communication. Hence, when the bandwidths of uplink and downlink transmissions are limited, both variations of the proposed method achieve better test accuracies. Turning our focus to the different variations of our scheme, it can be observed that, without partitioning the model for downlink compression, BICOMPFL-PR converges significantly slower than BICOMPFL-GR for any block allocation method. This highlights the intuition above that the additional MRC step in downlink incurs further noise, which reduces the convergence speed. However, when we partition the model in the downlink and only send disjoint parts to each client through MRC (BICOMPFL-PR-FixedSplitDL), the downlink communication cost reduces by a factor of n . In the regime of Fashion MNIST with a uniform data allocation, this comes without performance degradation, and is hence the method of choice in this regime. We additionally simulated BICOMPFL-GR with the suboptimal implementation (BICOMPFL-GR-Reconst-Fixed), in which the federator first reconstructs the global model, and then performs an additional MRC step for downlink transmission. This naturally reduces the convergence speed per iteration without gains in the communication cost. Hence, justifying the choice of BICOMPFL-GR. We show that, in conventional FL, BICOMPFL-GR-CFL substantially reduces the communication cost without loss in performance. In some cases, especially for non-i.i.d. data, we even observe improved performance, which we attribute to implicit regularization. Note that BICOMPFL-GR-CFL provides improvements even without error-feedback or momentum. However, our method is fully compatible with such techniques and can be used as a plug-in approach to further minimize the communication cost in many existing schemes. We study the convergence in Section 5.

We plot in Fig. 2(a) the schemes' average bitrates over the maximum test accuracy for MNIST and 4CNN. The average bitrate is reduced by more than a factor of 1000 compared to FedAvg, and more than a factor of 32 compared to DOUBLESQUEEZE, NEOLITHIC and LIEC, which perform best among the conventional bi-directional compression methods. We repeat the study for non-i.i.d.data allocation according to a Dirichlet distribution with parameter α = 0 . 1 , and show maximum test accuracies over average bitrates in Fig. 2(b). Partitioning the model in BICOMPFL-PR worsens the final accuracy of the model. While the model converges faster, it does not achieve the same accuracies as BICOMPFL-GR and BICOMPFL-PR without partitioning. This hints towards hybrid schemes for BICOMPFL-PR, where the training begins with partitioning on the downlink, which is later switched

10

1

10 0

10 1

Bitrate p. P .

0.990

0.991

0.992

0.993

0.994

0.995

0.996

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

BiCompFL-GR-CFL-Fixed

(a) MNIST 4CNN i.i.d.

10

1

10 0

10 1

Bitrate p. P .

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

BiCompFL-GR-CFL-Fixed

(b) MNIST 4CNN non-i.i.d.

10

1

10 0

10 1

Bitrate p. P .

0.625

0.650

0.675

0.700

0.725

0.750

0.775

0.800

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

BiCompFL-GR-CFL-Fixed

(c) CIFAR-10 6CNN i.i.d.

to full transmission. In Fig. 2(c), we provide the results for CIFAR-10 and uniform data allocation. BICOMPFL-GR and BICOMPFL-PR both achieve better results with a bitrate smaller by a factor of 5 than the best baselines. More detailed numerical results can be found in Appendices H and I.

The adaptive block allocation (Adaptive) of Isik et al. [2024] saves communication costs in many settings and provides better performance than the fixed block allocation (Fixed), due to more accurate MRC tailored to the exact divergences. The proposed low complexity adaptive strategy based on the average KL-divergence (Adaptive-Avg) per block can additionally save in communication (and computation) with no or little performance degradation. We refer the reader to Appendix H for further extensive experiments, graphs for accuracies over epochs, separate studies of uplink and downlink costs, and comparisons for the case of an available broadcast channel from federator to the clients. Finally, we refer to Appendix I for various ablation studies analyzing the sensitivity of BICOMPFL with respect to the choices of the priors, n , n DL, n IS, the block size d/B , and the learning rate η .

5 Theoretical Results

Convergence. In stochastic FL, the exact time dynamics of the system are challenging to analyze due to the round-dependent interplay of the learning procedure with the transmission noise. However, when using BICOMPFL for conventional FL with stochastic quantization (cf. BICOMPFL-GRCFL), convergence guarantees can be given. We prove the convergence for a general and widely used class of stochastic quantizers Q s ( · ) , which are natively unbiased. Q s ( · ) takes as input the entry g e of a gradient vector g ∈ R d and operates as follows. Let s be the number of quantization intervals, and let 0 ≤ τ e < s be an integer such that τ e s ≤ | g e | ∥ g ∥ ≤ τ e +1 s , then Q s ( g e ) outputs ∥ g ∥· sign ( g e )( τ e +1) /s with probability s | g e | / ∥ g ∥-τ e , and ∥ g ∥· sign ( g e ) τ e /s otherwise. Q s ( · ) is unbiased, i.e., E [ Q s ( x )] = x , and its variance satisfies E [ ∥ Q s ( x ) -x ∥ 2 ] ≤ min { d/s 2 , √ d/s }∥ x ∥ 2 2 [Alistarh et al., 2017]. See Remark 2 in Appendix B for a discussion on the choice of s .

Replacing stochastic SignSGD by Q s ( · ) in BICOMPFL-GR-CFL, the posterior is given by a Bernoulli distribution with parameter q t i,e = s | g t i,e | / ∥ g t i ∥-τ e . The values ∥ g ∥ , sign ( g ) , and τ e can be encoded independently, e.g., using Elias coding. With a slight abuse of notation, let C mrc ( Q s ( · ) , · ) denote the composition of Q s ( · ) and MRC with n IS samples per entry. The compression C mrc ( Q s ( g t i ) , · ) takes a gradient g t i and outputs samples from a distribution close to Q s ( g t i ) , and falls in the class of biased compressors. We can prove the following contraction property for C mrc ( Q s ( · ) , · ) , which will facilitate convergence analysis for uni- and bi-directional compression. This constitutes a substantial improvement over [Isik et al., 2024], where such guarantees were missing, and hence no convergence guarantees were given. A prominent biased contractive compressor is TopK.

Lemma 1. For any x ∈ R d and corresponding posterior q following Q s ( x ) , and a prior p ∈ [0 , 1] d , let ¯ ∆ := max e ∈ [ d ] q e p e -1 -q e 1 -p e , ¯ ∆ ′ := max e ∈ [ d ] q e ( p e q e + 1 -p e 1 -q e ) , and ¯ p := max e ∈ [ d ] p e . The compressor C mrc ( Q s ( · )) satisfies the following contraction property for n IS = O (max { √ 2 ¯ ∆ ′ , log(6¯ p ( ¯ ∆+ ¯ ∆ 2 )) √ 6¯ p ( ¯ ∆+ ¯ ∆ 2 ) } ) and s ≥ √ 2 d :

The underlying core result is a refinement of the MRC analysis, cf. Lemma 2 (Appendix A). Hence, for sufficiently large n IS, the compressor C mrc ( Q s ( · ) , · ) can be used as an alternative to common compressors such as Q s ( · ) . The use of MRC introduces a bias into the otherwise unbiased stochastic quantization. Based on the contraction property in Lemma 1, standard convergence results (cf. Theorem 2) follow easily by a straightforward extension of our conventional FL algorithm BICOMPFL-GR-CFL to error feedback (cf. Algorithm 3) as detailed in Appendix B.

̸

̸

Communication Cost. We analyze the communication cost in a specific iteration t and comment on the inter-round dependency later. When the latest global model estimate ˆ θ i,t is chosen as a prior in MRC, the uplink cost is determined by how far the model evolves during the client's training, i.e., dKL ( q t i || p t i,u ) = dKL ( q t i || ˆ θ i,t ) . After communicating samples of the posteriors, the federator obtains an estimate ˆ q t i for all i ∈ [ n ] . The cost of communication on the downlink to client i is then determined by dKL ( 1 n ∑ n i =1 ˆ q t i || ˆ θ i,t ) . While dKL ( q t i || ˆ θ i,t ) depends on the progress during client training, the core challenge is to bound the expected KL-divergence of each model estimate dKL (ˆ q t i || ˆ θ i,t ) in the presence of potentially different priors, i.e., ˆ θ i,t = ˆ θ j,t , i = j . For each client i , the overall communication cost is in the order of n DL exp ( d KL ( 1 n ∑ n i =1 ˆ q t i ∥ p t i,d )) + n UL exp ( dKL ( q t i || p t i,u ) ) . We will next quantify dKL ( 1 n ∑ n i =1 ˆ q t i || ˆ θ i,t ) for the case p t i,u = p t i,d , however, the analysis can be extended to p t i,u = p t i,d by an additional assumption on the divergence between the two priors.

̸

For the theoretical analysis, we focus on the scalar case for a single iteration t , where client i ∈ [ n ] has a posterior Q i , and the federator and client i share a common prior P i , both are Bernoulli distributions with parameters q i and p i , respectively. In the context of FL, the client locally trains P i and results with Q i . According to Chatterjee and Diaconis [2018] and the multi-client extension of Isik et al. [2024], the communication cost in the uplink is determined by exp( dKL ( Q i || P i )) . After uplink transmission, the federator obtains an estimate ˆ q i of q i ; and hence, the updated global model is given by 1 n ∑ n i =1 ˆ q i . The downlink cost for client i is determined by dKL ( 1 n ∑ n i =1 ˆ q i || p i ) .

We derive a new high probability upper bound on this quantity, refining previous MRC analysis for the special case of Bernoulli distributions. While a more general analysis can be conducted for other classes of distributions (cf. Remark 1 in Appendix A), the Bernoulli-based optimization method described earlier proves particularly efficient by enabling communication-regularized training, a unique property that renders our methods substantially more communication-efficient while preserving state-of-the-art performance.

Let X be a Bernoulli sample obtained through MRC. As an initial step, we bound the difference between q i and the probability Pr( X = 1) that the samples are drawn from, which vanishes when p i = q i (and hence dKL ( q i || p i ) = 0 ). We note that the bound of Chatterjee and Diaconis [2018, Theorem 1.1] does not satisfy this natural property. We formally state the result in Proposition 1 in Appendix A, which, however, does not yet capture the dependency on the number of samples n IS used in MRC to sample an index. We refine Proposition 1 with Lemma 2 (cf. Appendix A), which additionally captures this dependency, and will allow us to derive an upper bound on dKL ( 1 n ∑ n i =1 ˆ q i || p i ) . Lemma 2 is of independent interest and can be seen as a refinement of [Chatterjee and Diaconis, 2018] for Bernoulli distributions. It is required to prove Theorem 1.

For the statement of the following theorem, we assume that the progress by one local client training is bounded by | q j -p j | ≤ ρ for all j ∈ [ n ] . Using Pinsker's inequality to bound | q j -p j | ≤ 1 2 √ dKL ( q j || p j ) / 2 , this is a natural assumption given from the KL-proximity term of mirror descent (for one local iteration), and can be strictly enforced through the projection of q j onto a KL ball around p j of fixed divergence. We assume that the difference between the clients' priors, i.e., their global model estimates in our algorithms, are bounded as | p i -p j | ≤ ζ for all i, j ∈ [ n ] .

q 1 -q p + ζ 1 -p + ζ

By Chatterjee and Diaconis [2018], this provides an immediate bound on the cost of downlink transmission. The bound applies to both algorithms BICOMPFL-PR and BICOMPFL-GR. However, when all priors p j are the same (such as in BICOMPFL-GR-Reconst), i.e., ζ = 0 , the bound simplifies

accordingly. The explicit dependency on the factor 1 / √ n UL reflects the interplay between uplink and downlink cost. The parameter ζ gives rise to an inter-round dependency of the communication cost. The more accurate the estimation of the global model in the previous iteration (given the priors are chosen as ˆ θ i,t ), the smaller ζ , and hence the lower the transmission cost in the subsequent iteration. The proofs of Proposition 1, Lemma 2, and Theorem 1 can be found in Appendix A.

6 Related Work

Following the introduction of FL [McMahan et al., 2017], lossy compression of gradients or model updates has been a long-studied narrative in FL, with prominent representatives such as SignSGD, also known as 1-bit Stochastic Gradient Descent (SGD) [Seide et al., 2014], QSGD [Alistarh et al., 2017], TernGrad [Wen et al., 2017], SignSGD with error feedback [Karimireddy et al., 2019], vectorquantized SGD [Gandikota et al., 2021] and natural compression [Horvóth et al., 2022]. Such methods retain satisfactory final model accuracy even with aggressive quantization. Sparsificationbased methods have also been considered as alternatives, e.g., TopK [Wangni et al., 2018, Shi et al., 2019]. The importance of bi-directional gradient compression in many settings was outlined by Philippenko and Dieuleveut [2020]. Many schemes were proposed that leverage combinations of gradient compression in the uplink and downlink, error-feedback, and momentum, e.g., Mem-SGD [Stich et al., 2018], DoubleSqueeze [Tang et al., 2019], block-wise SignSGD with momentum [Zheng et al., 2019], communication-efficient SGD with error reset (Cser) [Xie et al., 2020], Artemis [Philippenko and Dieuleveut, 2020], Neolithic [Huang et al., 2022], DOCOFL [Dorfman et al., 2023], EF21-P and friends [Gruntkowska et al., 2023], 2Direction [Tyurin and Richtárik, 2023], M3 [Gruntkowska et al., 2024], and LIEC [Cheng et al., 2024]. With the exception of the methods MCM [Philippenko and Dieuleveut, 2021] and M3 [Gruntkowska et al., 2024], each client receives the same broadcast, potentially compressed, global gradient or model update. Isik et al. [2024] studied uplink compression for stochastic FL and showed significant communication reduction with competitive performance. Their framework, termed KLMS, applies to a variety of stochastic compressors and to Bayesian FL settings, e.g., QLSD Vono et al. [2022]. The compression is based on importance sampling and MRC, thoroughly studied by Chatterjee and Diaconis [2018] and Havasi et al. [2019]. Such methods, known as relative entropy coding, have been used in FL in conjunction with differential privacy, cf. DP-REC [Triastcyn et al., 2022].

Since the lottery ticket hypothesis [Frankle and Carbin, 2019], finding sparse subnetworks of neural networks that achieve satisfactory accuracy was investigated. Ramanujan et al. [2020] showed that randomly weighted networks contain suitable subnetworks of large neural networks capable of achieving competitive performance. Isik et al. [2023] formulated a probabilistic method of training neural network masks collaboratively in an FL context.

7 Conclusion

We illuminated bi-directional compression in stochastic FL via federated probabilistic mask training, which we showed to inherently optimize both the learning objective and the communication costs. By leveraging side information through carefully chosen prior distributions, the total communication costs are reduced by factors between 5 -32 compared to non-stochastic FL baselines, while achieving state-of-the-art accuracies on classification tasks, for both homogeneous and heterogeneous data. We thus close the gap of downlink compression for stochastic FL and complement the existing literature on bi-directional compression for standard FL. Applying our methods to stochastic quantization in conventional FL, we paved the way to convergence analysis for MRC-based compression. Allowing different priors among all clients, this work opens the door to studying compression under sideinformation in decentralized stochastic FL , where a central coordinator is missing. Our theoretical results are of independent interest and may be applied in various scenarios where MRC is used.

Client privacy and fairness are important directions beyond the primary focus of this work. Compression methods commonly strengthen the clients' privacy in FL. Particularly, the noise introduced through our sampling methods dilute the individual contributions of the clients observed by the federator, naturally enhancing the clients' privacy and reducing the risk of privacy breaches through, e.g., membership inference attacks. Quantifying the privacy-utility trade-off that arises in the presence of bi-directional compression is left as an interesting direction for future work.

Acknowledgments

The research of R. Bitar is supported by the German Research Foundation (DFG) under grant agreement number BI 2492/1-1. The research of N. Weinberger was partially supported by the Israel Science Foundation (ISF), grant no. 1782/22. D. Gündüz acknowledges funding from UKRI under the ERC-Consolidator project AI-R (EP/X030806/1) and INFORMED-AI Hub (EP/Y028732/1).

References

Abdi and F. Fekri. Nested dithered quantization for communication reduction in distributed training. arXiv preprint arXiv:1904.01197 , 2019.

Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems , volume 30, 2017.

Chatterjee and P. Diaconis. The sample size required in importance sampling. The Annals of Applied Probability , 28(2):1099-1135, 2018.

Cheng, L. Shen, L. Xu, X. Qian, S. Wu, Y. Zhou, T. Zhang, D. Tao, and E. Chen. Communication-efficient distributed learning with local immediate error compensation. arXiv preprint arXiv:2402.11857 , 2024.

Dorfman, S. Vargaftik, Y. Ben-Itzhak, and K. Y. Levy. DoCoFL: Downlink compression for crossdevice federated learning. In International Conference on Machine Learning , pages 8356-8388, 2023.

Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019.

Gandikota, D. Kane, R. Kumar Maity, and A. Mazumdar. vqSGD: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics , volume 130, pages 2197-2205, 2021.

Gruntkowska, A. Tyurin, and P. Richtárik. EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning , pages 11761-11807, 2023.

Gruntkowska, A. Tyurin, and P. Richtárik. Improving the worst-case bidirectional communication complexity for nonconvex distributed optimization under function similarity. arXiv preprint arXiv:2402.06412 , 2024.

Havasi, R. Peharz, and J. M. Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations , 2019.

Horvóth, C.-Y. Ho, L. Horvath, A. N. Sahu, M. Canini, and P. Richtarik. Natural compression for distributed deep learning. In Proceedings of Mathematical and Scientific Machine Learning , volume 190, pages 129-141, 2022.

Huang, Y. Chen, W. Yin, and K. Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. Advances in Neural Information Processing Systems , 35:18955-18969, 2022.

Isik, F. Pase, D. Gunduz, T. Weissman, and Z. Michele. Sparse random networks for communication-efficient federated learning. In International Conference on Learning Representations , 2023.

Isik, F. Pase, D. Gunduz, S. Koyejo, T. Weissman, and M. Zorzi. Adaptive compression in federated learning via side information. In International Conference on Artificial Intelligence and Statistics , pages 487-495, 2024.

Kairouz and H. B. McMahan. Advances and open problems in federated learning. Foundations and trends in machine learning , 14(1-2):1-210, 2021.

P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In International Conference on Machine Learning , volume 97, pages 3252-3261, 2019.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

T. Li. Channel simulation: Theory and applications to lossy compression and differential privacy. Foundations and Trends® in Communications and Information Theory , 21(6):847-1106, 2024.

McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics , volume 54, pages 1273-1282, 2017.

Milasheuski, L. Barbieri, S. Kianoush, M. Nicoli, and S. Savazzi. Bayesian federated learning for continual training. arXiv preprint arXiv:2504.15328 , 2025.

Philippenko and A. Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. arXiv preprint arXiv:2006.14591 , 2020.

Philippenko and A. Dieuleveut. Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems , 34:2387-2399, 2021.

Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in a randomly weighted neural network? In IEEE/CVF conference on computer vision and pattern recognition , pages 11893-11902, 2020.

Richtárik, I. Sokolov, and I. Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems , 34:4384-4396, 2021.

Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Interspeech , pages 1058-1062, 2014.

Shi, Q. Wang, K. Zhao, Z. Tang, Y. Wang, X. Huang, and X. Chu. A distributed synchronous SGD algorithm with global top-k sparsification for low bandwidth networks. In International Conference on Distributed Computing Systems (ICDCS) , pages 2238-2247, 2019.

Srinivasan. Importance sampling: Applications in communications and detection . Springer Science & Business Media, 2002.

U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified SGD with memory. Advances in Neural Information Processing Systems , 31, 2018.

Tang, C. Yu, X. Lian, T. Zhang, and J. Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In International Conference on Machine Learning , pages 6155-6165, 2019.

Theis and N. Y. Ahmed. Algorithms for the communication of samples. In International Conference on Machine Learning , pages 21308-21328, 2022.

Triastcyn, M. Reisser, and C. Louizos. DP-REC: Private & communication-efficient federated learning, 2022.

Tyurin and P. Richtárik. 2Direction: theoretically faster distributed training with bidirectional communication compression. In Conference on Neural Information Processing Systems , 2023.

Vono, V. Plassier, A. Durmus, A. Dieuleveut, and E. Moulines. QLSD: quantised langevin stochastic dynamics for bayesian federated learning. In International Conference on Artificial Intelligence and Statistics , pages 6459-6500, 2022.

Wangni, J. Wang, J. Liu, and T. Zhang. Gradient sparsification for communication-efficient distributed optimization. Advances in Neural Information Processing Systems , 31, 2018.

Weinberger and M. Yemini. Multi-armed bandits with self-information rewards. IEEE Transactions on Information Theory , 2023.

Wen, Z. Zhang, Y. Lan, Z. Cui, J. Cai, and W. Zhang. A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics , 14(2):513-535, 2023.

Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. TernGrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems , volume 30, 2017.

Xie, S. Zheng, S. Koyejo, I. Gupta, M. Li, and H. Lin. Cser: Communication-efficient sgd with error reset. Advances in Neural Information Processing Systems , 33:12593-12603, 2020.

Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y . Gao. A survey on federated learning. Knowledge-Based Systems , 216:106775, 2021.

Zhang, Y. Li, W. Li, K. Guo, and Y. Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning , pages 26293-26310, 2022.

Zheng, Z. Huang, and J. Kwok. Communication-efficient distributed blockwise momentum SGD with error-feedback. Advances in Neural Information Processing Systems , 32, 2019.

A Proofs and Intermediate Results

In the following, we provide the formal statements of Proposition 1 and Lemma 2 including their proofs. Parts of the proof of Proposition 1 will be used to prove Lemma 2. We prove Theorem 1 afterward.

Proposition 1. For a sample X ℓ transmitted by MRC with posterior and prior Bernoulli distributions with parameters q and p , we have

Proof of Proposition 1. Assume a party wants to sample from a Bernoulli distribution Q with parameter q , which is held by another party. Both parties share a common prior P in the form of a Bernoulli distribution with parameter p and have access to shared randomness. Fix any sample index ℓ for the moment (this index will be needed for the proof of Theorem 1). Both parties sample Kn IS i.i.d. samples X ℓ,i ∼ P for i ∈ [ n IS ] independently and identically from P . The party holding Q constructs an auxiliary distribution

from which it samples to obtain an index I ℓ . The index is transmitted to the other party, which reconstructs the corresponding sample X ℓ,I ℓ .

To bound the difference | Pr( X ℓ = 1) -q | , i.e., the target Bernoulli parameter compared to the parameter which the sample is drawn from, by the independence of the samples X ℓ,I ℓ for different ℓ ,

we focus on a single sample ℓ ∈ [ K ] , for which it holds that

where ( a ) follows from symmetry, ( b ) follows since by permutation invariance, the inner probability only depends on the number of ones in { x 2 , . . . , x n IS } .

The inner probability is given by the distribution W ℓ ( i ) . Given that X ℓ, 1 = 1 and that ∑ n IS i =2 X ℓ,ℓ = L , it holds that

Hence,

which is independent of the exact choice of { x 2 , . . . , x n IS } given their sum ∑ n IS i =2 X ℓ,i = L . Since Pr( X ℓ, 1 = 1 , X ℓ, 2 = x 2 , . . . , X ℓ,n IS = x n IS ) = p L+1 (1 -p ) n IS -L -1 by the Bernoulli distribution assumption, we have

Defining a binary random variable M with sample space { q p , 1 -q 1 -p } , for a Bernoulli distribution Ber ( L+1 n IS ) with success probability parameter L+1 n IS , where a success refers to the outcome M= q p , we can write that

where the outer expectation is over the binomial distribution with n IS -1 trials and success probability p , i.e., L ∼ Binomial ( n IS -1 , p ) , and where ( a ) follows from Jensen's inequality over the inner expectation. Hence,

Since 1 E Ber ( L+1 n IS ) [M] ≥ 2 -E Ber ( L+1 n IS ) [M] , it also follows from (1) that

from which we have

Combining the upper and lower bound in (2) and (3), respectively, we derive

This concludes the proof.

Lemma 2. For a sample X ℓ transmitted via MRC with posterior and prior being Bernoulli distributions with parameters q and p , ∆ := q p -1 -q 1 -p and ∆ ′ := q ( p q + 1 -p 1 -q ) , we have

Proof of Lemma 2. The proof starts with the same derivations as for the proof of Proposition 1, which we follow until (1) to get

Since L is a random quantity that follows a Binomial distribution, we bound | Pr( X ℓ,I ℓ = 1) -q | using a concentration bound on L . The relative (multiplicative) Chernoff bound states that

for any ε ∈ [0 , 1] . Setting ε = √ 3 log(2 /δ ) n IS p implies that

with probability at most δ . Setting δ = 1 n 2 , we obtain for a concentration parameter 2 η δ :=

IS

with probability Pr( E ) ≤ 1 n 2 IS .

Then, we can write

Assume for now that q < p (we will later proof the opposite event), then 1 E Ber ( L+1 n IS ) [M] is strictly non-increasing in L since q p < 1 -q 1 -p , and hence, when E c holds and hence L concentration around the average that

where the last step is by Taylor expansion. Using (4) and the monotonicity of 1 E Ber ( L+1 n IS ) [M] , we write

and hence

Similarly, we get by bounding 1 E Ber ( L+1 n IS ) [M] ≥ 1 E Ber ( (L+1) · ( p + η δ ) n IS ) [M] and using (4) that

When p ≤ q , then 1 E Ber ( L+1 n IS ) [M] is strictly non-decreasing, hence, under E , we have

and thus from (4) that

Since 0 ≤ p + η δ ≤ 1 and 1 ≥ p -η δ ≥ 0 by an appropriate choice of the concentration intervals, we have by approximations up to second order terms that

This concludes the proof.

Proof of Lemma 1. Using Lemma 2, we can show the following. Recall the following probability law of the stochastic quantizer Q s ( · ) [Alistarh et al., 2017] using s > 0 quantization intervals, which takes as input the entry x e of a gradient x ∈ R d vector. Let 0 ≤ τ e < s be an integer such that τ e s ≤ | x e | ∥ x ∥ ≤ τ e +1 s , then Q s ( x e ) is defined as Ber ( | x e | ∥ x ∥ s -τ e ) , which outputs ∥ x ∥ · sign ( x e )( τ e +1) /s in case of success, and ∥ x ∥ · sign ( x e ) τ e /s otherwise.

Focusing on an entry x e , we prove a contraction property for MRC with stochastic quantization with posterior q e = | x e | ∥ x ∥ s -τ e , and an arbitrary prior p e . In fact, the MRC methodology C mrc ( · ) leads to sampling from an approximate distribution with parameter ˜ q e . To be more specific, C mrc ( x e ) outputs ∥ x ∥ · sign ( x e )( τ e +1) /s with probability ˜ q e , and ∥ x ∥ · sign ( x e ) τ e /s with probability 1 -˜ q e . We established in Lemma 2 an upper bound on | q e -˜ q e | , which will be useful in the following.

To prove a contraction property of the kind

we can write

where

Substituting the result in (5), obtain

where, by Lemma 2, we have for ∆ e := q e p e -1 -q e 1 -p e and ∆ ′ e := q e ( p e q e + 1 -p e 1 -q e ) that

Let ¯ ∆ := max e ∈ [ d ] q e p e -1 -q e 1 -p e , ¯ ∆ ′ := max e ∈ [ d ] q e ( p e q e + 1 -p e 1 -q e ) , and ¯ p := max e ∈ [ d ] p e . We will ensure that ¯ ∆ ′ n 2 IS + O ( ( ¯ ∆+ ¯ ∆ 2 ) √ 6¯ p log(2 n IS ) n IS ) ≤ 1 by making each of the individual terms ≤ 1 2 . By choosing n IS ≥ √ 2 ¯ ∆ ′ , we have ¯ ∆ ′ n 2 IS ≤ 1 2 . To ensure that ( ¯ ∆ + ¯ ∆ 2 ) √ 6¯ p log(2 n IS ) n IS ≤ 1 2 , we require log(2 n IS ) n IS ≤ 1 √ 6¯ p ( ¯ ∆+ ¯ ∆ 2 ) . By Weinberger and Yemini [2023, Lemma 15], this holds when n IS = O (log(6¯ p ( ¯ ∆+ ¯ ∆ 2 )) √ 6¯ p ( ¯ ∆+ ¯ ∆ 2 )) . Hence, choosing n IS = O (max { √ 2 ¯ ∆ ′ , log(6¯ p ( ¯ ∆+ ¯ ∆ 2 )) √ 6¯ p ( ¯ ∆+ ¯ ∆ 2 ) } ) , we have ¯ ∆ ′ n 2 IS + O ( ( ¯ ∆+ ¯ ∆ 2 ) √ 6¯ p log(2 n IS ) n IS ) ≤ 1 . Thus, we have 0 ≤ δ ≤ 1 if 2 d s 2 ≤ 1 , and hence s ≥ √ 2 d . This concludes the proof.

Proof of Theorem 1. Assume a party estimates the Bernoulli distributions Q j with parameters q j held by parties j ∈ [ n ] . The estimating party shares with each of the other parties a common prior P j in the form of a Bernoulli distribution with parameter p j and access to unlimited shared randomness. To help estimate Q j , the j -th party sends K samples to the estimator through MRC. Therefore, both parties sample Kn IS i.i.d. samples X ℓ,i ∼ P j for ℓ ∈ [ K ] , i ∈ [ n IS ] , independently and identically from P j . The party holding Q j constructs for each ℓ ∈ [ K ] an auxiliary distribution

from which it samples to obtain an index I ℓ . The index is transmitted to the estimating party, which reconstructs the corresponding sample X ℓ,I ℓ . Averaging the samples for all ℓ ∈ [ K ] gives an estimate ˆ q j of q j , i.e., ˆ q j = 1 K ∑ K ℓ =1 X ℓ,I ℓ . This process is repeated for all j ∈ [ n ] .

We assume that | q j -p j | ≤ ρ for all i, j ∈ [ n ] , and that the difference between the priors, is bounded as | p i -p j | ≤ ζ for all i, j ∈ [ n ] . The goal is to bound dKL ( 1 n ∑ n j =1 ˆ q j || p i ) from above for any i ∈ [ n ] .

By the convexity of KL-divergence, we have

To bound dKL (ˆ q j || p i ) for any i, j ∈ [ n ] , by the triangle inequality, we can write

| ˆ q j -p i | ≤ | ˆ q j -Pr( X ℓ = 1) | + | Pr( X ℓ = 1) -q j | + | q j -p j | + | p j -p i | , where | ˆ q j -Pr( X ℓ = 1) | is bounded by Lemma 2. By Hoeffding's inequality, we have with probability at least 1 -δ ′ that

Thus, with probability at least 1 -δ ′ , since p j ≤ p i + ζ , we have with ∆ j := q j p j -ζ -1 -q j 1 -p j + ζ and

This holds under the assumption that p j > ζ for all j ∈ [ n ] . By the reversed Pinsker's inequality, we obtain

The statement of the theorem follows by the convexity of KL-divergence.

Remark 1. Note that our analysis can be extended to other parametric distributions, such as multi-variate Gaussians. The key ingredient is to replace the specific upper bound on the bias in Lemma 2 with the generic results from Chatterjee and Diaconis [2018], which holds for all classes of distributions. Using this upper bound, one can follow our derivations to prove the communication costs in Theorem 1, i.e., using the convexity of KL divergence and decomposing the error in parameter estimation into a bias term and a concentration term to bound the sampling error. Standard concentration results can be utilized to bound the latter from above, e.g., Hoeffding's inequality for sub-Gaussian random variables. The remaining steps follow analogously to our proof. Similar adaptations apply to the contraction property in Lemma 1 necessary to establish convergence guarantees for conventional FL.

B Convergence Analysis

Using the contraction property derived in Lemma 1, we can show that a straightforward extension of BICOMPFL-GR-CFL to error-feedback as used in [Richtárik et al., 2021] leads to the following convergence guarantee. The algorithmic details of the extension can be found in Algorithm 3. Therefore, assume that for all for x , y ∈ R d and i ∈ [ n ] , the following Lipschitz property holds:

Let F ( θ ) := 1 n ∑ n i =1 ∇ F ( θ, D i ) be the global loss function and L ′ := √ 1 n ∑ n i =1 L i .

Theorem 2. If F ⋆ := inf θ ∈ R d { F ( θ ) } > -∞ and E [ ∥ g t - ∇ F ( θ t ) ∥ 2 ] ≤ σ 2 , then with η ≤ ( L + L ′ √ 1 -δ (1 - √ 1 -δ ) 2 ) -1 , L = 1 , s ≥ √ 2 d , and n IS satisfying Lemma 1 in every iteration t , we have for Algorithm 3 that

Similarly, guarantees can be derived for other algorithms, such as modified versions of BICOMPFLPR with error-feedback and momentum, using Lemma 1. However, we emphasize the generality of BICOMPFL, reaching beyond conventional FL with stochastic compression to pure stochastic narratives.

Remark 2. The choice of s depends on the model architecture and dataset complexities, affecting the number of iterations required until convergence. The simpler the learning task, the fewer quantization intervals are sufficient for convergence. Beyond generic arguments, we particularly highlight that s should be carefully selected, respecting the expected variance of the gradients. If gradients exhibit inherently large variance, e.g., through small mini-batch sizes, large variance of the local datasets, or substantially over-parameterized networks, fewer quantization intervals might be efficient, and increasing that number would only negligibly improve the performance. We also note that one-bit quantization has been shown to be often remarkably effective despite its simplicity [Seide et al., 2014, Karimireddy et al., 2019].

C Gradient Descent with a KL-Proximity

Mirror descent employs point-wise optimization in the form of a first-order approximation of F ( ˆ θ t , D i ) with proximity term D F ( p, q ) , where D F is the Bregman divergence associated with function F ( · ) . When F ( x ) = ∥ x ∥ 2 , and hence the Bregman divergence is the Euclidean distance, this is known as gradient descent. Let now p and q be vectors with the entries corresponding to independent Bernoulli parameters. When we choose F ( x ) = x log( x ) + (1 -x ) log(1 -x ) , the Bregman divergence becomes D F ( p, q ) = ∑ d k =1 D KL ( p k ∥ q k ) . Hence, we are optimizing with respect to a KL-proximity constraint. The mapping between dual and primal spaces is then given by ∇ F ( x ) = log( x ) -log(1 -x ) and ( ∇ F ( x )) -1 = 1 e -x +1 , respectively; also known as the inverse sigmoid and the sigmoid functions.

D Block Allocation

The simplest yet effective strategy for block allocation is to partition the model into equally-sized blocks of size d/B for MRC (Fixed). The partitioning into blocks is required to make MRC practically feasible in this setting. It is known that for vanishing MRC error, the number of samples n IS from a block p t i,u,b of the prior is supposed to be in the order of exp ( D KL ( q t i,b ∥ p t i,u,b )) , where q t i,b is the b -th block of posterior q t i . It was observed by Isik et al. [2024] that the KL-divergence decreases as the training progresses with the global model used as a prior, which is intuitive since the local training will change the posterior less and less as training converges. To adapt the block size according to the

Algorithm 3 BICOMPFL-GR-CFL with stochastic quantization Q s ( · ) and EF21 from Richtárik et al. [2021]

Require: Both clients and federator initialize the same global model θ 0 using a shared seed

Ensure: Set t = 0 , clients set prior p t = ˆ θ 0 = θ 0 , ∀ i ∈ [ n ] , clients compute and broadcast v 0 i = C mrc ( Q s ( g t i ) , p t ) , with g t i the local gradient for θ 0 ; hence, v 0 = 1 n ∑ n i =1 v 0 i public

1: Update ∀ i : ˆ θ t +1 = ˆ θ t -η v t +1

2: repeat

3: for Client i ∈ [ n ] do

4: Compute gradient g t i by local training over L local iterations

6: Sample indices I b i,ℓ , ℓ ∈ [ n UL ] , b ∈ [ B ] from q t i with prior p t and transmit to federator to reconstruct ˆ q t i = C mrc ( q t i , p t )

5: Stochastic compression q t i ← Q s ( g t i -v t i )

7: Update v t +1 i = v t i + ˆ q t i

9: Federator reconstructs and computes v t +1 = v t + 1 n ∑ n j =1 ˆ q t j from { I b i,ℓ }

8: end for

10: Federator updates θ t +1 = θ t -η v t +1

11: Federator relays to client j the other clients' indices { I b i,ℓ } ℓ ∈ [ n UL ] ,b ∈ [ B ] ,i ∈ [ n ] \{ j }

12: for Clients i ∈ [ n ] do

13: Reconstruct and compute v t +1 = v t + 1 n ∑ n j =1 ˆ q t j from { I b i,ℓ }

14: Update ˆ θ t +1 = ˆ θ t -η v t +1 from { I b i,ℓ }

15: end for

16: Clients and federator set prior p t = ˆ θ t +1

17: t ← t +1

18: until Convergence

divergence from the posterior with respect to the prior, Isik et al. [2024] proposed an adaptive block allocation strategy (Adaptive), where upon realizing a large deviation from the target KL-divergence per block, clients partition their model into blocks with equal sums of parameter-wise KL-divergences and transmit the block intervals to the federator. The federator aggregates the indices of all the clients, and broadcasts the updated block allocation.

We propose in this work a low complexity solution that adapts the block size according to the average KL-divergence per block (Adaptive-Avg). This alleviates the cost of computing and transmitting the exact block partitions, where the transmission of each block size requires log 2 ( b max ) bits, with b max the maximum pre-defined block size. Instead, the transmission of one size is enough in our solution. If the average KL per block D KL ( q t i,b ∥ p t i,u,b ) deviates more than a given factor, the clients request to update the blocks. In the next iteration, each client proposes a block size, and the federator averages and broadcasts an updated size.

E Additional Experimental Details

We use the cross-entropy loss and a batch size of 128 in all our experiments. We use Adam [Kingma and Ba, 2015] as an optimizer with learning rate η = 0 . 0003 for all non-stochastic methods, and η = 0 . 1 for probabilistic mask training. For non-stochastic FL, we use a federator (server) learning rate of 0 . 1 , i.e., the clients' gradients are averaged, and the federator updates the global model with learning rate 0 . 1 , and with a learning rate of 0 . 005 for BICOMPFL-GR with SignSGD. For M3, we use a federator learning rate of 0 . 02 to obtain reliable results. For LIEC and CSER, we use an average period of 50 global iterations (cf. [Cheng et al., 2024, Xie et al., 2020]). For M3, we use TopK with K = ⌊ d/n ⌋ . To run the simulations, we use a cluster of different architectures, which we list in the following table.

The details of the CNN architectures used in our experiments are summarized in the following. The parameter count is 61706 for LeNet5, 1933258 for 4CNN, and 2262602 for 6CNN.

For the sake of clarity, in the paper we restrict the analysis to a fixed number of importance samples n IS, block sizes B , and choice of priors p t i,u , p t i,d . Our experiments have shown that, while increasing n IS beyond the ones used in our algorithms slightly improves the convergence over the number of epochs, the convergence with respect to the communication cost did not significantly improve. The block size is mainly limited by the system resources at hand, and one would choose the largest possible for best efficiency while complying with memory resources. We investigated many different

Algorithm 4 Local Training at Client i

Require: Model θ

ˆ i,t

1: Map model to scores in the dual space: s (0) i,t = σ -1 ( ˆ θ i,t ) = log ( ˆ θ i,t 1 -ˆ θ i,t )

2: for Local iterations m ∈ [ L ] do

3: s ( m ) i,t = s ( m -1) i,t -η ∇ s ( m -1) i,t F ( ˆ θ ( m -1) i,t , D i ) , where ˆ θ ( m -1) i,t = σ ( s ( m -1) i,t )

4: end for

5: Map back to primal space: q t i = σ ( s ( L ) i,t )

prior choices and found the former global model to be reasonably good in almost all cases. With high heterogeneity, it might be beneficial to use different convex combinations as priors, which mix the former global model with the latest posterior estimate of a certain client, but the gains we experienced were minor. Hence, we settled on the former global estimate for simplicity in presenting the algorithm.

F Federated Probabilistic Mask Training

The idea in federated probabilistic mask training (FedPM) Isik et al. [2023] is to collaboratively train a probabilistic mask that determines which weights to maintain from a randomly initialized network. The motivation stems from the lottery-ticket hypothesis [Frankle and Carbin, 2019], which claims that randomly initialized networks contain sub-networks capable of reaching accuracy comparable to that of the full network. The weights w of the network are randomly initialized at the start of training, and remain fixed. The federator and clients only train a mask, which determines for each parameter whether it is activated or not, i.e., identifying an efficient subnetwork within the given fixed network. The probabilistic masks θ t are described by Bernoulli distributions, i.e., θ t ∈ [0 , 1] d contains a Bernoulli parameter to be trained for each weight of the network. These parameters determine the probability of retaining the corresponding weights. During inference, the weights w are masked with samples x t ∈ { 0 , 1 } d ∼ θ t from the distribution θ t , i.e., the inference is conducted on a network with weights w ⊙ x t . In FedPM, clients sample from their locally trained models, and send these samples to the federator, which, in turn, updates the global model by averaging these samples. The communication cost of this scheme is fixed for all iterations, even though the communication cost can be reduced since the KL-divergence between the global model and the locally trained models diminishes as the training progresses.

We adopt the following federated learning procedure for collaboratively learning network masks, and highlight in the following the parallels to mirror descent by referring to primal and dual spaces. Starting from a common model θ 0 , at iteration t , each client i locally trains the model ˆ θ i,t in L local iterations. To enable gradient descent, the model ˆ θ i,t is mapped to scores s (0) i,t in a dual space by the inverse Sigmoid function s (0) i,t = σ -1 ( ˆ θ i,t ) = log( ˆ θ i,t ) -log(1 -ˆ θ i,t ) . The scores are then trained for L local iterations m ∈ [ L ] by computing the gradient ∇ s ( ℓ -1) i,t F ( ˆ θ ( m -1) i,t , D i ) , where the straight-through estimator is used to compute the gradient of the non-differentiable Bernoulli sampling operation based on the distribution ˆ θ ( m -1) i,t = σ ( s ( ℓ -1) i,t ) , i.e., the gradient equals the Bernoulli parameter. By mapping the model back to the primal space, each client i obtains a model update in terms of a posterior q t i = σ ( s ( L ) i,t ) . The client training process is summarized in Algorithm 4.

G Minimal Random Coding (MRC)

Isik et al. [2024] proposed a method, called KL minimization with side information (KLMS), to reduce the cost of transmitting the local models q t i to the federator. Consequently, the communication cost depends on the KL-divergence between the desired distribution and the common prior. This method utilizes the common side information available at both the clients and the federator, as well as shared randomness. The idea is that instead of sampling locally and sending the samples to the

federator, the federator in the KLMS method samples from the desired distribution through MRC. In a nutshell, MRC [Havasi et al., 2019] is based on importance sampling [Srinivasan, 2002] and makes use of a common prior to sample from a desired distribution. Consider two distributions P and Q , where P is known to both parties, and Q is only known to the client. To make the federator sample from Q , both parties sample n IS samples { X i } i ∈ [ n IS ] from P . The client forms an auxiliary distribution W ( i ) = Q ( X i ) /P ( X i ) ∑ n IS i =1 Q ( X i ) /P ( X i ) capturing the importance of the samples. A sample from W is fully described by its index i , which can be transmitted with log 2 ( n IS ) bits, and approximates a sample from Q . Chatterjee and Diaconis [2018] shown that importance sampling with posterior Q and prior P requires n IS to be in the order of Θ(exp(D KL ( Q ∥ P ))) , where D KL ( Q ∥ P ) denotes the KL-divergence between distributions Q and P . In what follows, we will also denote the KL-divergence between two Bernoulli distributions Q and P with parameters q and p by dKL ( q || p ) .

H Additional Experiments

We provide in the following experiments for both uniform (i.i.d.) and heterogeneous (non-i.i.d.) data distributions for training LeNet5 and a 4-layer CNN on MNIST, a 4-layer CNN on Fashion MNIST, and a 6-layer CNN on CIFAR-10. The details of the neural networks can be found in Tables 2 to 4. For each setting and method depicted, we show the average of three simulation runs with different seeds. We plot for each setting the test accuracies over the communication cost in bits, and the maximum test accuracy over the bitrate. We provide tables summarizing the maximum test accuracies with their standard deviation over multiple runs, the total bitrates and the bitrates split into uplink and downlink. The overall bitrates per parameter (bpp) are computed assuming point-to-point links between all participants, i.e., uplink and downlink costs have equal weight. For the case when a broadcast (BC) link between the federator and the clients is available, the bitrate per parameter for all baseline schemes reduces by a factor of n . BICOMPFL-GRprofits similarly from the broadcast link, but BICOMPFL-PRcannot profit due to the absence of shared randomness, giving the same overall bitrate compared to the point-to-point link scenario. We highlight for each of the measures the scheme with the best result. Consistently throughout all experiments, BICOMPFL achieves order-wise savings in the bitrates per parameter while reaching state-of-the-art accuracies in the classification task. While the sampling can introduce an additional computational overhead depending on the implementation, the storage cost is similar to the baselines. Since we leverage as priors the former global model, the additional storage cost incurred is limited to storing until the next iteration the estimate of the former global model at each client, i.e., where the training started, which is usually not a bottleneck. This can be cheaper than some baselines, which require storing data for momentum and error-feedback.

10 5

10 6

10 7

10 8

10 9

10 10

Communication Cost in Bits

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.93

0.94

0.95

0.96

0.97

0.98

0.99

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

For LeNet5 on MNIST, it can be observed that all our proposed methods converge significantly faster to satisfying accuracies with respect to the communication cost, while achieving higher maximum accuracies after 200 epochs than the non-stochastic baselines. Partitioning the model on the downlink can help to further reduce the communication cost with only a minor loss in performance, especially in the i.i.d. setting. For non-i.i.d. data distribution, the loss in performance is larger than for i.i.d.

distribution. However, at the beginning of the training, the model improves faster with respect to the communication cost than all other schemes. The bitrates are comparable for all our methods, with the exception of BICOMPFL-PR-Fixed-SplitDL. Further, BICOMPFL-GR-Reconst-Fixed does not suffer notable performance degradation from employing an additional MRC step (especially for i.i.d. data allocation).

10 5

10 6

10 7

10 8

10 9

10 10

Communication Cost in Bits

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.70

0.75

0.80

0.85

0.90

0.95

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

For 4CNN trained on MNIST, the differences between the proposed approaches become more visible. In the i.i.d. setting, we can observe that the adaptive block allocations (both Adaptive and AdaptiveAvg) can drastically reduce the average bitrate in BICOMPFL-GR. Partitioning the model in the downlink (BICOMPFL-PR-Fixed-SplitDL) improves the accuracy over bitrate significantly compared to BICOMPFL-PR-Fixed.

10 6

10 7

10 8

10 9

10 10

10 11

Communication Cost in Bits

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.990

0.991

0.992

0.993

0.994

0.995

0.996

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

10 6

10 7

10 8

10 9

10 10

10 11

Communication Cost in Bits

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.92

0.93

0.94

0.95

0.96

0.97

0.98

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

In the non-i.i.d. case of 4CNN on MNIST, the adaptive average allocation strategy provides a significant reduction in the bitrate for BICOMPFL-GR, with similar loss in the accuracy as SplitDL for BICOMPFL-PR. In this setting, it is also apparent that the reconstruction in BICOMPFL-GR degrades the performance without gains in the bitrate compared to the proposed Algorithm 1.

10 6

10 7

10 8

10 9

10 10

10 11

Communication Cost in Bits

0.2

0.4

0.6

0.8

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.890

0.895

0.900

0.905

0.910

0.915

0.920

0.925

0.930

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

10 6

10 7

10 8

10 9

10 10

10 11

Communication Cost in Bits

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.78

0.80

0.82

0.84

0.86

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

The results for Fashion MNIST are similar compared to the MNIST case. However, it becomes clear that BICOMPFL-PR can significantly suffer from the unavailability of shared randomness in terms of the achieved accuracy when data is highly heterogeneous.

For 6CNN trained on CIFAR-10, the negative effects of missing global shared randomness and reconstructing in the case of BICOMPFL-GR are prominent. For non-i.i.d. data distributions, the adaptive average allocation shows improvements over the fixed or the average block allocation. Partitioning the model is not a viable option in this setting, especially under non-i.i.d. data.

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Communication Cost in Bits

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

10

1

10 0

10 1

Bitrate p. P .

0.625

0.650

0.675

0.700

0.725

0.750

0.775

0.800

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(b) Test Accuracy over Bitrate

10 6

10 7

10 8

10 9

10 10

10 11

10 12

Communication Cost in Bits

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

10

1

10 0

10 1

Bitrate p. P .

0.40

0.45

0.50

0.55

0.60

0.65

Maximum Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) Test Accuracy over Communication

(b) Test Accuracy over Bitrate

For completeness, we present in Fig. 11 the test accuracies over the number of trained epochs for all scenarios considered above. The setting of interest to this work is that of limited communication cost, and in particular, which performance is achievable given a fixed communication budget. Nonetheless, we can find that our proposed methods are not inferior in convergence speed over epochs compared to the baselines.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(a) MNIST LeNet i.i.d.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(b) MNIST LeNet non-i.i.d.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(c) MNIST 4CNN i.i.d.

10 0

10 1

10 2

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(d) MNIST 4CNN non-i.i.d.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(e) Fashion MNIST 4CNN i.i.d.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

1.0

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(f) Fashion MNIST 4CNN non-i.i.d.

10 0

10 1

10 2

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(g) Cifar-10 6CNN i.i.d.

10 0

10 1

10 2

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

FedAvg

Doublesqueeze

Memsgd

Liec

Cser

Neolithic

M3

BiCompFL-GR-Adaptive

BiCompFL-GR-Adaptive-Avg

BiCompFL-GR-Fixed

BiCompFL-GR-Reconst-Fixed

BiCompFL-PR-Fixed

BiCompFL-PR-Fixed-SplitDL

(h) Cifar-10 6CNN non-i.i.d.

I Ablation Studies

I.1 Number of Clients

We study in what follows the sensitivity to various hyperparameters of our algorithms. For comparability, we conduct all experiments on the model 4CNN, Fashion MNIST, and i.i.d.data. We plot for all experiments the accuracies over the number of epochs, and over the communication cost in bits. We first evaluate in Fig. 12 the effectiveness of BICOMPFL-PR and BICOMPFL-PR for different numbers of clients. It can be found that both algorithms exhibit satisfying performance even for n = 50 , given that the same data is now distributed on more clients. The overall communication cost increases by roughly the factor of the increase in the number of n . To illustrate this further, we additionally plot in Fig. 13 the bitrates per parameter.

10 0

10 1

10 2

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

BiCompFL-GR-C10

BiCompFL-GR-C50

BiCompFL-PR-C10

BiCompFL-PR-C50

(a) Test Accuracy over Epochs

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

BiCompFL-GR-C10

BiCompFL-GR-C50

BiCompFL-PR-C10

BiCompFL-PR-C50

(b) Test Accuracy over Communication Cost

10

1

10 0

Bitrate p. P .

0.900

0.905

0.910

0.915

0.920

0.925

Maximum Test Accuracy

BiCompFL-GR-C10

BiCompFL-GR-C50

BiCompFL-PR-C10

BiCompFL-PR-C50

I.2 Optimization of the Prior

As described in the main body of the paper, BICOMPFL-PR allows for optimizing the choice of the prior at the clients by optimizing the convexity parameter λ that mixes the global model estimate with the posterior transmitted by the client an iteration ahead, i.e., p t i,u = λ ˆ θ i,t +(1 -λ )ˆ q t i to reduce the communication cost. To evaluate the potential of this method, we optimize λ so that it minimized the KL-divergence between the current posterior q t i (to be transmitted) and the prior p t i,u , representative for the uplink communication cost. The KL-minimizing λ is transmitted to the federator, which is necessary for the federator to reconstruct the importance samples. This optimization is conducted at each iteration individually at the clients. We present in Fig. 14 the performance of this method compared with the algorithms that use as priors exclusively the global model estimates of the clients. Note that optimizing the prior individually at the clients is only possible for BICOMPFL-PR ˙ We plot

the performance of BICOMPFL-GR for reference only. To assess the potential, we ignore for the moment the cost of transmitting λ , which could be reduced by further compression techniques and leveraging the inter-round dependencies of the choice of λ .

10 0

10 1

10 2

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

BiCompFL-GR

BiCompFL-PR

BiCompFL-PR-OP

(a) Test Accuracy over Epochs

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

BiCompFL-GR

BiCompFL-PR

BiCompFL-PR-OP

(b) Test Accuracy over Communication Cost

It can be found that, while optimizing the prior improves the accuracy over epochs and with respect to the communication cost compared to BICOMPFL-PR the improvements are rather insignificant. We therefore present for clarity the algorithm with a fixed choice of the prior as the former global model estimate, which additionally reduce the computation overhead at the clients by avoiding the optimization over λ . Nonetheless, we note that in certain edge cases, there can be merit in the optimization approach, for instance when the number n DL of samples on the downlink is very small, and hence the global model estimate is inaccurate.

I.3 Number of Samples

We continue to assess the impact of the number n DL of samples on the downlink. We therefore evaluate the performance of BICOMPFL-PR for n DL ∈ { 5 , 10 , 20 } . We evaluate the differences on BICOMPFL-PR ˙ The results in Fig. 15 reflect the obvious: the larger n DL, the better the accuracy when plotted over the number of epochs. On the contrary, the larger n DL, the larger the communication cost per epoch. The final accuracies do not show substantial differences, and hence, n DL = 5 is sufficient in this setting. To avoid assessing our method overly optimistic and provide a fair comparison to other methods, we choose n DL = 10 in all our experiments, noting that the communication can further be reduced in certain scenarios by lowering n DL without notable performance loss.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-PR-DL10-UL1

BiCompFL-PR-DL20-UL1

BiCompFL-PR-DL5-UL1

(a) Test Accuracy over Epochs

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-PR-DL10-UL1

BiCompFL-PR-DL20-UL1

BiCompFL-PR-DL5-UL1

(b) Test Accuracy over Communication Cost

I.4 Block Size

We compare in Fig. 16 the performance of BICOMPFL-GR for different block sizes BS = d/B ∈ { 128 , 256 , 512 } . As expected, fixing n IS, larger block sizes worsen the performance of the algorithm when evaluated over the number of epochs. However, larger block sizes simultaneously reduce the communication cost, and can hence be beneficial in many scenarios. However, we also note that larger block sizes comes at the expense of increases sampling complexities, and hence, the maximum block sizes are also dominated by the resources of the clients and the federator.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-GR-BS256

BiCompFL-GR-BS128

BiCompFL-GR-BS512

(a) Test Accuracy over Epochs

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-GR-BS256

BiCompFL-GR-BS128

BiCompFL-GR-BS512

(b) Test Accuracy over Communication Cost

I.5 Number of Importance Samples

In Fig. 17, we study the sensitivity of our algorithms with respect to the number of importance samples n IS at the example of BICOMPFL-GR. While larger number of n IS slightly improves the performance as of the epoch number, the improvements do not outweigh the additional communication costs. Overall, our algorithm proves rather stable within reasonable ranges for n IS . We fix in all our experiments n IS = 256 , presenting a good trade-off between performance and efficiency.

10 0

10 1

10 2

Epochs

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-GR-S256

BiCompFL-GR-S128

BiCompFL-GR-S512

(a) Test Accuracy over Epochs

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-GR-S256

BiCompFL-GR-S128

BiCompFL-GR-S512

(b) Test Accuracy over Communication Cost

I.6 Learning Rate

Our main claims are centered around the per-client bitrates per parameter, rendering the choices of learning rate secondary to our reasoning. Nonetheless, we tune the learning rates of all methods so that the baselines and BICOMPFL achieve roughly the same final accuracies, allowing a fair comparison of resulting communication costs. We analyze the impact of the learning rate choice

10 5

10 6

10 7

10 8

Communication Cost in Bits

0.2

0.4

0.6

0.8

Test Accuracy

BiCompFL-GR,

=0.1

BiCompFL-GR,

=0.01

BiCompFL-GR,

=0.05

BiCompFL-GR,

=0.2

BiCompFL-GR,

=0.5

on BICOMPFL in Fig. 18, for η ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } . It is particularly noteworthy that BICOMPFL exhibits stable performance across most learning rates we study, which we attribute to the regularization effects that occur in stochastic FL, detailed in the main body of the paper. Only for η = 0 . 01 , the final performance is decreased, indicating that BICOMPFL is not able to escape local optima in this setting. Although η = 0 . 05 provides the best communication efficiency, we choose a moderate learning rate of η = 0 . 1 not to overestimate our method compared to other approaches.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and the introduction are later justified in detail by the introduction of the schemes in Section 4 together with additional details in Appendices C and D, the various experiments in Section 4 and Appendix H, the formal theoretical guarantees (cf. Section 5), and the proofs provided in Appendix A.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The theoretical and experimental limitations are discussed in the paper. For instance, we discuss in Sections 3 and 4 how the availability of shared randomness affects our protocols. Further, we discuss in Section 5 the assumptions required for our main results to hold. We provide in Appendix I details on the impact of various hyperparameters, including the limitations of optimizing the prior distributions, and how the block size is affected by the computational resources. The space constraints didn't allow for adding a separate section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover

limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are provided along with the statements of our theoretical results. We provide the full set of proofs in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides all experimental details to reproduce the results in Sections 3 and 4 and the corresponding Appendices D to F. The code to reproduce our experiments is included in the supplementary material. This includes parameter files for the various experiments and recommended environments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The source code required to reproduce our experiments is included in the supplementary material, including parameter files for the various experiments and instructions on how to use the provided code and how to prepare the publicly available datasets. We also provide the Python environment used to generate the results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all training and test details necessary to understand the results in Section 4 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Each experiment is repeated for multiple runs to provide statistical confidence. The depicted results are averaged performance measures, consistently shown together with their respective standard deviations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The exact details of our simulation clusters are provided in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms with the NeurIPS Code of Ethics. The research does not involve any human subjects or participants, and all datasets used are standard and available to the public.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper presents a method for bi-directional compression in stochastic (Bayesian) FL, with the aim of reducing communication overhead and improving scalability. The contribution is methodological and does not target a specific downstream application. While the work does not explicitly discuss societal impacts, such efficiency improvements may enable more practical and energy-efficient deployment of machine learning systems in resource-constrained environments, such as edge devices, or indirectly support privacypreserving machine learning in settings such as medical healthcare. We do not foresee any immediate negative societal impacts arising from the work as presented.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets are properly credited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Alongside the paper, we introduce new source code that we provide as supplementary material. This material is well documented with particular instructions on how to use the provided framework.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
comments: SUMMARY:
The paper considers stochastic federated learning to reduce communication costs in both uplink and downlink communication. The techniques are based on Minicam random coding, an existing remote source generation tool. Both theoretical guarantees and empirical validations are provided.

STRENGTHS AND WEAKNESSES:
** Strengths**:

- The paper is well written and well structured.
- The contribution is clear, and the literature review is thorough.
- The work makes a solid contribution, both from theoretical and experimental perspectives.

** Weaknesses**:

There are no obvious major weaknesses. I would suggest the author consider also trying other remote source generation schemes to see if any of them achieve better results in FL tasks.

QUESTIONS:
I have a question about the choice of the remote source generation scheme. The authors used Minicam random coding, which is non-exact: the output distribution is distorted in the sense that the remotely generated sample only follows $\hat{Q}$, rather than $Q$ exactly.I believe the authors are aware of this (as noted in lines 96–97). My question is whether the authors have considered exact schemes, such as the Poisson functional representation by Li and El Gamal (2018) or dithered-quantization-based schemes. It is fine if these were not considered, I'm just curious whether exactness might make a difference in FL tasks, for example in terms of accuracy.

LIMITATIONS:
There are no new limitations from my side. The authors have already discussed some limitations, e.g.,  the availability of shared randomness and some required assumptions.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a bidirectional compression method for a wild federated learning setting in which uplink and downlink communications are both constrained. The underlying motivation is connected to the Bayesian federated learning mechanism, which employs an alternative view that the masks of weights are learned.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n---\n*  The paper is well-written and the motivation and algorithm design are both convincing. There are tons of additional analyses that thankfully address most of my preliminary concerns raised during reading the manuscript.\n* The consideration of uplink bottlenecks and global/private randomness, both of which are overlooked in academic papers, is impressive.\n* The distributional convergence of a global model is sound and the analysis of communication cost related to the divergence is also intriguing.\n* The empirical validation is conducted extensively on both IID and non-IID scenarios using widely-used benchmark datasets with moderate number of clients.\n\n### Weaknesses\n---\n* Based on Algorithm 1, the proposed method seems not support partial client pariticpation, which is often required in low-bandwith settings.\n* To better illustrate the practical strength and justification of the small number of clients in the experiments, the authors could narrow down the target federation setting to a cross-silo federated learning setting (see Table 1 of Kairouz et al., 2018), or specify/exemplify plausible scenarios in which the proposed methods are most applicable.\n* (minor question) Is there any (theoretical) tips to set the number of quantization intervals, $s$?\n\n- (Kairouz et al., 2018) Advances and Open Problems in Federated Learning\n\nQUESTIONS:\nPlease refer to the *weaknesses* subsection above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the significant communication overhead in both uplink (client-to-server) and downlink (server-to-client) directions for ​**stochastic (Bayesian) Federated Learning (FL)​**, where probabilistic models are trained to capture parameter uncertainty. Unlike prior work focused on deterministic FL or unidirectional compression, the authors propose ​**BICOMPFL**, a novel framework leveraging ​**remote source generation**​ via ​**Minimal Random Coding (MRC)​**​ to enable bidirectional communication efficiency. \n\nThe core insight exploits shared randomness and ​**side information**​ (in the form of prior distributions) to allow clients and the server to sample from each other's local/global posteriors rather than transmitting explicit updates. This transforms communication into a sampling problem, where transmission costs scale with the KL-divergence between updated distributions and their priors. Two algorithm variants are introduced: ​**BICOMPFL-GR**​ (using global shared randomness) enables efficient relay-based compression by synchronizing priors across all parties, while ​**BICOMPFL-PR**​ (using private randomness per client-server pair) handles settings without globally synchronized randomness, albeit with slightly higher overhead. \n\nKey contributions include:  \n1) ​**Inherent communication regularization**​ through a mirror-descent optimization structure, treating communication costs as part of the learning objective.  \n2) ​**Substantial empirical gains**: Experiments on MNIST, Fashion MNIST, and CIFAR-10 demonstrate ​**5–32× reduction in total communication costs**​ (bits per parameter) while maintaining competitive accuracy against strong baselines (e.g., FedAvg, Doublesqueeze, M3). Adaptive block allocation strategies further optimize costs.  \n[](@replace=1)\n  \n3) ​**Theoretical innovations**: Refined bounds for MRC compression (for Bernoulli distributions) quantify uplink/downlink trade-offs, and a new contraction property is proven for biased MRC operators, enabling convergence guarantees in conventional FL settings.  \n4) ​**Methodological generality**: Extension to conventional FL via stochastic quantization (e.g., SignSGD) shows broader applicability without error feedback mechanisms.  \n\nThe work bridges stochastic FL with practical communication constraints, demonstrating that principled probabilistic modeling and remote sampling can dramatically reduce bidirectional bandwidth while preserving model performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper demonstrates ​**exceptional technical quality**​ through rigorous theoretical development and comprehensive experimental validation. The introduction of ​**BICOMPFL**​ represents a significant advancement in stochastic federated learning, with novel bidirectional compression mechanisms based on Minimal Random Coding (MRC) and remote source generation. The theoretical contributions are particularly strong, including refined MRC bounds for Bernoulli distributions and a new contraction property for biased compressors (Lemma 1), which enables convergence guarantees in conventional FL settings. Experimentally, the work is thorough, validating claims across three datasets (MNIST, Fashion MNIST, CIFAR-10) and both i.i.d. and non-i.i.d. settings, consistently demonstrating ​**5–32× communication reduction**​ without accuracy degradation. The adaptive block allocation strategies and extension to conventional FL showcase methodological versatility. The significance is high, addressing a critical bottleneck in FL deployment—communication overhead—through a principled probabilistic framework that inherently treats communication as an optimization objective via mirror descent. The approach is highly original, being the first to solve bidirectional compression in stochastic FL through remote sampling and prior-based side information. [](@replace=1)\n visually encapsulates this breakthrough by showing order-of-magnitude bitrate improvements over baselines.\n\n### Weaknesses\nWhile technically solid, the paper has ​**clarity issues**​ in theoretical sections where derivations become overly dense (e.g., Lemma 2 proof in Appendix A). The distinction between BICOMPFL-GR and BICOMPFL-PR could be better motivated visually—a conceptual diagram would help readers grasp the remote generation workflow. Experimentally, the ​**quality is slightly compromised by limited scope**: Evaluations use only small CNNs rather than modern architectures (e.g., ResNets), and comparisons exclude recent Bayesian FL methods like FedBE. The significance claims about scalability lack validation beyond 50 clients, and energy/runtime metrics—critical for edge deployment—are absent. The theoretical ​**originality is constrained to Bernoulli distributions**, with no discussion of extending MRC analysis to Gaussian posteriors common in Bayesian FL. The ​**significance of broader impacts**​ is underdeveloped: While reduced communication benefits sustainability, potential privacy risks from compressed updates (e.g., membership inference) aren't addressed. [](@replace=2)\n reveals another weakness: convergence instability in BICOMPFL-PR under high heterogeneity, which isn't sufficiently analyzed. Finally, adaptive block allocation's overhead isn't quantified, leaving practical trade-offs unclear.\n\nQUESTIONS:\n​**1. Scalability Validation to Modern Architectures**​  \nThe experimental validation relies exclusively on small CNNs (LeNet, 4-6 layer CNNs). To strengthen claims of broader applicability, please evaluate BICOMPFL on larger models (e.g., ResNet-18/50, ViT-Tiny) and complex datasets (e.g., ImageNet subset or CIFAR-100). Specifically, quantify how communication costs scale with model size and heterogeneity under your framework. \n\n​**2. Generalization of Theoretical Guarantees**​  \nThe theoretical analysis (Section 5) focuses entirely on Bernoulli distributions for posterior/prior modeling. Clarify whether Lemma 1’s contraction property and Theorem 1’s communication bounds extend to other distributions common in Bayesian FL (e.g., Gaussian posteriors). Provide either a proof sketch for multivariate Gaussians or empirical validation via Gaussian-based stochastic FL experiments. Should these extensions hold, emphasize their implications in the text; if not, explicitly state the limitations of the current theory and propose mitigation strategies (e.g., parameter transformation techniques).\n\n​**3. Algorithm Workflow Clarity**​  \nThe distinction between BICOMPFL-GR and BICOMPFL-PR is not intuitively conveyed in the pseudocode (Algorithms 1-2). Consider adding a schematic diagram illustrating the bidirectional remote sampling workflow, highlighting critical differences in prior synchronization, index relaying, and noise propagation. [](@replace=1)\n demonstrates performance gaps but lacks visual workflow context. A figure showing how shared randomness reduces downlink variance would clarify why BICOMPFL-GR outperforms BICOMPFL-PR.\n\n​**4. Privacy and Fairness Implications**​  \nThe paper does not address how compressed bidirectional updates affect privacy vulnerabilities (e.g., membership inference/gradient inversion attacks) or fairness across heterogeneous clients. Analyze:  \n- How MRC-based sampling compares to DP-SGD in privacy-utility tradeoffs  \n- Whether adaptive block allocation biases aggregation against clients with high KL-divergence  \n- Test accuracy variance across demographic groups in non-i.i.d. settings  \nIncluding a \"Broader Impacts\" section addressing these points would align with NeurIPS ethics guidelines and strengthen societal relevance.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes BICOMPFL, a family of federated learning (FL) algorithms that cut both uplink and downlink communication by exploiting the stochastic/Bayesian view of model parameters and using Minimal Random Coding (MRC) with shared priors (“side information”) to remotely generate samples at the receiver instead of transmitting full local samples or model updates. The paper also provides experimental results to show the approach cuts total communication by about 5–32× while maintaining competitive accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper aims to reduce the communication overhead in FL training and proposes a probabilistic compression method, which is novel to the my knowledge.\n2. The paper also provides the convergence rate to show the proposed algorithm leads to similar convergence rate as FedAvg.\n3. The paper includes some empirical evaluations and ablation studies to show the effectiveness of the algorithm.\n\nWeakness:\n1. Empirical validation limited to small vision benchmarks.\n2. Minimal Random Coding needs n_{IS} samples which scales exponentially w.r.t. the KL divergence between the prior and posterior, which makes the high-dimensional models hard to reach the condition.\n3. The most communication-efficient variant (BICOMPFL-GR) assumes all clients share global pseudo-randomness, but this shared randomness is hard to maintain in real world FL systems, establishing and maintaining perfectly synchronized randomness can be difficult. (But this is a minor point compared to 1 and 2)\n\nQUESTIONS:\nQuestions:\n1. Can you provide evaluations results on large-scale benchmarks and models?\n2. On-device, how does BICOMPFL’s runtime and energy compare to vanilla FedAvg?\n3. When client data are extremely heterogeneous, does the KL gap explode for some clients?\n4. Could BICOMPFL be stacked with other sparsification or structured pruning-based compression methods?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-21"}