Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Authors: Terje Mildner
Oliver Hamelijnck
Paris Giampouras
Theodoros Damoulas
Venue: ICML 2025 spotlightposter (2025)
Keywords: Federated Learning
Probabilistic Machine Learning
Model Misspecification
Robustness
Generalised Variational Inference
Primary area: probabilistic_methods->variational_inference

Abstract: We introduce FedGVI, a probabilistic Federated Learning (FL) framework that is robust to both prior and likelihood misspecification. FedGVI addresses limitations in both frequentist and Bayesian FL by providing unbiased predictions under model misspecification, with calibrated uncertainty quantification. Our approach generalises previous FL approaches, specifically Partitioned Variational Inference (Ashman et al., 2022), by allowing robust and conjugate updates, decreasing computational complexity at the clients. We offer theoretical analysis in terms of fixed-point convergence, optimality of the cavity distribution, and provable robustness to likelihood misspecification. Further, we empirically demonstrate the effectiveness of FedGVI in terms of improved robustness and predictive performance on multiple synthetic and real world classification data sets.

Federated Generalised Variational Inference:
A Robust Probabilistic Federated Learning Framework
Terje Mildner' Oliver Hamelijnck
Paris Giampouras! Theodoros Damoulas 12
Abstract
cally and averaging updates the server. This has sparked a large body of research on issues such as communication ef- ficiency, data privacy, and data heterogeneity across clients (Hamer et al., 2020; Malinovsky et al., 2020; Reddi et al., 2021; Chen et al., 2022; Tenison et al., 2023; Tziotis et al., 2023; Li et al., 2024; Demidovich al., 2025). There has been some work addressing robustness adversarial clients (Allouah et al., 2024; Bao et al., 2024) and data and system heterogeneity (Chen et al.. 2022; Zhao et al., 2023; Heikkilä et al., 2023). However, these only provide point estimates, and do not allow principled uncertainty quantification, as required in many FL applications (Jonker et al., 2024).
We introduce FIDGVI, a probabilistic Federated Learning (FL) framework that is robust to both prior and likelihood misspecification. FEDGVI addresses limitations in both frequentist and Bayesian FL by providing unbiased predictions under model misspecification, with calibrated un- certainty quantification. Our approach generalises previous FL approaches, specifically Partitioned Variational Inference (Ashman et al., 2022), by allowing robust and conjugate updates, decreas- ing computational complexity at the clients. We offer theoretical analysis in terms of fixed-point convergence, optimality of the cavity distribution, and provable robustness to likelihood misspecifi- cation. Further, we empirically demonstrate the effectiveness of FEDGVI in terms of improved ro- bustness and predictive performance on multiple synthetic and real world classification data sets.
In contrast, Bayesian FL approaches aim to update be- liefs of global model with data partitioned across clients. This largely builds on distributed inference methods such as the Bayesian Committee Machine (Tresp, 2000), paral- lel MCMC (Ahn et al., 2014; Mesquita et al., 2020), or Divide&Conquer SMC (Chan et al., 2023). Expectation Propagation (Minka, 2001; Vehtari et al., 2020) is natu- rally applicable the distributed setting where local sites are iteratively refined. This requires computing the cavity distribution that removes local sites from the current approx- imation. Partitioned Variational Inference (PVI, Bui et al., 2018; 8; Ashman et al., 2022) takes this idea and proposes a distributed variational inference algorithm, which has been extended through MCMC (Guo et al., 2023) and Stochas- tic Gradient Langevin Dynamics (SGLD) (Mekkaoui et al., 2021). Whilst these approaches quantify uncertainty, they are susceptible to model misspecification which can lead to inaccurate, overconfident predictions (Bernardo & Smith, 2000; Bissiri et al., 2016; Knoblauch et al., 2022).
1. Introduction
Federated learning (FI.) a framework for the collaborative training of a global model by collection of clients, without requiring proprietary data to be shared with central server or other participating clients (McMahan et al., 2017). This decentralised approach allows FL to be used on applications with strict data privacy constraints, such as in finance or healthcare (Kairouz et al., 2021). However, due to the sensi- tive nature and complexity of these domains, both privacy and robustness model misspecification are paramount.
Current approaches to FL are inherently non-robust to model misspecification which leads to compromised performance and uncalibrated uncertainty quantification. We address these challenges by departing from the traditional Bayesian paradigm and propose distributed Generalised Variational Inference framework that allows to deal with model mis- specification. In summary, our contributions are:
The frequentist formulation of Fl. aims minimise global loss function by aggregating local gradients from clients. Early works include Federated Averaging (FEDAVG, McMa- han et al., 2017) which iterates between training clients lo-
University of Warwick, Department of Computer Science, Coventry, United Kingdom University of Warwick, Department of Statisties, Coventry, United Kingdom. Correspondence to: Terje Mildner
warwick.ac.uk>..
We introduce Federated Generalised Variational In- ference (FEDGVI), a family of robust probabilistic algorithms for federated learning.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
We prove that FEDGVI is robust to likelihood misspec- ification (Theorem 4.12).
include Expectation Propagation (Minka, 2001; Opper & Winther, 2005; Hasenclever al., 2017; Vehtari al., 2020), and consensus based Monte Carlo (Scott et al., 2016). In the Federated setting this assumption is often violated, as data is not split homogeneously and IID across participating de- vices. From this perspective, most approaches to Bayesian FL can be categorised into finding an approximate posterior through variational inference (Corinzia et al., 2021; Ash- man al., 2022; Kassab & Simeone, 2022; Heikkilä et al., 2023; Hassan et al., 2024; Vedadi et al., 2024; Swaroop et al., 2025), Markov Chain Monte Carlo (Al-Shedivat et al., 2021; Mckkaoui et al., 2021; Kotelevskii et al., 2022; Guo et al., 2023; Hasan et al., 2024), Gaussian Processes (Achi- tuve et al., 2021), or directly learning a Bayesian neural network (Yurochkin et al., 2019; Zhang et al., 2022). Per- sonalised or hierarchical Bayesian FL (Kotelevskii al., 2022; Zhang et al., 2022; Kim & Hospedales, 2023; Hassan et al., 2023; 2024; Vedadi et al., 2024) allows for additional expressibility of client posteriors, especially under hetero- geneity. However, none of these are inherently robust to contamination and model misspecification.
We demonstrate that FEDGVI generalises standard approaches such as PVI and FEDAVG (Remarks 4.1 and 4.2) and theoretically justify the use of the cavity distribution (Theorem 4.9).
We prove that, under suitable conditions, VEDGVI con- verges to Generalised Bayesian posteriors (Lemma 4.6 and Proposition 4.10) that are computationally tractable.
We evaluate FEDGVI on a range of synthetic and real- world datasets, across multiple models, demonstrating improved robustness and predictive performance.
In Section 2 we define model misspecification and recall methods that mitigate it in the non-distributed setting. Sec- tion introduces our framework, which builds on these concepts and extends them to the federated setting. We analyse the theoretical properties of FEDGVI in Section 4, including provable robustness. Finally, Section 5 studies the empirical performance and gains of FEDGVI with multiple models and real world datasets such as Bayesian Neural Networks on MNIST and FASHIONMMIST..
Robust Bayesian Inference Although the existing Bayesian FL methods address some of the challenges of fed- crated learning, such as communication constraints and data heterogeneity, they still aim to approximate the Bayesian posterior, which in itself is a flawed objective under model misspecification (Walker, 2013; Berk, 1966; Bernardo & Smith, 2000). In the global, non-federated case, several methods have been proposed to combat misspecification in the Bayesian setting (Grünwald, 2012), with the most promising direction being Generalised Bayesian Inference (Hooker & Vidyashankar, 2014; Bissiri et al., 2016; Ghosh & Basu, 2016a; Jewson et al., 2018; Miller, 2021: Alquier, 2021; Knoblauch et al., 2022; Matsubara et al., 2022). In this work we capitalise on this front and bring robustness to model misspecification in the federated setting.
1.1. Related Work
Robust Frequentist Federated Learning In the freeuen- tist setting, building on the seminal paper of McMahan et al. (2017), many approaches have aimed at mitigating challenges in FL, such as robustness to adversarial servers through secure aggregation (Chen et al., 2022), to strag- glers (Tziotis et al.. 2023), heterogenous data in out-of- distribution generalisation (Tenison et al., 2023), heteroge- neous and asynchronous clients (Fraboni et al., 2023), or finding weaknesses in communications (Zhu et al., 2019; Zhao et al., 2023). More recently, work on robust server ag- gregations achieves robustness against Byzantine clients that aim to deteriorate model performance (Allouah et al., 2024; Bao et al., 2024). However these do not allow principled uncertainty quantification.
2. Preliminaries
2.1. Notation and Model Misspecification
Let (2, Po) be probability space where Po is the data generating process, generating the observable random vari- ables X,,.., X, = X taking values in the measurable space (E, Further, let Y be observable random vari- ables depending on X respectively, taking values (T, Denote their realisations {X, = Fj, Y; = yi ) which are assumed to be partitioned across M clients (xm. ym Im each of size n. Consider hypothesis measures Po where 0 takes values in (9, T), a measurable space, admitting densities Pa- We study elements of P(0), the set of all probability measures on (9, T), starting with prior II and updated to Q, dominated by some common measure μ., and
Federated Bayesian Inference Federated and distributed Bayesian methods aim to approximate the posterior as if it had been computed with the data of all clients available at a central server. Early work on distributed Bayesian inference includes Bayesian opinion pools (Genest, 1984; Carvalho et al., 2023), and the Bayesian Committee machine (Tresp, 2000), which aim to find consensus among a collection of Bayesian beliefs. Works that aim to operationalise this in the distributed setting, where data is split IID across clients,
Cooee to reproduce experiments be found https:// github. www.bb..omm/rrre jew MM/FdGVV..
2

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Algorithm 1 FEDGVV SERVER
Algorithm 2 FEDGVI CLIENT
1: Input: =0)). Q. D.
1: Input: ga " (0), 2, {xm: ym Lmm eh ( D
(0),
2: Define: 600 (0) = 0, ( (0) - 0, (0) (0) (0)
2: Optimise q //m (0) according to Equation (3)
3: for = I...., T do
3: Optimise qm (0) according to Equation (4)
4: for m = 1,,., M in parallel do
4: Set Am (0) according to Equation (5)
5: Am (0) CLCIINN (t-1) (0), 2, m)
(0),
5: Set e (0) +- 1 ( + m. (0)
6: end for
7: Set e (0) x ( + Em= Am (
6: return: Communicate A m (0) to SERVER
(0) 8: Optimise a (0) according to Equation (7)
9:
end for
(2016) and Miller (2021) formalised a coherent Bayesian framework using loss functions leading to Gibbs posteri- ors (Alquier et al., 2016). This was further utilised to deal with likelihood misspecification through robust losses, e.g Knoblauch et al. (2018). Let L : θ x E x Y - IR be such a loss, then the GBI posterior is given by:
admitting densities x and respectively. Naive Bayes up- dates πT (0) to (0) through
M n (0) π (0) - (opymmmmmm)))
Xm) /Z
(1)
[GBI (0) = π (0) 3EM T(ym; O xm) 1z (2)
M
where f Im== (m: Xm) [I(d0) is the marginal likelihood. Since we do not suppose that the prior II, nor the likelihood Pe are well specified, i.e. Po & P(O), we are in the M-open setting (Bernardo& & Smith, 2000), the model misspecified, and the Bayesian posterior inappropriate.
M with Z So expl- -11-- T.ym: xmxm ) [[(d).. Here, B E R is a learning rate parameter that determines how much weight we place on the observed data, similar to power posteriors in VI (Grünwald, 2012; Kallioinen et al., 2024). This recovers (0) when the loss the negative log-likelihood and 8 1.
2.2. Model Misspecification
There are several different ways we can think about model misspecification under the M-open assumption.
Generalised Variational Inference (GVT) In Knoblauch et al. (2022) GBI is generalised within variational frame- work that explicitly accounts for prior and likelihood mis- specification. Let D P(0) x P(0) + R_ be divergence then the GVI posteriors are defined as:
Prior Misspecification The traditional Bayesian paradigm assumes that the prior encodes the best available judgement about 0, which beyond simple settings, is never realised (Berger, 1985; Knoblauch et al., 2018). Such misspecification is common; e.g. it is standard to use zero-mean Gaussian distributions on the weights of Bayesian Neural networks. This can have dire effects, for instance Diaconis & Freedman (1986) demonstrate that multimodal priors in a location model can cause the posterior to not accumulate around Po. even when the DGP well specified, i.e. when Po E P()).
1. (Gvi(() arg min {Ee((o) [L 0. xx + D(q m) 1EQ
where @ C P(0), making inference tractable. This allows for targeting a larger subspace of posteriors, and through dif- ferent divergences the effect of the prior can be controlled.
3. Federated Generalised Variational Inference
Likelihood Misspecification One such example is where the hypothesis of interest is contaminated, and frac- tion of the data (input and/or output variables) has some unknown data source. Formalising this we follow the defini- tion of Huber (1964):
3.1. Methodology
In this section, we present the proposed federated learning framework, named FEDGVI, that explicitly addresses like- lihood and prior misspecification. We aim to learn a robust approximate posterior g, (0) using partitioned observations across M clients. FEDGVI VI iterates consist of two steps: a) sending of the current approximate posterior to each client, which is updated through robust variational objective, and b) aggregating the updates on the server, resulting in robust approximate posterior; summarised in Algorithms 1 and 2.
Definition 2.1 (Huber contamination). Given an E E (0, ) and the uncontaminated distribution Pe of inliers and some contaminating distribution G of outliers, then Po is said to be an e-corrupted version of Po; Po == (1 Po + sG.
2.3. Robust Bayesian Methods
Generalised Bayesian Inference (GBI) Instead of link- ing the parameter and data through likelihoods, Bissiri et al.
Initialisation
We set the initial server posterior as the prior, qe ((00) = x (0), and the local and server loss ap-
3

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
proximations to be zero, e (0) 0 and 4
3.1.1. HYPERPARAMETERS
respectively; m denotes a specific client and s the server.
Ashman al. (2022) set the damping parameter to 7 x M throughout their experiments. This turns out, see Proposi- tion 4.3, to be reasonable choice when r 1 in combi- nation with D, = DxL since this causes the posterior at the server to be a logarithmic opinion pool induced by an externally Bayesian pooling operator (Genest et al., 1986), ensuring stable convergence. Other hyperparameters arising from the choice of losses and divergences are dependent on the expected amount of model misspecification.
Until Convergence Fort 1, 2, T, we synchronously compute updates locally at each client, and accumulate these the server to form the new global posterior q (0).
M
Client The client receives the current approximate pos- terior from the server. This will be used as the prior from which client can compute an updated posterior using their local data. First, however the information of the client's data must be removed by computing the cavity distribution. The cavity distribution acts as the local prior incorporating all previous information from all other clients and given by:
3.2. Robustness to Likelihood Misspecification
Within our framework we are free to choose the client side losses. We consider the Density-Power divergence based loss (Ghosh & Basu, 2016b), often referred to as 3- divergence loss Ls. the d-vivereence based losses (Hung et al., 2018), Ly. as well as a score matching loss, LSM- based on the Hyvärinen divergence (Hyvärinen, 2005; Al- tamirano et al., 2023). In the classification setting, we con- sider the generalised cross cross-entropy loss
(
(3)
exp
eh (0)}
The client then computes robust local approximate poste- rior with it's local data set (Xm; ym and it's loss function L (-), which regularised by the divergence, D, and cav- ity distribution
(6) La e (y: ..)) (1 pe (y y;xz)))
(0) arg min (0) ( (ymi 0, xm)] + D(q : ))
qEQ
for some 5 (0, 1] (Zhang & Sabuncu, 2018). These losses are robust to misspecification because they have finite supremum (see Definition 4.1 It is important to highlight that GVI and FEDGVI may underperform when using ro- bust losses in the case of correct likelihood specification; see Knoblauch et al. (2022). We can use a Sequential Monte Carlo sampler to estimate the | or y hyperparameters C3 and Lyy (Yonekura & Sugasawa, 2023) or cross valida- tion to select optimal parameters (Altamirano et al., 2024).
(4) This GVI style objective allows the client be robust both likelihood misspecification well prior misspecification arising due to the cavity. To update the global posterior at the server, the client computes the negative log ratio of the local and global posteriors. In line with existing Bayesian FL (Ashman et al., 2022; Guo et al., 2023), we use a damping parameter 7m E (0, 1], which analogous to a learning rate as in frequentist FL, compute the update:
qm (0)
(8)
3.3. Robustness to Prior Misspecification
A (t) (0) Tm log
go
(1-1)
(0)
We mainly consider the weighted Kullback-Leine diver- gence, 1 DKL. (Kullback & Leibler, 1951)
The client stores e (0) dh (t-11
(0) Al (1)
(0) and com-
1
1 E log ((0) n (0)
municates A (t) (0) to the server.
g0 DKzlq : T) = w w
Server The loss at the server is updated based on the received client updates,
and the Alpha-Rényi divergence, D(a)
[
((0) T (())
D Di
(alq::) T) = a(a 1) log (x(())
( 0) - e(
Σ M
- !
(c)
(6)
By only incorporating clients' updates that have changed we can trivially allow for batched and asynchronous scheduling of clients. The updated loss is then used to compute the new server posterior though a GVI optimisation procedure:
As examined in Knoblauch et al. (2022), D(a) allows for different prior regularisation depending on how much we trust the prior by placing different weights on it. In future work it would be simple to explore other divergences such as the f-divergences, Dy, (Amari, 2016; Alquier, 2021). 1). Similarly to the losses, we can perform cross validation to select the a parameter, however as demonstrated in the ablation study (Figure 6) FedGVI performs favourably under a range of a (and 5) values.
n (0)
arg min E,t q
(()) + Dz(g: m)
(7)
This posterior and loss are passed back to the clients for further refinement at the next iteration until convergence.

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
4. Theoretical Results
4.2. Damping Bayesian Logarithmic Opinion Pool
We now present a theoretical analysis of FEDGVI. We begin by examining the relationship FEDGVI with other FL algorithms while recovering some of them as special cases, study the damping parameter, and examine the conver- gence behaviour of FEDG VI. Then, we turn our attention on robustness to likelihood misspecification, where we first study FEDGVI as distributed GBI, from which we derive a theorem on the necessity of the cavity distribution. Finally, we derive result for computationally tractable and conju- gate FEDGVI, enabling us to present the main theorem on robustesssss of FEDGVI.
Choosing the damping parameter be 1/M results in a logarithmic opinion pool. In fact choosing damping parameters such that all of them sum to unity also forms valid logarithmic opinion pool (Genest et al., 1986).
Proposition 4.3. Assume D, = DKL. and that Em Tm
where Tm 2 0 Vm, then the posterior at the server is an externally Bayesian logarithmic opinion pool of the form
II M (1)
m= 9m (0)) Tm
(
TM do 0 a.c.
Se II== (9m (0))
Since it is an open problem where global GVI posteriors converge to under arbitrary divergences, we often have to restrict ourselves to consider the server divergence to be the Kullback-Leibliee divergence. This ensures that the posterior the server will have the structure of GBI posterior,
See Appendix B.2 for the proof. This results provides theoretical justification on the previously heuristic use of the damping parameter (as used in PVI, Ashman et al., 2022). Specifically it ensures that this selection of 7 leads to a valid distribution and results in more stable convergence.
(0) x exp {- -MM
(0)) क (0)
4.3. Fixed Points of FEDGVI
In this section we study the properties of FEDGVI posteriors when these converge to some fixed point. Specifically, we generalise the fixed point result of PVI (Ashman al., 2022, Property 2.3) to arbitrary losses.
where we incorporate prior robustness and tractability through the approximate losses.
4.1. Recovering Existing Methods as a Special Case
Proposition 4.4. Let De DKL D DKL w > and Q C P(O). then if q: (0) π (0)) exp( (0)) (0))
By choosing specific divergences, loss functions, and varia- tional families, we can recover existing methods as special cases of our framework, which summarise in Figure 1: Remark 4.1. Choosing the Kullback-Leibler divergence and the negative log- ligeleieloood as a loss function recovers the PVI algorithm of Ashman et al. (2022).
such that Vm E [M]. A, (0) = O, then q (0) is a local minimiser of the following GVI objective:
1
E,,0,)
Σ Lm (y O xm) -m==
+ DKilq: w
(9)
Remark 4.2. When = D, 0, and Q (6g (0) 0 E }}. with og being the Dirac-delta measure at some element 0, we recover FEDAVG of McMahan et al. (2017).
Remark 4.5. If the loss in Equation (9) is convex, then a fixed point of FEDGVI is a global minimum of GVI.
This illustrates that if FEDGVI converges, then the posterior is a (local) minimiser of the GVI objective. We refer to such distributions as fixed points. This recovers Kassab Simeone (2022, Theorem 1) (which deals with the restricted case of Q P())) with novel proof; see Appendix B.3.
FEDAVG
PVI log po, D DKL. Q. M, D3 DKL
L, D = 0, (50):
M,
FEDGVI D, Q M, D
4.4. Generalised Bayesian Inference
ERM
VI
As a consequence of Proposition 4.4 and Remark 4.1, FEDGVI will recover the GBI posterior when P(O).
L, D = 0, (60)
log po: DKL Q M 1, D, DRL.
M- 1, D.
Lemma 4.6. Assuming Q P(@). D 1 DKL with 3 > 0, D. DKL. and = 1, then FVDGVI will recover the GBI posterior after the first iteration.
Figure 1: We illustrate the relationship of FEDGVI- characterised by the loss L, the client divergence D, the variational family Q. the number of clients M, and the di- vergence at the server D.-to Partitioned Variational Infer- ence (PVI), Variational Inference (VI), Federated Averaging (FEDAvG), and Empirical Risk Minimisation (ERM).
q. ()) = ( ym v M 1M))
exp( Smpl---m=iiis 0 xm (0)
This posterior is invariant under subsequent iterations of FEDGVI, having reached a fixed point.
5

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Moreover, for a damping rate T = 1/M, the posterior at the server converges pointwise a.e. in O to the GBI posterior,
model is Gaussian, or has the form of squared exponential, and that the natural parameters of the DGP are ()) = 0. then the client approximation will have a conjugate form. Proposition 4.10. Assume that the hypothesis pe(y (y) has dif- ferentiable, exponential family distribution with (7) - 0, Lm - and D v D and the variational family Q is the multivariate Gaussians, then the local posteriors at the clients are conjugate Gaussians. Moreover, Equa- tion (7) will have flosedorm if D, has closed form between Gaussian distributions.
L (0) T (anr( (0l{Xm= Mm /m/m=1 m=m)) 1) 0 - a.c. a.e.
This result, proven in Appendix B.4, is the first step towards likelihood robustness. If we were able to find the GBI posterior efficiently with some robust loss, then the posterior would be robust and computable. Here however, the loss may not vary over different iterations of FEDGVI as in Equation (4) and the normaliser may be intractable.
See Appendix B.6 for the proof. The loss may now de- pend on the client and iteration t. Most exponential family distributions satisfy the conditions of the proposition, and there are several divergences that allow closed form expres- sions between Gaussians, such as the Alpha-Rényi or the a, 3, diveerggess of Cichocki & Amari (2010), Further, this enables the use of intractable likelihood models.
4.5. The Cavity Distribution is Necessary
By further investigating the relationship of FEDGVI with the GBI posterior, we can extend Lemma 4.6 and derive a Theorem under which we are required to use the cavity distribution to regularise the client update, This is in contrast to both PVI, where ''s use is heuristically justified, and to other Bayesian FL approaches where the previous posterior is used instead. For this we recall two natural assumptions that any such distribution must satisfy in a federated setting. Assumption 4.7. No client can have access to the data set of another client.
4.7. Provable Robustness to Outliers
For a robust loss function at the clients, and using the weighted KI. divergence at the clients and the KI. divergence at the server, guarantees that after T iterations, the posterior computed at the server will also be robust to outliers. This means we can achieve robustness at the server by leverag- ing the robust losses that were derived for GVI. In this, we mean robustness as defined by Ghosh & Basu (2016a) and further developed in Matsubara et al. (2022). We define the empirical DGP of a client as 1n.... := 1 Emmm 8x,, and of EM Pm When this the entire data set as Pn := n contaminated by some fraction of data centred some ad- versarially chosen data point E, the misspecified DGP is defined as IP Fn,E,2 == (1 e) P) R + gōg.
Assumption 4.8, Each client generates their update equiva- lently to other clients.
These assumptions combined with Lemma 4.6 lead us to the necessity of the cavity distribution.
Theorem 4.9. Let the assumptions be in Lemma 4.6 with = 1, and assume that the Assumptions 4.7 and 4.8 are satisfied, then (1.) holds if and only if (2.) holds.
1. FEDGVI recovers the generalised Bayesian posterior (0) which is invariant under further FEDGVI updates.
Definition 4.11. We say that loss L (0; IP (. w.r.t. some prior distribution π (0), is robust to outliers, if the following hold:
2. The cavity regularises the client optimisation problem.
This provides principled justification for the use of the cavity distribution, as defined in Equation (3), in FEDGVI. We provide the proof in Appendix B.5.
1. sup #E
de m (0; P Fmn...))
(t)
S (7(m) (0),
e=0
sup
x(0) (0) (0) < 00, and
(t)
4.6. Conjugate Client Updates
OE
3. / π (0)
(t) 0 pld(d) <0 (m)
Before we present our main result provable robustness to likelihood misspecification, we first show that we can find a GBI posterior under specific losses in a computationally tractable manner. Assuming that the data generating process has some exponential family distribution, where y ~ pyy),,
These conditions ensure that the influence of arbitrary con- tamination on the local posterior is not arbitrarily bad. In particular the auxiliary function 7m ensures that the influ- ence of an adversarial data point 2 on the posterior over infinitesimal contaminations, d (t) de 40 (0: (r..,,)))) are fi- nite over all 0 and z. Condition 2 ensures the loss increases slowly enough for the local posterior to concentrate around the data, and condition 3 ensures the resulting posterior will be normalisable.
poly) ( T ((y)- A(n(O)) h(y)),
such that this is differentiable in y. by using the weighted score matching loss of Altamirano et al. (2023), CyM: then client updates, using the weighted KL divergence locally, are available in closed form. If we further assume that our
6

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Theorem 4.12. Let D, DEL D =11DDa DLL -=P(@), further assume that the prior is upper bounded and the
5.2. Influence Function
loss is lower bounded, then if vt E [T] and Vmn E [M] T. (0; s.a) robust, then the posterior generated by FEDGVI will be robust to outliers.
2.0
IMQ
Influence
1.0
SE
B-Loss
1.0
The proof is in Appendix B.7. This result together with Proposition 4.10 is significant as we have robustness under intractable optimisation, and we can choose provably robust, conjugate loss generate robust FEDGVI posteriors, which are then computationally efficient compute.
0.5
0.0
15
Distance to True mean
Figure 3: We plot the influence of a single outlier on the server posterior. PVI is not robust to likelihood misspeci- fication through outliers, because it uses the negative log- likelihood (NLL).
5. Experiments
We evaluate FEDGVI against several other methods, specif- ically PVI (Ashman et al., 2022), FEDAVG (McMahan al., 2017), the nouparametric DSVGD (Kassab & Sime- one, 2022), the distributed MCMC based DSGLD (Ahn et al., 2014), federated MCMC based FEDPA (Al-Shedivat et al., 2021), and the one shot BCM based approach 8- PREDBAYES (Hasan et al., 2024). We provide further de- tails about experiments in Appendix D.
To demonstrate robustness to likelihood misspecification as in Theorem 4. 12, we consider the influence of a single out- lier at one of seven clients on the server posterior. Figure demonstrates that the negative log likelihood not robust in the federated setting, whereas different robust divergence based losses allow only limited influence of outliers on the posterior. We plot this the divergence between the pos- terior, had we observed the outlier value at the true mean, against the posteriors that have the outlier be farther from the true mean, using the Fisher-Rao distance (Nielsen, 2023).
5.1. 1D Clutter Problem
0.40
MIE wio outliers
with oulliers
0.35
PV] wño outliers
5.3. 2D Misspecified Logistic Regression
outleess
FedCV SM loss
Distribution
0.25
RedGV1 loss
0.20
Outliers
0.15
0.10
FalCVV
0.05
PVI Chutiers.
0.00
Clask
-
Cliss
-2
Outllers
Figure 2: Robustness to outliers can be achieved through varying losses with FEDGVI, while traditional Bayesian methods fail.
Figure 4: Logistic Regression decision boundaries (0.2, 0.5, 0.8) for PVI without outliers, PVI with misspecification, and FEDGVI with misspecification. The synthetic data is split homogeneously across 5 clients where PVI negatively skews the decision boundary, while FEDGVI does not.
We first examine the effect of misspecified likelihoods through the well known clutter problem (Minka, 2001). We generate 100 observations from Gaussian location model that is contaminated through Definition 2.1 with 0.25 Gaussian noise. The aim to infer the location parameter of the uncontaminated data. We compare VV with both L. and Lsas vs PVI with and without misspecifica- tion. We also provide the corresponding MLE results. See Figure Under misspecification both the MLE and PVI fail to recover the true whereas VIDGVI can easily handle different levels of contamination.
We next consider a 2D logistic regression example where we generate 100 linearly separable samples from Gaussian mixture distribution. We inject outliers generated by a third Gaussian distribution and assign them to one of the classes so that the data is no longer linearly separable. We compare FEDGVI with c (0.7) and D (1.5) against PVI, both with clients. Again, the target is given by PVI only trained on the uncontaminated data. As expected PVI is severely impacted

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
by outliers, whereas FEDGVI is robust them and closely
8=0.0
8s0.2 0.0.2
8s0.
8=0.6
8=0.8
8=1.0
recovers the target posterior.
a=0.0
6.16
3.14
1.92
2.26
2.55
2.83
5.4. Real-Worle Cover Type Dataset
a=0.5
3.68
3.09
2.03
2.01
2.24
2.48
0.76
1.0
3.46
3.02
2.08
1.95
2.08
2.28
Clussification Accuracy
0.74
1.5
3.32
3.93
2.33
1.96
2.04
2.12
0.72
0.70
a=2.5
3.16
5.21
2.81
1.83
1.91
2.13
0.68
PVI
FolArg
a=5.0
2.63
7.04
4.19
2.05
1.89
1.95
(5)
0.06
DSVGD
FedGVI, (0.5)
Figure 6:
An ablation study the hyperparameters of FEDGVI with c and D(a) We plot the maximum results achieved as percentage errors on uncontaminated test data after training clients on 10% contaminated data.
0.64
2
10
Server Iterations
Figure 5: Results on the COVERTYPE data set. We place Gaussian distribution over the weights and average over 10 different train/test splits; see Appendix for details.
Table 2: Classification accuracy (highest in bold) on uncon- taminated test data after training on different amounts of contaminated FASHIONMNIST data. For FEDGVI we have fixed 2.5 for the a- Rényi divergence. Each Method has data split homogencously across Clients. We report the best performance during all server iterations.
In this experiment we follow the experimental setup of Kassab & Simeone (2022) and average accuracy over random 80/20 train-test splits, where the training data is split homogeneously across 2 clients. We do not add any label contamination. The results plotted in Figure 5. The non-robust methods all eventually achieve similar accuracy, however FEDGVI is able to outperform all competing meth- ods, which we argue due to FEDGVI putting less weight on data points that are less likely to belong to the class.
MODEL
CONTAMINATION
0%
10%
20%
40%
FEDAVO
85.7.10.5
1..0
71.241.5
39.0:16.5
FEDPA
88.1=0.3
87.4=0.2
86.5+0.2
45.4
8 PREDBAYES
87.61:0.1
87.2+0.1
86.8+0.1
85.8
PVI
86.2110.2
85.110.11
84.4:00.1
82.810.1
5.5. Bayesian Neural Networks on MNIST and FASHIONMNIST
FEDGVT 5 0.0
87.120.1
86.2-10.2
85.6:10.1
83.8:10.1
FEDGV1
88.7=002
88.620.11
87.0-t0.4
78.1
FEDGVI 5 = 0.5
89.0:10.2
88.62:0.2
88.4:10.2
85.1:10.7
FEDGVI
88.64.0.0
88.420.1
88.0++0.0
87.2-00.1
Table 1: Classification accuracy (highest in bold) on un- contaminated test data after training on 10% contaminated MNIST data. We report the best performance across all server iterations.
FEDGVI 5 1.0
88.1#0.1
87.820.1
87.5=0022
86.0=t0.3
ing set while leaving the test set unchanged and evaluate performance in this. For MNIST, we add 10% of class dependent label noise, see Figure and Table 1. We further carry out an ablation study on the hyperparameter selection in FEDGVI with the Alpha-Rényi divergence and the gen- eralised cross entropy loss, see Figure 6. This demonstrates that FEDGVI performs well under variety of different loss and divergence parameters. Note that = 1 recovers the KL divergence, a 0 the reverse KL divergence, i.e. D D л) = DRKL x) DR q). and that 8 0 recovers the negative logelikeli.ood.
ACCURACY STD.
MODEL
CLIENTS
CLIENTS
FEDAVG
96.64.1 0.07
96.34 1: 0.20
FEDPA
94.2541 0.39
95.314 0.35
3- PREDBAYES
94.90 0.08
96.731 0.08
PVI
95.56 0.18
96.68 0.07
FEDGVI DAR
96.36 0.09
97.13
FEDGVT LECE
97.06-t 0.03
98.04 0.07
FEDGVI DARthGGE
97.50+ 0.07
98.13 0.08
VI CLIENT)
(96.96+t 0.17)
For FASHIONMIIST, in Table 2, we vary the amount of ran- dom label contamination, showcasing performance drops under different amounts of misspecification. We use an MI.P, for FEDGVI and PVI with hidden layer of 200
GVI (1 CLIENT)
198.13+ 0.07)
We create label contamination by adding noise to the train-
8

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
(2.6)
c(0.8)
K.L GNLL DA LGGE VIKLENLL
FaiCivi D (X
GGGE
r(0.5)
PVI KT. M.K.L.CNNL CNLL
% Error (10 Chients)
FedGVL (2.0) ANLL
NLL (10 Clients)
FedGv1 (
LNLL
GVI D12.5) c.0.8)
VIKL, CNLL
FedGVI KL, CGC
FadGivi GOB
Server Iierations t
Server lterations t
FedGVI 19
LCC
GVI VIKL,ENLL
3) GGE
FalCVVI AR
GGGE
GN AR CooE
% Ertor Clients)
PVI KL, LNLL
NLL (3 Clients)
KL, CNZz
VI MKL.CNE.
10
20
25
10
10
20
25
Server Iterations
Server Tterations
Figure 7: Accuracy (% Error) and Negative Log Likelihood (NLL) results when running fully connected BNNs, with Mean-Field Gaussian distribution, the MNIST data set with FEDGVI. The training data set is contaminated by 10% random label flipping, fixed across all repetitions. We average over five runs with random, homogeneous client splits.
neurons; for FEDAVG, FEDPA, and B-PRRDBAYES, two hidden layers with 100 neurons in each. Data distributed homogeneously across clients, using different, randomly chosen seeds. We demonstrate that under model misspeci- fication, FEDGVI significantly outperforms competing FL methods. Furthermore, FEDGVI incurs no additional com- putational complexity when compared to PVI. This is due to the KL and Jlpha--énny divergences having closed form solutions between Multivariate Gaussians with complexity of O(1) in each other, and as we require O(1) additional, constant operations get the GCE from the NIL..
at fixed points of FEDGVI as solving a global GVI opti- misation problem, extending existing theory. Our result on provable robustness to outliers through FEDGVI allows for closed form, conjugate posteriors that are computationally efficient, and robust model misspecification. deriving this, we have also shown that the cavity distribution is nec- essary as predictions would otherwise be overly confident and biased. The robustness of FEDGVI was further demon- strated empirically on multiple synthetic and real-world data sets, showing outperformance of existing FL methods across model architectures and misspecification levels.
We provide further experiments Appendix on the run- time of FEDGVI against PVI, learning rate selection, stabil- ity of posteriors under small perturbations in the robust loss parameters, and showing that using a single hidden layer NN for the competing methods would either negatively, or not significantly, affect their performance.
An interesting future direction is extend FedGVI within personalised FL settings (Kotelevskii et al., 2022) and hi- erarchical Bayesian FL through latent variables (Kim & Hospedales, 2023) as well as through the use of a structured posterior approximation (Hassan et al., 2024), in order to incorporate client level variations. Incorporating the hierar- chical model structures and additional inductive biases from such settings, while maintaining conjugacy and favourable computational complexity, remain open challenges. In future work, further aim to address the robust Bayesian nonparametric setting of FL through FEDGVI, as well as in- vestigate other types of robustness, including to adversarial and Byzantine attacks, by for instance using a robust aggre- gator Equation (6), and addressing the open problem of provable robustness prior misspecification GVI.
6. Conclusions and Future Work
We have introduced FEDGVI, a novel probabilistic ap- proach to federated learning that is provably robust to model misspecificatiio, and allows for faster, conjugate client up- dates. The theoretical analysis FEDGVI demonstrates it's appealing properties; we easily recover existing methods as restricted cases, and characterise the convergence behaviour

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Acknowledgements
Machine Learning Research, pp. 1078-1114. PMLR, 21-
27 Jul 2024.
OH, TM and TD acknowledge support from a UKRI Turing AI Acceleration Fellowship [EP/V02678X/1] and a Turing Impact Award from the Alan Turing Institute. For the pur- pose of open access, the authors have applied Creative Commons Attribution (CC-BY) license to any Author Ac- cepted Manuscript version arising from this submission. The authors acknowledge the University of Warwick Re- search Technology Platform for assistance in the research described in this paper.
Alquier, P. Non-exponentially weighted aggregation: Regret bounds for unbounded loss functions. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pp. 207-218. PMLR, 18-24 Jul 2021.
Alquier, P., Ridgway, J., and Chopin, N. On the properties variational approximations of gibbs posteriors. Journal of Machine Learning Research, 17(236):1 -41, 2016.
Impact Statement
This paper presents work on robust federated learning, a framework that aims to not only advance the field of ma- chine learning, but also to develop methods that ensure the privacy of data sources, whilst aiming to achieve optimal performance even under contamination of the data. This approach, however, may discard low probability, tail events that could represent minority groups. Hence, the trade off between robustness and inclusivity is fundamental ethical challenge for decision makers.
A Altamirano, M., Briol, F.-X., and Knoblauch, J. Robust and scalable Bayesian online changepoint detection. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 642-663. PMLR, 23-29 Jul 2023.
Altamirano, M., Briol, F.X.,, and Knoblauch, J. Robust and conjugate Gaussian process regression. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 1155-1188 PMLR, 21-27 Jul 2024.
References
Achituve. I., Shamsian, A., Navon, A., Chechik, G., and Fetaya, F. Personalized federated learning with Gaussian processes. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
Amari, S.-i. Information Geometry and Its Applications. Springer, Tokyo, Japan, 2016. ISBN 9784431559771.
Ahn, S., Shahbaba, B., and Welling, M. Distributed stochas- tic gradient MCMC. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Ma- chine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1044-1052, Bejing, China, 2014. PMLR.
Ashman, M., Bui, T. D., Nguyen, C. V., Markou, S., Weller, A., Swaroop, S., and Turner, R. E. Partitioned varia- tional inference: A framework for probabilistic federated learning. arXiv preprint arXiv:2202. 12275, 2022.
Bao, W., Wu, J., and Hc, J. BOBA: Byzantine- fed- erated learning with label skewness. In Dasgupta, S., Mandt, S., and Li, Y. (eds.). Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 892-900. PMLR, 02-04 May 2024.
Al-Shedivat, M., Gillenwater, J., Xing, E., and Ros- tamizadeh, A. Federated learning via posterior averaging: A new perspective and practical algorithms. In Interna- tional Conference on Learning Representations, 2021.
Ali, S. M. and Silvey, S. D. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B (Methodological), 28( 18(1):1311142, 1966.
Berger, J. O. Statistical Decision Theory and Bayesian Analysis. Springer-Verlag, New York, 1985. ISBN 9781475742862.
Allouah, Y., Farhadkhani, S., Guerraoui, R., Gupta, N., Pinot, R., Rizk, G., and Voitovych, S. Byzantine-robust federated learning: Impact of client subsampling and local updates. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of
Berk, R. H. Limiting behavior of posterior distributions when the model is incorrect. The Annals of Mathematical Statistics, 37(1):51 - 58, 1966.
Bernardo, J. M. and Smith, A. F. M. Bayesian theory. Wiley Series in Probability and Statistics, Chichester, England, 2000. ISBN 9780470316870
10

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Bissiri, P. G., Holmes, C., and Walker, S. G. A general framework for updating belief distributions. Journal of the Royal Statistical Society. Series B (Statistical Method- ology), (8(5):1103-1133) 2016.
Genest, C., McConway, K. J., and Schervish, M. Charac- terization of externally Bayesian pooling operators. The Annals of Statistics, 14(2):487- 501, 1986.
Ghosh, A. and Basu, Robust Bayes estimation using the density power divergence. Annals of the Institute of Statistical Mathematics, 68(2):413-437, 6a.
Blei, D. M., Kucukelbir, A., and McAuliffe, D. Varia- tional inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-
Ghosh, A. and Basu, A. Robust estimation in generalized linear models: the density power divergence approach. TEST, 25(2):269 2016b.
2017.
Bui, T. D., Nguyen, C. V., Swaroop, S., and Turner, R. E. Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv preprint / I.1 /206, 2018.
Grünwald, P. The safe Bayesian. In Bshouty, N. H., Stoltz, G., Vayatis, N., and Zeugmann, T. (eds.), Algorithmic Learning Theory, pp. 169-183, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
Carvalho, M., Villela, D. A. M., Coelho, F. C., and Bastos, L. S. Bayesian inference for the weights in logarithmic pooling. Bayesian Analysis, 18(1):223 251, 2023.
Guo, H., Greengard, P., Wang, H., Gelman, A., Kim, Y., and Xing, E. Federated learning as variational inference: A scalable expectation propagation approach. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Chan, R. S., Pollock, M., Johansen, A. M., and Roberts, G. O. Divide-and-conquer fusion. Journal of Machine Learning Research, 24(193):1-82 2023.
Chen, W.-N., Choquette-Cher C. A., Kairouz, P., and Suresh, A. T. The fundamental price of secure aggre- gation differentially private federated learning. Chaudhuri, K., Jegelka, S.. Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th In- ternational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 3056-3089, PMLR, 17-23 Jul 2022.
Hamer, J., Mohri, M., and Suresh, A. T. FedBoost:
communication-effice algorithm for federated learning. III, H. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3973-3983. PMLR, 2020.
Hasan, M., Zhang, G., Guo, K., Chen, X., and Poupart, P Calibrated one round federated learning with Bayesian inference in the predictive space. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1):12333 12321, 2024.
Cichocki, A. and Amari, S.-i. Families of alpha- beta- and gamma- divergences: Flexible and robust measures of similarities. Entropy, 12(6):1532-156) 2010.
Corinzia, L., Beuret, A., and Buhmann, J. M. Varia- tional federated multi-task learning. arXiv preprint arXiv: 1606.066888 2021.
Hasenclever, L.,, Webb, S., Lienart, T., Vollmer, S., Laksh- minarayanan, B., Blundell, C., and Teh, Y. W. Distributed Bayesian learning with stochastic natural gradient expec- tation propagation and the posterior server. Journal of Machine Learning Research, 18(1)3374437700 2017.
Demidovich, Y., Ostroukhov, P., Malinovsky, G., Horváth, S., Takáč, M., Richtárik, P., and Gorbunov, E. Meth- ods with local steps and random reshuffling for generally smooth non-convex federated optimization. In The Thir- teenth International Conference Learning Representa- tions, 2025.
Hassan, C., Salomone, R., and Mengersen, K. Federated variational inference methods for structured latent vari- able models. arXiv preprint arXiv:2302.0333: 2023.
Hassan, C., Sutton, M., Mira, A., and Mengersen, K. Scal- able vertical federated learning via data augmentation and amortized inference. arXiv preprint 2024.
Diaconis, P. and Freedman, On the consistency of Bayes estimates. The Annals of Statistics, 14(1):1 26, 1986.
04043,
Fraboni, Y., Vidal, R., Kameni, L., and Lorenzi, M. A gen- eral theory for federated optimization with asynchronous and heterogeneous clients updates. Journal of Machine Learning Research, 24(110):--43, 2023.
Heikkilä, M., Ashman, M., Swaroop, S., Turner, R., and Honkela, A. Differentially private partitioned variational inference. Transactions on machine learning research, 2023(4), 2023.
Genest, C. A characterization theorem for externally Bayesian groups. The Annals of Statistics, 12(3):1100- 1105, 1984.
Hooker, G. and Vidyashankar, A. N. Bayesian model ro- bustness via disparities. TEST, 23(3):5565844, 2014.

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Huber, P. J. Robust estimation of a location parameter. Annals of Mathematical Statistics, 35:73-101, 1964.
Knoblauch, J., Jewson, J. E., and Damoulas, T. Doubly robust Bayesian inference for non-stationary streaming data with 3-divergences. In Advances in Neural Informa- tion Processing Systems, volume 31, pp. 64-75. Curran Associates, Inc., 2018.
Hung, H., Jou, Z.-Y., and Huang. S.-Y. Robust mislabel lo- gistic regression without modeling mislabel probabilities. Biometrics, 74(1):145-1144 2018.
Knoblauch, J., Jewson, J., and Damoulas, T. An optimization-ceme view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(132):1-109, 2022.
Hyvärinen, A. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
Jewson, J., Smith, J. Q. and Holmes, C. Principles of Bayesian inference using general divergence criteria. En- tropy, 20(6):442, 2018.
Kotelevskii, N. Y., Vono, M., Durmus, A., and Moulines, E. FedPop: A Bayesian approach for personalised fed- erated learning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Jonker, M. A., Pazira, H., and Coolen, A. C. Bayesian feder- ated inference for estimating statistical models based non-shared multicenter data sets, Statistics in Medicine, pp. 1-18, 2024.
Kullback, S. and Leibler, R. A. On Information and Suffi- ciency. The Annals of Mathematical Statistics, 22(1):79- 86, 1951.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Ben- nis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cor- mode, G., Cummings, R., D' D'Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Har- chaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Koneený, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi H., Ramage, D., Raskar, R., Raykova, M., Song. D.. Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, E. Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q. Yu, F. X., Yu, H., and Zhao, S. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 1):2278-22333 1998.
Li, H., Acharya, K., and Richtárik, The power of extrapo- lation in federated learning. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. Malinovsky, G., Kovalev, D., Gasanov, E., Condat, L., and Richtarik, P. From local SGD to local fixed- methods for federated learning. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6692-6701. PMLR, 2020.
Kallioinen, N., Paananen, T., Bürkner, P.-C., and Vehtari, A.
Detecting and diagnosing prior and likelihood sensitivity with power-scaling. Statistics and Computing, 34(1):57, 2024.
Matsubara, T., Knoblauch, J., Briol, F-X.,, and Oates, C. J. Robust generalised Bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):997-1022, 04 2022.
Kassab, R. and Simeone, O. Federated generalized Bayesian learning via distributed Stein variational gradient descent. IEEE Transactions on Signal Processing, 70:2180-2199 2022.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
Arcas, B. A. y. Communication-effiiinn learning of deep networks from decentralized data. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings Machine Learning Research, pp. 1273- 1282. PMLR, 2017.
Katsevich, A. and Rigollet, P. On the approximation ac- curacy of Gaussian variational inference. The Annals of Statistics, 52(4):1384 1409, 2024.
Kim, M. and Hospedales, T. FedHB: Hierarchical Bayesian federated learning. arXiv preprint arXiv:2305.04979, 2023.
Mekkaoui, K. e., Mesquita, D., Blomstedt, P., and Kaski, S. Federated stochastic gradient Langevin dynamics. In de Campos, C. and Maathuis. M. H. (eds.), Proceedings of the Thirry-Seennth Conference on Uncertainty in Artifi- cial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1703-1712. PMLR, 2021.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization. In 3rd International Conference on Learning Representations, 2015.
12

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Mesquita, D., Blomstedt, P., and Kaski, S. Embarrassingly parallei MCMC using deep invertible transformations. In Adams, R. P. and Gogate, V. (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Confer- ence, volume 115 of Proceedings of Machine Learning Research, pp. 1244-1252. PMLR, 2020.
Tziotis, I., Shen, Z., Pedarsani, R., Hassani, H., and Mokhtari, A. Straggler-resilient personalized federated learning. Transactions Machine Learning Research, 2023.
Vedadi, E., Dillon, J. V., Mansfield, P. A., Singhal, K., Afkanpour, A., and Morningstar, W. R. Federated varia- tional inference: Towards improved personalization and generalization. Transactions on Machine Learning Re- search, 2024.
Miller, J. W. Asymptotic normality, concentration, and coverage of generalized posteriors. Journal of Machine Learning Research, 22(168): 2021.
Vehtari, A., Gelman, A., Sivula, T., Jylänki, P., Tran, D., Sahai, S., Blomstedt, P., Cunningham, J. P., Schiminovich, D., and Robert, C. P. Expectation propagation as way of life: A framework for Bayesian inference on partitioned data. Journal of Machine Learning Research, 21(1), 2020.
Minka, Expectation propagation for approximate Bayesian inference. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence. pp. 362-369, San Francisco, CA, USA, 2001.
Nielsen, F. An elementary introduction to information ge- ometry. Entropy, 22(10):1 100, 2020.
Walker, S. G. Bayesian inference with misspecified models. Journal of Statistical Planning and Inference, 143(10): 1621-1633, 2013.
Nielsen, F. A simple approximation method for the fisher-rao distance between multivariate normal distri- butions. Entropy, 25(4), 2023.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprini arXiv: / 0778.77747, 2017.
Opper, M. and Winther, O. Expectation consistent approxi- mate inference. Journal of Machine Learning Research, 6(73):2177-2044 2005.
Yonekura, S. and Sugasawa, S. Adaptation of the tuning parameter in general Bayesian inference with robust di- vergence. Statistics and Computing, 33(2):39, 2023.
Pardo Llorente, L. Statistical inference based on diver- gence measures. Chapman & Hall/CRC, 2006. ISBN 9781558 0781584886006.
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K.,
Hoang, N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7252- 7261. PMLR, 09-15 Jun 2019.
Pinski, F. J., Simpson, G., Stuart, A. M., and Weber, H. Kullback-leible approximation for probability measures on infinite dimensional spaces. SIAM Journal on Mathe- matical Analysis, 47(6):4091-411 2015.
Reddi, S., Charles, Z. B., Zaheer, M., Garrett, Z., Rush, K., Konečný. J., Kumar, S., and McMahan, B. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
Zellner, A. Optimal information processing and Bayes's the- orem. The American Statistician, 42(4):278-22 2800, 1988.
Zhang, X., Li, Y., Li, W., Guo, K., and Shao, Y. Personal- ized federated learning via variational Bayesian inference. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26293-26310. PMLR, 17-23 Jul 2022.
Scott, S. L., Blocker, A. W., Bonassi, F. V., Chipman, H. A., George, E. I., and McCulloch, R. E. Bayes and big data: The consensus monte carlo algorithm. Interna- tional Journal of Management Science and Engineering Management, 11:78-88, 2016.
Swaroop, S., Khan, M. E.,, and Doshi-Velez. F. Connecting federated ADMM to Bayes. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Zhang, Z. and Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Tenison, I., Sreeramadas, S. A., Mugunthan, V., Oyallon, E.,, Rish, I., and Belilovsky, E. Gradient masked averaging for federated learning. Transactions on Machine Learning Research, 2023.
Zhao, Z., Luo, M., and Ding, W. Deep leakage from model in federated learning. In Conference on Parsimony and Learning, volume 234 of Proceedings of Machine Learn- ing Research, pp. 324 340. PMLR, 2023.
Tresp, V. A Bayesian committee machine. Neural computa- tion, 12:2719-41, 2000.
13

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Zhu, L., Liu, Z., and Han, S. Deep leakage from gradi- ents. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
14

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Supplementary Material for:
Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
The appendix structured as follows: Appendix A summarises the notation used throughout the paper and in the proofs. In Appendix B we present complete proof' of all theorems, propositions and lemmas given in the main paper. Appendix C clarifies the requirements of Definition 4. I, the GBI learning rate, and places FEDGVI in the broader GVI literature. Lastly, Appendix D gives additional details about the implementation of FEDGVI and additional experiments.
A. Notation
In this section, we give definitions of the symbols used throughout the paper and the appendix.
Po
The abstract and unknown probability measure, also called data generating process, acting on some abstract measurable space F) which gives rise the data
(Zi, W
Entire data set of all clients, also written {-f.=") } for x; E and yılr, E T
(Xm: Vm )
The entire set of data points split across M clients labelled m E [M] == 21,
The data space, which assumed to have Polish topology
The output space, which can categorical such as classification where T = [C], or real valued as in regression
T CC
In the parametric setting this is the parameter space 0 E θ, assumed admit Polish topology
P()) The space of probability measures over the measurable space T), We refer to distributions in this space, where we mean distribution functions given rise to by measures in this space. Note that these need not be continuous, and could only be defined almost everywhere in O.
variational family distributions such that Q C P(0) and, in terms of distributions, = {0(0))) P(O): K)
where K is set of variational parameters
(0)
The prior distribution, given rise to by the prior measure II (0,7)
The local loss of client m, at iteration E [T], on the local data set (mm: not necessarily the same across clients nor iterations, and associated with the parameters
Line (y 0, X
(t)
Local loss approximation of L (ym: 0, xm) and the impact of the data of client m on the posterior at the server
A m (0) Local update, Equation (5), that represents the change in the approximate posteriors, and the de facto change in the local loss approximation. It has associated damping parameter
(t)
Global loss approximation of all clients aggregated the server
4m (0)
Local posterior computed through Equation (4)
(1) (0))
Global approximate posterior after server-side optimisation step, Equation (7)
P(L, D. Q)
The Rule of Three (Knoblauch et al., 2022) that defines global GVI objective
Any statistical divergence D PP()) x -P()) - R20 (for a detailed definition see Nielsen, 2020); D, denotes the divergence at the server.
D
Eg The expectation with respect to g())
B. Proofs of Theorems, Propositions, and Lemmas
Here, we provide the full proofs of the theorems stated in the paper. Throughout, we assume that all the losses, distributions and approximate losses, are measurable with respect to some dominating measure (d(d)). This can be the Lebesgue measure in finite dimensional spaces, or more generally the Haar measure. For infinite dimensional measure spaces, which are of interest in the study Bayesian inverse problems and nonparametrics, we could assume p(d0) to be Gaussian measure as
15

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
in Pinski et al. (2015).
B.1. Equivalence Between The KL Divergence and Weighted KL Divergence
First, we present a well known auxiliary lemma that will be used throughout the proofs. states that the weighted KL divergence is equivalent to using a tempered or weighted likelihood in the optimisation procedure, and hence lead to equivalent inference problems (Knoblauch et al., 2022; Bissiri et al., 2016). So without loss of generality, we can push the weighting term of the KL divergence inside the loss, by defining the loss to be L L, which does not change the optimisation procedure. We show this result for f diveergges,,s, which we define as in Ali & Silvey (1966) and Amari (2016).
Lemma B.1. For u the posteriors computed by the weighted J-divergence, = +o2 Dy and loss L, and the posterior through the f-divergence D Dy and weighted loss w L are equivalent, i.e.,
P(L, Dr, 2) P(( L, 2)
Proof
1 E (()) [L 0xx] + Dy T)
P(L. Dj, Q) = arg min
9EQ arg min 1E
e0) [L 0, x)]
1
By(o)
(
w
(1())
arg min q
(0)
w N.: +
w
(0))
[((())
arg min v
g0)
0(y;0.x)
((())
arg min
{ (0)) [W- Lyy x)] + Dy(q: [)] = P(w L, Dy
REQ
Therefore, when referring to the loss in the following we mean it to the weighted loss so that we can utilise the weighted KL divergence. This easily recovers the KL-divergence for f u log u.
B.2. Proposition 4.3: Logarithmic Opinion Pool through Damping
Proof Consider the server update at some iteration 4, where we gather the client updates. Under the KL divergence, we then solve the server optimisation procedure as:
( arg min Eg000) [e) + KL(qq m]] arg min Q
Ey((0) log
(
qe
n (()) exp 1 (0))
we know that this is minimised at:
d (0)( x ा ex)e ( (0) exp (((-1)(0)
Σ
(t)
M
(1) (0)
M
(L)
x (0) exp p(t-1) exp
Tm log
qm
qm
ocq's (0
m=1
(1-1)
m==1
g
(1-1)
(
(gf (1-1) m=
Fm
By assumption we have that EM Fm = 1, therefore (g= (t- 1) (0)) Dmal Tm = 98 (0) and:
M
(0) II
(0) TM
16

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
m==1 (mm (T)
(2) (0) = T m= (mm (0)
1 - a.e.
Fm p(d0)
This forms an externally Bayesian logarithmic opinion pool (Genest, 1984; Genest et al., 1986).
B.3. Proof of Proposition 4.4
The proof of Proposition 4.4 is adapted from that for Partitioned Variational Inference in Ashman et al. (2022). We show the proof of Proposition 4.4 by comparing the derivatives with respect to the variational parameters of (00)k) of the sum of local objectives with those of the global objective. This motivated by the equivalence of sum of local GVI objectives (from each client) with some added constants and the global GVI objective, demonstrated in Appendix B.3.1. The main proof is in Appendix B.3.2.
B.3.1. RECOVERING A GLOBAL GVI OBJECTIVE FROM LOCAL OBJECTIVES
First, we provide an analogue of Ashman et al. (2022, Property 2) which states that the sum of the local (client) VEDGVI objectives and some constant, which we find to be the negative log normalising constants of the cavity and the server distributions, equals the global GVI objective. We define the following:
M
1
(
0
exp
e
(0)
Z
g
m=1
m 0
1
(0) exp(-
(1) (0))
y 0
expl - - (0)}
Obj(m, ( == O)(
[m (ymi 0, xmm]
+ DKL(q: (m)))
w
M
Obj(g!") = Eg(0))
Σ
Lm (ymi 0, xm)
Dxi(q:m)
w
Then we can recover the global objective by summing over the local objectives and subtracting the log normalising constants of the cavity distributions and the current server posterior.
1 Obj(m, (12) (log Z 2000 + Em= log Z Zmrm)
M
Ey(()) 0, xm)) + w
DKLLq (9::99:999) g)m))
1
(log
Z
=IM log km)
(/4)
w
M
M
(0)
Σ
By( [m 0, x] +
E ())
log
(log Z [00
+ M +IM log Z pm
W
mm (0)
W
%
m=1
m=1
M Lm (ymi 0, xm)
1
M
q(0)
+
-
E
(0)
log
(log 2 n
+ >M log 2 (m)m)
q m (0)
Lm=1
-==1
1
exp
(0)
(0))
Lm (Ymi 0, x.x))
E
(()
log
(log Z /
+ EM log Z km)
Lm=1
W
m=1
9
0
Σ
Lm (mmi 0, xm)
1
log (()) exp
m=
M
m
(t)
(0))
1
log Z
Lm=1
w
q
(
w
9
M
(())
Ey(()
Σ
Lm mim O, xm)
Ey(0)
log
log (9)
=bbjgg
w
π (0)/Z
w
m=1
17

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Hence, by using the weighted KL divergence at the clients optimisation step, can we recover a global GVI objective by summing over the local objectives and adding some constants independent of the variational parameters of interest in the optimisation problem. We note that the added logarithms of the normalising constants are independent of K, since these are fixed through the current posterior and cavity distribution and do not depend on the variational parameters.
B.3.2. PROPOSITION 4.4: FIXED POINTS RECOVERS A GLOBAL FIXED POINT
We denote a fixed point of the algorithm as q. (0 K such that for all m E [MI] we have g (0)re)) E arg min Obj(m, g.), then we have the property that no update will change the posterior found. Recall:
Proposition 4.4 Let D s Dxi, al the clients, local loss Lm and Q {{(0)s) K. E K} C (P()) as variational family, Assume that FEDGVI VI finds a fixed point q. (0)k²)) ). such that for all clients we have that o² (0 k<) E arg min Obj(m, oz) Then, it holds that g. (0)x²)
arg min REo Obj(g)).
Proof First we note that we consider only the KL divergence in this proof, which is equivalent to saying we modify the loss L to be multiplied by w > O, which results in the equivalent formulation, as shown in Knoblauch et al. (2022) where P(L, L DKL Q) = P(w- w L, LL, 2), see also Lemma B.1.
Note that the condition Vm E [M] we have that (0)x") E arg min Obj(m, g.) is equivalent to requiring that At, (0) O, since this means that the local loss approximations remain unchanged and hence (0) remains unchanged. This then implies that the posterior at the server will not change. This is the same as saying that the client optimisation step
has found the global solution and hence 9.m (0) and o (0) will be the same which implies that A (0) = O.
In the following all integrals are assumed to be over the parameter space θ even when we don't 't make it explicit.
We can furthermore show that we can express the derivative of the local objective as a single integral under the weighted KL divergence.
Obj(m,, = Vx Eyto) [Lm (ommyyii,. 0,x,) )1+ + DKL (q: eepp - K5- }
C. (0)x=)
Vx / 9(0))) expl-Lane (ymm 0, xm]] + +0((0) (log ((0lx) -ea (0)r²)) }}
1
log Zin u(dd)
(0(x)²))
0
= Vx ((0)=) (000=*)) expl Lom (y mi i O, xm)] uld0) + Vx log Z f (0(0ik) u(do)
(0)) k<)
(0)x=)}
Now we first show that the fixed point is an extremum of the global objective and then that it is minimum. We do this by first differentiating the local objective with respect to the variational parameters and then that the sum of the local derivatives evaluated at K = x² equal the derivative of the global objective.
(00)xc)
V re Obb(mm Vx (01ks) log (0)x²) exp( Lm (y 0 xmm) mid0)
Vn (00)k) (ymi 0, xm) (0(x²)) ((d))= - Vn / /((0))) log g(0kk)
2. (0 m(d0)
Vn. (Lm) xmx (0)x")) - (VVn((0))))
([V()=-(VVv(((01))) log
(0)Kk)
9/ (0) (u(d0)
where first line follows since we can compose the expectation and (weighted) KL divergence and the normalising constant of the cavity distribution is constant with respect to k. The last line follows from the fact that d du f(x) log f(r) (x(x) log (xx + (x) and that we can exchange the order of integration and differentiation. We further note that at v(0)k))
convergence, where K = x", that log (0)m=²) m= 0. Evaluating the expression above at k² then yields:
V Obj(m, R=K" -vx/ (00))) (ymi 0, xm) Er (0)x=))) p(d0)
N=N!
18

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Summing over all these client objectives then yields the following expression:
M
M
x /
n Obj(m,
(Im))))()m(m)))))))))) 0m -E (0'k=)) px(d0)
K=K"
K=K"
= Vv / ((00)))(
M
Σ
Li my Xm)
Σ m=1
fm (0(=")) p(d0)
R=
m=1
= - - log
9
(0
kvd0)
(0) exp |m==
M
Lm my xm)))
K=
x / (00 2,-- mu(d0)
To compare this with a global fixed point we differentiate the global objective at q", not yet assumed to be a minimiser of the global objective, with respect to the variational parameters.
Vx.Obi(())) Vn / (00)k)) log (0) exp M
g(0kx)
uld0)
(y O, (.mm)
= -V(7)((01))) a (0) exp exp(Em M
(0)
Tem (ymi 0 xmm) (ld0) + [ [aq(011))p(
Then,
vxObj((t)))) = = (((n)((0=))) log
(00kk) M Lm (ymi O, xm)}
M VxObj((,,
((d0)
(0) exp
m=
And since o (0(x)) is fixed point of each client, we have that V Obb(m,, -===-
0. Therefore,
V Obj(m, g.)
vxObj(
This means that of (0)k") ) is an extremum of FEDGVI, and further that it is also an extremum of GVI with D = DrL. We now show that it is further minimum of the global GVI objective. We consider the Hessian VVx and proceed like before.
VV N O0.(Oijmmmpp dt)) vvn / (0)xx) log ((0kk) exp( - e; (f)re)))
uld0)
(0)xx²) exp Lm (y mi 0, xm
(0) (Im (ym Xm.) er (0) (0)x (udd)) VV (00)k: log ((0(k) 4: (0)x))) m(d0)
v..
= V.
(())x) (ym: 0, xm) Cr. (0) ((dd)
+ Vn
(/(n0(011)) log (0)kz
x0)) m(d0) + / |vllognt(m)
(2) u(d))
VVx
((0))s) (ym; xmx)) En (0)x))) uldd)
+ / (V log
0f ( k*) u(d0)= + / (V))(()<<)(V log (0)])) u(d0)
(0kk k)
Ashman et al. (2022) point out that this last term can equivalently be expressed through it's transpose.
( (o (((())))))(())))))
19

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
= vx / x(00x)(Vx (0)k=) a(d0) + / VV (Vx u(d0)
Vx / (00))))) ((0))z)) mdd0) 0
Evaluating this Hessian at K = R=:
v (b)(m,
V / (Lm)
+ / (VVn:(0k<)))) g(0tx)
(ym: 0, 0, xm) - (0 (x=) m(d0)
0, 0, m=*) uidO)
Therefore, when summing over the individual Hessians of the clients, we get:
M
M Σ m=1
V /
VvxObj(m,,
2.)
(Im xm)) - - (0)r=))) u(d0))
m=
M Tom (ym; xm)
M
= -V ((0))))
Σ
em (0)x=)) ((()))))))))))))))))))))))))(()
k=
m=1
m=1
(5(0 15*))
= v. ((()mx) log
1(d0)
π (0) exp ex)x > -m=
.7
Lm (ym: 0, xm)]
K=K"
+ V
H(g Zg m(d0)
4. (0k= k==)
Vv k:)) log
u(d0)
ग (0) expl M Lm (ym: 0, xm)}
which is a sum of positive definite matrices, and therefore, the extremum at the fixed point is a minimum. We now compare this with the Hessian of the global objective of GVI.
VV Vbjl()))) Vvx / ((0)x) log
q(0 k)
x x00) c(0)xxpf- EM Lm (Ymm O xm)] m(d0)
Vx (/VASTO122) log
g0
u(d0)
(0) exp( Em= Lm (ymi 0 xm
+ / (vx (00)
k)) m(d))
k))
V (x log
T Em= Lm (ymi xmxm) u(d0)
M
+ / ( (00)))()) u(d0)
Therefore, can see that, evaluated at K RR"
/ (V]=9(()k)) log
((0(==)
VV Obj(()) K=K
((0))xp( (10)exp(-Ent) -M - Tom (ym xmx) m(d0)
k==="
/ (VV (0))))))))))))))))))))))))))))) log
(0(r²)
( expl EM (Nm: xmx)) m(10)
K=K"
M
V xc Obj(m, qi)
k=
20

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Hence, the Hessian of the global GVI objective is positive definite and therefore have found local minimum at oz (0)x)) k*)
through FEDGVI.
B.4. Proof of Lemma 4.6
By combining Remark 4.1 and Proposition 4.4, we can show that, under infinite computational resources, specifically if we are able to optimise over the entire space of possible distribution parametrised by E O, then we are able to recover the Generalised Bayesian Posterior of Bissiri et al. (2016) in distributed fashion by partitioning the input data and solving several smaller optimisation problems in parallel. This is achieved by using the weighted Kullback -ullback-Leibler divergence at the clients and the regular KI. divergence the server.
Under the assumption that the prior is not misspecified, we can perform distributed Bayesian updating with our framework, similar to the Bayesian Committee Machine (Tresp, 2000) where we combine local posterior distributions. We aim to recover the Generalised Bayesian Posterior (Bissiri et al., 2016):
(1;; x)) (0)) 9GnI (0)k) = Ss exp(-3L(y: 0, 0,x)) (d))
where 3 is some parameter that controls the learning rate from the data.
We will show that using w = B at the clients will recover this GBI posterior after single iteration of our algorithm, and further that the algorithm shows convergence for any subsequent iteration. We assume that Q = P(O) and that Br (0)x y x) E Q. Furthermore, for simplicity we assume that the loss function L(-) is the additive across clients and that the data set is partitioned such that there are no intersections.
Proof The M clients have data sets {xm; m JM such that X nx3 @ for all k * j and write U = x and UM = yd to symbolise the entire data set.
Then we can rewrite the GBI posterior as:
exp(-8 ΣM L(ymi 0, xmm) (())
9GBI
M (0 = Se expl- -M' L(ymi 0, xm)) π (1)
The FEDGVI approximation then takes the following form: q. ())
M I expf- d100 (0) (
22, and as we initiate
(0) = we have that g. (0) π (0).
Then in parallel, the each client [M] carries out their optimisation step:
The cavity distribution can be found through division as:
x(0) 1 = π (0)
m (0) x expf (0)
(0))
And the Generalised Variational Inference step with the cavity distribution as local prior solves the following optimisation problem:
1 (Kl(q:m)
Pm
arg min
Eyton [L(ym: 0, xm)) )] +
B
.EQ
(0)
arg min E,(0)
log (T) cxpl- BBI m; 0 0,xm) xm))
Q
(0) exp(- : 0, /24 gm
Where (1) follows through the equivalence between the weighted KL divergence and the tempered loss as discussed in Appendix B.I, and (2) follows due to the properties of a statistical divergence which is minimised when the inside of the expectation is zero and since Q = P()).
21

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
This then implies that the update we send to the server is of the form:
Im (1) (0)
0, / =(0)
2 (0)
log (0)
-log
9
(y mi O, xm) + log gm
At the server, we can combine these such that we get:
M
(
Σ BLyym; 0, Xm) +
am ( = BLy x
4m
As GBI depends the prior and hence trusts it, the KL divergence the server, which is optimal with respect the GBI posterior (Zellner, 1988; Knoblauch et al., 2022). Thus, the GVI objective at the server becomes:
4
0
arg min Q
(yy(o)
(0)
DDIll:: })}
M
arg min gE@
E
BL M 0. 0. +
Σ
Z qm
+
DxL(q:
m=1
arg min gEQ
Eyror [3 0, x()]
DKI(q:m)
4m
((0)
arg min E (0)
log π (0)) exp(-BL(y 0, x }}
QEQ
i -L M, 0, M 13/2 00
(3) follows since Zg.. does not depend on 0, nor the variational parameters, and hence does not affect our optimisation problem. Line (4) is a result of Q = P()) and the assumption that the GBI posterior is contained within this set.
This implies that the posterior that we find at the server the Generalised Bayesian Inference posterior.
(1) ( = ग exp BLy x,x (0
Thereby, we have shown that FEDGVI recovers the GBI posterior under the assumptions and that this occurs after the first iteration. It remains to be shown that any further iteration steps will not change the posterior, and hence that we have recovered fixed point as defined in Proposition 4.4.
We repeat the client optimisation steps in parallel. We first find the cavity distribution:
mm
gs
(0)
(0) expl L(yx; O, xx)) (0) exp1-3 (xp(-3I((ym 0, xm))
Σ Lyva; 0 xR)} leff
exp( (I m: 0, xmm)
Note that we ignore the normalising constant, since, similar to the server side optimisation step before, it does not depend on the variational parameters nor 0.
The optimisation step is then given through:
( arg min Ey(or [L(ym; m 0, xm)] }] + D
TEQ
(())
arg min (,))) log q) exp( (I xmxm) ge Q
22

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
This statistical divergence is minimised
2 (0) g \m exp( - BL 0, xm 1/2 9m
π (0) expl -B Σ
L(y:; 0, xx)) -xpp--II(y.. 0, x/ qm
M
(0) exp -33
Σ
I(y 1/Z 9m
m=1
where we note that Z (2) = ( and we have recovered the GBI posterior we currently have as our server distribution. As
a result, A (0) = - (log qm (0) log q= (0)) = - log = 0 for all m E [M].
This satisfies the conditions for Proposition 4.4 and hence we have achieved a fixed point, which will not change the server distribution, since:
M
(0) (0) = 3 0,*
which means that the server optimisation routine would not be different from the one during the previous iteration.
L (0) arg min ()))
(0) + Dxilq:00
:)) m)}
arg min
(Eyo [o) + DKL(q: : m)} = (222
geQ
And thus qa
(0) (2) 0 go 2)
CG (0 (0ly ).
For the moreover part, we define the damping parameter 5 = 37 and show that de (0) CGBI ( x as + co. As the data here implicit, we simplify notation by denoting the losses of client as Lm and the GBI posterior as (G)) Furthermore, we assume that the GBI learning rate parameter is implicitly included in each client's loss. Then by the usual modes of convergence, we show that:
L (0) qGBs (0) +
Note that under KI. divergences at the server and client, will have that
(()
EM ell (0)
(see proof of Remark 4.1).
M plt) (0)
M Lm (0) Em
M
M
(0) exp
(0)
exp
(0))
exp
(0)
exp
Σ Lm 0
This converges when the exponents are equal, hence is sufficient to prove that Vm E [M] we have e!1) (0) - I (0)
Since for all E [M]. at each iteration we have that under the KL divergences:
4
( x (0) expf Σ M (t-
mm
(0))
Σ Eeh- (0)
x
= (0)
exp(
(0)}
expl- - 1t-
(0)}
(t) ( x exp Lm 1)]g (m)))
exp Lm (0) | (0) exp { - 1x
A (0) log
M L. (0)
1
p(c- p(c-1)(0)
x (0) expf Σ AM (
M
1
J.m (0) + M - plt- (0)
(0) = c 1) (()) + A (1) (0)
M
M
By expansion of dt (t)1))(0), by recursively applying the definition above, we get the following closed form expression: (0),
(t)))(0)= ((
1
M 1
M
1
1
M
M
+
M
M
M
+
M
M
p00)
m
(0)
Lm (0)
23

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
written as summation and recalling that ( = by definition, we can interpret this as the series:
[-1
M -1-1 )
e ( Lm (0)
Σ
M
M
2=0
which is a geometric series. And since
M-1
E (0, 1) by elementary analysis this converges, as t -> 00, to the limit M
lim e (0) Im (0) M t-000
M
Im
(1) (0) + QG (0) 0 almost everywhere. We can only guarantee almost everywhere pointwise Therefore, - 30 q=
convergence, since integral operators such as the KL. divergence only guarantee equivalence up to null sets.
Notably, the reason for using the cavity distribution instead of some other effective prior for the client optimisation step is that we want to recover the (generalised) Bayesian posterior eventually with our framework assuming that we can optimise over the entire space of probability measures that characterise their respective probability distributions. We further assume that we can find a global minimiser of any optimisation problem. Then, under these assumptions, we would like to not change the current posterior any further after recovering the GBI posterior.
We have previously shown that our algorithm achieves just this, and can furthermore show that the cavity distribution is indeed the only choice in the client update that causes this.
B.5. Proof of Theorem 4.9
We are interested in verifying whether the cavity distribution is necessary in Equation (4). It acts regularise the optimisation problem at the client, which we restate here, using some arbitrary probability density p E P(0):
( Im
arg min gEQ
(.(())
[ (ym: 0 xm) + D(q:p))
where it is regularised by D(: p). It is clear that this should not be the prior distribution after the server has additional information about client data available since we would not be doing anything different for subsequent updates and this would result in a Bayesian Committee Machine where each client does not learn from the others. Therefore it is imperative to ask what this "effective prior' should be? And in fact it turns out that it needs to be the cavity distribution.
We will approach this problem by considering the case where we know what we would want to target in the optimization problem and hence the sequence {q. (0)),e,, should converge to. We, however, have to restrict ourselves to the Federated Learning scenario and therefore any distribution that we come up with needs to satisfy the Assumptions 4.7 and 4.8. For this we require the following assumption so that we are able to target the GBI posterior.
Assumption B.2. We are able to find global minimisers over the entire space of probability distributions parametrised by 0, P(O).
Then it turns out that this regularising distribution is uniquely described by Theorem 4.9, which we restate here.
Theorem 4.9 Let the assumptions be as in Lemma 4.6, i.e. Q = P(O), D DL for 3 > 0, D2 = DKL I L,, and
Tm L, and further assume that Assumptions 4.7 and 4.8 are satisfied, then the following are equivalent:
(0)= 1. 3t E for which of = qGBI(() (a.e.) is invariant under further FEDGVI updates.
2. The cavity distribution regularises the client optimisation problem.
Proof (2 - 1) This is a direct consequence of Lemma 4.6 and can easily be seen by iterating through the algorithm with
the cavity distribution.
(1 - 2) Without loss of generality we consider the GBI posterior to be found after the first iteration. We show that the unique way that satisfies the axioms and does not change the GBI posterior at the second iteration (or any further iterations) is uniquely achieved by the cavity distribution. By the statement we have
() (0) exp( ( 0)) (0) expl-ee" (00)= (0)
24

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
We now need to relate this the client updates and hence the solutions of the client optimization problem.
2
(0)
exp(-4 -xp((=
(g) = (0) log Z!!)
(0)= CER
M
def
Σ
A (0) 60 (0) = ()) m
M
(0) C
(2)
m
M
9m = C
log
98 (0)
m12
4m (0) K (0)) M K k=c
(10)
Now, for some transformation operator &m P(@) + P(O) acting on the information available at the client from the server the form of the current approximate posterior, which we denote as Em [g (0). that satisfies the Assumptions 4.7 and 4.8, we get the client optimisation problem Vm:
arg min (()))) Ey(() [L(ym: xm)] + Dxllg Emlo!"1) }
1 3
Pm
0
log (()) Em Igs (()
arg min (EP(()
1o) 1- log (xpp-L(xx 0, xm]Hl + - 2000
3
(0)
arg min
3 Ey)(
log
(EP(O)
expl (yyl- O, xm) ) 5m 11s
(1) ((0)
2) mm
0)
exp(-3L(y O, xm)) (mm (g)))
Substituting this into Equation (10) and using the definition of q: approximations.
(1)
we can derive a relation between the individual client
M
2) (0) = (65 (0)) M
II
m=1
M
&m [gs
(1) (0) ( (y 0, xm)
M
L(y O, xm) } (2
II 122
((T(O)) exp
-M3 Σ
(22)
II Em
L(ymi O, (,xmm (0 (0)/2/) K(T(O)) exp -(M -(M--18 -1)8 Σ
} (12())(2)) (2()) M
(1) (()
II Em (0)
II TT (0) exp
3 Σ 1(y=;0,x, kymm
exp( (ym 0 x.,))
Here, proportional 'xx" means equivalent up to some constant independent of 0. To see that the cavity distribution is in fact the only choice that satisfies the above equation, need to recall the two axioms: (Assumption 4.8) &m (q. (0) needs to be generated in the same way across clients, and (Assumption 4.7) since we are in federated learning, each client will only be able to access it's own data. This implies that we can write Em [g (()) as function of the current approximation and
25

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
the client data, Sm Ga (1) E(1e (1) ym, (f).
(1) qs
(0)
II xm.
II
( pmm)
The only client that would have access an explicit expression for the denominator would client m, which the data {Xm= J belongs, and hence it must be entirely contained within that client's regularisation term Em. Therefore, we can
conclude that (2) (0) = qs (()(()) (0) and find closed form for Em[gs ((0). Note that this implies C' = -0 and hence K = 1.
exp ( 0, Xm )m (0)) = expl-
Σ
3 0, Xm) (0)/2
9m
m==
expl >m m= BL mi O, xm ) (0)/Z 40
Sm 1o0 (0) x
expl -B(ym: mi 0 xm //
qm
This is exactly the cavity distribution as described in Equation (3).
This gives a justification for using the cavity distribution in our algorithm, since under the assumption that the prior is well specified, we would like to converge to the generalised Bayesian posterior distribution. Furthermore, we can note that this single step of FEDGVI recovers the principle of the Bayesian Committce Machine (BCM) of Tresp (2000) where we use generalised loss functions instead of the negative log likelihood in our formulation. Furthermore, single pass through FEDGVI-with the divergences as described above- will recover generalised version of the BCM irregardless of the space we optimise over.
Remark B.3. For the last two proof' we have assumed that we can find the global minimisers of the equations. This isn't t strictly necessary to have since the use of the (weighted) Kullback-Leible divergence allows us to formulate a closed form expression for what these will look like.
B.6. Proof of Proposition 4.10
This proposition is direct result of Proposition 3.1 in Altamirano et al. (2023) and the proof is analogous, we merely include here for completeness. And while the stated result is in a regression setting, can be be extended the classification setting similar to Altamirano al. (2024) where Gaussian Processes considered.
We assume that each client has data set Hxm of size 7mm The divergence operator V [xx is defined in the usual way as the inner product between the vector of partial derivative operators and the vector of some vector valued function fxx as V fxx = ((0/0z1,.. ..., (a/a ( (x), fa(x)) T), and Vxg(x) is the Jacobian, the vector of partial derivatives of g(x). We further assume that = C R&d, and that pa (x) E P(O).
Proof The loss of some client m E [M] at some arbitrary iteration f [Z'] given by
D(0, IP Pm.....
=
Vx log (x:))1 +2 +
(t) (t) (w un m V
log po lo
(x:))
(1)
(2)
where Vx (x:) Vxn (x,)) <xxss). We can then expand the terms in the above terms which we then give equal up an additive constant independent of 0.
(1) = (wit) (vxΦ(x;) (0)) + vah(x))) (w)(l) ( (vxσ(x;) T "(() + vxh(x;)))
= (wl)) vxΦ(x;) (0)) ((())) (w(h) vx0(x,) + (wll) vxh(x;)) T (w!l vxh(x;))
2(w
(1) VxΦ(x,) (T((x,,))((()))))))))))))))))( ((0) ((w(()) vxh(x,)))
n)( T VxΦ(x))wl!) wo()) T Vxh(x;) =o (0) V Vx w with VxΦ(x,) T (10)
where the last line follows since the middle terms are independent of O as long as the weight function is independent of e.
(2) = (w/(1) ((()) Vxn(0) ((xx)) V (we(t)
(mn()) ()( vxh(x;))
26

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
= (V. (w0 will
vx((xx)))
Then, this has the form D IP) -o ()) T Am ((0) + n() "mm, where
1
(x0((x) w w (xx(x)) and
2 m
2
( (0 T (wil
vxΦ(x:)).
m
1m
1lm
==1
The first art follows by setting qm x
(())
Am (0) expl 1mmmmmm77 (7) T Am (0) + (n) [m)) Then, if (0) = 0, and
the local cavity distribution has the form
((0)
km (0) x exp( - - (0
Am pm (0 M mm }} then the local posterior is
(t) T (1)
conjugate and is given by (1) (0) x exp (0- pm (1) ) Σ (0- pm }} where
ai (0) x 4(t)
( exp( - mm (7) T (1) (0) + n !
x expl
(0-
(1) (1) -1 (0- μ mm
(t) exp 3mm (0 A 0 + 0 [))
2
mm
x expl
0
x(()))
T
20
(t)
im + 2m3 0- A + 2.3 0- v [
2
mm
m
1 (0- (
x expf
(e (
2
where in the last line, we complete the square and get parameters
0 -1 m = (1)) m +
A (1)
and
p 10 = 1
\m
p Mm
-A ((9)
The moreover part can now easily be seen. The update will be quadratic in @ and hence summing these results in quadratic function, and since the posterior will have Gaussian distribution, the expectation with respect to the posterior of this quadratic function will have closed form. Therefore, the divergence at the server allows for closed form solutions between Multivariate Gaussians, then the entire Equation (7) will have a closed form optimisation procedure that does not require sampling to approximate integrals.
Note have implicitly used the weighted KL divergence with parameter 3mm Note also that this does not immediately follow from Lemma 4.6 since the weighting function is allowed to change depending on the iteration and the client. In our experiments, we for instance use the weighting function as measuring some deviation of a data point to the cavity mean. Furthermore, the weighting function does depend on the data point, but we suppress this dependence here to lighten notation.
B.7. Proof of Theorem 4.12
This result is more involved to prove where we show by induction that at each iteration, the posterior generated at the server is robust to outliers through the robustness of each client's loss function to outliers. To prove this result, we first consider what we mean by robustness and introduce some terminology. We consider the empirical data distribution of all clients ΣT 6x. which perturbed by some Huber contamination with parameter E at some adversarially chosen data point F. n2 P n, = (1 (1-=)1P,, E) + E6g. where the subscript n indicates how many data points are drawn from the distribution. Note that Pn = 1 M m= Σ 8xm. = 1 EM Pm nmm. We then write qu
2 E
(f) (0; .P....:) to indicate a distribution with respect to data generated from the specified DGP. We first recall the notion of robustness introduced by Ghosh & Basu (2016a). Note that we suppress the measure p(d0) in the following and simply write d0. The posterior influence is given by:
(4) go (0; Pn,,s) 2:th) ( (0: (0.P.) d 1 (0:PP
(I P.). = lim $10
E
de
where the last line follows by L'Hopital's rule. Ghosh & Basu (2016a) further show, and one can easily check, that for (c)
L (0: P Pn....) = x =0f) exp (::(0=)000000
(0: P. PM,,,))// (x00))xp(()) (P,,,,)))d0 this is equal to:
d (- (0: (O 1emo + / d o000 (0: e (- ds
PIF (2. = 0h 12 (O.P.)
We call loss robust if it has finite posterior influence, ie. supece supzes [IIF 0, P.)) < co. To this end, we now state a Lemma due to Matsubara et al. (2022), adapted to our notation for FEDGVI which we have rephrased into Definition 4.1 1.
27

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
(1) Lemma B.4 (Matsubara al. (2022)). Let 4:))
(0:P..) be a posterior computed at the server or the client with fixed n E
with loss (0:Pa) and prior π (0). Suppose that g(() (0; n.) is lower bounded and that (0) is upper bounded over 0 + for any Pn. Then if there exists some function
O R such that
(0: P n. S (0)
1. sup =E de
2. sup π (0) (0) (0) < 00, and
OEO
3. / π T(0) (() (0) 0
hold, then 9:) (0:P.) is globally bias-robus.
We provide further clarification on these conditions in Appendix C.2, and we are now able to give the proof of Theorem 4.12. Proof By the Lemma B.4, need show that
d (0,P S ())
sup ZEE de
and that this - satisfies conditions (2.) and (3.) of the Lemma. Per assumption we know that the clients are robust to likelihood misspecification, so we need to relate the server loss to the client posterior influence functions. To this end, we consider the loss at the server.
(0 (n.,))
Σ A (0: + (O:
where for each client, the update is given through Equation (5)
(()) (0; Pn,, exp( mm (t) (0; IP km)s
Am
9ie (0; (2,,2, expl 3m I 1! (0; P. n.c,;) d0
!!!) (0:P,,e,s))
log
Qm
(0; PP ).)..))
-loog
( 1) (0: P n,,,.))
(
() P Pn...)
go
18
(1-1)
(0; Pn,.2)
exp
Bm
(t
(A; P. P.,,,)))
exp(-3nmmde (0: (nme))))
[t 1)
0 1),,s)))
10
exp 8m Lu
(0; (..,.)))))
4s
[t-1)
( P. F...))
f exp 8m
L
(11-
(0 (....))
exp( - 33mmmm (1) (0)
(0: P s.
qu
(t-1)
(0; P Pn...))
d0
exp(- 1m3 T
(1-1)
( PP Pn...)))
log
(t 1)
(0; P,,,,))
exp - 3rm Li (0; pmmm)))))
exp Bn Lu (0: p))
(t-
log
9s 0 (2...))))
(1 (t-1) 1)
expl - -B Lill (0; (F (mm...)))dd
exp( { 3m (t-1) (0; TP (n,.aa)}
exp( Bm / (t)
(0; )mmi)t))
- log
+ log 2(t) (Pn,,.)
exp( 104--3mm 3m (1- 00 Pn,,,))
28

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Therefore,
M exp( - 3m Lim (0; Pn nm em))
(1)
e (0;10 n,,,2)
log
+ log Z (Pn,e,)) + + 1 F,,,,))
exp( Bm Lm (1-1) (0: P [n,,,)))
m=1
M
M
log expl- -m mm (f; P
umme
log 7 (Pn,e
m=1
m=
1=1
Σ
81mm I (0;
P. mm
log 7 (P (Pnn,,))
We will now show by induction on that the posterior at the server is robust. Concretely we will show that Vt E TT. T E N, and M N finite, then
d
d 1 (0:P
d 2
sup xES
pl0 (0: ==0 S 8
m sup ZEE| de
(0:Pmmmph mm e.)) c==
sup =EE
log
(IP (Pn,,,)) c=0) S (0)
de
de
such that this function ?/ (1) (0) satisfies the conditions of Lemma B.4. Note that the first inequality follows by Minkowski's inequality.
We begin by considering the case where t = 1, then we have qe (0: = x (0) and ( (0; (0:Fn. Fn.,,)) = 0 as
initialised in the algorithm.
Consider the term de log (0) (P (nneei))ee then we have
d
d
d 7m (P k,,,
log 1 (Pn.e,)))
de
de
( (P n, 1==0
l 9: (1-1)
(0; P. Pn,,,s) cxpf expp-Bnnnn (I
d0
de
expl Bm Lm (1-1) ( pmm,,,))))
2=0
2 (Pnm)
{ π (0)
expl -mm (1) (0; P ()mme)))))
c= d0
2m (I mm
= =-/ -1(
3mm L (0; PP nm s, x )e=
(1) Pr.m. d0
de
where the last equation follows since de exp([(x)) exp (xp( ) ((x). Consequently, using Jensen's inequality
d
sup =
(!1)
(0: P
P.n,,
de
M ())0; m.m....)) 1c-o
M Σ
/
d 3r
$.3
Rmm
sup
de
sup m=1 SE
de
1 (1) P. p.m.)e)))_))))))
(I)
(0; PP Pr.. dd
=E
m=1
M
d
< 3
Rym
sup #EE
L (1) (0; (m))))) le=0
+
sup =E= de
(1) (0;
P
(1) (0; Pm dd
ds
m=1
Then, if 1 ) is robust, then there exists some function
7m (0)
(1)
R such that
(1) (0; P
(1) (0) and which satisfies:
sup
1=20
7m
sup ग (0) (1),0)))))))))))))) (0) < co, and / (0)) < co.
Ae
29

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Substituting this into the above, we have that
M
sup
= de
d -
(1) (0:PP,
5=0
S B
m
(0)+
Im
7m (0)g (0;P )d0
m=1
Now recall that qm O0 p..)
x (0) expl - mm ) (0; pm }1/2()) (I nn.)) and per the assumption we
have that the loss is lower bounded and that < ( (Pnm) < therefore (2) (0:PP I
(0) expl- -- inface Li (0; 11/2 (IP mm S C(l) π (0) so that,
M 3
d (0:
(1) (0) + C(1) / (0) =: (0)
Σ
sup EEE de
nm
m=l
We now verify that the conditions hold. For condition 2, we have
sup
(0)
< 3 Mmm
M
(O))
sup ा (0)
c. f 7mm (0)=
< 00
OE
m==
which follows by the assumptions on the robustness of the loss and that the prior is upper bounded, as well the finiteness of 3, T'm and CHll
Condition 3 follows similar reasoning.
/ 19 (2) (0)d0
Im (0) CC0 / / (0) (O) T (0)dd
Σ 7m
(0)
m=l
nmm (f x (0) ( OO)d
π (0)) (ʃ (0) (0)(0) d0
Σ 7mm (fz n (0)-
M
+ C( / mm (1) (0)=
3
00
Since the loss is robust, the integrals are finite, and since all other terms are finite, we conclude that condition 3 is also satisfied. Therefore, for = 1 the posterior computed at the server satisfies the conditions of Lemma B.4 and is therefore globally bobus.. remains to be shown that this holds for all E N such that t S T, i.c. is finite.
We now show by induction that if the posterior the server is robust for k. then will also be robust for
d (k+1) (0: (0:Pn...))
d (
d
ds
3
Tmm
de
Lm
(0:Prm,,e,)))
0=
de
log z (Pnne,=)lemd
m=1
d (k+1) (0;
k+1
de (IP (Pn,,,z)lem
d
(+)
3
nm
I'm
de
Z (P (Pn,e,))e=
(1)
To show the boundedness of this, we need to consider the expansion of (1) above.
d
(t-1) g (0; P. (.....:)exp)) exp( Br I (0; P Pmmn):)))
d0
d de log 2 (I ((P.,,)))==0
d 7 (TP (Pn.e,))=== 2i!ll (Pr,,,))ee le=o
de exp
3r 1tth (0: P Pmm.e.))
2!!t (Pa))
Now we consider the integral in the numerator. Using the chain rule when differentiating under the integral sign:
f de exp( Br (1- 10-1)(())))) () nmm,,,))
d 93 (t- (t-1) (0: (0:P.. Pm...) expf -3 I (mm...)))
d0
===
30

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
expl-
mm I. 1 mmm)
(0: Pn,
de 9s
exp(
mm Im (t-1) ( IP mm..
s=0
(t-
(0: (n) expl 3m m Lm (0:P }} (-Brm L (0; Pnmeee))
+ qa
expl - 3m Lm - (0; 1n
de
==0
qa (0: ,.. - Li (0:P }} d
de (-
km))))))
c=0 d0
expl- 3mm - (0:1Pm.) mn
Bringing the denominator back, and recalling the definition of qm (0; Pr..... ). then we can simplify.
expf
31 L (0: a...a..)
d
expl - mm Lm
11-11) (0: P
}1
d
log (PP,,,, (Pn,,,)))e=00 =
(0: (n.,o)
de
Z (Pn)
ds
5=0
q 1) (0: P,) exp( 3n Lm mm: }}
expl Bmm Lm
(0; 10:1Pma nn))
d
(3 (0: (m:Pn.e)))) 1=0
7 (Pn)
de
=yh"'(0: (O:Pm))
(t- 1) (0; P,,) exx))pp - Br
(1) (0; Pm
expl 3m Lim (6-111 (0; PP (nn)) )}
d
(8nm - [(t-1)(0) nm))))))
d0
z (Pm)
ds
==0
=qm (t) (0:Pmm)
exp Br (0: (0:Pmm))
(2)
d (0;
(1) (IPn) expl 3r 1 (0:P,, (m.P,.)) }} de 9s
==
d (Brm (0)(()))))))))))))))
(0; PP
mm (0; Pr, Pmms5,))) ))
de
==0
d
mm
(0; IP
mm
(1 (0; P F
d0
c-0
Consider now the derivative of the previous server posterior with respect to evaluated which we can write as:
d
d π (0)
exp
E
00 Pn.e.))) )
4s (0-")OPP
de
==0 de
1 (P....))
c==0
[expl 1 c (0: d
(0)
(- (0: ==
z 11 (Pr)
de
exp(-e5 (0-1)) (0: pn)]
(()) d de d d( (0: (0:
expl - (0: ==0
d0
(Z. 1 1) (PP,))2
(0:Pm)
d
(0:P,) s= - f 00 (0:
d (0.P..s,e)
dt ((
d0
de
de
==0
where we have used the definition of
dz (t 1) (0)P,) ) by distributing the common terms outside the brackets and for the second term, since the normalising constant does not depend on 0, we can take one of them inside the integral. Furthermore, using the fact that
(0; exp( - Bm L (0; (nFFn)) expl -Bmmmmmmm [ (1-1) 1)
(0:P.,. mm }}
(1) (0: P m.
2 (Pnn)
31

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
then substituting the result for de 24= (0:Pne,)))) into d log 7nd (IP (Pa,.,)))==0 we get that: de
d de
d (0; (n,,)),,)))==oo log / (en,,,))===0 = alh (0: (- c=0 + / (0:
de
d 1 (0 mnmmee)))) + fmm d ( nm ..=)) (==0) d0
-m
de
de
Substituting this expression back into the original equation for 4 1000 (0; Pn. taking the supremum over 2 E E of de
the absolute value of this, and applying Minkowski's inequality, results in the following upper bound.
d g (k; (0:PP, .,...:))
sup FEE
de
M
mm sup =E
d Lm
(0; P nm e,,
( (n))
sup
d f (0: (0: (M:
de
LEE de
m=1
m=1 1=1
sup
d
p (1-1)(0) (0: PeP.n.ee, n,,
c=0 9s
1/-0111 1)
(0: (0:PPn)) (.)
Bm sup =EΣ
d de
(0: nm e,=)
=EE
de
d
+
3m sup
=E
de
I.
mm
(t-1)) (0; P. Rm.
<=0
d0
By the inductive assumption V/ E [k+1]] 1]. 3T (1-1) (0) such that sup de 1 (0; (0;Pn,,,)e==ll S 7/0 (0). Additionally,
as, by assumption, the loss is lower bounded and robust 3y/00 (0) Vt E [k + 1 such that sup,e= - (0: P (..Fne,e))eee -
Im (0). Furthermore, these functions satisfy the conditions of Lemma B.4. Note also that q
(t)
- (0: (0: S C c! ( (0).
since the normalising constant of this distribution is finite and the loss is lower bounded per the inductive assumption, we
get 93 (t-1) (0; P,)) < (0) expf- infoce e((-1) (0:Pn))/Z," (1-1) (P.)) S ((t-1) (0), as seen in similar arguments before.
Utilising this, we conclude:
M
am (0; (0, P nm {f 1 ( (0)
nm Ti
(0)
+
m=1t=
+ Bm 1 (!)(() + 3n 1m (t- (0) d0 : 11+++))))) (8)
We now need to show that this satisfies conditions (2) and (3) of Lemma B.4. Let's recall what these conditions state:
(2) = sup π 0
(k+1) (0) (0) < 00
(11)
(3) = / ग (0)) (k+1) (0)d0- < 0
(12)
We first verify that condition (2) holds.
sup (0) oee
(k- 1) (0)
sup π (0)
Bnn Tm (k+1)(0)
oc
M A+1
(1) (0; Prom )
{ 7e (1) + c((-11) / (0)= + 3m m (0) + 3m 7m 3m
(t-11
(0) }
Um
M
M k+1
< <80m
sup π 10 (k+1)(0))
+ sup π (0)
ΣΣΣ
/
am
(2)
(0:Pmm) (2(0)(0)
0E
o
m=l
+ 1)--1) / (t 1) (0) (0)d0 + 3m (1) (0) + 3m 1 (00)
d0 <x
Since B, Tym, and M are finite, and any finite linear combination of finite terms is finite, we can easily see that the first part finite. This follows since 7m (k-1)(0) (0) satisfies condition (2) of Lemma B.4. Furthermore, since I (0) is upper bounded, we
32

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
now need verify whether the inside of the curly brackets is finite. Since this is finite sum, we need to verify if Vim [M] and Vt E [k: 1], the following holds:
{ (0) cL--1) / ( (0) + -m,mm. ( + 3mm 7m (0) d0 < 00
(0: Pr..)
By the inductive step, this is true VI E [k], so we need to show that it also holds for ( = k + So.
/ [k+ (0; (0) + c / % (x))) (0) (0)) + Bmm 7m
(()+))(()))))))))))))))))))))( 3m 700 (0) d0 < <00
(A++1) (0; is equal (0) exp(-Bnm Lm (0; In. )) expl- Eifm n
(++ =! and (1) (0; In: 13//z/+
Note that qm since the normalising constants finite and positive, and the losses lower bounded, then can write
pm gm
(0: I π ex)) fmm inf L (... )} expl-x 0E0
Σ i&m
n, inf [,(0) (M. (t.PP.)///htt22/) AEO
C(k+1)
where 0 (k+1)
<0. Thereby, we have
Am f
(k-1))
(0: n
(0)
(0) + c((K) / ( (0) - Bm mm (0) + Bnm m (0)
(0)(() d0
clk+1) /
(((0))((+)) (0)))) Cm C( (ʃ (0) (/ π (0))0
+mn C(K++) f (0))mm (k+1) (0) + Bnn.. ((k+1) / (0) (8) (0) < 00
This expression is finite since the individual integrals must be finite by the definition of the bounding functions 7 as these need satisfy condition (3) of Lemma B.4 with the prior (0). Hence, have shown that condition (2) holds for 7s and Equation (11) is indeed finite.
(k-1) (0)
remains to be shown that condition (3), Equation (12), also holds. Using the same expression for 7 (0) as before, we have:
M Σ
/ क (0) (k+1 (O)d0
= f T())
Brom (k+1) (0)
7m
m==1
M
( (O; mm of (t---1 (0)-
(
f
+ 3m, 20 (0)=
+ Bn a
Tm (0) do
m=
Since, the summations are finite, we can exchange the integrals and sums to get
M
) 2(-1-) c( f (/0 (((-1)
Bmm
((t))
+ (0)d0
Σ
am (0: P nm
<oc VmeEM
+ Bm (0) B3 7m (0) (0) )
(1-1)
(k+1) where the first part is finite since for each 7m (0), we have by definition that this expression is finite as it needs to satisfy condition (3). Therefore, we need to show that the summation is finite. By the inductive step, this is true VI E [kr] and we will now show that Vm E [M]] it also is finite for = + 11
f nn (k+1) (1;1)
(1) + C(()) / (0)= (0)d0 BT 7 3 (k++))()) Br 7 (0) 7o0
33

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Recall from before that (m (b++1((0: (r..) S C (0) and hence, is now immediate to see that by the same argument as in the proof of condition (2), this integral finite. Therefore, condition (3) of Lemma B.4 also holds and Equation (12) is true.
We conclude that all conditions of Lemma B. are satisfied.
Therefore, by induction, as long as we have a robust loss function (in the sense of Ghosh & Basu, 2016a; Matsubara et al., 2022) at the clients, then irregardless of the current iteration by using the weighted KL divergence at the clients and the KL divergence at the server, FEDGVI achieves global bias robustness to outliers.
Note that when assuming that (1++1) (0; nn.) I C have used that the normalising constants in the well specified case are finite. This is necessary to hold, since otherwise we will not have valid distributions, and furthermore we can always choose prior distribution that bounded above so this will always be finite. However, this finiteness is not assumed for the normalising constants that are contaminated by the outliers, so the proof is needed to show boundedness of the posterior influence under contamination.
т (0), or similarly at the server in the uncontaminated case, we
C. Additional Details on VEDGVI
We present some additional details on FEDGVI in order to aid clarity and contextualise it in the broader literature.
C.1. A Note on the Learning Rate Parameter in GBI
The 3 parameter comes from the power/cold/temperrd posteriors of e.g. Grünwald (2012), where the likelihood in Bayesian posteriors is raised to some power of 3 > O. This was originally done to add some robustness to the posterior, down- weighting observations if < and up weighting these for 3 > 1. Through a known result (Knoblauch et al., 2022) which we highlight in Lemma B.1 in the Appendix, this equivalent to having weighted Hallback-Leibler divergence, KL. This also allows us define if want to trust the prior more weighting the prior and vice versa.
less / > since up weighting the data means down
C.2. A Note on Definition 4.11
The three conditions combined allow us to say whether the client posterior (or simply the posterior in global, Client, GBI setting) derived from such robust loss is provably robust to Huber contamination.
From Condition 1 we are able to bound an infinitesimal change in the loss with the contaminating data point 2 by some auxiliary function 7, possibly infinite for some values of 0.
Condition 2 states that the product function, (0) has finite uniform norm. This ensures that this product under the worst case contamination and the worst parameter 0, is finite and hence it cannot be made arbitrarily bad, which does not hold for the negative log likelihood in general. Alternatively, the prior decays to zero faster than the auxiliary function can diverge to infinity in 0.
Condition 3 further says that (0) is finitely integrable, i.e. that this is in L' L',,, This, in effect bounds the normalising constant of the contaminated posterior and will ensure that this is finite.
Taking all these conditions together tells us that the product function x (0) is in ()),L)) ju). and that is in fact finite everywhere. These two conditions that are mutually independent so both Condition and need to hold. Equivalently, we require that (0) bounded and integrable with respect to the prior probability measure π (d)) =: II(d0).
These conditions characterise the notion of robustness use for Theorem 4.12, with derivation in Appendix B.7, by considering the worst choice for the contamination and the parameter 0 with respect to small perturbations of the resulting posterior through s. The influence of the contamination z and parameter 0 on the posterior is defined as de 9m (0; (P......=)le-d which is bounded through the conditions. Our result then implies that the posterior is 'globally bias robust", i.c. robust to Huber contamination.
d (t)
C.3. VEDGVI in the Context of GVT and FL
When viewing GVI/GBI as an optimisation problem on the space of probability distributions P(0), Bayesian inference, VI, hierarchical Bayes/VI, all target single element of this space. These methods either target the standard Bayesian posterior
34

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
explicitly, or the posterior within some variational family with closest Kullback-Leibler distance the Bayesian one (Blei al., 2017; Walker, 2013). Through GBI and GVI
able target different elements of subspace of P(O), then simply single point; in that regard, these approaches 'generalise" Bayes. In this paper, 'generalised" is inherited from GVI and GBI. We should note that in the FEDGVI setting, GBI and GVI allow us to generalise PVI or FEDAVG to a broader subspace of possible posteriors. Figure displays FEDGVI in regards to the related GVI and GBI literature as well as the FL literature as in Figure! 1.
GVI D,D,Q. e M = 1, D, = DKL
GBI L, DKL, P(0),
BAYES log DKL P(O), M = 1, D. DKL
M = D, = DKL
ERM
VI log po. D 2 M = 1, D. = D
FEDGVI L, Q, M,D,
L, D = 0, (
M = =
FEDAYG L, D 0, {60}
PVI
log pe: DKL 2. M, D. DKL
M. D 0
Figure 8: We illustrate the relationship of FEDDGI by the loss I., the client divergence D, the variational family Q. the number of clients M, and the divergence at the server D,-to to Generalised Variationall/aayeiian Inference (GVI/GBI), Partitioned Variational Inference (PVI), Variational Inference (VT), Federated Averaging (FEDAVG), Empirical Risk Minimisation (ERM), and Bayes.
D. A Additional Details on Experiments
For reproducibility we give additional details on the experiments that we have carried out to empirically support our contributions. Code to reproduce these can be found at:
https:///ii thu..itthbbb...
D.1. Normal-Location Model
We assume the following well known model for the Data Generating Process and prior, with some unspecified prior mean Ma, in order to allow for prior misspecification:
0 N( 12) = x (0)
21.N 0 id N(,, 12) := ())
The true Data Generating Process under model misspecification through Huber contamination given by:
No(00.53
35

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
T1:N
iid
(1 (1- (1-E)
2), 19) + + 3),
where the second term represents some noise fraction that is added to the data. Our aim is to find the location of the first term in the above model, while modelling out the noise from the second term.
We consider PVI where the client optimisation is given by (- KL,N), and the server optimisation step by P. ((e.)( (0), .L..). Under the assumption of likelihood misspecification,, consider the following divergences and losses at the clients, while leaving the server optimisation step unchanged: The weighted Kullback-Leibe divergence 1 DXL. the Alpha-Rényi divergence AR the Fisher-Rao divergence DER: the score matching losses 4 the beta-divergence based loss - and the gamma divergenee based loss 4 Expect for P... ( w NV). which allows for conjugate updates by Proposition 4.10, we have to resort optimisation. This however does not require Monte-Carte sampling since the divergence terms and the losses have closed forms under Gaussian distributions, remaining losses, Pardo Llorente (2006) for the KL and Alpha-Rényí divergences and Nielsen (2023) for the Fisher-Rao divergence. For the optimisation, we use the Adam optimiser with learning rate of 0.001, leaving all other parameters at their default values.
Knoblauch et al. (2022) for the
Explicit Losses and Divergences used As mentioned in Section 3.1, we employ a range of different loss functions and divergences throughout the experiments. The main one being the robust generalised cross entropy used in the real world experiments. For the syntheties, for instance in Figure 3 we compare four different losses with two different implementations for the Score-Matching loss.
For this example, where only have sequence of data points Vi.N which are assumed be independent, the losses
1. The Negative Log Likelihood:
LNLL(TIP po) = log (1:)
2. The Density-Power Divergence based loss (Ghosh & Basu, 2016a;b):
1 (zi, po) = - - + + B po(x) u(dx)
3. The Gamma divergence based loss (Hung et al., 2018):
c ( Po) (7 11 Po(=)))))
7
(o=ro(z)) u(dx)) =-1
4. The weighted Score Matching Loss (Altamirano et al., 2023):
(wl)) C
(I,. = || (x:) T Ve po + 2V (w (z, (0 (II) V (o
SM
(1) We use two different weight functions wm where ym is the mean of the cavity distribution:
(a) The Squared Exponential Kernel (SE):
(r; - pl )2
(1) (xx) = Bexp
2c
(b) The Inverse Multi-Quadress Kernel (IMQ):
(x:
pl
(4)
)2
(II) 3
1
2002
See Appendix B.6 for details on what this posterior looks like using this particular loss.
36

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
All the above losses have closed form objectives under the expectation with respect to the approximating distribution g and the assumed Gaussian likelihood. Furthermore, the negative log likelihood and the score matching loss admit conjugate updates under the KL divergence.
We also use different divergences, mainly:
1. The Kullback-Leible divergence (Kullback & Leibler, 1951):
OKL(q:m] m) -/o 9(0) log (()
(0) p(d0)
2. The Reverse KL divergence:
DRKI(q: σ) x== DxL(T: :q)
3. The Alpha-Rényi divergence:
D (a) A
a(1 a) log q(0) (0)) (0)) 1-a u(d0)
1
4. Weighted Divergences of the form:
Dig : T)
3
For Gaussian distributions, these have closed form solutions.
D.1.1. INFLUENCE FUNCTIONS
For the influence functions experiment in Figure 3, we still assume the same likelihood function, but have different data generating process. We generate 99 data points from the following student-t distribution with 4 degrees of freedom, mean 0 and scale 1.:
T1:99 Student T(0, 4)
T100 (z)x, y RR
We place Huber contamination the hypothesis, where we add an additional observation one of seven clients that is increasingly farther from the true mean, and calculate the posteriors with this outlier, y. We have used the losses described previously for the posteriors and the Billback-Leibler divergence, rumning all experiments to convergence. We compare the resulting distributions using the Fisher-Rao divergence (Nielsen, 2023), which has closed form between two univariate Gaussians g(0) ~ (M,, o and T (0) ~ (Nu,,
A (juy Bq : Mo, xx)
Den(g : ω) v2 log
AlUgg 0g 0x)
(c a)² + (d b))
Ala,b:c,d) =
(d, e E
a)2 (d + b)2*
D.2. Logistic Regression with Gaussian Design
We place mean field Gaussian distribution over the parameters of linear model T + by augmenting the data to = [ ] in order allow for non-normalised data sets. We assume that the labels. y; {0, 1}. follow Bernoulli
distribution with sigmoid probabilities:
yv (0 x;))
where (a) (1 - is the sigmoid function. This allows to define the likelihood follows:
ply 00 x;) ex (x] 0))
37

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
where v(a)= log(1 + e²), which gives rise the sigmoid through ((a) (f((a) (Katsevich Rigollet, 2024). We this exponential family form above since taking the logarithm for the negative log-likelihood is easily achieved by removing the exponential and allows for slightly faster calculations during the optimisation. Further, assume that the prior π ()) = (0 Σ). where Σ is fixed but generated through sampling from a Gamma distribution and averaging over the samples. More specifically we sampled 100 samples from a Gamma distribution with E1: ~ Gamma (1, 1/0.01), and use their mean, ç. to define Σ = LI. This was done to ensure fairness with the Distributed Stein Variational Gradient Descent approach of Kassab & Simeone (2022), who use an Gaussian inverse Gamma prior, which we for ease of implementation forgo (the results of the experiments show that we easily match their performance, if not surpass slightly). For the prediction, we use an approximation to the expectation with respect to the final distribution found, qa (0) ~ NNs,, ((ss,2) Es) where 3g a diagonal matrix, as Ashman et al. (2022).
iid
Rg Xnew
p(y new 1 new) = E 92 (T) (0) [p(ynew new 1 Xnew)] z
+ TT now I Nnww
This allows to forgo Monte Carlo sampling to evaluate this expectation.
Remark D. Since neither GVI, nor FEDGVV targets the Bayesian posterior under different divergences or loss functions in comparison vanilla VI, we cannot truly speak of this expectation approximating the Bayesian posterior predictive distribution, however since our aim is find distribution that is more valuable decision maker, using FEDG VI posterior should allow make more informed predictions depending on what the DM wants model. This be better uncertainty quantification through changing the divergence, and/or better prediction accuracy through changing the loss.
D.2.1. FURTHER EXPERIMENTS
In Figure 9 we compare the predictive performance of FEDGVI with two clients against that of GVI with only one client.
0.77
0.76
Classification Accurucy
0.75
0.74
0.73
0.72
0.71
PVI
VI
0.70
FaiCVI.
AR
GVL.
(3))
FalGVI, (00.5
0.69
10
Server Iterations
Figure 9: Comparing Logistic Regression with FEDGVI where the data set split across clients, to GVI where the entire data set is available.
D.3. Bayesian Neural Networks
The model architecture is a fully connected multi-layer perceptron with RELU activation.
D. MNIST (LECUn ET AL., 1998) DETAILS AND ADDITIONAL EXPERIMENTS
For the hyperparameters of the competing methods in the BNNs, follow Hasan et al. (2024) in using SGD with momentum with learning rate of 0.1 for FEDAVG, and 3-PREDB AYES, and 0.01 for FEDPA. The architecture for these is 2 hidden layer fully connected neural network, where each hidden layer has 100 neurons.
For FEDGVI and PVI, we follow the set up of (Ashman et al., 2022) in using the ADAM optimiser (Kingma & Ba, 2015) with learning rate 0.0005, leaving all other parameters the default values in PyTorch. Here we use a fully connected NN with hidden layer of 200 neurons.
38

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
The contamination maps all contaminated data points of one class to single other class. In both cases, carried mini-batch optimisation.
Since we use different architectures for the BNN experiments for the MNIST data set in Table 1, we additionally report results for BNNs when we use the same Neural Network architecture and still retain superior performance of FEDGVI, see Table 3. We notice that the choosing the implementation with the two hidden layer NN for the competing methods performs better or on an equivalent level (within one standard deviation) of each other, while FEDGVI performs better on the single layer NN.
When examining their convergence behaviour under the different architectures, further notice that the competing methods perform worse than FEDGVI VI, and FEDAVG and FEDPA exhibit no stability in their accuracy in the contaminated setting, see Figures 10 and 11. This phenomenon occurs even in the uncontaminated case as reported in Al-Shedivat et al. (2021), where we have chosen the optimiser and learning rates as suggested in their paper, and hence we conjecture that contamination further exacerbates this.
Table 3:
Classification accuracy (highest in bold) uncontaminated test data after training on 10% contaminated MNIST data. Here, we compare the results with fully connected Neural Network with hidden layer of 200 Neurons to the results of fully connected Neural Network with 2 hidden layers of 100 Neurons each. We report the best performance across all server iterations.
I HIDDEN LAYER
2 HIDDEN LAYERS
MODEL
10 CLIENTS
3 CLIENTS
10 CLIENTS
3 CLIENTS
FEDAVG
94.79:0.44
91.76100.08
96.64 0.07
96.34 + 0.20
FEDPA
94.53=0.15
95.7410.08
94.25+ 0.39
95.31 0.35
3- PREDDAYES
94.9620.06
96.6710.07
94.901
96.731 0.08
PVI
95.56+0.18
96.68= 0.07
95.68+0.10
97.31+0.08
FEDGVI DAR
96.36+
97.13
95.78+0.17
97.24+0.05
FEDGVI LGCE
97.06 0.03
98.04
96.57-0.04
97.74-0.11
FEDGVI DAR LACE
97.50 0.07
98.13 0.08
96.7710.10
97.7910.11
(1 CLIENT)
(96. (96.96 0.17)
(90.87+0.50)
GVI CLIENT)
(98.13+t 0.07)
(97.56=0.005)
Table Classification accuracy (highest in bold for each learning rate) on uncontaminated test data after training on 10% contaminated MNIST data split across 3 clients. Here, compare the results of different initialisations of FEDGVI with the Alpha-Rényí divergence and generalised cross entropy loss achieved when optimising the posteriors with different learning rates of ADAM. We report the best performance after all server iterations.
LEARNING RATE 1)
MODEL
1e - 2
5e 3
3
1e 1e-3 3
5e -44
1e ---4 4
5e - 3
PVI
96.3410.16
96.50+0.18
96.72+0.06
96.76:1000
96.01
95,3910.00
FEDGVI (2.5)
96.84-1.12
96.9110.02
97.16+0..04
97.18
96.5110.19
95.65-10.03
FEDGVI C LOEE
98.22-0.07
98.30110.03
98.1510.01
98.0810.08
97.07-1.006
95.8411.00
FEDGVT D GCE
98.31++0.10
98.24400.07
98.23+00066
98.06+0.09
97.50+0000
96.35+00088
In Table we compare different learning rates of ADAM with different initialisations of FEDGVI showing that we fact underperform with the learning rate selected for the experiments in Figures 6 and 7, and Table for the outperforming methods of FEDGVI.
In Table 5 we further investigate the stability of FEDGVI posteriors when slightly varying the robustness parameter. This shows no significant variations in the accuracy achieved by FEDGVI when slightly perturbing 5 = 0.8.
We also want to highlight that by not carefully selecting the hyperparameters of FEDGVI, as well as the learning rate, and keeping these constant across the BNN experiments, we have shown that you do not require extensive knowledge to adapt existing PVT approaches FEDGVI and outperform. For instance, FEDGVI performs even better for the robust losses at
39

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
Table 5: We fix a 2.5 in the Alpha-Rényi divergence, and vary ó, of the generalised cross entropy loss of Zhang & Sabuncu (2018), around 0.8. We report accuracies on uncontaminated test data after training on 10% contaminated MNIST data split across clients. These accuracies vary very little demonstrating stability the FEDGVI posterior at slight perturbations the loss parameter.
6 OF Ct0)
MODEL
0.75
0.775
0.79
0.8
0.81
0.825
8.85
PEOGVI ((2.5) cid)
98. 08.05::0.02 02
98.14: 09
98. 04:40.00000 09
98.06:20.00 06
98 06:00.000
97. 07.99
97.98:10.03
100
100
PedGVI ((2.5) c(0.8)
FedPA
FodiiVi (22.5) c(o.8) AR GGR
FedPA
Tedovi (223) CXL.
FadAvg
FedGivi ((2.6) ENLL
FodAvg P P-PreBBayes
% Eroo (10 Clients)
B-Prrellayes
% Eroor (3 Clients)
(0.8) PeaGHT KL, ECCE
30
CCE
WIL,, CNLL
PVI WILL, ANLL
10
Server lterations
Server Iterations
Figure 10: Accuracy Fully Connected Neural Networks with 1 Hidden Layer. We demonstrate convergence of the different approaches examined in the first multicolumn of Table 3. The models are trained on 10% label-contaminated data, and prediction accuracy is assessed on uncontaminated test data.
100
100
WE
FedGVI PAR GGG
FedPA
FedGiVi (2.6), 2(0.8) AR GCR
FedPA
FedGVI (2255 CXLL
FedAvg S Predilayes
FedGVi (2.5) -NLL
FedAvg
Error (10 Clients)
% Error (3 Clients)
B PredBayes
(0.8) PVI LYLL
30
FaRGVT KL, CCCS
PVI WL. ENLL
10
10
15
20
25
10
20
25
Server Itcrations
Server Iterations
Figure 11: Accuracy on Fully Connected Neural Networks with 2 Hidden Layers. We demonstrate convergence of the different approaches examined in the second multicolumn Table 3. The models are trained on 10% cobel-comtaminatedd data, and prediction accuracy is assessed uncontaminated test data.
higher learning rate, but we have shown in Table I that it still outperforms even when not carefully selecting learning rate. Furthermore, choosing = 0.6 and a = 2.5 would have performed better when varying only the robustness parameters of FedGVI, as seen in Figure 6.
Lastly, Figure 12 shows that even when the loss is not available in a conjugate, closed form way, that FEDGVI still incurs no significant computational overhead through choosing the Alpha-Rém Rényi divergence or the generalised cross entropy loss.
40

---

Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework
100
Fedivii GGB Ai
FeddVI (0.8) GCE
FedGVi GGE
(0.0) (2.5)
Eiror Clients)
30
FedtiVi (2.5)
AR
10
150
500
Total Computation time (sec) per Client
Figure 12: Wall-clock times for FEDGVI iterations per client. We plot the classification error against the computation time taken per client during each server iteration, where train 5 Clients 10% contaminated MNIST data. If we do state the loss divergence in the legend, it is LNLL and Dx respectively. Here, FEDGVI outperforms PVI in terms of accuracy while having similar runtimes.
D.3.2. FASHIONMNIST (XIAO ET AL.. 2017) DETAILS
We vary the amount of contamination from 0.0, 0.1, 0.2, 0.4, where the contamination is random and assigns each contaminated data point a different class uniformly at random. The model architecture, prior, learning rate, and optimiser remain unchanged and are as before.
FEDGVI uses the Alpha-Rényi divergence with an alpha value of 2.5 for all, and the robust generalised cross entropy loss, where 5 0.0 indicates the negative log likelihood. Table 6 specifies Table 2 to higher precision but the results identical.
Table 6: Classification accuracy (highest in bold) on uncontaminated test data after training on different amounts of contaminated FASHIONMNIST data. Each Method has data split homogeneously across 3 Clients. report the best performance during all server iterations for each method.
CONTAMINATION
MODEL
0%
10%
20%
40%
FEDAVG
85.72-1.552
78.9911.90
71.16.11.33
48.97-6.51
FEDPA
88.0810.30
87.3610.11
86.5410.16
85.36
/ PREDBAYES
87.58=0.13
87.20=0.12
86.82+0.07
85.77=0.10
PVI
86.21#0.21
85.1440.13
84.36+0.12 12
82.81=0.05
FEDGVI 0.0
87.12=0.12
86.23:0.15
85.56#0.11
83.78=0.09
FEDGVI 0.4
88.73-10.21
88.60+10099
87.0110.37
78.1440.39
FEDGVI 0.5
89.02-0.188
88.5740.16
88.39:1.21
85.06-00.67
FEDGVI 0.8
88.59-00.33
88.4440.07
87.95+0.04
87.21
FEDGVI 8 = 1.0
88.0910.00
87.8310.14
87.5410.15
85.9740.27


Reviews:
[{"review_id": "IECB6FT5Vx", "reviewer": "Reviewer_KG8f", "summary": "This work presents FedGVI, an extension of partitioned variational inference (PVI) to generalised variational inference (GVI). The core benefit of GVI is that it permits robustness to model misspecification. The authors demonstrate a number of advantageous properties of FedGVI, most notably theoretical results that show it convergeses to GVI approximate posteriors, and is thus robust to model misspecification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and positive review that highlights the clarity of our writing, the theoretical results on FedGVI, and its provable robustness. By addressing below the weaknesses you mentioned, we notably improve the clarity of the paper and its ease of understanding for the reader.\n\n**Weakness 1** $\\beta$ parameter in Eq. 2 undefined:\n\nThank you for spotting this typo, it has now been corrected, and we will add the following clarification:\n\n*Here, $\\beta\\in\\mathbb{R}_{>0}$ is a learning rate parameter that determines how much weight we place on the observed data, similar to power posteriors in VI (Grünwald, 2012; Kallioinen et al., 2024).*\n\nThe $\\beta$ parameter comes from the power/cold/tempered posteriors of e.g. Grünwald (2012), where the likelihood in Bayesian posteriors is raised to some power of $\\beta>0$. This was originally done to add some robustness to the posterior, down-weighting observations if $\\beta< 1$ and up weighting these for $\\beta > 1$. Through a known result (Knoblauch et al., 2022), which we highlight in Lemma B.1 in the Appendix, this is equivalent to having a weighted Kullback-Leibler divergence, $\\frac1\\beta\\mathrm{KL}$. This also allows us to define if we want to trust the prior more $\\beta<1$ or less $\\beta>1$, since up weighting the data means down weighting the prior and vice versa.\n\nWe will add this discussion to Appendix B.1 to give further intuition behind GBI.\n\n**Weakness 2** Intuition behind Definition 4.10 and Conditions 2 and 3:\n\nThe three conditions combined allow us to say whether the client posterior (or simply the posterior in a global, 1 Client, GBI setting) derived from such a robust loss is provably robust to Huber contamination.\n\nFrom *Condition 1* we are able to bound an infinitesimal change in the loss with the contaminating data point $z$ by some auxiliary function $\\gamma$, possibly infinite for some values of $\\theta$.\n\n*Condition 2* states that the product function, $\\gamma(\\theta)\\pi(\\theta)$ has finite uniform norm. \nThis ensures that this product under the worst case contamination and the worst parameter $\\theta$, is finite and hence it cannot be made arbitrarily bad, which does not hold for the negative log likelihood in general. Alternatively, the prior decays to zero faster than the auxiliary function can diverge to infinity in $\\theta$.\n\n*Condition 3* further says that $\\gamma(\\theta)\\pi(\\theta)$ is finitely integrable, i.e. that this is in $L^1(\\Theta)$. This, in effect, bounds the normalising constant of the contaminated posterior and will ensure that this is finite.\n\nTaking all these conditions together tells us that the product function $\\pi(\\theta)\\gamma(\\theta)$ is in $L^1(\\Theta)$ and that it is finite everywhere; two conditions that are mutually independent, i.e. one doesn't follow form the other.\n\nThese conditions characterise the notion of robustness we use for Theorem 4.11, a derivation of this notion is shown in Appendix B.7, by considering the worst choice for the contamination $z$ and the parameter $\\theta$ with respect to small perturbations of the resulting posterior through $\\epsilon$. \nThe influence of the contamination $z$ and parameter $\\theta$ on the posterior is defined as $\\frac d{d\\epsilon}q_m^{(t)}(\\theta;\\mathbb{P}_{n_m,\\epsilon, z})|_0$ (evaluated at $\\epsilon=0$), which is bounded through the conditions. This then implies that the local posterior is *globally bias robust*, i.e. robust to Huber contamination of all $z$ for all parameters $\\theta\\in\\Theta$. \n\n\nThank you for pointing this out, we will provide the following clarification between Definition 4.10 and Theorem 4.11.\n\n*These conditions ensure that the influence of arbitrary contamination on the local posterior is not arbitrarily bad. \nIn particular the auxiliary function $\\gamma_m^{(t)}$ ensures that the influence of an adversarial data point $z$ on the posterior over infinitesimal contaminations,\n$$\\frac d{d\\epsilon} q_m^{(t)}(\\theta;\\mathbb{P}_{n_m,\\epsilon, z}) {\\Big|}_0$$\nevaluated at $\\epsilon=0$, are finite over all $\\theta$ and $z$. Condition 2 ensures the loss increases slowly enough for the local posterior to concentrate around the data, and condition 3 ensures the resulting posterior will be normalisable.*\n\n**References not in Paper**\n\nKallioinen, N., Paananen, T., Bürkner, P.-C., and Vehtari, A. Detecting and diagnosing prior and likelihood sensitivity with power-scaling. Statistics and Computing, 34(1):57, 2024.", "claims_and_evidence": "Yes, both theoretical and empirical.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Related to the field of federated probabilistic learning, which has wide spread applications.", "essential_references_not_discussed": "No,", "other_strengths_and_weaknesses": "Strengths\n\n* The paper is very well written, not only in presenting FedGVI but also in the presentation of PVI (and extending the current understanding of how it works). \n* FedGVI itself is easy to implement (a relatively straightforard extension of PVI).\n* The theoretical results are supported by a number of experimantal results demonstrating the superiority of FedGVI relative to baseline methods (including PVI) when data contains outliers.\n\nWeaknesses\n* In equation 2 $\\beta$ is undefined. Also---and I could be mistaken---should the normalisation constant include a $\\beta$ in the integral?\n* Whilst I understand the intuition behind 1. in Definition 4.10, I struggle more with the intuition behind points 2. and 3.. If there exists relatively compact intuition, perhaps the paper would benefit from its inclusion here.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTvMGCj6JH", "reviewer": "Reviewer_gKSi", "summary": "The paper presents a new framework, Federated Generalised Variational Inference (FEDGVI), for robust probabilistic federated learning. The authors argue that standard Bayesian and frequentist federated learning methods are vulnerable to model misspecification (e.g., contaminated data, incorrect priors, or mismatched likelihood models). Building on the theory of Generalised Variational Inference (GVI), they propose a method that can (1) systematically handle likelihood and prior misspecifications, (2) generalize existing frameworks like Partitioned Variational Inference (PVI) and FedAvg, and (3) provide stronger robustness guarantees with calibrated uncertainty estimates. Key contributions include:\n\n- A unifying algorithmic framework for GVI in the federated learning (FL) setting, with theoretical convergence guarantees.\n- Proofs that FEDGVI is robust to outliers and model misspecification under suitable choices of loss functions and divergences.\n- Empirical evaluations on both synthetic tasks (e.g., the 1D “clutter” problem and logistic regression) and real data (CoverType, MNIST, FashionMNIST), demonstrating improved robustness and accuracy compared to baselines such as PVI, DSGLD, and FedAvg.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their in-depth, constructive, and positive review.\n\n**Question 1, Weakness 1, and Experiments**\n\n> ...method’s sensitivity to the choice of robust hyperparameters [...] how stable...\n\n> How sensitive is FEDGVI to [...] hyperparameters [...]? Would small changes degrade performance significantly?\n\nWe presented results on FashionMNIST while varying $\\delta$ with different contaminations in Fig. 2. We now extend these to further contamination levels (0, 0.1, 0.2, 0.4) and $\\delta$ (0, 0.4, 0.5, 0.8, 1),\nas well as FedAvg, FedPA, and $\\beta$-PredBayes, to demonstrate hyperparameter sensitivity:\n\nhttps://anonymous.4open.science/r/Resources-3CEF/FashionMNIST.png\n\nWe also now offer an ablation study on MNIST for hyperparameter selection: https://anonymous.4open.science/r/Resources-3CEF/ablation_study.png (see also response to reviewer *4k22*).\n\nTo show the stability of FedGVI under small perturbations of $\\delta$, we fix $\\alpha=2.5$, vary $\\delta$ around $0.8$, and report accuracies on uncontaminated test data after training on 10% contaminated MNIST data split across 5 clients:\n\nhttps://anonymous.4open.science/r/Resources-3CEF/Perturbations.png\n\n**Question 2, Weakness 2** Computational Complexity/Overhead\n\n**W2**\n> Communication cost or runtime overhead is not directly benchmarked...\n\n**Q2** \n> do [...] loss functions [...] incur any significant computational overhead [...] updates?\n\nAs the number of clients increases, the comp. time at each client is reduced due to less data per client. By Proposition 4.9 we can choose a robust loss at each client that enables conjugate client updates whereas the negative log likelihood might not have closed form. These include intractable likelihood models (Matsubara et al., 2022) where the normalising constant is intractable but we can tractably solve.\n\nThe comp. complexity of the divergence is solely dependent on the choice of variational family, prior, and divergence without dependence on number of clients or data per client. The divergence may be more comp. expensive in the case where no closed form solutions exist, e.g. for the Total Variation Distance. However, for exponential family distributions, many robust divergences with closed form solutions exist (Pardo Llorente, 2006; Knoblauch et al., 2022).\n\nThe KL and A-R divergences between Gaussians (Sec. 5.5) are available in closed form, scaling only in the number of parameters. The A-R can be computed in $\\mathcal{O}(1)$ times the comp. complexity of the KL, driven by the determinant of the covariances; assuming $\\Theta\\subset\\mathbb{R}^d$, and using a Cholesky decomposition this naively takes $\\mathcal{O}(d^3)$ for both; assume these are (block)-diagonal, makes this cheaper.\nThe discrete losses in the classification setting are non-differentiable. We cannot apply the conjugate score matching loss, and require sampling from the approximation. We can transform the NLL in $\\mathcal{O}(1)$ to the GCE, thereby, as the likelihood is used for the GCE and it's log for the NLL, these are of the same order of magnitude.  \n\nWe will add in Section 5.5:\n\n*FedGVI incurs no additional computational complexity when compared to PVI. This is due to the KL and Alpha-Rényi divergences having closed form solutions between Multivariate Gaussians with complexity of $\\mathcal{O}(1)$ in each other, and because we require $\\mathcal{O}(1)$ additional, constant operations to get the GCE from the NLL.*\n\nTo empirically validate this, we now report average wall-clock times at each client for FedGVI and PVI, see https://anonymous.4open.science/r/Resources-3CEF/runtime.png.\n\n**Comments**\n\n> Heuristics for selecting robustness parameters\n\nThese include Knoblauch et al. (2018) and Yonekura &  Sugasawa (2023) for the Density Power loss, with the latter a theoretically principled Sequential Monte Carlo sampler for selecting $\\beta$, or Altamirano et al. (2024) for weighted score matching, through cross validation. \n\nWe propose to add this in Sections 3.2 and 3.3:\n\n3.2: *We can use a Sequential Monte Carlo sampler to estimate the $\\beta$ or $\\gamma$ hyperparameters in $\\mathcal{L}_B$ and $\\mathcal{L}_G$ (Yonekura &  Sugasawa, 2023) or use cross validation to select optimal parameters (Altamirano et al., 2024).*\n\n3.3: *Similarly to the losses, we can perform cross validation to select the $\\alpha$ parameter, however as demonstrated in the ablation study (Figure TBD) FedGVI performs favourably under a range of $\\alpha$ values.*\n\n> Robust Aggregator strategies at the server\n\nThis is an area we are actively working on. We are exploring ways in which the summation in Eq. 6 can be replaced by a robust aggregator, such as Nearest Neighbour Mixing (Allouah et al., 2024) or indeed coordinate-wise median and trimmed mean, to achieve Byzantine robustness. We will discuss this in Sec. 6.\n\n**References**\n\nYonekura, S., Sugasawa, S. Adaptation of the tuning parameter in general Bayesian inference with robust divergence. Stat Comput 33, 39, 2023.", "claims_and_evidence": "1. FEDGVI is robust to both likelihood and prior misspecification. The authors develop theoretical results (Section 4) showing that if each client employs a robust (bounded) loss function, the global posterior remains stable even when some portion of data is contaminated (Theorem 4.11). \n2. FEDGVI recovers existing federated learning methods (e.g., PVI, FedAvg) as special cases: In Section 4.1, the authors show that by choosing particular divergences and losses, the method reduces to either standard partitioned variational inference (for Bayesian FL) or FedAvg (for frequentist FL).\n3. The approach is scalable to real-world tasks and yields superior predictive performance compared to baselines: Empirical studies on real datasets (Cover Type, MNIST, FashionMNIST) demonstrate that FEDGVI obtains higher classification accuracy under mismatched or noisy training. \n\nOverall, the manuscript provides both proofs of conceptual claims and corresponding empirical corroboration.", "methods_and_evaluation_criteria": "The proposed method relies on Generalised Variational Inference to handle potential model misspecification. Clients use robust local objectives (like Density–Power or Generalised Cross-Entropy losses) while the server employs a chosen divergence-based penalty (often KL or an $\\alpha$-Rényi divergence) on the global posterior. Convergence is examined in terms of fixed points—where the global posterior stops changing under repeated local-global updates.\n\nThe proposed evaluation criteria seems solid.", "theoretical_claims": "The authors extend existing PVI theory by showing that if a robust generalised Bayesian update is done at each client and a KL-based GVI update at the server, the global solution is itself robust. Key points:\n1. Fixed point analysis (Proposition 4.4): Demonstrates that, if the algorithm converges, the final server distribution is a stationary minimizer of a global GVI functional.\n2. Equivalence to GBI (Lemma 4.5): With certain parameter choices (e.g., negative log-likelihood as the local “loss,” or a robust alternative + KL at the server), the final solution coincides with the standard or robust GBI posterior.\n3. Cavity distribution necessity (Theorem 4.8): Shows that removing the cavity from the client update would lead to systematically biased or overconfident global updates. This result clarifies why each client must “subtract out” the impact of its own data from the global prior, rather than simply using the existing posterior.\n\nI did not uncover any obvious mistakes in the proofs; they seem consistent with prior results on partitioned variational methods. That said, verifying every detail would require deeper familiarity with some of the advanced robust GVI and measure-theoretic derivations, which I do not fully possess.", "experimental_designs_or_analyses": "The authors systematically vary contamination rates (e.g., flipping labels randomly on MNIST) and track how well each method recovers clean performance. They compare with classical federated baselines (FedAvg) and Bayesian baselines (PVI, DSGLD, DSVGD, etc.) across varied contamination levels. The data partition is typically homogeneous or split randomly, though the paper briefly remarks it is straightforward to allow non-i.i.d. partitions.\n\nThe authors provide clarity that the number of clients (M) or the chosen “damping” parameters can be adapted.\nSome additional exploration of the method’s sensitivity to the choice of robust hyperparameters (e.g., $\\beta$, $\\delta$, $\\gamma$) might help readers see how stable the procedure is under default vs. tuned settings.", "relation_to_broader_scientific_literature": "This work builds on the recognized problem of model misspecification in Bayesian methods, referencing established works like Bissiri et al. (2016) for generalised Bayes, and the robust M-estimation approaches from Ghosh & Basu (2016) or Knoblauch et al. (2022). Overall, the framework is well-situated within existing lines of research but has a distinct novelty in bridging the gap between robust GVI and federated Bayesian methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The framework is conceptually simple to implement: each client modifies its update with a robust local objective and the server does a GVI-based merge.\n2. The theoretical story on why robust GVI helps in the federated setting is coherent and well-linked with standard references.\n3. Strong empirical demonstration that the method truly mitigates outlier influence.\n\n**Weaknesses**\n\n1. Some of the hyperparameter tuning for the robust losses (like $\\beta$ or $\\gamma$ for L$\\beta$/GCE) is not deeply explored in the main text; it might require extra user expertise.\n2. Communication cost or runtime overhead is not directly benchmarked: presumably the overhead is comparable to standard PVI or FedAvg, but actual run-time comparisons might be valuable to demonstrate practical feasibility.", "other_comments_or_suggestions": "The authors might add small clarifications on the recommended heuristics for selecting the “robustness parameters” (β, γ, etc.) in real-world use, or mention a rule of thumb.\n\nIt might be beneficial to discuss whether robust aggregator strategies at the server (like coordinate-wise trimming or medians) could be combined with robust GVI for even stronger resilience against malicious clients.", "questions_for_authors": "1. How sensitive is FEDGVI to the choice of the robust loss hyperparameters (e.g., $\\delta$ in the generalised cross-entropy or $\\beta$ in the density–power loss)? Would small changes degrade performance significantly?\n\n2. In practical setups with many clients ($M>>1$), do the introduced robust loss functions or divergences incur any significant computational overhead relative to standard negative log-likelihood updates?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSnMv9IFi6", "reviewer": "Reviewer_RwrC", "summary": "The paper introduces *FEDGVI* (Federated Generalized Variational Inference), a probabilistic federated learning framework designed to be robust against both prior and likelihood misspecification. FEDGVI generalizes Partitioned Variational Inference by integrating robust and conjugate updates, thus reducing computational complexity at client devices. It provides theoretical results demonstrating fixed-point convergence, optimality of the cavity distribution, and robustness to model specification. Empirical evaluations on synthetic and real-world datasets highlight improved robustness and predictive accuracy compared to existing FL methods.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our theoretical results and especially for taking the time to review our proofs.\n\n**Claims & Evidence**\n\n> the examples provided are [...] specifically chosen to highlight FEDGVI's strengths over PVI\n\nFedGVI with the negative log likelihood and KL divergence is equivalent to PVI, i.e. FedGVI(NLL,KLD)=PVI. In the idealised setting where the DGP $p_0=p_\\theta$ is known, PVI with a correctly specified likelihood is preferable to FedGVI(RobustLoss, KLD), i.e. using a robust loss will degrade performance; see Knoblauch et al. (2022), Jewson et al. (2018), Bissiri et al. (2016). Arguably, we rarely have accurate likelihood functions, since these are either too complex to work with or practitioners opt for standard likelihoods without integrating domain expertise.\n\nWe will amend Sec. 3.2 and add:\n\n*It is important to highlight that GVI and FedGVI may underperform when using robust losses in the case of correct likelihood specification.*\n\n**Relation to Literature**\n\nWhen viewing GVI/GBI as an optimisation problem on the space of probability distributions $\\mathcal{P}(\\Theta)$, Bayesian inference, VI, hierarchical Bayes/VI, all target a single element of this space. These methods either target the standard Bayesian posterior explicitly, or the posterior within some variational family with closest Kullback-Leibler distance to the Bayesian one. \nThrough GBI and GVI we are able to target different elements of a  subspace of $\\mathcal{P}(\\Theta)$, then simply a single point; in that regard, these approaches do 'generalise' Bayes. In the paper, 'generalised' is inherited from GVI and GBI. We should note that in the FedGVI setting, GBI and GVI allow us to generalise PVI or FedAvg to a broader subspace of possible posteriors. We will clarify this naming distinction in the main paper. \n\nWe have made a figure to highlight this, see https://anonymous.4open.science/r/Resources-3CEF/FedGVI.png, which we will include in the paper.\n\n**Question 2, Missed References, Hierarchical Bayes**\n\nFollowing your suggestion we will add and discuss the proposed references in relation to FedGVI and GVI, giving an additional discussion on personalised and hierarchical Bayesian FL in Sec 1.1. Furthermore, we propose to add the following to Sec. 6:\n\n*An interesting future direction is to extend FedGVI within personalised FL settings (Kotelevskii et al., 2020) and hierarchical Bayesian FL through latent variables (Kim & Hospedales, 2023), as well as the use of a structured posterior approximation (Hassan et al., 2023; 2024), in order to incorporate client level variations. Incorporating the hierarchical model structures and additional inductive biases from such settings, while maintaining conjugacy and favourable computational complexity, remain as open challenges.*\n\n**Weakness 1, Comments, Experimental Design**\n\nWe have now carried out further empirical and ablation studies, https://anonymous.4open.science/r/Resources-3CEF, see also our responses to reviewers *4k22* and *gKSi*, which we will include in the revised paper. Nevertheless, as we have shown even the 'simplistic' MNIST setting already provides challenges to traditional Bayesian FL making it worth studying. \n\nWe also agree with the reviewer that extending FedGVI to more complex data sets and scenarios, such as investigating different aspects of FL for instance privacy constraints, the cross-device setting, and massively distributed scenarios, present intriguing future directions for FedGVI. We thank the reviewer for pointing this out and will be included in Sec. 6. \n\nFor 'computational constraints' see our next answer.\n\n**Weakness 2, Question 1**\n\n> unique methodological contributions that the use of [GBI] could specifically enable in [FL]\n\nTo the best of our knowledge, we have proposed the first theoretically grounded probabilistic FL framework that deals with model misspecification. Our framework brings the following methodological advances:\n\nFedGVI can be more computationally efficient than PVI through the use of GBI, as we have shown in Prop. 4.9. The conjugacy with exponential family likelihoods through the robust score matching loss enables faster computation than PVI since it does not require sampling at the local clients. See also our response on computational complexity to Reviewer *gKSi*. \n\nThrough the guaranteed posterior robustness by Theorem 4.11, our prediction accuracy is less affected by outliers which would be detrimental for instance in medical settings; see e.g. Jonker et al. (2024) for FL in medicine.\n\nThe server divergence and the potential for robust aggregation at the servers (Sec. 6) without fundamentally changing our approach allows for unique optimisation on a global level not provided by Bayesian FL approaches.\n\nWe thank the reviewer for encouraging us to flesh these points out and we will include these in the revised paper.", "claims_and_evidence": "The claims made in the paper regarding robustness and improved predictive performance are generally supported by the theoretical analysis and experimental results. However, the examples provided are standard in statistical and ML literature and are specifically chosen to highlight FEDGVI's strengths over PVI, which clearly struggles in some of these settings. The claims related to applicability in more general settings may require further validation.\n\nIndeed, federated hierarchical Bayesian models (which I provide references to below) would likely outperform FEDGVI in many of these settings. While hierarchical models may be out of scope for this work, methods designed for \"personalised\" federated settings are addressing similar issues. Such methods would be more convincing baselines, and they should at least be mentioned and referenced in the related work. Future work combining generalized Bayesian methods and hierarchical models in federated learning would be very interesting.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria, including theoretical robustness under various divergence-based losses and experimental evaluation on common benchmark datasets, are reasonable.", "theoretical_claims": "The theoretical results, such as the proofs of robustness and convergence of FEDGVI (e.g., Theorems 4.8 and 4.11, Lemma 4.5, Proposition 4.9), are clearly stated, and are **the biggest strength of the paper**. I reviewed proofs provided in the supplementary material, and they appear sound and rigorous. No issues were identified.", "experimental_designs_or_analyses": "Experimental designs involving the clutter problem, logistic regression, MNIST, and FASHIONMNIST datasets are sound overall, though somewhat limited in terms of diversity and complexity. Experiments were explicitly crafted to demonstrate FEDGVI’s advantages, which may slightly exaggerate the practical benefits.", "relation_to_broader_scientific_literature": "FEDGVI effectively integrates robust Bayesian inference methods (GBI and GVI) with federated learning. It builds upon prior work in Bayesian FL and robust inference, clearly identifying its position and contributions within the broader context. However, the common claim in generalized Bayesian methods literature that each new approach inherently \"generalizes\" existing Bayesian approaches is somewhat overstated. Generalization in this context is an expected consequence of employing generalized Bayesian inference, rather than a novel contribution specific to each individual application of Generalized Bayes.", "essential_references_not_discussed": "As mentioned previously I think the following references should at least be mentioned as hierarchical/personalized approaches to Bayesian federated inference and offer an alternative or additional approach to dealing with some of the issues that might arise in the numerical examples provided in the paper: \n- Kotelevskii, Nikita, et al. \"Fedpop: A bayesian approach for personalised federated learning.\" Advances in Neural Information Processing Systems 35 (2022): 8687-8701 (was cited, but just in a citation dump of various FL methods)\n- Kim, Minyoung, and Timothy Hospedales. \"Fedhb: Hierarchical bayesian federated learning.\" arXiv preprint arXiv:2305.04979 (2023).\n- Hassan, Conor, Robert Salomone, and Kerrie Mengersen. \"Federated variational inference methods for structured latent variable models.\" arXiv preprint arXiv:2302.03314 (2023).\n- Zhang, Xu, et al. \"Personalized federated learning via variational bayesian inference.\" International Conference on Machine Learning. PMLR, 2022.", "other_strengths_and_weaknesses": "**Strengths** of the paper include the theoretical analysis, clear positioning within existing literature, and the presentation of explicit theoretical gurantees of robustness and convergence. Additionally, the paper offers a systematic integration of GBI with federated learning, which can be valuable for further theoretical explorations. **Weaknesses** of the paper are that the experimental setups are simplistic and somewhat contrived, limiting the empirical evidence of generalization to more challenging, realistic scenarios. The paper's methodology is straightforward (applying generalized Bayesian inference in FL context), potentially limiting originality and novelty in methodological contribution.", "other_comments_or_suggestions": "The paper would benefit from additional complex real-world experiments beyond simple label contamination scenarios, such as realistic non-IID client data distributions, large-scale deployments, or federated learning tasks with genuine privacy or computational constraints.", "questions_for_authors": "1. Do the authors see any unique methodological contributions that the use of generalized Bayesian inference could specifically enable in federated learning settings?\n\n2. What future directions do the authors envision for further integrating generalized Bayesian methods with hierarchical or personalized federated learning approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OlwGGA4tqH", "reviewer": "Reviewer_4k22", "summary": "The paper introduces FEDGVI, a novel probabilistic framework for federated learning that is designed for both prior and likelihood misspecification. ## update after rebuttal I will retain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of the clarity of our paper and the theoretical results, as well as their important suggestion for an ablation study that we have now conducted and which will significantly improve the empirical study of this work. We will include these in the revised version of the main paper. \n\n**Hyperparameters of FedGVI**\n\nIn particular, we have carried out an ablation study on the selection of the hyperparameters of FedGVI in the training of a Bayesian Neural Network on 10% contaminated MNIST data, split across 5 clients, extending the results discussed in Section 5.5. We vary the $\\delta$ parameter of the Generalised Cross Entropy, and the $\\alpha$ parameter of the Alpha-Rényi divergence in the client optimisation step (Eq. 4), and record the predictive accuracies on the uncontaminated test data for the FedGVI posteriors found with the respective hyperparameters:\n\n| | $\\delta=0.0$ | $\\delta=0.2$ | $\\delta=0.4$ | $\\delta=0.6$ | $\\delta=0.8$ | $\\delta=1.0$ |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n| $\\alpha=0.0$ | 93.84 | 96.86 | 98.08 | 97.74 | 97.45 | 97.17 |\n| $\\alpha=0.5$ | 96.32 | 96.91 | 97.97 | 97.99 | 97.76 | 97.52 |\n| $\\alpha=1.0$ | 96.54 | 96.98 | 97.92 | 98.05 | 97.92 | 97.72 |\n| $\\alpha=1.5$ | 96.68 | 96.07 | 97.67 | 98.04 | 97.96 | 97.88 |\n| $\\alpha=2.5$ | 96.84 | 94.79 | 97.19 | 98.17 | 98.09 | 97.87 |\n| $\\alpha=5.0$ | 97.37 | 92.96 | 95.81 | 97.95 | 98.11 | 98.05 |\n\nwhere we note that $\\delta=0$ implies the negative log likelihood since the loss converges to it as $\\delta$ tends down to zero (Zhang & Sabuncu, 2018). For the Alpha-Rényi divergence, $\\alpha=1$ implies the Kullback-Leibler divergence, and $\\alpha=0$ implies the reverse Kullback-Leibler divergence, i.e. $D_{AR}^{(0)}(q:q^{\\backslash m})=D_{RKL}(q:q^{\\backslash m}):=D_{KL}(q^{\\backslash m}:q)$ (Amari, 2016). This means that for $\\alpha=1.0$ and $\\delta=0.0$, we recover PVI.\n\nWe propose to add these results on hyperparameter selection in FedGVI as an annotated heat map to Section 5.5 in the revised version of the paper; the figure can be viewed here: https://anonymous.4open.science/r/Resources-3CEF/ablation_study.png. \n\nIn the figure, we present the maximum result achieved across all server iterations and plot the percentage errors on uncontaminated test data for the different hyperparameter combinations. As evident from the table most of the combinations of $\\alpha$ and $\\delta$ explored show stability of the posterior regarding classification accuracy with different hyperparameters, however some care should be placed in selecting these  since the wrong choice could significantly degrade model performance, e.g. $\\alpha=5.0$ and $\\delta=0.2$, where we do not sufficiently filter out outliers but place increased weight on the cavity distribution through the high alpha. Nevertheless, the majority of settings outperform PVI, especially for $\\delta=0.6$.\n\n**Learning Rate Selection**\n\nFurthermore, we also present additional results on learning rate selection for the ADAM optimiser (Kingma & Ba, 2015) in the 3 Client FedGVI and PVI regimes presented in Tab. 1 of the paper. So far, we have fixed the learning rate to be 5e-4 in the BNN experiments of the paper, which we now vary while keeping the divergence and loss parameters fixed for the respective method. We highlight the best result for each learning rate in bold.\n\n| Learning Rate $\\eta$ | 1e-2 | 5e-3 | 1e-3 | 5e-4 | 1e-4 | 5e-5 |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n| PVI | 96.34$\\pm$0.16 | 96.50$\\pm$0.18 | 96.72$\\pm$0.06 | 96.76$\\pm$0.07 | 96.01$\\pm$0.05 | 95.39$\\pm$0.06  |\n| FedGVI $D_{AR}^{(2.5)}$ | 96.84$\\pm$0.12 | 96.91$\\pm$0.02 | 97.16$\\pm$0.04 | 97.18$\\pm$0.03 | 96.51$\\pm$0.19 | 95.65$\\pm$0.03  |\n| FedGVI $\\mathcal{L}_{GCE}^{(0.8)}$ | 98.22$\\pm$0.07 | **98.30$\\pm$0.03** | 98.15$\\pm$0.01 | **98.08$\\pm$0.08** | 97.07$\\pm$0.06 | 95.84$\\pm$0.04 |\n| FedGVI $D_{AR}^{(2.5)}$ + $\\mathcal{L}_{GCE}^{(0.8)}$ | **98.31$\\pm$0.10** | 98.24$\\pm$0.07 | **98.23$\\pm$0.06** | 98.06$\\pm$0.09 | **97.50$\\pm$0.01** | **96.35$\\pm$0.08** |\n\nHere, FedGVI with a robust loss outperforms PVI in every scenario.\n\nWe also want to point out that by not carefully selecting the hyperparameters of FedGVI, as well as the learning rate, and keeping these constant across the BNN experiments, we have shown that you don't require extensive knowledge to adapt existing PVI approaches to FedGVI and outperform. For instance, FedGVI performs even better for the robust losses with a higher learning rate, but we have shown in Tab. 1 in the paper that it outperforms even when not carefully selecting a learning rate.\nFurthermore, choosing $\\delta=0.6$ and $\\alpha=2.5$ would have performed better when varying only the robustness parameters of FedGVI.\n\n**References not in paper**\n\nKingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.", "claims_and_evidence": "Good.", "methods_and_evaluation_criteria": "Fair.", "theoretical_claims": "Good.", "experimental_designs_or_analyses": "Fair.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n1. The writing of the paper is clear.\n2. The author propose solid theoretical analysis and empirical validation.\n\nWeaknesses:\n1. There is no ablation study regarding the hyperparameter selection.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission considers generalised variational inference in the federated settings, extending the partitioned variational inference framework of Ashman et al (2022) to handle potential model misspecification. The theoretical contributions are rigorous and validated by a suite of experiments. All reviewers' questions are comprehensively addressed. I therefore recommend acceptance.
Decision: Accept (spotlight poster)