Federated ADMM from Bayesian Duality

## Abstract

Abstract ADMM is a popular method for federated deep learning which originated in the 1970s and, even though many new variants of it have been proposed since then, its core algorithmic structure has remained unchanged.
Here, we take a major departure from the old structure and present a fundamentally new way to derive and extend federated ADMM. We propose to use a structure called Bayesian Duality which exploits a duality of the posterior distributions obtained by solving a variational-Bayesian reformulation of the original problem.
We show that this naturally recovers the original ADMM when isotropic Gaussian posteriors are used, and yields non-trivial extensions for other posterior forms.
For instance, full-covariance Gaussians lead to Newton-like variants of ADMM, while diagonal covariances result in a cheap Adam-like variant. This is especially useful to handle heterogeneity in federated deep learning, giving up to 7% accuracy improvements over recent baselines.
Our work opens a new Bayesian path to improve primal-dual methods. Code is at https://github.com/team-approx-bayes/bayes-admm .

## 1 Introduction

Alternating Direction Method of Multipliers (ADMM) forms the backbone of many federated deep-learning algorithms [1, 64, 24, 59, 40, 65], and all such algorithms continue to use ADMM in more or less the original form proposed in the 1970s [22, 21].
Federated ADMM is derived by using a Lagrangian which constrains the server parameters $\bar{\boldsymbol{\theta}}$ to be equal to the parameter $\boldsymbol{\theta}_{k}$ of the client $k$ by using dual vectors $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$.
The resulting algorithm consists of three steps: (1) local optimization of $\boldsymbol{\theta}_{k}$ at each client, (2) local update of $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, and (3) a global server update of $\bar{\boldsymbol{\theta}}$ using the aggregate $\bar{}\mbox{$\mbox{$\mathbf{v}$}$}$ of all the $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$.
The optimality condition then yields a closed loop as shown in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1), connecting the two dual spaces of the $\boldsymbol{\theta}$ and $\mathbf{v}$ vectors.
This core ADMM structure continues to be used to this day, not only in deep learning but also in many other fields such as image processing, inverse problems, science, economics and finance; see [10, 47] for surveys.
It is clear that ADMM and its core structure are a foundational idea for federated, distributed, and decentralized optimization.

In modern times, we need extensions that can solve the most challenging issues in deep learning, for example, new methods that can deal with heterogeneity and uncertainty arising due to distributed nature of the problem.
This would make AI development cheaper (saving compute and money) while also improving privacy, security, and sustainability.
However, existing ADMM variants do not fundamentally depart from the original ADMM updates [e.g., 16, 9, 61, 5, 58, 57].
In this work, we propose a new Bayesian way to derive and extend federated ADMM.
This allows us to use Bayesian federated learning [3, 34, 25, 62, 37, 46], which use fundamentally different methodologies to ADMM, to directly address heterogeneity and uncertainty in modern federated deep learning.

Here, we present a fundamentally new Bayesian way to derive and extend federated ADMM. We reformulate the original federated problem as a variational-Bayesian (VB) problem where the goal is to learn a posterior distribution $\bar{q}(\boldsymbol{\theta})$ instead of the point estimate $\bar{\boldsymbol{\theta}}$. This lifting gives rise to a new duality structure associated with the solutions of the VB objective, which not only resembles the ADMM structure but naturally extends it
([Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1)). We call this the Bayesian Duality structure, and show that it naturally recovers the original ADMM when isotropic Gaussian posteriors are used in the VB objective. New variants are also automatically obtained by simply choosing different kinds of exponential-family posterior forms. We use multivariate Gaussians to obtain a new Newton-like variant which can exploit posterior covariance to handle heterogeneity. Unlike regular ADMM, the Newton-like variant can find the optimum in a single communication round on quadratic objectives. Such results do not exist in the ADMM literature so far. Using diagonal covariances, we get a cheap Adam-like version
of ADMM, and this improves over existing baselines across a variety of federated deep learning benchmarks.
Our work exploits the power of Bayesian methods to extend ADMM for federated deep learning, opening a new Bayesian path for distributed optimization.

Figure: Figure 1: The left figure shows the core structure of federated ADMM to update the parameters of the server $\bar{\boldsymbol{\theta}}$ and clients $\boldsymbol{\theta}_{k}$ by using the dual vectors $\mbox{$\mbox{$\mathbf{v}$}$}_{k}=\nabla\ell_{k}$ for each local loss $\ell_{k}$, which are aggregated into $\bar{}\mbox{$\mbox{$\mathbf{v}$}$}$. The loop is derived from the optimality condition of the Lagrangian and drawn analogously to the closed circuit by Rockafellar [52, Fig. 2]. The right figure shows our new Bayesian Duality structure derived by using the optimality condition of a lifted VB objective over the space of distributions $q(\boldsymbol{\theta})$. Analogous to the dual vector $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, we now have $\widehat{\boldsymbol{\lambda}}_{k}=\nabla_{\text{$\mbox{$\mbox{$\boldsymbol{ \mu}$}$}$}}\mathbb{E}_{q_{\text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}$}}}[ \ell_{k}]$ which are set to *natural* gradients instead of gradients (precise definitions are in the main text). Our main result is to derive and extend federated ADMM by using the above Bayesian Duality.
Refer to caption: x1.png

## 2 Federated ADMM and Connections to Bayes

Federated learning aims to train a global model with parameters $\bar{\boldsymbol{\theta}}$ at a central server by communicating with $K$ clients without ever gaining access to their local data. The aim is to obtain the optimal $\bar{\boldsymbol{\theta}}^{*}$ that minimizes the sum over each client’s loss, denoted by $\ell_{k}(\boldsymbol{\theta})$, computed over their local data. This problem is solved by introducing local $\boldsymbol{\theta}_{k}$ at each client and by forming a Lagrangian, which uses dual vectors $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ to enforce the
constraint $\bar{\boldsymbol{\theta}}=\boldsymbol{\theta}_{k}$. More formally, the goal is to solve the problem shown below in the left, and federated ADMM uses the Lagrangian shown on the right where the inner-product term $\langle\mbox{$\mbox{$\mathbf{v}$}$}_{k},\boldsymbol{\theta}_{k}-\bar{
\boldsymbol{\theta}}\rangle$ appears due to the equality constraints,

$$ $\bar{\boldsymbol{\theta}}^{*}=\arg\min_{\bar{\boldsymbol{\theta}}}\,\,\sum_{k= 0}^{K}\ell_{k}(\bar{\boldsymbol{\theta}})\,\,\implies\,\,\mathcal{L}(\bar{ \boldsymbol{\theta}},\boldsymbol{\theta}_{1:K},\mbox{$\mbox{$\mathbf{v}$}$}_{1 :K})=\sum_{k=1}^{K}\left[\ell_{k}(\boldsymbol{\theta}_{k})\!+\!\langle\mbox{$ \mbox{$\mathbf{v}$}$}_{k},(\boldsymbol{\theta}_{k}\!-\!\bar{\boldsymbol{\theta }})\rangle\right]+\ell_{0}(\bar{\boldsymbol{\theta}}).$ (1) $$

For notational convenience, we denote by $\boldsymbol{\theta}_{1:K}$ and $\mbox{$\mbox{$\mathbf{v}$}$}_{1:K}$ the set of all $\boldsymbol{\theta}_{k}$ and $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ for $k=1$ to $K$. We also use $\ell_{0}$ to denote the regularizer which in the Lagrangian is defined over $\bar{\boldsymbol{\theta}}$.

The two formulations are equivalent because $\bar{\boldsymbol{\theta}}^{*}$ can be recovered by the optimality condition of the Lagrangian minimized with respect to $(\bar{\boldsymbol{\theta}},\boldsymbol{\theta}_{1:K})$ and maximized with respect to $\mbox{$\mbox{$\mathbf{v}$}$}_{1:K}$. Below, we give expressions for the optimality conditions of the original objective (shown on the left) and the Lagrangian (the four equations on the right) for the case when $\ell_{0}(\boldsymbol{\theta})=\mbox{$\frac{1}{2}$}\|\boldsymbol{\theta}\|^{2}$:

$$ $\bar{\boldsymbol{\theta}}^{*}=-\sum_{k=1}^{K}\nabla\ell_{k}(\bar{\boldsymbol{ \theta}}^{*})\,\,\implies\,\,\boldsymbol{\theta}_{k}^{*}=\bar{\boldsymbol{ \theta}}^{*},\quad\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{*}=-\nabla\ell_{k}( \boldsymbol{\theta}_{k}^{*}),\quad\bar{}\mbox{$\mbox{$\mathbf{v}$}$}^{*}=\sum_ {k=1}^{K}\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{*},\quad\bar{\boldsymbol{\theta}}^{ *}=\bar{}\mbox{$\mbox{$\mathbf{v}$}$}^{*}.$ (2) $$

All equations are straightforwardly obtained by simply taking the derivatives of the objectives and setting them to 0. For the Lagrangian, we have used an additional vector $\bar{}\mbox{$\mbox{$\mathbf{v}$}$}$ to highlight the closed loop shown in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1). The four equations on the right indicate the four respective edges in the loop.

Federated ADMM closely follows this loop, implementing an algorithm to obtain the optimality condition of the Lagrangian. The core algorithm is shown in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). To derive the algorithm, first, we collect the terms depending on $\boldsymbol{\theta}_{k}$, and add a quadratic proximity term to optimize the objective $\ell_{k}(\boldsymbol{\theta})+\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}
\boldsymbol{\theta}+\mbox{$\frac{\rho}{2}$}\|\boldsymbol{\theta}-\bar{
\boldsymbol{\theta}}\|^{2}$ with $\rho>0$ as the (inverse) step-size. We locally optimize this to update $\boldsymbol{\theta}_{k}$, which is shown in the first line of
[Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). Then we do the same with all $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ and $\bar{\boldsymbol{\theta}}$ to get the update in the second and third lines respectively. The second line simply accumulates the discrepancy between $\boldsymbol{\theta}_{k}$ and $\bar{\boldsymbol{\theta}}$, while the third line computes averages and sum of $\boldsymbol{\theta}_{1:K}$ and $\mbox{$\mbox{$\mathbf{v}$}$}_{1:K}$ respectively.
An interesting property is that, after the updates in lines 1 and 2, the duals are equal to the gradients: $\mbox{$\mbox{$\mathbf{v}$}$}_{k}=-\nabla\ell_{k}(\boldsymbol{\theta}_{k})$. As a result, optimality is achieved in the third line when $\boldsymbol{\theta}_{k}=\bar{\boldsymbol{\theta}}$, that is, we get $\bar{\boldsymbol{\theta}}=\text{\tt{Sum}}(\mbox{$\mbox{$\mathbf{v}$}$}_{k})$ at convergence.

Figure: Figure 2: FederatedADMM algorithm iteratively updates $(\boldsymbol{\theta}_{1:K},\mbox{$\mbox{$\mathbf{v}$}$}_{1:K},\bar{\boldsymbol {\theta}})$ as shown in line 1-3. Each update is obtained by collecting respective terms from the Lagrangian in [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1) and optimizing with a quadratic proximal term (with learning rate $\rho$ and $\alpha=1/(1+\rho K)$). The functions $\text{\tt{Mean}}(\boldsymbol{\theta}_{1:K})$ and $\text{\tt{Sum}}(\mbox{$\mbox{$\mathbf{v}$}$}_{1:K})$ compute the mean and sum over $\boldsymbol{\theta}_{k}$ and $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ respectively. BayesADMM (which we introduce in [Sec. 3](https://arxiv.org/html/2506.13150v1#S3)) lifts the optimization to probability distributions over the parameters and extends the updates to dual coordinate systems of an exponential family $(\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{1:K},\widehat{\boldsymbol{\lambda}}_{1:K} ,\bar{\boldsymbol{\lambda}})$.

As discussed earlier, the ADMM structure in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1) and the core algorithm in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) continues to be used to this day in more or less the same form in a variety of fields. Then, a natural question arises: Are there more general ways to derive and extend ADMM? Existing literature is full of many new variations and extensions of ADMM, but they more or less use the same derivation as the original method. For instance, a long line of research is concerned with accelerating the convergence of ADMM, such as by considering overrelaxation [16, 11] or momentum [9, 12, 11].
These approaches simply introduce additional variables but do not alter much the form of the main updates. Another line of research considers preconditioned variants where the quadratic proximal terms are replaced by scaled (Mahalanobis) norms [48, 23, 20, 61, 5] or even more general Bregman distances [58, 57]. This changes the form of the updates but the overall structure still remains the same.

Ideally, we want extensions that can solve the most pressing issues for federated deep learning, for instance, methods that can deal with heterogeneity and uncertainty arising from the distributed nature of the problem. With such applications in mind, we turn to Bayesian methods, as they are naturally suited to handle such issues. Bayesian methods have a rich history for distributed learning, starting with algorithms such as Kalman filtering [28], forward-backward algorithm [50], expectation maximization [14], and their junction-tree versions [35].
These have been used to solve decentralized and distributed problems in signal processing [42, 15], but work on federated deep learning is not extensive yet. Existing works in Bayesian federated learning [3, 34, 25, 62, 37, 46] use fundamentally different methodologies to ADMM.
Our goal here is bridge this gap, so that Bayesian ideas can be naturally incorporated in ADMM and used to improve it.

Our work is inspired by a recent work by Swaroop et al. [54] who show a connection between federated ADMM and a variational Bayesian method called Partitioned Variational Inference (PVI) [6]. They show a line-by-line correspondence between the ADMM and PVI updates, and use it to derive new extensions.
The server update in [54] is different from that used in ADMM and corresponds what is used in the Alternating Minimization Algorithm (AMA) [55] (we discuss these details in [App. A](https://arxiv.org/html/2506.13150v1#A1)).
As a result, their framework is not able to derive and extend ADMM.
In the next section, we describe our new proposal to fix these issues, as well as provide further understanding of the intriguing similarities between PVI and ADMM using the Bayesian duality structure from [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1).
These insights will lead to algorithms that are both computationally cheaper and empirically better than Swaroop et al. [54].

## 3 A New Bayesian Duality Structure

Here, we present our new Bayesian duality structure shown in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1) (right), and then use it to derive the BayesADMM algorithm shown in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). Bayesian duality follows from the optimality condition of a VB objective which lifts [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1) to the space of exponential-family (EF) distributions $\mathcal{Q}$,

$$ $\bar{q}^{*}=\underset{{q\in\mathcal{Q}}}{\operatorname{argmin}}~{}\sum_{k=1}^{ K}\mathbb{E}_{\bar{q}}[\ell_{k}(\boldsymbol{\theta})]+{\operatorname{KL}}{} \left(q\,\|\,\pi_{0}\right){},\,\text{ where }\,q(\boldsymbol{\theta})=\exp \left(\langle\boldsymbol{\lambda},\text{\mbox{$\mbox{$\mathbf{T}$}$}}( \boldsymbol{\theta})\rangle-A(\boldsymbol{\lambda})\right).$ (3) $$

Here, the second term is the Kullback-Leibler (KL) divergence between an exponential family $q(\boldsymbol{\theta})$ and the prior $\pi_{0}(\boldsymbol{\theta})\propto e^{-\ell_{0}(\boldsymbol{\theta})}$.
The distribution $q$ is defined using natural parameters $\boldsymbol{\lambda}$, sufficient statistics $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$, and an inner product $\langle\boldsymbol{\lambda},\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})\rangle$ between them.
The log-partition function $A(\boldsymbol{\lambda})$ is a convex function to ensure that $q$ normalizes to one, but it also defines a dual-coordinate system which is important for us to derive Bayesian Duality. Essentially, we can define a dual-coordinate $\mbox{$\mbox{$\boldsymbol{\mu}$}$}=\nabla A(\boldsymbol{\lambda})=\mathbb{E}_{
q}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})]$ to $\boldsymbol{\lambda}$, also known as the expectation parameter. The pair $(\boldsymbol{\lambda},\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ constitutes a dual-coordinate system [4] which we use to define the two space shown in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1) (right).
A self-contained introduction to EFs is in [App. B](https://arxiv.org/html/2506.13150v1#A2) and more details are in Wainwright et al. [56].

### 3.1 Bayesian Duality and BayesADMM

We start by writing the stationary condition for the VB objective, which takes a similar to the stationarity condition in [Eq. 2](https://arxiv.org/html/2506.13150v1#S2.E2). This is derived by taking the derivative with respect to $\boldsymbol{\mu}$,

$$ $\nabla_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}{\operatorname{KL}}{}\left( \bar{q}^{*}\,\|\,\pi_{0}\right){}=\nabla_{\text{\mbox{$\mbox{$\boldsymbol{\mu} $}$}}}\sum_{k=1}^{K}\mathbb{E}_{\bar{q}^{*}}[\ell_{k}(\boldsymbol{\theta})] \quad\implies\quad\bar{\boldsymbol{\lambda}}^{*}=-\sum_{k=0}^{K}\nabla_{\text{ \mbox{$\mbox{$\boldsymbol{\mu}$}$}}}\mathbb{E}_{\bar{q}^{*}}[\ell_{k}( \boldsymbol{\theta})]\bigl{|}_{\text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$}$}= \text{$\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$}^{*}$}}.$ (4) $$

The simplification at the right follows as shown by Khan & Rue [31, Eq. 5]. They also show that the gradient with respect to $\boldsymbol{\mu}$ is the natural-gradient with respect to $\boldsymbol{\lambda}$ [31, Eq. 4].
An important aspect of the condition is that it exploits the $(\boldsymbol{\lambda},\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ coordinate system where gradients computed in the $\boldsymbol{\mu}$-space are mapped to the $\boldsymbol{\lambda}$-space. This is analogous to the use of the $(\mbox{$\mbox{$\mathbf{v}$}$},\boldsymbol{\theta})$ pair in [Eq. 2](https://arxiv.org/html/2506.13150v1#S2.E2), where the gradient in $\mathbf{v}$-space is mapped to an element in the $\boldsymbol{\theta}$-space. For VB, the gradient is
replaced by natural-gradients $\nabla_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}$ and the two spaces $(\boldsymbol{\lambda},\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ are duals of each other, with the map $\nabla A$.

We will exploit the dual-coordinate system to generalize the Lagrangian from [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1). We denote the server and client distributions by $\bar{q}$ and $q_{k}$ respectively, and their respective dual-pairs are denoted by $(\bar{\boldsymbol{\lambda}},\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ and $(\boldsymbol{\lambda}_{k},\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k})$. Then, instead of the constraint $\bar{\boldsymbol{\theta}}=\boldsymbol{\theta}_{k}$, we constrain the respective expectation parameters $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}=\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$}$. This choice of coordinates is essential to our proposal. Exponential families are derived using expectation constraints, and natural parameters are Lagrange multipliers (see [App. B](https://arxiv.org/html/2506.13150v1#A2)). Using $\boldsymbol{\mu}$ as a primal variable aligns with this duality of exponential families. We will also see that the use of $\boldsymbol{\mu}$-coordinates in the primal yields addition and subtraction in $\boldsymbol{\lambda}$-space in the updates which is natural as it corresponds to multiplication or division of exponential family distributions. This property is used to obtain mirror-descent formulations of Bayesian learning [31, Eq. 22]. With these considerations in mind, we propose the following Lagrangian,

$$ $\mathcal{L}(\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$},\mbox{$\mbox{$\boldsymbol {\mu}$}$}_{1:K},\widehat{\boldsymbol{\lambda}}_{1:K})=\sum_{k=1}^{K}\left( \mathbb{E}_{q_{k}}[\ell_{k}(\boldsymbol{\theta})]\!+\!\langle\widehat{ \boldsymbol{\lambda}}_{k},\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}\!-\!\bar{} \mbox{$\mbox{$\boldsymbol{\mu}$}$}\rangle\right)+{\operatorname{KL}}{}\left( \bar{q}\,\|\,\pi_{0}\right){},$ (5) $$

which is analogous to the one shown in [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1) but now formulated in dual-coordinates. There are four main differences: the loss $\ell_{k}$ is replaced by its expectation, the inner product corresponds to the one used to define $q$, Lagrange multipliers are denoted by $\smash{\widehat{\boldsymbol{\lambda}}_{k}}$, and instead of $\ell_{0}$ we now have a KL term to the prior $\pi_{0}$.
As before, by taking the derivative of the Lagrangian over the individual arguments, we get the first three conditions,

$$ $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}^{*}=\bar{}\mbox{$\mbox{$\boldsymbol{\mu }$}$}^{*},\quad\widehat{\boldsymbol{\lambda}}_{k}^{*}=-\nabla_{\text{$\mbox{$ \mbox{$\boldsymbol{\mu}$}$}$}}\mathbb{E}_{q_{k}}[\ell_{k}]\big{|}_{\text{$ \mbox{$\mbox{$\boldsymbol{\mu}$}$}$}=\text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$} _{k}^{*}$}},\quad\bar{\boldsymbol{\lambda}}^{*}=\sum_{k=0}^{K}\widehat{ \boldsymbol{\lambda}}_{k}^{*},\quad\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$}^{* }=\nabla A(\bar{\boldsymbol{\lambda}}^{*}),$ (6) $$

and the last equation is the EF’s dual map. These four equations together give us the Bayesian Duality structure shown in [Fig. 1](https://arxiv.org/html/2506.13150v1#S1.F1). The structure mirrors the ADMM one but uses the more general $(\boldsymbol{\lambda},\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ coordinates. The four edges of Bayesian Duality can be implemented in the same manner as ADMM. Lagrangian formulations of variational-Bayesian problems have been used by Khan et al. [32, 29] for dual optimization of Gaussian latent models, but they do not exploit the duality of exponential families, rather using mean-covariance parametrizations of Gaussians. Similar to us, Adam et al. [2] propose a Lagrangian in $(\mbox{$\mbox{$\boldsymbol{\mu}$}$},\boldsymbol{\lambda})$-coordinates but with a different goal of improving SVGPs.

An algorithm to solve the optimality condition in [Eq. 6](https://arxiv.org/html/2506.13150v1#S3.E6) is shown in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). We call it BayesADMM because it follows the same steps as ADMM but is derived from the VB objective. The updates of $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}$ and $\bar{\boldsymbol{\mu}}$ follow by collecting all their terms in the Lagrangian, and then adding a proximal term. One main difference is that instead of using the quadratic proximity term, we use a KL term. The resulting updates are shown below:

$$ $\displaystyle\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}\leftarrow\operatorname{ argmin}_{\text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}$}}\,\mathbb{E}_{q_{k}}[ \ell_{k}(\boldsymbol{\theta})]+\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox {$\mbox{$\boldsymbol{\mu}$}$}_{k}\rangle+\rho{\operatorname{KL}}{}\left(q_{k} \,\|\,\bar{q}\right){},$ (7) $\displaystyle\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$}\leftarrow\operatorname{ argmin}_{\text{$\bar{}\mbox{$\mbox{$\boldsymbol{\mu}$}$}$}}\,\sum_{k=1}^{K} \langle\widehat{\boldsymbol{\lambda}}_{k},-\bar{}\mbox{$\mbox{$\boldsymbol{\mu }$}$}\rangle+{\operatorname{KL}}{}\left(\bar{q}\,\|\,\pi_{0}\right){}+\rho\sum _{k=1}^{K}{\operatorname{KL}}{}\left(\bar{q}\,\|\,q_{k}\right){}.$ (8) $$

These give us lines 1 and 3 of BayesADMM in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). To get line 1, we simply use the definition of $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}=\mathbb{E}_{q_{k}}[\mbox{$\mbox{$
\mathbf{T}$}$}(\boldsymbol{\theta})]$ and linearity of the expectation. To get
line 3, we use the expression for the gradient of KL [31, Eq. 23]. To simplify notation in line 3, we denote $\widehat{\boldsymbol{\lambda}}_{0}$ as the natural gradient of $\mathbb{E}_{\bar{q}}[\ell_{0}(\boldsymbol{\theta})]$.
Compared to PVI [6, 54], BayesADMM (i) has a different server update for $\bar{q}$, and (ii) there are learning rates $\rho$ in all of our updates, matching the step-sizes from ADMM. These differences lead to quicker convergence, which we show in [Fig. 3a](https://arxiv.org/html/2506.13150v1#S3.F3.sf1). The details of the experiment in that figure are described in [Sec. F.1](https://arxiv.org/html/2506.13150v1#A6.SS1). We show in [App. A](https://arxiv.org/html/2506.13150v1#A1) that the PVI updates are closer to the Alternating Minimization Algorithm (AMA) of Tseng [55] than they are to ADMM. The difference of AMA to ADMM is that no proximal term in [Eq. 8](https://arxiv.org/html/2506.13150v1#S3.E8) is used, see Tseng [55, Eq. 3.4a], and that convergence is only guaranteed for certain choices of $\rho$, see Tseng [55, Eq. 3.4d].

The derivation of line 2 of BayesADMM fundamentally departs from the standard procedure. Typically, we would simply maximize the Lagrangian with respect to $\widehat{\boldsymbol{\lambda}}_{k}$, but we do not do this. We view $\widehat{\boldsymbol{\lambda}}_{k}$ as Lagrange multipliers to expectation constraints, and use $\boldsymbol{\lambda}$-coordinates in the dual update, as they play a similar role (see [App. B](https://arxiv.org/html/2506.13150v1#A2)). This choice aligns with the Bayesian Duality structure and ensures that the dual variables are natural gradients:

$$ $\widehat{\boldsymbol{\lambda}}_{k}\leftarrow\widehat{\boldsymbol{\lambda}}_{k} +\rho(\boldsymbol{\lambda}_{k}-\bar{\boldsymbol{\lambda}})\quad\implies\quad \widehat{\boldsymbol{\lambda}}_{k}=-\widetilde{\nabla}\mathbb{E}_{q_{k}}[\ell_ {k}].$ (9) $$

The second equation follows from the optimality condition of lines 1 and 2. This update step also marks a fundamental departure from other generalizations of ADMM used in the optimization literature [58, 57, 7, 44], which use increments in $\boldsymbol{\mu}$-space, and also from nonlinear augmented Lagrangian approaches [38, 17, 8, 45]. In practice, these differences also improve convergence of our method, see [Figs. 3a](https://arxiv.org/html/2506.13150v1#S3.F3.sf1) and [3b](https://arxiv.org/html/2506.13150v1#S3.F3.sf2) for a comparison to BregmanADMM [58].

We also note that this matches with the (damped) dual update in PVI [6, 54] which we explain in [App. A](https://arxiv.org/html/2506.13150v1#A1).
In PVI, the updates follow a Bayesian framework without any connection to duality, Lagrange multipliers or ADMM-like methods. PVI can diverge without an additional damping term and when only damping in the dual update is used [6], convergence is still slower.

We also show in [App. A](https://arxiv.org/html/2506.13150v1#A1) that the dual update in PVI with damping is related to a local variant of the Bayesian learning rule (BLR) [31].
The BLR update [31],
$\bar{\boldsymbol{\lambda}}\leftarrow(1-\alpha)\bar{\boldsymbol{\lambda}}+
\alpha\sum_{k=0}^{K}\nabla_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}\mathbb{
E}_{\bar{q}}[-\ell_{k}]$,
also solves the variational-Bayesian problem in [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3). BayesADMM mimics the BLR steps when the client updates are iterated until convergence before doing a server update.

###### Proposition 3.1 .

If the client steps 1 and 2 in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) are iterated until convergence for a given $\bar{\boldsymbol{\lambda}}$, then the following server step in line 3 is equivalent to the BLR of Khan & Rue [31].

The proof (and other proofs) are given in [App. E](https://arxiv.org/html/2506.13150v1#A5).
We can see that BayesADMM evaluates the natural gradients at local variables $q_{k}$ unlike BLR’s global $\bar{q}$ by comparing the BLR update with the third step of BayesADMM in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) and using [Eq. 9](https://arxiv.org/html/2506.13150v1#S3.E9).
This allows us to view BayesADMM as an inexact but distributed version of the BLR, showing how it is related to existing Bayesian algorithms (see [31, Table 1]), unlike existing ADMM methods.

Figure: Algorithm 1 BayesADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) for Gaussians with diagonal covariance. Additional steps when compared to FederatedADMM are highlighted in red. Implementation details are in [App. D](https://arxiv.org/html/2506.13150v1#A4).

###### Proposition 3.1 .

### 3.2 Deriving Federated ADMM from Bayesian Duality

We now show that BayesADMM naturally recovers the original ADMM when isotropic Gaussian posteriors are used in the VB objective, and later extend ADMM by using a more flexible posterior form. Given the form of the posterior, the procedure is straightforward. We just do the following:

- 1.
Identify the EF’s $(\boldsymbol{\lambda},\mbox{$\mbox{$\boldsymbol{\mu}$}$})$ pairs and $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$, for example, using Nielsen & Garcia [43].
- 2.
Derive the form of the natural gradient, for example, using Khan & Rue [31].

We illustrate the procedure to derive ADMM by using isotropic Gaussians $q(\boldsymbol{\theta})=\mbox{${\cal N}$}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{
$\mathbf{m}$}$},\mbox{$\mbox{$\mathbf{I}$}$})$. For this choice, we can write all the necessary quantities for step 1 and 2 below,

$$ $\boldsymbol{\lambda}=\mbox{$\mbox{$\mathbf{m}$}$},\quad\mbox{$\mbox{$ \boldsymbol{\mu}$}$}=\mbox{$\mbox{$\mathbf{m}$}$},\quad\mbox{$\mbox{$\mathbf{T }$}$}(\boldsymbol{\theta})=\boldsymbol{\theta},\quad\nabla_{\text{$\mbox{$ \mbox{$\boldsymbol{\mu}$}$}$}}\mathbb{E}_{q_{k}}[\ell_{k}]=\mathbb{E}_{q_{k}}[ \nabla\ell_{k}].$ (10) $$

The last equality is due to Bonnet’s theorem [51] which implies that $\widehat{\boldsymbol{\lambda}}_{k}$ lies in the gradient space, similarly to $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$. Because the KL divergence between two Gaussians with identity covariance is simply the squared Euclidean distance, we can write line 1 of BayesADMM as,

$$ $\mbox{$\mbox{$\mathbf{m}$}$}_{k}\leftarrow\operatorname{argmin}_{\text{$\mbox{ $\mbox{$\mathbf{m}$}$}_{k}$}}\,\mathbb{E}_{q_{k}}[\ell_{k}(\boldsymbol{\theta} )]+\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$\mathbf{m}$}$}_{k} \rangle+\frac{\rho}{2}\|\mbox{$\mbox{$\mathbf{m}$}$}_{k}-\bar{}\mbox{$\mbox{$ \mathbf{m}$}$}\|^{2}.$ (11) $$

Line 1 in ADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) is recovered by approximating $\mathbb{E}_{q}[\nabla\ell_{k}]\approx\ell_{k}(\mbox{$\mbox{$\mathbf{m}$}$})$. This is the so-called delta approximation [31, Sec. 1.3.1], which is also used by Swaroop et al. [54] to derive connections to ADMM. In general, we can perform a delta approximation in the optimality condition of line 1 of BayesADMM to obtain a method which solves the original objective in [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1), which for Gaussians is related to the Laplace approximation method. Derivation of other lines follow in a similar fashion. A detailed derivation is in [App. C](https://arxiv.org/html/2506.13150v1#A3). It is also possible to keep the expectation in the update, in which case this will use a Gaussian loss smoothing.
This is closely related to sharpness-aware minimization (SAM) [19, 41] which works well in federated learning [49, 18]. Our new BayesADMM naturally includes such ideas.

### 3.3 Extending Federated ADMM using Bayesian Duality

We show a simple extension by considering a Gaussians with diagonal covariances which then leads to a fast and scalable variant of ADMM with Adam-like diagonal adaptive preconditioning. We consider such extensions to exploit posterior covariance and better handle heterogeneity. Denoting $q(\boldsymbol{\theta})=\mbox{${\cal N}$}(\boldsymbol{\theta}|\mbox{$\mbox{$
\mathbf{m}$}$},\mbox{$\mbox{diag}$}(\mbox{$\mbox{$\mathbf{s}$}$})^{-1})$ where $\mathbf{s}$ is the precision vector, we can write the following,

$$ $\boldsymbol{\lambda}=(\mbox{$\mbox{$\mathbf{s}$}$}\mbox{$\mbox{$\mathbf{m}$}$} ,-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{s}$}$}),\,\mbox{$\mbox{$ \boldsymbol{\mu}$}$}=(\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$\mbox{$\mathbf{m}$}$ }^{2}+1/\mbox{$\mbox{$\mathbf{s}$}$}),\,\mbox{$\mbox{$\mathbf{T}$}$}( \boldsymbol{\theta})=(\boldsymbol{\theta},\boldsymbol{\theta}^{2}),\,\nabla_{ \text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$}$}}\mathbb{E}_{q_{k}}[\ell_{k}]=\left (\mbox{$\mbox{$\mathbf{g}$}$}_{k}-\mbox{$\mbox{$\mathbf{h}$}$}_{k}\mbox{$\mbox {$\mathbf{m}$}$}_{k},\,\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{h}$}$}_{k} \right).$ (12) $$

Here, all multiplication (such as $\mathbf{s}$$\mathbf{m}$) and divisions (such as $1/\mbox{$\mbox{$\mathbf{s}$}$}$) are element wise. The last equation is obtained from Khan & Rue [31, Eq. 10-11] where we denote the expected gradient and Hessian by $\mbox{$\mbox{$\mathbf{g}$}$}_{k}=\mathbb{E}_{q_{k}}[\nabla\ell_{k}]$ and $\mbox{$\mbox{$\mathbf{h}$}$}_{k}=\mathbb{E}_{q_{k}}[\mbox{$\mbox{diag}$}(
\nabla^{2}\ell_{k})]$, respectively. Using these in line 1-3 of BayesADMM gives us a new Adam-like extension of ADMM shown in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1). A detailed derivation is in [App. C](https://arxiv.org/html/2506.13150v1#A3).

We now briefly describe the algorithm to show that its implementation is nearly identical to the standard ADMM and as easy. Line 4 in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) implements the first line in BayesADMM. This is obtained by essentially expanding defining $\widehat{\boldsymbol{\lambda}}_{k}=(\mbox{$\mbox{$\mathbf{v}$}$}_{k},-\mbox{$
\frac{1}{2}$}\mbox{$\mbox{$\mathbf{u}$}$}_{k})$ with an additional dual variable $\mbox{$\mbox{$\mathbf{u}$}$}_{k}$ to handle the constraints in the precision $\mbox{$\mbox{$\mathbf{s}$}$}_{k}$. Therefore, the dual term becomes $\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$\mathbf{T}$}$}(
\boldsymbol{\theta})\rangle=\boldsymbol{\theta}^{\top}\mbox{$\mbox{$\mathbf{v}
$}$}_{k}-\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}(\mbox{$\mbox{$\mathbf{
u}$}$}_{k}\boldsymbol{\theta})/2$. The second term is new and is highlighted in red in line 4. Similarly, the KL divergence between the Gaussian gives rise to a scaling of the quadratic term by $\bar{}\mbox{$\mbox{$\mathbf{s}$}$}$ which is also highlighted in red. We propose to implement line 4 by using the IVON optimizer [53] which automatically yields both $\mbox{$\mbox{$\mathbf{m}$}$}_{k}$ and $\mbox{$\mbox{$\mathbf{s}$}$}_{k}$ for the VB training. It has no additional overhead over standard training with Adam.

Line 5 and 6 implement the second line in BayesADMM and they are straightforwardly obtained by simply plugging the definitions from [Eq. 12](https://arxiv.org/html/2506.13150v1#S3.E12). Line 6 updates the dual variable of $\mbox{$\mbox{$\mathbf{s}$}$}_{k}$ but just involves addition and subtraction which are easy to implement. Finally, line 9-11 implement the third line of BayesADMM and these are simple additions of natural parameters and the dual variables. These are also easy to implement. In line 6, we use a different step-size $\gamma$ for the dual
update which we found to give better results (also a choice that is common in the literature [57]).

To summarize, the new extension has almost the same cost as Adam and is as easy to implement. It adds three new lines (6,10,11) all of which involve simple addition and subtraction, while the line 4 can be implemented by IVON whose cost is the same as Adam. Communication cost is doubled compared to federated ADMM as we send both a mean and diagonal covariance vector, similarly to other preconditioned or Bayesian methods [54]. However, this is expected to help in cases with
heterogeneity.

On the toy illustrations in [Figs. 3a](https://arxiv.org/html/2506.13150v1#S3.F3.sf1) and [3b](https://arxiv.org/html/2506.13150v1#S3.F3.sf2), we use BayesADMM with full covariances, which is derived again using the above two-step procedure. The derivation is in [App. C](https://arxiv.org/html/2506.13150v1#A3). Similar to message passing algorithms [60, 33], this method can converge in a single round of communication, which is unlike ADMM and BregmanADMM [58].

###### Proposition 3.2 .

If $\ell_{k}(\boldsymbol{\theta})=-\langle\mbox{$\mbox{$\mathbf{t}$}$}_{k},\mbox{$
\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})\rangle$ (for example, $\ell_{k}(\boldsymbol{\theta})=\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}
\mbox{$\mbox{$\mathbf{A}$}$}\boldsymbol{\theta}+\mbox{$\mbox{$\mathbf{b}$}$}^{
\top}\boldsymbol{\theta}$ in the case of full Gaussian distributions), step-size $\rho=1/K$ and initializing the server at the prior $\bar{\boldsymbol{\lambda}}=\widehat{\boldsymbol{\lambda}}_{0}$, then BayesADMM ([Eqs. 7](https://arxiv.org/html/2506.13150v1#S3.E7), [9](https://arxiv.org/html/2506.13150v1#S3.E9) and [8](https://arxiv.org/html/2506.13150v1#S3.E8)) converges to the solution of [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3) after one communication round.

In [Fig. 3b](https://arxiv.org/html/2506.13150v1#S3.F3.sf2), we show a federated ridge regression problem with full Gaussian posteriors (experiment details in [Sec. F.2](https://arxiv.org/html/2506.13150v1#A6.SS2)), where BayesADMM converges in a single round. This is similar to Newton-methods which converge in a single step on quadratic functions, but generalizations of such results to the ADMM setting have been missing so far.

Figure: a Logistic Regression
Refer to caption: x2.png

###### Proposition 3.2 .

## 4 Numerical Experiments

#### Illustrative example.

We show how BayesADMM can better deal with heterogeneous and noisy data than standard ADMM in [Fig. 4](https://arxiv.org/html/2506.13150v1#S4.F4). We compare federated ADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) to full-covariance BayesADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) in 2D linear classification: data is split over two clients, and client 1 contains an outlier.
We see how federated ADMM takes many (five) communication rounds to resolve the issue due to the outlier.
In contrast, BayesADMM maintains high uncertainty for the outlier, and quickly adapts in a single round, illustrating the benefits of maintaining posterior uncertainties.

Figure: Figure 4: A single outlier that slows down ADMM (top row) poses no issues for our new Bayesian version BayesADMM (bottom row). The server (left column) with ADMM takes 5 iterations while with Bayes only needs 2. Client 1 (middle column) is the source of the issue which takes 5 iterations to ignore the outlier, while with Bayes, it is much faster due to the use of uncertainty (gray lines). The rightmost column shows client 2 where there are no outliers.
Refer to caption: x4.png

#### Evaluation on Federated Deep Learning Benchmarks.

We follow a similar experimental setup to the recent work of Swaroop et al. [54] and run fully-connected and convolutional neural networks on image datasets.
We consider different levels of data heterogeneity on the clients and different numbers of clients.
We focus on comparing our IVON implementation of BayesADMM in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) to
(i) the distributed Bayesian algorithms FedLap and FedLap-Cov [54],
(ii) FedDyn [1], which is the best-performing federated ADMM-style algorithm, and
(iii) standard baseline algorithms for
federated learning, FedAvg [39] and FedProx [36].
Further hyperparameters and details are in [App. G](https://arxiv.org/html/2506.13150v1#A7).
Overall, we find that BayesADMM outperforms baselines in most scenarios; BayesADMM has lower test loss than baselines, showing the benefits of a Bayesian method; and BayesADMM is significantly computationally cheaper than the second-best method FedLap-Cov. Ensembling predictions over the learned BayesADMM posterior at test-time usually improves performance more.

**Table 1: Test accuracy and test NLL for different (10 and 100-client) scenarios after 10, 25 and 50 rounds, with mean and standard deviations over 3 runs. We see that BayesADMM significantly outperforms all baselines on FashionMNIST (* note that this uses 10% of total data, following previous work) and on the second CIFAR-10 CNN, while being worse on the first CNN. Averaging over the posterior in BayesADMM (as opposed to BayesADMM@$\mathbf{m}$) often improves performance.**
|  |  | Test accuracy ($\uparrow$ larger is better) | Test NLL ($\downarrow$ smaller is better) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Scenario | Method | 10 rounds | 25 rounds | 50 rounds | 10 rounds | 25 rounds | 50 rounds |
| MLP, 10 clients<br>homogeneous<br>FMNIST^(∗) | FedAvg | 72.3$\pm$0.4 | 77.7$\pm$0.3 | 80.0$\pm$0.2 | 0.70$\pm$0.00 | 0.61$\pm$0.01 | 0.56$\pm$0.01 |
| FedProx | 72.2$\pm$0.3 | 77.4$\pm$0.1 | 80.3$\pm$0.1 | 0.71$\pm$0.00 | 0.61$\pm$0.01 | 0.56$\pm$0.01 |  |
| FedDyn | 75.3$\pm$0.8 | 77.5$\pm$0.8 | 78.2$\pm$0.5 | 0.67$\pm$0.01 | 0.63$\pm$0.02 | 0.63$\pm$0.02 |  |
| FedLap | 72.1$\pm$0.2 | 77.1$\pm$0.1 | 80.2$\pm$0.1 | 0.71$\pm$0.01 | 0.62$\pm$0.01 | 0.57$\pm$0.01 |  |
| FedLap-Cov | 75.0$\pm$0.6 | 79.8$\pm$0.4 | 81.8$\pm$0.1 | 0.67$\pm$0.01 | 0.59$\pm$0.01 | 0.56$\pm$0.01 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 80.4$\pm$0.2 | 83.1$\pm$0.1 | 83.4$\pm$0.1 | 0.56$\pm$0.00 | 0.53$\pm$0.00 | 0.60$\pm$0.01 |  |
| BayesADMM | 80.6$\pm$0.2 | 83.5$\pm$0.1 | 84.1$\pm$0.2 | 0.55$\pm$0.00 | 0.50$\pm$0.00 | 0.53$\pm$0.01 |  |
| MLP, 10 clients<br>hetereogeneous<br>FMNIST^(∗) | FedAvg | 70.4$\pm$0.9 | 74.3$\pm$0.5 | 76.0$\pm$0.7 | 0.78$\pm$0.04 | 0.72$\pm$0.03 | 0.67$\pm$0.04 |
| FedProx | 69.9$\pm$0.4 | 74.7$\pm$0.6 | 76.9$\pm$0.9 | 0.80$\pm$0.04 | 0.71$\pm$0.03 | 0.66$\pm$0.03 |  |
| FedDyn | 73.0$\pm$0.6 | 74.6$\pm$0.4 | 74.6$\pm$0.5 | 0.75$\pm$0.04 | 0.70$\pm$0.03 | 0.77$\pm$0.05 |  |
| FedLap | 71.3$\pm$0.9 | 74.3$\pm$0.4 | 77.6$\pm$0.7 | 0.75$\pm$0.04 | 0.71$\pm$0.06 | 0.65$\pm$0.05 |  |
| FedLap-Cov | 74.6$\pm$0.7 | 78.3$\pm$1.0 | 80.5$\pm$0.6 | 0.70$\pm$0.04 | 0.63$\pm$0.04 | 0.60$\pm$0.04 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 77.0$\pm$0.8 | 81.4$\pm$0.4 | 82.1$\pm$0.1 | 0.65$\pm$0.02 | 0.53$\pm$0.01 | 0.51$\pm$0.00 |  |
| BayesADMM | 77.0$\pm$0.8 | 81.5$\pm$0.5 | 82.3$\pm$0.2 | 0.65$\pm$0.02 | 0.52$\pm$0.01 | 0.50$\pm$0.00 |  |
| MLP, 100 clients<br>heterog.<br>FMNIST | FedAvg | 73.9$\pm$0.3 | 78.7$\pm$0.3 | 81.8$\pm$0.3 | 0.73$\pm$0.01 | 0.60$\pm$0.01 | 0.53$\pm$0.01 |
| FedProx | 73.3$\pm$0.4 | 78.4$\pm$0.1 | 81.6$\pm$0.1 | 0.74$\pm$0.01 | 0.61$\pm$0.02 | 0.53$\pm$0.02 |  |
| FedDyn | 75.7$\pm$0.4 | 81.4$\pm$0.2 | 82.2$\pm$0.3 | 0.66$\pm$0.01 | 0.53$\pm$0.02 | 0.51$\pm$0.01 |  |
| FedLap | 73.9$\pm$0.3 | 79.4$\pm$0.2 | 82.4$\pm$0.3 | 0.68$\pm$0.01 | 0.59$\pm$0.00 | 0.53$\pm$0.01 |  |
| FedLap-Cov | 76.9$\pm$0.7 | 81.3$\pm$0.2 | 83.0$\pm$0.1 | 0.64$\pm$0.01 | 0.54$\pm$0.00 | 0.49$\pm$0.01 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 78.7$\pm$0.1 | 82.2$\pm$0.1 | 82.9$\pm$0.2 | 0.62$\pm$0.00 | 0.50$\pm$0.00 | 0.48$\pm$0.00 |  |
| BayesADMM | 78.7$\pm$0.1 | 82.3$\pm$0.1 | 83.0$\pm$0.2 | 0.62$\pm$0.00 | 0.50$\pm$0.00 | 0.48$\pm$0.00 |  |
| CNN from<br>Zenke et al. [63]<br>10 clients<br>heterogeneous<br>CIFAR-10 | FedAvg | 73.8$\pm$0.5 | 75.0$\pm$0.5 | 75.1$\pm$0.3 | 0.9$\pm$0.1 | 1.1$\pm$0.1 | 1.5$\pm$0.1 |
| FedProx | 73.8$\pm$1.5 | 75.0$\pm$0.9 | 75.4$\pm$0.8 | 0.9$\pm$0.0 | 1.2$\pm$0.1 | 1.5$\pm$0.1 |  |
| FedDyn | 72.7$\pm$0.9 | 77.4$\pm$0.6 | 79.4$\pm$0.4 | 1.0$\pm$0.0 | 0.8$\pm$0.0 | 0.7$\pm$0.0 |  |
| FedLap | 74.8$\pm$1.3 | 78.0$\pm$1.2 | 79.5$\pm$1.4 | 0.7$\pm$0.0 | 0.6$\pm$0.0 | 0.6$\pm$0.0 |  |
| FedLap-Cov | 75.1$\pm$1.1 | 77.6$\pm$0.7 | 79.2$\pm$0.9 | 0.7$\pm$0.0 | 0.7$\pm$0.0 | 0.6$\pm$0.0 |  |
| BayesADMM@m | 71.5$\pm$1.8 | 78.9$\pm$0.8 | 80.3$\pm$0.6 | 0.8$\pm$0.0 | 0.6$\pm$0.0 | 0.6$\pm$0.0 |  |
| BayesADMM | 71.5$\pm$1.8 | 79.0$\pm$0.8 | 80.3$\pm$0.6 | 0.8$\pm$0.0 | 0.6$\pm$0.0 | 0.6$\pm$0.0 |  |
| CNN from<br>Acar et al. [1]<br>10 clients<br>heterogeneous<br>CIFAR-10 | FedAvg | 62.8$\pm$3.1 | 65.4$\pm$1.8 | 66.0$\pm$1.5 | 1.1$\pm$0.1 | 1.8$\pm$0.1 | 2.3$\pm$0.1 |
| FedProx | 64.3$\pm$2.0 | 65.9$\pm$1.6 | 66.3$\pm$1.4 | 1.7$\pm$0.1 | 2.2$\pm$0.1 | 2.6$\pm$0.1 |  |
| FedDyn | 63.6$\pm$1.1 | 64.7$\pm$0.6 | 65.4$\pm$1.0 | 2.0$\pm$0.6 | 3.6$\pm$1.3 | 2.6$\pm$0.1 |  |
| FedLap | 60.2$\pm$2.4 | 66.4$\pm$1.1 | 66.5$\pm$1.2 | 1.1$\pm$0.0 | 1.3$\pm$0.1 | 1.8$\pm$0.1 |  |
| FedLap-Cov | 58.4$\pm$2.4 | 65.4$\pm$1.1 | 67.5$\pm$1.2 | 1.2$\pm$0.0 | 1.0$\pm$0.0 | 1.0$\pm$0.0 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 63.8$\pm$1.4 | 69.5$\pm$0.8 | 70.2$\pm$0.7 | 1.0$\pm$0.0 | 1.0$\pm$0.0 | 1.5$\pm$0.0 |  |
| BayesADMM | 63.8$\pm$1.4 | 69.5$\pm$0.8 | 70.3$\pm$0.6 | 1.0$\pm$0.0 | 1.0$\pm$0.0 | 1.4$\pm$0.0 |  |

#### Scenarios and splits.

We consider fully connected neural networks [39, 36] on the MNIST
and FashionMNIST datasets.
Following the setup by Swaroop et al. [54] we use 10% of the data in FMNIST but use the full MNIST dataset.
We consider homogeneous, heterogeneous and highly-heterogeneous splits on either $K=10$ or $K=100$ clients. We also consider two different convolutional neural network architectures on CIFAR-10.
One CNN is from Zenke et al. [63] which has Dropout, and the other CNN is from Acar et al. [1] which does not have Dropout and uses more fully connected layers.
On CIFAR-100 we use a ResNet-20 architecture.
For all heterogeneous splits, we follow previous work and sample Dirichlet distributions that decide how many points per class go into each client [54].
For the highly-heterogeneous split on MNIST with $K=100$ clients, we assign data from only two random classes per client [39].
We perform hyperparameter sweeps as in Swaroop et al. [54].

Our results are summarized in [Tables 1](https://arxiv.org/html/2506.13150v1#S4.T1) and [2](https://arxiv.org/html/2506.13150v1#S4.T2).
For BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ we evaluate using server’s $\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$, while for BayesADMM, we ensemble predictions across 32 samples from $\bar{q}$. We show accuracy and test loss (NLL) averaged over three seeds and the three previous rounds after $10$, $25$ and $50$ rounds.

#### Methods and hyperparameters.

We compare our implementation of BayesADMM to five baselines: FedAvg [39], FedProx [36] and FedDyn [1] and FedLap/FedLap-Cov [54]. FedDyn [1] is a well-performing implementation of standard ADMM for federated learning, and FedLap/FedLap-Cov are implementations of the distribution Bayesian learning algorithm in [54] using Laplace approximations. Hyperparameters are described in [App. G](https://arxiv.org/html/2506.13150v1#A7).

#### BayesADMM is overall the best method.

We compare BayesADMM to other baselines for both 10 and 100-client splits on FMNIST and CIFAR-10 ([Table 1](https://arxiv.org/html/2506.13150v1#S4.T1)). It is much better than all five baselines on FMNIST in terms of test NLL and accuracy, at all rounds.
On CIFAR-10, BayesADMM is comparable to the best method in terms of NLL but significantly better than others in accuracy for both CNNs from Acar et al. [1] and Zenke et al. [63].

#### For longer runs in more challenging settings, BayesADMM significantly outperforms all other methods.

We compare BayesADMM to baselines for a 100-client highly heterogeneous MNIST split, and a 10-client heterogeneous CIFAR-100 split, in [Table 2](https://arxiv.org/html/2506.13150v1#S4.T2).
We see that BayesADMM significantly outperforms all baselines, showing the benefit of the method in heterogeneous settings with more clients and over more communication rounds.
After 100 rounds on the highly-heterogeneous split on MNIST, accuracy is improved by 0.6% and test NLL by 0.03.
After 100 rounds on CIFAR-100, accuracy is improved by 6.7% and test NLL by 0.9.

#### BayesADMM has the lowest test loss (NLL) on all datasets.

In all but one experiment in [Tables 1](https://arxiv.org/html/2506.13150v1#S4.T1) and [2](https://arxiv.org/html/2506.13150v1#S4.T2), BayesADMM(@$\mathbf{m}$) usually has the lowest test loss.
We would expect this from a Bayesian method, and we see better performance than FedLap and FedLap-Cov too.
This shows how BayesADMM is not overfitting to the data, and this often also shows as improved test accuracy.

**Table 2: Test accuracy and NLL after 25, 50 and 100 rounds, with mean and standard deviations over 3 runs. We see that BayesADMM outperforms all baselines in both scenarios.**
|  |  | Test accuracy ($\uparrow$ larger is better) | Test NLL ($\downarrow$ smaller is better) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Scenario | Method | 25 rounds | 50 rounds | 100 rounds | 25 rounds | 50 rounds | 100 rounds |
| MLP, 100 clients<br>highly heterog.<br>MNIST | FedAvg | 89.4$\pm$0.2 | 91.8$\pm$0.3 | 94.1$\pm$0.0 | 0.67$\pm$0.01 | 0.41$\pm$0.01 | 0.24$\pm$0.00 |
| FedProx | 89.6$\pm$0.4 | 91.8$\pm$0.5 | 94.0$\pm$0.4 | 0.59$\pm$0.01 | 0.36$\pm$0.01 | 0.22$\pm$0.01 |  |
| FedDyn | 89.8$\pm$0.1 | 92.6$\pm$0.2 | 94.9$\pm$0.1 | 0.60$\pm$0.01 | 0.33$\pm$0.01 | 0.18$\pm$0.00 |  |
| FedLap | 89.7$\pm$0.3 | 92.0$\pm$0.4 | 94.4$\pm$0.3 | 0.63$\pm$0.01 | 0.38$\pm$0.01 | 0.22$\pm$0.01 |  |
| FedLap-Cov | 91.0$\pm$0.4 | 92.9$\pm$0.4 | 94.9$\pm$0.3 | 0.47$\pm$0.01 | 0.29$\pm$0.01 | 0.18$\pm$0.01 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 88.8$\pm$1.5 | 93.6$\pm$0.1 | 95.6$\pm$0.1 | 0.39$\pm$0.06 | 0.22$\pm$0.01 | 0.15$\pm$0.00 |  |
| BayesADMM | 88.9$\pm$1.5 | 93.5$\pm$0.1 | 95.5$\pm$0.1 | 0.40$\pm$0.06 | 0.22$\pm$0.01 | 0.15$\pm$0.00 |  |
| ResNet-20,<br>10 clients,<br>heterog.<br>CIFAR-100 | FedAvg | 39.8$\pm$0.5 | 39.8$\pm$0.8 | 39.3$\pm$0.6 | 2.6$\pm$0.3 | 3.3$\pm$0.3 | 4.0$\pm$0.2 |
| FedProx | 37.9$\pm$0.4 | 40.4$\pm$0.9 | 39.8$\pm$0.8 | 2.4$\pm$0.0 | 2.6$\pm$0.1 | 3.4$\pm$0.1 |  |
| FedDyn | 38.4$\pm$0.5 | 39.2$\pm$0.8 | 39.6$\pm$0.4 | 3.5$\pm$0.3 | 3.2$\pm$0.2 | 3.2$\pm$0.1 |  |
| FedLap | 40.1$\pm$1.1 | 40.4$\pm$1.0 | 39.7$\pm$0.7 | 2.9$\pm$0.1 | 3.0$\pm$0.0 | 3.2$\pm$0.0 |  |
| BayesADMM@$\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ | 40.0$\pm$1.0 | 46.2$\pm$0.4 | 46.5$\pm$0.6 | 2.3$\pm$0.0 | 2.1$\pm$0.0 | 2.3$\pm$0.1 |  |
| BayesADMM | 40.0$\pm$1.0 | 46.2$\pm$0.4 | 46.6$\pm$0.6 | 2.3$\pm$0.0 | 2.1$\pm$0.0 | 2.2$\pm$0.1 |  |

#### BayesADMM has less overhead than FedLap-Cov.

BayesADMM (with IVON) has similar computation cost and runtime to FedAvg, FedProx, FedDyn and FedLap, despite estimating a (diagonal) covariance matrix, as we use the IVON optimizer.
However, FedLap-Cov requires a Laplace approximation to the (diagonal) covariance matrix, and this is computationally slow and memory-intensive: we use LaplaceRedux [13] to calculate the diagonal Hessian, but FedLap-Cov is still four times slower than FedLap on 100-client MNIST, 7 times slower on CIFAR-10, and prohibitively slow on CIFAR-100. BayesADMM with IVON does not have such overheads.

## 5 Discussion

We introduce a Bayesian Duality structure, from which an extension of ADMM that learns distributions $\bar{q}(\boldsymbol{\theta})$ instead of point estimates $\bar{\boldsymbol{\theta}}$ naturally follows. For Gaussians with fixed variance, we recover standard ADMM and general Gaussians give Newton-like methods. These show good performance when compared to recent baselines. Other approximating distributions may lead to new interesting splitting algorithms, and more generally, our work opens up new research path to extend and improve primal-dual splitting optimization algorithms using Bayesian ideas.

There are some limitations of the approach. ADMM is usually formulated for general linear (or even nonlinear) constraints, whereas here we only consider the ‘consensus’ (federated) setting. To make the method easily scalable for deep learning, we used diagonal covariances. While this performed better than federated ADMM (which uses a fixed scalar covariance) and all other baselines, more general structured covariances are expected to give larger benefits.

#### Acknowledgements.

This work is supported by JST CREST Grant Number JP-MJCR2112. This work used computational resources on the TSUBAME4.0 supercomputer provided by Institute of Science Tokyo.
This material is based upon work supported by the National Science Foundation under Grant No. IIS-2107391. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

## References

- Acar et al. [2021]
Acar, D. A. E., Zhao, Y., Matas, R., Mattina, M., Whatmough, P., and Saligrama,
V.
Federated learning based on dynamic regularization.
In *International Conference on Learning Representations
(ICLR)*, 2021.
- Adam et al. [2021]
Adam, V., Chang, P. E., Khan, M. E., and Solin, A.
Dual parameterization of sparse variational gaussian processes.
In *Advances in Neural Information Processing Systems
(NeurIPS)*, 2021.
- Al-Shedivat et al. [2021]
Al-Shedivat, M., Gillenwater, J., Xing, E., and Rostamizadeh, A.
Federated learning via posterior averaging: A new perspective and
practical algorithms.
In *International Conference on Learning Representations
(ICLR)*, 2021.
- Amari [2016]
Amari, S.
*Information Geometry and its Applications*.
Springer, 2016.
- Applegate et al. [2021]
Applegate, D., Díaz, M., Hinder, O., Lu, H., Lubin, M., O’Donoghue, B.,
and Schudy, W.
Practical large-scale linear programming using primal-dual hybrid
gradient.
In *Advances in Neural Information Processing Systems
(NeurIPS)*, 2021.
- Ashman et al. [2022]
Ashman, M., Bui, T. D., Nguyen, C. V., Markou, S., Weller, A., Swaroop, S., and
Turner, R. E.
Partitioned variational inference: A framework for probabilistic
federated learning.
*arXiv:2202.12275*, 2022.
- Babagholami-Mohamadabadi et al. [2015]
Babagholami-Mohamadabadi, B., Yoon, S., and Pavlovic, V.
D-MFVI: Distributed mean field variational inference using
Bregman ADMM.
*arXiv:1507.00824*, 2015.
- Bertsekas [2014]
Bertsekas, D. P.
*Constrained optimization and Lagrange multiplier methods*.
Academic press, 2014.
- Bot & Csetnek [2014]
Bot, R. I. and Csetnek, E. R.
An inertial alternating direction method of multipliers.
*arXiv:1404.4582*, 2014.
- Boyd et al. [2011]
Boyd, S. P., Parikh, N., Chu, E., Peleato, B., and Eckstein, J.
Distributed optimization and statistical learning via the alternating
direction method of multipliers.
*Found. Trends Mach. Learn.*, 3(1):1–122,
2011.
- Chambolle & Pock [2016]
Chambolle, A. and Pock, T.
On the ergodic convergence rates of a first-order primal–dual
algorithm.
*Math. Progam.*, 159(1-2):253–287, 2016.
- Chen et al. [2015]
Chen, C., Chan, R. H., Ma, S., and Yang, J.
Inertial proximal ADMM for linearly constrained separable convex
optimization.
*SIAM J. Imaging Sci.*, 8(4):2239–2267,
2015.
- Daxberger et al. [2021]
Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and
Hennig, P.
Laplace redux–effortless Bayesian deep learning.
In *Advances in Neural Information Processing Systems
(NeurIPS)*, 2021.
- Dempster et al. [1977]
Dempster, A. P., Laird, N. M., and Rubin, D. B.
Maximum likelihood from incomplete data via the EM algorithm.
*J. R. Stat. Soc. Ser. B Methodol.*, 39(1):1–22, 1977.
- Durrant-Whyte [2001]
Durrant-Whyte, H.
Data fusion in decentralised sensing networks.
In *International Conference on Information Fusion*, 2001.
- Eckstein & Bertsekas [1992]
Eckstein, J. and Bertsekas, D. P.
On the Douglas-Rachford splitting method and the proximal point
algorithm for maximal monotone operators.
*Math. Progam.*, 55(1):293–318, 1992.
- Eckstein & Ferris [1999]
Eckstein, J. and Ferris, M. C.
Smooth methods of multipliers for complementarity problems.
*Math. Progam.*, 86(1):65–90, 1999.
- Fan et al. [2024]
Fan, Z., Hu, S., Yao, J., Niu, G., Zhang, Y., Sugiyama, M., and Wang, Y.
Locally estimated global perturbations are better than local
perturbations for federated sharpness-aware minimization.
In *International Conference on Machine Learning (ICML)*, 2024.
- Foret et al. [2021]
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B.
Sharpness-aware minimization for efficiently improving
generalization.
In *International Conference on Learning Representations
(ICLR)*, 2021.
- Fougner & Boyd [2018]
Fougner, C. and Boyd, S.
Parameter selection and preconditioning for a graph form solver.
In *Emerging Applications of Control and Systems Theory*, pp. 41–61. 2018.
- Gabay & Mercier [1976]
Gabay, D. and Mercier, B.
A dual algorithm for the solution of nonlinear variational problems
via finite element approximation.
*Computers & Mathematics with Applications*, 2(1):17–40, 1976.
- Glowinski & Marroco [1975]
Glowinski, R. and Marroco, A.
Sur l’approximation, par éléments finis d’ordre un, et la
résolution, par pénalisation-dualité d’une classe de
problèmes de Dirichlet non linéaires.
*Revue française d’automatique, informatique, recherche
opérationnelle. Analyse numérique*, 9(R2):41–76,
1975.
- Goldstein et al. [2013]
Goldstein, T., Li, M., Yuan, X., Esser, E., and Baraniuk, R.
Adaptive primal-dual hybrid gradient methods for saddle-point
problems.
*arXiv:1305.0546*, 2013.
- Gong et al. [2022]
Gong, Y., Li, Y., and Freris, N. M.
FedADMM: A robust federated deep learning framework with adaptivity
to system heterogeneity.
In *International Conference on Data Engineering (ICDE)*, 2022.
- Guo et al. [2023]
Guo, H., Greengard, P., Wang, H., Gelman, A., Kim, Y., and Xing, E.
Federated learning as variational inference: A scalable expectation
propagation approach.
In *International Conference on Learning Representations
(ICLR)*, 2023.
- He et al. [2016]
He, K., Zhang, X., Ren, S., and Sun, J.
Deep residual learning for image recognition.
In *IEEE Conference on Computer Vision and Pattern Recognition
(CVPR)*, 2016.
- Jaynes [1957]
Jaynes, E. T.
Information theory and statistical mechanics.
*Phys. Rev.*, 106(4):620, 1957.
- Kalman [1960]
Kalman, R. E.
A new approach to linear filtering and prediction problems.
*Transactions of the ASME – Journal of Basic Engineering*,
82(Series D):35–45, 1960.
- Khan [2014]
Khan, M. E.
Decoupled variational Gaussian inference.
In *Advances in Neural Information Processing Systems
(NeurIPS)*, 2014.
- Khan & Lin [2017]
Khan, M. E. and Lin, W.
Conjugate-computation variational inference: Converting variational
inference in non-conjugate models to inferences in conjugate models.
In *International Conference on Artificial Intelligence and
Statistics (AISTATS)*, 2017.
- Khan & Rue [2023]
Khan, M. E. and Rue, H.
The Bayesian learning rule.
*J. Mach. Learn. Res. (JMLR)*, 24(281):1–46, 2023.
- Khan et al. [2013]
Khan, M. E., Aravkin, A., Friedlander, M., and Seeger, M.
Fast dual variational inference for non-conjugate latent Gaussian
models.
In *International Conference on Machine Learning (ICML)*, 2013.
- Koller & Friedman [2009]
Koller, D. and Friedman, N.
*Probabilistic graphical models: principles and techniques*.
MIT press, 2009.
- Kotelevskii et al. [2022]
Kotelevskii, N., Vono, M., Durmus, A., and Moulines, E.
FedPop: A Bayesian approach for personalised federated learning.
In *Advances in Neural Information Processing Systems
(NeurIPS)*, 2022.
- Lauritzen & Spiegelhalter [1988]
Lauritzen, S. L. and Spiegelhalter, D. J.
Local computations with probabilities on graphical structures and
their application to expert systems.
*J. R. Stat. Soc. Ser. B Methodol.*, 50(2):157–194, 1988.
- Li et al. [2020]
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V.
Federated optimization in heterogeneous networks.
In *Proceedings of Machine Learning and Systems*, 2020.
- Louizos et al. [2021]
Louizos, C., Reisser, M., Soriaga, J., and Welling, M.
An expectation-maximization perspective on federated learning.
*arXiv:2111.10192*, 2021.
- Luque [1986]
Luque, J.
The nonlinear poximal point algorithm and multiplier methods.
Laboratory for Information and Decision Systems, MIT, 1986.
- McMahan et al. [2016]
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A.
Communication-efficient learning of deep networks from decentralized
data.
In *International Conference on Artificial Intelligence and
Statistics (AISTATS)*, 2016.
- Mishchenko et al. [2022]
Mishchenko, K., Malinovsky, G., Stich, S., and Richtárik, P.
ProxSkip: Yes! local gradient steps provably lead to communication
acceleration! finally!
In *International Conference on Machine Learning (ICML)*, 2022.
- Möllenhoff & Khan [2023]
Möllenhoff, T. and Khan, M. E.
SAM as an optimal relaxation of Bayes.
In *International Conference on Learning Representations
(ICLR)*, 2023.
- Mutambara [1998]
Mutambara, A. G. O.
*Decentralized estimation and control for multisensor systems*.
Routledge, 1998.
- Nielsen & Garcia [2009]
Nielsen, F. and Garcia, V.
Statistical exponential families: A digest with flash cards.
*arXiv:0911.4863*, 2009.
- Nodozi & Halder [2023]
Nodozi, I. and Halder, A.
Wasserstein consensus ADMM.
*arXiv:2309.07351*, 2023.
- Oikonomidis et al. [2023]
Oikonomidis, K. A., Bodard, A., Laude, E., and Patrinos, P.
Global convergence analysis of the power proximal point and augmented
lagrangian method.
*arXiv:2312.12205*, 2023.
- Pal et al. [2024]
Pal, S., Gupta, A., Sarwar, S., and Rai, P.
Simple and scalable federated learning with uncertainty via
Improved Variational Online Newton.
In *NeurIPS Workshop on Optimization for Machine Learning
(OPT)*, 2024.
- Parikh & Boyd [2013]
Parikh, N. and Boyd, S.
Proximal algorithms.
*Foundations and Trends in Optimization*, 1:123–231,
2013.
- Pock & Chambolle [2011]
Pock, T. and Chambolle, A.
Diagonal preconditioning for first order primal-dual algorithms in
convex optimization.
In *International Conference on Computer Vision (ICCV)*, 2011.
- Qu et al. [2022]
Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Z.
Generalized federated learning via sharpness aware minimization.
In *International Conference on Machine Learning (ICML)*, 2022.
- Rabiner & Juang [1986]
Rabiner, L. and Juang, B.
An introduction to hidden Markov models.
*IEEE ASSP Magazine*, 3(1):4–16, January
1986.
- Rezende et al. [2014]
Rezende, D. J., Mohamed, S., and Wierstra, D.
Stochastic backpropagation and approximate inference in deep
generative models.
In *International Conference on Machine Learning (ICML)*, 2014.
- Rockafellar [1967]
Rockafellar, R. T.
Duality and stability in extremum problems involving convex
functions.
*Pacific J. Math.*, 21(1):167–187, 1967.
- Shen et al. [2024]
Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Bazan, C., Yokota,
R., Gurevych, I., Cremers, D., Khan, M. E., and Möllenhoff, T.
Variational learning is effective for large deep networks.
In *International Conference on Machine Learning (ICML)*, 2024.
- Swaroop et al. [2025]
Swaroop, S., Khan, M. E., and Doshi-Velez, F.
Connecting federated ADMM to Bayes.
In *International Conference on Learning Representations
(ICLR)*, 2025.
- Tseng [1991]
Tseng, P.
Applications of a splitting algorithm to decomposition in convex
programming and variational inequalities.
*SIAM J. Control Optim.*, 29(1):119–138,
1991.
- Wainwright et al. [2008]
Wainwright, M. J., Jordan, M. I., et al.
Graphical models, exponential families, and variational inference.
*Foundations and Trends® in Machine Learning*,
1(1–2):1–305, 2008.
- Wang et al. [2014]
Wang, F., Xu, Z., and Xu, H.-K.
Convergence of Bregman alternating direction method with
multipliers for nonconvex composite problems.
*arXiv:1410.8625*, 2014.
- Wang & Banerjee [2014]
Wang, H. and Banerjee, A.
Bregman alternating direction method of multipliers.
*Advances in Neural Information Processing Systems (NeurIPS)*,
2014.
- Wang et al. [2022]
Wang, H., Marella, S., and Anderson, J.
FedADMM: A federated primal-dual algorithm allowing partial
participation.
*Conference on Decision and Control (CDC)*, 2022.
- Winn et al. [2005]
Winn, J., Bishop, C. M., and Jaakkola, T.
Variational message passing.
*J. Mach. Learn. Res. (JMLR)*, 6(4), 2005.
- Ye et al. [2020]
Ye, Z., Möllenhoff, T., Wu, T., and Cremers, D.
Optimization of graph total variation via active-set-based
combinatorial reconditioning.
In *International Conference on Artificial Intelligence and
Statistics (AISTATS)*, 2020.
- Yurochkin et al. [2019]
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., and Khazaeni,
Y.
Bayesian nonparametric federated learning of neural networks.
In *International Conference on Machine Learning (ICML)*, 2019.
- Zenke et al. [2017]
Zenke, F., Poole, B., and Ganguli, S.
Continual learning through synaptic intelligence.
In *International Conference on Machine Learning (ICML)*, 2017.
- Zhang et al. [2021]
Zhang, X., Hong, M., Dhople, S., Yin, W., and Liu, Y.
FedPD: A federated learning framework with adaptivity to non-iid
data.
*IEEE Trans. Signal Process.*, 69:6055–6070, 2021.
- Zhou & Li [2023]
Zhou, S. and Li, G. Y.
Federated learning via inexact ADMM.
*IEEE Trans. Pattern Anal. Mach. Intell. (PAMI)*, 45(8):9699–9708, 2023.

## Appendix A Partitioned Variational Inference (PVI)

Here, we discuss the difference between BayesADMM and PVI. Unlike PVI, BayesADMM in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) uses the learning rate $\rho$ in all three updates ($\alpha$ is also derived from $\rho$), whereas PVI does not use any learning rates in its raw form [6]. The learning rate is essential for convergence and it is well-known that splitting algorithms can diverge if learning-rates are not carefully chosen [55, Eq. 3.4d]. To get a method that works well in practice, Swaroop et al. [54] used a learning rate $\rho$ in the dual update with a justification of damping (similarly to Ashman et al. [6]) but they did not use it in the update of $q_{k}$.

To see the above, we rewrite PVI as introduced in Ashman et al. [6, Alg. 1], bringing it closer to BayesADMM. The first step is a local update of the distribution $q_{k}$ which takes the following form,

$$ $\displaystyle q_{k}$ $\displaystyle\leftarrow\operatorname{argmax}_{q\in\mathcal{Q}}~{}\int q( \boldsymbol{\theta})\log\frac{\bar{q}(\boldsymbol{\theta})p(\mbox{$\mbox{$ \mathbf{y}$}$}_{k}|\boldsymbol{\theta})}{q(\boldsymbol{\theta})\hat{t}_{k}( \boldsymbol{\theta})}\mathrm{d}\boldsymbol{\theta}$ (13) $\displaystyle=\operatorname{argmin}_{q\in\mathcal{Q}}~{}\mathbb{E}_{q}[\ell_{k }(\boldsymbol{\theta})+\log\hat{t}_{k}(\boldsymbol{\theta})]+{\operatorname{KL }}{}\left(q\,\|\,\bar{q}\right){},$ (14) $$

where for the equality step, we switched from maximization to minimization and replaced the likelihood with a loss $\ell_{k}$. We can see that the second step for $\log\hat{t}_{k}(\boldsymbol{\theta})=\langle\widehat{\boldsymbol{\lambda}}_{k}
,\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})\rangle$ matches line 1 in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2), but as discussed above, there is no step-size $\rho$ in front of the KL-term. $\hat{t}_{k}(\boldsymbol{\theta})$ are so-called site-functions [6] which aim to approximate the likelihood, but as we can see, they play a similar rule to Lagrange multipliers.
Ashman et al. [6, Alg. 1] then update $\hat{t}_{k}$ as follows:

$$ $\displaystyle\hat{t}_{k}(\boldsymbol{\theta})\leftarrow\frac{q_{k}(\boldsymbol {\theta})}{\bar{q}(\boldsymbol{\theta})}\hat{t}_{k}(\boldsymbol{\theta})\quad \implies\quad\log\hat{t}_{k}(\boldsymbol{\theta})\leftarrow\log\hat{t}_{k}( \boldsymbol{\theta})+(\log q_{k}(\boldsymbol{\theta})-\log\bar{q}(\boldsymbol{ \theta})).$ (15) $$

Taking the log, we can see that this is the dual update step in line 2 of [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2), since we can write the update purely in terms of $\widehat{\boldsymbol{\lambda}}_{k}$, $\boldsymbol{\lambda}_{k}$ and $\bar{\boldsymbol{\lambda}}$. But again, there is no step-size. Ashman et al. [6] also consider a damping factor, which then recovers exactly line 2 and matches [Eq. 9](https://arxiv.org/html/2506.13150v1#S3.E9).

Finally, the server update in Ashman et al. [6, Alg. 1] is given by,

$$ $\displaystyle\bar{q}^{(t)}(\boldsymbol{\theta})$ $\displaystyle\propto\bar{q}^{(t-1)}(\boldsymbol{\theta})\prod_{k=1}^{K}\frac{ \hat{t}_{k}^{(t)}(\boldsymbol{\theta})}{\hat{t}_{k}^{(t-1)}(\boldsymbol{\theta })}\propto\pi(\boldsymbol{\theta})\prod_{k=1}^{K}\hat{t}_{k}^{(t)}(\boldsymbol {\theta}),$ (16) $$

where the last step holds when $\hat{t}_{k}^{(0)}(\boldsymbol{\theta})=1$ and $\bar{q}^{(0)}(\boldsymbol{\theta})=\pi(\boldsymbol{\theta})$, which is the initialization condition in Ashman et al. [6, Alg. 1]. The last step can be seen by recursively applying the definition of $\bar{q}$ and noticing that successive terms in the product cancel out in a telescoping fashion.
Again, taking the log, this recovers line 3 in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) when $\alpha=1$.

The use of $\alpha=1$ in the server update of PVI makes the method more similar to the Alternating Minimization Algorithm (AMA) of Tseng [55] than ADMM which uses a general $\alpha$ (derived from $\rho$). AMA does not use a proximal-term on the server, which is similar to the PVI update. However, AMA also uses a step-size in the primal and dual updates, whereas PVI only uses a step-size in the dual (when damping is enabled). From the convergence proof of AMA, it is clear that step-sizes are required to obtain a convergent method, see [55, Eq. 3.4d].

Finally, we connect PVI with damping to the Bayesian learning rule.
Taking the optimality condition of [Eq. 14](https://arxiv.org/html/2506.13150v1#A1.E14) for an exponential-family $q$ and inserting it into the dual step [Eq. 15](https://arxiv.org/html/2506.13150v1#A1.E15) with damping, we get:

$$ $\widehat{\boldsymbol{\lambda}}_{k}\leftarrow(1-\rho)\widehat{\boldsymbol{ \lambda}}_{k}+\rho\nabla_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}\mathbb{E} _{q_{k}}[\ell_{k}]\bigl{|}_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}=\text{$ \mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}$}}.$ (17) $$

This is a reformulation of the Bayesian learning rule [31] in terms of local parameters, see Khan & Rue [31, Eq. 60] (see also Khan & Lin [30]), except that the natural gradients are evaluated at $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}$ instead of $\boldsymbol{\mu}$.

**Table 3: A summary of exponential-families used in the paper showing natural parameters $\boldsymbol{\lambda}$, sufficient statistics $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$, and expectation parameters $\boldsymbol{\mu}$, reproduced from [31, Table 2].**
| Distribution | $\boldsymbol{\lambda}$ | $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$ | $\boldsymbol{\mu}$ | Resulting Method |
| --- | --- | --- | --- | --- |
| $\mbox{${\cal N}$}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$<br>\mbox{$\mathbf{I}$}$})$ | $\mathbf{m}$ | $\boldsymbol{\theta}$ | $\mathbf{m}$ | ADMM |
| $\mbox{${\cal N}$}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$<br>\mbox{$\mathbf{S}$}$}^{-1})$, fixed $\mathbf{S}$ | $\mathbf{S}$$\mathbf{m}$ | $\boldsymbol{\theta}$ | $\mathbf{m}$ | Preconditioned<br>ADMM |
| $\mbox{${\cal N}$}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$<br>\mbox{$\mathbf{S}$}$}^{-1})$, $\mbox{$\mbox{$\mathbf{S}$}$}=\mbox{$\mbox{diag}$}(\mbox{$\mbox{$\mathbf{s}$}$})$ | $\left(\begin{array}[]{c}\mbox{$\mbox{$\mathbf{m}$}$}\\<br>-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{s}$}$}\end{array}\right)$ | $\left(\begin{array}[]{c}\boldsymbol{\theta}\\<br>\boldsymbol{\theta}^{2}\end{array}\right)$ | $\left(\begin{array}[]{c}\mbox{$\mbox{$\mathbf{m}$}$}\\<br>\mbox{$\mbox{$\mathbf{m}$}$}^{2}+1/\mbox{$\mbox{$\mathbf{s}$}$}\end{array}\right)$ | [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) |
| $\mbox{${\cal N}$}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$<br>\mbox{$\mathbf{S}$}$}^{-1})$ | $\left(\begin{array}[]{c}\mbox{$\mbox{$\mathbf{m}$}$}\\<br>-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{S}$}$}\end{array}\right)$ | $\left(\begin{array}[]{c}\boldsymbol{\theta}\\<br>\boldsymbol{\theta}\boldsymbol{\theta}^{\top}\end{array}\right)$ | $\left(\begin{array}[]{c}\mbox{$\mbox{$\mathbf{m}$}$}\\<br>\mbox{$\mbox{$\mathbf{m}$}$}\mbox{$\mbox{$\mathbf{m}$}$}^{\top}+\mbox{$\mbox{$<br>\mathbf{S}$}$}^{-1}\end{array}\right)$ | [Sec. C.4](https://arxiv.org/html/2506.13150v1#A3.SS4) |

## Appendix B Background on Exponential Families

Many commonly used distributions such as Gaussians, Categorical or Gamma belong to an exponential family.
Viewing them as such provides us with a rich dual structure which we exploint in this paper. [Table 3](https://arxiv.org/html/2506.13150v1#A1.T3) provides an overview over all exponential families used in this paper.
Here, we give a short introduction and refer the interested reader to Wainwright et al. [56, Sec. 3] for more details and a rigorous treatment. Following Jaynes [27], we introduce exponential families as maximum-entropy solutions of a constrained optimization problem. We assume a base measure $h(\boldsymbol{\theta})$ and then search for a distribution $q$ in the space of all possible distributions which has maximum entropy (relative to $h$) and whose expectation under sufficient statistics $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$ matches some given moments $\boldsymbol{\mu}$. There are many distributions which share the same moments $\boldsymbol{\mu}$, but only one has the maximum entropy (see [Fig. 5a](https://arxiv.org/html/2506.13150v1#A2.F5.sf1) for an example with uniform base-measure $h(\boldsymbol{\theta})=1$). The maximum entropy distribution is given as the solution to the following optimization problem:

$$ $q_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}(\boldsymbol{\theta})= \operatorname{argmin}_{q(\boldsymbol{\theta})}~{}{\operatorname{KL}}{}\left(q( \boldsymbol{\theta})\,\|\,h(\boldsymbol{\theta})\right){}\,\text{ s.t. }\, \mathbb{E}_{q}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})]=\mbox{$\mbox {$\boldsymbol{\mu}$}$},\,q(\boldsymbol{\theta})\geq 0,\,\int q(\boldsymbol{ \theta})\mathrm{d}\boldsymbol{\theta}=1.$ (18) $$

Similar as in Jaynes [27, Sec. 2], introducing Lagrange multipliers $\boldsymbol{\lambda}$ for the moment constraints and another Lagrange multiplier $\alpha$ for the normalization constraint, we get the Lagrangian,

$$ $\mathcal{L}(q,\boldsymbol{\lambda},\alpha)={\operatorname{KL}}{}\left(q( \boldsymbol{\theta})\,\|\,h(\boldsymbol{\theta})\right){}+\langle\boldsymbol{ \lambda},\mathbb{E}_{q}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})]- \mbox{$\mbox{$\boldsymbol{\mu}$}$}\rangle+\alpha\left(1-\int q(\boldsymbol{ \theta})\mathrm{d}\boldsymbol{\theta}\right).$ (19) $$

Taking the first variation of the Lagrangian with respect to $q$ and setting it to zero, we arrive at,

$$ $q_{\text{\mbox{$\mbox{$\boldsymbol{\mu}$}$}}}(\boldsymbol{\theta})=h( \boldsymbol{\theta})\exp(\langle\boldsymbol{\lambda},\mbox{$\mbox{$\mathbf{T}$ }$}(\boldsymbol{\theta})\rangle-\alpha),$ (20) $$

which for $\alpha=A(\boldsymbol{\lambda})=\log\int\exp(\langle\boldsymbol{\lambda},\mbox{
$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})\rangle)h(\boldsymbol{\theta})
\mathrm{d}\boldsymbol{\theta}$ is the definition of the exponential family as provided in the main text in [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3), where we did set $h(\boldsymbol{\theta})=1$. We also see that the Lagrange multipliers $\boldsymbol{\lambda}$ are the natural parameters of the exponential family.

The Lagrangian we used in the main text to derive our BayesADMM method closely follows this dual structure of the exponential family. The above derivation provides an additional motivation why one should use the natural parameters $\boldsymbol{\lambda}$ in the dual update of our method in [Eq. 9](https://arxiv.org/html/2506.13150v1#S3.E9): They live in the same space of Lagrange multipliers to expectation constraints.

As mentioned in the main text, Lagrange multipliers $\boldsymbol{\lambda}$ and moments $\boldsymbol{\mu}$ form a dual coordinate system, where $\mbox{$\mbox{$\boldsymbol{\mu}$}$}=\nabla A(\boldsymbol{\lambda})=\mathbb{E}_{
q}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})]$. The map $\nabla A:\Omega\to\mathcal{M}$ maps from the space of valid natural parameters $\Omega=\{\boldsymbol{\lambda}:A(\boldsymbol{\lambda})<\infty\}$ to the space of valid moments $\mathcal{M}=\{\mathbb{E}_{q}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})
]:q\in\mathcal{Q}\}$. The inverse is given by $\nabla A^{*}:\mathcal{M}\to\Omega$, where $A^{*}$ denotes the convex conjugate of $A$. For an illustration of the two spaces $\Omega$ and $\mathcal{M}$ on the example of Gaussian distributions, see [Fig. 5b](https://arxiv.org/html/2506.13150v1#A2.F5.sf2).

Figure: a Maximum Entropy Distribution
Refer to caption: x5.png

## Appendix C Details about the Derivations in Sec. 3.2 and 3.3

Here, we provide details about the derivations of BayesADMM with Gaussians in [Sec. 3.2](https://arxiv.org/html/2506.13150v1#S3.SS2) and [3.3](https://arxiv.org/html/2506.13150v1#S3.SS3).

### C.1 Some Background on Federated ADMM

For reader’s convenience, we provide a short derivation of federated ADMM as shown in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2). The method is a standard application of ADMM to the consensus problems with regularization in [Eq. 1](https://arxiv.org/html/2506.13150v1#S2.E1), see Boyd et al. [10, Section 7.1.1]. Compared to the algorithm presented by Boyd et al. [10, Section 7.1.1], we reorder the variables so that the dual update is performed directly after the client step. The reordering is still a valid instance of ADMM and falls within the reach of standard theory (the role of the two functions in the splitting can be reversed). The updates are as follows:

$$ $\displaystyle\boldsymbol{\theta}_{k}\leftarrow\operatorname{argmin}_{ \boldsymbol{\theta}_{k}}\,\ell_{k}(\boldsymbol{\theta}_{k})+\mbox{$\mbox{$ \mathbf{v}$}$}_{k}^{\top}\boldsymbol{\theta}_{k}+\smash{\frac{\rho}{2}\| \boldsymbol{\theta}_{k}-\bar{\boldsymbol{\theta}}\|^{2},}$ (21) $\displaystyle\mbox{$\mbox{$\mathbf{v}$}$}_{k}\leftarrow\mbox{$\mbox{$\mathbf{v }$}$}_{k}+\rho(\boldsymbol{\theta}_{k}-\bar{\boldsymbol{\theta}}),$ (22) $\displaystyle\bar{\boldsymbol{\theta}}\leftarrow\operatorname{argmin}_{\bar{ \boldsymbol{\theta}}}\,\ell_{0}(\bar{\boldsymbol{\theta}})+\textstyle\sum_{k=1 }^{K}\left(\frac{\rho}{2}\|\bar{\boldsymbol{\theta}}-\boldsymbol{\theta}_{k}\| ^{2}-\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\bar{\boldsymbol{\theta}}\right).$ (23) $$

The first two steps are the same as in Federated ADMM as shown in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2).

To see how the third step matches the one in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) we use $\ell_{0}(\boldsymbol{\theta})=\mbox{$\frac{1}{2}$}\|\boldsymbol{\theta}\|^{2}$. Then, the server $\bar{\boldsymbol{\theta}}$-update has a simple solution:

$$ $\displaystyle\bar{\boldsymbol{\theta}}+\sum_{k=1}^{K}\left(\rho(\bar{ \boldsymbol{\theta}}-\boldsymbol{\theta}_{k})-\mbox{$\mbox{$\mathbf{v}$}$}_{k} \right)=0\quad\implies\quad(1+K\rho)\bar{\boldsymbol{\theta}}=\frac{1}{K}\sum_ {k=1}^{K}K\rho\boldsymbol{\theta}_{k}+\sum_{k=1}^{K}\mbox{$\mbox{$\mathbf{v}$} $}_{k}.$ (24) $$

Dividing the above equation by $1+K\rho$ we get,

$$ $\displaystyle\bar{\boldsymbol{\theta}}=(1-\alpha)\,\text{\tt{Mean}}( \boldsymbol{\theta}_{1:K})+\alpha\,\text{\tt{Sum}}(\mbox{$\mbox{$\mathbf{v}$}$ }_{1:K}),$ (25) $$

where $\alpha=1/(1+K\rho)$. This is the third step in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2).

### C.2 Recovering Federated ADMM

As mentioned in [Sec. 3.2](https://arxiv.org/html/2506.13150v1#S3.SS2), we recover federated ADMM when using Gaussians with fixed covariance.
These form an exponential family as described in [Eq. 10](https://arxiv.org/html/2506.13150v1#S3.E10), see also [Table 3](https://arxiv.org/html/2506.13150v1#A1.T3).
From [Eq. 10](https://arxiv.org/html/2506.13150v1#S3.E10), we see that the natural gradients are simply an average of the regular gradients. We denote $\widehat{\boldsymbol{\lambda}}_{k}=\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ since they play a similar role. Using these definitions, line 1 in BayesADMM can be written as follows:

$$ $\displaystyle\mbox{$\mbox{$\mathbf{m}$}$}_{k}$ $\displaystyle\leftarrow\operatorname{argmin}_{\text{$\mbox{$\mbox{$\mathbf{m}$ }$}_{k}$}}\,\mathbb{E}_{\text{$\mbox{${\cal N}$}(\boldsymbol{\theta}|\mbox{$ \mbox{$\mathbf{m}$}$}_{k},\mbox{$\mbox{$\mathbf{I}$}$})$}}[\ell_{k}( \boldsymbol{\theta})+\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$ \mathbf{T}$}$}(\boldsymbol{\theta})\rangle]+\smash{\rho{{\operatorname{KL}}{} \left(q_{k}\,\|\,\bar{q}\right){}}}$ (26) $\displaystyle=\operatorname{argmin}_{\text{$\mbox{$\mbox{$\mathbf{m}$}$}_{k}$} }\,\mathbb{E}_{\text{$\mbox{${\cal N}$}(\boldsymbol{\theta}|\mbox{$\mbox{$ \mathbf{m}$}$}_{k},\mbox{$\mbox{$\mathbf{I}$}$})$}}[\ell_{k}(\boldsymbol{ \theta})]+\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\mbox{$\mbox{$\mathbf{m}$}$}_ {k}+\frac{\rho}{2}\|\mbox{$\mbox{$\mathbf{m}$}$}_{k}-\bar{}\mbox{$\mbox{$ \mathbf{m}$}$}\|^{2}.$ (27) $$

To go from [Eq. 26](https://arxiv.org/html/2506.13150v1#A3.E26) to [Eq. 27](https://arxiv.org/html/2506.13150v1#A3.E27) we used $\mathbb{E}_{q_{k}}[\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$
\mathbf{T}$}$}(\boldsymbol{\theta})\rangle]=\langle\widehat{\boldsymbol{
\lambda}}_{k},\mathbb{E}_{q_{k}}[\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{
\theta})]\rangle=\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$
\boldsymbol{\mu}$}$}_{k}\rangle=\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\mbox{$
\mbox{$\mathbf{m}$}$}_{k}$ and that the KL between two Gaussians is simply the squared Euclidean norm. [Eq. 27](https://arxiv.org/html/2506.13150v1#A3.E27) then recovers [Eq. 11](https://arxiv.org/html/2506.13150v1#S3.E11) from the main text.
The dual update in line 2 directly follows by $\widehat{\boldsymbol{\lambda}}_{k}=\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, $\boldsymbol{\lambda}_{k}=\mbox{$\mbox{$\mathbf{m}$}$}_{k}$, $\bar{\boldsymbol{\lambda}}=\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$.
For the server update, let us first assume that $\ell_{0}(\boldsymbol{\theta})=\mbox{$\frac{1}{2}$}\|\boldsymbol{\theta}\|^{2}$. Again by the above definitions, we see that line 3 in BayesADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) is the same as line 3 in federated ADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)).

### C.3 Derivation of Alg. 1 by Using Gaussians with Diagonal Covariances

Plugging the definitions of a diagonal Gaussian as an EF in [Eq. 12](https://arxiv.org/html/2506.13150v1#S3.E12) (see [Table 3](https://arxiv.org/html/2506.13150v1#A1.T3)) into the BayesADMM updates in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) is straightforward and mechanical calculation. We denote the dual variables as $\widehat{\boldsymbol{\lambda}}_{k}=(\mbox{$\mbox{$\mathbf{v}$}$}_{k},-\mbox{$
\frac{1}{2}$}\mbox{$\mbox{$\mathbf{u}$}$}_{k})$ to match the structure of the natural parameter $\boldsymbol{\lambda}_{k}=(\mbox{$\mbox{$\mathbf{s}$}$}_{k}\mbox{$\mbox{$
\mathbf{m}$}$}_{k},-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{s}$}$}_{k})$ which is also a Lagrange multiplier (see [App. B](https://arxiv.org/html/2506.13150v1#A2)). Unlike $\boldsymbol{\lambda}_{k}$ which belongs to $\Omega$, $\widehat{\boldsymbol{\lambda}}_{k}$ can be arbitrary (and also zero), so we do not use $\mbox{$\mbox{$\mathbf{u}$}$}_{k}\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, but rather just $\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ for the first argument.

By $\langle\widehat{\boldsymbol{\lambda}}_{k},\mbox{$\mbox{$\mathbf{T}$}$}(
\boldsymbol{\theta})\rangle=\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\boldsymbol
{\theta}-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{u}$}$}_{k}^{\top}(
\boldsymbol{\theta})^{2}=\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\boldsymbol{
\theta}-\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}(\mbox{$\mbox{$\mathbf{u
}$}$}_{k}\boldsymbol{\theta})$, we get for line 1 in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2),

$$ $\displaystyle\mbox{$\mbox{$\mathbf{m}$}$}_{k},\mbox{$\mbox{$\mathbf{s}$}$}_{k}$ $\displaystyle\leftarrow\operatorname{argmin}_{\text{$\mbox{$\mbox{$\mathbf{m}$ }$}_{k}$},\text{$\mbox{$\mbox{$\mathbf{s}$}$}_{k}$}}\,\mathbb{E}_{q_{k}}[\ell_ {k}(\boldsymbol{\theta})+\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\boldsymbol{ \theta}-\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}(\mbox{$\mbox{$\mathbf{u }$}$}_{k}\boldsymbol{\theta})]+\rho{\operatorname{KL}}{}\left(q_{k}\,\|\,\bar{ q}\right){},$ (28) $$

where the KL-divergence is given by:

$$ $\displaystyle{\operatorname{KL}}{}\left(q_{k}\,\|\,\bar{q}\right){}=\mbox{$ \frac{1}{2}$}\left(\sum_{i=1}^{P}\left[\frac{\bar{}\mbox{$\mbox{$\mathbf{s}$}$ }^{i}}{\mbox{$\mbox{$\mathbf{s}$}$}_{k}^{i}}+\log\mbox{$\mbox{$\mathbf{s}$}$}_ {i}\right]+\|\mbox{$\mbox{$\mathbf{m}$}$}-\bar{}\mbox{$\mbox{$\mathbf{m}$}$}\| _{\text{$\bar{}\mbox{$\mbox{$\mathbf{s}$}$}$}}^{2}\right)+\text{const.}$ (29) $$

Line 4 in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) solves [Eq. 28](https://arxiv.org/html/2506.13150v1#A3.E28) using IVON [53]. Here, we modify IVON to handle a general prior and the two Lagrange multipliers. The resulting method is shown in [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) with modifications over Shen et al. [53] highlighted it red. [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) solves the following objective function,

$$ $\lambda\mathbb{E}_{q}\left[\ell(\boldsymbol{\theta})\!+\!\mbox{$\mbox{$\mathbf {v}$}$}^{\top}\boldsymbol{\theta}\!-\!\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^ {\top}(\mbox{$\mbox{$\mathbf{u}$}$}\boldsymbol{\theta})\right]\!+\!{ \operatorname{KL}}{}\left(q\,\|\,p\right){},$ (30) $$

where $\ell(\boldsymbol{\theta})$ is a generic loss and $\hat{\nabla}\ell(\boldsymbol{\theta})$ denotes a stochastic gradient. The objective [Eq. 30](https://arxiv.org/html/2506.13150v1#A3.E30) matches the subproblem in [Eq. 28](https://arxiv.org/html/2506.13150v1#A3.E28). We provide further details on how to use [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) in practice in [App. D](https://arxiv.org/html/2506.13150v1#A4).

Figure: Algorithm 2 IVON for minimizing [Eq. 30](https://arxiv.org/html/2506.13150v1#A3.E30). Hyperparameters $h_{0}$, $\alpha_{t}$, $\beta_{1}$, $\beta_{2}$ are chosen following Shen et al. [53, Appendix A].

### C.4 BayesADMM with Full Covariances

The derivation with full covariances mirrors the one for diagonal covariances. We have the following forms for the posterior distributions

$$ $\displaystyle\bar{q}(\boldsymbol{\theta})=\mathcal{N}(\boldsymbol{\theta}\,|\, \mbox{$\mbox{$\mathbf{m}$}$},\mbox{$\mbox{$\mathbf{S}$}$}^{-1}),\quad q_{k}( \boldsymbol{\theta})=\mathcal{N}(\boldsymbol{\theta}\,|\,\mbox{$\mbox{$\mathbf {m}$}$}_{k},\mbox{$\mbox{$\mathbf{S}$}$}_{k}^{-1}).$ (31) $$

These fit our setup (see [Table 3](https://arxiv.org/html/2506.13150v1#A1.T3)) through,

$$ $\displaystyle\boldsymbol{\lambda}=(\mbox{$\mbox{$\mathbf{S}$}$}\mbox{$\mbox{$ \mathbf{m}$}$},-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{S}$}$}),\,\,\,\mbox{ $\mbox{$\boldsymbol{\mu}$}$}=(\mbox{$\mbox{$\mathbf{m}$}$},\mbox{$\mbox{$ \mathbf{m}$}$}\mbox{$\mbox{$\mathbf{m}$}$}^{\top}+\mbox{$\mbox{$\mathbf{S}$}$} ^{-1}),\,\,\,\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})=(\boldsymbol{ \theta},\boldsymbol{\theta}\boldsymbol{\theta}^{\top}),$ (32) $\displaystyle\nabla_{\text{$\mbox{$\mbox{$\boldsymbol{\mu}$}$}$}}\mathbb{E}_{q _{k}}[\ell_{k}]=\left(\mbox{$\mbox{$\mathbf{g}$}$}_{k}-\mbox{$\mbox{$\mathbf{H }$}$}_{k}\mbox{$\mbox{$\mathbf{m}$}$}_{k},\mbox{$\frac{1}{2}$}\mbox{$\mbox{$ \mathbf{H}$}$}_{k}\right),\,\,\widehat{\boldsymbol{\lambda}}_{k}=(\mbox{$\mbox {$\mathbf{v}$}$}_{k},-\mbox{$\frac{1}{2}$}\mbox{$\mbox{$\mathbf{V}$}$}_{k}),$ (33) $$

where $\mbox{$\mbox{$\mathbf{H}$}$}_{k}=\mathbb{E}_{q_{k}}[\nabla^{2}\ell_{k}]$ is now the full expected Hessian which determines the form of the second dual variable to be a matrix. We now repeat the same derivations as in the diagonal covariance case. For the client in [Eq. 7](https://arxiv.org/html/2506.13150v1#S3.E7), we get:

$$ $\displaystyle\mbox{$\mbox{$\mathbf{m}$}$}_{k},\mbox{$\mbox{$\mathbf{S}$}$}_{k}$ $\displaystyle\leftarrow\operatorname{argmin}_{\text{$\mbox{$\mbox{$\mathbf{m}$ }$}_{k}$},\text{$\mbox{$\mbox{$\mathbf{S}$}$}_{k}$}}\,\mathbb{E}_{q_{k}}[\ell_ {k}(\boldsymbol{\theta})+\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\boldsymbol{ \theta}-\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}\mbox{$\mbox{$\mathbf{V} $}$}_{k}\boldsymbol{\theta}]+\rho{\operatorname{KL}}{}\left(q\,\|\,q_{k}\right ){},$ (34) $$

where the KL-divergence is given as,

$$ ${\operatorname{KL}}{}\left(q\,\|\,q_{k}\right){}=\mbox{$\frac{1}{2}$}\left( \text{tr}(\mbox{$\mbox{$\mathbf{S}$}$}^{-1}\bar{}\mbox{$\mbox{$\mathbf{S}$}$}) +\log\det(\mbox{$\mbox{$\mathbf{S}$}$})+\|\mbox{$\mbox{$\mathbf{m}$}$}-\bar{} \mbox{$\mbox{$\mathbf{m}$}$}\|_{\text{\mbox{$\mbox{$\mathbf{S}$}$}}}^{2}\right )+\text{const.}$ (35) $$

By substitution, the dual update in line 2 of BayesADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) is,

$$ $\displaystyle\mbox{$\mbox{$\mathbf{v}$}$}_{k}$ $\displaystyle\leftarrow\mbox{$\mbox{$\mathbf{v}$}$}_{k}+\rho(\mbox{$\mbox{$ \mathbf{S}$}$}_{k}\mbox{$\mbox{$\mathbf{m}$}$}_{k}-\bar{}\mbox{$\mbox{$\mathbf {S}$}$}\bar{}\mbox{$\mbox{$\mathbf{m}$}$}),\quad\mbox{$\mbox{$\mathbf{V}$}$}_{ k}\leftarrow\mbox{$\mbox{$\mathbf{V}$}$}_{k}+\rho(\mbox{$\mbox{$\mathbf{S}$}$} _{k}-\bar{}\mbox{$\mbox{$\mathbf{S}$}$}).$ (36) $$

Finally, the server update in line 3 of BayesADMM ([Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2)) can be written as:

$$ $\displaystyle\bar{}\mbox{$\mbox{$\mathbf{S}$}$}$ $\displaystyle\leftarrow(1-\alpha)\,\text{\tt{Mean}}\left(\mbox{$\mbox{$\mathbf {S}$}$}_{1:K}\right)+\alpha\left[\delta\mbox{$\mbox{$\mathbf{I}$}$}+\text{\tt{ Sum}}(\mbox{$\mbox{$\mathbf{V}$}$}_{1:K})\right],$ (37) $\displaystyle\bar{}\mbox{$\mbox{$\mathbf{m}$}$}$ $\displaystyle\leftarrow\bar{}\mbox{$\mbox{$\mathbf{S}$}$}^{-1}\left[(1-\alpha) \text{\tt{Mean}}(\mbox{$\mbox{$\mathbf{S}$}$}_{1:K}\mbox{$\mbox{$\mathbf{m}$}$ }_{1:K})+\alpha\text{\tt{Sum}}(\mbox{$\mbox{$\mathbf{v}$}$}_{1:K})\right].$ (38) $$

We use the full method for the experiments in [Figs. 3a](https://arxiv.org/html/2506.13150v1#S3.F3.sf1), [3b](https://arxiv.org/html/2506.13150v1#S3.F3.sf2) and [4](https://arxiv.org/html/2506.13150v1#S4.F4) where the variational subproblem is solved using VON [31, Eq. 12].

## Appendix D Details on using Alg. 2 to Implement BayesADMM ( Alg. 1 )

Here, we provide additional details on how line 4 in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) is implemented in our experiments. As commonly done, we consider a tempered version of the variational-Bayesian problem in [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3),

$$ $\bar{q}^{*}=\underset{\bar{q}\in\mathcal{Q}}{\operatorname{argmin}}~{}\sum_{k= 1}^{K}\mathbb{E}_{\bar{q}}[\ell_{k}(\boldsymbol{\theta})]+\tau{\operatorname{ KL}}{}\left(\bar{q}\,\|\,\pi_{0}\right){},$ (39) $$

where $\tau>0$ is a temperature parameter. Dividing the whole objective by $\tau$ does not change the minimizer and gives rescaled client losses $\ell_{k}(\boldsymbol{\theta})/\tau$ on the original objective [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3).

For the tempered problem, the local client optimisation problem in BayesADMM for Gaussians with diagonal covariance reads as follows,

$$ $\operatorname{argmin}_{q_{k}}\,\mathbb{E}_{q_{k}}\left[\sum_{i=1}^{N_{k}}\ell_ {k}^{(i)}(\boldsymbol{\theta})/\tau+\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top} \boldsymbol{\theta}-\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}(\mbox{$ \mbox{$\mathbf{u}$}$}_{k}\boldsymbol{\theta})\right]+\smash{\rho{\operatorname {KL}}{}\left(q_{k}\,\|\,\bar{q}\right){}},$ (40) $$

which when pulling the KL-terms into the first expectations gives us the modified loss in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1). $\ell_{k}^{(i)}$ denote the per-example loss functions and $N_{k}$ is the number of data examples on the client $k$.

We now bring this problem into a form that allows us to directly apply [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2). Dividing by $\rho$ we have the equivalent form:

$$ $\operatorname{argmin}_{q_{k}}\,\frac{N_{k}}{\rho\tau}\mathbb{E}_{q_{k}}\left[ \frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\ell_{k,i}(\boldsymbol{\theta})+\frac{\tau}{N _{k}}\mbox{$\mbox{$\mathbf{v}$}$}_{k}^{\top}\boldsymbol{\theta}-\frac{\tau}{N_ {k}}\mbox{$\frac{1}{2}$}\boldsymbol{\theta}^{\top}(\mbox{$\mbox{$\mathbf{u}$}$ }_{k}\boldsymbol{\theta})\right]+\smash{{\operatorname{KL}}{}\left(q_{k}\,\|\, \bar{q}\right){}}.$ (41) $$

Matching the forms of the above equation with [Eq. 30](https://arxiv.org/html/2506.13150v1#A3.E30) gives us $\lambda=N_{k}/(\rho\tau)$, $\mbox{$\mbox{$\mathbf{v}$}$}=(\tau/N_{k})\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, $\mbox{$\mbox{$\mathbf{u}$}$}=(\tau/N_{k})\mbox{$\mbox{$\mathbf{u}$}$}_{k}$. For each client subproblem in line 4 in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1) we are calling [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) as a subroutine with the following inputs: $\ell_{k}$, $(\bar{}\mbox{$\mbox{$\mathbf{m}$}$},\bar{}\mbox{$\mbox{$\boldsymbol{\sigma}$}$
}^{2})$, $\frac{N_{k}}{\rho\tau}$, $\frac{\tau}{N_{k}}\mbox{$\mbox{$\mathbf{v}$}$}_{k}$, and $\frac{\tau}{N_{k}}\mbox{$\mbox{$\mathbf{u}$}$}_{k}$. [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) then returns the needed ($\mbox{$\mbox{$\mathbf{m}$}$}_{k},\mbox{$\mbox{$\mathbf{s}$}$}_{k}$) which minimizes the client objective in [Eq. 28](https://arxiv.org/html/2506.13150v1#A3.E28).

## Appendix E Proofs

See [3.1](https://arxiv.org/html/2506.13150v1#S3.Thmprop1)

###### Proof.

To show this, we note that if we run line 1 and line 2 iteratively until convergence, then $\mbox{$\mbox{$\boldsymbol{\mu}$}$}_{k}=\bar{\boldsymbol{\mu}}$ and $\boldsymbol{\lambda}_{k}=\bar{\boldsymbol{\lambda}}$, and the dual vector $\widehat{\boldsymbol{\lambda}}_{k}$ is also equal to the natural gradient at $\bar{\boldsymbol{\mu}}$. This
way, the average over $\boldsymbol{\lambda}_{k}$ in line 3 in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) is equal to $\bar{\boldsymbol{\lambda}}$ and the update becomes the BLR [31, Eq. 6].
∎

See [3.2](https://arxiv.org/html/2506.13150v1#S3.Thmprop2)

###### Proof.

From the optimality condition of [Eq. 7](https://arxiv.org/html/2506.13150v1#S3.E7), we get that $\boldsymbol{\lambda}_{k}=\bar{\boldsymbol{\lambda}}+\frac{1}{\rho}\mbox{$\mbox
{$\mathbf{t}$}$}_{k}$ as we initialize $\widehat{\boldsymbol{\lambda}}_{k}=0$.
Noticing that $\widehat{\boldsymbol{\lambda}}_{k}=\mbox{$\mbox{$\mathbf{t}$}$}_{k}$ after the dual update, the server update is:

$$ $\displaystyle\bar{\boldsymbol{\lambda}}$ $\displaystyle=(1-\alpha)\frac{1}{K}\sum_{k=1}^{K}\left(\widehat{\boldsymbol{ \lambda}}_{0}+\frac{1}{\rho}\mbox{$\mbox{$\mathbf{t}$}$}_{k}\right)+\alpha \left(\widehat{\boldsymbol{\lambda}}_{0}+\sum_{k=1}^{K}\mbox{$\mbox{$\mathbf{t }$}$}_{k}\right)$ (42) $\displaystyle=\widehat{\boldsymbol{\lambda}}_{0}+\sum_{k=1}^{K}\mbox{$\mbox{$ \mathbf{t}$}$}_{k},$ (43) $$

where in the second equality we used $\rho=1/K$. This is the optimality condition of ([3](https://arxiv.org/html/2506.13150v1#S3.E3)) for a conjugate prior $\widehat{\boldsymbol{\lambda}}_{0}$.
In the second client step, the Lagrange multiplier term cancels out with the loss (since $\widehat{\boldsymbol{\lambda}}_{k}=\mbox{$\mbox{$\mathbf{t}$}$}_{k}$ and $\ell_{k}(\boldsymbol{\theta})=-\mbox{$\mbox{$\mathbf{t}$}$}_{k}^{\top}\mbox{$
\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})$), and minimizing the KL leads to $\boldsymbol{\lambda}_{k}=\bar{\boldsymbol{\lambda}}$, and therefore all clients and the server reach the optimal solution after one communication round.
∎

###### Proof.

###### Proof.

## Appendix F Experimental Details for the Illustrative Examples

### F.1 Logistic Regression Convergence Plot in Fig. 3a

The setup is a Bayesian logistic multiclass regression problem on MNIST. We have $10$ classes, split onto $K=5$ clients where the data is split up in a heterogeneous way: [(0,1), (2,3), (4,5), (6,7), (8,9)].
We run all methods PVI, PVI with damping [54, 6], BregmanADMM [58] and BayesADMM for full Gaussian posteriors. BregmanADMM is implemented similar to BayesADMM but using the $\boldsymbol{\mu}$-coordinates in the dual update. This is possible, since the KL-divergence between two exponential families is a Bregman divergence. The client subproblem is solved by running the Variational Online Newton method [31, Eq. 12] until convergence. PVI with damping uses $\rho=1/K$ on the dual update (larger $\rho$ did not converge, smaller were slower), whereas for our method we set $\rho=1$. The horizontal dashed line in [Fig. 3a](https://arxiv.org/html/2506.13150v1#S3.F3.sf1) is the global minimum of the Bayesian logistic regression problem in terms of the objective in [Eq. 3](https://arxiv.org/html/2506.13150v1#S3.E3), computed by running the VON [31, Eq.12] on all data (batch mode).

### F.2 One-Step Convergence on Ridge Regression in Fig. 3b

The one step convergence plot in [Fig. 3b](https://arxiv.org/html/2506.13150v1#S3.F3.sf2) uses the same data as in [Sec. F.1](https://arxiv.org/html/2506.13150v1#A6.SS1), but we consider a linear regression loss $\ell_{k}(\boldsymbol{\theta})=\mbox{$\frac{1}{2}$}\|\mbox{$\mbox{$\mathbf{X}$}
$}\boldsymbol{\theta}-\mbox{$\mbox{$\mathbf{y}$}$}\|^{2}$. We also use a quadratic regularizer $\ell_{0}(\boldsymbol{\theta})=\frac{\delta}{2}\|\boldsymbol{\theta}\|^{2}$. Note that this fits the assumption of the one-step convergence result in [Prop. 3.2](https://arxiv.org/html/2506.13150v1#S3.Thmprop2) for sufficient statistics $\mbox{$\mbox{$\mathbf{T}$}$}(\boldsymbol{\theta})=(\boldsymbol{\theta},
\boldsymbol{\theta}\boldsymbol{\theta}^{\top})$. We compare the federated ADMM method as it is written on the left in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) to our BayesADMM with full Gaussian covariances and BregmanADMM [58] which uses $\boldsymbol{\mu}$-coordinates in the dual update of $\widehat{\boldsymbol{\lambda}}_{k}$.

### F.3 Toy Example in Fig. 4

For the toy example in [Fig. 4](https://arxiv.org/html/2506.13150v1#S4.F4) we run the standard federated ADMM algorithm outlined in [Fig. 2](https://arxiv.org/html/2506.13150v1#S2.F2) (but with tuned quadratic regularizer $\frac{\delta}{2}\|\boldsymbol{\theta}\|^{2}$) and BayesADMM with full covariances from [Sec. C.4](https://arxiv.org/html/2506.13150v1#A3.SS4). The dataset consists of the shown points, where we append a single dimension to the features to have a bias present in the linear classifier. We use a binary cross entropy loss (logistic regression) and train with $\delta=0.2$. The step-sizes are set to $\rho=0.2$ for both methods.

## Appendix G Hyperparameters, Architectures and Datasets

We provide further details about the hyperparameters, model architectures and datasets used in the experiments in [Tables 1](https://arxiv.org/html/2506.13150v1#S4.T1) and [2](https://arxiv.org/html/2506.13150v1#S4.T2).

For the hyperparameters in BayesADMM we perform a coarse grid search over the step-sizes $\rho$ and $\gamma$ in [Alg. 1](https://arxiv.org/html/2506.13150v1#alg1), the prior precision $\delta$ and the temperature $\tau$. Compared to FedProx, there are only two additional hyperparameters (the dual step-size and the temperature). Typically $0.1$ works well for both and we did not further tune them.
The subproblem is solved using the IVON optimizer outlined in [Alg. 2](https://arxiv.org/html/2506.13150v1#alg2) whose hyperparameters we set according to [53, Appendix A].

For all baseline methods, we follow the hyperparameter tuning procedure from Swaroop et al. [54].
We also ensure that the random dataset splits for our experiments match those from the reported results in Swaroop et al. [54].

Details on heterogeneous splits.
We follow the heterogeneous sampling procedure from Swaroop et al. [54, App E1] to generate heterogeneous splits of MNIST, FashionMNIST and CIFAR-10.
When there are 10 clients, 90% of all data is usually within 6 clients, with 2 clients having 50%. Within each client, usually 60-95% of client data belongs to just 4 classes.

### G.1 MNIST

We train on the full MNIST dataset of 60000 examples. All methods use batch-size 32. The model is a fully connected neural network with sigmoid activation functions and two hidden layers with $200$ and $100$ neurons, respectively.

Results for baselines are taken from Swaroop et al. [54] as our setup is identical to theirs.

For the highly-heterogeneous 100-client setting, we use the same split as in McMahan et al. [39], where each client has data from 2 classes only (300 examples from each class in each client).
This is a particularly difficult setting as each client only has data from 2 classes.
We perform hyperparameter sweeps over baselines following Swaroop et al. [54, App E4], except we also allow each hyperparameter to be one order of magnitude smaller (as there are more clients).

### G.2 FashionMNIST

For the $10$ client splits, we train on $10\%$ of the FMNIST dataset. The $100$ client split uses the full dataset. All methods use batch-size 32, and the model is again the same fully connected neural network as for MNIST.

Results for baselines are taken from Swaroop et al. [54] as our setup is identical to theirs.

### G.3 CIFAR-10

We train on the full CIFAR-10 dataset. We consider two models, one is a convolutional network used in Swaroop et al. [54] (from Zenke et al. [63]) which uses dropout and the other model is the convolutional network used in Acar et al. [1] which does not use dropout and has less convolutional but more fully connected layers. All methods use batch-size 64.

Results for baselines for the first CNN are taken from Swaroop et al. [54] as our setup is identical to theirs.
For the second CNN, we rerun hyperparameter sweeps similar settings to Swaroop et al. [54, App E6].
In addition to those parameters, we also sweep over Adam learning rate ($10^{-3}$ or $10^{-4}$).
We also try SGD with learning rate $0.1$ and sweep over learning rate decay ($0.992$ or $1$), so that we follow the hyperparameter sweep settings from Acar et al. [1].

### G.4 CIFAR-100

We train on the full CIFAR-100 dataset, split across 10 clients using the same Dirichlet parameters. We use a ResNet-20 model [26]. All methods use batch-size 64.

We use the same hyperparameter sweeps as in Swaroop et al. [54] for CIFAR-10. We note that FedLap-Cov is now prohibitively slow (the Laplace estimates of the covariance every communication round takes a long time), and so we do not provide results for it.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: Why is it valid to just switch back and forth between variational inference and MLE? I was not convinced that equation 5 was equivalent to equation 1. Along this line, the notation in the paper blurred distinction between parameters and distributions (see equation 4 for example and line 976).
In equation 12-14, $\bar{\lambda}$ is not defined. Perhaps include an update equation for it. Is that because it is a deterministic mapping from $\bar{\mu}$? Similarly for $\bar{q}$, is the update from equation 4?
Under what circumstances does BayesADMM converge?
soundness: 2
strengths: The paper proposes a novel ADMM-like extension to federated learning, with good experimental results.
summary: The authors propose an analogous extension to federated ADMM in the context of variational inference. This provides an extension of ADMM-like federated procedures based on the duality of various exponential distributions.
weaknesses: It seemed that the argument was more by analogy than exact equality. The claim that ADMM is recovered exactly is misleading because it requires an approximation and therefore is not necessarily recovered exactly.

A couple minor points:
- In equation 3, a subscript k is missing
- On line 195 A* should be defined in the main text rather than just in the appendix.
- In figure 4, it would be helpful to mention that each line is numbered with the iteration number.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: In the federated learning ADMM framework, there are theoretical guarantees for communication complexity and iterative complexity. Can the author briefly discuss the communication complexity and iteration complexity of Bayesian ADMM.
soundness: 3
strengths: Authors introduced a Bayesian duality, from which an extension of ADMM that optimizes over distributions naturally follows. For Gaussians with fixed variance, they recover regular ADMM and general Gaussians give Newton-like methods and IVON-ADMM. These show good performance when compared to recent baselines. Other approximating distributions may lead to new interesting splitting algorithms, and more generally, which opens up new research paths to extend and improve primal-dual algorithms using Bayesian ideas.
summary: Authors propose a new Bayesian approach to derive and extend the federated Alternating Direction Method of Multipliers (ADMM). We show that the solutions of variational-Bayesian objectives are associated with a duality structure that not only resembles ADMM but also extends it， which opens a new direction to use Bayes to extend ADMM and other primal-dual methods.
weaknesses: In the federated learning ADMM framework, there are theoretical guarantees for communication complexity and iterative complexity. Can the author briefly discuss the communication complexity and iteration complexity of Bayesian ADMM.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. What specific mechanism makes BayesADMM or IVON-ADMM handle heterogeneity and uncertainty better than existing method? Can you clarify the role of the posterior covariance in this improvement and demonstrate it experimentally?
2. In Fig. 4, how are the gray uncertainty contours generated? Are they derived from posterior covariance?
3. Why does the exposition (Eqns. 1, 2, 10, 11) rely on the plain Lagrangian, while the ADMM's implementation (Eqns. 3, 12-14) uses the augmented Lagrangian version?
4. The notation $\hat{\lambda}_k$ seems to play dual roles -- as natural parameters of site functions in Sec. 3.2 and as dual variables in Sec. 3.3. Are both interpretations valid? If so, explain their precise connection.
soundness: 2
strengths: __Conceptual novelty:__ The paper establishes a novel Bayesian duality perspective that unifies ADMM and VB under a single framework. This is an interesting connection that could inspire extensions of primal-dual optimization methods.

__Clear motivation and exposition:__ The introduction and backgrounds are well written and clearly position the work relative to prior ADMM and PVI approach (Swaroop et al., 2025).

__Framework generality:__ The proposed Bayesian duality formulation provides a principled way to derive new ADMM-like algorithms by changing the exponential-family posterior.

__Readable presentation:__ For the most part, the paper is well structured and logically progress from classical ADMM to its Bayesian interpretation and finally to _IVON-ADMM_.
summary: The paper proposes a general Bayesian perspective on federated ADMM. Under this framework, the classical ADMM emerges as a special case corresponding to isotropic Gaussian posteriors, while more expressive exponential-family posteriors yield new variants. One such variant called _IVON-ADMM_ is derived using diagonal Gaussian covariance and is claimed to perform better in heterogeneity and uncertainty cases of federated learning.
weaknesses: __Soundness of the Formulation:__ While the high-level idea is promising, the derivation in section 3.3 raises concerns about mathematical consistency:
- The "Bayesian ADMM" updates (Eqns. 12-14) are expected to follow from alternating optimization of the Lagrangian in Eqn. 11. However, the replacement of the dual update term $\mu_k - \bar{\mu}$ with $\lambda_k - \bar{\lambda}$ lacks justification within the Lagrangian formulation. The reasoning provided in Appendix E.2, appealing to Bayesian intuition, seems heuristic rather derivational.
- Equation 11 itself may need reconsideration: shifting the linear term $<\hat{\lambda}_k, \mu_k - \hat{\mu}>$ between the sub problems while keeping others fixed breaks symmetry between local and global updates, potentially undermining the claimed equivalence to ADMM.
- Overall, the theoretical grounding of the "Bayesian duality" remains somewhat fragile: the proposed updates look reasonable by analogy but are not rigorously shown to correspond to valid saddle-point dynamics of the stated objective.

__Relation to Existing Work:__
- The method appears to be a straightforward extension of _PVI_ with modified update equations. The novelty over _PVI_ is mainly the introduction of the step size $\rho$ and reinterpretation of dual variables. The paper should make a stronger argument for why this constitutes a _fundametal_ new framework rather  than a variant of _PVI_ with heuristic scaling.
A direct empirical or theoretical  comparison with _PVI_ (as Eqn. 4) is missing. Including  such results would make the claimed advantages more credible.


__Experimental Evaluation__:  The experiments, while broad, are not yet conclusive about the claimed benefits in heterogeneity and uncertainty.
- Figure 3-4 provide illustrative  but small-scale toy examples; they show qualitative improvement but not a clear quantitative advantage.
- The key claim that _IVON_ handles heterogeneity better by leveraging posterior covariance is not substantiated with ablation or analysis showing the role of uncertainty.
- Comparisons with both BayesADMM (without _IVON_) and _PVI_ are missing. Including them would help isolate what _IVON_ adds.
- The computational overhead relative to FedDyn should be quantified to see the computation gain compared to the performance gain.

__Clarity and Notation:__ Several presentation issues reduce readability and reproducibility.
- Step 3 of Fig. 2 is valid only for $\alpha=\frac{1}{1+\rho K}$.
- Many symbols and methods are used before being introduced:
    + BLR first appears on lines 294 and 297 without citation.
    + $q_{1:K}, \hat{t}_{1:K},\bar{q}$ are used before definition in line 215.
- The discussion of natural vs. ordinary gradients is confusing. The paper should use distinct and properly defined notations for both.
- Section 3.2  could be organized better: the correspondence between Eqns. 2 and 10 is conceptually interesting but presented unclearly, with inconsistent references to $\hat{t}_k, \lambda_k, \text{ and } \mu_k$.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.In practical federated learning scenarios, clients often have different computational capabilities and communication bandwidths. How does BayesADMM adapt to this system heterogeneity? Specifically, what is the algorithm's robustness when certain clients cannot complete the full variational inference update?
2.From Figure 3(a), it can be seen that PVI diverges without damping while BayesADMM remains stable. Could you provide a detailed analysis of BayesADMM's convergence guarantees, particularly theoretical guarantees for non-convex optimization problems?
3.IVON-ADMM uses diagonal covariance approximation in its implementation, which may lead to information loss in deep learning models. How much performance improvement do you think using low-rank approximation or other covariance structures would bring? How much would the computational overhead increase?
4.In Algorithm 1, you mention "implementation details in Appendix G," but the provided PDF excerpt does not contain this section. Could you briefly explain the key implementation differences between IVON-ADMM and standard ADMM, particularly the techniques used when handling high-dimensional parameter spaces?
5.The paper states that BayesADMM can converge in just one communication round for certain loss functions. Is this property limited to specific types of loss functions? Could you provide more general conditions that specify when the algorithm can converge quickly?
soundness: 3
strengths: 1. The paper proposes a novel theoretical framework that integrates Bayesian variational inference with the ADMM algorithm, providing a unified perspective for federated learning. This approach of connecting optimization algorithms with probabilistic inference has theoretical depth.
2. The authors clearly demonstrate how traditional federated ADMM serves as a special case of BayesADMM under specific conditions. This theoretical connection provides a new perspective for understanding existing methods and naturally leads to more powerful extensions.
3. The implementation details of IVON-ADMM showcase a practical and efficient algorithm variant. Compared to existing federated learning optimizers (such as FedAvg and FedAdam), it incurs limited additional computational overhead while delivering significant performance improvements.
4. The experimental design is comprehensive, covering both homogeneous and heterogeneous data distribution scenarios, and validates the method's effectiveness across multiple standard datasets. Particularly, Figure 3(b) demonstrates BayesADMM's property of converging in just one communication round for certain loss functions, which is a compelling empirical result.
5. The authors situate their method within the broader context of optimization and Bayesian inference literature, clearly highlighting connections and distinctions with prior works such as Partitioned Variational Inference (PVI) and Bregman ADMM.
summary: The paper proposes a novel method that combines Bayesian variational inference with the Alternating Direction Method of Multipliers (ADMM) for federated learning scenarios. Based on the concept of "Bayesian Duality," the authors extend traditional parameter optimization to distribution optimization, developing the BayesADMM algorithm. This method not only encompasses standard federated ADMM as a special case (when using fixed-variance Gaussian distributions) but also naturally extends to more complex distributions, producing update rules similar to Newton's method. The authors further derive the IVON-ADMM variant, a computationally efficient implementation suitable for deep learning models. Experimental results show that this method outperforms existing baselines on multiple datasets (MNIST, FashionMNIST, and CIFAR-10), particularly excelling in heterogeneous data distribution scenarios.
weaknesses: 1.Although the paper claims BayesADMM is theoretically superior, it lacks detailed analysis of computational complexity. Specifically, how do the algorithm's computational and communication costs scale with model parameter size when using more complex distributions (such as full-covariance Gaussians)? This is crucial in practical federated learning scenarios.
2.The experimental section only reports average performance without providing standard deviations or statistical significance tests, making it difficult to assess result reliability. Given the stochastic nature of federated learning (such as client selection and data partitioning), such statistical analysis is particularly necessary.
3.Table 4 shows IVON-ADMM's performance across multiple rounds but does not analyze the trade-off between convergence speed and communication rounds. In practical applications, early stopping may be more useful, but the paper does not explore performance comparisons of different methods under limited communication budgets.
4.While the authors mention the concept of "Bayesian Duality," their explanation of its theoretical foundation is not sufficiently deep. The derivation of the BayesADMM algorithm in Section 3.3 is relatively brief, and certain key steps (such as the transition from Equation 26 to 27) lack detailed explanation, which may affect reader comprehension.
5.The paper does not adequately discuss the impact of hyperparameter selection, particularly the sensitivity of regularization parameter ρ and temperature parameter τ. Although the appendix mentions hyperparameter search, it lacks systematic analysis of how these parameters affect algorithm performance.
6.The comparison with recent state-of-the-art federated learning methods (such as Scaffold and FedProx) is not comprehensive. While the paper compares with FederatedADMM and BregmanADMM, these methods are relatively outdated, and more comparisons with recent works should be included.


Decision: Accept (Poster)