# Communication-Efficient Federated Learning with Adaptive Consensus ADMM
## 3 FedACADMM: Adaptive Consensus ADMM for Federated Learning
### Federated Optimization Problem  
The classical federated learning problem involves constructing a single, global statistical model from numerous clients. The primary objective of federated learning is to enable the model to be learned using client-generated data while staying within the constraints of local storage and processing. During periodic updates, clients communicate only with the central server. The ultimate aim of federated learning is typically to minimize the following objective function:  
\[\min_{w}F(w),\text{ where }F(w):=\sum_{k=1}^{m}p_{k}F_{k}(w) \tag{1}\]  
Here, \(m\) represents the total number of devices, \(p_{k}\geq 0\), and \(\sum_{k}p_{k}=1\). Additionally, \(F_{k}\) represents the local objective function for the \(k\)-th device. The local objective function is commonly defined as the empirical risk computed over local data. The local objective function is used to optimize the parameters of the model by minimizing the empirical risk over the local data. The optimization is performed locally, and the results are then aggregated to obtain a global model that can be used for inference or further training. A typical local objective function used in federated learning is \(F_{k}(w)=\frac{1}{n_{k}}\sum_{j_{k}=1}^{n_{k}}f_{j_{k}}\left(w;x_{j_{k}},y_{j_ {k}}\right)\), where \(n_{k}\) is the number of samples available locally. To adjust the weight of each device, a user-defined term \(p_{k}\) is introduced, where \(p_{k}\) is the weight assigned to device \(k\) and can be set to \(\frac{1}{n}\) or \(\frac{n_{k}}{n}\) depending on the desired effect. The total number of samples across all devices is denoted as \(n=\sum_{k}n_{k}\), which determines the scale of the learning task. Convex objective refers to optimization problems in which the objective function is convex. Convex functions have the property that for any two points in their domain, the line segment connecting them lies on top of the graph of the function. Convex objectives are important in optimization because they have desirable mathematical properties that make them easier to optimize. For convex targets, the distributed local update primitive dual method utilizes a duality structure to effectively decompose the global target into sub-problems that can be solved in parallel in each round of communication. Hence, in this article, we decompose the objective function of federated learning into sub-problems using this method,  
\[\min_{x\in\mathbb{R}^{d}}\left\{F(x):=f(x)+g(x)=\frac{1}{n}\sum_{i=1}^{m}f_{i} (x)+g(x)\right\} \tag{2}\]  
where \(m\) is the number of devices and the weighted local training loss is represented by \(f_{i}\), which is considered to be L-smooth and nonconvex, and \(g\) is a proper, closed, and convex regularizer. We emphasize that the use of regularizers \(g\) has been emphasized in various works, including [33]. The goal of federated learning is to solve (2) for the model parameters \(\theta\) using data distributed across the clients. \begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Definition** \\ \hline \(\theta^{t}\) & The global model of the \(t\)-th round \\ \(w_{i}\) & The local model computed by client \(i\) \\ \(S_{t}\) & A subset of clients chosen at the \(t\)-th round \\ \(\varepsilon\) & The stopping criterion \\ \(\rho^{t}\) & The self-adaptive penalty parameter at the \(t\)-th round \\ \(y_{i}\) & The local dual variable kept by client \(i\) \\ \(\delta_{i}^{t}\) & The local update at the \(t\)-th round by client \(i\) \\ \(\eta\) & The global step-size \\ \(t\) & The \(t\)-th round global iteration \\ \(\tau_{i}\) & Hyperparameters for regulating the range of \(\rho^{t+1}\) variations \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Symbols and DefinitionsLet \(\text{dom}(F):=\left\{x\in\mathbb{R}^{d}:F(x)<+\infty\right\}\) be the domain of F and \(\partial g\) be the subdifferential of \(g\)[34]. Because (2) is nonconvex, we only anticipate discovering a stationary point, which meets the next optimality condition.