# Communication-Efficient Federated Learning with Adaptive Consensus ADMM
## 5 Experiments
### Choosing the Initial Penalty Parameter  
When designing the evaluation method for FedACADMM, the first consideration in this paper was determining the appropriate initial self-adaptive penalty parameter, denoted as \(\rho^{0}\), which is crucial for (8). To achieve a balanced initial residual \(\left\|w_{i}^{t}-\theta^{t}\right\|\) and pairwise residual \(\left\|\theta^{t}-\theta^{t-1}\right\|\), an empirical rule-based heuristic optimization hyperparameter method was employed. This is because if one of the two terms approaches zero gradually, the algorithm would converge slowly. Another consideration in proposing the rule is to prevent additional computation. In particular, a practical approach in practice would be to run FedACADMM with various values of \(\rho^{0}\) in parallel to obtain multiple final global models, and then select between these based on performance (e.g., accuracy) on the validation data. In this paper, we adopted an empirical rule-based heuristic optimization method to narrow down the search space and select a value for \(\rho^{0}\) from a limited set of candidates, namely \(\left\{10^{-3},10^{-2},0.1,1,10\right\}\), following the experimental setups of [9; 32].  
Figure 2 displays a comparison of test accuracy curves and training loss curves across various initial value penalty values, denoted as \(\rho^{0}\). For clarity, we omit the curves associated with poor experimental results when \(\rho^{0}=10\). Our experiments demonstrate that FedACADMM is a robust approach for both homogeneous and heterogeneous data distributions. Furthermore, our results indicate that FedACADMM performs well within a limited number of communication rounds for different datasets, and is not highly sensitive to the choice of initial variable parameters \(\rho^{0}\). FedACADMM (\(\rho>0\)) has better performance when the initial value is \(\rho^{0}=10^{-2}\).  
Although it is not as good as \(\rho^{0}=10^{-3}\) in most cases, its performance is very stable without large deviations, and it maintains robust convergence. Therefore, we chose \(\rho^{0}=10^{-2}\) as the initial penalty parameter value for the comparison experiment with the baselines. When the initial penalty parameter value is \(\rho^{0}=10^{-2}\), the global model can effectively integrate local data and achieve robust and fast updates. It maintains a nice balance between \(\left\|w_{i}^{t}-\theta^{t}\right\|\) and \(\left\|\theta^{t}-\theta^{t-1}\right\|\). Figure 2d shows that \(\rho^{0}=10^{-3}\) stalls, which is why we did not choose it as the initial penalty parameter value. The penalty parameter that is chosen will affect how quickly ADMM converges in practice.  
When we choose the appropriate initial penalty parameter value, the proposed algorithm is robust and can achieve good results in both homogeneous and heterogeneous situations. In practical applications, the convergence speed of FedACADMM is dependent on  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **IID** & **Non-IID** \\ \hline MNIST & 98\% & 96\% \\ CIFAR-10 & 50\% & 45\% \\ FMNIST & 85\% & 80\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Target Accuracy. For better representation of figures and tables, we set different target accuracy for different data distributions in the same dataset.  
the chosen penalty parameter. The actual convergence of FedACADMM is particularly sensitive to this parameter selection. For instance, when the initial penalty parameters are set to \(\rho^{0}=\left\{10^{-3},10^{-2},10\right\}\), the convergence is usually very slow. However, FedACADMM displays robustness to the choice of initial penalty parameters\(\rho^{0}\). When appropriate values of the initial penalty parameters are selected, the FedACADMM algorithm demonstrates strong robustness and achieves excellent model training results and fast convergence in both IID and non-IID data distribution cases.