# On the Adaptive Penalty Parameter Selection in ADMM
## 3 Numerical Experiments
### Consensus \(\ell_{1}\)-Regularized Logistic Regression  
As a first experiment, we test the proposed algorithms on the solution of consensus \(\ell_{1}\)-regularized logistic regression (see Section 4.2 in [1; 7]). Let us consider a dataset consisting of \(M\) training pairs \((D_{i},y_{i})\in\mathbb{R}^{n}\times\{0,\,1\}\). The aim is to build a linear classifier by minimizing a regularized logistic regression functional, exploiting a distributed computing architecture. One can do so by partitioning the original dataset into \(S\) subsets of size \(m_{1},\,\ldots,\,m_{S}\), such that \(\sum_{s=1}^{S}m_{s}=M\), and solving the optimization problem  
\[ \begin{array}{ll}\min_{x_{1},\ldots,x_{S}\in\mathbb{R}^{n},z\in\mathbb{R}^{ n}}&\sum_{s=1}^{S}\sum_{j=1}^{m_{s}}\log\biggl{(}1+e^{-y_{s,j}D_{s,j}^{ \top}x_{s}}\biggr{)}+\lambda\|z\|_{1}\\ \text{s.t.}&x_{s}-z=0,\qquad for\;s=1,\,\ldots,\,S,\end{array}  \tag{22}\]  
where \(x_{s}\) is the local variable on the \(s\)-th computational node, which acts as a linear classifier of the \(s\)-th subset, \(\bigl{(}D_{s,j},\,y_{s,j}\bigr{)}\) represents the \(j\)-th training pair of the \(s\)-th subset, and \(z\) is the global variable. We can reformulate problem (22) as  
\[ \begin{array}{ll}\min_{u,v}&H(u)+G(v)\\ \text{s.t.}&u-Fv=0\end{array}  \tag{23}\]  
where we set \(u=(x_{1}^{\top},\,\ldots,x_{S}^{\top})^{\top}\in\mathbb{R}^{nS}\), \(v=z\), \(H(u)=\sum_{s=1}^{S}\sum_{j=1}^{m_{s}}\log\biggl{(}1+e^{-y_{s,j}D_{s,j}^{\top}x_ {s}}\biggr{)}\), \(G(v)=\lambda\|v\|_{1}\), and \(F=(I_{n},\,\ldots,\,I_{n})^{\top}\) (i.e., the stacking of \(S\) identity matrices of order \(n\)). Scaling the dual variable, the augmented Lagrangian function associated with problem (23) is  
\[\mathcal{L}_{A}=H(u)+G(v)+\frac{\tau}{2}\biggl{\|}u-Fv-\frac{\xi}{\tau}\biggr{\|} ^{2}-\frac{\tau}{2}\biggl{\|}\frac{\xi}{\tau}\biggr{\|}^{2},\]  
where \(\xi=(\xi_{1}^{\top},\,\ldots,\xi_{S}^{\top})^{\top}\in\mathbb{R}^{nS}\), \(\tau>0\). Starting from given estimates \(u^{0}\), \(v^{0}\), \(\xi^{0}\), and \(\tau^{0}\), and observing that the minimization problem in \(u\) can be split into \(S\) independent optimization problems in \(x_{1},\,\ldots,\,x_{S}\), at each iteration \(k\) the adaptive ADMM updates the estimates as  
\[x_{1}^{k+1} = \operatorname*{argmin}_{x\in\mathbb{R}^{n}}\sum_{j=1}^{m_{1}}\log \biggl{(}1+e^{-y_{1,j}D_{1,j}^{\top}x}\biggr{)}+\frac{\tau^{k}}{2}\left\|x-v^{k }-\frac{\xi_{1}^{k}}{\tau^{k}}\right\|^{2},\] \[\vdots \vdots \tag{24}\] \[x_{S}^{k+1} = \operatorname*{argmin}_{x\in\mathbb{R}^{n}}\sum_{j=1}^{m_{S}}\log \biggl{(}1+e^{-y_{S,j}D_{5,j}^{\top}x}\biggr{)}+\frac{\tau^{k}}{2}\left\|x-v^{k }-\frac{\xi_{S}^{k}}{\tau^{k}}\right\|^{2},\] \[v^{k+1} = \operatorname*{argmin}_{v}\lambda\|v\|_{1}+\frac{\tau^{k}}{2} \left\|u^{k+1}-Fv-\frac{\xi^{k}}{\tau^{k}}\right\|^{2},\] (25) \[\xi^{k+1} = \xi^{k}+\tau^{k}(u^{k+1}-Fv^{k+1}). \tag{26}\]  
The \(S\) problems in (24), which are smooth unconstrained optimization problems, are solved approximately via BFGS with a stopping criterion on the gradient and the objective function with a tolerance set to 10 times the tolerance given to the ADMM scheme. The minimization in (25) can be performed via a soft thresholding operator. We considered 4 datasets from the LIBSVM ([[https://www.csie.ntu.edu.tw/~cjlin/lib](https://www.csie.ntu.edu.tw/~cjlin/lib)]([https://www.csie.ntu.edu.tw/~cjlin/lib](https://www.csie.ntu.edu.tw/~cjlin/lib)) svmtools/datasets/, accessed on 1 February 2023) dataset collection; to simulate real-life conditions, we created the instances by randomly extracted a training set consisting of 70% of the available data. Table 1 reports the final number of training pairs (\(M\)) and the problem dimension (\(n\)) for each of the datasets. We considered \(\lambda=1\) for all problems to enforce sparsity in the solutions and ran all the algorithms with \(\tau_{0}=1\), stopping when the relative residual was below \(\mathit{tol}=10^{-4}\) and/or the number of iterations reached 500.