## 2 Method
### ADMM Principle  
ADMM mainly solves the minimization problem of the objective function of two variables with equality constraints, and it mixes the decomposability of the dual rise algorithm with the superior convergence of the multiplier method, thus providing an effective method to solve the non-convex problem with combinatorial constraints [14; 15]. In the process of model pruning, ADMM decomposes the original pruning problem into two subproblems. It first solves subproblem 1 by the traditional gradient descent method, and then it introduces a quadratic term iteration to solve subproblem 2; finally, it performs system weight pruning, and its weight-optimization problem is shown in Equation (1) [16]:  
\[minimize\ f(\{W_{i}\},\{b_{i}\})+\sum_{i=1,...,\ N}^{N}g_{i}(W_{i}) \tag{1}\]  
where \(Z_{i}\) = \(W_{i}\), i = 1,..., N. \(S_{i}\) = \(\{W_{i}\}\)\(\mid\) card(\(W_{i}\)) \(\leq\) i\(\mid\), i = 1,..., N. \(S_{i}\),..., \(S_{N}\) are a nonconvex set. \[g_{i}(W_{i})= \begin{cases}0&if\ \ W_{i}\ \in S_{i}\\ +\infty&else\end{cases} \]  
where \(S_{i}\) is a set of weights for different pruning methods, and \(g(\cdot)\) is an indicator function containing weight sparsity constraints. Then, Equation (1) is rewritten from the augmented Lagrange formula to Equation (2):  
\[L_{\rho}(\{Z_{i}\},\{W_{i}\},\{\gamma_{i}\},\{b_{i}\})=f(\{W_{i}\},\{b_{i}\})+ \sum_{i=1}^{N}g_{i}(Z_{i})+\sum_{i=1}^{N}tr\Big{[}\gamma_{i}^{T}(W_{i}-Z_{i}) \Big{]}+\sum_{i=1}^{N}\frac{\rho_{i}}{2}\|W_{i}-Z_{i}\|_{F}^{2} \tag{2}\]  
where \(\gamma_{i}\) is the Lagrangian multiplier of the equal constraint of \(Z_{i}\) = \(W_{i}\) in Equation (1), and it is the same as the \(W_{i}\) dimension. The positive scalar \(\{\rho_{1},\rho_{2}\),..., \(\rho_{N}\}\) is the augmentation parameter, \(\text{tr}(\cdot)\) represents the trace of the matrix, and \(\|\cdot\|_{2}^{2}\) represents the Frobenius norm. _Ui\(=(1/\rho)\gamma i\)_, and Equation (2) can be rewritten as Equation (3):  
\[L_{\rho}(\{Z_{i}\},\{W_{i}\},\{\tau_{i}\},\{b_{i}\})=f(\{W_{i}\},\{b_{i}\})+\sum_{ i=1}^{N}g_{i}(Z_{i})+\sum_{i=1}^{N}\frac{\rho_{i}}{2}\|W_{i}-Z_{i}+U_{i}\|_{F}^{2}+ \sum_{i=1}^{N}\frac{\rho_{i}}{2}\|U_{i}\|_{F}^{2} \tag{3}\]  
The following steps (4), (5), and (6) are repeated. \[\left\{Z_{i}^{k+1}\right\}:=\underset{\{Z_{i}\}}{argmin}\ L\rho\Big{(}\left\{ b_{i}^{k+1}\right\},\{Z_{i}\},\left\{W_{i}^{k+1}\right\},\left\{U_{i}^{k}\right\} \Big{)} \tag{4}\]  
\[\left\{W_{i}^{k+1},b_{i}^{k+1}\right\}:=\underset{\{W_{i}\},\{b_{i}\}}{argmin }\ L\rho\Big{(}\left\{Z_{i}^{k}\right\},\{W_{i}\},\{bi\},\left\{U_{i}^{k+1} \right\}\Big{)} \tag{5}\]  
\[U_{i}^{k+1}:=U_{i}^{k}+\rho\Big{(}W_{i}^{k+1}-Z_{i}^{k+1}\Big{)} \tag{6}\]  
until the following conditions are met:  
\[\|W_{i}^{k+1}-Z_{i}^{k+1}\|\leq\varepsilon_{i},\ \|Z_{i}^{k+1}-\text{U}_{i}^{k+1} \|\leq\varepsilon_{i},\]  
where \(k\) is the iteration argument, and Equation (4) can be expressed as Equation (7). \[\underset{Zi}{min}E(x,y)\ \sim D[\max L(Z,x,y)]+\frac{\rho}{2}\underset{i=1}{ \sum}\|Z_{i}-W_{i}^{k}+U_{i}^{k}\|_{2}^{2} \tag{7}\]  
Equation (5) can be expressed as Equation (8). \[minimize\ f(\{W_{i}\},\{bi\})+\sum_{i=1}^{N}\frac{\rho^{i}}{2}\|W_{i}-Z_{i}^{ k+1}+U_{i}^{k}\|_{F}^{2} \tag{8}\]  
The former term of Equation (8) is a loss function, and the latter term can be used as a special L2 regularization. Since the two terms before and after Equation (8) is differentiable, this paper uses the stochastic gradient descent method to solve the formula.