# Pruning Deep Neural Network Models via Minimax Concave Penalty Regression
## 2 Methodological Models
### MCP Regression and Solution  
The standard linear regression model is as follows:  
\[y_{i}\ =\ \alpha+\sum_{j}\beta_{j}x_{ij}+\varepsilon_{i},\ i\ =\ 1,2,\ldots,N \tag{2}\]  
In the equation above, \(\alpha\) and \(\beta\) represent the model's intercept and coefficients, respectively, while \(N\) denotes the total number of samples. Here, \(y_{i}\) is the response variable of the \(i^{th}\) sample, and \(x_{ij}\) is the \(j^{th}\) predictor variable of the \(i^{th}\) sample. In contrast to the least squares estimation (LSE) of the model, Lasso estimation incorporates L1 regularization into the loss function to strike a balance between model fitting and model complexity. The functional form of traditional Lasso regression is expressed as follows:  
\[(\hat{\alpha},\hat{\beta})\ =\ \text{argmin}\{\sum_{i\ =\ 1}^{N}\left(y_{i}- \alpha-\sum_{j}\beta_{j}x_{ij}\right)^{2}\}+\lambda\sum_{j}\lvert\beta_{j}\rvert \tag{3}\]  
The regularization parameter, \(\lambda\), is employed to control model complexity, reducing the model's parameters by penalizing the sum of the absolute values of the coefficients, thereby encouraging a preference for model simplicity. Through the application of L1 regularization, the coefficients are reduced toward zero, leading to the elimination of redundant features. The Lasso penalty function, being convex and inducing biased estimation, mitigates the overall prediction error by reducing the variance in the prediction model. However, it is susceptible to the drawback of excessively compressing the coefficients. In pursuit of further minimizing the total prediction error, [PERSON] and [PERSON] [40] introduced the SCAD penalty. This penalty not only compresses very small regression coefficients to zero but also provides an approximately unbiased estimation. [PERSON] [41] evidenced the Minimax Concave Penalty (MCP) as another notable penalty, validating its significance in the field. It differentiates penalties on regression coefficients, preserving the advantages of the SCAD penalty while yielding more precise estimates. The functional form of MCP regression is expressed as follows:  
\[(\hat{\alpha},\hat{\beta})\ =\ \text{argmin}\{\sum_{i\ =\ 1}^{N}\left(y_{i}- \alpha-\sum_{j}\beta_{j}x_{ij}\right)^{2}\}+\sum_{j}MCP(\beta_{j}) \tag{4}\]  
Figure 4: Effect diagram of a single filter. iteration is equivalent to one iteration of gradient descent. It solves local optimization by fixing the other variables and optimizing along one direction at a time. Specifically, in the iterative solving process for p-dimensional features, \(\{\beta_{1},\beta_{2},\ldots\beta_{j},\ldots\beta_{p}\}\), the other variables are held constant at their most recently updated values. Consequently, MCP regression can be viewed as optimized through a univariate solution scheme. In the context of simple linear regression, the least squares solution employing the unpunished function is represented by \(z~{}=~{}n^{-1}x^{\prime}y\), where \(x\) denotes the normalized \(x\), resulting in \(x^{\prime}x~{}=~{}n\). In MCP regression,  
\[\hat{\beta}~{}=~{}f_{MCP}(z,\lambda,\alpha)~{}=~{}\left\{ \begin{array}{ll} \frac{S(z,\alpha)}{1-1/\alpha}&|z|\leq\alpha\lambda\\ z&|z|>\alpha\lambda\end{array} \right. \tag{8}\]  
The variable \(S(z,\alpha)\) represents the univariate solution derived from Lasso regression and is expressed as  
\[S(z,\alpha)~{}=~{}\left\{ \begin{array}{ll}z-\lambda&z>\lambda\\ 0&|z|\leq\lambda\\ z+\lambda&z<-\lambda\end{array} \right. \tag{9}\]  
From the aforementioned equation, the solutions of MCP regression and Lasso regression coincide when \(\alpha\rightarrow\infty\). The univariate solution of MCP is attained through the coordinate descent algorithm, aiming to minimize the coordinates of the objective function. The symbol \(-j\) is introduced to denote the remaining portion of column \(j\) after removing its elements, and the partial residual of \(x_{j}\) is represented by \(r_{-j}~{}=~{}y-X_{-j}\beta_{-j}\), where \(\beta_{-j}\) denotes the most recently updated value of \(\beta\).