# Pruning Deep Neural Network Models via Minimax Concave Penalty Regression
## 2 Methodological Models
### Preliminary Investigation of Convolutional Layer Parameters  
Figure 7 is a scatter plot of the parameter distribution in the convolutional layers of the VGG19 network trained on the CIFAR100 dataset. For the VGG19 network with convolutional layers with 64, 128, 256, and 512 filters and for Layer1, Layer2, Layer3, and Layer4 in ResNet34, the quantiles are set at 0.99, 0.999, 0.9999, and 0.9999, respectively.  
Figure 7 reveals that the distribution of parameters is mostly concentrated near zero, as indicated by the blue area, yet there are many points that are far from the center, represented by the red area. Using the L1-Norm method to measure the importance of filters may not be accurate. In the field of statistics, the Minimax Concave Penalty (MCP) introduces differentiated penalties for coefficients. When the absolute value of parameters exceeds a threshold, all parameters are replaced with a fixed value, and when the absolute value is below the threshold, different degrees of penalty are applied based on their magnitude. [PERSON] (2019) [42] noted that in the presence of tail errors or outliers, sparse penalties can lead to more accurate and robust models. [PERSON] (2021) [43] discussed the selection problem when variables have outliers and applied the MCP penalty to logistic regression models, using a weighted method to mitigate the impact of outliers. Building on these findings, this study incorporates the MCP function into filter pruning to address the issue of \"outliers\" in parameters. For parameters whose absolute values are above the threshold, they are considered equally important, and for those below the threshold, different degrees of compression are applied according to their magnitude, ensuring that larger parameters  
Figure 6: Schematic diagram of the pruning process.  
Figure 7: VGG19 convolutional layer parameter distribution diagram. Blue represents weight values close to 0, red represents weight values a little further away from 0.  
signifies the number of samples; n represents the number of output channels; and \(k_{h}\) and \(k_{\omega}\) denote the convolution sizes. For the sake of representation convenience, the bias term is not considered. The optimization function during pruning is  
\[\underset{\beta,W}{\text{argmin}}\frac{1}{2N}(Y-\sum_{i\,=\,1}^{c}\,\beta_{i}X_ {i}W_{i}^{T})^{2}+\sum_{j\,=\,1}^{c}\,MCP(\beta_{j}) \tag{14}\]  
In the given equation, \(X_{i}\) represents the matrix of the \(i\)-\(th\) channel of \(X\) after data transformation, with dimensions of \(N\times k_{h}k_{\omega}\). Similarly, \(W_{i}\) denotes the \(i\)-\(th\) channel of the data transformation of the weight earmarked for pruning, having dimensions of \(n\times k_{h}k_{\omega}\). In this context, \(X\) and \(Y\) denote the intermediate inputs generated by the selected pruning samples through the trained network. Specifically, \(X\) signifies the input of the current convolutional layer; \(Y\) represents the output of the current convolutional layer; and \(W\) stands for the parameter of the convolutional layer, already trained at the current layer.