## 2 Related Works  
### Model Compression and Acceleration  
#### 2.1.1 Pruning and Sparse Connection  
Pruning is a well-studied method for reducing the computation and storage costs of deep neural network models. Initially, connections were pruned based on the lowest saliency [11] through the computation of the Hessian or inverse Hessian matrix for every parameter, as shown in the Optimal Brain Damage (OBD) and Optimal Brain Surgeon (OBS) methods [11; 12]. However, this approach is not feasible for state-of-the-art deep neural network models like AlexNet and VGG-16, which have 60 million and 138 million parameters, respectively.  
To address this challenge, the deep compression method introduced a threshold-based approach to remove connections [5]. This method mainly prunes the connections that are in the fully connected layers, which account for 90% of the total parameters and only 1% of the overall floating-point operations (FLOPs) [13].  
Convolutional layers require Sparse Basic Linear Algebra Subprogram (BLAS) libraries [5] or special hardware to deal with sparse matrices [14]. To overcome these limitations, researchers have proposed numerous pruning methods that do not require specific hardware, such as vector-level sparsity (1D), kernel-level sparsity (2D), and filter-level sparsity (3D). The use of 1D, 2D, and 3D filter pruning is a natural structural way of pruning that does not require BLASs or specialized hardware [13; 15; 16] and can reduce FLOPs by more than 30%, 48%, and 52%, respectively.  
Dynamic channel pruning prunes filters dynamically during training based on their contribution to the network's loss [17]. This approach achieves higher sparsity and better accuracy than traditional pruning methods while also being computationally efficient.  
The Lottery Ticket Hypothesis suggests that a sparse subnetwork of a larger neural network can achieve accuracy comparable to that if the original dense network [18]. This method is based on iterative pruning, where weights below a certain threshold are pruned at each iteration, and the remaining weights are then fine-tuned to recover accuracy. The Lottery Ticket Hypothesis has been applied to various architectures and tasks and shown to achieve high levels of sparsity with little accuracy degradation.  
Group sparsity regularization prunes filters by promoting group sparsity in the network [19]. This method imposes a penalty on the L1 norm of groups of filters in each layer, encouraging the network to learn sparser representations. Group sparsity regularization achieves high levels of sparsity with minimal accuracy degradation and can be applied to both convolutional and fully connected layers.  
Efficient dense module search uses a search algorithm to find the most efficient dense module for a given task [20]. The efficient dense module search method prunes filters in each layer and selects the remaining filters based on their contribution to the network's accuracy. This approach outperforms other pruning methods in terms of accuracy and efficiency while also reducing the number of parameters in the network.  
Finally, the progressive sparse learning method reduces the computational cost of iterative pruning and retraining the network [21]. Progressive sparse learning starts by training the network with a low sparsity level and gradually increases the sparsity level over multiple iterations. This method achieves high levels of sparsity with minimal accuracy degradation while also reducing the computational cost of training and inference.