## 3 Related Works on Models Compression
### Parameter Pruning
#### 3.1.3 Granulation  
Depending on the structure, pruning can be classified in two main categories: unstructured pruning and structured pruning. The unstructured pruning approach, also called fine-grained, removes individual weights unconstrained from the local structure. However, pruning can removeentire structures (kernel, filter, etc.). Figure 1 illustrates the different level of granularity neural network pruning.  
Fine-grained pruning methods remove network parameters independently to the data structure. Any unimportant parameters in the convolutional kernels can be pruned. This consists of using salience criteria to rank individual weights then remove the less important. Recently the authors of [33] proposed a deep compression framework to compress deep neural networks in three steps: pruning, quantization, and Huffman encoding. By using this method, AlexNet could be compressed by a factor of 35 without drops in accuracy. The work uses a polynomial decay function to control the step sparsity level. High sparsity levels can be achieved, so the model can be compressed to require less memory space and bandwidth. However, a poor pruning can lead to a drop in accuracy. In addition, these approaches result in unstructured parsimony of the model. Without specialized parallelization techniques and hardware that can perform computations on sparse tensors, this does not accelerate the computation. Another strategy is to remove entire data structures, such as kernels or filters.  
Vector-level pruning methods prune 1D vectors in the convolutional kernels, and kernel-level pruning methods prune 2D kernels in the filters. Like fine-grained pruning, vector-level pruning and kernel-level convert dense connectivity into sparse connectivity, but in a structured way [40].  
Group-level pruning tries to eliminate network parameters according to the same sparse pattern on the filters [41]. In this way, convolutional computation can be efficiently implemented with reduced matrix multiplication. In [42], the authors propose a group-wise convolutional kernel pruning approach inspired by optimal brain damage [36]. The authors of [35] used group Lasso to learn a sparse structure of neural networks.  
Filter-level pruning reduces convolutional layers size by removing unimportant filters. This leads to a simplification of the network architecture. Layer filters are removed, and also the size of the output features-map is reduced and directly impacts on the convolutional layers' speedup. Pruning filters with low absolute weights sum is similar to pruning low magnitude weights. Filter-level pruning maintains dense tensors which leads to a lower compression ratio but is more efficient for accelerating the model with less convolution operation. The authors of [43] proposed ThiNet, a filter-level pruning framework to simultaneously accelerate and compress convolutional neural networks. They used the next layer's feature map to guide the filter pruning in the current layer.  
Unstructured sparsity produces very high compression rates but allows very low accuracy. Structured sparsity is more beneficial for direct savings of computational resources on embedded systems, in parallel computing environments and efficient inference acceleration. As proposed by [33], pruning can be combined with quantization to achieve maximal compression ratio.  
Figure 1: Neural network pruning granularity.