## 1 Introduction  
Different from traditional hand-crafted feature extraction methods [1, 2, 3, 4], which involve abundant engineering skills and domain expertise, the convolutional neural network (CNN), a typical deep feature learning method, automatically extracts features at various scales. Supported by immense technological advancements, CNN models have grown deeper and wider. This growth in complexity has significantly intensified the learning ability of CNNs, which renders these models more suitable for numerous computer vision applications, such as object detection, object recognition, image classification, semantic segmentation, and many others [5; 6; 7; 8; 9; 10; 11]. Despite their surprising success, these deep models come at the cost of a vast amount of parameters and heavy computations, hampering their deployment on mobile or embedded devices with limited computational resources. Therefore, this problem has attracted significant attention from the research community in recent years. A prominent solution is to perform network compression without ominously degrading the model's performance. To compress a deep model, numerous approaches have been suggested including knowledge distillation [12; 13; 14], network quantization [15], a lightweight architecture [16; 17; 18; 19], low-rank approximations [20; 21], and network pruning [22; 23; 24; 25]. In knowledge distillation, the knowledge gathered by a large network (called the Teacher network) trained on large dataset is transferred to a smaller network (also known as the Student network). The Student model tries to mimic the Teacher model by learning its generalization capability and obtains a compact network with similar or higher accuracy. However, acquiring a well-learned Teacher model and transferring the rich knowledge from the Teacher to a Student are very challenging in knowledge distillation. Network quantization tries to cut down the number of bits occupied by weights to obtain a compact network model. Here, weights are represented by lowering the number of bits required per weight to store each weight. The weights can be quantized to 16-bit, 8-bit, 4-bit, or even with 1-bit. Quantization can be applied either during or after the training of the neural network, where relevancy among the network parameters is completely disregarded. Despite obtaining remarkable performance in network compression, these methods experience the threat of accuracy degradation. Most recent attempts have focused on factorization-based methods, for instance the methods where standard convolution modules are replaced with depthwise separable convolution modules, to obtain lightweight neural network architectures. Many classic lightweight architectures have been developed using the above process, such as MobileNet, SqueezeNet [26], SqueezeNext [27], ESPNet, etc. Although these models have achieved significant performance, they tend to have strong domain dependency, which makes it difficult to balance between model size and its performance. Low-rank approximation methods obtain a compressed model by using a low-rank matrix to approximate the original weight matrix. These methods discards weak network components layer by layer. Singular-value decomposition (SVD) is one of the popular low-rank approximation methods. These methods are suitable for fully connected layers, since operations in these layers can be performed easily using matrix operations. However, these are not suitable for convolutional layers, where most of the computations of the network take place. Despite the significant compression abilities of the above-mentioned approaches, network pruning has been widely explored in recent years due to its simplicity and effectiveness. Network pruning is the desirable way to compress the model by removing unwanted parameters in a structured or unstructured manner. Unstructured pruning removes the connections from the model with small magnitude, resulting in an irregular sparse structure of the CNN [15; 22; 28; 29; 30].