PRUNING WITH HINTS: AN EFFICIENT FRAMEWORK FOR MODEL ACCELERATION

In this paper , we propose an efficient framework to accelerate convolutional neural networks . We utilize two types of acceleration methods : pruning and hints . Pruning can reduce model size by removing channels of layers . Hints can improve the performance of student model by transferring knowledge from teacher model . We demonstrate that pruning and hints are complementary to each other . On one hand , hints can benefit pruning by maintaining similar feature representations . On the other hand , the model pruned from teacher networks is a good initialization for student model , which increases the transferability between two networks . Our approach performs pruning stage and hints stage iteratively to further improve the performance . Furthermore , we propose an algorithm to reconstruct the parameters of hints layer and make the pruned model more suitable for hints . Experiments were conducted on various tasks including classification and pose estimation . Results on CIFAR-10 , ImageNet and COCO demonstrate the generalization and superiority of our framework . 1 INTRODUCTION . In recent years , convolutional neural networks ( CNN ) have been applied in many computer vision tasks , e.g . classification Krizhevsky & Hinton ( 2009 ) ; Deng et al . ( 2016 ) , objects detection Everingham et al . ( 2010 ) ; Ren et al . ( 2015 ) , and pose estimation Lin et al . ( 2014 ) . The success of CNN drives the development of computer vision . However , restricted by large model size as well as computation complexity , many CNN models are difficult to be put into practical use directly . To solve the problem , more and more researches have focused on accelerating models without degradation of performance . Pruning and knowledge distillation are two of mainstream methods in model acceleration . The goal of pruning is to remove less important parameters while maintaining similar performance of the original model . Despite pruning methods ’ superiority , we notice that for many pruning methods with the increase of pruned channel number , the performance of pruned model drops rapidlly . Knowledge distillation describes teacher-student framework : use high-level representations from teacher model to supervise student model . Hints method Romero et al . ( 2015 ) shares a similar idea of knowledge distillation , where the feature map of teacher model is used as high-level representations . According to Yosinski et al . ( 2014 ) , the student network can achieve better performance in knowledge transfer if its initialization can produce similar features as the teacher model . Inspired by this work , we propose that pruned model outputs similar features with original model ’ s and provide a good initialization for student model , which does help distillation . And on the other hand , hints can help reconstruct parameters and alleviate degradation of performance caused by pruning operation . Figure 1 illustrates the motivation of our framework . Based on this analysis , we propose an algorithm : we do pruning and hints operation iteratively . And for each iteration , we conduct a reconstructing step between pruning and hints operations . And we demonstrate that this reconstructing operation can provide a better initialization for student model and promote hints step ( See Figure 2 ) . We name our method as PWH Framework . To our best knowledge , we are the first to combine pruning and hints together as a framework . Our framework can be applied on different vision tasks . Experiments on CIFAR-10 Krizhevsky & Hinton ( 2009 ) , ImageNet Deng et al . ( 2016 ) and COCO Lin et al . ( 2014 ) datasets show the effectiveness of our framework . Furthermore , our method is a framework where different pruning and hints methods can be included . To summarize , the contributions of this paper are as follows : ( 1 ) We analyze the properties of pruning and hints methods and show that these two model acceleration methods are complementary to each other . ( 2 ) To our best knowledge , this is the first work that combines pruning and hints . Our framework is easy to be extended to different pruning and hints methods . ( 3 ) Sufficient experiments show the effectiveness of our framework on different datasets for different tasks . 2 RELATED WORK . Recently , model acceleration has received a great deal of attention . Quantization methods Rastegari et al . ( 2016 ) ; Courbariaux et al . ( 2016 ; 2015 ) ; Juefei-Xu et al . ( 2017 ) ; Zhou et al . ( 2017 ) reduce model size by quantizing float parameters to fixed-point parameters . And fixed-point networks can be speeded up on special implementation . Group convolution based methods Howard et al . ( 2017 ) ; Chollet ( 2017 ) separates a convolution operation into several groups , which can reduce computation complexity . Several works exploit linear structure of parameters and approach parameters using lowrank way to reduce computational parameters Denton et al . ( 2014 ) ; Jaderberg et al . ( 2014 ) ; Alvarez & Petersson ( 2016 ) ; Zhang et al . ( 2016 ) . In our experiments , we use two of current mainstream model acceleration way : pruning and knowledge distillation . 2.1 PRUNING . Network pruning has been proposed for a long time , such as Hanson & Pratt ( 1989 ) ; Hassibi et al . ( 1993 ) ; LeCun et al . ( 1990 ) . Recent pruning methods can be roughly adopted in two levels , i.e . channel-wise Molchanov et al . ( 2017 ) ; He et al . ( 2017 ) ; Sajid Anwar ( 2015 ) ; Hu et al . ( 2017 ) and parameter-wise Han et al . ( 2016 ; 2015 ) ; Yang et al . ( 2017 ) ; Li et al . ( 2017a ) ; Wen et al . ( 2015 ) ; Luo et al . ( 2017 ) .In this paper , we use channel-wise approach as our pruning method . There are many methods in channel-wise family . He et al . He et al . ( 2017 ) prune channels in LASSO regression way from sample feature map . Proposed in Liu et al . ( 2017 ) , the scale parameters in Batch Normalization layers are used to evaluate the importance of different channels . Molchanov et al . Molchanov et al . ( 2017 ) use taylor formula to analyze each channel ’ s contribution to the loss and prune the lowest contribution channel . We utilize this method in our framework . Despite the superiority of pruning methods , we find that the effectiveness of them will observably decrease with the increase of pruned channel numbers . 2.2 DISTILLATION . Knowledge distillation ( KD ) Hinton et al . ( 2015 ) is the pioneering work of this field . Hinton et al . Hinton et al . ( 2015 ) define soft targets and use it to supervise student networks . Beyond soft targets , hints are introduced in Fitnets Romero et al . ( 2015 ) , which can be explained as whole feature map mimic learning . Several researches have focused on hints . Zagoruyko et al . Zagoruyko & Komodakis ( 2017 ) propose atloss to mimic combined output of an ensemble of large networks using student networks . Furthermore , Li et al . Li et al . ( 2017b ) demonstrate a mimic learning strategy based on region of interests to improve small networks ’ performance for object detection . However , most of these works train student model from scratch and ignore the significance of student networks ’ initialization . 3 OUR APPROACH . In this section , we will describe our method in details . First we show hints and pruning methods which have been used in our framework . Then we introduce reconstructing operation and analyze its effectiveness . Finally , combining hints , pruning and reconstructing operation , we propose PWH Framework . 3.1 PRUNING STEP . The pruning method we use in this paper is based on Molchanov et al . ( 2017 ) . The algorithm can be described as a combinatorial optimization problem : min W ′ ∣∣∣∣C ( D|W ′ ) − C ( D|W ) ∣∣∣∣ st ||W ′||0 ≤ B , ( 1 ) Where C ( · ) is the cost function of the task , D is the training samples , W and W ′ are the parameters of original and pruning networks . In the optimization problem , ||W ′||0 bounds the number of nonzero parameters in W ′ . The parameter Wi whether to be pruned completely depends on its outputs hi . And the problem can be redescribed as minimizing ∣∣∆C ( hi ) ∣∣ = ∣∣C ( D , hi = 0 ) − C ( D , hi ) ∣∣ . However , it ’ s hard to find global optimal solution of the problem . Using taylor expansion can get approximate formula of the objective function : C ( D , hi = 0 ) ≈ C ( D , hi ) − δC δhi hi∣∣∆C ( hi ) ∣∣ ≈ ∣∣C ( D , hi ) − δC δhi hi − C ( D , hi ) ∣∣ = ∣∣∣∣ δCδhihi ∣∣∣∣ . ( 2 ) During backpropagation , we can get gradient δCδhi and activation hi . So this criteria can be easily implemented for channel pruning . 3.2 HINTS STEP . Hints can provide an extra supervision for student network , and it usually combines with task loss . The whole loss function of hints learning is represented as follows : L = Lt + λLh ( 3 ) Where Lt is the task loss ( e.g . classification loss ) and Lh is the hints loss . λ is hints loss weight which determines the intensity of hints supervision . There are many types of hints methods . Different hints methods are suitable for different tasks and network architectures . To demonstrate the superiority and generalization of our framework , we try three kinds of hints methods in our experiments : L2 hints , normalization hints and adaption hints . We introduce L2 hints , normalization hints in appendix . Lm = 1 N ‖ft − r ( fs ) ‖22 ( 4 ) Where r ( · ) is adaption layer . And for convolutional neural networks , adaption layer is 1× 1 convolution layer . 3.3 RECONSTRUCTING STEP . The objective function of reconstructing step is : min W ‖Y −WX‖22 ( 5 ) Where Y is the feature map of original ( teacher ) model , X is the input of hints layer and W is the parameter of hints layer . The optimization problem has close-form solution using least square method : W = ( XTX ) −1XTY . However , because some datasets ( e.g . ImageNet Deng et al . ( 2016 ) ) have numerous images , X will be high-dimension matrix . And it ’ s impossible to solve the problem which involves such huge matrix in one time . So we randomly sample images in dataset and compute X according to these images . Due to the randomness , the reconstructed weights may be worse than original weights . Thus , we finally use a switch to select better weights ( See equation 6 ) . Wf = arg min W ‖Y0 −WX0‖22 W ∈ { Wo , Wr } ( 6 ) Where Wf , Wo and Wr are the final weights , original weights and reconstructed weights of hints layer . Y0 and X0 are computed from the whole dataset . The objective function of Normalized L2 loss ( See equation 16 ) is different from L2 loss , but we explain that the reconstruction step is also effective for normalized L2 loss . The details of proof is available in supplementary material .

Summary: This paper proposes a new framework which combines pruning and model distillation techniques for model acceleration. Though the ``pruning” (Molchanov et al. (2017)) and hint components already exists, the authors claim to be the first to combine them, and experimentally show the benefit of jointly and iteratively applying the two techniques. The authors show better performance of their new framework over baseline, pruning only method and hint only method on a few standard Vision data set.