PRUNING WITH HINTS: AN EFFICIENT FRAMEWORK FOR MODEL ACCELERATION

In this paper , we propose an efficient framework to accelerate convolutional neural networks . We utilize two types of acceleration methods : pruning and hints . Pruning can reduce model size by removing channels of layers . Hints can improve the performance of student model by transferring knowledge from teacher model . We demonstrate that pruning and hints are complementary to each other . On one hand , hints can benefit pruning by maintaining similar feature representations . On the other hand , the model pruned from teacher networks is a good initialization for student model , which increases the transferability between two networks . Our approach performs pruning stage and hints stage iteratively to further improve the performance . Furthermore , we propose an algorithm to reconstruct the parameters of hints layer and make the pruned model more suitable for hints . Experiments were conducted on various tasks including classification and pose estimation . Results on CIFAR-10 , ImageNet and COCO demonstrate the generalization and superiority of our framework . 1 INTRODUCTION . In recent years , convolutional neural networks ( CNN ) have been applied in many computer vision tasks , e.g . classification Krizhevsky & Hinton ( 2009 ) ; Deng et al . ( 2016 ) , objects detection Everingham et al . ( 2010 ) ; Ren et al . ( 2015 ) , and pose estimation Lin et al . ( 2014 ) . The success of CNN drives the development of computer vision . However , restricted by large model size as well as computation complexity , many CNN models are difficult to be put into practical use directly . To solve the problem , more and more researches have focused on accelerating models without degradation of performance . Pruning and knowledge distillation are two of mainstream methods in model acceleration . The goal of pruning is to remove less important parameters while maintaining similar performance of the original model . Despite pruning methods ’ superiority , we notice that for many pruning methods with the increase of pruned channel number , the performance of pruned model drops rapidlly . Knowledge distillation describes teacher-student framework : use high-level representations from teacher model to supervise student model . Hints method Romero et al . ( 2015 ) shares a similar idea of knowledge distillation , where the feature map of teacher model is used as high-level representations . According to Yosinski et al . ( 2014 ) , the student network can achieve better performance in knowledge transfer if its initialization can produce similar features as the teacher model . Inspired by this work , we propose that pruned model outputs similar features with original model ’ s and provide a good initialization for student model , which does help distillation . And on the other hand , hints can help reconstruct parameters and alleviate degradation of performance caused by pruning operation . Figure 1 illustrates the motivation of our framework . Based on this analysis , we propose an algorithm : we do pruning and hints operation iteratively . And for each iteration , we conduct a reconstructing step between pruning and hints operations . And we demonstrate that this reconstructing operation can provide a better initialization for student model and promote hints step ( See Figure 2 ) . We name our method as PWH Framework . To our best knowledge , we are the first to combine pruning and hints together as a framework . Our framework can be applied on different vision tasks . Experiments on CIFAR-10 Krizhevsky & Hinton ( 2009 ) , ImageNet Deng et al . ( 2016 ) and COCO Lin et al . ( 2014 ) datasets show the effectiveness of our framework . Furthermore , our method is a framework where different pruning and hints methods can be included . To summarize , the contributions of this paper are as follows : ( 1 ) We analyze the properties of pruning and hints methods and show that these two model acceleration methods are complementary to each other . ( 2 ) To our best knowledge , this is the first work that combines pruning and hints . Our framework is easy to be extended to different pruning and hints methods . ( 3 ) Sufficient experiments show the effectiveness of our framework on different datasets for different tasks . 2 RELATED WORK . Recently , model acceleration has received a great deal of attention . Quantization methods Rastegari et al . ( 2016 ) ; Courbariaux et al . ( 2016 ; 2015 ) ; Juefei-Xu et al . ( 2017 ) ; Zhou et al . ( 2017 ) reduce model size by quantizing float parameters to fixed-point parameters . And fixed-point networks can be speeded up on special implementation . Group convolution based methods Howard et al . ( 2017 ) ; Chollet ( 2017 ) separates a convolution operation into several groups , which can reduce computation complexity . Several works exploit linear structure of parameters and approach parameters using lowrank way to reduce computational parameters Denton et al . ( 2014 ) ; Jaderberg et al . ( 2014 ) ; Alvarez & Petersson ( 2016 ) ; Zhang et al . ( 2016 ) . In our experiments , we use two of current mainstream model acceleration way : pruning and knowledge distillation . 2.1 PRUNING . Network pruning has been proposed for a long time , such as Hanson & Pratt ( 1989 ) ; Hassibi et al . ( 1993 ) ; LeCun et al . ( 1990 ) . Recent pruning methods can be roughly adopted in two levels , i.e . channel-wise Molchanov et al . ( 2017 ) ; He et al . ( 2017 ) ; Sajid Anwar ( 2015 ) ; Hu et al . ( 2017 ) and parameter-wise Han et al . ( 2016 ; 2015 ) ; Yang et al . ( 2017 ) ; Li et al . ( 2017a ) ; Wen et al . ( 2015 ) ; Luo et al . ( 2017 ) .In this paper , we use channel-wise approach as our pruning method . There are many methods in channel-wise family . He et al . He et al . ( 2017 ) prune channels in LASSO regression way from sample feature map . Proposed in Liu et al . ( 2017 ) , the scale parameters in Batch Normalization layers are used to evaluate the importance of different channels . Molchanov et al . Molchanov et al . ( 2017 ) use taylor formula to analyze each channel ’ s contribution to the loss and prune the lowest contribution channel . We utilize this method in our framework . Despite the superiority of pruning methods , we find that the effectiveness of them will observably decrease with the increase of pruned channel numbers . 2.2 DISTILLATION . Knowledge distillation ( KD ) Hinton et al . ( 2015 ) is the pioneering work of this field . Hinton et al . Hinton et al . ( 2015 ) define soft targets and use it to supervise student networks . Beyond soft targets , hints are introduced in Fitnets Romero et al . ( 2015 ) , which can be explained as whole feature map mimic learning . Several researches have focused on hints . Zagoruyko et al . Zagoruyko & Komodakis ( 2017 ) propose atloss to mimic combined output of an ensemble of large networks using student networks . Furthermore , Li et al . Li et al . ( 2017b ) demonstrate a mimic learning strategy based on region of interests to improve small networks ’ performance for object detection . However , most of these works train student model from scratch and ignore the significance of student networks ’ initialization . 3 OUR APPROACH . In this section , we will describe our method in details . First we show hints and pruning methods which have been used in our framework . Then we introduce reconstructing operation and analyze its effectiveness . Finally , combining hints , pruning and reconstructing operation , we propose PWH Framework . 3.1 PRUNING STEP . The pruning method we use in this paper is based on Molchanov et al . ( 2017 ) . The algorithm can be described as a combinatorial optimization problem : min W ′ ∣∣∣∣C ( D|W ′ ) − C ( D|W ) ∣∣∣∣ st ||W ′||0 ≤ B , ( 1 ) Where C ( · ) is the cost function of the task , D is the training samples , W and W ′ are the parameters of original and pruning networks . In the optimization problem , ||W ′||0 bounds the number of nonzero parameters in W ′ . The parameter Wi whether to be pruned completely depends on its outputs hi . And the problem can be redescribed as minimizing ∣∣∆C ( hi ) ∣∣ = ∣∣C ( D , hi = 0 ) − C ( D , hi ) ∣∣ . However , it ’ s hard to find global optimal solution of the problem . Using taylor expansion can get approximate formula of the objective function : C ( D , hi = 0 ) ≈ C ( D , hi ) − δC δhi hi∣∣∆C ( hi ) ∣∣ ≈ ∣∣C ( D , hi ) − δC δhi hi − C ( D , hi ) ∣∣ = ∣∣∣∣ δCδhihi ∣∣∣∣ . ( 2 ) During backpropagation , we can get gradient δCδhi and activation hi . So this criteria can be easily implemented for channel pruning . 3.2 HINTS STEP . Hints can provide an extra supervision for student network , and it usually combines with task loss . The whole loss function of hints learning is represented as follows : L = Lt + λLh ( 3 ) Where Lt is the task loss ( e.g . classification loss ) and Lh is the hints loss . λ is hints loss weight which determines the intensity of hints supervision . There are many types of hints methods . Different hints methods are suitable for different tasks and network architectures . To demonstrate the superiority and generalization of our framework , we try three kinds of hints methods in our experiments : L2 hints , normalization hints and adaption hints . We introduce L2 hints , normalization hints in appendix . Lm = 1 N ‖ft − r ( fs ) ‖22 ( 4 ) Where r ( · ) is adaption layer . And for convolutional neural networks , adaption layer is 1× 1 convolution layer . 3.3 RECONSTRUCTING STEP . The objective function of reconstructing step is : min W ‖Y −WX‖22 ( 5 ) Where Y is the feature map of original ( teacher ) model , X is the input of hints layer and W is the parameter of hints layer . The optimization problem has close-form solution using least square method : W = ( XTX ) −1XTY . However , because some datasets ( e.g . ImageNet Deng et al . ( 2016 ) ) have numerous images , X will be high-dimension matrix . And it ’ s impossible to solve the problem which involves such huge matrix in one time . So we randomly sample images in dataset and compute X according to these images . Due to the randomness , the reconstructed weights may be worse than original weights . Thus , we finally use a switch to select better weights ( See equation 6 ) . Wf = arg min W ‖Y0 −WX0‖22 W ∈ { Wo , Wr } ( 6 ) Where Wf , Wo and Wr are the final weights , original weights and reconstructed weights of hints layer . Y0 and X0 are computed from the whole dataset . The objective function of Normalized L2 loss ( See equation 16 ) is different from L2 loss , but we explain that the reconstruction step is also effective for normalized L2 loss . The details of proof is available in supplementary material .

Summary: This paper uses pruning and model distillation iteratively to reduce the model sizes. The pruning step is based on Molchanov et al. (2017). This is followed by a hints step that minimizes the feature map difference between student and teacher. Finally, a reconstruction step is used to restore original weights. Results are shown on CIFAR-10, Imagenet and COCO datasets for classification and pose estimation tasks where PWH reduces model costs with a small loss in accuracy.