Title: KNOWLEDGE DISTILLATION FROM FEW SAMPLES

Abstract: Current knowledge distillation methods require full training data to distill knowledge from a large "teacher" network to a compact "student" network by matching certain statistics between "teacher" and "student" such as softmax outputs and feature responses. This is not only time-consuming but also inconsistent with human cognition in which children can learn knowledge from adults with few examples. This paper proposes a novel and simple method for knowledge distillation from few samples. Taking the assumption that both "teacher" and "student" have the same feature map sizes at each corresponding block, we add a 1 × 1 conv-layer at the end of each block in the student-net, and align the block-level outputs between "teacher" and "student" by estimating the parameters of the added layer with limited samples. We prove that the added layer can be absorbed/merged into the previous conv-layer to formulate a new conv-layer with the same size of parameters and computation cost as the previous one. Experiments verify that the proposed method is very efficient and effective to distill knowledge from teacher-net to student-net constructing in different ways on various datasets.

1. INTRODUCTION:

Deep neural networks (DNNs) have demonstrated extraordinary success in a variety of fields such as computer vision (Krizhevsky & Hinton, 2012; He et al., 2016) , speech recognition (Hinton et al., 2012) , and natural language processing (Mikolov et al., 2010) . However, DNNs are resource-hungry which hinders their wide deployment to some resource-limited scenarios, especially low-power embedded devices in the emerging Internet-of-Things (IoT) domain. To address this limitation, extensive works have been done to accelerate or compress deep neural networks. Putting those works on designing (Chollet, 2016) or automatically searching efficient network architecture aside (Zoph & Le, 2016) , most studies try to optimize DNNs from four perspectives: network pruning (Han et al., 2016; Li et al., 2016) , network decomposition (Denton et al., 2014; Jaderberg et al., 2014) , network quantization (or low-precision networks) (Gupta et al., 2015; Courbariaux et al., 2016; Rastegari et al., 2016) and knowledge distillation (Hinton et al., 2015; Romero et al., 2015) . Among these method categories, knowledge distillation is somewhat different due to the utilization of information from the pre-trained teacher-net. The concept was proposed by (Bucila et al., 2006; Ba & Caruana, 2014; Hinton et al., 2015) for transferring knowledge from a large "teacher" model to a compact yet efficient "student" model by matching certain statistics between "teacher" and "student". Further research introduced various kinds of matching mechanisms in the field of DNN optimization. The distillation procedure designs a loss function based on the matching mechanisms and enforces the loss during a full training process. Hence, all these methods usually require time-consuming training procedure along with fully annotated large-scale training dataset.

Meanwhile, some network pruning (Li et al., 2016; Liu et al., 2017) and decomposition (Zhang et al., 2016; Kim et al., 2016) methods can produce extremely small networks, but with large accuracy drops so that time-consuming fine-tuning is required for possible accuracy recovery. Usually, it may still not be able to recover the accuracy drops with the original cross-entropy loss due to its low representation capacity. Hence, knowledge distillation may be used to alleviate the problem, since the compact student-net can sometimes be trained to match the performance of the teacher-net. For instance, Crowley et al. (2017) uses cheap group convolutions and pointwise convolutions to build a small student-net and adopts knowledge distillation to transfer knowledge from a full-sized "teacher-net" to the "student-net". However, it still suffers from high training cost. (2) add 1×1 conv-layer at the end of each block of student-net (before ReLU), and align teacher and student by estimating the parameter using least-squared regression;

(3) absorb or merge the added 1×1 conv-layer into the previous conv-layer to obtain final student-net.

As is known, children can learn knowledge concept from adults with few examples. This cognition phenomenon has motivated the development of the few-shot learning (Fei-Fei et al., 2006; Bart & Ullman, 2005) , which aims to learn information about object categories from a few training samples, and focuses more on image classification task. Nevertheless, it inspires people to consider the possibility of knowledge distillation from few samples. Some recent works on knowledge distillation address this problem by constructing "pseudo" training data (Kimura et al., 2018; Lopes et al., 2017) with complicated heuristics and heavy engineering, which are still costly.

This paper proposes a novel and simple three-step method for few-sample knowledge distillation (FSKD) as illustrated in Figure 1 , including student-net design, teacher-student alignment, and absorbing added conv-layer. We assume that both "teacher" and "student" nets have the same feature map sizes at each corresponding block. However, the relatively small student-net can be obtained in various ways, such as pruning/decomposing the teacher-net, and fully redesigned network with random initialization. We add a 1×1 conv-layer at the end of each block of the student-net and align the block-level outputs between "teacher" and "student", which is done by estimating the parameters of the added layer with few samples using least square regression. Since the added 1×1 conv-layers have relatively few parameters, we can get a good approximation from a small number of samples. We further prove that the added 1×1 conv-layer can be absorbed/merged into the previous conv-layer when certain conditions fulfill, so that the new conv-layer has the same number of parameters and computation cost as the older/previous one.

We argue that FSKD has many potential applications, especially when fine-tuning or full training is not feasible in practice. We just name a few such cases below. First, edge devices have limited computing resources, while FSKD offers the possibility of on-device learning to compress deep models with a limited number of samples. Second, FSKD may help software/hardware vendors optimizing the deep models from their customers when full training data is unavailable due to privacy or confidential issues. Third, FSKD enables fast model deployment optimization when there is a strict time budget. Our major contributions can be summarized as follows:

(1) To the best of our knowledge, we are the first to show that knowledge distillation can be done with few samples within minutes on a desktop PC.

(2) The proposed FSKD method is widely applicable not only for fully redesigned student-nets but also for compressed networks from pruning and decomposition-based methods.

(3) We demonstrate significant performance improvement of the student-net by FSKD, comparing to existing distillation techniques on various datasets and network structures.

2. RELATED WORK:

Knowledge Distillation (KD) transfers knowledge from a pre-trained large "teacher" network (or even an ensemble of networks) to a small "student" network, for facilitating the deployment at test time. Originally, this is done by regressing the softmax output of the teacher model (Hinton et al., 2015) . The soft continuous regression loss used here provides richer information than the label based loss, so that the distilled model can be more accurate than training on labeled data with cross-entropy loss. Later, various works have extended this approach by matching other statistics, including intermediate feature responses (Romero et al., 2015; Chen et al., 2016) , gradient (Srinivas & Fleuret, 2018) , distribution (Huang & Wang, 2017) , Gram matrix (Yim et al., 2017) , etc. Deep mutual learning (Zhang et al., 2018) trains a cohort of student-nets and teaches each other collaboratively with model distillation throughout the training process. All these methods require a large amount of data (known as the "transfer set") to transfer the knowledge, whereas we aim to provide a solution with a limited number of samples. We need emphasize that FSKD has a quite different philosophy on aligning intermediate responses to the closest knowledge distillation method FitNet (Romero et al., 2015) . FitNet re-trains the whole student-net with intermediate supervision using a larger amount of data, while FSKD only estimates parameters for the added 1 × 1 conv-layer with few samples. We will verify in experiments that FSKD is not only more efficient but also more accurate than FitNet.

Network Pruning methods obtain a small network by pruning weights from a trained larger network, which can keep the accuracy of the larger model if the prune ratio is set properly. Han et al. (2015) proposes to prune the individual weights that are near zero. Recently, channel pruning has become increasingly popular thanks to its better compatibility with off-the-shelf computing libraries, compared with weights pruning. Different criteria have been proposed to select the channel to be pruned, including norm of weights (Li et al., 2016) , scales of multiplicative coefficients (Liu et al., 2017) , statistics of next layer (Luo et al., 2017) , etc. It is usually required iterative loop between pruning and fine-tuning for achieving better pruning ratio and speedup. Similar gradually adjusting trick is also applied to train very-deep neural networks (Smith et al., 2016) . Meanwhile, Network Decomposition methods try to factorize heavy layers in DNNs into multiple lightweight ones. For instance, it may adopt low-rank decomposition to fully-connection layers (Denton et al., 2014) , and different kinds of tensor decomposition to conv-layers (Zhang et al., 2016; Kim et al., 2016) . However, aggressive network pruning or network decomposition usually lead to large accuracy drops, thus fine-tuning is required to alleviate those drops (Li et al., 2016; Liu et al., 2017; Zhang et al., 2016) .

As aforementioned, KD is more accurate than directly training on labeled data, it is of great interest to explore KD on extremely pruned or decomposed networks, especially under the few-sample setting.

Learning with few samples has been extensively studied under the concept of one-shot or few-shot learning. One category of methods directly model few-shot samples with generative models (Fei-Fei et al., 2006; Lake et al., 2011) , while most others study the problem under the notion of transfer learning (Bart & Ullman, 2005; Ravi & Larochelle, 2017) . In the latter category, meta-learning methods (Vinyals et al., 2016; Finn et al., 2017) solve the problem in a learning to learn fashion, which has been recently gaining momentum due to their application versatility. Most studies are devoted to the image classification task, while it is still less-explored for knowledge distillation from few samples. Recently, some works tried to address this problem. Kimura et al. (2018) 

3. FEW-SAMPLE KNOWLEDGE DISTILLATION (FSKD):

3.1 OVERVIEW Our FSKD method consists of three steps as shown in Figure 1 . First, we design a student-net either by pruning/decomposing the teacher-net, or by fully redesigning a small student-net with random initialization. Second, we add a 1×1 conv-layer at the end of each block of the student-net and align the block-level outputs between "teacher" and "student" by estimating the parameters for the added layer from few samples. Third, we absorb the added 1×1 conv-layer into the previous conv-layer without introducing extra parameters and computations into the student-net.

Two reasons make this idea work efficiently. First, the 1×1 conv-layers have relatively few parameters, which do not require too many data for the estimation. Second, the block-level output from teacher-net provides rich information as shown in FitNet (Romero et al., 2015) . Below, we will first provide the theoretical derivation why the added 1×1 conv-layer could be absorbed/merged into the previous conv-layer. Then we provide details on how we do the block-level output alignment.

3.2. ABSORBABLE 1 × 1 CONV-LAYER:

Let's first give some mathematic notions for different kinds of convolutions before moving to the theoretical derivation. A regular convolution consists of multi-channel and multi-kernel filters which build both cross-channel correlations and spatial correlations. Formally, a regular convolution layer can be represented by a 4-dimensional tensor W ∈ R no×ni×k×k , where n o and n i are the number of output and input channels respectively, and k × k is the squared spatial kernel size. The point-wise (PW) convolution, also known as 1 × 1 convolution (Lin et al., 2014) can be represented by a tensor P ∈ R no×ni×1×1 , which is actually degraded from a 4-dimensional tensor to a 2-dimensional matrix.

The depth-wise (DW) convolution (Chollet, 2016) does per-channel 2D convolution for each input channel, so that it can be represented by a tensor D ∈ R 1×ni×k×k . Due to no-correlation among output channels, it usually follows by a point-wise convolution to model their correlations. This combination (DW + PW) is also named as depth-wise separable convolution by Chollet (2016) .

Theorem 1. A pointwise convolution with tensor Q ∈ R n o ×n i ×1×1 can be absorbed into the previous convolution layer with tensor W ∈ R no×ni×1×1 to obtain the absorbed tensor W = Q • W, where

• is absorbing operator and W ∈ R n o ×ni×k×k if the following conditions are satisfied.

c1. The output channel number of W equals to the input channel number of Q, i.e., n o = n i . c2. No non-linear activation layer like ReLU (Nair & Hinton, 2010) between W and Q.

Due to the space limitation, we put the proof and the detailed form of the absorbing operator in Appendix-A. The number of output channels of W is n o , which is different from that of W (i.e., n o ). It is easy to have the following corollary.

Corollary 1. When the following condition is satisfied for Q, c3. the number of input and output channels of Q equals to the number of output channel of W, i.e.,

n i = n o = n o , Q ∈ R no×no×1×1 ,

the absorbed convolution tensor W has the same parameters and computation cost as

W, i.e. both W , W ∈ R no×ni×k×k .

This condition is required not only for ensuring the same parameter size and computing cost, but also for ensuring current layer output size matching/connectable to next layer input size.

3.3. BLOCK-LEVEL ALIGNMENT AND ABSORBING:

Now we consider the knowledge distillation problem. Suppose X s , X t ∈ R no×d are the block-level output in matrix form for the student-net and teacher-net respectively, where d is the per-channel feature map resolution size. We add a 1 × 1 conv-layer Q at the end of each block of student-net before non-linear activation, which satisfies condition c1 ∼ c3. As Q is degraded to the matrix form, it can be estimated with least squared regression as

EQUATION

where N is the number of samples used, and "*" here means matrix product. The number of parameters of Q is n o × n o , where n o is the number of output channels in the block, which is usually not too large so that we can estimate Q with a limited number of samples.

Suppose there are M corresponding blocks in the teacher and student networks required to align, to achieve our goal, we need minimize the following loss function

EQUATION

where Q j is the tensor for the added 1 × 1 conv-layer of the j-th block. In practice, we optimize this loss with a block-coordinate descent (BCD) algorithm (Hong et al., 2017) , which greedy handles each of the M terms/blocks in Equation 2 in the student-net sequentially as shown in algorithm 1 at Appendix-B, instead of optimizing this loss all together using standard SGD. The BCD algorithm for FSKD has the following advantages:

(1) The BCD algorithm processes each block greedy with a sequential update rule, and each Q can be solved much cheaper with a small number of samples by aligning the block-level responses between teacher and student networks, while SGD considers {Q j } all together which theoretically requires more data.

(2) The alignment procedure is very efficient, which can be usually done within several minutes for the entire network. (3) The alignment procedure itself does not require class label information of input data due to its regression nature. However, if we fully redesign the student-net from scratch with random weights, we may leverage SGD on a few labeled samples to initialize the network.

Our FSKD can still produce significant performance gains over SGD in this case. (4) Our FSKD works extremely well for student-net obtained by aggressively pruning/decomposing the teacher-net. It beats the standard fine-tuning based solution on the number of data required, processing speed, and accuracy of the output student-net.

4. EXPERIMENT:

We perform extensive experiments on different image classification datasets to verify the effectiveness of FSKD on various student-net construction methods. Student-nets can be obtained either from compressing the teacher-net or redesigning network structure with random initialization (termed "zero student network"). For the former case, we evaluate FSKD on three well-known compression methods, filter pruning (Li et al., 2016) , network slimming (Liu et al., 2017) , and network decoupling (Guo et al., 2018) . We implement the code with PyTorch, and conduct experiments on a desktop PC with Intel i7-7700K CPU and one NVidia 1080TI GPU.

4.1. STUDENT NETWORK FROM COMPRESSING TEACHER NETWORK:



FILTER PRUNING:

We first obtain the student-nets using the filter pruning method (Li et al., 2016) , which prunes out conv-filters according to the L 1 norm of their weights. The L 1 norm of filter weights are sorted and the smallest portion of filters will be pruned to reduce the number of filter-channels in a conv-layer.

We make a comprehensive study of VGG-16 (Simonyan & Zisserman, 2015) on CIFAR-10 dataset to evaluate the performance of FSKD along with different configuration settings. Following Li et al. (2016) , we first prune half of the filters in conv1_1, conv4_1, conv4_2, conv4_3, conv5_1, conv5_2, conv5_3 while keeps the other layer unchanged (scheme-A). We also propose another more aggressive pruning scheme named scheme-B, which pruned 10% more filters in the aforementioned layers, and also pruned 20% filters for the remaining layers. Scheme-A prunes 64% of total parameters with 7% accuracy drop. Scheme-B prunes 77% of total parameters with almost 50% accuracy drop. We use those two pruned networks as student-nets in this study. 1We illustrate in Figure 6 in Appendix-C how teacher and student are aligned at block-level. For the few-sample setting, we randomly select 100 (10 for each category) and 500 (50 for each category) images from the CIFAR-10 training set, and keep them fixed in all experiments. Table 1 lists the results of different methods of recovering a pruned network, including FitNet (Romero et al., 2015) , fine-tuning with limited data and full training data (Li et al., 2016) . Note that we optimize FSKD with two algorithms: FSKD-BCD uses the BCD algorithm on block-level and FSKD-SGD optimizes the loss (Equation 2) all together with SGD algorithm. In the BCD algorithm, we do not observe benefit for the iteration number T > 1 over T = 1, so that we set T = 1 in all our following experiments. This is consistent with the finding by (Hong et al., 2017 ) that the convergence is sublinear when each block is minimized exactly (here due to linear structure). Regarding the processing speed, FSKD-BCD can be done in 19.3 seconds for student-net from scheme-B with 500 samples, while FitNet requires 157.3 seconds when converged, which is about 8.1× slower. This verifies our previous claim that FSKD is more efficient than FitNet. It can be seen that in the few-sample setting, FSKD-BCD provides better accuracy recovery than both FitNet and the fine-tuning procedure adopted in Li et al. (2016) . For instance, for scheme-B with only 500 samples, FSKD can recover the accuracy from 47.9% to 91.2%, while few-sample fine-tuning can only recover the accuracy to 83.36%. When full training set available, it will take about 30 minutes for full fine-tuning to reach similar accuracy as FSKD. This demonstrates the big advantages of FSKD over full fine-tuning based solutions.

Figure 2 further studies the performance with different amount of training samples available. It can be observed that our FSKD-BCD keep outperforming FSKD-SGD, FitNet under the same training samples. In particular, FSKD-SGD and FitNet experience a noticeable accuracy drop when the number of samples is less than 100, while FSKD-BCD can still recover the accuracy of the pruned network to a high level. It is also interesting to note that fine-tuning experiences even larger accuracy drops than FitNet when the data amount is limited. This verifies that knowledge distillation methods like FitNet provide richer information than fine-tuning/re-training with label based loss.

As is shown, FSKD-BCD performs better and tends to be more sample-efficient than FSKD-SGD. Therefore, we choose it as the default algorithm, and denote it as FSKD for simplification in the following studies. We further illustrate the per-layer (block) feature responses difference between teacher and student before and after using FSKD in Figure 3a . Before applying FSKD, the correlation between teacher and student is broken due to the aggressive compression. However, after FSKD, the per-layer correlations between teacher and student are restored. This verifies the ability of FSKD for recovering lost information. We do see a decreasing trend with layer depth increased, which is possibly due to error accumulation through multiple convolutional layers. We also show the accuracy change during sequentially block-level alignment in Figure 3b , which clearly demonstrate the effectiveness of our sequentially block-by-block update in the BCD algorithm.

NETWORK SLIMMING:

We then study the student-net from another channel pruning method named network slimming (Liu et al., 2017) , which removes insignificant filter channels and corresponding feature maps using sparsified channel scaling factors. Network slimming consists of three steps: sparse regularized training, pruning and fine-tuning. Here, we replace the time-consuming fine-tuning step with our Channel-prune-ratio Top1-before(%) Top1-after(%) FLOPs(×10 FSKD, and follow the original paper (Liu et al., 2017) to conduct experiments to prune different networks on different datasets. We apply FSKD on networks pruned from VGG-19, ResNet-164, and DenseNet-40 (Huang et al., 2017) , on both CIFAR-10 and CIFAR-100 datasets. Table 2 lists results on CIFAR-10, while Table 3 lists results on CIFAR-100. Note that the channel-prune-ratio (like 70% in Table 2 ) means the portion of channels that are removed in comparison to the total number of channels in the network. It shows that FSKD consistently outperforms FitNet and fine-tuning with a notable margin under the few-sample setting on all evaluated networks and datasets.

NETWORK DECOUPLING:

Network decoupling (Guo et al., 2018) decomposes a regular convolution layer into the sum of several blocks, where each block consists of a depth-wise (DW) convolution layer and a point-wise (PW, 1×1) convolution layer. The ratio of compression increases as the number of blocks decreases, but the accuracy of the compressed model will also drop. Since each decoupled block ends with a 1×1 convolution, we can apply FSKD at the end of each decoupled block.

Following (Guo et al., 2018) , we obtain student-nets by decoupling VGG-16 and ResNet-18 pretrained on ImageNet with different T values, where T stands for the number of DW + PW blocks that a conv-layer decouples out. Figure 7 in Appendix-C illustrates how teacher and student are aligned at block-level in this case. For VGG-16, we also decouple half of the conv-layer with T = 1 and the other half T = 2, and denote the case as "T = 1mix". We evaluate the resulted network performance on the validation set of the ImageNet classification task. We randomly select one image from each of the 1000 classes in ImageNet training set to obtain 1000 samples as our FSKD training set. Table 4 shows the top-1 accuracy of student-net before and after applying FSKD on VGG-16 and ResNet-18.

It is quite interesting to see that in the case of T = 1mix for VGG-16 and T = 2 for ResNet-18, we can recover the accuracy of student-net from nearly random guess (0.12%, 0.21%) to a much higher level (51.3% and 49.5%) with only 1000 samples. In all the other cases, FSKD can recover the accuracy of a highly-compressed network to be comparable with the original network. One possible explanation is that the highly-compressed networks still inherit some representation power from the teacher-net i.e., the depth-wise 3×3 convolution, while lacking the ability to output meaningful predictions due to the inaccurate/degraded 1 × 1 convolution. The FSKD calibrates the 1 × 1 convolution by aligning the block-level responses between "teacher" and "student" so that the lost information in 1 × 1 convolution is compensated, and reasonable recovery is achieved2 .

Top1-before(%) Top1-after (%) GFLOPs Reduced #Param * (×10 

4.2. ZERO STUDENT NETWORK:

Finally, we evaluate FSKD on fully redesigned student-net with a different structure from the teacher and random initialized parameters (named as zero student-net). We conduct experiments on CIFAR-10 and CIFAR-100 with VGG-19 as the teacher-net and a shallower VGG-13 as the student-net. Due to the similar structure between VGG-13 and VGG-19, they can be easily aligned in block-level.

The random initialized network does not contain any information about the training set. Simply training this network using SGD with few samples will lead to poor generalization ability, as shown in Figure 4 . We propose two schemes to combine FSKD in the training procedure: SGD+FSKD and FitNet+FSKD. In the SGD+FSKD case, we first use SGD to train the student-net (without using teacher-net information) on the given few labeled samples with 150 epochs (multi-step learning-rate decay at every 50 epochs from 0.01 to 0.0001), and then apply FSKD to the obtained student-net using the same few-sample set. We repeat these two steps until the training loss converges. In the FitNet+FSKD case, we keep the same few-sample set, and simply replace the SGD with FitNet (using teacher-net information) to add supervision on intermediate responses during training.

We compare the results from four different recovery methods: running SGD until convergence, SGD+FSKD, running FitNet until convergence, and FitNet+FSKD. In order to better simulate the few-sample setting, we do not apply data augmentation to the training set. We randomly pick 100, 200, 500, 1000 samples from the CIFAR-10 training set, and 500, 1000, 1500, 2000 samples from the CIFAR-100 training set, and keep these few-sample sets fixed in this study. Figure 4 shows the comparison results on the four methods and four few-sample sets. It shows that FSKD+SGD takes a big jump over pure SGD, and FSKD+FitNet also takes a big jump over pure FitNet. FSKD+SGD performs much better than FitNet on CIFAR-10, while this is not true on CIFAR-100. There are two possible reasons. First, we did not enable data augmentation so that few-sample SGD is underfitting, which provides much less information than what the student-net can get from the teacher-net in FitNet. Second, CIFAR-100 is much more difficult than CIFAR-10 so that the performance is more sensitive to the number of samples. However, FSKD+SGD can still achieve accuracy on par with FitNet. We should also note here that the zero-student-nets have accuracy gaps with the fully-trained teacher-nets on both CIFAR-10 and CIFAR-100. This is reasonable and acceptable, considering that we did not use data augmentation and trained the model with very few samples. Nevertheless, it still demonstrates the advantages of our FSKD over SGD and FitNet based methods. To further illustrate the benefit of FSKD over SGD, we visualize the convolution kernel (in terms of decoupled pointwise convolutions) before SGD, after SGD, and after SGD+FSKD in Appendix-F.

5. CONCLUSION:

We proposed a novel and simple method for knowledge distillation from few samples (FSKD). The method works for student-nets constructed in various ways, including compression from teachernets and fully redesigned networks with random initialization on various datasets. Experiments demonstrate that FSKD outperforms existing knowledge distillation methods by a large margin in the few-sample setting, while requires significantly less computation budget. This advantage will bring many potential applications and extensions for FSKD. The accuracy of FSKD-BCD versus the number of iterations T is illustrated in Figure 5 , which shows that more iterations do not bring noticeable performance gain. This is because in each iteration, the sub-problem is a linear optimization problem so that we can find exact minimization. This is consistent with the finding by (Hong et al., 2017) . Therefore, in the paper, we only report the accuracy of T = 1 for FSKD-BCD.

APPENDIX-C: ILLUSTRATION OF FSKD ON PRUNING AND DECOUPLING:

Figure 6 illustrates how FSKD work for block-level alignment on the filter pruning (Li et al., 2016) and network slimming (Liu et al., 2017) cases. Figure 7 illustrates how FSKD works for block-level alignment on the network decoupling (Guo et al., 2018) case.

APPENDIX-D: ITERATIVE PRUNING AND FSKD:

Previous works show that one time extremely pruning may yield the pruned network unable to recovery from fine-tuning, while the iteratively pruning and fine-tuning procedure is observed effective to obtain extreme model compression (Han et al., 2016; Li et al., 2016; Liu et al., 2017) .

Inspired by these works, we proposed the iteratively pruning and FSKD procedure as described in algorithm 2 to achieve extremely compression rate. This solution is still much more efficient than iteratively pruning and fine-tuning due to the great efficiency of FSKD over fine-tuning. Table 5 : Performance comparison between FitNet, fine-tuning (Li et al., 2016) , FSKD-BCD/SGD with studentnets from filter pruning of VGG-16 with scheme-C on CIFAR-10. "Full fine-tuning"uses full training data.

Based on this procedure, we extremely prune VGG-16 on CIFAR-10 by 88% total parameters. Table 5 list the results comparison to fine-tuning, FitNet, etc. It verfies the effectiveness of our FSKD on this extremely pruned case.

APPENDIX-E: TRAINING ONLY POINTWISE CONV-LAYER IS ACCURATE ENOUGH:

People may challenge that learning 1 × 1-conv may loss representation power and ask why the added 1 × 1 convolution works so well with such few samples. According to the network decoupling theory (Lemma-1), any regular conv-layer could be decomposed into a sum of depthwise separable blocks, where each depthwise separable block consists of a depthwise (DW) convolution (for spatial correlation modeling) followed by a pointwise (PW) convolution (for cross-channel correlation modeling). The added 1 × 1 conv-layer is absorbed/merged into the previous PW layer finally. The decoupling yields that the number of parameters in PW-layer occupies most (>=80%) parameters of the whole network. We argue that learning 1 × 1-conv is still very powerful, and make a bold hypothesis in subsection 4.1 that PW conv-layer is more critical for performance than DW convlayer. To verify this hypothesis, we make experiments on VGG16 and ResNet50 on CIFAR-10 and CIFAR-100 under below different settings.

(1) We train the network from random initialization with 150 epoches.

(2) We start from a random initialized network (VGG16 or ResNet50), and do full rank decoupling (K = k 2 in Equation 3) so that channels in DW layers are orthogonal, and PW layers are still fully random. Note that Lemma-1 ensures the network before and after decoupling are equivalent (i.e., able to transfer back and force from each other). We keep all the DW-layers fixed (with random orthogonal basis), and train only the PW layers with 150 epochs. We denote this scheme as ND-1*1.

Note that except the setting explicitly described, all the other configurations (including training epochs, hyper-parameters, hardware platform, etc) are kept the same on both experimental cases. Table 6 lists the experimental results on these two cases on both datasets with two different network structures. It is obvious that the 2nd case (ND-1*1) clearly outperforms the 1st case. This verifies our

:

An extremely pruned case "scheme-C" is also provided in Appendix-D.

We thus make a bold hypothesis that point-wise is more critical for performance than depthwise, so that even depthwise

× 3 conv-layers are initialized to be orthogonal from random data, training only pointwise conv-layers could provide enough accurate results. We verify this in Appendix-E.

Figures and Tables:

Figure 1: Three steps of our few-sample knowledge distillation. (1) design student-net from scratch or by compressing teacher-net;(2) add 1×1 conv-layer at the end of each block of student-net (before ReLU), and align teacher and student by estimating the parameter using least-squared regression; (3) absorb or merge the added 1×1 conv-layer into the previous conv-layer to obtain final student-net.

constructs pseudo-examples using the inducing point method, and develops a complicated algorithm to optimize the model and pseudo-examples alternatively. Lopes et al. (2017) records per-layer meta-data for the teacher-net in order to reconstruct a training set, and then adopts a standard training procedure to obtain the student-net. Both are very costly due to the complicated and heavy training procedure. On the contrary, we aim for a simple solution for knowledge distillation from few samples.

Accuracy vs #samples on CIFAR-10. Student-net (a) scheme-A (b) scheme-B by filter pruning. (a) Layer-level output correlation between "teacher" and "student" before and after FSKD on student-nets (scheme-A) by filter pruning. (b) Accuracy change during sequentially block-level alignment.

Accuracy vs #samples for zero student network on (a) CIFAR-10, (b) CIFAR-100.

Figure 5: Accuracy vs #iterations of FSKD-BCD on CIFAR-10. Student-net is scheme-B by filter pruning.

Figure 6: Illustration of FSKD on filter pruning and network slimming. At each block, we copy weights of the unpruned part in teacher-net to student-net, and align the feature maps of student-net to those unpruned feature maps of teacher-net by adding a 1 × 1 convlayer (red-color) with L2-loss. The added 1 × 1 can be merged into the previous conv-layer in student-net.

Performance comparison between FitNet, fine-tuning(Li et al., 2016), FSKD with student-nets from filter pruning of VGG-16 with scheme-A/B on CIFAR-10. "Full fine-tuning" uses full training data.

Table content: <table><tr><td/><td colspan="7">Top1-before(%) Top1-after(%) FLOPs(×10 8 ) Reduced #Param(×10 6 ) Pruned #Samples</td></tr><tr><td>VGG-16</td><td>92.66</td><td>-</td><td>3.11</td><td>-</td><td>15</td><td>-</td><td>-</td></tr><tr><td>Scheme-A + FSKD-BCD</td><td>85.42</td><td>92.37</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>100</td></tr><tr><td>Scheme-A + FSKD-SGD</td><td>85.42</td><td>92.18</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>100</td></tr><tr><td>Scheme-A + FitNet</td><td>85.42</td><td>91.23</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>100</td></tr><tr><td>Scheme-A + FSKD-BCD</td><td>85.42</td><td>92.46</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>500</td></tr><tr><td>Scheme-A + FSKD-SGD</td><td>85.42</td><td>92.42</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>500</td></tr><tr><td>Scheme-A + FitNet</td><td>85.42</td><td>92.13</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>500</td></tr><tr><td>Scheme-A + Fine-tuning</td><td>85.42</td><td>90.25</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>500</td></tr><tr><td>Scheme-A + Full fine-tuning</td><td>85.42</td><td>92.54</td><td>2.06</td><td>34%</td><td>5.3</td><td>64%</td><td>50000</td></tr><tr><td>Scheme-B + FSKD-BCD</td><td>47.90</td><td>90.17</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>100</td></tr><tr><td>Scheme-B + FSKD-SGD</td><td>47.90</td><td>89.41</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>100</td></tr><tr><td>Scheme-B + FitNet</td><td>47.90</td><td>88.76</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>100</td></tr><tr><td>Scheme-B + FSKD-BCD</td><td>47.90</td><td>91.21</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>500</td></tr><tr><td>Scheme-B + FSKD-SGD</td><td>47.90</td><td>90.76</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>500</td></tr><tr><td>Scheme-B + FitNet</td><td>47.90</td><td>90.68</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>500</td></tr><tr><td>Scheme-B + Fine-tuning</td><td>47.90</td><td>83.36</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>500</td></tr><tr><td>Scheme-B + Full fine-tuning</td><td>47.90</td><td>91.53</td><td>1.33</td><td>58%</td><td>3.4</td><td>77%</td><td>50000</td></tr></table>

Performance comparison between FSKD, FitNet and fine-tuning(Liu et al., 2017) on different network structures obtained by network slimming with 100 samples randomly selected from CIFAR-10 training set.

Table content: <table><tr><td>8 ) Reduced #Param(×10 6 ) Pruned</td></tr></table>

Performance comparison between FSKD, FitNet and fine-tuning(Liu et al., 2017) on different network structures obtained by network slimming with 500 samples randomly selected from CIFAR-100 training set.

Table content: <table/>

Performance of FSKD on different student nets obtained by network decoupling VGG-16 and ResNet-18 with different parameters T on ImageNet dataset. " * " here means that parameters from FC-layer are not counted, only those from conv-layers are counted, since decoupling only handles the conv-layers.

Table content: <table/>

Algorithm 2: Iteratively pruning and FSKD Algorithm 1 Data: Teacher-net t, input data {Xi} N i=1 , prune-ratio-list {r k } K k=1 , number of iterations T 2 smax = ∅; 3 for t = 1 : T do Prune s with ratio r k to obtain student-net t;Run FSKD (algorithm 1) with s, t and {Xi} N i=1 , output s ; Evaluation s on validation set to obtain score q k ;

Table content: <table><tr><td>4</td><td>qmax = 0;</td><td/><td/><td/><td/><td/><td/></tr><tr><td>5</td><td>for k = 1 : K do</td><td/><td/><td/><td/><td/><td/></tr><tr><td>6</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>9</td><td>if q k &gt; qmqx then</td><td/><td/><td/><td/><td/><td/></tr><tr><td>10</td><td>qmax = q k ;</td><td/><td/><td/><td/><td/><td/></tr><tr><td>11</td><td>smax = s ;</td><td/><td/><td/><td/><td/><td/></tr><tr><td>12</td><td>end</td><td/><td/><td/><td/><td/><td/></tr><tr><td>13</td><td>end</td><td/><td/><td/><td/><td/><td/></tr><tr><td>14</td><td colspan="2">Update teacher t = smax;</td><td/><td/><td/><td/><td/></tr><tr><td>15 end</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">16 Result: final student-net smax.</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="8">Top1-before(%) Top1-after(%) FLOPs(×10 8 ) Reduced #Param(×10 6 ) Pruned #Samples</td></tr><tr><td colspan="2">VGG-16</td><td>92.66</td><td>-</td><td>3.11</td><td>-</td><td>15</td><td>-</td><td>-</td></tr><tr><td colspan="2">Scheme-C + FSKD-BCD</td><td>13.05</td><td>89.55</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>100</td></tr><tr><td colspan="2">Scheme-C + FSKD-SGD</td><td>13.05</td><td>89.01</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>100</td></tr><tr><td colspan="2">Scheme-C + FitNet</td><td>13.05</td><td>85.09</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>100</td></tr><tr><td colspan="2">Scheme-C + FSKD-BCD</td><td>13.05</td><td>90.41</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>500</td></tr><tr><td colspan="2">Scheme-C + FSKD-SGD</td><td>13.05</td><td>90.12</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>500</td></tr><tr><td colspan="2">Scheme-C + FitNet</td><td>13.05</td><td>88.31</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>500</td></tr><tr><td colspan="2">Scheme-C + Fine-tuning</td><td>13.05</td><td>78.13</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>500</td></tr><tr><td colspan="2">Scheme-C + Full fine-tuning</td><td>13.05</td><td>90.77</td><td>1.09</td><td>65%</td><td>1.8</td><td>88%</td><td>50000</td></tr></table>

Peer review (iclr2019):
 Model distillation can be tricky and in my own experience can take a lot of samples (albeit unlabeled, so cheaper and more readily available), as well as time to train. This simple trick seems to be doing quite well at training students quickly with few samples. However, it departs from most student-teacher training that find its primary purpose by actually outperforming students trained from scratch (on the full dataset without time constraints). This trick does not outperform this baseline, so its emphasis is entirely on quick and cheap. However, it*s unclear to me how often that is actually necessary and I don*t think the paper makes a compelling case in this regard. I am borderline on this work and could probably be swayed either way. Strengths: - It*s a very simple and fast technique. As I will cover in a later bullet point (under weaknesses), the paper does not make it clear why this type of model distillation is that useful (since it doesn*t improve the student model over full fine-tuning, unlike most student-teacher work). However, the reason why I do see some potential for this paper is because there might be a use case in quickly being able to adapt a pretrained network. It is very common to start from a pretrained model and then attach a new loss and fine-tune. Under this paradigm, it is harder to make architectural adjustments, since you are starting from a finite set of pretrained models made available by other folks (or accept the cost of re-training one yourself). However, it is unclear how careful one needs to treat the pretrained model if more fine-tuning is going to occur. If for instance you could just remove layers, drop some channels, glue it all together, and then that model would still be reasonable as a pretrained model since the fine-tuning stage could tidy everything up, then this method would not be useful in this situation. - The fact that least squares solvers can be used at each stage, without the need for a final end-to-end fine-tune is interesting. - It is good that the paper demonstrates improvements coupled with three separate compression techniques (Li et al., Liu et al., Guo et al.). - The paper is technically thorough. - It*s good that the method is evaluated on different styles of networks (VGG, ResNet, DenseNet). Weaknesses: - Limited application because it only makes the distillation faster and cheaper. The primary goal of student-teacher training in literature is to outperform a student trained from scratch by the wisdom of the teacher. It ties into this notion that networks are grossly over-parameterized, but perhaps that is where the training magic comes from. Student-teacher training acknowledges this and tries to find a way to benefit from the over-parameterized training and still end up with a small model. I think the same motivation is used for work in low-rank decomposition and many other network compression methods. However, in Table 1 the "full fine-tune" model is actually the clear winner and presented almost as an upper bound here, so the only benefit this paper presents is quick and cheap model distillation, not better models. Because of this, I think this paper needs to spend more time making a case for why this is so important. - Since this technique doesn*t outperform full fine-tuning, the goal of this work is much more focused on pure model compression. This could put emphasis on reducing model size, RAM usage reduction, or FLOPS reduction. The paper focuses on the last one, which is an important one as it correlates fairly well with power (the biggest constraint in most on-device scenarios). However, it would be great if the paper gave a broader comparison with compression technique that may have slightly different focus, such as low-rank decomposition. Size and memory usage could be included as columns in tables like 1, along with a few of these methods. - Does it work for aggressive compression? The paper presents mostly modest reductions (30-50%). I thin even if accuracy takes a hit, it could still work to various degrees. From what I can see, the biggest reduction is in Table 4, but FSKD is used throughout this table, so there is no comparison for aggressive compression with other techniques. - The method requires appropriate blocks to line up. If you completely re-design a network, it is not as straightforward as regular student-teacher training. Even the zero-student method requires the same number of channels at certain block ends and it is unclear from the experiments how robust this is. Actually, a bit more analysis into the zero student would be great. For instance, it*s very interesting how you randomly initialize (let*s say 3x3) kernels, and then the final kernels are actually just linear combinations of these - so, will they look random or will they look fairly good? What if this was done at the initial layer where we can visualize the filters, will they look smooth or not? Other comments: - A comparison with "Deep Mutual Learning" might be relevant (Zhang et al.). I think there are also some papers on gradually adjusting neural network architectures (by adding/dropping layers/channels) that are not addressed but seem relevant. I didn*t read this recently, but perhaps "Gradual DropIn of Layers to Train Very Deep Neural Networks" could be relevant. There is at least one more like this that I*ve seen that I can*t seem to find now. - It could be more clear in the tables exactly what cited method is. For instance, in Table 1, does "Fine-tuning" (without FitNet/FSKD) correspond to the work of Li et al. (2016)? I think this should be made more clear, for instance by including a citation in the table for the correct row. Right now, at a glance, it would seem that these results are only comparing against prior work when it compares to FitNet, but as I read further, I understood that*s not the case. - The paper could use a visual aid for explaining pruning/slimming/decoupling. Minor comments: - page 4, "due to that too much hyper-parameters" - page 4, "each of the M term" -> "terms" - page 6, methods like FitNet provides" -> "provide"