Teacher−student framework‐based knowledge transfer with incremental block‐wise retraining

Electronics LettersVolume 55, Issue 20 p. 1090-1093 Image and vision processing and display technologyFree Access Teacher−student framework-based knowledge transfer with incremental block-wise retraining J.-H. Bae, Corresponding Author J.-H. Bae baejh@etri.re.kr orcid.org/0000-0002-0035-5261 KSB Convergence Research Department, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea J.-H. Bae: Also with Daegu Catholic University, Gyeongsan, Gyeongbuk, Republic of Korea.Search for more papers by this authorJ. Yim, J. Yim Laboratory, DronID Inc., Seoul, Republic of KoreaSearch for more papers by this authorJ. Kim, J. Kim Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of KoreaSearch for more papers by this author J.-H. Bae, Corresponding Author J.-H. Bae baejh@etri.re.kr orcid.org/0000-0002-0035-5261 KSB Convergence Research Department, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea J.-H. Bae: Also with Daegu Catholic University, Gyeongsan, Gyeongbuk, Republic of Korea.Search for more papers by this authorJ. Yim, J. Yim Laboratory, DronID Inc., Seoul, Republic of KoreaSearch for more papers by this authorJ. Kim, J. Kim Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of KoreaSearch for more papers by this author First published: 01 October 2019 https://doi.org/10.1049/el.2019.1186 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This Letter proposes a stage-wise training method that uses block-wise retraining to transfer the useful knowledge of a pre-trained deep residual network (ResNet) in a teacher−student framework (TSF). To achieve this, flow-based hidden information transfer and hierarchically supervised retraining of the information are alternatively implemented from bottom to top between teacher and student ResNets in the TSF. To evaluate the effectiveness of the proposed method, the authors used well-known image data sets Canadian Institute For Advanced Research (CIFAR)-10, CIFAR-100, and street view house number. The results showed that the flow-based bottom-up knowledge transfer combined with incremental block-wise retraining can provide the improved small student ResNet with higher accuracy than the deep teacher ResNet. This approach will help extend the use of deep neural network models to limited computing environments. Introduction Knowledge distillation is a useful technique for exploiting the inherent and rich information of a well-optimised deep neural network (DNN). Generally, complex DNN models with large numbers of learning parameters provide high-quality performance, but there are potential limitations when directly reusing a pre-trained DNN architecture in applications with restricted computing resources, such as in an internet-of-things environment [1]. Therefore, efficient knowledge extraction and knowledge transfer are important methods for extending the application of DNN models to limited computing environments. To accomplish this objective, we propose a teacher−student framework (TSF)-based knowledge transfer approach combined with block-wise retraining. The TSF-based approach has the advantage of adaptively generating student models of various sizes depending on the application. For the TSF in the proposed method, both teacher and student models adopt a residual network (ResNet), which is a well-known recent DNN model [2]. In the first stage, flow-based distilled knowledge [3] in a pre-trained teacher ResNet is incrementally transferred to a student ResNet from bottom to top based on each residual block in which the feature map size changes. In this stage, block-wise retraining for each residual block is also performed on the student ResNet in a supervised manner with true labels. In other words, flow-based knowledge transfer and supervised block-wise retraining for each residual block are implemented alternately between the teacher ResNet and the student ResNet until incremental bottom-up knowledge transfer is completed. Finally, in the second stage, the additional teacher knowledge, which is known as the soft output distribution over classes [4], is transferred from the teacher to the student after completely delivering the multiple flow-based information extracted from the hierarchy in the previous stage. In the next section, we describe the proposed stage-wise knowledge transfer procedure coupled with block-wise retraining in detail. Proposed method Flow-based information across two different layers, which is defined as the flow-of-solution procedure (FSP) [3], is applied to TSF-based knowledge transfer using incremental block-wise retraining. The FSP is mathematically defined as the Gramian matrix , which computes the inner products between the feature maps across two different layers. For the k FSP matrices of a teacher ResNet {} and those of a student ResNet {}, the proposed knowledge transfer procedure in stage 1 can be described as follows: Step 1(a): estimate weights using by solving the following norm-based optimisation problem (Fig. 1a). (1)where x denotes the input images, λ is a weight decay for regularisation, denotes the fixed weights of the teacher ResNet, and denotes the weights to be trained in the student ResNet. Weight matrix , which consists of weights from the input layer to the first residual block, is randomly initialised. Fig 1Open in figure viewerPowerPoint TSF-based knowledge transfer procedure coupled with incremental block-wise retraining in stage 1 a Step 1(a): first flow-based knowledge transfer in the TSF b Step 1(b): supervised block-wise retraining using only the student c Step 2(a): second flow-based knowledge transfer in the TSF d Step 2(b): supervised block-wise retraining using only the student e Step 3(a): third flow-based knowledge transfer in the TSF f Step 3(b): supervised block-wise retraining using only the student Step 1(b): After loading into the student ResNet with the initial weights, perform supervised training with true labels on the student ResNet to obtain weights by adding a global average pooling (GAP) layer and a fully connected (FC) layer at the top of the first residual block, which consists of 2n residual modules (n = 1, 2, …, as shown in Fig. 1b). Note that the teacher ResNet does not participate in this step. Step 2(a): Remove the old GAP and FC layers in the student ResNet and create an additional pair between the teacher and student ResNets (Fig. 1c). Subsequently, estimate weights using the refined weights in Step 1(b) and the pair that was previously generated in Step 1(a) as follows: (2)where indicates the weighting factor for each loss term and the initial weights . Here, is composed of the randomly initialised weights of the second residual block in the student ResNet. Since we assume that each loss term is equally significant for knowledge transfer, we set = 1 for all steps. Step 2(b): After initially loading weights , perform supervised training using a new GAP layer and a FC layer in the student ResNet without the teacher ResNet, as shown in Fig. 1d. Step k(a): Generate an additional pair. Then, estimate weights using the (k − 1)th pre-trained . (3)where the initial weights and denotes the randomly initialised weights of the kth residual block of the student ResNet. Note that the previous k − 1 pairs of are also included in this step and that has the form of an empty set (Ø). Step k(b): To obtain the weights from the extracted in step k(a), perform supervised training without the teacher ResNet by adding new GAP and FC layers at the top of the kth residual block. For example, Figs. 1e and f illustrate the third flow-based knowledge transfer and the corresponding block-wise retraining when k = 3, respectively. The previous steps in stage 1 are then repeated until k reaches the number of residual blocks () required for knowledge transfer in the TSF. Therefore, the flow-based knowledge transfer of steps 1(a), 2(a), …, k(a) is hierarchically performed in conjunction with the incremental bottom-up retraining of steps 1(b), 2(b), …, k(b) between the teacher and the student ResNets. Finally, after completing stage 1, stage 2 of the proposed method implements the soft output distribution-based training using a softening factor (called the ‘temperature’ T [4]) in the final output layer. According to the recommendations in [4], we set T = 3 for all the experiments. A description of the proposed approach for the stage-wise knowledge transfer is summarised in Algorithm 1. Algorithm 1.Stage-wise knowledge transfer with incremental block-wise retraining in the TSF Initialisation: k = 0 and = Ø, select pairs of FSP matrices. Stage 1: Increment k by 1 and perform the following steps until the last weights are found. (1) Estimate the kth weights with (3) in Step k(a) using the (k − 1)th pre-trained and k pairs of . (2) Perform supervised block-wise retraining using the previously estimated to obtain refined weights in Step k(b). Stage 2: After initially loading the weights estimated from Stage 1, perform soft output distribution-based training in the TSF. Experimental results This Letter considered a ResNet structure with three residual blocks (K0 = 3) and a total of 6k + 2 stacked weighted layers (k = 1, 2, …), as reported in the Canadian Institute For Advanced Research (CIFAR)-10 case [2]. We first experimentally evaluated the proposed method for the well-known CIFAR-10 and CIFAR-100 benchmark images [5] with data pre-processing in [2], where a ResNet structure with {16, 32, 64} filters and {64, 128, 256} filters were considered for CIFAR-10 and CIFAR-100, respectively. We prepared 26-layer teacher ResNets with 91.6% accuracy for CIFAR-10 and 74.26% accuracy for CIFAR-100, where both cases were pre-trained using the same learning rates of 0.1, 0.01, and 0.001 for 32,000, 48,000, and 64,000 iterations, respectively, and a mini-batch size of 256. For the proposed method, flow-based knowledge transfer for each residual block was performed step-by-step using a learning rate of 0.1 over 21,000 iterations (3,000 iterations for , 5,000 iterations for , and 13,000 iterations for ) for CIFAR-10 and over 32,000 iterations (5,000 iterations for , 9,000 iterations for , and 18,000 iterations for ) for CIFAR-100. In addition, to obtain each refined , where i = 1, 2, and 3, each step of the incremental block-wise retraining was performed for the same number of iterations: 4000 iterations for CIFAR-10 and 5000 iterations for CIFAR-100. After completing stage 1, stage 2 was implemented over 64,000 iterations, as used in the training procedure of the teacher ResNet. In the case of the existing flow-based method, according to [3], we performed FSP-based training of stage 1 over 21,000 iterations, followed by a normal training of stage 2 using true labels over 64,000 iterations. Comparative results for each method on CIFAR-10 and CIFAR-100 are presented in Tables 1 and 2, respectively, where the accuracy results are calculated as the average of the top three accuracies over five trials. As the proposed method adopted additional teacher knowledge in stage 2, we also applied the soft output teacher knowledge [4] to stage 2 of the original flow-based method. As presented in Table 1, the proposed method clearly exhibits a classification accuracy that is better than that of other methods. In addition, we observe that the proposed method provides better initial weights for training stage 2 than both the FSP-based training with the soft output teacher knowledge and the original FSP-based training method. Furthermore, the resultant small 14-layer student ResNet using the proposed method demonstrated even higher performance than the deep 26-layer teacher ResNet for both CIFAR-10 and CIFAR-100. Table 1. Performance comparison results on CIFAR-10 when the pre-trained 26-layer teacher ResNet with an accuracy of 91.6% was used Methods Classification accuracy, % First Second Third Average original 14-layer student ResNet 90.49 90.54 90.21 90.41 FSP-based training method [3] 91.11 90.96 91.06 91.04 FSP-based training [3] with the soft output 91.62 91.59 91.44 91.55 Proposed knowledge transfer method 91.76 91.96 91.66 91.79 Table 2. Performance comparison results on CIFAR-100 when the pre-trained 26-layer teacher ResNet with an accuracy of 74.26% was used Methods Classification accuracy, % First Second Third Average original 14-layer student ResNet 72.69 72.69 72.57 72.65 FSP-based training method [3] 74.17 74.57 74.53 74.42 FSP-based training [3] with the soft output 74.81 74.93 74.74 74.82 proposed knowledge transfer method 75.13 75.32 75.15 75.2 Fig 2Open in figure viewerPowerPoint Classification accuracies on SVHN for 14-layer student ResNets using each method when the pre-trained 26-layer teacher ResNet with an accuracy of 96.39% was used Next, Fig. 2 presents the classification accuracies for 14-layer student ResNets using the three knowledge transfer methods on the street view house numbers (SVHNs) [6], which is also a common benchmark data set. In this experiment, we prepared a 26-layer teacher ResNet with {64, 128, 256} filters that achieved a 96.39% accuracy using a mini-batch size of 100 and learning rates of 0.1, 0.01, and 0.001 for 18,000, 27,000, and 36,000 iterations, respectively. In addition, for all knowledge transfer methods, we used the parameter settings of the CIFAR-10 case in Table 1, but employed different training iterations for stage 2. In other words, stage 2 followed the same training procedure up to 36,000 iterations as in the case of the 26-layer teacher ResNet. The accuracy results in Fig. 2 are also given as the average of the top three accuracies over five trials for all methods. As expected, all the trained 14-layer student ResNets using the three knowledge transfer methods performed better than the original student ResNets without any teacher knowledge. However, the proposed training approach outperformed the other two knowledge transfer methods and clearly provided an accuracy higher than that of the 26-layer teacher ResNet, although student ResNet had only 53.84% of the number of layers of the teacher ResNet. Conclusion In this study, we proposed a stage-wise knowledge transfer method coupled with incremental block-wise retraining in which hidden knowledge and the soft output knowledge of the teacher were utilised for the TSF. Unlike the original method, which transmits several flow-based streams of knowledge simultaneously, the proposed method is combined with block-wise retraining to gradually transfer multiple flow-based information streams hierarchically. As it integrates supervised block-wise retraining for each residual block into the intermediate process of the flow-based knowledge transfer, the student ResNet can acquire teacher knowledge more precisely through a block-wise refinement of the teacher information. The experimental results using the well-known benchmark data sets CIFAR-10, CIFAR-100, and SVHN indicate that the proposed method can deliver useful teacher information from a complex deep teacher ResNet to a small student ResNet more accurately and efficiently than other methods considered in this study. Acknowledgments This work was supported by the National Research Council of Science & Technology (NST) grant by the Korean government (MSIP) (grant CRC-15-05-ETRI). References 1Yun, J., Jang, B.-J.: ‘Ambient light backscatter communication for IoT applications’, J. Korea Electromagn. Eng. Soc. (JEES), 2016, 16, (4), pp. 214– 218, https://doi.org/10.5515/JKIEES.2016.16.4.214 (https://doi/org/10.5515/JKIEES.2016.16.4.214) 2He, K., Zhang, X., Ren, S. et. al.,: ‘ Deep residual learning for image recognition’. Proc. IEEE Conf. Computer Vision Pattern Recognition (CVPR), Las Vegas, USA, June–July 2016, pp. 770– 778 3Yim, J., Joo, D., Bae, J.-H. et. al.,: ‘ A gift from knowledge distillation: fast optimization, network minimization, and transfer learning’. Proc. IEEE Conf. Computer Vision Pattern Recognition (CVPR), Honolulu, USA, July 2017, pp. 7130– 7138 4Hinton, G., Vinyals, O., Dean, J.: ‘ Distilling the knowledge in a neural network’. Deep Learning Representation Learning Workshop, Neural Information Processing Systems (NeurIPS), Montreal, Canada, 2015, pp. 1– 19 5Krizhevsky, A., Hinton, G.: ‘ Learning multiple layers of features from tiny images’. Master's thesis, Department of Computer Science, University of Toronto, 2009 6Netzer, Y., Wang, T., Coates, A. et. al.,: ‘ Reading digits in natural images with unsupervised feature learning’. Deep Learning Unsupervised Feature Learning Workshop, Neural Information Processing Systems (NeurIPS), Granada, Spain, 2011, pp. 1– 9 Volume55, Issue20October 2019Pages 1090-1093 FiguresReferencesRelatedInformation