DD‐YOLO: An object detection method combining knowledge distillation and Differentiable Architecture Search

IET Computer VisionVolume 16, Issue 5 p. 418-430 ORIGINAL RESEARCHOpen Access DD-YOLO: An object detection method combining knowledge distillation and Differentiable Architecture Search Zhiqiang Xing, Zhiqiang Xing School of Information Science and Technology, North China University of Technology, Beijing, ChinaSearch for more papers by this authorXi Chen, Xi Chen orcid.org/0000-0001-9223-0302 School of Information Science and Technology, North China University of Technology, Beijing, ChinaSearch for more papers by this authorFengqian Pang, Corresponding Author Fengqian Pang fqpang@ncut.edu.cn School of Information Science and Technology, North China University of Technology, Beijing, China Correspondence Fengqian Pang, School of Information Science and Technology, North China University of Technology, Beijing 100144, China. Email: fqpang@ncut.edu.cnSearch for more papers by this author Zhiqiang Xing, Zhiqiang Xing School of Information Science and Technology, North China University of Technology, Beijing, ChinaSearch for more papers by this authorXi Chen, Xi Chen orcid.org/0000-0001-9223-0302 School of Information Science and Technology, North China University of Technology, Beijing, ChinaSearch for more papers by this authorFengqian Pang, Corresponding Author Fengqian Pang fqpang@ncut.edu.cn School of Information Science and Technology, North China University of Technology, Beijing, China Correspondence Fengqian Pang, School of Information Science and Technology, North China University of Technology, Beijing 100144, China. Email: fqpang@ncut.edu.cnSearch for more papers by this author First published: 02 April 2022 https://doi.org/10.1049/cvi2.12097AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Although YOLOv4 has a high detection accuracy, its backbone network CSPDarknet53 requires a large number of parameters, which reduces the inference speed of the detection network to some extent. To address this drawback, a high-efficiency detection network search strategy is proposed. First, the concept of Differentiable Architecture Search (DARTS) is used to search networks with a small number of parameters on the COCO2017 datasets; then, the backbone of YOLOv4 is redesigned by stacking cells. This strategy can reduce the number of parameters of the network. Second, the authors use ResNet101 as the teacher network and DD-YOLO searched by DARTS as the student network to refine the knowledge model and improve the detection accuracy of the model. The proposed model is evaluated by conducting experiments on the COCO2017 test-dev datasets. The results show that DD-YOLO achieves 43.5% mean average precision and 2.3% model accuracy improvement. Meanwhile, the model complexity can be reduced to 61.4% of the original. Moreover, compared with YOLOv4, DD-YOLO is more suitable for mobile deployment. 1 INTRODUCTION In recent years, with the rapid development of deep learning algorithms, object detection is widely used in various aspects. Object detection algorithms can be divided into one-stage object detection algorithms, such as a single-shot multibox detector (SSD) [1] and You Only Look Once (YOLOv1-YOLOv4) [2-5], and two-stage object detection algorithms, such as region-based convolution neural networks(RCNNs) [6], fast region-based convolution neural networks (Fast-RCNNs) [7], faster region-based convolution neural networks (Faster-RCNNs) [8], and mask region-based convolution neural networks (Mask-RCNNs) [9]. The YOLO object detection algorithm provides real-time detection with high inference speed, making it one of the most widely used excellent algorithms. Many current detection algorithms consume considerable computational resource, and their training on a single GPU leads to extremely slow convergence of the network. YOLOv4 requires low computing power and can be trained on a single GPU, such as GTX 1080 Ti and GeForce RTX 2080Ti. Hence, this study focusses on YOLOv4. The backbone network of YOLOv4 is CSPDarknet53 and contains a large number of parameters. In the feature extraction, the convolution block needs to be multiplexed multiple times, greatly limiting the detection speed. Therefore, it is difficult to deploy YOLOv4 on a mobile terminal or an embedded device because it causes a low inference speed. Therefore, designing a lightweight network model [10-12] and lightweight algorithm [13] has gained research attention. Most of these models are based on expert experience, which makes their execution time-consuming and labour-intensive. In addition, it is difficult to effectively adjust their parameters. In this study, we design a model that can be adapted to the requirements of various tasks. The search strategy of neural architecture search (NAS) [14] does not rely on expert experience to design lightweight networks. In this method, new network structures are designed by stacking a small number of cells. Thus, a lightweight model and automatic adjustment of hyperparameters are realised. Traditional NAS is based on reinforcement learning and evolutionary algorithms. Many search spaces are relatively discrete, which makes model training also discrete, thereby preventing the use of the gradient descent method to optimise the model. Differentiable Architecture Search (DARTS) [15] weakens the search space into a continuous space, allowing the use of the gradient descent method to optimise the model. The computing unit of DARTS is a cell, and through cell stacking, a new convolutional neural network (CNN) or recurrent neural network (RNN) can be designed. Therefore, the backbone network of YOLOv4 is designed by using cell stacking to reduce the number of model parameters. However, the lightweight network designed through the DARTS strategy reduces accuracy. This study uses knowledge distillation to improve the accuracy of the network. LeCun et al. [16] first proposed model compression by which relatively unimportant weights in the network structure are discarded, thus improving the generalisation ability of the network. In 2015, Hinton et al. proposed a training method for knowledge distillation by which the information of the teacher network is transferred to the student network so as to improve the student network accuracy. Thereafter, the upsurge of research on knowledge distillation began. In 2016, Han et al. [17] reviewed neural network model compression, offering a more comprehensive and systematic understanding of model compression. Wang et al. [18] applied knowledge distillation in YOLO to achieve detection speed improvement. Dai et al. [19] proposed a new method for discriminative instance extraction, achieving the state-of-the-art performance on two datasets. To address the above-mentioned problems of reducing the model parameters and improving the model accuracy, this study proposes an object detection method combining knowledge distillation and NAS. The method has three main contributions: We use the knowledge distillation method to optimise the search of NAS network structure and determine the optimal object identification model for complex environments. The results of the experiment conducted on the COCO [18] datasets demonstrate a reduction in model complexity to 61.4% and a 2.3% improvement in the mean average precision of the proposed model compared with the original YOLOv4 model. Our experiment show that the frame per seconds of DD-YOLO reaches 58, which can achieve the effect of real-time detection. 2 RELATED WORK 2.1 Object detection algorithm The use of deep learning methods usually reduces the time required for feature extraction. Deep learning object detection algorithms are mainly divided into one-stage and two-stage object detection algorithms. One-stage object detection algorithms can obtain all convolution results through a convolution network. By contrast, two-stage object detection algorithms predict the region of interest through a convolution network and then input the feature to another convolution network. Therefore, the inference speed of the two-stage object detection algorithms will be slightly inferior. One-stage object detection algorithms are further divided into algorithms with anchor frame detectors [20, 21] and without anchor frame detectors [22, 23]. Although one- and two-stage object detection algorithms have achieved great success, they are trained with self-supervised methods. Their training is inseparable from the collection and annotation of datasets, so they cannot be applied to large-scale categories. Lampert et al. [24] first proposed the concept of zero-shot learning, which uses the principle of attribute transfer between categories to predict unknown categories. Combining zero-sample classification with the object detection framework can achieve zero-sample detection, which allows the model to be effectively applied to larger scale categories. The zero-shot detection (ZSD) problem was first introduced by Rahman et al. [25], with a new loss function designed to simultaneously identify and localise novel classes of objects. Applying the ZSD problem to one- and two-stage detection algorithms can enhance the ability to identify and locate new targets in complex environments. Zhao et al. [26] proposed a generative adversarial network based on intersection over union (IoU) perception for the ZSD task. Yan et al. [27] proposed a semantics-preserving graph propagation model based on the ZSD task by using Faster RCNN as the research framework. It provides a good research direction for the detection of invisible categories in images. 2.2 Search strategy based on network architecture At present, the automatic search network structure shows good performance in many fields, such as image classification [28] and object detection [29]. Most NAS strategies require large calculations to search for realising high-performance networks. For example, Zoph et al. [30] used a reinforcement learning method to obtain a high-performance network structure with more than 500 GPUs on CIFAR-10 and ImageNet. Therefore, many acceleration methods have been advanced to solve the problem of NAS's high consumption of computational resources. Liu [31] proposed a hierarchical search space that uses the optimised search algorithm. Brock [32] tested the weighting and performance of a single network module. Elsken et al. [33] proposed a method of sharing or inheritance between network modules. However, in practical applications, the search efficiency of NAS is lower than that of random network search. In recent years, some scholars have proposed to use the shared network parameters for training in search spaces, which results in the invalidity of the NAS method. Benyahia et al. [34] found that if a large network contains multiple weight-sharing models, the overall model performance will worsen with training, a phenomenon called multi-model forgetting. They reduced the forgetting of multiple models by introducing weight plasticity loss. Zhang et al. [35] effectively solved the problem caused by weight sharing by designing a novel search-based architecture selection method, the novelty search-based architecture selection (NSAS) loss function. Experiments on multiple datasets demonstrated that NSAS can offset the performance degradation caused by NAS forgetting, providing a direction for future research. Li et al. [36] introduced the concept of knowledge distillation, which allows to perform end-to-end network architecture search by considerably reducing the problems caused by weight sharing to NAS. Applying the knowledge distillation method to NAS can reduce not only the number of network parameters but also the calculation time of the model. 2.3 Training method based on knowledge distillation Knowledge distillation is one of the commonly used model compression methods. The existing mainstream knowledge distillation methods can be broadly classified into two categories. In the first category, the student network is optimised using the classified soft labels and real labels generated by the teacher network. Jimmy Ba et al. [37] first proposed this type of method. Hinton et al. [38] referred to the teacher network output as a soft target. However, as the teacher network deepens, relying solely on soft labels to train the network will be problematic. Hence, the second category of knowledge distillation method was proposed [39-43]. The method uses the middle layer characteristics of the teacher network to guide the student network. Junho Yim et al. [44] proposed a method to imitate the characteristics of the multi-layer teacher network during student network training. Wang et al. [45] proposed a step-by-step distillation method, wherein the student network is allowed to learn the information in the middle layer of the teacher network in steps. This approach effectively reduces the difficulty of student network optimisation. When the steps increase, the performance of the student network differs more from that of the teacher network. Most studies believe that the knowledge of the network model mainly depends on the network parameters. 3 COMBINING KNOWLEDGE DISTILLATION AND DARTS FOR OBJECT DETECTION In this section, first, the overall flowchart of DD-YOLO is presented. Then, the DARTS unit and the YOLOv4 network structure are outlined. Finally, the search strategy of DARTS and the knowledge distillation method are analysed in detail. 3.1 DD-YOLO network structure To reduce the number of YOLOv4 parameters and balance the object detection accuracy of the network, an object detection method, DD-YOLO, which combines knowledge distillation and DARTS is proposed. First, the DARTS search strategy is applied to stack cells and replace the YOLOv4 extraction network CSPDarknet53. Then, the knowledge distillation method is used to train the cell-stacking network so that the accuracy of the detection model can be improved. The overall flowchart of DD-YOLO is shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint DD-YOLO overall flowchart Figure 1 consists of two parts: the network structure of DD-YOLO and the network structure of knowledge distillation. In the network structure of DD-YOLO, the YOLOv4 algorithm is selected for object detection. This algorithm consists of four basic modules: (1) CBM: Conv + BN + Mish activation function composition; (2) CBL: Conv + BN + Leaky_Relu activation function composition; (3) Concat: tensor splicing and expanding dimension; and (4) SPP [46]: this adopts the maximum pooling method of 1 × 1, 5 × 5, 9 × 9, and 13 × 13 to expand the receptive field. The input image (416 × 416 × 3) is first subjected to Focus and CBM operations before feature extraction to achieve scaling of the channel and output size. The Focus operation is shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Focus operation The feature map after scaling is extracted through the network Cell0–Cell13; the search unit structure of Cell0–Cell13 is shown in Figure 3. FIGURE 3Open in figure viewerPowerPoint Search unit of Cell0–Cell13 In Figure 3, S0 and S1 represent two feature maps. The feature map sizes and channels searched by DARTS on COCO2017 are shown in Table 1. TABLE 1. Feature map size and channels Feature map Output size Channels Input 416 × 416 3 Focus 208 × 208 12 Conv + BN 208 × 208 64 S0/S1 208 × 208 112 Cell0 104 × 104 144 Cell1 104 × 104 144 Cell2 104 × 104 144 Cell3 104 × 104 144 Cell4 104 × 104 288 Cell5 104 × 104 288 Cell6 104 × 104 288 Cell7 52 × 52 576 Cell8 52 × 52 576 Cell9 52 × 52 576 Cell10 52 × 52 576 Cell11 26 × 26 1152 Cell12 26 × 26 1152 Cell13 26 × 26 1152 Conv + BN 13 × 13 1024 YOLO Neck is a layer added between the backbone network and YOLO Head. The SPP module and the FPN + PAN [47] structure added by YOLO Neck can realise feature integration of different scales. YOLO Head divides the predicted feature map into grids of sizes (13 × 13), (26 × 26), and (52 × 52) for realising the classification of large and small objects. The YOLOv4 loss function consists of three components: classification loss, regression loss, and confidence loss. The algorithm is optimised for regression loss, replacing the mean square error (MSE) in YOLOv3 with complete intersection over union (CIoU). CIoU is calculated as follows: L CIOU = 1 − I O U + ρ 2 b , b g t c 2 + ∂ V (1) v = 4 π 2 arctan w g t h g t − arctan w h 2 (2) ∂ = v ( 1 − I O U ) + v (3) In Equation (1), b $b$ is the Euclidean distance between the centre point of the calculated prediction box and the real box. c is the diagonal length of the minimum box containing the prediction box and the real box. In Equation (2), w g t ${w}^{gt}$ represents the width of the real frame, h g t ${h}^{gt}$ represents the height of the real frame, and w $w$ represents the width of the predicted frame. In Equation (3), h is the height of the prediction frame, ∂ $\partial $ is the equivalent to weight, and v $v$ is used to measure the aspect ratio of the two test frames. The change in v $v$ can predict the degree of overlap between the real frame and the predicted frame. In the second part of the knowledge distillation structure, ResNet101 is selected for the teacher network used for distillation, and Cell0–Cell13 is a student network. Among them, Cell0, Cell7, and Cell11, indicated in pink, represent the reduction cells. The rest of the green cells represent the normal cells. The training process of the student network in knowledge distillation is divided into two parts. One part of the training process is the same as the teacher network training. The soft target obtained by the teacher network is subjected to cross-entropy loss with the result of the Softmax output by the student network. The training weight is λ $\lambda $ . By contrast, the student network undergoes cross-entropy loss through the output of Softmax and the hard target. The training weight is 1 − λ $\lambda $ . Finally, different weights are assigned to the cross-entropy loss, and after adding them, the total loss function is calculated. The above is a systematic overview of the DD-YOLO structure and knowledge distillation structure. The following sections provide the overview of the DARTS searching strategy on the COCO datasets and the knowledge distillation loss function as a replacement of the DD-YOLO classification loss. 3.2 DARTS search strategy 3.2.1 Network search space A computing unit is a feature-to-feature mapping structure, which is stacked to form a convolutional network. The search space refers to the set containing all operations of the convolutional network; that is, the computing unit is a directed acyclic graph composed of N-ordered node sequences. x ( j ) = ∑ i < j o ( i , j ) x ( i ) (4)where o ( i , j ) ${o}^{(i,j)}\,$ represents some operation between the current node, denoted by x ( j ) ${x}^{(j)}$ , and the predecessor node, denoted by x ( j ) ${x}^{(j)}$ . The search operation finds the directed edge with the largest weight in the computing unit and deletes the directed edge with a smaller weight. This process reduces the number of network parameters and calculations required. To enrich the search space of the conversion operation combination, it is assumed that there are seven characteristic nodes in a unit, whose structure is shown in Figure 4. FIGURE 4Open in figure viewerPowerPoint Unit internal connection structure In Figure 4, fout(i) represents the current output node; fin(i−1) and fin(i−2) represent the current input nodes; 0, 1, and 2, and 3 represents the current intermediate nodes. There are 14 conversion path search spaces of the computing unit, and each conversion operation OP is a set. The set mainly includes skip connection, pooling, and convolution. A total of eight DARTS operations are defined in this paper, as shown in Table 2. TABLE 2. Eight operations of DARTS Operations None max_pool_3 × 3 avg_pool_3 × 3 skip_connect sep_conv_3 × 3 sep_conv_5 × 3 dil_conv_3 × 3 dil_conv_5 × 5 In the operation of the calculation unit, separate convolution and dilated convolution are used instead of the standard convolution, thereby reducing the number of calculation parameters. The search space of DARTS is any combination of 8 operations and 14 conversion paths. Therefore, the search parameter of the convolution unit is an 8 × 14 matrix. Considering that some feature layers perform down-sampling operations, the search unit is divided into two types: a normal cell and a reduction cell. 3.2.2 Continuous relaxation and optimisation The DARTS is based on the concept of converting the search space into a continuous space, providing the directed edges with various operation options. During the training process, each operation option is assigned a weighting factor. All conversion operations are performed when feature nodes are converted by the directed edges. The conversion operations are multiplied by the corresponding weight coefficients to obtain the final result. To better normalise the weight coefficients, a set of weight coefficients is normalised to a probability vector, which is implemented with the Softmax function. The conversion is defined as follows. O P mix ( x ) = ∑ i = 1 n e a i ∑ j = 1 n e a j O P i ( X ) (5)where X denotes the current feature map, O P mix ( x ) $O{P}_{\text{mix}}(x)$ shows the new feature map, n is the number of OP operations in the set, O P i $O{P}_{i}$ represents any operation in the set of operations, and a i ${a}_{i}$ means the weight coefficient of the corresponding O P i $O{P}_{i}$ . With Equation (5), the discrete operation selection is turned into differentiable weights for learning. The weight coefficients a i ${a}_{i}$ are regarded as additional parameters of the search network, which can be adjusted by gradient descent optimisation, thus changing the network structure of the convolutional unit. Because the weight coefficients a i ${a}_{i\,}\,$ are differentiable, the training and searching of the convolution unit are performed alternately during the training process. The dataset is divided into the training set and validation set. The training set is used to optimise the weight of O P i $O{P}_{i}$ to obtain the training loss Ltrain. The validation set is used to optimise the weight coefficient a i ${a}_{i}$ to obtain the validation loss Lval. In the search process of the network, only the operation with the largest weight coefficient is retained. Equation (5) is transformed as follows: O P mix ( x ) = e a argmax ∑ j = 1 n e a j O P argmax ( X ) (6) Its specific algorithm flow is shown in Table 3. TABLE 3. Convolutional unit network structure search process Convolutional unit network structure search process Create an operation set O P set $O{P}_{\text{set}}$ for each operation edge of the network structure While does not converge: According to Ltrain, use the gradient descent method to update the weight of the operation w o p ${w}_{op}$ According to Lval, use the gradient descent method to update the weight of the operation a o p ${a}_{op}$ Replace the mixed operation ∑ i = 1 n e a i ∑ j = 1 n e a j O P i ( X ) $\sum\limits _{i=1}^{n}\,\frac{{e}^{{a}_{i}}}{{\sum }_{j=1}^{n}{e}^{{a}_{j}}}O{P}_{i}(X)$ with the operation e a argmax ∑ j = 1 n e a j O P argmax ( X ) $\frac{{e}^{{a}_{\mathrm{argmax}}}}{{\sum }_{j=1}^{n}{e}^{{a}_{j}}}O{P}_{\mathrm{argmax}}(X)$ 3.2.3 Generating a continuous network After training and optimisation, each operation edge in the convolutional unit network is assigned a weight value. The operation edges are filtered according to the weight, and the reduction cell and normal cell searched by DARTS on the COCO2017 datasets are obtained. In this paper, the backbone network of DD-YOLO is designed for 14 cell stacks, including 11 normal cells and 3 reduction cells. The convolutional unit network structures of the normal and reduction cells are shown in Figure 5. FIGURE 5Open in figure viewerPowerPoint Normal cell search network structure (a) and reduction cell search network structure (b) 3.3 Training method of knowledge distillation Knowledge distillation improves the performance of the student network by providing additional information beyond the label. The additional information is the classification probability vector output by the teacher network through the Softmax function. In knowledge distillation, the classification probability vector and the real label are used to supervise the learning of the student network during its training. This paper uses the knowledge distillation loss function to replace the DD-YOLO classification loss function, and its effectiveness is determined through ablation experiments. The loss function is defined as follows: L k d = ∂ T 2 ⋅ K L div log softmax S cls T , softmax T cls T + ( 1 − ∂ ) C E S cls , Y (7)where ∂ $\partial $ represents the harmonic coefficient, K L div $K{L}_{\text{div}}$ represents the divergence loss, T $T$ is the temperature, S cls ${S}_{\text{cls}}$ represents the student network classification, T cls ${T}_{\text{cls}}$ represents the teacher network classification, Y represents the true label value, and CE represents the cross-entropy loss function. 4 EXPERIMENTS AND ANALYSIS This section describes the details of the three parts of the experimental process: hardware and parameter settings of the experiment, model evaluation criteria, and comparative experiment results. The comparison of the experimental results involves determining qualitative visual effects, the ablation experiments with and without the use of knowledge distillation in DARTS; comparison of the number of experimental parameters required by the network as well as the experimental precision and recall ratios; and objective quantitative evaluation with other state-of-the-art detection methods. 4.1 Hardware and network parameter settings The hardware device configuration used for the experiments is TITAN XP 12G graphics card and Linux operating system. For the experimental study, the public dataset COCO2017 is selected to form the training set, test set, and validation set. The stochastic gradient descent (SGD) optimiser is selected for the optimisation of the model with an initial learning rate of 0.01 and a momentum value of 0.9. The entire network is trained for 300 epochs. 4.2 Model evaluation criteria Although the main evaluation criterion for the model in this experiment is the mean average precision (mAP), precision, recall, and mean average recall (mAR) are also used as additional criteria to evaluate the model. The formulas for precision, recall, and category average recall rates are as follows: A c c u r a c y = T P + T N T P + T N + F P + F N (8) P r e c i s i o n = T P T P + F P (9) R e c a l l = T P T P + F N (10) m A P = ∑ i = 1 N A P i N (11) m A R = ∑ i = 1 O A R i O (12) In the above equations, TP represents the positive samples that are correctly classified as positive; TN represents the negative samples that are correctly classified as negative; FP represents the negative samples that are incorrectly classified as positive; FN represents the positive samples that are incorrectly classified as negative. N and O indicate the number of total categories in the dataset; A P i $A{P}_{i}$ denotes the AP value of category i; A R i $A{R}_{i}$ denotes the AR value of category i. 4.3 Model visual recognition results To effectively test the DD-YOLO detection algorithm proposed in this study, four images are randomly selected from the test set for recognition. Figure 6 shows the results of the four randomly selected images from the COCO2017 dataset for classification regression detection. FIGURE 6Open in figure viewerPowerPoint DD-YOLO visual recognition results As shown in the images, DD-YOLO correctly recognises the single and multiple objects while displaying the type of objects and the confidence levels. Table 4 shows the average run time and the frame per second of the model; it compares the effect of implementing the knowledge distillation training method in DD-YOLO on the detection time for a batch size of 10. TABLE 4. Comparison of the average run time and the frame per second Backbone Distillation Feature size Average run time (s) Frame per second (Hz) CSPDarkNet53 - 416×416 0.0205 48 DARTS No 416 × 416 0.0186 54 DARTS Yes 416 × 416 0.0172 58 The results confirm that the DD-YOLO classification model using the knowledge distillation method has improved image detection speed, with the frame per second reaching 58. The confusion matrix is a table that explains the prediction results of the classification model in machine learning. The 80 object categories of the C