# A New Knowledge-Distillation-Based Method for Detecting Conveyor Belt Defects
## 2 Materials and Methods
### A New Method for Knowledge Distillation Detection of Conveyor Belt Defects Based on YOLOv5
#### 2.3.3 Design of Knowledge Distillation Network Based on YOLOv5  
In the field of object detection, the traditional method of knowledge distillation is to extract the feature map in the middle layer of the teacher network and the student network and calculate the distillation loss for the whole feature map. However, the network tends to learn the features of the target to be detected during training, while in the task of conveyor-belt-damage detection, the conveyor belt scratches and defects only account for a small part of the whole picture. At this time, a large amount of background information is regarded as a noisy part relative to the damage area, which negatively affects the training of the network. To address this problem, this paper chose to adopt a knowledge distillation strategy based on fine-grained feature simulation. The student network only simulates the teacher network for the area around the anchor frame instead of learning the features of the teacher network on the whole feature map, thus avoiding the impact of a large amount of useless background information on the network training. The flow chart of the knowledge distillation method based on the fine-grained feature simulation is shown in Figure 8. In this paper, we used YOLOv5m as the teacher network, and we used YOLOv5n and its pruned model, YOLOv5n-slim, as the student model. To localize the distillation targets, for each GT box, the IOU between it and all anchor boxes was first calculated to form a \(W\times H\times K\) IOU mapping graph, where \(W\) and \(H\) are the width and height of the feature map, respectively, and \(K\) denotes \(K\) pre-defined anchor boxes. Then, we found the maximum value of IOU \(M=\max(M)\), multiplied it by the threshold factor \(\psi\), and obtained the filter \(F=\psi*\text{M}\). Finally, we used \(F\) to filter all IOU mappings, keeping only the positions larger than F and using the OR operation to obtain the feature map of size \(W\times H\). The final fine-grained simulation mask \(I\) is obtained by merging all the GT boxes. When \(\psi=0\), the generated mask included all the positions on the feature map, and when \(\psi=1\), no position was retained. Different masks can be obtained by changing the value of \(\psi\). During the training process, to ensure the distillation effect, it is necessary to keep the student network and teacher network feature map sizes consistent, to which a feature adaptation layer is added after the feature map of the student network to solve the problem of mismatch between the student network and the teacher network feature map channels. Finally, the deviations of these anchor frames in the student network and the teacher network were obtained by combining the mask information as the simulated losses and were added to the loss function of the distillation training. Figure 7: Model-pruning algorithm process. We define \(s\) as the bootstrap feature map of the student network and \(t\) as the feature map of the corresponding teacher network. For each approaching target anchor location \((i,j)\) on the feature map with width \(W\) and height \(H\), the loss function between the student network and the teacher network is as follows:  
\[l=\sum_{c=1}^{C}(f_{\text{adap}}(s)_{ijc}-t_{ijc})^{2} \tag{9}\]  
combined with the mask information \(I\) to obtain the simulated loss function:  
\[L_{\text{{imitation}}}=\frac{1}{2N_{p}}\sum_{i=1}^{W}\sum_{j=1}^{H}\sum_{c=1}^{ C}I_{ij}\Big{(}f_{adap}(s)_{ijc}-t_{ijc}\Big{)}^{2} \tag{10}\]  
\[N_{p}=\sum_{i=1}^{W}\sum_{j=1}^{H}I_{ij} \tag{11}\]  
where \(N_{p}\) is the number of positive points in the mask and \(f_{adap}(\cdot)\) is the adaptation function that unifies the size of the feature map. The loss function of the final student model is:  
\[L=L_{gt}+\lambda L_{\text{{imitation}}} \tag{12}\]  
where \(L_{gt}\) is the detection loss of the student network on real data and \(\lambda\) is the weighting factor of the simulated loss.