# Real-Time Detection of Insulator Defects with Channel Pruning and Channel Distillation
## 5 Channel Distillation  
Knowledge distillation has significant applications in computer vision, particularly in the field of object detection, where it is used to reduce model parameters and improve the accuracy of existing models [31]. After channel pruning, to quickly restore the accuracy of the model, we assist the training of the pruned model through channel distillation. Unlike traditional knowledge distillation (KD), the channel distillation (CD) method transfers channel information from the teacher to the student [32]. In SENet [33], the channel-wise attention mechanism enables the model to learn the weights for each channel, which are then multiplied with the original channels. This process enhances the features of important channels while diminishing those of less significant ones, making the extracted features more directional and improving the network's predictive performance. The weights for each channel are calculated as follows:  
\[w_{c}=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{h=1}^{W}u_{c}(i,j) \tag{10}\]  
The variable \(w_{c}\) represents the weight of the \(c  
utilize Global Average Pooling (GAP) to assess the importance of the feature mapping for each channel, representing the attention information of each channel. These attention data are then treated as knowledge. Both the student and the teacher compute the attention information from their respective feature maps. The teacher then guides the student in learning and acquiring this attention information. In this way, the student gains insight into the teacher's channel-specific attention details, thereby enhancing its performance. Typically, the layers between the teacher and the student models do not align. For simplicity, we apply channel distillation only between networks where spatial resolution is reduced. Moreover, if the number of channels does not match, we adopt the FitNets approach, using \(1\times 1\) convolution to upscale dimensions [35]. First, we upscale the student's feature mapping to match the number of channels in the teacher model, followed by channel distillation. The formula for channel distillation (CD) loss is defined as follows:  
\[CD(s,t)=\frac{\sum_{i=1}^{n}\sum_{j=1}^{c}{(w_{s}^{ij}-w_{t}^{ij})}^{2}}{n \times c} \tag{11}\]  
In the provided formula, \(CD(s,t)\) represents the \(CD\) loss between the student and the teacher. The term \(w_{ij}\) denotes the weight of the \(j^{th}\) channel for the \(i^{th}\) sample, and \(c\) denotes the number of channels. After conducting channel distillation across different channels, we proceed with guided knowledge distillation (GKD) following the softmax predictions. Figure 12 illustrates the structure of our distillation process. GKD is designed based on the foundational principles of knowledge distillation (KD) [36]. The core concept of KD involves calculating the distribution of predictions between a teacher model and a student model. By progressively minimizing the discrepancy between these two distributions, the output distribution of the student model becomes similar to that of the teacher model. The formula for KD is as follows:  
\[p=softmax(\frac{a}{T}) \tag{12}\]  
\[KD(s,t)=\frac{\sum_{i=1}^{n}KL(p_{s}^{i},p_{t}^{i})}{n} \tag{13}\]  
In this context, \(p\) represents the probability distribution computed by the logit function and \(T\) denotes the temperature parameter. Employing a higher \(T\) value results in a softer class probability distribution. Here, \(n\) refers to the batch size. \(KD(s,t)\) represents the mean _Kullback-Leibler_ divergence (KL) between \(p_{s}^{i}\) and \(p_{t}^{i}\). Figure 12: The structure of our distillation process. Although the teacher network is more accurate than the student network, the teacher model still exhibits some prediction errors during batch processing, leading to significant issues. When the teacher makes incorrect predictions, this erroneous knowledge is transferred to the student, worsening the student's performance.