# A Defect Detection Model for Industrial Products Based on Attention and Knowledge Distillation

Authors: Ze-Kai Zhang; Ming-Le Zhou; Rui Shao; Min Li; Gang Li
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Industrial quality detection is one of the important fields in machine vision. Big data analysis, the Internet of Things, edge computing, and other technologies are widely used in industrial quality detection. Studying an industrial detection algorithm that can be organically combined with the Internet of Things and edge computing is imminent. Deep learning methods in industrial quality detection have been widely proposed recently. However, due to the particularity of industrial scenarios, the existing deep learning-based general object detection methods have shortcomings in industrial applications. This study designs two isomorphic industrial detection models to solve these problems: T-model and S-model. Both proposed models combine swin-transformer with convolution in the backbone and design a residual fusion path. In the neck, this study designs a dual attention module to improve feature fusion. Second, this study presents a knowledge distiller based on the dual attention module to improve the detection accuracy of the lightweight S-model. According to the analysis of the experimental results on four public industrial defect detection datasets, the model in this study is more advantageous in industrial defect detection.


---

## Body

## 1. Introduction

Quality detection is an important task in the industrial production process, which is of great significance to protect the personal safety of users and avoid economic losses. In industrial quality detection, big data analysis technology, the Internet of Things, edge computing, and other technologies are widely used. Studying an industrial detection algorithm that can be organically combined with the Internet of Things and edge computing is imminent. Early machine vision algorithms used manual feature selection and trained classifiers to identify defect features. This approach relies too much on the robustness of the extracted features, resulting in much time-consuming development.In recent years, convolutional neural networks (CNNs) have rapidly developed in image classification, object detection, and image segmentation. However, due to the particularity of industrial defect detection scenarios, deep learning has not been applied widely. As shown in Figure1, the brightness, colour, object size, and background discrimination of industrial pictures differ from natural scenes. Specifically, the industrial surface defect detection datasets scale is relatively small, unlike large-scale general datasets such as ImageNet, PASCAL VOC2007/2012, and COCO in classical computer vision tasks. Moreover, the industrial datasets differ significantly from the public datasets in the number of samples, sample scale, and proportion of positive and negative samples. This difference leads to less application of general-purpose object detectors in industrial product surface defect detection. In addition, most industrial detection scenarios are offline and require a light model, while existing general object detection models are challenging to meet actual needs.Figure 1
Comparison of industrial detection dataset with COCO dataset. The first row is the picture of the PCB defect dataset. The second row is the COCO dataset pictures. The third row is a picture of aluminium defects. It can be seen that in the field of industrial quality detection, there are many types of small objects, and the positive and negative samples are not balanced, making detection difficult.This study proposes a novel surface defect detection model for industrial products. The model consists of two submodels, T-model and S-model. T-model has a large depth and high detection accuracy and is suitable for scenarios with no obvious requirements for speed. S-model has a small depth and high speed and is more suitable for edge computing scenarios with high-speed requirements. Both models consist of a backbone, neck, and detector. In the backbone, this study combines CNN and swin-transformer. CNN extracts local information in shallow features, swin-transformer extracts global information in deep layers, and each layer outputs weighted features through a unified residual path. In neck, this study designs a dual attention structure to focus on the features of object regions. In detector, this study designs multiple predictions heads on T-model and S-model.In order to improve the detection effect of the lightweight S-model, a knowledge distiller is presented in this study. It contains the foreground attention-guided distillation of objects and global distillation. The knowledge distiller can significantly improve the detection accuracy of the S-model without extra overhead.Overall, this study organically combines some existing techniques to create two detection models. An attention module is independently designed inside the model, and a knowledge distiller is designed to transfer the teacher network’s knowledge for the lightweight network’s guided training. The detection model designed in this study can not only be used for defect detection but also has a wide range of application scenarios in the direction of processing a large amount of image data in the cloud platform combined with the Internet of Things.The main contributions of this study are listed as follows:(1)
This study proposes a novel model for surface defect detection of industrial products. Compared with the traditional CNN-based target detection methods, the model combines CNN and swin-transformer, which significantly improves the accuracy and can be used for application deployment in many image data analyses, edge computing, and other scenarios.(2)
This study designs two isomorphic object detection models. In backbone, this study extracts local and global important features by combining CNN and swin-transformer and designs a unified residual path to fuse features at different levels. In the neck, the attention module is designed to improve the effect of feature fusion. At the same time, this study designs the attention module to the decoupling detection head to improve detection accuracy.(3)
This study designs a novel attention-guided distillation strategy. The distiller uses the dual attention module to guide the generation of attention region features. The distiller transfers the knowledge of the T-model into the S-model, instructing the S-model to learn the T-model. The distiller in this study can obtain a lightweight, high-accuracy, and fast industrial quality detection model.


## 2. Related Work

### 2.1. Object Detection

In recent years, CNN-based object detection algorithms have been used in many domains. CNN-based object detection algorithms are generally divided into two categories: one stage and two stage. One-stage algorithms include YOLO [1–6], SSD [7], RetinaNet [8], and DSSD[9]. The methods are to directly divide the input image into multiple 1 × 1 grids, where each cell is responsible for detecting objects whose centre points fall within the grid. The methods significantly improve the detection speed, but the accuracy is slow. The two-stage algorithms include Rcnn [10], FastRcnn [11], FasterRcnn [12], and MaskRcnn [13]. These detection algorithms generate boxes via RPN, and the second-level detector uses boxes to conclude. The methods are better at detecting, but the speed is very low.The one-stage object detector usually contains three parts: backbone, neck, and detector. The backbone is generally composed of multiple groups of convolutions for feature extraction. The well-known backbones include ResNet [14], ResNext [15], VGG [16], DenseNet [17], MobileNet [18], CSPDarkNet [19], and EfficentNet [20], etc. The neck fuses feature maps at different levels in the backbone to enhance the semantic and fine-grained features. Typical structures of neck mainly include FPN [21], PANet [22], Bi-FPN [20], etc. Two types of detectors commonly used in one-stage object detection are as follows: coupled and decoupled.


### 2.2. Attention

The attention mechanism is to make the network pay more attention to the area of the object and ignore the unimportant areas. Its essence is to use the relevant features to learn the weight distribution and then apply the learned weight to the original features. Attention makes the network pay more attention to the target object. Classical attention networks include SENet [23], SKNet [24], ResNext [15], CBAM [25], and self-attention [26].


### 2.3. Vision-Transformer

The transformer [26] is an attention-based encoder-decoder architecture in deep learning. Compared with CNN, vision transformer (Vit) [27] can obtain more refined global attention features and achieve good performance on multiple benchmarks such as ImageNet, COCO, and ADE20k. However, it also has some drawbacks. First, the range of object scales for visual detection varies greatly, and the performance of the Vit [27] may not be optimal in different scenarios. Second, if the image resolution is high, transformer [26] requires much computation. Swin-transformer [28] solves this problem by shifting window partitions to calculate self-attention.


### 2.4. Knowledge Distillation

Knowledge distillation is a widely used method for model compression. Knowledge distillation is to transfer the knowledge of the T-model into S-model to improve the accuracy of lightweight models without adding extra computation. Knowledge distillation methods can be roughly categorized into response based [29–32], feature based [33, 34], and relation based [35]. Response-based methods use the output of the last layer of the teacher network to imitate the teacher’s final prediction. This method is simple and efficient, but it relies on the output of the last layer and cannot make the student model obtain the supervision of the middle layer of the teacher model. Feature-based methods use the features in the middle layer of the teacher model to guide the student network to perform feature selection. However, two drawbacks need to be solved urgently. One is choosing the intermediate layer, and the other is matching the feature representation between the intermediate layer and the guiding layer if the layers’ sizes are different. Relation-based methods take advantage of the inner product of the features between the two layers and employ the teacher structure as knowledge to guide the student model.


### 2.5. Application in Big Data, Industrial Internet of Things, Defect Detection, and Other Scenarios

Wang et al. proposed a deep learning model [36] combining GRU and LSTM and modeled the crack width of the dam, which can effectively predict the change of dam defects. Chen et al. proposed a training method for CNN and proposed a two-layer parallel training (BPT-CNN) architecture [37] in a distributed computing environment. BPT-CNN effectively improves the training performance of CNN, saves training time while maintaining accuracy, and has wide application fields.In addition, there are many excellent survey proposed in the fields of big data, industrial Internet of things, etc. Pu et al. proposed an automatic fetal ultrasound standard plane recognition (FUSPR) [38] based on deep learning in an Industrial Internet of Things (IIoT) Environment. Cao et al. proposed a novel BERT-based deep space network (BDSTN) [39] to learn the demand pattern of taxis. Chen et al. combined CNN and LSTM to propose multiple closed spatiotemporal CNNs (MGSTC) [40] for traffic flow prediction; Wang et al. [41] proposed the application of big data technology to data mining, data analysis, and data sharing in large amounts of data, and to create huge economic benefits by using the potential value of data. Zhang et al. [41] summarized the existing blockchain-based systems and applications, which have broad application prospects in different data processing and transmission scenarios. Zhang et al. proposed a combined method of Weber local descriptor (WLD) and local binary pattern (LBP) for seam carving forgery detection [42]. To sum up, some detection methods can be combined with existing technologies such as CNN, not only in defect detection scenarios, but also in different application scenarios such as IoT and big data analysis.


## 2.1. Object Detection

In recent years, CNN-based object detection algorithms have been used in many domains. CNN-based object detection algorithms are generally divided into two categories: one stage and two stage. One-stage algorithms include YOLO [1–6], SSD [7], RetinaNet [8], and DSSD[9]. The methods are to directly divide the input image into multiple 1 × 1 grids, where each cell is responsible for detecting objects whose centre points fall within the grid. The methods significantly improve the detection speed, but the accuracy is slow. The two-stage algorithms include Rcnn [10], FastRcnn [11], FasterRcnn [12], and MaskRcnn [13]. These detection algorithms generate boxes via RPN, and the second-level detector uses boxes to conclude. The methods are better at detecting, but the speed is very low.The one-stage object detector usually contains three parts: backbone, neck, and detector. The backbone is generally composed of multiple groups of convolutions for feature extraction. The well-known backbones include ResNet [14], ResNext [15], VGG [16], DenseNet [17], MobileNet [18], CSPDarkNet [19], and EfficentNet [20], etc. The neck fuses feature maps at different levels in the backbone to enhance the semantic and fine-grained features. Typical structures of neck mainly include FPN [21], PANet [22], Bi-FPN [20], etc. Two types of detectors commonly used in one-stage object detection are as follows: coupled and decoupled.


## 2.2. Attention

The attention mechanism is to make the network pay more attention to the area of the object and ignore the unimportant areas. Its essence is to use the relevant features to learn the weight distribution and then apply the learned weight to the original features. Attention makes the network pay more attention to the target object. Classical attention networks include SENet [23], SKNet [24], ResNext [15], CBAM [25], and self-attention [26].


## 2.3. Vision-Transformer

The transformer [26] is an attention-based encoder-decoder architecture in deep learning. Compared with CNN, vision transformer (Vit) [27] can obtain more refined global attention features and achieve good performance on multiple benchmarks such as ImageNet, COCO, and ADE20k. However, it also has some drawbacks. First, the range of object scales for visual detection varies greatly, and the performance of the Vit [27] may not be optimal in different scenarios. Second, if the image resolution is high, transformer [26] requires much computation. Swin-transformer [28] solves this problem by shifting window partitions to calculate self-attention.


## 2.4. Knowledge Distillation

Knowledge distillation is a widely used method for model compression. Knowledge distillation is to transfer the knowledge of the T-model into S-model to improve the accuracy of lightweight models without adding extra computation. Knowledge distillation methods can be roughly categorized into response based [29–32], feature based [33, 34], and relation based [35]. Response-based methods use the output of the last layer of the teacher network to imitate the teacher’s final prediction. This method is simple and efficient, but it relies on the output of the last layer and cannot make the student model obtain the supervision of the middle layer of the teacher model. Feature-based methods use the features in the middle layer of the teacher model to guide the student network to perform feature selection. However, two drawbacks need to be solved urgently. One is choosing the intermediate layer, and the other is matching the feature representation between the intermediate layer and the guiding layer if the layers’ sizes are different. Relation-based methods take advantage of the inner product of the features between the two layers and employ the teacher structure as knowledge to guide the student model.


## 2.5. Application in Big Data, Industrial Internet of Things, Defect Detection, and Other Scenarios

Wang et al. proposed a deep learning model [36] combining GRU and LSTM and modeled the crack width of the dam, which can effectively predict the change of dam defects. Chen et al. proposed a training method for CNN and proposed a two-layer parallel training (BPT-CNN) architecture [37] in a distributed computing environment. BPT-CNN effectively improves the training performance of CNN, saves training time while maintaining accuracy, and has wide application fields.In addition, there are many excellent survey proposed in the fields of big data, industrial Internet of things, etc. Pu et al. proposed an automatic fetal ultrasound standard plane recognition (FUSPR) [38] based on deep learning in an Industrial Internet of Things (IIoT) Environment. Cao et al. proposed a novel BERT-based deep space network (BDSTN) [39] to learn the demand pattern of taxis. Chen et al. combined CNN and LSTM to propose multiple closed spatiotemporal CNNs (MGSTC) [40] for traffic flow prediction; Wang et al. [41] proposed the application of big data technology to data mining, data analysis, and data sharing in large amounts of data, and to create huge economic benefits by using the potential value of data. Zhang et al. [41] summarized the existing blockchain-based systems and applications, which have broad application prospects in different data processing and transmission scenarios. Zhang et al. proposed a combined method of Weber local descriptor (WLD) and local binary pattern (LBP) for seam carving forgery detection [42]. To sum up, some detection methods can be combined with existing technologies such as CNN, not only in defect detection scenarios, but also in different application scenarios such as IoT and big data analysis.


## 3. Method

The flowchart of the proposed model is shown in Figure2. The model consists of two submodels: T-model and S-model. In order to improve the detection effect of the lightweight S-model, this study designs a knowledge distiller and a dual attention module. By generating attention features from the pretrained T-model, the S-model training process is guided to learn the features from the T-model. In this way, the S-model reaches or even exceeds the detection effect of the T-model.Figure 2
The flowchart of the model in this study. First, the dataset is expanded and enhanced by geometric and photometric transformations. Then, the data is input into the pretrained T-model and S-model, respectively, and the T-model output features guide the S-model training.The structure of the T-model and S-model is shown in Figure3. Both models propose a novel backbone combining convolutional layers with swin-transformer [28] to extract finer-grained image features. This study also adds weighted residual connection paths in the backbone. In the neck, a double-tower structure and dual attention modules are designed to improve the effect of feature fusion. Finally, multiple decoupled detectors are used to detect objects of different scales.Figure 3
T-model and S-model. The two models are similar in structure, and the depth of the S-model network is much smaller than the T-model. The blue block-B is the Conv layer, the red block-S is the swin-transformer layer, the purple block-N is the feature fusion layer with the dual attention module added, and the detector is the decoupling detection head.The main formula symbol table used in the rest of the method of this study is annotated as Table1.Table 1
The main formula symbol used in the rest of the method.
Formula symbol annotationLocationxi: the original input featureFormula (1)wi: the learnable weight of a single feature mapwj: the weight of any feature map to be accumulated.F: original input featureFormulas (2)–(4)McF: channel attentionMSF: spatial attentionF′: the superposition of channel attention and spatial attention and multiplied by the original input F.⊗: multiply by bitFi,j,kT: the characteristics of teachersFormula (5)Fi,j,kS: the characteristics of studentsf: converting Fi,j,kS to the same data dimension as Fi,j,kTH, W: specify the height and width of the featureC: channelsx, y: the horizontal and vertical coordinates of the area objectFormula (6)GT: the area position of the real frameMF: the process of generating attentionFormulas (9) and (10)F: the original feature mapT: distillation temperature, T = 20Lfg: the distillation loss of foreground and backgroundFormula (11)Masky,z: foreground mask1−Masky,z: background maskScaleMasky,z: scale maskAttentionMask: attention maskFx,y,zT−fFx,y,zS2: the difference between teacher feature and student featureMFx,y,zT: the attention feature map of the teacher modelFormula (13)MfFx,y,zS: the attention feature map of the student modelLfg: the distillation loss of foreground and backgroundFormula (17)LA: attention mask lossLG: distillation loss of the entire feature mapQFL: QFocal lossLCIOU: CIou lossα,β,γ,δ,ε: the weight parameters of the balance lossW: the updated weight parameterFormula (18)wk: the weight parameter before the updateLi: the value of each lossn: the number of weight parameters


### 3.1. S-Model and T-Model

#### 3.1.1. Backbone

Most general-purpose object detectors are based on CNN and employ large-scale convolution kernels in the first layer of the backbone to increase the receptive field. However, more detailed information will be lost with the increase of convolutional layers. Swin-transformer [28] uses window self-attention to significantly reduce computation load and extract small-scaled features. Therefore, this study designs a novel backbone based on swin-transformer [28].As shown in Figure3, our backbone includes block-B, block-S, SPP, and weighted residual fusion paths. The first layer selects a 3×3 convolution kernel to extract fine-scaled features. Block-B comprises three YOLOv5 C3, and three Conv cascaded. Block-S consists of swin-transformer [28], Conv cascade. The structure of the swin-transformer [28] is shown in Figure 4. In swin-transformer [28], the input features are divided into windows and encoded with relative position. The final output is obtained through multiple images down sampling (patch merging) and swin-block. In the steel surface classification experiment of swin-transformer [28], the effect of swin-transformer [28] with relative position encoding is better than that of swin-transformer [28] without it. Therefore, this study adds relative position encoding to block-S.Figure 4
Structure of block-B and block-S.In order to further improve the detection of small objects, this study designs a unified weighted residual path (Figure3) and performs fusion of features from the backbone. The fusion formula is as follows:(1)F=f∑i=1nConvconvxiexpwi∑1nexpwj,where xi is the original input feature and w is the adaptive learnable weight, ∑1nwj=1. xiconvolves with a 3×3 kernel to adjust the feature size, a 1×1 to integrate the channel, and then multiplies expwi/∑1nexpwi before fusion.


#### 3.1.2. Neck

The neck is designed to use better the features extracted by the backbone. In this study, the SPP structure is designed to enhance the invariance and robustness of image features. Figure5 shows the structure of block-N in the top-down path and the bottom-up path in the neck. In block-N, this study designs a dual attention module, which will be introduced in detail in Section 3.2.Figure 5
Block-N structure in neck, and block-N is reversed in the top-down and bottom-up paths.


#### 3.1.3. Decoupled Detector

In object detection, the role of the head in CNN is to regress the generated features to the bounding box and classified into some categories. Most methods use one head for classification and regression. This solution has limitations because classification relies more on fine-grained features, while regression relies more on semantic information to locate the object.To stress the issue, YOLOX proposes the decoupled detector, which divides classification and regression into two types of problems. In this study, the decoupled detector is innovated (Figure6), and a dual attention module is added to the classification branch. In our proposal, 6 decoupled detectors are used to detect features of the T-model, and four decoupled detectors to detect features of the S-model. The dense stacking of multiple decoupled detectors helps the models to detect objects of different scales.Figure 6
Decoupled detector.


### 3.2. Attention Module

Attention is usually embedded in CNNs and used to generate attention matrices to optimize features. CBAM [25] is a classic attention module that combines channel and spatial attention. Specifically, a 1 × 1 × C feature map will be obtained by pooling in channel attention, and a H × W × 1 feature map will be obtained by pooling in spatial attention. However, pooling leads to much information being lost. A mutual mapping between the three dimensions in a C × H × W image is crucial to extracting attention.The attention structure proposed in this study (Figure7) includes channel and spatial attention. The resulting attention features are multiplied by input features as their features are combined. In terms of channel attention, this study first uses 1 × 1 convolution instead of pooling to achieve channel interaction and information integration. It then inputs the features into a two-layer neural network (MLP) to amplify the relationship between channels and spatial dimensions. The number of neurons in the first layer in MLP is C/r (r is the reduction ratio, r = 16), the activation function is Relu, and the number of neurons in the second layer is C. In this study, the attention structure introduces BN to reduce the gradient dispersion and speed up the convergence speed. This study uses two dilated convolutions (d = 4, r = 16) identical to BAM [43] for spatial information integration to focus on spatial information.Figure 7
Attention flowchart. This study integrates spatial and channel attention and adds residual paths to them.The attention extraction formula in this study is as follows:(2)McF=sigmoidBNConvMLPConvF,(3)MSF=sigmoidBNConvConvF,(4)F′=McF+MsF⊗F,where McF denotes channel attention, MsF denotes spatial attention, and F′ denotes the superposition of channel attention and spatial attention and multiplied by the original input F.


### 3.3. Knowledge Distillation Module

The lightweight model has speed and memory consumption advantages, but the detection effect is challenging to meet the requirements. Knowledge distillation is an effective method to improve the detection accuracy of small models. The general knowledge distiller focuses on the extraction of the overall features. The formula is as follows:(5)LD=1CHW∑i=1C∑j=1H∑k=1WFi,j,kT−fFi,j,kS2,where FT/i,j,k and FS/i,j,k denote the characteristics of teachers and students, respectively, and f denotes converting FS/i,j,k to the same data dimension as FT/i,j,k. H, W specify the height and width of the feature, and C denotes the channel.This study uses knowledge distillation to improve the S-model’s performance. The image’s foreground and background are distilled separately. The positive and negative samples are separated by separating the foreground and background, which solves the imbalance of positive and negative samples in the image. This study also designs an attention module to generate attention regions, forcing the student model to learn the vital features of the image.In the distillation method of foreground and background, this study first sets the binary mask of foreground segmentation (the real position of the object frame) and sets objects within the ground truth box to 1 and objects outside the ground truth box to 0. It is expressed as follows:(6)Maskx,y=1,x,y∈GT,0,otherwise,where x and y denote the horizontal and vertical coordinates of the area object and GT denotes the area position of the real frame. When the position of x,y falls in the real frame area, it is set to 1; otherwise, it is 0.Due to the large-scale variation of objects in industrial detection datasets and the uneven distribution of positive and negative samples of objects, these will adversely affect the distillation effect. For this reason, this study uses the scaling mask to balance the object scale with reference to FGD [44] to solve the problem of an unbalanced object scale. The formula is as follows:(7)ScaleMaskx,y=1Hx,y×Wx,y,x,y∈GT,1∑x=1H∑y=1W1−Maskx,y,where GT is the region of the ground truth. In this study, the scaling mask is used to normalize the foreground and background pixels. When there are two objects in an image, the bounding box of the large object may cover small object. In this study, when small objects and large objects are in a bounding box, the following formula is used:(8)Mask=minGT,where GT represents the real box of the object. When a small object is surrounded by the box of a large object, the smallest bounding box is preferentially selected.After separating the foreground and background, this study uses the designed dual attention module to generate the attention mask. The attention mask formula is as follows:(9)MF=McF+MsF⊗F,(10)AttentionMaskF=softmaxMFT,where MF denotes the process of generating attention, F denotes the original feature map, McF denotes channel attention, MsF denotes spatial attention, T denotes distillation temperature, T = 20, and softmax is used to process features graph weights.In the distillation process of foreground and background, binary mask, scale mask, and attention mask are used for attention-guided distillation. The loss function is as follows:(11)Lfg=∑x=1C∑y=1H∑z=1WMasky,zScaleMasky,zAttentionMaskFx,y,zT−fFx,y,zS2+∑x=1C∑y=1H∑z=1W1−Maskx,yScaleMaskx,yAttentionMaskFx,y,zT−fFx,y,zS2,where Lfg,Masky,z,1−Masky,z,ScaleMasky,z,AttentionMaskare the distillation loss of foreground and background, foreground mask, background mask, scale mask, attention mask, respectively, and Fx,y,zT−fFx,y,zS2 is the difference between teacher feature and student feature.In addition, this study uses the attention loss function to let the student model learn the attention mask of the teacher model. The loss formula is as follows:(12)LA=SmoothL1AttentionMaskT,AttentionMaskS,where LA is the attention mask loss function, AttentionMaskT denotes the teacher’s attention mask, and AttentionMaskS denotes the student's attention mask.In the feature distillation of the foreground, this study uses the designed attention module to distil the entire feature map. The loss formula is as follows:(13)LG=MFx,y,zT−MfFx,y,zS2,where LG is the distillation loss function of the entire feature map, MFT/x,y,z is the attention feature map of the teacher model, and MfFS/x,y,z is the attention feature map of the student model.


### 3.4. Loss Function

In order to solve the unbalanced positive and negative samples in the PCB dataset, this study uses QFocal loss as the classification and confidence losses of the T-model and S-model. The formulas are as follows:(14)QFLσ=−αt×y−σβ×1−ylog1−σ+ylogσ,where y is the smoothed label in [0, 1] and σ is the prediction result. Focal loss introduces two factors αt and y−σβ, where αt=y×α+1−y×1−α is used to balance positive and negative samples, and y−σβ is used to stress difficult detected samples. In addition, this study introduces CIoU Loss as the prediction box regression loss of T-model and S-model. The formula is as follows:(15)LCIOU=1−IOU+RCIOUBpd,Bgt,where RCIOU is the penalty term for the prediction box Bpd and the object box Bgt. CIoU loss considers the overlapping area, centre point distance, and aspect ratio in the prediction frame regression, which solves the problem of inconsistency between the real frame and the predicted frame during object detection. The normalized distance and penalty term between the centre points of the two bounding boxes are defined as follows:(16)RCIOU=ρ2b,bgtC2+α4π2arctanwgthgt−arctanwh2,where b,bgt denote the centre points of Bpd,Bgt respectively, ρx is the Euclidean distance, and C is the diagonal length of the smallest bounding box covering these two boxes. α is a positive trade-off parameter. w,h are the width and height of the prediction box. In the selection of anchors, this study uses K-means to filter the anchors that meet the criteria.In summary, the total loss function proposed in this study is as follows:(17)TotalLoss=αLfg+βLA+γLG+2δQFL+εLCIOU,where Lfg,LA,LG,QFL,LCIOU denote the distillation loss of foreground and background, attention mask loss, distillation loss of the entire feature map, QFocal loss, cIou loss, respectively, α,β,γ,δ,ε are the weight parameters of the balance loss, respectively. In this study, an adaptive weight updater is designed to adjust the weight adaptively. The formula is as follows:(18)W=expLi∑i=1nexpLi×n×wk,where W denotes the updated weight parameter, wk denotes the weight parameter before the update, Li denotes the value of each loss, and n denotes the number of weight parameters. In this study, each weight parameter is set to one before the training starts, and then in each training round, the weight parameter is updated according to the proportion of each loss value in the total loss. In this way, the loss with a large value will get a large weight in the next round of optimization, and the weight with a small value will be further reduced. At the end of the training, all tasks can be optimized almost simultaneously.


## 3.1. S-Model and T-Model

### 3.1.1. Backbone

Most general-purpose object detectors are based on CNN and employ large-scale convolution kernels in the first layer of the backbone to increase the receptive field. However, more detailed information will be lost with the increase of convolutional layers. Swin-transformer [28] uses window self-attention to significantly reduce computation load and extract small-scaled features. Therefore, this study designs a novel backbone based on swin-transformer [28].As shown in Figure3, our backbone includes block-B, block-S, SPP, and weighted residual fusion paths. The first layer selects a 3×3 convolution kernel to extract fine-scaled features. Block-B comprises three YOLOv5 C3, and three Conv cascaded. Block-S consists of swin-transformer [28], Conv cascade. The structure of the swin-transformer [28] is shown in Figure 4. In swin-transformer [28], the input features are divided into windows and encoded with relative position. The final output is obtained through multiple images down sampling (patch merging) and swin-block. In the steel surface classification experiment of swin-transformer [28], the effect of swin-transformer [28] with relative position encoding is better than that of swin-transformer [28] without it. Therefore, this study adds relative position encoding to block-S.Figure 4
Structure of block-B and block-S.In order to further improve the detection of small objects, this study designs a unified weighted residual path (Figure3) and performs fusion of features from the backbone. The fusion formula is as follows:(1)F=f∑i=1nConvconvxiexpwi∑1nexpwj,where xi is the original input feature and w is the adaptive learnable weight, ∑1nwj=1. xiconvolves with a 3×3 kernel to adjust the feature size, a 1×1 to integrate the channel, and then multiplies expwi/∑1nexpwi before fusion.


### 3.1.2. Neck

The neck is designed to use better the features extracted by the backbone. In this study, the SPP structure is designed to enhance the invariance and robustness of image features. Figure5 shows the structure of block-N in the top-down path and the bottom-up path in the neck. In block-N, this study designs a dual attention module, which will be introduced in detail in Section 3.2.Figure 5
Block-N structure in neck, and block-N is reversed in the top-down and bottom-up paths.


### 3.1.3. Decoupled Detector

In object detection, the role of the head in CNN is to regress the generated features to the bounding box and classified into some categories. Most methods use one head for classification and regression. This solution has limitations because classification relies more on fine-grained features, while regression relies more on semantic information to locate the object.To stress the issue, YOLOX proposes the decoupled detector, which divides classification and regression into two types of problems. In this study, the decoupled detector is innovated (Figure6), and a dual attention module is added to the classification branch. In our proposal, 6 decoupled detectors are used to detect features of the T-model, and four decoupled detectors to detect features of the S-model. The dense stacking of multiple decoupled detectors helps the models to detect objects of different scales.Figure 6
Decoupled detector.


## 3.1.1. Backbone

Most general-purpose object detectors are based on CNN and employ large-scale convolution kernels in the first layer of the backbone to increase the receptive field. However, more detailed information will be lost with the increase of convolutional layers. Swin-transformer [28] uses window self-attention to significantly reduce computation load and extract small-scaled features. Therefore, this study designs a novel backbone based on swin-transformer [28].As shown in Figure3, our backbone includes block-B, block-S, SPP, and weighted residual fusion paths. The first layer selects a 3×3 convolution kernel to extract fine-scaled features. Block-B comprises three YOLOv5 C3, and three Conv cascaded. Block-S consists of swin-transformer [28], Conv cascade. The structure of the swin-transformer [28] is shown in Figure 4. In swin-transformer [28], the input features are divided into windows and encoded with relative position. The final output is obtained through multiple images down sampling (patch merging) and swin-block. In the steel surface classification experiment of swin-transformer [28], the effect of swin-transformer [28] with relative position encoding is better than that of swin-transformer [28] without it. Therefore, this study adds relative position encoding to block-S.Figure 4
Structure of block-B and block-S.In order to further improve the detection of small objects, this study designs a unified weighted residual path (Figure3) and performs fusion of features from the backbone. The fusion formula is as follows:(1)F=f∑i=1nConvconvxiexpwi∑1nexpwj,where xi is the original input feature and w is the adaptive learnable weight, ∑1nwj=1. xiconvolves with a 3×3 kernel to adjust the feature size, a 1×1 to integrate the channel, and then multiplies expwi/∑1nexpwi before fusion.


## 3.1.2. Neck

The neck is designed to use better the features extracted by the backbone. In this study, the SPP structure is designed to enhance the invariance and robustness of image features. Figure5 shows the structure of block-N in the top-down path and the bottom-up path in the neck. In block-N, this study designs a dual attention module, which will be introduced in detail in Section 3.2.Figure 5
Block-N structure in neck, and block-N is reversed in the top-down and bottom-up paths.


## 3.1.3. Decoupled Detector

In object detection, the role of the head in CNN is to regress the generated features to the bounding box and classified into some categories. Most methods use one head for classification and regression. This solution has limitations because classification relies more on fine-grained features, while regression relies more on semantic information to locate the object.To stress the issue, YOLOX proposes the decoupled detector, which divides classification and regression into two types of problems. In this study, the decoupled detector is innovated (Figure6), and a dual attention module is added to the classification branch. In our proposal, 6 decoupled detectors are used to detect features of the T-model, and four decoupled detectors to detect features of the S-model. The dense stacking of multiple decoupled detectors helps the models to detect objects of different scales.Figure 6
Decoupled detector.


## 3.2. Attention Module

Attention is usually embedded in CNNs and used to generate attention matrices to optimize features. CBAM [25] is a classic attention module that combines channel and spatial attention. Specifically, a 1 × 1 × C feature map will be obtained by pooling in channel attention, and a H × W × 1 feature map will be obtained by pooling in spatial attention. However, pooling leads to much information being lost. A mutual mapping between the three dimensions in a C × H × W image is crucial to extracting attention.The attention structure proposed in this study (Figure7) includes channel and spatial attention. The resulting attention features are multiplied by input features as their features are combined. In terms of channel attention, this study first uses 1 × 1 convolution instead of pooling to achieve channel interaction and information integration. It then inputs the features into a two-layer neural network (MLP) to amplify the relationship between channels and spatial dimensions. The number of neurons in the first layer in MLP is C/r (r is the reduction ratio, r = 16), the activation function is Relu, and the number of neurons in the second layer is C. In this study, the attention structure introduces BN to reduce the gradient dispersion and speed up the convergence speed. This study uses two dilated convolutions (d = 4, r = 16) identical to BAM [43] for spatial information integration to focus on spatial information.Figure 7
Attention flowchart. This study integrates spatial and channel attention and adds residual paths to them.The attention extraction formula in this study is as follows:(2)McF=sigmoidBNConvMLPConvF,(3)MSF=sigmoidBNConvConvF,(4)F′=McF+MsF⊗F,where McF denotes channel attention, MsF denotes spatial attention, and F′ denotes the superposition of channel attention and spatial attention and multiplied by the original input F.


## 3.3. Knowledge Distillation Module

The lightweight model has speed and memory consumption advantages, but the detection effect is challenging to meet the requirements. Knowledge distillation is an effective method to improve the detection accuracy of small models. The general knowledge distiller focuses on the extraction of the overall features. The formula is as follows:(5)LD=1CHW∑i=1C∑j=1H∑k=1WFi,j,kT−fFi,j,kS2,where FT/i,j,k and FS/i,j,k denote the characteristics of teachers and students, respectively, and f denotes converting FS/i,j,k to the same data dimension as FT/i,j,k. H, W specify the height and width of the feature, and C denotes the channel.This study uses knowledge distillation to improve the S-model’s performance. The image’s foreground and background are distilled separately. The positive and negative samples are separated by separating the foreground and background, which solves the imbalance of positive and negative samples in the image. This study also designs an attention module to generate attention regions, forcing the student model to learn the vital features of the image.In the distillation method of foreground and background, this study first sets the binary mask of foreground segmentation (the real position of the object frame) and sets objects within the ground truth box to 1 and objects outside the ground truth box to 0. It is expressed as follows:(6)Maskx,y=1,x,y∈GT,0,otherwise,where x and y denote the horizontal and vertical coordinates of the area object and GT denotes the area position of the real frame. When the position of x,y falls in the real frame area, it is set to 1; otherwise, it is 0.Due to the large-scale variation of objects in industrial detection datasets and the uneven distribution of positive and negative samples of objects, these will adversely affect the distillation effect. For this reason, this study uses the scaling mask to balance the object scale with reference to FGD [44] to solve the problem of an unbalanced object scale. The formula is as follows:(7)ScaleMaskx,y=1Hx,y×Wx,y,x,y∈GT,1∑x=1H∑y=1W1−Maskx,y,where GT is the region of the ground truth. In this study, the scaling mask is used to normalize the foreground and background pixels. When there are two objects in an image, the bounding box of the large object may cover small object. In this study, when small objects and large objects are in a bounding box, the following formula is used:(8)Mask=minGT,where GT represents the real box of the object. When a small object is surrounded by the box of a large object, the smallest bounding box is preferentially selected.After separating the foreground and background, this study uses the designed dual attention module to generate the attention mask. The attention mask formula is as follows:(9)MF=McF+MsF⊗F,(10)AttentionMaskF=softmaxMFT,where MF denotes the process of generating attention, F denotes the original feature map, McF denotes channel attention, MsF denotes spatial attention, T denotes distillation temperature, T = 20, and softmax is used to process features graph weights.In the distillation process of foreground and background, binary mask, scale mask, and attention mask are used for attention-guided distillation. The loss function is as follows:(11)Lfg=∑x=1C∑y=1H∑z=1WMasky,zScaleMasky,zAttentionMaskFx,y,zT−fFx,y,zS2+∑x=1C∑y=1H∑z=1W1−Maskx,yScaleMaskx,yAttentionMaskFx,y,zT−fFx,y,zS2,where Lfg,Masky,z,1−Masky,z,ScaleMasky,z,AttentionMaskare the distillation loss of foreground and background, foreground mask, background mask, scale mask, attention mask, respectively, and Fx,y,zT−fFx,y,zS2 is the difference between teacher feature and student feature.In addition, this study uses the attention loss function to let the student model learn the attention mask of the teacher model. The loss formula is as follows:(12)LA=SmoothL1AttentionMaskT,AttentionMaskS,where LA is the attention mask loss function, AttentionMaskT denotes the teacher’s attention mask, and AttentionMaskS denotes the student's attention mask.In the feature distillation of the foreground, this study uses the designed attention module to distil the entire feature map. The loss formula is as follows:(13)LG=MFx,y,zT−MfFx,y,zS2,where LG is the distillation loss function of the entire feature map, MFT/x,y,z is the attention feature map of the teacher model, and MfFS/x,y,z is the attention feature map of the student model.


## 3.4. Loss Function

In order to solve the unbalanced positive and negative samples in the PCB dataset, this study uses QFocal loss as the classification and confidence losses of the T-model and S-model. The formulas are as follows:(14)QFLσ=−αt×y−σβ×1−ylog1−σ+ylogσ,where y is the smoothed label in [0, 1] and σ is the prediction result. Focal loss introduces two factors αt and y−σβ, where αt=y×α+1−y×1−α is used to balance positive and negative samples, and y−σβ is used to stress difficult detected samples. In addition, this study introduces CIoU Loss as the prediction box regression loss of T-model and S-model. The formula is as follows:(15)LCIOU=1−IOU+RCIOUBpd,Bgt,where RCIOU is the penalty term for the prediction box Bpd and the object box Bgt. CIoU loss considers the overlapping area, centre point distance, and aspect ratio in the prediction frame regression, which solves the problem of inconsistency between the real frame and the predicted frame during object detection. The normalized distance and penalty term between the centre points of the two bounding boxes are defined as follows:(16)RCIOU=ρ2b,bgtC2+α4π2arctanwgthgt−arctanwh2,where b,bgt denote the centre points of Bpd,Bgt respectively, ρx is the Euclidean distance, and C is the diagonal length of the smallest bounding box covering these two boxes. α is a positive trade-off parameter. w,h are the width and height of the prediction box. In the selection of anchors, this study uses K-means to filter the anchors that meet the criteria.In summary, the total loss function proposed in this study is as follows:(17)TotalLoss=αLfg+βLA+γLG+2δQFL+εLCIOU,where Lfg,LA,LG,QFL,LCIOU denote the distillation loss of foreground and background, attention mask loss, distillation loss of the entire feature map, QFocal loss, cIou loss, respectively, α,β,γ,δ,ε are the weight parameters of the balance loss, respectively. In this study, an adaptive weight updater is designed to adjust the weight adaptively. The formula is as follows:(18)W=expLi∑i=1nexpLi×n×wk,where W denotes the updated weight parameter, wk denotes the weight parameter before the update, Li denotes the value of each loss, and n denotes the number of weight parameters. In this study, each weight parameter is set to one before the training starts, and then in each training round, the weight parameter is updated according to the proportion of each loss value in the total loss. In this way, the loss with a large value will get a large weight in the next round of optimization, and the weight with a small value will be further reduced. At the end of the training, all tasks can be optimized almost simultaneously.


## 4. Experiments and Analysis

In this section, experiments are carried out on the PCB defect dataset, the NEU surface defect dataset, and the aluminium defect dataset.


### 4.1. Experimental Environment and Parameter Settings

This article implements the code in the PyTorch framework, version 1.9.0. CUDA version 11.4, cuDNN version 8.0. This study’s model training and inference are performed on NVIDIA RTX 6000 × 1 and Intel i9-9900k@5 GHz × 1. Experimental platforms are GPU memory of 24 GB and CPU memory of 32 GB. The IDE used in the experiment is Pycharm 2019 Professional Edition.In the training process, this study uses YOLOv5 as the baseline to build the T-model, uses the pretraining weights of YOLOv5 in the initial training of the T-model, and saves T-model weights after training. This study uses T-model for pretraining, loads the pretraining weights of the T-model to S-model for training, and uses knowledge distillation during the training process. Unless otherwise specified, the network in this study is trained with Adam for 300 iterations with an initial learning rate of 0.001, and the learning rate is adjusted using cosine annealing. This article uses a weight decay of 0.0001 and a momentum of 0.9. Also, the input image is resized to 640× 640. The batch of the model in this study is eight during training, the batch is one during inferencing, and TensorRT is not used.


### 4.2. Object Detection and Evaluation Indicators

FPS [45]: in this study, the model inference is carried out under the same equipment conditions. The same size image is used to calculate FPS and evaluate the model’s processing speed.IOU [45]: object detection uses the IOU to calculate the degree of coincidence between the predicted box and the ground-truth box, which further measures the accuracy of detecting the corresponding object in a specific dataset.mAP [45]: mAP is the sum of the average precision of all categories divided by the number of all categories. mAP@.5 is the model accuracy index when the IOU is 0.5. mAP@.5:.95 is obtained by calculating an mAP every 0.5 from IOU from 0.5 to 0.95 and finally averaging these maps.


### 4.3. Experiments on the Aluminium Defect Dataset

The aluminium defect dataset is the images of aluminium surface defects published by Baidu AI (Figure8). It contains 412 images in total. There are four types of defects, i.e., Zhen_kong, ca_shang, zang_wu, and zhe_zhou, and one image may contain different types of defects. In this study, the dataset is processed with photometric and geometric distortion. Specifically, this study brightens the picture and then flips and pans the picture to expand the dataset, increasing the number of pictures in the dataset to 1236. Then, the extended images are divided into a training set, validation set, and test set with a ratio of 8 : 1:1. It can be seen from Figure 8 that the surface defect scale of aluminium material varies greatly, which brings difficulties to the detection.Figure 8
Example of aluminium defect.


#### 4.3.1. Comparative Experiments

Table2 lists the comparison experiment results of aluminium defect datasets. It can be seen that the T-model of this study surpasses most classic object detection models, indicating that the model is more suitable for the field of industrial detection. T-model size is smaller than YOLOv5X, YOLOv4, YOLOR-P6, Faster-R-CNN, etc. The F1-score and mAP of the T-model are all in the leading position. T-model surpassed the newly proposed YOLOR 1.13% on mAP@.5, and F1-score exceeded 0.05. YOLOX has better detection performance than T-model, but T-model accuracy is close to YOLOX. However, due to the swin-transformer, the FPS advantage of the T-model is not apparent. In contrast, the S-model is lighter, the reasoning speed is fast, and the accuracy after knowledge distillation is close to the T-model.Table 2
Comparative experiment of the aluminium defect dataset.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Model size (M)FPSSSDVGG-1686.5835.299.862.50.7799.7667Faster-rcnnResNet-5092.3542.373.8897.50.8452316YOLOv3DarkNet-5395.954.494.795.70.9523419YOLOv5SCSPDarkNet97.256.696.496.80.972650Efficientdet-d3EfficientNet-B393.5250.291.8590.910.9114.7820YOLOv5XCSPDarkNet94.354.995.194.70.9516721CentrenetHourglass-10460.338.651.0960.610.55124.6148RetinanetResNet-10196.5246.794.7993.320.94144.8420YOLOv4CSPDarkNet-5394.1359.183.3896.010.89245.5329YOLOR-P6CSPDarkNet9753.188.397.80.93140.8861YOLOX-sDarknet-5399.258.298.9199.30.9934.241T-modelOur backbone98.5358.898.8598.380.9881.619S-modelOur backbone97.8255.8697.8396.870.9725.136In order to more intuitively show the detection effect of the T-model, Figure9 shows the detection results of 16 pictures.Figure 9
Effect of aluminium defect detection.


#### 4.3.2. Ablation Experiments

Table3 lists the ablation experiments performed with YOLOv5S as the baseline. In the ablation experiment of the backbone, this study uses the B4, S3, and SPP outputs in the backbone as the input of YOLOv5S neck for experimenting. The results in the second row show that when using the backbone of the T-model, mAP@.5 is 0.63% higher than that of YOLOv5S, and the other indicators are also slightly improved. On the neck ablation experiment, this study experiments with three C3s in the YOLOv5S backbone with one SPP output as the neck input. The third row shows that the detection effect is improved when introducing the neck in this study.Table 3
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S97.256.696.496.80.97+Our backbone97.8356.9497.8496.870.97+Our neck97.7157.1397.197.380.98This paper also conducts ablation experiments with T-model as the baseline to verify the residual fusion path in the backbone and the down sampling path in neck. The ablation results are listed in Table4. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, mAP@.5 increases by 0.58%, and the F1-score increases by 0.02. The third rows show that when introducing the down sampling path in neck, mAP@.5 is improved by 0.52%, and the F1-score is improved by 0.02. In general, the residual fusion path and the down sampling path are beneficial to industrial detection scenarios with drastic changes in scale.Table 4
Ablation experiment with the T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model97.9157.4397.2497.870.97+Residual fusion path98.0257.5798.1698.470.99+Residual fusion path + down sampling path98.5358.898.8598.380.98


#### 4.3.3. Experiments on the Detector

This study examines several different detectors, including coupled detector, decoupled detector, and decoupled detector with added dual attention module. This study conducts experiments with T-model as the baseline. Specifically, this study experiments with these detectors on T-model, and the results are listed in Table5. The experimental results in the second row show that the mAP@.5 and F1-score of decoupled detector are improved by 0.78% and 0.05, respectively, compared with coupled detector. The third line shows that the detection accuracy is further improved when adding the dual attention module to the classification branch of decoupled detector, indicating that the dual attention module improves the classification accuracy.Table 5
Experiments on three detectors.
DetectormAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Coupled detector97.2457.0586.497.310.93Decoupled detector98.0257.798.1698.470.98Decoupled detector + attention98.5358.898.8598.380.98


#### 4.3.4. Experiments on the Dual Attention Module

This study analyses the features of CBAM. As shown in Figure10, it is a heat map comparison between CBAM and the dual attention module. It can be seen that the dual attention module pays more attention to object areas such as wrinkles.Figure 10
(a) Original image. (b) CBAM heatmap. (c) Dual attention module heatmap. The darker the yellow, the higher the attention.
(a)(b)(c)This study uses T-model as the baseline for experiments on the dual attention module (DA). The experimental results are listed in Table6. It can be seen that the dual attention module effect surpasses CBAM.Table 6
Comparison experiment between CBAM and dual attention module.
ApproachmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model without DA97.5257.296.7597.870.97+DA98.5358.898.8598.380.98+CBAM96.6853.1595.7596.830.96


#### 4.3.5. Knowledge Distillation

This study conducts experiments related to knowledge distillation on lightweight models. The experiments are performed on isomorphism object detectors and heterogeneous object detectors, respectively. On isomorphic object detectors, this study experiments T-model and S-model, Efficientdet-d7, and Efficientdet-d3. This study experiments with faster-R-CNN-ResNet50, faster-R-CNN-VGG16, YOLOv4, and YOLOv4-tiny on heterogeneous object detectors. In addition, this study also experimented with output layer knowledge distillation on T-model and S-model as a comparative experiment. Table7 lists the experimental results of various classic object detectors. It can be seen that knowledge distillation improves the detection performance of the S-model, especially when T-model and S-model are isomorphic. It proves that the knowledge distiller can improve the detection accuracy of small models without increasing the number of parameters.Table 7
Knowledge distiller experiments.
ModelsmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)S-model95.7654.2395.8295.25S-model with KD97.8255.8697.8396.87Faster-rcnn-VGG1687.2236.1877.4288.84Faster-rcnn-VGG16 with KD87.8736.2687.4686.67Efficientdet-d393.5250.291.8590.91Efficientdet-d3 with KD95.851.3491.8896.78YOLOv4-tiny84.4335.7980.4686.01YOLOv4-tiny with KD84.9435.680.4486.9


### 4.4. Experiments on the PCB Defect Dataset

The PCB defect dataset has 1386 images, and its annotation files contain the object location information and classification. It contains six defects: missing hole, mouse bite, open circuit, spur, short, and spurious copper. Each image may contain multiple defective objects of the same type (missing holes in Figure11). In the experiments in this study, 900 images of different defect types are selected for training (mainly minor defects that are difficult to identify). In our experiment, images and annotations are divided into the training set, validation set, and experiment set according to the ratio of 6 : 2 : 2.Figure 11
Missing hole.This study conducts a comparative experiment on the PCB surface defect dataset, omitting FPS since the input size is still 640. It should be noted that the effect of the S-model is the result of distillation through the knowledge distiller. The comparison experiments are listed in Table8. It can be seen that most of the classic object detection models are not effective for small object detection, and the T-model in this study shows a better performance.Table 8
Comparative experiment of PCB surface defects.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1617.11019.9354.640.29Faster-rcnnResNet-5078.3350.2467.7474.510.7YOLOv3DarkNet-5373.3247.5090.8552.660.67YOLOv5SCSPDarkNet-5391.251.881.8292.120.86Efficientdet-d3EfficicientNet-B371.83599.4449.510.66YOLOv5XCSPDarkNet-5391.850.795.291.30.93CentrenetHourglass-10443.414.5343.7952.20.48RetinanetResNet-10113.18566.674.310.08YOLOv4CSPDarkNet-5381.250.0689.3794.780.92YOLOR-P6CSPDarkNet94.754.593.193.10.94YOLOX-sDarknet-5395.8462.0387.1693.730.91T-modelOur backbone94.7958.0288.1495.650.91S-modelOur backbone92.5156.9883.7992.330.88In order to more intuitively show the detection effect of the T-model, Figure12 shows the detection results of 16 pictures.Figure 12
PCB defect dataset detection results.


#### 4.4.1. Ablation Experiment

Table9 lists the ablation experiments performed in this study with YOLOv5S as the baseline. The experimental operations are consistent with the experiments on the aluminium defect dataset. The results in the second row show that after using the backbone of the T-model in this study, mAP@.5 is 3.68% higher than that of YOLOv5S indicating that the backbone in this study is beneficial for small object detection. The third line shows that the detection effect is improved when introducing the neck of the T-model. The dual attention module makes the network pay more attention to the object and improves the small object detection effect.Table 9
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S91.251.881.8292.120.86+Our backbone91.9852.4896.0592.460.94+Our neck92.0553.0996.6293.60.95This study also uses T-model as the baseline to conduct ablation experiments to verify the residual fusion path in the backbone and the down sampling path in the neck. The experimental operations are consistent with the experiments on the aluminium defect dataset. The ablation test results are listed in Table10. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, the effect is significantly improved, indicating that the residual path can better fuse the fine-grained features in the backbone. The third row shows that by introducing the down sampling path in neck, mAP@.5 is improved by 2.4%, and the F1-score is improved by 0.03. It shows that the down sampling path is beneficial for small object detection because more fine-grained feature information is fused into neck. Overall, the residual fusion and the down sampling paths can improve the detection effect.Table 10
Ablation trials with T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model92.3954.2191.7189.840.88+Residual fusion path93.6556.4489.9494.430.92+Residual fusion path + down sampling path94.7958.0288.1495.650.91


### 4.5. Experiments on the NEU Surface Defect Dataset

North-eastern university releases the NEU surface defect dataset. This dataset collects six typical defects on the surface of the hot-rolled strip: rolled-in scale, patches, crazing, pitted surface, inclusion, and scratches. Each image has several defects of the same type. The label file marks the category and specific location of the defective object.Figure13 shows different kinds of steel defects. The categories’ defects have significant differences in appearance. Such scratches (last column) could be horizontal, vertical, slanted, etc. At the same time, each category has similarities in defects, such as rolled-in scale, crazing, and pitted surface. In addition, due to the influence of lighting and specific materials, the grey value of each category of defect images will also change. Object detection in the NEU surface defect dataset contains three difficulties: intraclass defects have significant appearance differences, interclass defects have similar aspects, and defect images are affected by changes in lighting and specific materials. According to the observation of the dataset, some cracks are concentrated in one direction because the steel is slender, and scratches are in any direction. Cracks and scratches are very similar, so scratches in specific directions are easily identified as cracks. Therefore, this study increases the data enhancement of cracks, rotates, splices crack pictures, and reduces the recognition error rate.Figure 13
Example of surface defects in steel.In order to show the generalization ability of this model in industrial detection, this study conducts a comparative experiment on the NEU surface defect database, where the S-model is the result of knowledge distillation. The comparative test results are listed in Table11. It can be seen that the T-model and S-model still have better performance, indicating that the object detection model of this study has a particular generalization ability in multiple industrial detection datasets. Figure 14 shows the different steel defect recognition results; it can be seen that all the defects are accurately identified.Table 11
Comparative experiments on NEU defect datasets.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1666.7326.687.2734.520.5Faster-rcnnResNet-5076.9539.944.1487.860.59YOLOv3DarkNet-5368.933.572.265.90.7YOLOv5SCSPDarkNet-5370.935.678.464.30.35Efficientdet-d3EfficicientNet-B365.653484.7648.440.77YOLOv5XCSPDarkNet-537237.766.872.50.69CentrenetHourglass-10439.2413.555.518.20.29RetinanetResNet-10166.5431.981.7344.620.57YOLOv4CSPDarkNet-5368.7332.995.640.540.56YOLOR-P6CSPDarkNet76.5238.350.8683.370.63YOLOX-sDarknet-5379.3840.2353.7384.880.66T-modelOur backbone75.5639.6853.4583.160.65S-modelOur backbone74.4139.249.0582.280.62Figure 14
Steel defect detection effect.


### 4.6. Experiments on Glass Bottle Bottom Mould Point Dataset

This is a dataset for recognising mould point sequences at the bottom of glass bottles, with different permutations representing different product lot numbers. The mould point identification on the bottom of glass bottles is mainly used to locate the batch of glass products online to trace the product. The dataset contains 900 glass bottle bottom mould point images, each with a native resolution of 800 × 780 and 18 types. Each image has a corresponding label file. In this study, the label files correspond to the images one by one, and the training set, test set, and validation set are divided according to the ratio of 6 : 2 : 2.Figure15 shows the images of different glass bottle bottom mould points, and the high similarity between objects makes detection difficult. In order to demonstrate the generalization ability of the model in industrial detection, this study conducts a comparative experiment on the glass bottle bottom mould point database, where the S-model results from knowledge distillation. The comparison test results are listed in Table 12. It can be seen that T-model and S-model still have better performance, indicating that the object detection model in this study has a specific generalization ability on multiple industrial detection datasets. Figure 16 shows the recognition effect of 16 different model point images. It can be seen that all model point objects are accurately recognized and positioned.Figure 15
Example of bottom mould point data.Table 12
Comparative experiments on NEU defect datasets.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1670.4432.5278.9449.480.5Faster-rcnnResNet-5083.6238.7374.8975.40.59YOLOv3DarkNet-5391.3740.6684.3992.330.7YOLOv5SCSPDarkNet-5393.2162.185.1991.440.35Efficientdet-D3EfficicientNet-B386.3955.6767.4989.90.77YOLOv5XCSPDarkNet-5398.464.891.4799.040.69CentrenetHourglass-10468.5143.1884.1448.740.29RetinanetResNet-10180.7949.8966.2877.560.57YOLOv4CSPDarkNet-5396.150.0689.3794.780.56YOLOR-P6CSPDarkNet99.359.953.5990.63YOLOX-sDarknet-5399.5570.6880.3299.90.66T-modelOur backbone99.3669.5381.199.50.89S-modelOur backbone98.165.279.4498.910.89Figure 16
Glass bottle bottom mould point recognition effect.


## 4.1. Experimental Environment and Parameter Settings

This article implements the code in the PyTorch framework, version 1.9.0. CUDA version 11.4, cuDNN version 8.0. This study’s model training and inference are performed on NVIDIA RTX 6000 × 1 and Intel i9-9900k@5 GHz × 1. Experimental platforms are GPU memory of 24 GB and CPU memory of 32 GB. The IDE used in the experiment is Pycharm 2019 Professional Edition.In the training process, this study uses YOLOv5 as the baseline to build the T-model, uses the pretraining weights of YOLOv5 in the initial training of the T-model, and saves T-model weights after training. This study uses T-model for pretraining, loads the pretraining weights of the T-model to S-model for training, and uses knowledge distillation during the training process. Unless otherwise specified, the network in this study is trained with Adam for 300 iterations with an initial learning rate of 0.001, and the learning rate is adjusted using cosine annealing. This article uses a weight decay of 0.0001 and a momentum of 0.9. Also, the input image is resized to 640× 640. The batch of the model in this study is eight during training, the batch is one during inferencing, and TensorRT is not used.


## 4.2. Object Detection and Evaluation Indicators

FPS [45]: in this study, the model inference is carried out under the same equipment conditions. The same size image is used to calculate FPS and evaluate the model’s processing speed.IOU [45]: object detection uses the IOU to calculate the degree of coincidence between the predicted box and the ground-truth box, which further measures the accuracy of detecting the corresponding object in a specific dataset.mAP [45]: mAP is the sum of the average precision of all categories divided by the number of all categories. mAP@.5 is the model accuracy index when the IOU is 0.5. mAP@.5:.95 is obtained by calculating an mAP every 0.5 from IOU from 0.5 to 0.95 and finally averaging these maps.


## 4.3. Experiments on the Aluminium Defect Dataset

The aluminium defect dataset is the images of aluminium surface defects published by Baidu AI (Figure8). It contains 412 images in total. There are four types of defects, i.e., Zhen_kong, ca_shang, zang_wu, and zhe_zhou, and one image may contain different types of defects. In this study, the dataset is processed with photometric and geometric distortion. Specifically, this study brightens the picture and then flips and pans the picture to expand the dataset, increasing the number of pictures in the dataset to 1236. Then, the extended images are divided into a training set, validation set, and test set with a ratio of 8 : 1:1. It can be seen from Figure 8 that the surface defect scale of aluminium material varies greatly, which brings difficulties to the detection.Figure 8
Example of aluminium defect.


### 4.3.1. Comparative Experiments

Table2 lists the comparison experiment results of aluminium defect datasets. It can be seen that the T-model of this study surpasses most classic object detection models, indicating that the model is more suitable for the field of industrial detection. T-model size is smaller than YOLOv5X, YOLOv4, YOLOR-P6, Faster-R-CNN, etc. The F1-score and mAP of the T-model are all in the leading position. T-model surpassed the newly proposed YOLOR 1.13% on mAP@.5, and F1-score exceeded 0.05. YOLOX has better detection performance than T-model, but T-model accuracy is close to YOLOX. However, due to the swin-transformer, the FPS advantage of the T-model is not apparent. In contrast, the S-model is lighter, the reasoning speed is fast, and the accuracy after knowledge distillation is close to the T-model.Table 2
Comparative experiment of the aluminium defect dataset.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Model size (M)FPSSSDVGG-1686.5835.299.862.50.7799.7667Faster-rcnnResNet-5092.3542.373.8897.50.8452316YOLOv3DarkNet-5395.954.494.795.70.9523419YOLOv5SCSPDarkNet97.256.696.496.80.972650Efficientdet-d3EfficientNet-B393.5250.291.8590.910.9114.7820YOLOv5XCSPDarkNet94.354.995.194.70.9516721CentrenetHourglass-10460.338.651.0960.610.55124.6148RetinanetResNet-10196.5246.794.7993.320.94144.8420YOLOv4CSPDarkNet-5394.1359.183.3896.010.89245.5329YOLOR-P6CSPDarkNet9753.188.397.80.93140.8861YOLOX-sDarknet-5399.258.298.9199.30.9934.241T-modelOur backbone98.5358.898.8598.380.9881.619S-modelOur backbone97.8255.8697.8396.870.9725.136In order to more intuitively show the detection effect of the T-model, Figure9 shows the detection results of 16 pictures.Figure 9
Effect of aluminium defect detection.


### 4.3.2. Ablation Experiments

Table3 lists the ablation experiments performed with YOLOv5S as the baseline. In the ablation experiment of the backbone, this study uses the B4, S3, and SPP outputs in the backbone as the input of YOLOv5S neck for experimenting. The results in the second row show that when using the backbone of the T-model, mAP@.5 is 0.63% higher than that of YOLOv5S, and the other indicators are also slightly improved. On the neck ablation experiment, this study experiments with three C3s in the YOLOv5S backbone with one SPP output as the neck input. The third row shows that the detection effect is improved when introducing the neck in this study.Table 3
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S97.256.696.496.80.97+Our backbone97.8356.9497.8496.870.97+Our neck97.7157.1397.197.380.98This paper also conducts ablation experiments with T-model as the baseline to verify the residual fusion path in the backbone and the down sampling path in neck. The ablation results are listed in Table4. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, mAP@.5 increases by 0.58%, and the F1-score increases by 0.02. The third rows show that when introducing the down sampling path in neck, mAP@.5 is improved by 0.52%, and the F1-score is improved by 0.02. In general, the residual fusion path and the down sampling path are beneficial to industrial detection scenarios with drastic changes in scale.Table 4
Ablation experiment with the T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model97.9157.4397.2497.870.97+Residual fusion path98.0257.5798.1698.470.99+Residual fusion path + down sampling path98.5358.898.8598.380.98


### 4.3.3. Experiments on the Detector

This study examines several different detectors, including coupled detector, decoupled detector, and decoupled detector with added dual attention module. This study conducts experiments with T-model as the baseline. Specifically, this study experiments with these detectors on T-model, and the results are listed in Table5. The experimental results in the second row show that the mAP@.5 and F1-score of decoupled detector are improved by 0.78% and 0.05, respectively, compared with coupled detector. The third line shows that the detection accuracy is further improved when adding the dual attention module to the classification branch of decoupled detector, indicating that the dual attention module improves the classification accuracy.Table 5
Experiments on three detectors.
DetectormAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Coupled detector97.2457.0586.497.310.93Decoupled detector98.0257.798.1698.470.98Decoupled detector + attention98.5358.898.8598.380.98


### 4.3.4. Experiments on the Dual Attention Module

This study analyses the features of CBAM. As shown in Figure10, it is a heat map comparison between CBAM and the dual attention module. It can be seen that the dual attention module pays more attention to object areas such as wrinkles.Figure 10
(a) Original image. (b) CBAM heatmap. (c) Dual attention module heatmap. The darker the yellow, the higher the attention.
(a)(b)(c)This study uses T-model as the baseline for experiments on the dual attention module (DA). The experimental results are listed in Table6. It can be seen that the dual attention module effect surpasses CBAM.Table 6
Comparison experiment between CBAM and dual attention module.
ApproachmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model without DA97.5257.296.7597.870.97+DA98.5358.898.8598.380.98+CBAM96.6853.1595.7596.830.96


### 4.3.5. Knowledge Distillation

This study conducts experiments related to knowledge distillation on lightweight models. The experiments are performed on isomorphism object detectors and heterogeneous object detectors, respectively. On isomorphic object detectors, this study experiments T-model and S-model, Efficientdet-d7, and Efficientdet-d3. This study experiments with faster-R-CNN-ResNet50, faster-R-CNN-VGG16, YOLOv4, and YOLOv4-tiny on heterogeneous object detectors. In addition, this study also experimented with output layer knowledge distillation on T-model and S-model as a comparative experiment. Table7 lists the experimental results of various classic object detectors. It can be seen that knowledge distillation improves the detection performance of the S-model, especially when T-model and S-model are isomorphic. It proves that the knowledge distiller can improve the detection accuracy of small models without increasing the number of parameters.Table 7
Knowledge distiller experiments.
ModelsmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)S-model95.7654.2395.8295.25S-model with KD97.8255.8697.8396.87Faster-rcnn-VGG1687.2236.1877.4288.84Faster-rcnn-VGG16 with KD87.8736.2687.4686.67Efficientdet-d393.5250.291.8590.91Efficientdet-d3 with KD95.851.3491.8896.78YOLOv4-tiny84.4335.7980.4686.01YOLOv4-tiny with KD84.9435.680.4486.9


## 4.3.1. Comparative Experiments

Table2 lists the comparison experiment results of aluminium defect datasets. It can be seen that the T-model of this study surpasses most classic object detection models, indicating that the model is more suitable for the field of industrial detection. T-model size is smaller than YOLOv5X, YOLOv4, YOLOR-P6, Faster-R-CNN, etc. The F1-score and mAP of the T-model are all in the leading position. T-model surpassed the newly proposed YOLOR 1.13% on mAP@.5, and F1-score exceeded 0.05. YOLOX has better detection performance than T-model, but T-model accuracy is close to YOLOX. However, due to the swin-transformer, the FPS advantage of the T-model is not apparent. In contrast, the S-model is lighter, the reasoning speed is fast, and the accuracy after knowledge distillation is close to the T-model.Table 2
Comparative experiment of the aluminium defect dataset.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Model size (M)FPSSSDVGG-1686.5835.299.862.50.7799.7667Faster-rcnnResNet-5092.3542.373.8897.50.8452316YOLOv3DarkNet-5395.954.494.795.70.9523419YOLOv5SCSPDarkNet97.256.696.496.80.972650Efficientdet-d3EfficientNet-B393.5250.291.8590.910.9114.7820YOLOv5XCSPDarkNet94.354.995.194.70.9516721CentrenetHourglass-10460.338.651.0960.610.55124.6148RetinanetResNet-10196.5246.794.7993.320.94144.8420YOLOv4CSPDarkNet-5394.1359.183.3896.010.89245.5329YOLOR-P6CSPDarkNet9753.188.397.80.93140.8861YOLOX-sDarknet-5399.258.298.9199.30.9934.241T-modelOur backbone98.5358.898.8598.380.9881.619S-modelOur backbone97.8255.8697.8396.870.9725.136In order to more intuitively show the detection effect of the T-model, Figure9 shows the detection results of 16 pictures.Figure 9
Effect of aluminium defect detection.


## 4.3.2. Ablation Experiments

Table3 lists the ablation experiments performed with YOLOv5S as the baseline. In the ablation experiment of the backbone, this study uses the B4, S3, and SPP outputs in the backbone as the input of YOLOv5S neck for experimenting. The results in the second row show that when using the backbone of the T-model, mAP@.5 is 0.63% higher than that of YOLOv5S, and the other indicators are also slightly improved. On the neck ablation experiment, this study experiments with three C3s in the YOLOv5S backbone with one SPP output as the neck input. The third row shows that the detection effect is improved when introducing the neck in this study.Table 3
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S97.256.696.496.80.97+Our backbone97.8356.9497.8496.870.97+Our neck97.7157.1397.197.380.98This paper also conducts ablation experiments with T-model as the baseline to verify the residual fusion path in the backbone and the down sampling path in neck. The ablation results are listed in Table4. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, mAP@.5 increases by 0.58%, and the F1-score increases by 0.02. The third rows show that when introducing the down sampling path in neck, mAP@.5 is improved by 0.52%, and the F1-score is improved by 0.02. In general, the residual fusion path and the down sampling path are beneficial to industrial detection scenarios with drastic changes in scale.Table 4
Ablation experiment with the T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model97.9157.4397.2497.870.97+Residual fusion path98.0257.5798.1698.470.99+Residual fusion path + down sampling path98.5358.898.8598.380.98


## 4.3.3. Experiments on the Detector

This study examines several different detectors, including coupled detector, decoupled detector, and decoupled detector with added dual attention module. This study conducts experiments with T-model as the baseline. Specifically, this study experiments with these detectors on T-model, and the results are listed in Table5. The experimental results in the second row show that the mAP@.5 and F1-score of decoupled detector are improved by 0.78% and 0.05, respectively, compared with coupled detector. The third line shows that the detection accuracy is further improved when adding the dual attention module to the classification branch of decoupled detector, indicating that the dual attention module improves the classification accuracy.Table 5
Experiments on three detectors.
DetectormAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1Coupled detector97.2457.0586.497.310.93Decoupled detector98.0257.798.1698.470.98Decoupled detector + attention98.5358.898.8598.380.98


## 4.3.4. Experiments on the Dual Attention Module

This study analyses the features of CBAM. As shown in Figure10, it is a heat map comparison between CBAM and the dual attention module. It can be seen that the dual attention module pays more attention to object areas such as wrinkles.Figure 10
(a) Original image. (b) CBAM heatmap. (c) Dual attention module heatmap. The darker the yellow, the higher the attention.
(a)(b)(c)This study uses T-model as the baseline for experiments on the dual attention module (DA). The experimental results are listed in Table6. It can be seen that the dual attention module effect surpasses CBAM.Table 6
Comparison experiment between CBAM and dual attention module.
ApproachmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model without DA97.5257.296.7597.870.97+DA98.5358.898.8598.380.98+CBAM96.6853.1595.7596.830.96


## 4.3.5. Knowledge Distillation

This study conducts experiments related to knowledge distillation on lightweight models. The experiments are performed on isomorphism object detectors and heterogeneous object detectors, respectively. On isomorphic object detectors, this study experiments T-model and S-model, Efficientdet-d7, and Efficientdet-d3. This study experiments with faster-R-CNN-ResNet50, faster-R-CNN-VGG16, YOLOv4, and YOLOv4-tiny on heterogeneous object detectors. In addition, this study also experimented with output layer knowledge distillation on T-model and S-model as a comparative experiment. Table7 lists the experimental results of various classic object detectors. It can be seen that knowledge distillation improves the detection performance of the S-model, especially when T-model and S-model are isomorphic. It proves that the knowledge distiller can improve the detection accuracy of small models without increasing the number of parameters.Table 7
Knowledge distiller experiments.
ModelsmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)S-model95.7654.2395.8295.25S-model with KD97.8255.8697.8396.87Faster-rcnn-VGG1687.2236.1877.4288.84Faster-rcnn-VGG16 with KD87.8736.2687.4686.67Efficientdet-d393.5250.291.8590.91Efficientdet-d3 with KD95.851.3491.8896.78YOLOv4-tiny84.4335.7980.4686.01YOLOv4-tiny with KD84.9435.680.4486.9


## 4.4. Experiments on the PCB Defect Dataset

The PCB defect dataset has 1386 images, and its annotation files contain the object location information and classification. It contains six defects: missing hole, mouse bite, open circuit, spur, short, and spurious copper. Each image may contain multiple defective objects of the same type (missing holes in Figure11). In the experiments in this study, 900 images of different defect types are selected for training (mainly minor defects that are difficult to identify). In our experiment, images and annotations are divided into the training set, validation set, and experiment set according to the ratio of 6 : 2 : 2.Figure 11
Missing hole.This study conducts a comparative experiment on the PCB surface defect dataset, omitting FPS since the input size is still 640. It should be noted that the effect of the S-model is the result of distillation through the knowledge distiller. The comparison experiments are listed in Table8. It can be seen that most of the classic object detection models are not effective for small object detection, and the T-model in this study shows a better performance.Table 8
Comparative experiment of PCB surface defects.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1617.11019.9354.640.29Faster-rcnnResNet-5078.3350.2467.7474.510.7YOLOv3DarkNet-5373.3247.5090.8552.660.67YOLOv5SCSPDarkNet-5391.251.881.8292.120.86Efficientdet-d3EfficicientNet-B371.83599.4449.510.66YOLOv5XCSPDarkNet-5391.850.795.291.30.93CentrenetHourglass-10443.414.5343.7952.20.48RetinanetResNet-10113.18566.674.310.08YOLOv4CSPDarkNet-5381.250.0689.3794.780.92YOLOR-P6CSPDarkNet94.754.593.193.10.94YOLOX-sDarknet-5395.8462.0387.1693.730.91T-modelOur backbone94.7958.0288.1495.650.91S-modelOur backbone92.5156.9883.7992.330.88In order to more intuitively show the detection effect of the T-model, Figure12 shows the detection results of 16 pictures.Figure 12
PCB defect dataset detection results.


### 4.4.1. Ablation Experiment

Table9 lists the ablation experiments performed in this study with YOLOv5S as the baseline. The experimental operations are consistent with the experiments on the aluminium defect dataset. The results in the second row show that after using the backbone of the T-model in this study, mAP@.5 is 3.68% higher than that of YOLOv5S indicating that the backbone in this study is beneficial for small object detection. The third line shows that the detection effect is improved when introducing the neck of the T-model. The dual attention module makes the network pay more attention to the object and improves the small object detection effect.Table 9
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S91.251.881.8292.120.86+Our backbone91.9852.4896.0592.460.94+Our neck92.0553.0996.6293.60.95This study also uses T-model as the baseline to conduct ablation experiments to verify the residual fusion path in the backbone and the down sampling path in the neck. The experimental operations are consistent with the experiments on the aluminium defect dataset. The ablation test results are listed in Table10. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, the effect is significantly improved, indicating that the residual path can better fuse the fine-grained features in the backbone. The third row shows that by introducing the down sampling path in neck, mAP@.5 is improved by 2.4%, and the F1-score is improved by 0.03. It shows that the down sampling path is beneficial for small object detection because more fine-grained feature information is fused into neck. Overall, the residual fusion and the down sampling paths can improve the detection effect.Table 10
Ablation trials with T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model92.3954.2191.7189.840.88+Residual fusion path93.6556.4489.9494.430.92+Residual fusion path + down sampling path94.7958.0288.1495.650.91


## 4.4.1. Ablation Experiment

Table9 lists the ablation experiments performed in this study with YOLOv5S as the baseline. The experimental operations are consistent with the experiments on the aluminium defect dataset. The results in the second row show that after using the backbone of the T-model in this study, mAP@.5 is 3.68% higher than that of YOLOv5S indicating that the backbone in this study is beneficial for small object detection. The third line shows that the detection effect is improved when introducing the neck of the T-model. The dual attention module makes the network pay more attention to the object and improves the small object detection effect.Table 9
Ablation experiment with YOLOv5S.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1YOLOv5S91.251.881.8292.120.86+Our backbone91.9852.4896.0592.460.94+Our neck92.0553.0996.6293.60.95This study also uses T-model as the baseline to conduct ablation experiments to verify the residual fusion path in the backbone and the down sampling path in the neck. The experimental operations are consistent with the experiments on the aluminium defect dataset. The ablation test results are listed in Table10. The first row shows the results without weighted fusion paths in the backbone and down sampling paths in neck. The second row shows that when adding the residual fusion path to the backbone, the effect is significantly improved, indicating that the residual path can better fuse the fine-grained features in the backbone. The third row shows that by introducing the down sampling path in neck, mAP@.5 is improved by 2.4%, and the F1-score is improved by 0.03. It shows that the down sampling path is beneficial for small object detection because more fine-grained feature information is fused into neck. Overall, the residual fusion and the down sampling paths can improve the detection effect.Table 10
Ablation trials with T-model.
Approach and improvementmAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1T-model92.3954.2191.7189.840.88+Residual fusion path93.6556.4489.9494.430.92+Residual fusion path + down sampling path94.7958.0288.1495.650.91


## 4.5. Experiments on the NEU Surface Defect Dataset

North-eastern university releases the NEU surface defect dataset. This dataset collects six typical defects on the surface of the hot-rolled strip: rolled-in scale, patches, crazing, pitted surface, inclusion, and scratches. Each image has several defects of the same type. The label file marks the category and specific location of the defective object.Figure13 shows different kinds of steel defects. The categories’ defects have significant differences in appearance. Such scratches (last column) could be horizontal, vertical, slanted, etc. At the same time, each category has similarities in defects, such as rolled-in scale, crazing, and pitted surface. In addition, due to the influence of lighting and specific materials, the grey value of each category of defect images will also change. Object detection in the NEU surface defect dataset contains three difficulties: intraclass defects have significant appearance differences, interclass defects have similar aspects, and defect images are affected by changes in lighting and specific materials. According to the observation of the dataset, some cracks are concentrated in one direction because the steel is slender, and scratches are in any direction. Cracks and scratches are very similar, so scratches in specific directions are easily identified as cracks. Therefore, this study increases the data enhancement of cracks, rotates, splices crack pictures, and reduces the recognition error rate.Figure 13
Example of surface defects in steel.In order to show the generalization ability of this model in industrial detection, this study conducts a comparative experiment on the NEU surface defect database, where the S-model is the result of knowledge distillation. The comparative test results are listed in Table11. It can be seen that the T-model and S-model still have better performance, indicating that the object detection model of this study has a particular generalization ability in multiple industrial detection datasets. Figure 14 shows the different steel defect recognition results; it can be seen that all the defects are accurately identified.Table 11
Comparative experiments on NEU defect datasets.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1666.7326.687.2734.520.5Faster-rcnnResNet-5076.9539.944.1487.860.59YOLOv3DarkNet-5368.933.572.265.90.7YOLOv5SCSPDarkNet-5370.935.678.464.30.35Efficientdet-d3EfficicientNet-B365.653484.7648.440.77YOLOv5XCSPDarkNet-537237.766.872.50.69CentrenetHourglass-10439.2413.555.518.20.29RetinanetResNet-10166.5431.981.7344.620.57YOLOv4CSPDarkNet-5368.7332.995.640.540.56YOLOR-P6CSPDarkNet76.5238.350.8683.370.63YOLOX-sDarknet-5379.3840.2353.7384.880.66T-modelOur backbone75.5639.6853.4583.160.65S-modelOur backbone74.4139.249.0582.280.62Figure 14
Steel defect detection effect.


## 4.6. Experiments on Glass Bottle Bottom Mould Point Dataset

This is a dataset for recognising mould point sequences at the bottom of glass bottles, with different permutations representing different product lot numbers. The mould point identification on the bottom of glass bottles is mainly used to locate the batch of glass products online to trace the product. The dataset contains 900 glass bottle bottom mould point images, each with a native resolution of 800 × 780 and 18 types. Each image has a corresponding label file. In this study, the label files correspond to the images one by one, and the training set, test set, and validation set are divided according to the ratio of 6 : 2 : 2.Figure15 shows the images of different glass bottle bottom mould points, and the high similarity between objects makes detection difficult. In order to demonstrate the generalization ability of the model in industrial detection, this study conducts a comparative experiment on the glass bottle bottom mould point database, where the S-model results from knowledge distillation. The comparison test results are listed in Table 12. It can be seen that T-model and S-model still have better performance, indicating that the object detection model in this study has a specific generalization ability on multiple industrial detection datasets. Figure 16 shows the recognition effect of 16 different model point images. It can be seen that all model point objects are accurately recognized and positioned.Figure 15
Example of bottom mould point data.Table 12
Comparative experiments on NEU defect datasets.
ApproachBackbonemAP@.5 (%)mAP@.5-.95 (%)Precision (%)Recall (%)F1SSDVGG-1670.4432.5278.9449.480.5Faster-rcnnResNet-5083.6238.7374.8975.40.59YOLOv3DarkNet-5391.3740.6684.3992.330.7YOLOv5SCSPDarkNet-5393.2162.185.1991.440.35Efficientdet-D3EfficicientNet-B386.3955.6767.4989.90.77YOLOv5XCSPDarkNet-5398.464.891.4799.040.69CentrenetHourglass-10468.5143.1884.1448.740.29RetinanetResNet-10180.7949.8966.2877.560.57YOLOv4CSPDarkNet-5396.150.0689.3794.780.56YOLOR-P6CSPDarkNet99.359.953.5990.63YOLOX-sDarknet-5399.5570.6880.3299.90.66T-modelOur backbone99.3669.5381.199.50.89S-modelOur backbone98.165.279.4498.910.89Figure 16
Glass bottle bottom mould point recognition effect.


## 5. Conclusions

This study proposes a deep learning model for industrial quality detection. The model consists of T-model and S-model, which aims to meet detection tasks under different conditions. The model uses a combination of swin-transformer and convolution to extract the global information of the image. A dual attention module is designed to improve the neck’s attention to important areas of the image, thereby improving the detection effect of the model. This study also designs a knowledge distiller using a dual attention module to improve the detection effect of the S-model. Finally, this study designs an adaptive loss weight updater to adjust the loss weights automatically. The experimental results show that the T-model in this study has high accuracy and is suitable for online data processing in scenarios such as IoT intelligent computing and big data analysis. The S-model in this study is fast and suitable for use in scenarios such as edge computing. In general, the model in this study can meet the needs of different scenarios and achieve a balance between accuracy and speed.



---

*Source: 6174255-2022-10-10.xml*