# Automatic Fabric Defect Detection Based on an Improved YOLOv5

Authors: Rui Jin; Qiang Niu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Fabric defect detection is particularly remarkable because of the large textile production demand in China. Traditional manual detection method is inefficient, time-consuming, laborious, and costly. A deep learning technique is proposed in this work to perform automatic fabric defect detection by improving a YOLOv5 object detection algorithm. A teacher-student architecture is used to handle the shortage of fabric defect images. Specifically, a deep teacher network could precisely recognize fabric defects. After information distillation, a shallow student network could do the same thing in real-time with minimal performance degeneration. Moreover, multitask learning is introduced by simultaneously detecting ubiquitous and specific defects. Focal loss function and central constraints are introduced to improve the recognition performance. Evaluations are performed on the publicly available Tianchi AI and TILDA databases. Results indicate that the proposed method performs well compared with other methods and has excellent defect detection ability in the collected textile images.


---

## Body

## 1. Introduction

The textile industry is the traditional advantageous industry in China’s economic development and is an important livelihood industry. The quality of textiles has a great influence on the textile industry. Fabric defects would reduce the cost and profit by 45%–65% [1]. Therefore, defect detection plays an important role in the control of textile quality. Traditional textile defect testing is usually achieved by training skilled operators with high training costs, and the manual detection efficiency is low (the detection speed is less than 20 m/min). The error and leakage rates are high due to personnel fatigue or other subjective factors. Hence, how to detect fabric defects by automatic means has become an engaging, difficult research spot in the field of textile industry and machine vision.The core of machine-vision-based fabric defect detection is extracting the characteristics related to defects from the textile images. A detailed review of the machine-vision-based fabric defect detection methods could be found in References [2, 3]. Thomas and Cattoen [4] used the gray-scale means of image rows and columns as defect-related characteristics, which are sensitive to illumination changes. Ye [5] presented the fuzzy inference based on image histogram statistical variables, which is robust to defects’ rotation and translation. However, handling complex image texture is difficult. For complex texture images, researchers proposed methods based on edges [6], local binary patterns [7, 8], contour waves [9], and gray co-occurrence matrix [10, 11]. These methods perform well in identifying defective images but have difficulty recognizing specific fabric defects. Moreover, several researchers used the characteristics of the high-frequency parts, such as Fourier transform methods [12, 13], Gabor filter methods [7, 14], and wavelet transform methods [15, 16]. Compared with fabric defect detection in the spatial domain, it has more space-time overheads in the frequency domain.Deep learning has been widely used in the fields of computer vision [17–19]. Researchers designed deep neural networks to realize fabric defect detection in a data-driven manner. Liu et al. [20] proposed using multistage GAN the detection of fabric defects through unsupervised data reconstruction. Hence, it could overcome the challenges of diversified fabric defects. Mei et al. [21] introduced a multiscale convolutional denoising autoencoder to learn the reconstruction of textile images. The reconstruction errors are utilized to realize automatic defect detection. Xian et al. [22] studied the problem of metallic surface defect detection that is similar to fabric defect detection. Convolutional neural network-based segmentation is used to detect and recognize defect regions. Wei et al. [23] used faster-RCNN to detect fabric defects automatically. It achieves satisfied detection performance benefiting from faster-RCNN’s strong feature engineering ability. However, faster-RCNN has large space-time complexity due to its two-stage object detection scheme. Jing et al. [24] improved YOLOv3, which is a single-stage object detection method with real-time detection performance. Then, it could better detect fabric defects.In addition, several researchers studied the model-driven fabric defect detection methods, such as Markov random field [25], autoregression [26], and sparse dictionary [27, 28]. After effective training, these methods could identify small-region defects. However, they are vulnerable to external signals such as noise and light.In conclusion, many researchers have proposed different methods to study how to detect fabric defects. However, detecting fabric defects is still challenging owing to many kinds of defects with large differences and uneven distributions. These problems lead to the difficulty of designing an effective system to detect and localize the fabric defects automatically. Moreover, the proposed system is required to operate faster and could be realized in an intelligent edge device platform.According to the above requirements, a lightweight fabric defect detection method is proposed by improving YOLOv5 [29] based on the special needs of the defect detection system. It could detect and recognize special fabric defects in real time. The main contributions of this article are as follows.A teacher-student architecture is introduced to detect fabric defects. The deep teacher network could precisely recognize fabric defects. After information distillation, a shallow student network could do the same thing in real time with minimal performance degeneration. The student network could be deployed in the edge equipment because of its low space-time overheads.To solve the problems of many kinds of fabric defects that are difficult to be distinguished, a multitask learning strategy is proposed to detect ubiquitous and specific defects simultaneously. Such a strategy could fully utilize the complementary between ubiquitous and specific defects. Moreover, an attention mechanism is used to enhance the defect-related features.To handle data imbalance and small-region defects better, the focal loss function [30] is employed to mitigate data imbalance. The center loss is introduced as a constraint to increase the interclass distance while reducing the intraclass distance, hence improving the recognition performance of specific defects.The proposed method is evaluated on the publicly available Tianchi AI and TILDA databases. The results reveal its ability to detect and recognize specific fabric defects. To verify the generalization capability of the proposed algorithm, it is tested on self-collected fabric defect images and achieves good results.


## 2. Related Technologies

### 2.1. Convolutional Neural Networks

Convolutional neural networks (CNNs) are widely used in computer vision tasks [31]. CNN is a kind of feed-forward neural network that contains convolutional computation and deep structure. It has the representation learning ability to learn structured and translation-invariant information from input images. Compared with fully connected operations, CNN has the advantage of small computational overhead. A common CNN-based computer vision system consists of the following parts:Input layer: it performs gray processing, normalization, and data augmentation on the input images.Convolutional layer: it performs convolutional operations in each layer to ensure the forward and backward transmission of the information. The feature map of the lth layer is derived from that of thel−1th layer using the convolutional operation, as follows:(1)yli,j=Kil∗xlrj,where Kil is the weight of ith convolutional kernel in the lth layer, and xlrj represents the jth local region being calculated in the lth layer.Activation layer: it always follows the convolutional layer to introduce nonlinearity. Hence, the network could have better representation learning ability. Commonly used activation functions contain Sigmoid, Tanh, ReLu, and their variants. Figure1 shows the curves of three different activation functions.(1)
Pooling layer: it is used to subsample the feature maps to decrease computational overheads. It could also mitigate the overfitting phenomenon. Commonly used pooling functions consist of the average and maximum pooling strategies.(2)
Output layer: it presents various structures according to different computer vision applications. For classification tasks, the SoftMax function is often used in the output layer to calculate the probability that the input belongs to each category, thus obtaining the classification results.Figure 1
Curves of the Sigmoid, Tanh, and ReLu activation functions.The five components above are used in the improved YOLOv5. They would not be introduced in detail in the following sections.


### 2.2. Object Detection Algorithm

Object detection is one of the essential issues in the field of computer vision. It enables the computer to discover and locate targets of interest from images automatically, such as flaws in the fabric. Deep learning-based object detection algorithms have achieved great successes recently. Commonly used methods include RCNN [32], fast-RCNN [33], faster-RCNN [34], SDD [35], and YOLO [36]. However, the above methods have difficulty meeting the real-time requirements of the fabric defect detection system because they have high computational overheads. To balance precision and speed, a lightweight object detection network, named YOLOv5, is used in this work. The traditional YOLOv5 is improved based on the characteristics of the fabric defects, such that it can be applied to the fabric defect detection system.Figure2 demonstrates the structure of the traditional YOLOv5, which mainly includes Bakbone, PANet, and Output. Bakbone is used to perform feature engineering from input images. PANet could obtain visual features robust to scale changes due to the used pyramid structure. The positions are output, and the regions of interest are classified simultaneously. Assuming the input image size as 608 ∗ 608 ∗ 3 (height ∗ width ∗ channels), the Output part could output three different scales of features with dimensions of 76 ∗ 76 ∗ 255, 38 ∗ 38 ∗ 255, and 19 ∗ 19 ∗ 255. Specific details of the YOLOv5 network could be found in [29].Figure 2
Pipeline of YOLOv5.


### 2.3. Attention Mechanism

The attention mechanism draws on human’s selective attention characteristic. Specifically, a human being could quickly scan the global image and concentrate on the regions of interest. Then, detail information of these regions are obtained, and useless information is suppressed. Based on different applications, attention mechanism could be divided into temporal attention, spatial attention, and channel attention. Temporal attention [37] could assign different weights to sequence features. Then, the model could automatically focus on important sequence features, thus enhancing the ability to process sequence data without increasing the computational costs. Spatial attention [38] transforms the spatial information in the original image into another space and retains the key information, thereby identifying the substantial areas and increasing the attention on these areas. Channel attention [39] excavates effective features from the channel dimension and suppresses task-independent features, thus improving network performance.For fabric defect detection, temporal attention cannot be used because the input is a static image. Considering that the defects may occupy a small proportion in the overall image, spatial attention can be used to pay more attention to small-region defects. Moreover, channel attention is used to refine features and improve the algorithm performance. Figure3 shows that input feature F is initially processed by max-pooling (MaxPool) and average-pooling (AvgPool). Then, channel and spatial attention realize feature transformation with the shared three-layer MLP and convolutional operation, respectively. Finally, the sigmoid activation function is used to calculate different attention weights.Figure 3
Illustration of spatial and channel attention mechanisms.


## 2.1. Convolutional Neural Networks

Convolutional neural networks (CNNs) are widely used in computer vision tasks [31]. CNN is a kind of feed-forward neural network that contains convolutional computation and deep structure. It has the representation learning ability to learn structured and translation-invariant information from input images. Compared with fully connected operations, CNN has the advantage of small computational overhead. A common CNN-based computer vision system consists of the following parts:Input layer: it performs gray processing, normalization, and data augmentation on the input images.Convolutional layer: it performs convolutional operations in each layer to ensure the forward and backward transmission of the information. The feature map of the lth layer is derived from that of thel−1th layer using the convolutional operation, as follows:(1)yli,j=Kil∗xlrj,where Kil is the weight of ith convolutional kernel in the lth layer, and xlrj represents the jth local region being calculated in the lth layer.Activation layer: it always follows the convolutional layer to introduce nonlinearity. Hence, the network could have better representation learning ability. Commonly used activation functions contain Sigmoid, Tanh, ReLu, and their variants. Figure1 shows the curves of three different activation functions.(1)
Pooling layer: it is used to subsample the feature maps to decrease computational overheads. It could also mitigate the overfitting phenomenon. Commonly used pooling functions consist of the average and maximum pooling strategies.(2)
Output layer: it presents various structures according to different computer vision applications. For classification tasks, the SoftMax function is often used in the output layer to calculate the probability that the input belongs to each category, thus obtaining the classification results.Figure 1
Curves of the Sigmoid, Tanh, and ReLu activation functions.The five components above are used in the improved YOLOv5. They would not be introduced in detail in the following sections.


## 2.2. Object Detection Algorithm

Object detection is one of the essential issues in the field of computer vision. It enables the computer to discover and locate targets of interest from images automatically, such as flaws in the fabric. Deep learning-based object detection algorithms have achieved great successes recently. Commonly used methods include RCNN [32], fast-RCNN [33], faster-RCNN [34], SDD [35], and YOLO [36]. However, the above methods have difficulty meeting the real-time requirements of the fabric defect detection system because they have high computational overheads. To balance precision and speed, a lightweight object detection network, named YOLOv5, is used in this work. The traditional YOLOv5 is improved based on the characteristics of the fabric defects, such that it can be applied to the fabric defect detection system.Figure2 demonstrates the structure of the traditional YOLOv5, which mainly includes Bakbone, PANet, and Output. Bakbone is used to perform feature engineering from input images. PANet could obtain visual features robust to scale changes due to the used pyramid structure. The positions are output, and the regions of interest are classified simultaneously. Assuming the input image size as 608 ∗ 608 ∗ 3 (height ∗ width ∗ channels), the Output part could output three different scales of features with dimensions of 76 ∗ 76 ∗ 255, 38 ∗ 38 ∗ 255, and 19 ∗ 19 ∗ 255. Specific details of the YOLOv5 network could be found in [29].Figure 2
Pipeline of YOLOv5.


## 2.3. Attention Mechanism

The attention mechanism draws on human’s selective attention characteristic. Specifically, a human being could quickly scan the global image and concentrate on the regions of interest. Then, detail information of these regions are obtained, and useless information is suppressed. Based on different applications, attention mechanism could be divided into temporal attention, spatial attention, and channel attention. Temporal attention [37] could assign different weights to sequence features. Then, the model could automatically focus on important sequence features, thus enhancing the ability to process sequence data without increasing the computational costs. Spatial attention [38] transforms the spatial information in the original image into another space and retains the key information, thereby identifying the substantial areas and increasing the attention on these areas. Channel attention [39] excavates effective features from the channel dimension and suppresses task-independent features, thus improving network performance.For fabric defect detection, temporal attention cannot be used because the input is a static image. Considering that the defects may occupy a small proportion in the overall image, spatial attention can be used to pay more attention to small-region defects. Moreover, channel attention is used to refine features and improve the algorithm performance. Figure3 shows that input feature F is initially processed by max-pooling (MaxPool) and average-pooling (AvgPool). Then, channel and spatial attention realize feature transformation with the shared three-layer MLP and convolutional operation, respectively. Finally, the sigmoid activation function is used to calculate different attention weights.Figure 3
Illustration of spatial and channel attention mechanisms.


## 3. Proposed Method

Figure4 illustrates the flow chart of the overall algorithm: (1) training stage: fabric images after data augmentation are sent to the teacher network to detect specific fabric defects. Then, the defect-related knowledge is distilled from the teacher network to the lightweight student network. (2) Testing stage: the student network is used to detect specific fabric defects in real-time performance and with minimal performance degradation. The testing stage requires deploying the student network on the NVIDIA JETSON TX2 platform based on TensorRT, which is used to accelerate the student network.Figure 4
Flow chart of the overall algorithm.


### 3.1. Teacher Network Structure

The structure of the proposed teacher network is shown in Figure5. The feature extraction part and multiscale information extraction part of the teacher network are implemented using Backbone and PANet of the YOLOv5 network. Their specific structures have been introduced in Section 2.2 and are not repeated here. Two improvements are presented to perform better fabric defect detection.(1)
Attention enhancement mechanism: the defect areas may occupy small regions in the overall textile image. Extracting defect-related features from these small regions is still a problem, even if PANet could extract the context information. Hence, the attention enhancement mechanism is introduced to mitigate the problem. First, spatial attention is used to enhance the network’s sensitivity to small defect areas. Then, channel attention is used to suppress the nondefective features, thus highlighting the defective features. Assuming that the output of PANet isF, spatial attention weight As(F) and channel attention weights Ac(F) could be calculated as follows:(2)AcF=σMLPFavgs+MLPFmaxs,AsF=σConvFavgs;Fmaxs,where MLP () represents a shared multilayer perceptron (three layers, the number of neurons is m, m/4, m, respectively; m represents the channel dimension of F.) and Conv () represents a convolution operation with the kernel size 7 ∗ 7. The attention enhancement mechanisms used in this work are defined as follows:(3)F′=AcF⊗F,F″=AsF′⊗F′.(2)
Multitask learning strategy: the fabric defect detection task is usually divided into ubiquitous defect detection and specific defect recognition. Complementarity exists between these two tasks. Hence, the multitask learning strategy is introduced to utilize the complementarity fully. Specifically, two detection heads are designed to detect ubiquitous defects and recognize specific defects. A fusion model is then proposed to fuse the outputs of two detection heads to predict a more accurate defect recognition probability. Details are as follows:(1)
For the detection head to detect ubiquitous defects, the defective probability of the ROI with the largest defective probability is defined asPA. Then, the normal probability of the given fabric image is defined as PN = 1 − PA.(2)
For the detection head to recognize specific defects, the defective probability of each ROI is defined asPj (j = 1, …, M), where M indicates the number of ROIs.(3)
PN and all Pj are concatenated, and the concatenated vector is then sent into the SoftMax activation function for normalization. Then, the probability that the given fabric image belongs to a normal sample or a certain defect could be obtained.Figure 5
Structure of the proposed teacher network.


### 3.2. Student Network Structure

Figure6 exhibits the structure of the proposed student network. Different from the teacher network, the student network performs the following lightweight processing:(1)
The backbone part is thin. Specifically, only two sets of BottleNeckCSP modules are preserved in the new backbone part. Details of the BottleNeckCSP module could be found in [29].(2)
The PANet is removed to reduce the space-time complexity. The student network relies on the knowledge distilled from the teacher network to extract multiscale features.Figure 6
Structure of the proposed student network.The rest of the student network, including the attention enhancement, multitask learning strategy, and information fusion, are the same with the teacher network.


### 3.3. Loss Functions

The network is trained in a multitask learning manner, and a weighted combined loss function is presented to optimize the network. The loss functions used consist of the following sections:(1)
The ubiquitous defect detection is termed as a binary classification problem. A cross-entropy loss functionLT is used and defined as follows:(4)LT=−1N∑i=1Nyilnpi+1−yiln1−pi,where yi represents the sample label, pi represents the output probability of the ubiquitous defects detection head, and N represents the number of samples.(2)
The specific defect detection is termed as a multiclass problem. A SoftMax loss functionLs is used and defined as follows:(5)LS=−∑i=1Ky˜ilogsi,where K represents the kinds of specific defects, y˜i represents the one-hot encoding of the ground truth label, and si indicates the probability that the sample belongs to the ith defect.(3)
Considering the sample imbalance in the ubiquitous defect detection head, focal loss functionLF is used to mitigate the problem. LF is defined as follows:(6)LF=−α1−piγlogpi,where the hyperparameters α and γ are used to alleviate the imbalance problem of positive and negative samples and difficult samples, respectively.(4)
To improve the feature discriminability in the specific defect detection head, central loss functionLC is employed to increase the interclass distances while reducing the innerclass distances of learned features. LC is defined as follows:(7)LC=12∑i=1Nxi−cyi22,where xi represents the sample encoding, and cyi is the center of the corresponding category, which xi belongs to.The final loss function of the proposed method is calculated in a weighted manner as follows:(8)L=w1LT+w2LS+w3LF+w4Lc,where the weights w1,w2,w3,andw4 are set to 0.4, 0.4, 0.1, and 0.1, respectively. Settings of different weights are obtained based on the crossvalidation on the publicly available databases.


## 3.1. Teacher Network Structure

The structure of the proposed teacher network is shown in Figure5. The feature extraction part and multiscale information extraction part of the teacher network are implemented using Backbone and PANet of the YOLOv5 network. Their specific structures have been introduced in Section 2.2 and are not repeated here. Two improvements are presented to perform better fabric defect detection.(1)
Attention enhancement mechanism: the defect areas may occupy small regions in the overall textile image. Extracting defect-related features from these small regions is still a problem, even if PANet could extract the context information. Hence, the attention enhancement mechanism is introduced to mitigate the problem. First, spatial attention is used to enhance the network’s sensitivity to small defect areas. Then, channel attention is used to suppress the nondefective features, thus highlighting the defective features. Assuming that the output of PANet isF, spatial attention weight As(F) and channel attention weights Ac(F) could be calculated as follows:(2)AcF=σMLPFavgs+MLPFmaxs,AsF=σConvFavgs;Fmaxs,where MLP () represents a shared multilayer perceptron (three layers, the number of neurons is m, m/4, m, respectively; m represents the channel dimension of F.) and Conv () represents a convolution operation with the kernel size 7 ∗ 7. The attention enhancement mechanisms used in this work are defined as follows:(3)F′=AcF⊗F,F″=AsF′⊗F′.(2)
Multitask learning strategy: the fabric defect detection task is usually divided into ubiquitous defect detection and specific defect recognition. Complementarity exists between these two tasks. Hence, the multitask learning strategy is introduced to utilize the complementarity fully. Specifically, two detection heads are designed to detect ubiquitous defects and recognize specific defects. A fusion model is then proposed to fuse the outputs of two detection heads to predict a more accurate defect recognition probability. Details are as follows:(1)
For the detection head to detect ubiquitous defects, the defective probability of the ROI with the largest defective probability is defined asPA. Then, the normal probability of the given fabric image is defined as PN = 1 − PA.(2)
For the detection head to recognize specific defects, the defective probability of each ROI is defined asPj (j = 1, …, M), where M indicates the number of ROIs.(3)
PN and all Pj are concatenated, and the concatenated vector is then sent into the SoftMax activation function for normalization. Then, the probability that the given fabric image belongs to a normal sample or a certain defect could be obtained.Figure 5
Structure of the proposed teacher network.


## 3.2. Student Network Structure

Figure6 exhibits the structure of the proposed student network. Different from the teacher network, the student network performs the following lightweight processing:(1)
The backbone part is thin. Specifically, only two sets of BottleNeckCSP modules are preserved in the new backbone part. Details of the BottleNeckCSP module could be found in [29].(2)
The PANet is removed to reduce the space-time complexity. The student network relies on the knowledge distilled from the teacher network to extract multiscale features.Figure 6
Structure of the proposed student network.The rest of the student network, including the attention enhancement, multitask learning strategy, and information fusion, are the same with the teacher network.


## 3.3. Loss Functions

The network is trained in a multitask learning manner, and a weighted combined loss function is presented to optimize the network. The loss functions used consist of the following sections:(1)
The ubiquitous defect detection is termed as a binary classification problem. A cross-entropy loss functionLT is used and defined as follows:(4)LT=−1N∑i=1Nyilnpi+1−yiln1−pi,where yi represents the sample label, pi represents the output probability of the ubiquitous defects detection head, and N represents the number of samples.(2)
The specific defect detection is termed as a multiclass problem. A SoftMax loss functionLs is used and defined as follows:(5)LS=−∑i=1Ky˜ilogsi,where K represents the kinds of specific defects, y˜i represents the one-hot encoding of the ground truth label, and si indicates the probability that the sample belongs to the ith defect.(3)
Considering the sample imbalance in the ubiquitous defect detection head, focal loss functionLF is used to mitigate the problem. LF is defined as follows:(6)LF=−α1−piγlogpi,where the hyperparameters α and γ are used to alleviate the imbalance problem of positive and negative samples and difficult samples, respectively.(4)
To improve the feature discriminability in the specific defect detection head, central loss functionLC is employed to increase the interclass distances while reducing the innerclass distances of learned features. LC is defined as follows:(7)LC=12∑i=1Nxi−cyi22,where xi represents the sample encoding, and cyi is the center of the corresponding category, which xi belongs to.The final loss function of the proposed method is calculated in a weighted manner as follows:(8)L=w1LT+w2LS+w3LF+w4Lc,where the weights w1,w2,w3,andw4 are set to 0.4, 0.4, 0.1, and 0.1, respectively. Settings of different weights are obtained based on the crossvalidation on the publicly available databases.


## 4. Experimental Results

### 4.1. Databases

One public database comes from the Xuelang Tianchi AI Challenge. It contains 3,331 labeled images with the rectangular locations to label the defects. The number of normal pictures is 2,163, and the number of defective pictures is 1,168. It has 22 kinds of defects, including jumps, knots, stains, puncture holes, and lacking warp. The data distribution on the database shows an unbalanced state in which the number of normal pictures is much higher than the number of defective pictures. Using the same experimental protocol as [19], the specific defect category is reintegrated into puncture hole, knots, rubbing hole, thin spinning, jumps, hanging warp, lacking warp, brushed hole, stains, and others. In experiments, 70% of the entire database is taken as the training set, and the remaining 30% are the test set. Several training samples and their labels are shown in Figure 7.Figure 7
Illustrations of some training samples and their labels on the Tianchi AI database.Another used public database is TILDA, a well-known fabric texture database containing eight kinds of representative fabric categories. Seven error classes and a correct class are defined according to the textile atlas analysis. Similar to [40], 300 fabric images are chosen and are divided into six categories, such as holes, scratch, knots, stain, carrying, and normal. Each class consists of 50 fabric images, and each image is resized to 256 × 256 pixels. In experiments, 70% of the entire database is taken as the training set, and the remaining 30% are the test set. Figure 8 demonstrates several samples and their labels.Figure 8
Illustrations of several samples and their labels on the TILDA database.


### 4.2. Evaluation Metrics

The defect detection algorithm proposed in this work could distinguish between normal and defect images and identify specific fabric defects. Therefore, area under the ROC curve (AUC) and mean average precision (mAP) are used as metrics for evaluation. The former reflects the algorithm’s ability to distinguish between normal and defective fabric images, whereas the latter reflects the algorithm’s ability to recognize specific fabric defects. To calculate AUC and mAP, precision (P) and recall (R) are calculated initially, as follows:(9)P=TPTP+FP,R=TPTP+FN,where TP (true positive) represents the number of samples whose labels are positive, and the actual forecasts are positive. FP (false positive) indicates the number of samples whose labels are negative, and the actual forecasts are positive. FN (false negative) represents the number of samples whose labels are positive, and the actual forecasts are negative. Based on the calculated P and R, the P-R curve could be obtained. Then, the ROC curve could be obtained. The cover area of the ROC curve is AUC.mAP represents the mean of different APs, where AP represents the area under the P-R curve. mAP is calculated as follows:(10)mAP=1K∑i=1KRi−Ri−1Pi,where k represents the number of categories.


### 4.3. Qualitative Analysis

A qualitative analysis of the proposed method is performed from three aspects: (1) the ability of the proposed teacher network to detect specific defects on public databases is evaluated, and OurNet is used for comparison; (2) the accuracy of the proposed teacher network to locate the defect areas is evaluated, and the improved YOLOv3 proposed by Jing et al. [24] is used for comparison; and (3) comparisons between the teacher and student networks are performed on self-collected fabric images to verify the generalization performance of the proposed method. Quantitative comparisons between the teacher and student networks will be introduced in the following section.Figure9 demonstrates the comparisons between the proposed teacher network and OurNet in detecting specific defects on the Tianchi AI database. The results show that our method successfully recognizes different defect types benefiting from the used multitask learning, focal loss function, and the center loss constraint. By contrast, OurNet fails to identify the puncture hole defects. It also mistakes the brushed hole and thin spinning defects for others and jumps defects, respectively.Figure 9
Comparisons between the teacher network and OurNet in detecting specific defects.Figure10 shows the location results between the proposed teacher network and the improved YOLOv3 proposed by Jing et al. [24] on the Tianchi AI database. Types of specific defects are labeled under each subfigure for a clearer view. In the subfigure, the green box represents the real defect area, the red box is the positioning result of the proposed teacher network, and the yellow box is the positioning result of the improved YOLOv3. Figure 10 shows that the defect regions predicted by the proposed method are more accurate than those predicted by the improved YOLOv3. Such superiority may be a benefit from the strong YOLOv5 and our improvements. The improved YOLOv3 suffers from positioning small defect areas, although it could detect most defects. For example, it fails to detect the hanging warp and jump defects.Figure 10
Comparisons between the teacher network and the improved YOLOv3 in positioning defect areas.Figure11 compares the teacher and student networks on self-collected fabric images, specifically, their performance in positioning defect areas. In each subfigure, the green box represents the real defect area, the red box is the positioning result of the teacher network, and the yellow box is the positioning result of the student network. The teacher network could more accurately identify the defect areas. The defect detection performance of the student network is slightly weaker than that of the teacher network. However, the student network has lower space-time overheads; thus, it is more suitable to be arranged for embedded systems.Figure 11
Comparisons between the teacher and student networks on self-collected fabric images.


### 4.4. Quantitative Analysis Results

An ablation study is performed on the Tianchi AI database to verify the effects of different improvement methods, including multitask learning, focal loss, and central loss constraints. The results are presented in Table1. The ablation study of the teacher network shows that the student network has similar results.Table 1
Ablation study of the teacher network on the Tianchi AI database.
Attention moduleMultitask learningFocal lossCentral loss constraintsAUCmAP0.9380.403√0.9570.412√√0.9650.431√√√0.9710.441√√√0.9730.442√√√√0.9810.447Table1 shows that the teacher network is degraded into traditional YOLOv5 when none of the improvements is used. Compared with the YOLOv5-based detection method, the introduced attention module could lead to an improved performance with increased AUC and mAP. Then, AUC and mAP are further improved by simultaneously detecting ubiquitous and specific defects with the proposed multitask learning strategy because of the complementarity between different tasks. Based on the multitask learning strategy, the introduction of the focal loss function and central loss constraint could further improve the defect detection results. Simultaneously using all improvements achieves the best performance on the Tianchi AI database, which verifies the effects of different improvement methods.A quantitative comparison between the teacher and student networks is presented in Table2. The identification times are tested on an Nvidia JETSON TX2. The table shows that the student network could still meet the needs of fabric defect detection, despite the performance degradation observed compared with the teacher network. More importantly, the identification time of the student network is approximately half of the teacher network. Its identification time guarantees the real-time performance on embedded devices.Table 2
Quantitative comparisons between the teacher and student networks on the Tianchi AI database.
AUCmAPIdentification times (ms)YOLOv50.9570.41232Teacher network0.9810.44735Student network0.9520.40616Finally, comparisons with other mainstream methods are performed to verify the effectiveness of the proposed method. The improved YOLOv3 [24] and the pretrained deep CNN [40] are selected as the fabric defect detection algorithms. Faster-RCNN [34] and YOLOv5 [29] are selected as the universal object detection methods. The comparison results are presented in Table 3.Table 3
Comparisons of different fabric defect detection algorithms on the Tianchi AI database.
AUCmAPOurNet [41]0.7870.104OurNet-VGG160.8480.288OurNet-ResNet0.8820.311Improved YOLOv3 [24]0.9270.372Jing et al. [40]0.9320.382YOLOv5 [29]0.9570.412Faster-RCNN [34]0.9560.413Student network0.9520.406Teacher network0.9810.447The above table shows that the original OurNet based on AlexNet has poor defect detection performance because it fails to handle small defect areas well. Two variants of OurNet, namely, OurNet-VGG16 and OurNet-ResNet, obtain better performance benefit from extracting better features with deeper structures. Jing et al. [24] achieves better defect detection performance using improved YOLOv3 networks. A pretrained CNN is also beneficial in boosting the defect detection performance as proposed by Jing et al. [40]. YOLOv5 and faster-RCNN achieve similar defect detection performance benefiting from their strong power in object detection. Both methods are superior to the student network proposed in this work, but the time overhead is relatively large. The proposed teacher network achieves the best fabric defect detection performance, whereas the student network provides an alternative to detect fabric defects with acceptable accuracy on embedded devices.Table4 presents the comparisons between different methods on the TILDA database. OurNet [41] and its variants perform much better than on the Tianchi AI database because the TILDA database contains fewer categories and equal samples per category. Improved YOLOv3 [24] proposed by Jing et al. [40] achieve similar performance due to the reason discussed above. Similar to the comparisons on the Tianchi AI database, two state-of-the-art detectors, YOLOv5 [29] and faster-RCNN [34], obtained higher AUC and mAP compared with that of the proposed student network. The proposed teacher network still achieves the best defect detection performance, which verifies the accuracy of the proposed method.Table 4
Comparisons of different fabric defect detection algorithms on the TILDA database.
AUCmAPOurNet [41]0.8660.301OurNet-VGG160.9120.346OurNet-ResNet0.9260.382Jing et al. [40]0.9580.411YOLOv5 [29]0.9700.442Faster-RCNN [34]0.9720.443Student network0.9650.428Teacher network0.9880.451


## 4.1. Databases

One public database comes from the Xuelang Tianchi AI Challenge. It contains 3,331 labeled images with the rectangular locations to label the defects. The number of normal pictures is 2,163, and the number of defective pictures is 1,168. It has 22 kinds of defects, including jumps, knots, stains, puncture holes, and lacking warp. The data distribution on the database shows an unbalanced state in which the number of normal pictures is much higher than the number of defective pictures. Using the same experimental protocol as [19], the specific defect category is reintegrated into puncture hole, knots, rubbing hole, thin spinning, jumps, hanging warp, lacking warp, brushed hole, stains, and others. In experiments, 70% of the entire database is taken as the training set, and the remaining 30% are the test set. Several training samples and their labels are shown in Figure 7.Figure 7
Illustrations of some training samples and their labels on the Tianchi AI database.Another used public database is TILDA, a well-known fabric texture database containing eight kinds of representative fabric categories. Seven error classes and a correct class are defined according to the textile atlas analysis. Similar to [40], 300 fabric images are chosen and are divided into six categories, such as holes, scratch, knots, stain, carrying, and normal. Each class consists of 50 fabric images, and each image is resized to 256 × 256 pixels. In experiments, 70% of the entire database is taken as the training set, and the remaining 30% are the test set. Figure 8 demonstrates several samples and their labels.Figure 8
Illustrations of several samples and their labels on the TILDA database.


## 4.2. Evaluation Metrics

The defect detection algorithm proposed in this work could distinguish between normal and defect images and identify specific fabric defects. Therefore, area under the ROC curve (AUC) and mean average precision (mAP) are used as metrics for evaluation. The former reflects the algorithm’s ability to distinguish between normal and defective fabric images, whereas the latter reflects the algorithm’s ability to recognize specific fabric defects. To calculate AUC and mAP, precision (P) and recall (R) are calculated initially, as follows:(9)P=TPTP+FP,R=TPTP+FN,where TP (true positive) represents the number of samples whose labels are positive, and the actual forecasts are positive. FP (false positive) indicates the number of samples whose labels are negative, and the actual forecasts are positive. FN (false negative) represents the number of samples whose labels are positive, and the actual forecasts are negative. Based on the calculated P and R, the P-R curve could be obtained. Then, the ROC curve could be obtained. The cover area of the ROC curve is AUC.mAP represents the mean of different APs, where AP represents the area under the P-R curve. mAP is calculated as follows:(10)mAP=1K∑i=1KRi−Ri−1Pi,where k represents the number of categories.


## 4.3. Qualitative Analysis

A qualitative analysis of the proposed method is performed from three aspects: (1) the ability of the proposed teacher network to detect specific defects on public databases is evaluated, and OurNet is used for comparison; (2) the accuracy of the proposed teacher network to locate the defect areas is evaluated, and the improved YOLOv3 proposed by Jing et al. [24] is used for comparison; and (3) comparisons between the teacher and student networks are performed on self-collected fabric images to verify the generalization performance of the proposed method. Quantitative comparisons between the teacher and student networks will be introduced in the following section.Figure9 demonstrates the comparisons between the proposed teacher network and OurNet in detecting specific defects on the Tianchi AI database. The results show that our method successfully recognizes different defect types benefiting from the used multitask learning, focal loss function, and the center loss constraint. By contrast, OurNet fails to identify the puncture hole defects. It also mistakes the brushed hole and thin spinning defects for others and jumps defects, respectively.Figure 9
Comparisons between the teacher network and OurNet in detecting specific defects.Figure10 shows the location results between the proposed teacher network and the improved YOLOv3 proposed by Jing et al. [24] on the Tianchi AI database. Types of specific defects are labeled under each subfigure for a clearer view. In the subfigure, the green box represents the real defect area, the red box is the positioning result of the proposed teacher network, and the yellow box is the positioning result of the improved YOLOv3. Figure 10 shows that the defect regions predicted by the proposed method are more accurate than those predicted by the improved YOLOv3. Such superiority may be a benefit from the strong YOLOv5 and our improvements. The improved YOLOv3 suffers from positioning small defect areas, although it could detect most defects. For example, it fails to detect the hanging warp and jump defects.Figure 10
Comparisons between the teacher network and the improved YOLOv3 in positioning defect areas.Figure11 compares the teacher and student networks on self-collected fabric images, specifically, their performance in positioning defect areas. In each subfigure, the green box represents the real defect area, the red box is the positioning result of the teacher network, and the yellow box is the positioning result of the student network. The teacher network could more accurately identify the defect areas. The defect detection performance of the student network is slightly weaker than that of the teacher network. However, the student network has lower space-time overheads; thus, it is more suitable to be arranged for embedded systems.Figure 11
Comparisons between the teacher and student networks on self-collected fabric images.


## 4.4. Quantitative Analysis Results

An ablation study is performed on the Tianchi AI database to verify the effects of different improvement methods, including multitask learning, focal loss, and central loss constraints. The results are presented in Table1. The ablation study of the teacher network shows that the student network has similar results.Table 1
Ablation study of the teacher network on the Tianchi AI database.
Attention moduleMultitask learningFocal lossCentral loss constraintsAUCmAP0.9380.403√0.9570.412√√0.9650.431√√√0.9710.441√√√0.9730.442√√√√0.9810.447Table1 shows that the teacher network is degraded into traditional YOLOv5 when none of the improvements is used. Compared with the YOLOv5-based detection method, the introduced attention module could lead to an improved performance with increased AUC and mAP. Then, AUC and mAP are further improved by simultaneously detecting ubiquitous and specific defects with the proposed multitask learning strategy because of the complementarity between different tasks. Based on the multitask learning strategy, the introduction of the focal loss function and central loss constraint could further improve the defect detection results. Simultaneously using all improvements achieves the best performance on the Tianchi AI database, which verifies the effects of different improvement methods.A quantitative comparison between the teacher and student networks is presented in Table2. The identification times are tested on an Nvidia JETSON TX2. The table shows that the student network could still meet the needs of fabric defect detection, despite the performance degradation observed compared with the teacher network. More importantly, the identification time of the student network is approximately half of the teacher network. Its identification time guarantees the real-time performance on embedded devices.Table 2
Quantitative comparisons between the teacher and student networks on the Tianchi AI database.
AUCmAPIdentification times (ms)YOLOv50.9570.41232Teacher network0.9810.44735Student network0.9520.40616Finally, comparisons with other mainstream methods are performed to verify the effectiveness of the proposed method. The improved YOLOv3 [24] and the pretrained deep CNN [40] are selected as the fabric defect detection algorithms. Faster-RCNN [34] and YOLOv5 [29] are selected as the universal object detection methods. The comparison results are presented in Table 3.Table 3
Comparisons of different fabric defect detection algorithms on the Tianchi AI database.
AUCmAPOurNet [41]0.7870.104OurNet-VGG160.8480.288OurNet-ResNet0.8820.311Improved YOLOv3 [24]0.9270.372Jing et al. [40]0.9320.382YOLOv5 [29]0.9570.412Faster-RCNN [34]0.9560.413Student network0.9520.406Teacher network0.9810.447The above table shows that the original OurNet based on AlexNet has poor defect detection performance because it fails to handle small defect areas well. Two variants of OurNet, namely, OurNet-VGG16 and OurNet-ResNet, obtain better performance benefit from extracting better features with deeper structures. Jing et al. [24] achieves better defect detection performance using improved YOLOv3 networks. A pretrained CNN is also beneficial in boosting the defect detection performance as proposed by Jing et al. [40]. YOLOv5 and faster-RCNN achieve similar defect detection performance benefiting from their strong power in object detection. Both methods are superior to the student network proposed in this work, but the time overhead is relatively large. The proposed teacher network achieves the best fabric defect detection performance, whereas the student network provides an alternative to detect fabric defects with acceptable accuracy on embedded devices.Table4 presents the comparisons between different methods on the TILDA database. OurNet [41] and its variants perform much better than on the Tianchi AI database because the TILDA database contains fewer categories and equal samples per category. Improved YOLOv3 [24] proposed by Jing et al. [40] achieve similar performance due to the reason discussed above. Similar to the comparisons on the Tianchi AI database, two state-of-the-art detectors, YOLOv5 [29] and faster-RCNN [34], obtained higher AUC and mAP compared with that of the proposed student network. The proposed teacher network still achieves the best defect detection performance, which verifies the accuracy of the proposed method.Table 4
Comparisons of different fabric defect detection algorithms on the TILDA database.
AUCmAPOurNet [41]0.8660.301OurNet-VGG160.9120.346OurNet-ResNet0.9260.382Jing et al. [40]0.9580.411YOLOv5 [29]0.9700.442Faster-RCNN [34]0.9720.443Student network0.9650.428Teacher network0.9880.451


## 5. Discussion and Conclusion

An automatic fabric defect detection method based on YOLOv5 is proposed because of the considerable role of fabric defect detection in the textile industry. A teacher-student architecture is used in considering the real-time requirements of the fabric defect detection. The deep teacher network could precisely detect specific fabric defects. After knowledge distillation, the shallow student network could perform fabric defects in real time with an acceptable accuracy. A multitask learning strategy is introduced to detect ubiquitous and specific defects simultaneously, and better utilize the complementarity between different tasks. Focal loss and center loss constraints are introduced for better defect detection performance. Evaluations are performed on the public databases and self-collected fabric images. Comparisons with other mainstream methods indicate that the proposed method is applicable to the automatic detection task of textile defects, which can greatly improve the accuracy and efficiency of defect detection and enhance the automation level of the textile industry.



---

*Source: 7321394-2021-09-30.xml*