# Surface Defect Detection Method Based on Improved Semisupervised Multitask Generative Adversarial Network

Authors: Lin Zhu; Du Baolin; Zhao Xiaomeng; Fang Shaoliang; Che Zhen; Zhou Junjie; Chen Shumin
Journal: Scientific Programming
 (2022)

---

## Abstract

The detection methods based on deep learning networks have attracted widespread interest in industrial manufacture. However, the existing methods are mainly trapped by a large amount of training data with excellent labels and also show difficulty for the simultaneous detection of multiple defects in practical detection. Therefore, in this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the training data are manually labeled according to the types of defects, and the generative adversarial network (GAN) is constructed according to the reliable annotations about defects. Thus, a classification decision surface for the detection of multitype defects is formed in the discriminative network of GAN in an integrated manner. Moreover, the semisupervised samples generated by the discriminative network give the generative network feedback for enhancing the image features and avoiding gradient disappearance or overfitting. Finally, the experimental results show that the proposed method can generate high-quality image features compared with the classic GAN. Furthermore, this increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively.


---

## Body

## 1. Introduction

With the development of artificial intelligence and precision machining techniques, the quality requirements of industrial products are also increasing. However, machining defects are inevitable in industrial manufacture. Since machined surface defects may have a disadvantageous effect on the properties and quality of industrial products, the defect detection and analysis of the industrial products are regarded as an important process for controlling the quality and increasing the commercial value. Recently, machine vision technology has been used extensively to test the production quality, such as defect detection [1], industrial diagnostics [2], and other fields [3]. At the same time, to increase the accuracy of defect detection, academic researchers and the industry have developed many effective detection methods and conducted many research activities, such as deep neural network (DNN) [4, 5], logistic regression (LR) [6], deep belief network (DBN) [7], and other methods [8]. Among these approaches, with the advancement and research of deep learning techniques, the defect detection methods based on DNN have become popular, which automatically learn image features from a large scale of training data through the multilayer neural networks [9, 10]. Therefore, the methods have automatic feature extraction and powerful learning capabilities and have been successful in various fields of power electronics [11], microelectronics [6], and other fields [12]. The feature maps of different layers and deep CNN were used by Chen et al. [13] to classify the fastener defects of the support devices, and the public dataset with excellent labels was used in the experiments. Mei et al. [14] proposed a method to localize and detect defect-free data for model training, and the performance of model was evaluated on the KTH-TIPS2 datasets. He et al. [15] proposed an effective defect detection method based on DNN and focused on industrial manufacture: steel plate defect inspection. The performance of DNN was evaluated on the defect datasets: NEU-DET and NEU-CLS, and each class of the defect datasets has 300 images. The defect detection methods based on DNN have been extensively used in industrial application. However, the methods have their own disadvantages. The DNNs, which perform entirely on benchmark datasets, may perform poorly in practical detection [16, 17]. Generally, the advantageous performance of these detection methods based on DNN depends on a large amount of training data with excellent labels [18]. However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. Most prior works have been limited by small training sample sizes, which impact the power to detect effects and the reproducibility of results.For the problem of small samples, many academic researchers use the image enhancement method to expand the training samples. Three image enhancement approaches based on the characteristics of the synthetic image were designed by Ding et al. [19] to expand the training dataset. These methods could increase the performance of CNN in defect detection. However, this method can only expand the single image, and the quality of the generated samples is poor, and the training dataset expansion effect is not ideal. As a solution, the generative adversarial network (GAN) can quickly reconstruct or generate images with various patterns or enhance the quality of the training dataset, which has displayed hopeful performance in generating sample images [20, 21]. At present, the GAN has been extensively used in different fields, such as image restoration [22] and speech enhancement [23]. A deep convolutional GAN (DCGAN) was proposed by Radford et al. [24], which normalizes the input in batches, and it has achieved better results in image generation. Although GAN has gained popularity, it still has some fatal deficiencies. The GAN has some problems that the network structure is unstable, and training GAN itself is difficult. To solve the problem, several methods [25, 26] were proposed. An efficient detection approach based on improved GAN (iGAN) was developed by Wang et al. [27] for machined surfaces. The iGAN is similar to an extension of the classic GAN. On this basis, many methods have produced satisfying efficacies in semisupervised image classification by improving the structure of the DCGAN model. An improved conditional DCGAN model was proposed by Yang et al. [28] which could effectively improve the accuracy of image recognition, but this method was considered to be commercially time-consuming. The improved GAN has a more stable architecture than the classic DCGAN by applying some constraints on GAN. Thus, it is necessary to optimize the constraints of the DCGAN. The information maximizing generative adversarial net (InfoGAN) was designed by Chen et al. [29] to learn entangled representations in an unsupervised method. An adversarial learning approach was developed by Hu et al. [30] to increase the accuracy of semantic segmentation by connecting the cross-entropy loss with the adversarial loss of the proposed method. For the problem that the existing methods mostly depend on weakly labeled images, Zhang et al. [20] proposed a semisupervised GAN (SSGAN) with two subnetworks to expand the training dataset, and the proposed method could improve semantic segmentation accuracy with fewer annotations. The performance of the above methods is enhanced to a certain extent compared with CNN. The GAN has been widely used in many related detection tasks and requires significant computational resources in software implementations and a large amount of data for training. Because training GAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. In other words, these techniques pursue prediction accuracy at the expense of computational efficiency. Moreover, the discriminator of the GAN is mainly dichotomous classification, which is challenging to apply on the detection of multitype defects. However, the application of the multitask GANs, exceptionally for surface detection using small samples, is still rare, which deserves more attention.In practical detection, the existing methods are mainly trapped by a large amount of training data with excellent labels and also show difficulty for the simultaneous detection of multiple defects. If the number of the training datasets is not sufficiently large and the training datasets are imbalanced, training the defect detection method from scratch is likely prone to overfitting in the detection of multitype defects. Therefore, in this article, a surface defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed to reduce overfitting caused by the lack of training samples. The primary contributions of this paper are summarized as follows:(1)
The training dataset is manually labeled according to the type of the defect, and a classification decision surface for the detection of multitype defects is formed in the adversity-discriminator link of iSSMT-GAN in an integrated manner. The discriminative network inputs both real and fake samples and distinguishes whether its input is real or not. Its output result is used as the basis for the classification decision surface for the detection of multitype defects.(2)
The structure of iSSMT-GAN is improved, and the multitask noise is introduced in the generative network of GAN. Moreover, the semisupervised samples generated by the discriminative network give the generative network feedback to guide the generative network, which is an efficient way to avoid overfitting. More specifically, in the discriminator of iSSMT-GAN, the probability of judging the authenticity of the multitask sample output by the discriminator is returned to the generator, and the resulting latent code from the discriminator can be fed back to guide the generative model.This article is organized in the following manner. Section2 indicates the problem formulation and motivation and introduces the network structure of GAN. Section 3 presents the improved semisupervised multitask generative adversarial network (iSSMT-GAN), and the semisupervised samples generated by the discriminative network give the generative network feedback. The experiment results are reported to verify the efficacy of the proposed methodology in Section 4. Finally, Section 5 gives the conclusions and future work.


## 2. Problem Formulation and Motivation

### 2.1. Problem Formulation

In practical manufacture, the industrial products usually include multitype defects, such as bump defects, hole defects, burr defects, oxidation defects, and so on. Collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and it leads to a series of difficulties in the defect detection process due to imbalanced datasets. The main problems of surface defect detection are summarized as follows:(1)
In practical detection, the detection methods based on DNN are prone to overfitting and weak generalization ability during training due to the unbalanced training dataset and the small number of samples. It further leads to low accuracy and recall in actual product testing [6, 10, 11].(2)
At this stage, the discriminator of the GAN is mainly dichotomous classification, which is challenging to apply on the detection of multitype defects. However, the detection of multitype defects is a significant topic in actual industrial applications.(3)
Although many experts and scholars have adopted GAN for image defect detection at this stage, the problem of gradient disappearance or overfitting of the GAN is straightforward to occur during the learning process of data.In this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the iSSMT-GAN is used to predict the potential distribution of the training dataset and reconstruct the image of multitype defects. By integrating the advantages of GAN, it can solve the problems, such as small sample size, not noticeable features, and weak generalization ability in practical detection. From the perspective of sample feature quality and multitask classification recognition, the classification accuracy and recall are improved.


### 2.2. Generative Adversarial Network

The GAN is a framework for estimating generative models through an adversarial process [31]. The GAN framework contains two separate networks, the generative network G and the discriminative network D. The basic framework of GAN is shown in Figure 1.Figure 1
The basic framework of GAN.In Figure1, the generative network G is trained to generate the fake samples from the latent variable z. In GAN, the generating network can quickly learn the target data distribution. The objective function of the generative network is shown in the following equation [31]:(1)minGmaxDEx∼PrlnDx+Ex∼Pgln1−Dx,where E· refers to the calculation of the expected values, Dx is the output of the discriminative network, and Pr and Pg refer to the distribution of real and generated sample data, respectively.The discriminative network inputs both real data and generated data and distinguishes whether its input is real or not. The objective function of the discriminative network is shown in the following formula:(2)maxDEx∼PrlnDx+Ex∼Pgln1−Dx.Therefore, the optimization problem of GAN is a minimization-maximization problem. The discriminative network and the generative network play the two-player mini-max game with the value functionVD,G [19]:(3)VD,G=Ex∼pdataxlnDx+Ez∼pzzln1−DGz,where x represents the actual sample data used for training, pdatax is the distribution of samples, Dx is the output result of the discriminative network, z is the noise of the input, pzz is the distribution of known noise z, and DGz represents the probability that the generated data are differentiated to be an actual sample after passing through the discriminator.


## 2.1. Problem Formulation

In practical manufacture, the industrial products usually include multitype defects, such as bump defects, hole defects, burr defects, oxidation defects, and so on. Collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and it leads to a series of difficulties in the defect detection process due to imbalanced datasets. The main problems of surface defect detection are summarized as follows:(1)
In practical detection, the detection methods based on DNN are prone to overfitting and weak generalization ability during training due to the unbalanced training dataset and the small number of samples. It further leads to low accuracy and recall in actual product testing [6, 10, 11].(2)
At this stage, the discriminator of the GAN is mainly dichotomous classification, which is challenging to apply on the detection of multitype defects. However, the detection of multitype defects is a significant topic in actual industrial applications.(3)
Although many experts and scholars have adopted GAN for image defect detection at this stage, the problem of gradient disappearance or overfitting of the GAN is straightforward to occur during the learning process of data.In this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the iSSMT-GAN is used to predict the potential distribution of the training dataset and reconstruct the image of multitype defects. By integrating the advantages of GAN, it can solve the problems, such as small sample size, not noticeable features, and weak generalization ability in practical detection. From the perspective of sample feature quality and multitask classification recognition, the classification accuracy and recall are improved.


## 2.2. Generative Adversarial Network

The GAN is a framework for estimating generative models through an adversarial process [31]. The GAN framework contains two separate networks, the generative network G and the discriminative network D. The basic framework of GAN is shown in Figure 1.Figure 1
The basic framework of GAN.In Figure1, the generative network G is trained to generate the fake samples from the latent variable z. In GAN, the generating network can quickly learn the target data distribution. The objective function of the generative network is shown in the following equation [31]:(1)minGmaxDEx∼PrlnDx+Ex∼Pgln1−Dx,where E· refers to the calculation of the expected values, Dx is the output of the discriminative network, and Pr and Pg refer to the distribution of real and generated sample data, respectively.The discriminative network inputs both real data and generated data and distinguishes whether its input is real or not. The objective function of the discriminative network is shown in the following formula:(2)maxDEx∼PrlnDx+Ex∼Pgln1−Dx.Therefore, the optimization problem of GAN is a minimization-maximization problem. The discriminative network and the generative network play the two-player mini-max game with the value functionVD,G [19]:(3)VD,G=Ex∼pdataxlnDx+Ez∼pzzln1−DGz,where x represents the actual sample data used for training, pdatax is the distribution of samples, Dx is the output result of the discriminative network, z is the noise of the input, pzz is the distribution of known noise z, and DGz represents the probability that the generated data are differentiated to be an actual sample after passing through the discriminator.


## 3. Improved Semisupervised Multitask Generative Adversarial Network

For generating better image features and improving the classification accuracy, an improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed to reduce overfitting caused by the lack of training samples. In the iSSMT-GAN, the multitask noise is introduced in the generator of GAN according to the reliable annotations about multitype defects. In generating multitask samples, a low-rank space is formed, and a classification decision surface is formed in the adversarial discriminator of GAN. At the same time, the resulting latent code from the discriminative network can be fed back to guide the generative model. The framework of the improved semisupervised multitask generative adversarial network (iSSMT-GAN) is illustrated in Figure2.Figure 2
The structure of the iSSMT-GAN.The framework of iSSMT-GAN contains four parts: the generative network, the discriminative network, the semisupervised samples generated by the discriminative network giving the generative network feedback, and the classification decision surface for the detection of multitype defects.(1)
The generative networkG is constrained with MSE, and the prediction of total variation loss and image features extracted by CNN, which forces the generated data to be more realistic [24] and the batch normalization layer can be used to solve the problems of gradient disappearance and gradient explosion.(2)
The discriminative networkD inputs both real and fake data and distinguishes whether its input is real or not, and the output result is used as the basis for the classification decision surface for the detection of multitype defects. At the same time, the semisupervised samples generated by the discriminative network give the generative network feedback.(3)
The classification decision surface for the detection of multitype defects performs multitask classification tasks based on the scene-prejudged defect labels and the output results of the discriminative network. Thus, a classification decision surface for the detection of multitype defects is formed in the adversity-discriminator link of iSSMT-GAN in an integrated manner.The difference between the proposed iSSMT-GAN and the classic GAN is summarized as follows:(1)
In the discriminative network of iSSMT-GAN, a classification decision surface for the detection of multitype defects is formed by labeling sample defect images and supervising the discriminator.(2)
According to the annotation result and the results of the discriminative network, the semisupervised samples generated by the discriminative network give the generative network feedback to guide the generative network, which is an efficient way to avoid overfitting, so that many more characteristic sample images are generated through the generation network.


### 3.1. Multitask Discriminative Network

In the iSSMT-GAN, a multitask discriminative network is built by constructing a classification decision surface for the detection of multitype defects according to reliable annotations about multitype defects. Firstly, the discriminator makes a preliminary judgment based on the manual labels of defects. Then, the GAN performs dichotomous classification tasks in different defects. Finally, a classification decision surface for detecting multitype defects is formed in an integrated manner. The schematic diagram for the multitask discriminative network is shown in Figure3.Figure 3
The schematic diagram for multitask discriminative network.At the same time, the multitasking noisew is treated as an input variable to the generative network. In the training process, the GAN can generate samples closer to the real samples distribution from the given multitasking noise w. The multitasking noise w is introduced in the detection task facing multiple scene modes, and a low-rank space shared by multiple tasks is formed in generating multitask samples.The discriminative network is constructed using a convolution layer and fully connected layers, and LeakyReLU is selected as each convolution layer’s activation function. Each convolutional block consists of three major components, including a convolutional layer, a LeakyReLU activation layer, and a batch normalization layer [32].The loss function is an essential part of the GAN, and it can directly determine the performance of the network model. According to the structure of GAN, the total loss value can be calculated as the sum of the counter loss and content loss. Its application is shown in the following formula:(4)L=LGAN+λLX,where LGAN is the counter loss, LX is the content loss, and λ is a constant.To optimize the parameters of our proposed model, the loss function using mean square error (MSE) can be defined as follows [23]:(5)Lossp=1m∑i=1mPdataxi−Pzxi2,where Pdatax is the real image, Pzx is the generated image, and m is the number of training datasets.The optimization of the model is achieved by minimizing the confrontation loss and the residual feedback of MSE. The loss function of GAN is shown as follows:(6)Loss=LossG+LossP.


### 3.2. The Semisupervised Sample Feedback

The structure of iSSMT-GAN uses a simple factored continuous input noise vectorz. Therefore, the noise vector z may be used by the generative network in an extremely involved mode, causing various scales of the noise vector z to be discordant to the semantic characteristics of the image [24].In this article, the input noise vectors are divided into two parts instead of using a single incompressible noise vector: (i)z represents the source of unstructured noise; (ii) c represents the latent code. Define the set of methodic latent variables by c1,c2,…,cL. Then, a factored distribution can be assumed, given by the following formula [24]:(7)Pc1,c2,…,cL=∏i=1LPci.For the concise and clear notation, the latent codec will be used to represent the connection of all latent variables ci.An unsupervised way is proposed to discover these latent factors. Then, the input to the generative network is the latent codec and the incompressible noise, and the form of generative network becomes Gz,c. However, the generative network can neglect the attached latent code c by searching a solution that meets the following condition:(8)PGxc=PGx.Moreover, in order to generate or reconstruct the images that are more suitable for defect detection, the semisupervised samplesc generated by the discriminative network are fed back to the generative network. Therefore, the defect detection is performed after high-resolution images are generated, and the accuracy of detection is improved by improving the structure of GAN model. The structure of the semisupervised sample feedback is shown in Figure 4.Figure 4
The structure of the semisupervised sample feedback.In iSSMT-GAN, the structure of the semisupervised sample feedback is formed according to the discriminant results. In the discriminator of iSSMT-GAN, the probability of judging the authenticity of the multitask sample output by the discriminator is returned to the generator and the semisupervised samples generated by the discriminative network give the generative network feedback. In the feedback process, the classification result after the image label is carried corresponds to the signal input to the generator after scene prediction, which has a semisupervised effect.


## 3.1. Multitask Discriminative Network

In the iSSMT-GAN, a multitask discriminative network is built by constructing a classification decision surface for the detection of multitype defects according to reliable annotations about multitype defects. Firstly, the discriminator makes a preliminary judgment based on the manual labels of defects. Then, the GAN performs dichotomous classification tasks in different defects. Finally, a classification decision surface for detecting multitype defects is formed in an integrated manner. The schematic diagram for the multitask discriminative network is shown in Figure3.Figure 3
The schematic diagram for multitask discriminative network.At the same time, the multitasking noisew is treated as an input variable to the generative network. In the training process, the GAN can generate samples closer to the real samples distribution from the given multitasking noise w. The multitasking noise w is introduced in the detection task facing multiple scene modes, and a low-rank space shared by multiple tasks is formed in generating multitask samples.The discriminative network is constructed using a convolution layer and fully connected layers, and LeakyReLU is selected as each convolution layer’s activation function. Each convolutional block consists of three major components, including a convolutional layer, a LeakyReLU activation layer, and a batch normalization layer [32].The loss function is an essential part of the GAN, and it can directly determine the performance of the network model. According to the structure of GAN, the total loss value can be calculated as the sum of the counter loss and content loss. Its application is shown in the following formula:(4)L=LGAN+λLX,where LGAN is the counter loss, LX is the content loss, and λ is a constant.To optimize the parameters of our proposed model, the loss function using mean square error (MSE) can be defined as follows [23]:(5)Lossp=1m∑i=1mPdataxi−Pzxi2,where Pdatax is the real image, Pzx is the generated image, and m is the number of training datasets.The optimization of the model is achieved by minimizing the confrontation loss and the residual feedback of MSE. The loss function of GAN is shown as follows:(6)Loss=LossG+LossP.


## 3.2. The Semisupervised Sample Feedback

The structure of iSSMT-GAN uses a simple factored continuous input noise vectorz. Therefore, the noise vector z may be used by the generative network in an extremely involved mode, causing various scales of the noise vector z to be discordant to the semantic characteristics of the image [24].In this article, the input noise vectors are divided into two parts instead of using a single incompressible noise vector: (i)z represents the source of unstructured noise; (ii) c represents the latent code. Define the set of methodic latent variables by c1,c2,…,cL. Then, a factored distribution can be assumed, given by the following formula [24]:(7)Pc1,c2,…,cL=∏i=1LPci.For the concise and clear notation, the latent codec will be used to represent the connection of all latent variables ci.An unsupervised way is proposed to discover these latent factors. Then, the input to the generative network is the latent codec and the incompressible noise, and the form of generative network becomes Gz,c. However, the generative network can neglect the attached latent code c by searching a solution that meets the following condition:(8)PGxc=PGx.Moreover, in order to generate or reconstruct the images that are more suitable for defect detection, the semisupervised samplesc generated by the discriminative network are fed back to the generative network. Therefore, the defect detection is performed after high-resolution images are generated, and the accuracy of detection is improved by improving the structure of GAN model. The structure of the semisupervised sample feedback is shown in Figure 4.Figure 4
The structure of the semisupervised sample feedback.In iSSMT-GAN, the structure of the semisupervised sample feedback is formed according to the discriminant results. In the discriminator of iSSMT-GAN, the probability of judging the authenticity of the multitask sample output by the discriminator is returned to the generator and the semisupervised samples generated by the discriminative network give the generative network feedback. In the feedback process, the classification result after the image label is carried corresponds to the signal input to the generator after scene prediction, which has a semisupervised effect.


## 4. Experiments and Discussion

In this section, the experiments are divided into two parts: Section4.1. presents the experimental methods and Section 4.2. presents the experimental results.


### 4.1. Experimental Methods

#### 4.1.1. Dataset

To verify the efficacy of the proposed method, comparative evaluation experiments using the NEU-DET dataset and the PCB dataset (self-built dataset) were conducted, and the performance of the proposed iSSMT-GAN is further tested on the NEU-DET dataset and PCB dataset. NEU surface defect is a public defect classification dataset [16], including six different classes from hot-rolled steel plates, such as inclusion, patches, crazing, rolled-in scales, scratches, and pitted surface. Each class has 300 images. Examples of defect images are shown in Figure 5.Figure 5
Examples of defect images with annotations in NEU. (a) Inclusion. (b) Patches. (c) Crazing. (d) Pitted surface. (e) Rolled-in scale. (f) Scratches.
(a)(b)(c)(d)(e)(f)However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. The highly imbalanced datasets are common in many pattern recognition tasks. The severe class imbalance of training datasets is still a significant point that needs to be addressed. The PCB dataset is collected by AOI detection equipment from real detection scenarios of a company. The PCB dataset (self-built dataset) contains 3569 images with a size of 161 × 161. There are six types of defects, including open circuit defects, bump defects, hole defects, residual copper defects, burr defects, and oxidation defects. We manually label 3569 images at first. The dataset contains 2,923 training images and 646 test images of 6 classes. Table1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. The size of all the images was adjusted to 96  ×  96. Figure 6 shows the different types of defects for the dataset.Table 1
The distribution of sample sizes in the PCB dataset.
CategoryTraining dataTest dataOriginal datasetThe dataset expanded by DCGANThe dataset expanded by iSSMT-GANOriginal datasetDCGAN expandiSSMT-GAN expandOpen circuit defects14030030040100100Bump defects1105300300196100100Hole defects1115300300204100100Residual copper defects23630030097100100Burr defects12730030047100100Oxidation defects20030030062100100Total data292318001800646600600Figure 6
Different types of defects for dataset. (a) Open circuit defects. (b) Bump defects. (c) Hole defects. (d) Residual copper defects. (e) Burr defects. (f) Oxidation defects.
(a)(b)(c)(d)(e)(f)


#### 4.1.2. Experimental Environment

The data are employed to verify the proposed method. The experimental environment is described as follows: deep learning open source framework PyTorch [6]. The experiments are performed on the computer with Intel Core i7 processor at 3.4 GHz with 8GB memory.


#### 4.1.3. Evaluation Indicators

In this experiment, model discrimination was assessed using machine learning evaluation metrics, including precision,F1 score, accuracy, and recall. Among them, accuracy is used to evaluate the method's ability to accurately find samples. The recall is used to evaluate the method's ability to find a certain type of sample in the dataset. These indexes are defined as follows:(9)precision=TPTP+FP,recall=TPTP+FN,F1=2⋅precision⋅recallprecision+recall,accuracy=TN+TPTN+FN+TP+FP,where TN, FN, TP, and FP represent the number of true negatives, false negatives, true positives, and false positives, respectively.


### 4.2. Experiment Result

Comparative evaluation experiments were conducted using PCB dataset. Table1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. In response to the problems, the proposed iSSMT-GAN and the traditional DCGAN were used to expand the size of the training dataset and generate a balanced training dataset. Finally, the defect detection approaches based on deep learning were applied for the expanded datasets in the experiments.


#### 4.2.1. Training of the Proposed iSSMT-GAN

To verify the efficacy of the proposed iSSMT-GAN to generate images, the classic DCGAN [24] and the proposed iSSMT-GAN are trained on each category of training data for 2000 epochs, respectively. The number of image channels was set to 3. In the training process, the number of training epochs was 2000, the optimizer used in this experiment was Adam, the initial learning rate was 0.002, and a batch size of 25 was used.The iSSMT-GAN and the DCGAN were used for enhancing dataset from the real sample images of patches of the NEU-DET dataset, respectively. Figure7 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. Figure 8 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs.Figure 7
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 8
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)The iSSMT-GAN and the DCGAN were used for enhancing the dataset from the real sample images of hole defects of PCB dataset, respectively. Figure9 shows the real sample images of hole defects in the dataset. Figure 10 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. Figure 11 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs.Figure 9
The real sample image of hole defects.Figure 10
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 11
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figures10 and 11 show that since the generative network and the discriminative network are continually changing during training, the generated image will change. The quality of samples suggests that the quality of training data impacts the generated images performance of the iSSMT-GAN and DCGAN methods. Figures 10(a), 10(b), 11(a), and 11(b) show that the image features generated by proposed iSSMT-GAN are more apparent and more obvious than the images generated by DCGAN using the NEU-DET dataset and the PCB dataset, respectively. From Figures 10(a) and 11(a), when the number of training epochs is 1000, there will be a gradient explosion phenomenon. Increasing the number of images in training data can eliminate overfitting, strengthen generalizability, and improve the performance of the training dataset.In contrast, the generated images using iSSMT-GAN disclose much-detailed high-resolution features, and the image quality is uniform with the training data. It has been proven that selecting the appropriate training images in the iSSMT-GAN can efficiently improve the dataset quality. The iSSMT-GAN approach requires a long training time, and real-time capability was influenced by the computational resources. Figure12 shows the loss results of the generator and the discriminator on the NEU-DET dataset. Figure 13 shows the loss results of the generator and the discriminator on the PCB dataset.Figure 12
The loss results of the generator and the discriminator on the NEU-DET dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)Figure 13
The loss results of the generator and the discriminator on the PCB dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)The proposed method is also compared with that based on DCGAN [24] in the experiments. Since 2018, the DCGAN has been successful in addressing the problem of limited labeled training samples in the CIFAR-10 dataset and ImageNet-1k dataset. From Figures 12(a) and 13(a), the experiment results indicate that the DCGAN model presents a phenomenon of overfitting during training. Because training DCGAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. From Figures 12(a) and 13(a), the experiment results show that the training DCGAN model suffers from a certain degree of overfitting, and the experiment results are not unexpected. On the one hand, this may be due to the small size of the training dataset. On the other hand, the training dataset contains more complex data than the CIFAR-10 dataset. In the comparative experiment, the proposed iSSMT-GAN is trained using the same dataset. The proposed method displays better results under the same conditions.Figures12 and 13 show the loss changes of the generator and discriminator during training. From Figures 12(a) and 13(a), the training loss values of the generator and discriminator oscillate more and more violently. Because the discriminator and the generator are in a dynamic game, the training process of iSSMT-GAN is commonly unsteady. From Figures 12(b) and 13(b), it is clear that the generator and the discriminator improved via the adversarial process. The training loss values of the generator and discriminator converge well, despite the fluctuating phenomenon in the process of training. At the same time, the generator learns to reconstruct or generate more realistic images, which become pyramidally difficult for the discriminative network to discriminate as not real, as observed from the downward trend in the loss values of the discriminative network and upward trend in the loss values of the generative network curves in early epochs. In the iSSMT-GAN, the loss value decreased significantly during the training period of 0–500 epochs and then decreased steadily. The loss value of the discriminator closer to 0 indicates that the discriminator can discriminate the real image and the generated image.In practice, we believe that the improved discriminative network can improve the performance of the generative network to some extent.


#### 4.2.2. Defect Detection Experiment

In the experiment, the PCB dataset was separated into two parts, including the training dataset and the test dataset. At the same time, the proposed iSSMT-GAN and DCGAN are used to increase the size of the dataset. The distribution of sample sizes in the dataset is shown in Table1. In contrast, the generated image method can change the distribution of the dataset.To verify the efficacy of two different training datasets expanded by the proposed iSSMT-GAN and the classic DCGAN [24], respectively, the defect detection models, such as AlexNet-based transfer learning [9], the VGG-19 model [10], the MobileNet v3 model [12], and the RegNet model, were trained with the two different datasets. From Table 1, the expanded dataset has the problem of the imbalance problem in the training data. To avoid overfitting, we also tested the detection model on the imbalanced datasets by 10-fold cross-validation. The number of training epochs was set as 300, the batch size was 64, and the learning rate was 0.002.The training accuracy and loss value of AlexNet-based transfer learning training are shown in Figure14. The training accuracy and loss value of VGG-19 model training are shown in Figure 15. The testing performance of each model is shown in Table 2.Figure 14
The training accuracy and loss value of AlexNet-based transfer learning training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Figure 15
The training accuracy and loss value of RegNet model training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Table 2
The testing performance of each model.
ModelIn the dataset expanded by DCGANIn the dataset expanded by iSSMT-GANAlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)AlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)Accuracy95.3190.6293.0190.6298.4393.1095.3193.75F1 score95.8384.3497.8791.4498.1793.4094.7292.66Recall rate95.1083.1698.4291.5897.7793.7195.8292.50Precision97.0888.0693.0191.5898.7594.0497.7293.98From Figures14 and 15 and Table 2, the accuracy increases with epoch number and eventually reaches the maximum (98.43%, 93.10%, 95.31%, and 93.75%) in the training process of AlexNet-based transfer learning, the VGG-19 model, the MobileNet v3 model, and the RegNet model, respectively. From Table 2, the experiment results show that the classification accuracy of AlexNet-based transfer learning method is 98.43%, in the training dataset expanded by iSSMT-GAN. The classification accuracy of AlexNet-based transfer learning method is 95.31%, in the training dataset expanded by DCGAN. Figures 14(a) and 14(b) show that the training loss values of the AlexNet-based transfer learning training converge well, and the loss value at the beginning of the model training is tremendous, but as the training process progresses, the loss value extends to decrease on the whole. On the training dataset, the VGG-19 model begins to converge after about 75 epochs, and the convergence process is stable.This increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. In the dataset expanded by iSSMT-GAN, AlexNet-based transfer learning showed theF1 score of 0.9817, reflecting a superior recall performance and precision. Therefore, the dataset expanded by the proposed iSSMT-GAN turns out to be the best option, considering the performance and accuracy of defect detection.In summary, through the analysis of classification quality and classification cost indicators, the proposed system can not only generate better image feature but also improve the classification accuracy. The proposed iSSMT-GAN model displays better results under the same conditions. In contrast, the value of classification accuracy in VGG-19 model only increases slightly from 90.62% to 93.10%. In general, the developed iSSMT-GAN model is efficient for generating better image features compared with DCGAN and improving the classification accuracy using the same dataset.


## 4.1. Experimental Methods

### 4.1.1. Dataset

To verify the efficacy of the proposed method, comparative evaluation experiments using the NEU-DET dataset and the PCB dataset (self-built dataset) were conducted, and the performance of the proposed iSSMT-GAN is further tested on the NEU-DET dataset and PCB dataset. NEU surface defect is a public defect classification dataset [16], including six different classes from hot-rolled steel plates, such as inclusion, patches, crazing, rolled-in scales, scratches, and pitted surface. Each class has 300 images. Examples of defect images are shown in Figure 5.Figure 5
Examples of defect images with annotations in NEU. (a) Inclusion. (b) Patches. (c) Crazing. (d) Pitted surface. (e) Rolled-in scale. (f) Scratches.
(a)(b)(c)(d)(e)(f)However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. The highly imbalanced datasets are common in many pattern recognition tasks. The severe class imbalance of training datasets is still a significant point that needs to be addressed. The PCB dataset is collected by AOI detection equipment from real detection scenarios of a company. The PCB dataset (self-built dataset) contains 3569 images with a size of 161 × 161. There are six types of defects, including open circuit defects, bump defects, hole defects, residual copper defects, burr defects, and oxidation defects. We manually label 3569 images at first. The dataset contains 2,923 training images and 646 test images of 6 classes. Table1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. The size of all the images was adjusted to 96  ×  96. Figure 6 shows the different types of defects for the dataset.Table 1
The distribution of sample sizes in the PCB dataset.
CategoryTraining dataTest dataOriginal datasetThe dataset expanded by DCGANThe dataset expanded by iSSMT-GANOriginal datasetDCGAN expandiSSMT-GAN expandOpen circuit defects14030030040100100Bump defects1105300300196100100Hole defects1115300300204100100Residual copper defects23630030097100100Burr defects12730030047100100Oxidation defects20030030062100100Total data292318001800646600600Figure 6
Different types of defects for dataset. (a) Open circuit defects. (b) Bump defects. (c) Hole defects. (d) Residual copper defects. (e) Burr defects. (f) Oxidation defects.
(a)(b)(c)(d)(e)(f)


### 4.1.2. Experimental Environment

The data are employed to verify the proposed method. The experimental environment is described as follows: deep learning open source framework PyTorch [6]. The experiments are performed on the computer with Intel Core i7 processor at 3.4 GHz with 8GB memory.


### 4.1.3. Evaluation Indicators

In this experiment, model discrimination was assessed using machine learning evaluation metrics, including precision,F1 score, accuracy, and recall. Among them, accuracy is used to evaluate the method's ability to accurately find samples. The recall is used to evaluate the method's ability to find a certain type of sample in the dataset. These indexes are defined as follows:(9)precision=TPTP+FP,recall=TPTP+FN,F1=2⋅precision⋅recallprecision+recall,accuracy=TN+TPTN+FN+TP+FP,where TN, FN, TP, and FP represent the number of true negatives, false negatives, true positives, and false positives, respectively.


## 4.1.1. Dataset

To verify the efficacy of the proposed method, comparative evaluation experiments using the NEU-DET dataset and the PCB dataset (self-built dataset) were conducted, and the performance of the proposed iSSMT-GAN is further tested on the NEU-DET dataset and PCB dataset. NEU surface defect is a public defect classification dataset [16], including six different classes from hot-rolled steel plates, such as inclusion, patches, crazing, rolled-in scales, scratches, and pitted surface. Each class has 300 images. Examples of defect images are shown in Figure 5.Figure 5
Examples of defect images with annotations in NEU. (a) Inclusion. (b) Patches. (c) Crazing. (d) Pitted surface. (e) Rolled-in scale. (f) Scratches.
(a)(b)(c)(d)(e)(f)However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. The highly imbalanced datasets are common in many pattern recognition tasks. The severe class imbalance of training datasets is still a significant point that needs to be addressed. The PCB dataset is collected by AOI detection equipment from real detection scenarios of a company. The PCB dataset (self-built dataset) contains 3569 images with a size of 161 × 161. There are six types of defects, including open circuit defects, bump defects, hole defects, residual copper defects, burr defects, and oxidation defects. We manually label 3569 images at first. The dataset contains 2,923 training images and 646 test images of 6 classes. Table1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. The size of all the images was adjusted to 96  ×  96. Figure 6 shows the different types of defects for the dataset.Table 1
The distribution of sample sizes in the PCB dataset.
CategoryTraining dataTest dataOriginal datasetThe dataset expanded by DCGANThe dataset expanded by iSSMT-GANOriginal datasetDCGAN expandiSSMT-GAN expandOpen circuit defects14030030040100100Bump defects1105300300196100100Hole defects1115300300204100100Residual copper defects23630030097100100Burr defects12730030047100100Oxidation defects20030030062100100Total data292318001800646600600Figure 6
Different types of defects for dataset. (a) Open circuit defects. (b) Bump defects. (c) Hole defects. (d) Residual copper defects. (e) Burr defects. (f) Oxidation defects.
(a)(b)(c)(d)(e)(f)


## 4.1.2. Experimental Environment

The data are employed to verify the proposed method. The experimental environment is described as follows: deep learning open source framework PyTorch [6]. The experiments are performed on the computer with Intel Core i7 processor at 3.4 GHz with 8GB memory.


## 4.1.3. Evaluation Indicators

In this experiment, model discrimination was assessed using machine learning evaluation metrics, including precision,F1 score, accuracy, and recall. Among them, accuracy is used to evaluate the method's ability to accurately find samples. The recall is used to evaluate the method's ability to find a certain type of sample in the dataset. These indexes are defined as follows:(9)precision=TPTP+FP,recall=TPTP+FN,F1=2⋅precision⋅recallprecision+recall,accuracy=TN+TPTN+FN+TP+FP,where TN, FN, TP, and FP represent the number of true negatives, false negatives, true positives, and false positives, respectively.


## 4.2. Experiment Result

Comparative evaluation experiments were conducted using PCB dataset. Table1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. In response to the problems, the proposed iSSMT-GAN and the traditional DCGAN were used to expand the size of the training dataset and generate a balanced training dataset. Finally, the defect detection approaches based on deep learning were applied for the expanded datasets in the experiments.


### 4.2.1. Training of the Proposed iSSMT-GAN

To verify the efficacy of the proposed iSSMT-GAN to generate images, the classic DCGAN [24] and the proposed iSSMT-GAN are trained on each category of training data for 2000 epochs, respectively. The number of image channels was set to 3. In the training process, the number of training epochs was 2000, the optimizer used in this experiment was Adam, the initial learning rate was 0.002, and a batch size of 25 was used.The iSSMT-GAN and the DCGAN were used for enhancing dataset from the real sample images of patches of the NEU-DET dataset, respectively. Figure7 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. Figure 8 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs.Figure 7
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 8
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)The iSSMT-GAN and the DCGAN were used for enhancing the dataset from the real sample images of hole defects of PCB dataset, respectively. Figure9 shows the real sample images of hole defects in the dataset. Figure 10 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. Figure 11 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs.Figure 9
The real sample image of hole defects.Figure 10
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 11
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figures10 and 11 show that since the generative network and the discriminative network are continually changing during training, the generated image will change. The quality of samples suggests that the quality of training data impacts the generated images performance of the iSSMT-GAN and DCGAN methods. Figures 10(a), 10(b), 11(a), and 11(b) show that the image features generated by proposed iSSMT-GAN are more apparent and more obvious than the images generated by DCGAN using the NEU-DET dataset and the PCB dataset, respectively. From Figures 10(a) and 11(a), when the number of training epochs is 1000, there will be a gradient explosion phenomenon. Increasing the number of images in training data can eliminate overfitting, strengthen generalizability, and improve the performance of the training dataset.In contrast, the generated images using iSSMT-GAN disclose much-detailed high-resolution features, and the image quality is uniform with the training data. It has been proven that selecting the appropriate training images in the iSSMT-GAN can efficiently improve the dataset quality. The iSSMT-GAN approach requires a long training time, and real-time capability was influenced by the computational resources. Figure12 shows the loss results of the generator and the discriminator on the NEU-DET dataset. Figure 13 shows the loss results of the generator and the discriminator on the PCB dataset.Figure 12
The loss results of the generator and the discriminator on the NEU-DET dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)Figure 13
The loss results of the generator and the discriminator on the PCB dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)The proposed method is also compared with that based on DCGAN [24] in the experiments. Since 2018, the DCGAN has been successful in addressing the problem of limited labeled training samples in the CIFAR-10 dataset and ImageNet-1k dataset. From Figures 12(a) and 13(a), the experiment results indicate that the DCGAN model presents a phenomenon of overfitting during training. Because training DCGAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. From Figures 12(a) and 13(a), the experiment results show that the training DCGAN model suffers from a certain degree of overfitting, and the experiment results are not unexpected. On the one hand, this may be due to the small size of the training dataset. On the other hand, the training dataset contains more complex data than the CIFAR-10 dataset. In the comparative experiment, the proposed iSSMT-GAN is trained using the same dataset. The proposed method displays better results under the same conditions.Figures12 and 13 show the loss changes of the generator and discriminator during training. From Figures 12(a) and 13(a), the training loss values of the generator and discriminator oscillate more and more violently. Because the discriminator and the generator are in a dynamic game, the training process of iSSMT-GAN is commonly unsteady. From Figures 12(b) and 13(b), it is clear that the generator and the discriminator improved via the adversarial process. The training loss values of the generator and discriminator converge well, despite the fluctuating phenomenon in the process of training. At the same time, the generator learns to reconstruct or generate more realistic images, which become pyramidally difficult for the discriminative network to discriminate as not real, as observed from the downward trend in the loss values of the discriminative network and upward trend in the loss values of the generative network curves in early epochs. In the iSSMT-GAN, the loss value decreased significantly during the training period of 0–500 epochs and then decreased steadily. The loss value of the discriminator closer to 0 indicates that the discriminator can discriminate the real image and the generated image.In practice, we believe that the improved discriminative network can improve the performance of the generative network to some extent.


### 4.2.2. Defect Detection Experiment

In the experiment, the PCB dataset was separated into two parts, including the training dataset and the test dataset. At the same time, the proposed iSSMT-GAN and DCGAN are used to increase the size of the dataset. The distribution of sample sizes in the dataset is shown in Table1. In contrast, the generated image method can change the distribution of the dataset.To verify the efficacy of two different training datasets expanded by the proposed iSSMT-GAN and the classic DCGAN [24], respectively, the defect detection models, such as AlexNet-based transfer learning [9], the VGG-19 model [10], the MobileNet v3 model [12], and the RegNet model, were trained with the two different datasets. From Table 1, the expanded dataset has the problem of the imbalance problem in the training data. To avoid overfitting, we also tested the detection model on the imbalanced datasets by 10-fold cross-validation. The number of training epochs was set as 300, the batch size was 64, and the learning rate was 0.002.The training accuracy and loss value of AlexNet-based transfer learning training are shown in Figure14. The training accuracy and loss value of VGG-19 model training are shown in Figure 15. The testing performance of each model is shown in Table 2.Figure 14
The training accuracy and loss value of AlexNet-based transfer learning training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Figure 15
The training accuracy and loss value of RegNet model training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Table 2
The testing performance of each model.
ModelIn the dataset expanded by DCGANIn the dataset expanded by iSSMT-GANAlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)AlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)Accuracy95.3190.6293.0190.6298.4393.1095.3193.75F1 score95.8384.3497.8791.4498.1793.4094.7292.66Recall rate95.1083.1698.4291.5897.7793.7195.8292.50Precision97.0888.0693.0191.5898.7594.0497.7293.98From Figures14 and 15 and Table 2, the accuracy increases with epoch number and eventually reaches the maximum (98.43%, 93.10%, 95.31%, and 93.75%) in the training process of AlexNet-based transfer learning, the VGG-19 model, the MobileNet v3 model, and the RegNet model, respectively. From Table 2, the experiment results show that the classification accuracy of AlexNet-based transfer learning method is 98.43%, in the training dataset expanded by iSSMT-GAN. The classification accuracy of AlexNet-based transfer learning method is 95.31%, in the training dataset expanded by DCGAN. Figures 14(a) and 14(b) show that the training loss values of the AlexNet-based transfer learning training converge well, and the loss value at the beginning of the model training is tremendous, but as the training process progresses, the loss value extends to decrease on the whole. On the training dataset, the VGG-19 model begins to converge after about 75 epochs, and the convergence process is stable.This increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. In the dataset expanded by iSSMT-GAN, AlexNet-based transfer learning showed theF1 score of 0.9817, reflecting a superior recall performance and precision. Therefore, the dataset expanded by the proposed iSSMT-GAN turns out to be the best option, considering the performance and accuracy of defect detection.In summary, through the analysis of classification quality and classification cost indicators, the proposed system can not only generate better image feature but also improve the classification accuracy. The proposed iSSMT-GAN model displays better results under the same conditions. In contrast, the value of classification accuracy in VGG-19 model only increases slightly from 90.62% to 93.10%. In general, the developed iSSMT-GAN model is efficient for generating better image features compared with DCGAN and improving the classification accuracy using the same dataset.


## 4.2.1. Training of the Proposed iSSMT-GAN

To verify the efficacy of the proposed iSSMT-GAN to generate images, the classic DCGAN [24] and the proposed iSSMT-GAN are trained on each category of training data for 2000 epochs, respectively. The number of image channels was set to 3. In the training process, the number of training epochs was 2000, the optimizer used in this experiment was Adam, the initial learning rate was 0.002, and a batch size of 25 was used.The iSSMT-GAN and the DCGAN were used for enhancing dataset from the real sample images of patches of the NEU-DET dataset, respectively. Figure7 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. Figure 8 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs.Figure 7
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 8
The generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)The iSSMT-GAN and the DCGAN were used for enhancing the dataset from the real sample images of hole defects of PCB dataset, respectively. Figure9 shows the real sample images of hole defects in the dataset. Figure 10 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. Figure 11 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs.Figure 9
The real sample image of hole defects.Figure 10
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figure 11
The generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs. (a) The DCGAN. (b) The proposed iSSMT-GAN.
(a)(b)Figures10 and 11 show that since the generative network and the discriminative network are continually changing during training, the generated image will change. The quality of samples suggests that the quality of training data impacts the generated images performance of the iSSMT-GAN and DCGAN methods. Figures 10(a), 10(b), 11(a), and 11(b) show that the image features generated by proposed iSSMT-GAN are more apparent and more obvious than the images generated by DCGAN using the NEU-DET dataset and the PCB dataset, respectively. From Figures 10(a) and 11(a), when the number of training epochs is 1000, there will be a gradient explosion phenomenon. Increasing the number of images in training data can eliminate overfitting, strengthen generalizability, and improve the performance of the training dataset.In contrast, the generated images using iSSMT-GAN disclose much-detailed high-resolution features, and the image quality is uniform with the training data. It has been proven that selecting the appropriate training images in the iSSMT-GAN can efficiently improve the dataset quality. The iSSMT-GAN approach requires a long training time, and real-time capability was influenced by the computational resources. Figure12 shows the loss results of the generator and the discriminator on the NEU-DET dataset. Figure 13 shows the loss results of the generator and the discriminator on the PCB dataset.Figure 12
The loss results of the generator and the discriminator on the NEU-DET dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)Figure 13
The loss results of the generator and the discriminator on the PCB dataset. (a) The DCGAN [24]. (b) The proposed iSSMT-GAN.
(a)(b)The proposed method is also compared with that based on DCGAN [24] in the experiments. Since 2018, the DCGAN has been successful in addressing the problem of limited labeled training samples in the CIFAR-10 dataset and ImageNet-1k dataset. From Figures 12(a) and 13(a), the experiment results indicate that the DCGAN model presents a phenomenon of overfitting during training. Because training DCGAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. From Figures 12(a) and 13(a), the experiment results show that the training DCGAN model suffers from a certain degree of overfitting, and the experiment results are not unexpected. On the one hand, this may be due to the small size of the training dataset. On the other hand, the training dataset contains more complex data than the CIFAR-10 dataset. In the comparative experiment, the proposed iSSMT-GAN is trained using the same dataset. The proposed method displays better results under the same conditions.Figures12 and 13 show the loss changes of the generator and discriminator during training. From Figures 12(a) and 13(a), the training loss values of the generator and discriminator oscillate more and more violently. Because the discriminator and the generator are in a dynamic game, the training process of iSSMT-GAN is commonly unsteady. From Figures 12(b) and 13(b), it is clear that the generator and the discriminator improved via the adversarial process. The training loss values of the generator and discriminator converge well, despite the fluctuating phenomenon in the process of training. At the same time, the generator learns to reconstruct or generate more realistic images, which become pyramidally difficult for the discriminative network to discriminate as not real, as observed from the downward trend in the loss values of the discriminative network and upward trend in the loss values of the generative network curves in early epochs. In the iSSMT-GAN, the loss value decreased significantly during the training period of 0–500 epochs and then decreased steadily. The loss value of the discriminator closer to 0 indicates that the discriminator can discriminate the real image and the generated image.In practice, we believe that the improved discriminative network can improve the performance of the generative network to some extent.


## 4.2.2. Defect Detection Experiment

In the experiment, the PCB dataset was separated into two parts, including the training dataset and the test dataset. At the same time, the proposed iSSMT-GAN and DCGAN are used to increase the size of the dataset. The distribution of sample sizes in the dataset is shown in Table1. In contrast, the generated image method can change the distribution of the dataset.To verify the efficacy of two different training datasets expanded by the proposed iSSMT-GAN and the classic DCGAN [24], respectively, the defect detection models, such as AlexNet-based transfer learning [9], the VGG-19 model [10], the MobileNet v3 model [12], and the RegNet model, were trained with the two different datasets. From Table 1, the expanded dataset has the problem of the imbalance problem in the training data. To avoid overfitting, we also tested the detection model on the imbalanced datasets by 10-fold cross-validation. The number of training epochs was set as 300, the batch size was 64, and the learning rate was 0.002.The training accuracy and loss value of AlexNet-based transfer learning training are shown in Figure14. The training accuracy and loss value of VGG-19 model training are shown in Figure 15. The testing performance of each model is shown in Table 2.Figure 14
The training accuracy and loss value of AlexNet-based transfer learning training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Figure 15
The training accuracy and loss value of RegNet model training. (a) In the dataset expanded by DCGAN [24]. (b) In the dataset expanded by iSSMT-GAN.
(a)(b)Table 2
The testing performance of each model.
ModelIn the dataset expanded by DCGANIn the dataset expanded by iSSMT-GANAlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)AlexNet model (%)VGG-19 model (%)MobileNet v3 model (%)RegNet model (%)Accuracy95.3190.6293.0190.6298.4393.1095.3193.75F1 score95.8384.3497.8791.4498.1793.4094.7292.66Recall rate95.1083.1698.4291.5897.7793.7195.8292.50Precision97.0888.0693.0191.5898.7594.0497.7293.98From Figures14 and 15 and Table 2, the accuracy increases with epoch number and eventually reaches the maximum (98.43%, 93.10%, 95.31%, and 93.75%) in the training process of AlexNet-based transfer learning, the VGG-19 model, the MobileNet v3 model, and the RegNet model, respectively. From Table 2, the experiment results show that the classification accuracy of AlexNet-based transfer learning method is 98.43%, in the training dataset expanded by iSSMT-GAN. The classification accuracy of AlexNet-based transfer learning method is 95.31%, in the training dataset expanded by DCGAN. Figures 14(a) and 14(b) show that the training loss values of the AlexNet-based transfer learning training converge well, and the loss value at the beginning of the model training is tremendous, but as the training process progresses, the loss value extends to decrease on the whole. On the training dataset, the VGG-19 model begins to converge after about 75 epochs, and the convergence process is stable.This increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. In the dataset expanded by iSSMT-GAN, AlexNet-based transfer learning showed theF1 score of 0.9817, reflecting a superior recall performance and precision. Therefore, the dataset expanded by the proposed iSSMT-GAN turns out to be the best option, considering the performance and accuracy of defect detection.In summary, through the analysis of classification quality and classification cost indicators, the proposed system can not only generate better image feature but also improve the classification accuracy. The proposed iSSMT-GAN model displays better results under the same conditions. In contrast, the value of classification accuracy in VGG-19 model only increases slightly from 90.62% to 93.10%. In general, the developed iSSMT-GAN model is efficient for generating better image features compared with DCGAN and improving the classification accuracy using the same dataset.


## 5. Conclusions

In this article, a defect detection method based on an improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed. The training data are classified and manually labeled according to the type of defects, and the iSSMT-GAN is constructed according to reliable annotations about defects. Thus, a classification decision surface for the detection of multitype defects is formed in the discriminative network of GAN. The applicability and feasibility of the proposed iSSMT-GAN model are verified on the PCB dataset (self-built dataset) to expand the size of the training dataset, and this increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. Experiment results are presented to verify the proposed method. This research mainly focuses on the defect detection on the PCB surface. It is noted that micron-level defects are widespread in industrial products. Our future research will focus on reconstructing or generating micron-level image, so that the test approach adapts to the classification of micron-level defects.



---

*Source: 4481495-2022-01-19.xml*