# Railway Fastener Fault Diagnosis Based on Generative Adversarial Network and Residual Network Model

Authors: Dechen Yao; Qiang Sun; Jianwei Yang; Hengchang Liu; Jiao Zhang
Journal: Shock and Vibration
 (2020)

---

## Abstract

The present work aimed at the problems of less negative samples and more positive samples in rail fastener fault diagnosis and low detection accuracy of heavy manual patrol inspection tasks. Exploiting the capacity of a Convolution Neural Network (CNN) to process unbalanced data to solve tedious and inefficient manual processing, a fault diagnosis method based on a Generative Adversarial Network (GAN) and a Residual Network (ResNet) was developed. First, GAN was used to track the distribution of rail fastener failure data. To study the noise distribution, the mapping relationship between image data was established. Additional real fault samples were then generated to balance and extend the existing data sets, and these data sets were used as input to ResNet for recognition and detection training. Finally, the average accuracy of multiple experiments was used as the evaluation index. The experimental results revealed that the fault diagnosis of rail fastener based on GAN and ResNet could improve the fault detection accuracy in the case of a serious shortage of fault data.


---

## Body

## 1. Introduction

With the rapid development of the economy, rail transit has become the primary travel medium for people. However, substantial traffic and transportation pressure are the two main challenges to the safety of rail transit [1–3]. Many advanced fault detection methods have been adopted for trains, and the safety of tracks should not be ignored [4–6]. As fasteners bear the change of alternating stresses in a long-term reversion process, they are prone to fracture [7, 8]. At present, the inspection of track fasteners mainly relies on manual inspection, and the state of fasteners is judged by walking along the track. This method is inefficient and expensive and relies on the experience of the workers. With the rapid development of image processing, computer vision, machine learning, and neural networks [9, 10], advanced technologies are considered to reduce the time and cost and improve detection accuracy [11, 12].Liu et al. [13] proposed a method based on image fusion features and Bayesian compression image classification recognition to detect the state of fasteners by extracting IEOH and MSLBP features. Mao et al. [14] obtained a precise and extremely dense point cloud of fasteners through structural light sensors and proposed a complex cylindrical surface center extraction method based on the vanilla vector to extract the centerline of ordinary fastener metal clips. Finally, the looseness of the fasteners was evaluated according to the extracted metal clip centerline. Gilbert et al. [15] improved the robustness of fastener detection by adjusting the training data, mining the difficult data, and analyzing the features extracted from these edges with a support vector machine (SVM) classifier [16].Wei et al. [17] proposed a new method for defect detection of fasteners based on dense sieve features. Visual geometry group 16 (VGG16) [18] was used for feature extraction of fasteners, using a faster r-CNN fastener for defect detection, to improve the detection rate and efficiency. Liu et al. [19] proposed a new method to detect partially worn and intact missing fasteners.In recent years, the problem of data imbalance has been prevalent in face recognition, satellite image detection, medical diagnostic decision-making, and other fields [20]. Class imbalance can impair the predictive ability of the classification algorithm because the algorithm pursues the overall classification accuracy. Therefore, increasing attention is paid to the problem of data imbalance.In the traditional approach, the target cost function is modified by weighting small-category data and large-category data samples differently and assigning higher weights to small-category data samples, and thus imposing more severe penalties for the misclassification of small-category data samples. Sampling methods include undersampling and oversampling, the reduction of the size of sample categories by deleting samples from some large data categories, and the increase of samples of subclasses. Random undersampling and random oversampling are the main methods of undersampling and oversampling.The existing intelligent fault diagnosis methods work on the premise of obtaining balanced training data, which indicates that the same number of marker samples should be used for training under different experimental conditions. Suh et al. [21] reduced the level of data imbalance through oversampling and improved the accuracy. Hao and Liu [22] showed that the proposed the K-means synthetic minority oversampling technique (SMOTE) method is then used for oversampling the minority class samples.Generative Adversarial Network (GAN) does well in some specific fields, such as human–computer interaction, computer vision, natural language processing, etc. [23, 24]. Bissoto et al. [25] employed pix2pixHD GAN for image synthesis. They synthesized from semantic maps and instance maps, rather than from random noise. However, for the intelligent detection of track fasteners, it is impossible to obtain a large amount of fault data in a short time to balance data sets for network training. Learning from unbalanced data of different categories is a long-standing problem, and the distribution of training data between different object classes becomes severely skewed [26]. In the case of unbalanced training data, most of the existing learning algorithms generate inductive bias to a big data category, leading to poor recognition performance and low accuracy of small data classes [27]. Figure 1 expresses that unbalanced data cause errors in data classification and explains the process of using manually generated data to compensate for unbalanced classification of classes. With the development of the convolutional neural network (CNN) [28] technology, great success has been achieved in the field of image recognition and detection; however, these algorithms are not designed for rail transit. The present work proposes a fault diagnosis method for track fasteners based on deep learning. The method consisted of two parts: (1) the generative adversarial network (GAN) was used to learn the distribution of real data samples, and additional fault samples were generated to balance and train the data sets; (2) residual network (ResNet) was used for track fastener fault diagnosis and classification, and the extended data set was used for group training and validation.Figure 1
Classification of balanced data using synthetic data. (a) Balanced data, (b) unbalanced data, and (c) composite data.
(a)(b)(c)The main contributions of the present paper are as follows:(1)
The proposed method used GAN and ResNet models for rail fasteners in unbalanced data(2)
The good performance of GAN in solving unbalanced data was applied to rail fasteners(3)
The excellent performance of ResNet in image classification was combined with the data generation capability of GAN to expand the amount of data(4)
It provides an idea for intelligent detection of failure data that cannot be obtained by experiment or within a short timeThe rest of the paper is organized as follows. An introduction to theoretical fundamentals is provided in Section2; Section 3 presents the experimental data; Section 4 completes the experimental verification of data and methods; and Section 5 concludes the paper.


## 2. Theoretical Background

Deep learning models manifest a relatively advanced performance in different tasks using high-dimensional balanced data sets, and these models can optimize millions of parameters through tagged data training [29]. Due to a large number of model parameters, it is easy to overfit small data and thus make the generalization performance proportional to the category proportion of marked data. Some traditional methods, such as variable target cost function, sampling methods, and manual data generation, are available to deal with class imbalance.


### 2.1. Generative Adversarial Network

Most of the existing sampling methods do not consider the underlying distribution of real data and result in incorrect samples. Artificial neural networks (ANNs) have a good application prospect in data generation [30, 31]. In 2014, Goodfellow et al. from the University of Montreal first proposed ANNs. GAN is used to solve the problem of data imbalance as well as to generate mechanical data samples [32–34]. Khan et al. [35] used GAN for the damage detection and life prediction of bearings. Wang et al. [36] also applied GAN for the fault detection of a planetary gearbox with unbalanced data.GAN generally consists of two modules: the generating network (G) and discriminant network (D), which are parameterized deep neural networks. Figure 2 presents a structural diagram of the GAN algorithm. Input is the input of G, and it is generally the noise data generated by the Gaussian random distribution. The output of G is G(Z), and for real data, it is usually a picture. For the output of D, the distribution variable is expressed by X, and it is possible to judge the possibility of X, where G(Z) represents the noise distribution.Figure 2
Structural diagram of GAN.The goal of GAN is to use random noise learning to generate the distributed input of a network on real data. The identification network is updated through resistance training to accurately identify true and false data, optimize the generation network, and generate real samples that cannot be recognized by the identifier. The principle of GAN can be summarized as follows:(1)minGmaxDG,D=EXpdataXlogDX+EZPZZLOG1−DGZ,where Pdata is the true data distribution. By generating a confrontation training between the network G and the discriminant network D, the noise distribution GZ can be projected to a position similar to Pdata, thus generating more real negative sample data. For a specific sample, it may come from a true distribution or it may come from a generated distribution, so its contribution to the discriminator’s loss function is(2)−PrxlogDX−Pgxlog1−Dx,where pr is the true distribution and pg is the generated distribution. If the derivative with respect to D(x) is zero, the global optimal solution of D(x) can be obtained as follows:(3)D∗x=prxprx+pgx.The optimization function of the generator can be obtained by substituting the optimal discriminator into(4)Ex∼prlogprxprx+pgx+Ex∼pglogpgxprx+pgx.The expressions of JS divergence and KL divergence are as follows:(5)KLp1p2=Ex∼p1logp1xp2x,JSp1p2=12KLp1p1+p22+12KLp2p1+p22.Clearly, equation (5) is similar to the JS divergence form, which can be converted into(6)Ex∼prlogprx1/2prx+pgx+Ex∼pglogpgx1/2prx+pgx−2log2=JSprpg−2log2.In conclusion, it can be considered that when the discriminator is too optimal, the loss of the generator can be approximately equivalent to the JS divergence of the real sample distributionpr and the generated sample distribution pg by the generator.


### 2.2. Residual Learning Module

The basic network of deep learning is from AlexNet and VGG to GoogLeNet. A deeper network structure can extract more complex features; however, the deeper the network, the lower the accuracy. A network model can solve the problems of network gradient disappearance or explosion and precision degradation, and it is easier to optimize the model.ResNet network is a reference to VGG19 network, based on this modification, and through the short circuit of the residual unit. In the change of ResNet, the stride-2 of convolution is directly used as sampling, and the global average pool is used to replace the whole connection layer. An important design principle of ResNet is to maintain the complexity of the network layer by doubling the number of maps when the feature size of the feature map is reduced. ResNet adds a short circuit mechanism between every two layers compared with ordinary networks, which results in residual learning. Table1 shows the different depths of ResNet. In this article, we select Resnet50 network for rail fastener detection.Table 1
Different depths of ResNet.
Layer nameOutput size18-layer34-layer50-layer101-layer152-layerConv1112×1127×7, 64, stride23×3 max pool, stride2Conv2_x56×563×3,643×3,64×23×3,643×3,64×31×1,643×3,641×1,256×31×1,643×3,641×1,256×31×1,643×3,641×1,256×3Conv3_x28×283×3,1283×3,128×23×3,1283×3,128×41×1,1283×3,1281×1,512×41×1,1283×3,1281×1,512×41×1,1283×3,1281×1,512×8Conv4_x14×143×3,2563×3,256×23×3,2563×3,256×61×1,2563×3,2561×1,1024×61×1,2563×3,2561×1,1024×231×1,2563×3,2561×1,1024×36Conv5_x7×73×3,2563×3,256×23×3,5123×3,512×31×1,5123×3,5121×1,2048×31×1,5123×3,5121×1,2048×31×1,5123×3,5121×1,2048×31×1Average pool, 1000-d fc, softmaxFLOPs1.8×1093.6×1093.8×1097.6×10911.3×109ResNet introduces residual learning withX as the input and HX as the output. A convolution neural network learns HX directly through training, whereas residual learning uses several network layers with parameters to learn the residual FX=HX−X between input and output, and the final output of the residual unit is FX+X [37, 38].Residual learning module, shown in Figure3, is defined as follows:(7)y=Fx,Wi+x,where x and y are the input and output vectors of the network layer, and the function Fx,Wi represents the residual mapping to be learned.Figure 3
Residual learning module.From Figure4, the function Fx,Wi can represent multiple convolutional layers. To simplify the notation, ReLU activation function and deviation are omitted. The F(x) + X operation is realized by fast connection and element-by-element addition on a channel-by-channel basis. Assuming that the two-layer neural network is activated in the L layer to obtain xl+1, it can be simply written as follows:(8)xl+1=xl+Fxl,Wl.Figure 4
Diagnostic flowchart for track fasteners.By continuously stacking the number of network layers, the expression of the characteristics of any deep unitL can be obtained as follows:(9)xL=xl+∑i=lL−1Fxi,Wi.Assuming thatε is a loss function, the following equation can be obtained from the chain rule of backpropagation:(10)∂ε∂xL=∂ε∂xL∂xL∂xl=∂ε∂xL1+∂∂xl∑i=lL−1Fxi,wi.This derivative can be divided into two parts:∂ε/∂XL not passing through the weight layer and ∂ε/∂xL∂/∂xl∑i=lL−1Fxi,wi passing through the weight layer. The former provides a guarantee for the signal to be transmitted directly to the shallow layer and avoids the phenomenon of gradient disappearance because ∂/∂xl∑i=lL−1Fxi,wi cannot be −1.


## 2.1. Generative Adversarial Network

Most of the existing sampling methods do not consider the underlying distribution of real data and result in incorrect samples. Artificial neural networks (ANNs) have a good application prospect in data generation [30, 31]. In 2014, Goodfellow et al. from the University of Montreal first proposed ANNs. GAN is used to solve the problem of data imbalance as well as to generate mechanical data samples [32–34]. Khan et al. [35] used GAN for the damage detection and life prediction of bearings. Wang et al. [36] also applied GAN for the fault detection of a planetary gearbox with unbalanced data.GAN generally consists of two modules: the generating network (G) and discriminant network (D), which are parameterized deep neural networks. Figure 2 presents a structural diagram of the GAN algorithm. Input is the input of G, and it is generally the noise data generated by the Gaussian random distribution. The output of G is G(Z), and for real data, it is usually a picture. For the output of D, the distribution variable is expressed by X, and it is possible to judge the possibility of X, where G(Z) represents the noise distribution.Figure 2
Structural diagram of GAN.The goal of GAN is to use random noise learning to generate the distributed input of a network on real data. The identification network is updated through resistance training to accurately identify true and false data, optimize the generation network, and generate real samples that cannot be recognized by the identifier. The principle of GAN can be summarized as follows:(1)minGmaxDG,D=EXpdataXlogDX+EZPZZLOG1−DGZ,where Pdata is the true data distribution. By generating a confrontation training between the network G and the discriminant network D, the noise distribution GZ can be projected to a position similar to Pdata, thus generating more real negative sample data. For a specific sample, it may come from a true distribution or it may come from a generated distribution, so its contribution to the discriminator’s loss function is(2)−PrxlogDX−Pgxlog1−Dx,where pr is the true distribution and pg is the generated distribution. If the derivative with respect to D(x) is zero, the global optimal solution of D(x) can be obtained as follows:(3)D∗x=prxprx+pgx.The optimization function of the generator can be obtained by substituting the optimal discriminator into(4)Ex∼prlogprxprx+pgx+Ex∼pglogpgxprx+pgx.The expressions of JS divergence and KL divergence are as follows:(5)KLp1p2=Ex∼p1logp1xp2x,JSp1p2=12KLp1p1+p22+12KLp2p1+p22.Clearly, equation (5) is similar to the JS divergence form, which can be converted into(6)Ex∼prlogprx1/2prx+pgx+Ex∼pglogpgx1/2prx+pgx−2log2=JSprpg−2log2.In conclusion, it can be considered that when the discriminator is too optimal, the loss of the generator can be approximately equivalent to the JS divergence of the real sample distributionpr and the generated sample distribution pg by the generator.


## 2.2. Residual Learning Module

The basic network of deep learning is from AlexNet and VGG to GoogLeNet. A deeper network structure can extract more complex features; however, the deeper the network, the lower the accuracy. A network model can solve the problems of network gradient disappearance or explosion and precision degradation, and it is easier to optimize the model.ResNet network is a reference to VGG19 network, based on this modification, and through the short circuit of the residual unit. In the change of ResNet, the stride-2 of convolution is directly used as sampling, and the global average pool is used to replace the whole connection layer. An important design principle of ResNet is to maintain the complexity of the network layer by doubling the number of maps when the feature size of the feature map is reduced. ResNet adds a short circuit mechanism between every two layers compared with ordinary networks, which results in residual learning. Table1 shows the different depths of ResNet. In this article, we select Resnet50 network for rail fastener detection.Table 1
Different depths of ResNet.
Layer nameOutput size18-layer34-layer50-layer101-layer152-layerConv1112×1127×7, 64, stride23×3 max pool, stride2Conv2_x56×563×3,643×3,64×23×3,643×3,64×31×1,643×3,641×1,256×31×1,643×3,641×1,256×31×1,643×3,641×1,256×3Conv3_x28×283×3,1283×3,128×23×3,1283×3,128×41×1,1283×3,1281×1,512×41×1,1283×3,1281×1,512×41×1,1283×3,1281×1,512×8Conv4_x14×143×3,2563×3,256×23×3,2563×3,256×61×1,2563×3,2561×1,1024×61×1,2563×3,2561×1,1024×231×1,2563×3,2561×1,1024×36Conv5_x7×73×3,2563×3,256×23×3,5123×3,512×31×1,5123×3,5121×1,2048×31×1,5123×3,5121×1,2048×31×1,5123×3,5121×1,2048×31×1Average pool, 1000-d fc, softmaxFLOPs1.8×1093.6×1093.8×1097.6×10911.3×109ResNet introduces residual learning withX as the input and HX as the output. A convolution neural network learns HX directly through training, whereas residual learning uses several network layers with parameters to learn the residual FX=HX−X between input and output, and the final output of the residual unit is FX+X [37, 38].Residual learning module, shown in Figure3, is defined as follows:(7)y=Fx,Wi+x,where x and y are the input and output vectors of the network layer, and the function Fx,Wi represents the residual mapping to be learned.Figure 3
Residual learning module.From Figure4, the function Fx,Wi can represent multiple convolutional layers. To simplify the notation, ReLU activation function and deviation are omitted. The F(x) + X operation is realized by fast connection and element-by-element addition on a channel-by-channel basis. Assuming that the two-layer neural network is activated in the L layer to obtain xl+1, it can be simply written as follows:(8)xl+1=xl+Fxl,Wl.Figure 4
Diagnostic flowchart for track fasteners.By continuously stacking the number of network layers, the expression of the characteristics of any deep unitL can be obtained as follows:(9)xL=xl+∑i=lL−1Fxi,Wi.Assuming thatε is a loss function, the following equation can be obtained from the chain rule of backpropagation:(10)∂ε∂xL=∂ε∂xL∂xL∂xl=∂ε∂xL1+∂∂xl∑i=lL−1Fxi,wi.This derivative can be divided into two parts:∂ε/∂XL not passing through the weight layer and ∂ε/∂xL∂/∂xl∑i=lL−1Fxi,wi passing through the weight layer. The former provides a guarantee for the signal to be transmitted directly to the shallow layer and avoids the phenomenon of gradient disappearance because ∂/∂xl∑i=lL−1Fxi,wi cannot be −1.


## 3. Data Preparation

### 3.1. Experimental Flow

By integrating the rail fastener fault diagnosis module into the confrontation training program, it was optimized by real data and generated data. A rail fastener fault diagnosis method was proposed based on GAN and ResNet. The method consisted of two parts. In the first part, GAN was used to learn the distribution of real data samples, and the training data sets were balanced and expanded by generating additional real negative samples. The second part used a ResNet network to train ended data sets during identification and diagnosis. The diagnostic flowchart for rail fasteners is presented in Figure4. Fault class samples were generated, and the training data sets were expanded explicitly to further improve the performance of the fault diagnosis model.


### 3.2. Data Sources

The data of the current experiment consisted of a picture of a rail fastener of a subway operation line, which contained a large number of complete fasteners and a small number of faulty fasteners (the main faults were loss and fracture). Figure5 presents a sample of real data. These fastener images have some characteristics, the texture feature is different from other parts, and the length and width of the fastener do not exceed the length and width of the bottom plate.Figure 5
Samples of true fasteners. (a) Intactness, (b) fracture, and (c) loss.
(a)(b)(c)Figure6 presents the data acquisition equipment. The data acquisition equipment can adapt to track data acquisition of normal gauge by installing a charge coupled device (CCD) linear array camera on the acquisition equipment to collect images. The field of view of the collected images included the rail and fasteners on both sides. The images were preprocessed, the fastener area was extracted, and the feature description of the fastener was established.Figure 6
Data acquisition equipment.


### 3.3. Data Generation

A GAN network was used to learn the distribution of track fastener image data and generate simulated real samples to expand the data set. The fault diagnosis of track fasteners was studied under different states using multiple networks by distributed learning. Multiple learning modules were set up to generate different types of fasteners at the same time. Each module learned the data distribution of each rail fastener individually. Figure7 exhibits the data generation structure and multiple learning modules.Figure 7
Data generation structure and multiple learning modules.The input noise is a standard Gaussian distributionZ. In the generation network, a filter with two input full-connection layers and three convolution layers (filter sizes were 128, 64, and 32) was used. The identification network used two convolution layers with filter sizes of 64 and 32. The ReLU unit activation function was generally used throughout the network, and the Adam optimizer was used in this model. Tables 2 and 3 present the algorithms for generating and identifying networks, respectively.Table 2
Generator network algorithm.
Generating network algorithmInput: Gaussian noiseNOutput: Composite dataG(Z)1 Initialization of relevant parametersSet up the training cycle (e): epochSet up the batch sample size (m): batch size2 Begin3 Fori = 1 in e do4  For batch = 1 in m do5G=ReLUGZW1G+b1G6   Optimize loss functions7   Update weight and bias8  End9 End10 EndTable 3
Discriminator network algorithm.
Discriminator network algorithmInput: Fractured track fastener pictureOutput: Probability that the output data is true or false1 Initialization of relevant parametersSet up the training cycle (e): epochSet up the batch sample size (m): batch_size2 Begin3 Fori = 1 in e do4  For batch = 1 in m do5D=ReLuGZW1D+b1D6   Optimize loss functions7   Update weight and bias8  End9 End10 EndThe generator network consists of 100 input neurons, a hidden layer containing 128 neurons, and an output of 784 = 28 × 28 neurons. The discriminator network is composed of 784 = 28 × 28 input neurons, a hidden layer containing 128 neurons, and 1 output neuron. The output neuron is the value between 0 and 1, which represents the probability that the output is true. Figures8 and 9, respectively, represent the structure of the generator network and the discriminator network. Figure 10 shows the dynamic architecture of GAN.Figure 8
Generator network architecture.Figure 9
Discriminator network architecture.Figure 10
GAN dynamic architecture.During the continuous training of the confrontation neural network, when the two networks reach the Nash equilibrium point, the losses of the generated and discriminator network decreased and gradually approached the minimum value. Figure11 presents the observed G-loss curve of the generated network and the D-loss curve of the distinguishing network. The partial magnification of the loss curves reveals that the two networks gradually reached Nash equilibrium after 3000 steps.Figure 11
Loss curves of the generator and discriminator networks.To improve the model recognition accuracy, GAN was used to generate images of fractured track fasteners with different noises and lost track fasteners in different training cycles. Figure12 displays the sample data generated in different training cycles.Figure 12
Generation of sample data in different training cycles. (a) 1000 epochs, (b) 1500 epochs, (c) 2000 epochs, (d) 2500 epochs, and (e) 3000 epochs.
(a)(b)(c)(d)(e)


## 3.1. Experimental Flow

By integrating the rail fastener fault diagnosis module into the confrontation training program, it was optimized by real data and generated data. A rail fastener fault diagnosis method was proposed based on GAN and ResNet. The method consisted of two parts. In the first part, GAN was used to learn the distribution of real data samples, and the training data sets were balanced and expanded by generating additional real negative samples. The second part used a ResNet network to train ended data sets during identification and diagnosis. The diagnostic flowchart for rail fasteners is presented in Figure4. Fault class samples were generated, and the training data sets were expanded explicitly to further improve the performance of the fault diagnosis model.


## 3.2. Data Sources

The data of the current experiment consisted of a picture of a rail fastener of a subway operation line, which contained a large number of complete fasteners and a small number of faulty fasteners (the main faults were loss and fracture). Figure5 presents a sample of real data. These fastener images have some characteristics, the texture feature is different from other parts, and the length and width of the fastener do not exceed the length and width of the bottom plate.Figure 5
Samples of true fasteners. (a) Intactness, (b) fracture, and (c) loss.
(a)(b)(c)Figure6 presents the data acquisition equipment. The data acquisition equipment can adapt to track data acquisition of normal gauge by installing a charge coupled device (CCD) linear array camera on the acquisition equipment to collect images. The field of view of the collected images included the rail and fasteners on both sides. The images were preprocessed, the fastener area was extracted, and the feature description of the fastener was established.Figure 6
Data acquisition equipment.


## 3.3. Data Generation

A GAN network was used to learn the distribution of track fastener image data and generate simulated real samples to expand the data set. The fault diagnosis of track fasteners was studied under different states using multiple networks by distributed learning. Multiple learning modules were set up to generate different types of fasteners at the same time. Each module learned the data distribution of each rail fastener individually. Figure7 exhibits the data generation structure and multiple learning modules.Figure 7
Data generation structure and multiple learning modules.The input noise is a standard Gaussian distributionZ. In the generation network, a filter with two input full-connection layers and three convolution layers (filter sizes were 128, 64, and 32) was used. The identification network used two convolution layers with filter sizes of 64 and 32. The ReLU unit activation function was generally used throughout the network, and the Adam optimizer was used in this model. Tables 2 and 3 present the algorithms for generating and identifying networks, respectively.Table 2
Generator network algorithm.
Generating network algorithmInput: Gaussian noiseNOutput: Composite dataG(Z)1 Initialization of relevant parametersSet up the training cycle (e): epochSet up the batch sample size (m): batch size2 Begin3 Fori = 1 in e do4  For batch = 1 in m do5G=ReLUGZW1G+b1G6   Optimize loss functions7   Update weight and bias8  End9 End10 EndTable 3
Discriminator network algorithm.
Discriminator network algorithmInput: Fractured track fastener pictureOutput: Probability that the output data is true or false1 Initialization of relevant parametersSet up the training cycle (e): epochSet up the batch sample size (m): batch_size2 Begin3 Fori = 1 in e do4  For batch = 1 in m do5D=ReLuGZW1D+b1D6   Optimize loss functions7   Update weight and bias8  End9 End10 EndThe generator network consists of 100 input neurons, a hidden layer containing 128 neurons, and an output of 784 = 28 × 28 neurons. The discriminator network is composed of 784 = 28 × 28 input neurons, a hidden layer containing 128 neurons, and 1 output neuron. The output neuron is the value between 0 and 1, which represents the probability that the output is true. Figures8 and 9, respectively, represent the structure of the generator network and the discriminator network. Figure 10 shows the dynamic architecture of GAN.Figure 8
Generator network architecture.Figure 9
Discriminator network architecture.Figure 10
GAN dynamic architecture.During the continuous training of the confrontation neural network, when the two networks reach the Nash equilibrium point, the losses of the generated and discriminator network decreased and gradually approached the minimum value. Figure11 presents the observed G-loss curve of the generated network and the D-loss curve of the distinguishing network. The partial magnification of the loss curves reveals that the two networks gradually reached Nash equilibrium after 3000 steps.Figure 11
Loss curves of the generator and discriminator networks.To improve the model recognition accuracy, GAN was used to generate images of fractured track fasteners with different noises and lost track fasteners in different training cycles. Figure12 displays the sample data generated in different training cycles.Figure 12
Generation of sample data in different training cycles. (a) 1000 epochs, (b) 1500 epochs, (c) 2000 epochs, (d) 2500 epochs, and (e) 3000 epochs.
(a)(b)(c)(d)(e)


## 4. Validation of the Method

### 4.1. Building Data Sets

A deep learning model for rail fastener fault diagnosis was established based on the ResNet model. The failure data of rail fasteners generated by the GAN network were combined with the intact rail fastener data to form a large data set. As the output of the CNN model follows a probability distribution, the input label must follow a probability distribution so that the cross-entropies between the model output and the input label can be calculated and the network can be optimized. Aiming at the above problem, all samples in the data set were labeled with the one-hot code, which converted the number into a vector with only one bit. Table4 presents the tags with one-hot code.Table 4
Label coding.
Original labelOne-hot coding labelFastener status0100Intactness1010Loss2001This experimental algorithm was designed and developed based on the open-source deep learning framework of TensorFlow, and all experimental data sets were stored through the unified storage format “TFRecord” provided by TensorFlow.


### 4.2. Experimental Results and Analysis

The current experimental algorithm was developed in the framework of TensorFlow and used an ResNet 18-layer network as the fault diagnosis model. The hardware of the model consisted of two Intel Xeon CPU E5-2678 V3 processors (frequency = 2.5 GHz), the WIN10 64-bit operating system, and two NVIDIA’s TITAN RTX display cards. In each type of data set, 4000 samples were randomly selected to establish a training set of 12,000 samples, and 1000 samples were randomly selected from the remaining data set to form a test set of 3,000 samples. The batch size was set to 100, the number of training iterations was 2000, and the learning rate was 0.001.In this case, fault identification experiments will be conducted for the rail fasteners. Three health conditions are simulated in the experiments, which are intactness, fracture, and loss. To improve the reliability of the obtained results, the experiment was divided into five groups. The training set includes a total of 2400 fault samples in each group, and each health condition has 800 fault samples. The validation set and test set each contain 600 failure samples, and each failure type has 200 failure samples. Finally, the average value was taken as the final accuracy of the GAN + ResNet method. Table5 and Figure 13 present the accuracy of all experimental training and test sets.Table 5
Data composition for experimental verification.
Validation setData sourcesMixed dataA + B + C + D + E + true dataKA + true dataLB + true dataMC + true dataND + true dataOE + true dataFigure 13
Accuracy of training and test sets.Table6 presents the IDs of different epochs of the GAN network training data group. Table 6 presents the data composition for experimental verification, and Table 7 depicts the accuracy of different methods. Figure 14 presents the accuracies of the different methods; LBP represents the local binary mode that can effectively describe textures, and HOG represents the directional gradient histogram, which is a common feature to describe local textures in computer vision and pattern recognition. VGG16 is a classical deep learning model with a good classification ability.Table 6
ID of the data group.
Training cycles1000 epochs1500 epochs2000 epochs2500 epochs3000 epochsID of data setABCDETable 7
Accuracies of different methods.
MethodMixed data (%)K (%)L (%)M (%)N (%)O (%)Average (%)LBP + SVM96.2190.6191.8692.5693.9394.0193.20HOG + SVM97.3292.2493.5593.8995.0295.7894.63VGG1698.8493.9294.4395.4497.8998.9796.58GAN + ResNet99.5695.8797.4198.2999.0399.3498.25Figure 14
Accuracies of the different methods.The same data set was used in LBP + SVM, HOG + SVM, VGG16, and GAN + ResNet. The t-SNE algorithm could reduce the dimensionality, visualize the model forecast on the test set, and display the output multidimensional forecast data on the two-dimensional space. It was found that the GAN + ResNet method could classify the data well, and its classification effect was clearly better than other methods. Figure15 displays the test data distributions for different models.Figure 15
Test data distributions for different modes. (a) LBP + SVM, (b) HOG + SVM, (c) VGG16, and (d) GAN + ResNet.
(a)(b)(c)(d)To further analyze the diagnostic effect of each method on different faults, the classification results of each method were visualized by the confusion matrix. In Figure16, the green squares are the correct number of samples to diagnose and the blue squares are the number of samples to diagnose errors.Figure 16
Visualization of data set classification results. (a) LBP + SVM, (b) HOG + SVM, (c) VGG16, and (d) GAN + ResNet.
(a)(b)(c)(d)The recall rate and classification accuracy of different methods were calculated by the confusion matrix. Tables8 and 9 present the recall rates and classification accuracies of different methods, respectively. It is evident that the proposed method could accurately classify each type of fastener with a general accuracy of over 98%.Table 8
Recall rates of different methods.
t (%)LossIntactnessLBP + SVM96.0093.0091.00HOG + SVM94.5093.0096.50VGG1694.5096.5097.00GAN + ResNet97.5098.0099.50Table 9
Diagnostic accuracies of different methods.
Fracture (%)Loss (%)Intactness (%)LBP + SVM91.4393.4795.29HOG + SVM93.5695.3895.07VGG1697.4194.6195.57GAN + ResNet98.4898.4998.03Through the above experiments, it can be seen that ResNet has the best effect. Classification algorithms are not sensitive to unbalanced data sets; therefore, it is difficult to classify unbalanced data sets. The main processing methods of unbalanced data sets are sampling, which can be divided into undersampling and oversampling.To verify the performance of GAN, fault identification experiments will be carried out on track fasteners. The fracture and loss are grouped together and the two kinds of failure of the coupler are simulated as health and disease, respectively. Among them, there were 200 samples of health and 40 samples of disease, with a ratio of 5:1. The methods of oversampling and undersampling are, respectively, adopted to verify on ResNet. Then, 80 disease samples were generated by GAN, and the proportion of healthy samples and disease samples was 1:1. Input the sample into the ResNet network for training. Finally, the visualization is carried out through a confusion matrix.Figure17 shows the confounding matrix of different data processing methods and Table 10 shows the classification result index. Through comparison, it is found that undersampling is used to reduce the sample number of majority classes which is inevitable to cause information loss problem, thus resulting in the worst effect. Oversampling method is to copy a few samples to increase the number of samples in a few classes. However, an overfitting problem can be emerged since there is no extra new information incorporated. The synthetic images from the GAN model were added to the training set to help train the ResNet for better classification performance.Figure 17
Confounding matrix of different data processing methods. (a) Undersampling + ResNet, (b) oversampling + ResNet, and (c) GAN + ResNet.
(a)(b)(c)Table 10
Classification result index.
Accuracy (%)Precision (%)Recall (%)Undersampling + ResNet81.2580.4982.50Oversampling + ResNet95.0095.4594.50GAN + ResNet98.2598.4998.00


## 4.1. Building Data Sets

A deep learning model for rail fastener fault diagnosis was established based on the ResNet model. The failure data of rail fasteners generated by the GAN network were combined with the intact rail fastener data to form a large data set. As the output of the CNN model follows a probability distribution, the input label must follow a probability distribution so that the cross-entropies between the model output and the input label can be calculated and the network can be optimized. Aiming at the above problem, all samples in the data set were labeled with the one-hot code, which converted the number into a vector with only one bit. Table4 presents the tags with one-hot code.Table 4
Label coding.
Original labelOne-hot coding labelFastener status0100Intactness1010Loss2001This experimental algorithm was designed and developed based on the open-source deep learning framework of TensorFlow, and all experimental data sets were stored through the unified storage format “TFRecord” provided by TensorFlow.


## 4.2. Experimental Results and Analysis

The current experimental algorithm was developed in the framework of TensorFlow and used an ResNet 18-layer network as the fault diagnosis model. The hardware of the model consisted of two Intel Xeon CPU E5-2678 V3 processors (frequency = 2.5 GHz), the WIN10 64-bit operating system, and two NVIDIA’s TITAN RTX display cards. In each type of data set, 4000 samples were randomly selected to establish a training set of 12,000 samples, and 1000 samples were randomly selected from the remaining data set to form a test set of 3,000 samples. The batch size was set to 100, the number of training iterations was 2000, and the learning rate was 0.001.In this case, fault identification experiments will be conducted for the rail fasteners. Three health conditions are simulated in the experiments, which are intactness, fracture, and loss. To improve the reliability of the obtained results, the experiment was divided into five groups. The training set includes a total of 2400 fault samples in each group, and each health condition has 800 fault samples. The validation set and test set each contain 600 failure samples, and each failure type has 200 failure samples. Finally, the average value was taken as the final accuracy of the GAN + ResNet method. Table5 and Figure 13 present the accuracy of all experimental training and test sets.Table 5
Data composition for experimental verification.
Validation setData sourcesMixed dataA + B + C + D + E + true dataKA + true dataLB + true dataMC + true dataND + true dataOE + true dataFigure 13
Accuracy of training and test sets.Table6 presents the IDs of different epochs of the GAN network training data group. Table 6 presents the data composition for experimental verification, and Table 7 depicts the accuracy of different methods. Figure 14 presents the accuracies of the different methods; LBP represents the local binary mode that can effectively describe textures, and HOG represents the directional gradient histogram, which is a common feature to describe local textures in computer vision and pattern recognition. VGG16 is a classical deep learning model with a good classification ability.Table 6
ID of the data group.
Training cycles1000 epochs1500 epochs2000 epochs2500 epochs3000 epochsID of data setABCDETable 7
Accuracies of different methods.
MethodMixed data (%)K (%)L (%)M (%)N (%)O (%)Average (%)LBP + SVM96.2190.6191.8692.5693.9394.0193.20HOG + SVM97.3292.2493.5593.8995.0295.7894.63VGG1698.8493.9294.4395.4497.8998.9796.58GAN + ResNet99.5695.8797.4198.2999.0399.3498.25Figure 14
Accuracies of the different methods.The same data set was used in LBP + SVM, HOG + SVM, VGG16, and GAN + ResNet. The t-SNE algorithm could reduce the dimensionality, visualize the model forecast on the test set, and display the output multidimensional forecast data on the two-dimensional space. It was found that the GAN + ResNet method could classify the data well, and its classification effect was clearly better than other methods. Figure15 displays the test data distributions for different models.Figure 15
Test data distributions for different modes. (a) LBP + SVM, (b) HOG + SVM, (c) VGG16, and (d) GAN + ResNet.
(a)(b)(c)(d)To further analyze the diagnostic effect of each method on different faults, the classification results of each method were visualized by the confusion matrix. In Figure16, the green squares are the correct number of samples to diagnose and the blue squares are the number of samples to diagnose errors.Figure 16
Visualization of data set classification results. (a) LBP + SVM, (b) HOG + SVM, (c) VGG16, and (d) GAN + ResNet.
(a)(b)(c)(d)The recall rate and classification accuracy of different methods were calculated by the confusion matrix. Tables8 and 9 present the recall rates and classification accuracies of different methods, respectively. It is evident that the proposed method could accurately classify each type of fastener with a general accuracy of over 98%.Table 8
Recall rates of different methods.
t (%)LossIntactnessLBP + SVM96.0093.0091.00HOG + SVM94.5093.0096.50VGG1694.5096.5097.00GAN + ResNet97.5098.0099.50Table 9
Diagnostic accuracies of different methods.
Fracture (%)Loss (%)Intactness (%)LBP + SVM91.4393.4795.29HOG + SVM93.5695.3895.07VGG1697.4194.6195.57GAN + ResNet98.4898.4998.03Through the above experiments, it can be seen that ResNet has the best effect. Classification algorithms are not sensitive to unbalanced data sets; therefore, it is difficult to classify unbalanced data sets. The main processing methods of unbalanced data sets are sampling, which can be divided into undersampling and oversampling.To verify the performance of GAN, fault identification experiments will be carried out on track fasteners. The fracture and loss are grouped together and the two kinds of failure of the coupler are simulated as health and disease, respectively. Among them, there were 200 samples of health and 40 samples of disease, with a ratio of 5:1. The methods of oversampling and undersampling are, respectively, adopted to verify on ResNet. Then, 80 disease samples were generated by GAN, and the proportion of healthy samples and disease samples was 1:1. Input the sample into the ResNet network for training. Finally, the visualization is carried out through a confusion matrix.Figure17 shows the confounding matrix of different data processing methods and Table 10 shows the classification result index. Through comparison, it is found that undersampling is used to reduce the sample number of majority classes which is inevitable to cause information loss problem, thus resulting in the worst effect. Oversampling method is to copy a few samples to increase the number of samples in a few classes. However, an overfitting problem can be emerged since there is no extra new information incorporated. The synthetic images from the GAN model were added to the training set to help train the ResNet for better classification performance.Figure 17
Confounding matrix of different data processing methods. (a) Undersampling + ResNet, (b) oversampling + ResNet, and (c) GAN + ResNet.
(a)(b)(c)Table 10
Classification result index.
Accuracy (%)Precision (%)Recall (%)Undersampling + ResNet81.2580.4982.50Oversampling + ResNet95.0095.4594.50GAN + ResNet98.2598.4998.00


## 5. Conclusions

The present paper proposed a rail fastener fault diagnosis method based on deep learning. GAN and ResNet were used to solve the problem of data imbalance. The main observations are as follows:(1)
The mapping relationship between noise distribution and real track fault data was established through the countermeasure training of the generated network and the discriminant network and was used to generate additional negative samples to balance and further expand the training data set.(2)
Based on the experimental verification of faulty and intact fastener samples, it was observed from the t-SNE algorithm that the detection effect of the GAN + ResNet method was more obvious than the other methods.The proposed rail fastener fault diagnosis method based on GAN and ResNet could improve the accuracy of fault detection in the case of insufficient fault data and provide a new idea for fault diagnosis in the case of an unbalanced rail direction data category.



---

*Source: 8823050-2020-11-07.xml*