# Defect Detection of Pandrol Track Fastener Based on Local Depth Feature Fusion Network

Authors: Zhaomin Lv; Anqi Ma; Xingjie Chen; Shubin Zheng
Journal: Complexity
 (2021)

---

## Abstract

There are three main problems in track fastener defect detection based on image: (1) The number of abnormal fastener pictures is scarce, and supervised learning detection model is difficult to establish. (2) The potential data features obtained by different feature extraction methods are different. Some methods focus on edge features, and some methods focus on texture features. Different features have different detection capabilities, and these features are not effectively fused and utilized. (3) The detection of the track fastener clip will be interfered by the track fastener bolt subimage. Aiming at the above three problems, a method for track fastener defects detection based on Local Deep Feature Fusion Network (LDFFN) is proposed. Firstly, the track fastener image segmentation method is used to obtain the track fastener clip subimage, which can effectively reduce the interference of bolt subimage features on the track fastener clip detection. Secondly, the edge features and texture features of track fastener clip subimages are extracted by Autoencoder (AE) and Restricted Boltzmann Machine (RBM), and the features are fused. Finally, the similarity measurement method Mahalanobis Distance (MD) is used to detect defects in track fasteners. The effectiveness of the proposed method is verified by real Pandrol track fastener images.


---

## Body

## 1. Introduction

With the continuous growth of the world economy, railway transportation plays a more and more important role in the economic lifeline. In order to meet the growing demand for economic development, railway construction has also increased year by year. Railway track is composed of several key parts, such as sleeper, rail, and fastener. As the connecting part of fixed track and sleeper, track fastener is an important part to ensure the safety of railway operation. Huge safety risks and even major accidents were caused by the defective track fasteners, such as breaks or missing. Defective track fastener is caused by the contact friction and vibration impact between the hub and the track, as well as the impact of the natural environment [1, 2].At present, the state of track fasteners is mainly detected by manual inspection. Manual inspection method is inefficient, and many uncontrollable risk factors may be caused by manual inspection, such as missed inspection and false inspection. With the rapid development of automation technology [3–6], manual inspection has been difficult to adapt to the rapid development of track transit. The inspection of track traffic needs to be automated. Automating the track fastener detection technology [7, 8] is significant to track operation and maintenance. However, the existing track fastener defect detection methods are limited, and the detection accuracy is low.In recent years, with the rapid development of artificial intelligence technology [9, 10], image processing methods based on deep learning have become a research hotspot [11–13]. Deep learning networks have achieved important research results in image classification [14, 15]. The defect detection method of track fastener based on deep learning avoids complicated feature extraction. The manual design and selection of image features are replaced by automatic extraction of image features by computer. The deep learning network automatically learns the pattern recognition system of features from the data set, and the information in the image can be extracted more comprehensively [16]. Although the feature extraction of deep learning is shown to be far superior to traditional “artificial design features,” the existing track fastener defect detection network based on deep learning is a supervised network, and a large amount of labeled sample data is required for a supervised network. For example, when using convolutional neural networks to detect defects in track fasteners [17], a large number of labeled training sample images are required. However, the actual situation is that the number of abnormal samples of track fastener is rare, which is limited for the establishment of supervised learning model.Because the supervised network is limited by the number of abnormal fastener images, it is difficult to establish, so the unsupervised network is adopted. However, different deep learning networks have different structure and depth. With different deep learning networks, the features of track fastener image are extracted differently, and the detection ability of different features is also different. The feature information of track fastener image extracted by single network is not comprehensive enough, and the feature information of track fastener image cannot be used well. For example, the edge feature of the track fastener image is extracted [18–20]. Edge feature is an important feature of image, and image edge is the discontinuity of image features such as pixel gray distribution. Image edge is the set of pixels whose characteristics around the image change step or ridge. The edge extraction is mainly to retain the sharply changed area of the image, and the contour edge of the track fastener can be obtained. The edge contour feature of the track fastener image is helpful to distinguish the state of track fastener loss and track fastener clip loosening. The texture feature of track fastener is extracted [21–23]. The image texture feature is a global feature, and the visual features of homogeneity in the image are reflected. The arrangement properties of the surface structure of the object surface with slow or periodic changes are reflected. The texture feature can effectively represent the clip detail feature of track fastener image, and the clip detail feature of track fastener image is helpful to identify the fracture state of track fastener clip. These features are not well integrated and utilized at the same time, and the feature information of track fastener image is not fully expressed, which leads to a high rate of missing alarm in track fastener defect detection [24–26].Because, for Pandrol fasteners, the track fastener picture is composed of the track fastener clip subimage and the track fastener bolt subimage, the defect of the track fastener is mainly determined by the state of the track fastener clip subimage. When the deep learning network is used to extract the full image feature, the bolt subimage feature in the image will also be extracted into the discriminant feature, which will interfere with the detection of track fastener defects.Therefore, in view of the above-mentioned problems in the detection of track fastener defects: the number of abnormal samples of track fastener images is sparse; image features are not fully utilized; and track fastener detection is interfered by the feature of track fastener bolt subimage. A method that is based on LDFFN to detect the Pandrol track fastener defects is proposed. Firstly, the Pandrol track fastener picture is segmented by the automatic segmentation algorithm of track fastener pictures. The bolt information in the picture is removed to obtain the track fastener clip subpicture, so as to avoid the interference of the bolt subgraph feature on the track fastener defect detection. Secondly, AE [27, 28] and RBM [29, 30] are used to extract clip subimage features of track fastener, because AE focuses on edge feature for extracting subimage features of track fastener clip [31], while RBM focuses on texture feature [32, 33]. Considering that the edge feature is advantageous for detecting the state of track fastener loss and track fastener clip loosening, the texture feature is advantageous for detecting the broken state of the track fastener clip. Therefore, the subimage features of the track fasteners extracted by AE and RBM are fused to obtain the locally deeply integrated track fastener features [34]. Finally, the similarity measurement method MD [35] is used to detect the status of track fasteners. The image features of track fastener can be fully extracted by unsupervised LDFFN, the interference of bolt diagram of track fastener on defect detection of track fastener can be avoided, and the problem that the number of abnormal images of track fastener is rare can be solved.The second part of this article briefly introduces two deep learning networks, AE and RBM. The third part describes the proposed defect detection method of track fasteners based on LDFFN. The fourth part verifies the effectiveness of the LDFFN-based track fastener defect detection method through experimental comparative analysis. The fifth part summarizes and prospects the full text.


## 2. Neural Network

### 2.1. AE

AE is an unsupervised neural network model that can learn the hidden features of the input data, which is called encoding. At the same time, the original input data can be reconstructed with the learned new features, which is called decoding [36]. From an intuitive point of view, AE can be used for feature dimensionality reduction, similar to principal component analysis but more compared to the performance of principal component analysis, which is due to the fact that AE has stronger nonlinear feature extraction capabilities. The main structure of AE is shown in Figure 1.Figure 1
AE structure diagram.X is represented as the input of AE encodes, H is represented as the new feature, and X′ is represented as the output of AE decodes.The encoding process is as follows:(1)H=fWX+b.The decoding process is as follows:(2)X′=gW′H+d,where f and g are represented as activation functions of the encoding network and the decoding network, respectively; {W, b} and {W′, d} are represented as the connection weights and biases corresponding to the network. Usually, W′=WT, and θ=W,b,d is represented as the parameter of AE. When S=Xji=1N is represented as the sample data set, the overall loss function of AE is(3)JAEθ=∑x∈SLX,gfX,where LX,gfX is represented as the reconstruction error function and LX,gfX is determined by functions f and g. By calculating the minimum value of JAEθ, the parameter θ of the AE neural network can be solved.


### 2.2. RBM

Boltzmann machines are a large class of neural network models, but the most used model in practical applications is RBM. The RBM model itself is very simple, just a two-layer neural network. The main structure of RBM is shown in Figure2.Figure 2
RBM structure diagram.v1,v2,…,vm is represented as the RBM network visible nodes. h1,h2,…,hn is represented as the RBM network hidden nodes. Each visible node is only related to n hidden nodes and is independent of other visible nodes. That is, the state of this visible node is only affected by the influence of nhidden nodes. For each hidden node, it is only affected by m visible nodes. a=a1,a2,…,am is represented as the offset of the visible node. b=b1,b2,…,bn is represented as the offset of the hidden node.


## 2.1. AE

AE is an unsupervised neural network model that can learn the hidden features of the input data, which is called encoding. At the same time, the original input data can be reconstructed with the learned new features, which is called decoding [36]. From an intuitive point of view, AE can be used for feature dimensionality reduction, similar to principal component analysis but more compared to the performance of principal component analysis, which is due to the fact that AE has stronger nonlinear feature extraction capabilities. The main structure of AE is shown in Figure 1.Figure 1
AE structure diagram.X is represented as the input of AE encodes, H is represented as the new feature, and X′ is represented as the output of AE decodes.The encoding process is as follows:(1)H=fWX+b.The decoding process is as follows:(2)X′=gW′H+d,where f and g are represented as activation functions of the encoding network and the decoding network, respectively; {W, b} and {W′, d} are represented as the connection weights and biases corresponding to the network. Usually, W′=WT, and θ=W,b,d is represented as the parameter of AE. When S=Xji=1N is represented as the sample data set, the overall loss function of AE is(3)JAEθ=∑x∈SLX,gfX,where LX,gfX is represented as the reconstruction error function and LX,gfX is determined by functions f and g. By calculating the minimum value of JAEθ, the parameter θ of the AE neural network can be solved.


## 2.2. RBM

Boltzmann machines are a large class of neural network models, but the most used model in practical applications is RBM. The RBM model itself is very simple, just a two-layer neural network. The main structure of RBM is shown in Figure2.Figure 2
RBM structure diagram.v1,v2,…,vm is represented as the RBM network visible nodes. h1,h2,…,hn is represented as the RBM network hidden nodes. Each visible node is only related to n hidden nodes and is independent of other visible nodes. That is, the state of this visible node is only affected by the influence of nhidden nodes. For each hidden node, it is only affected by m visible nodes. a=a1,a2,…,am is represented as the offset of the visible node. b=b1,b2,…,bn is represented as the offset of the hidden node.


## 3. Defect Detection Method of Track Fasteners Based on LDFFN

### 3.1. Automatic Segmentation of Track Fastener Pictures

The Pandrol track fastener image is composed of the track fastener clip subgraph and the track fastener bolt subgraph. Normally, if the fastener bolt is damaged, it will not be reflected in the image, and the damage of the fastener clip will be reflected in the image. Track fastener defect detection will be disturbed by the track fastener bolt which is not damaged. In this paper, in order to reduce the interference of track fastener bolts, the image of track fastener will be automatically segmented.In the first step, the input image is grayscale processed [37]. Because the color image data of track fasteners is relatively large, the calculation is time-consuming. The grading empirical formula is expressed as follows:(4)Grayx,y=0.299∗Redx,y+0.587∗Greenx,y+0.114∗Bluex,y,where Redx,y is represented as pixels in the red channel. Greenx,y is represented as pixels in the green channel. Bluex,y is represented as pixels in the blue channel.In the second step, histogram equalization is performed on the gray image [38]. The contrast and detail of the image are improved.(5)pxi=px=i=nin,0≤i≤L−1,where n is represented as the number of pixels in the image, ni is represented as the number of pixels in each gray level of the original image, L is represented as the number of gray levels of the original image, and pxi is the probability of occurrence of a pixel of gray i in the image. Equation (6) is cumulative normalization.(6)cdfxi=∑j=0ipxj,0≤i≤L−1,(7)gi=round255×cdfxi,0≤i≤L−1,where cdfxi is represented as the cumulative distribution probability, round indicates that the number is rounded, L is represented as the number of gray levels of the original image, 255 represents the maximum gray level after histogram equalization, and gi represents the pixel value of the pixel histogram whose gray level is i in the original image after equalization.In the third step, the gray image is denoised by Gaussian filtering [39]. Image noise can be removed by Gaussian filter. Gaussian filter denoising can keep the edge of the image well, and more of the overall gray distribution characteristics of the image can be preserved.In the fourth step, a sample image is made for the subimage of track fastener clip. Thereafter, grayscale, grayscale histogram equalization, and Gaussian filtering are performed on the sample picture.In the fifth step, find the location of the track fastener clip subimage in the track fastener image. The most similar place with the track fastener clip subgraph sample is the target area. The normalized correlation coefficient [40] is used to locate the best similar region, and the subimage of track fastener clip is segmented.


### 3.2. Track Fastener Defect Detection Based on LDFFN

Firstly, AE and RBM are used to extract the image features of the training set of track fastener clip subgraph. Then the AE extracted features and RBM extracted features are combined in series, and MD is used to detect the track fastener defects.MD is used to measure the distance between a sample pointX and a set of data distribution D. X is represented as a sample point. D is represented as the set of data. u is represented as the mean value of D. v is represented as the covariance of D. Then the MD between the sample and the set is(8)DMX,u=X−uTv−1X−u.The covariance matrix, inverse matrix, and mean vector of training set are calculated, and then MD is calculated. The maximum value of the training set MD is selected as the detection threshold, and the image to be tested is processed in the same way to calculate the MD. If MD is less than or equal to the threshold value, the test image is judged as normal fastener, and if MD is greater than the threshold value, the test image is judged as abnormal fastener.Flow chart of detection of defects in Pandrol track fasteners based on LDFFN is shown in Figure3. The first part is to import the training set of track fastener image. Firstly, the track fastener image is segmented by the automatic segmentation algorithm of track fastener image to obtain the track fastener clip subgraph. Secondly, AE and RBM are used to extract the subimage features of track fastener clip, and then the extracted subimage features of track fastener clip are fused. Finally, MD of fusion features is calculated, MD of training set is selected as threshold of detection image, and feature fusion network model is saved after training. The second part is the test part. Firstly, the test image of track fastener image is imported. Secondly, the training set image is segmented by the automatic segmentation algorithm of track fastener image, and the training set image of track fastener clip subimage is obtained. Finally, through the trained feature fusion network model, the MD of each image to be tested is calculated to determine the status of track fastener image. If the calculated MD is less than or equal to the threshold value, the tested track fastener image is judged as normal track fastener image. If the calculated MD is greater than the threshold value, the tested track fastener image is judged as abnormal track fastener image.Figure 3
Flow chart of detection of defects in Pandrol track fasteners based on LDFFN.


## 3.1. Automatic Segmentation of Track Fastener Pictures

The Pandrol track fastener image is composed of the track fastener clip subgraph and the track fastener bolt subgraph. Normally, if the fastener bolt is damaged, it will not be reflected in the image, and the damage of the fastener clip will be reflected in the image. Track fastener defect detection will be disturbed by the track fastener bolt which is not damaged. In this paper, in order to reduce the interference of track fastener bolts, the image of track fastener will be automatically segmented.In the first step, the input image is grayscale processed [37]. Because the color image data of track fasteners is relatively large, the calculation is time-consuming. The grading empirical formula is expressed as follows:(4)Grayx,y=0.299∗Redx,y+0.587∗Greenx,y+0.114∗Bluex,y,where Redx,y is represented as pixels in the red channel. Greenx,y is represented as pixels in the green channel. Bluex,y is represented as pixels in the blue channel.In the second step, histogram equalization is performed on the gray image [38]. The contrast and detail of the image are improved.(5)pxi=px=i=nin,0≤i≤L−1,where n is represented as the number of pixels in the image, ni is represented as the number of pixels in each gray level of the original image, L is represented as the number of gray levels of the original image, and pxi is the probability of occurrence of a pixel of gray i in the image. Equation (6) is cumulative normalization.(6)cdfxi=∑j=0ipxj,0≤i≤L−1,(7)gi=round255×cdfxi,0≤i≤L−1,where cdfxi is represented as the cumulative distribution probability, round indicates that the number is rounded, L is represented as the number of gray levels of the original image, 255 represents the maximum gray level after histogram equalization, and gi represents the pixel value of the pixel histogram whose gray level is i in the original image after equalization.In the third step, the gray image is denoised by Gaussian filtering [39]. Image noise can be removed by Gaussian filter. Gaussian filter denoising can keep the edge of the image well, and more of the overall gray distribution characteristics of the image can be preserved.In the fourth step, a sample image is made for the subimage of track fastener clip. Thereafter, grayscale, grayscale histogram equalization, and Gaussian filtering are performed on the sample picture.In the fifth step, find the location of the track fastener clip subimage in the track fastener image. The most similar place with the track fastener clip subgraph sample is the target area. The normalized correlation coefficient [40] is used to locate the best similar region, and the subimage of track fastener clip is segmented.


## 3.2. Track Fastener Defect Detection Based on LDFFN

Firstly, AE and RBM are used to extract the image features of the training set of track fastener clip subgraph. Then the AE extracted features and RBM extracted features are combined in series, and MD is used to detect the track fastener defects.MD is used to measure the distance between a sample pointX and a set of data distribution D. X is represented as a sample point. D is represented as the set of data. u is represented as the mean value of D. v is represented as the covariance of D. Then the MD between the sample and the set is(8)DMX,u=X−uTv−1X−u.The covariance matrix, inverse matrix, and mean vector of training set are calculated, and then MD is calculated. The maximum value of the training set MD is selected as the detection threshold, and the image to be tested is processed in the same way to calculate the MD. If MD is less than or equal to the threshold value, the test image is judged as normal fastener, and if MD is greater than the threshold value, the test image is judged as abnormal fastener.Flow chart of detection of defects in Pandrol track fasteners based on LDFFN is shown in Figure3. The first part is to import the training set of track fastener image. Firstly, the track fastener image is segmented by the automatic segmentation algorithm of track fastener image to obtain the track fastener clip subgraph. Secondly, AE and RBM are used to extract the subimage features of track fastener clip, and then the extracted subimage features of track fastener clip are fused. Finally, MD of fusion features is calculated, MD of training set is selected as threshold of detection image, and feature fusion network model is saved after training. The second part is the test part. Firstly, the test image of track fastener image is imported. Secondly, the training set image is segmented by the automatic segmentation algorithm of track fastener image, and the training set image of track fastener clip subimage is obtained. Finally, through the trained feature fusion network model, the MD of each image to be tested is calculated to determine the status of track fastener image. If the calculated MD is less than or equal to the threshold value, the tested track fastener image is judged as normal track fastener image. If the calculated MD is greater than the threshold value, the tested track fastener image is judged as abnormal track fastener image.Figure 3
Flow chart of detection of defects in Pandrol track fasteners based on LDFFN.


## 4. Experiments and Applications

### 4.1. Data Set

The picture of track fasteners in this article comes from the Shijiazhuang-Taiyuan high-speed rail line, called Pandrol fasteners. The state of common Pandrol track fasteners is shown in Figure4, where (a) is the image of normal track fasteners; (b) is the image of broken track fastener; (c) is the image of loose track fastener.Figure 4
Common track fastener status. (a) The image of normal track fasteners. (b) The image of broken track fastener. (c) The image of loose track fastener.
(a)(b)(c)A total of 1064 pictures and 84 pictures of normal and abnormal track fasteners, respectively, on Pandrol are presented. Among them, there are 542 left fasteners for the normal track, 42 left fasteners for the abnormal track, 522 right fasteners for the normal track, and 42 right fasteners for the abnormal track. In this paper, the detection indexes in reference are cited for comparison of detection efficiency.(1)
Segmentation rate (SR): NO is represented as the number of original images, and NS is represented as the number of pictures successfully segmented [41].(9)SR=NSNO×100%.(2)
False alarm rate (FAR), missed alarm rate (MAR), and error rate (ER): DAA indicates that the abnormal track fastener detection result is abnormal; DAN indicates that the abnormal track fastener test result is normal; DNN indicates that the normal track fastener test result is normal; DNA indicates that the normal track fastener test result is abnormal [41].(10)FAR=DNADNN+DNA×100%,MAR=DANDAA+DAN×100%,ER=DNA+DANDNN+DNA+DAA+DAN×100%.


### 4.2. Experimental Process and Results

#### 4.2.1. Reconstruction of Picture Features of Track Fasteners

Feature reconstruction images of track fastener are shown in Figure5. (a) is the reconstruction image of track fastener image extracted by AE, and (b) is the image feature reconstruction map of track fastener extracted by RBM. The edge feature of track fastener contour can be better extracted by AE, and the internal texture feature of track fastener can be better extracted by RBM. Therefore, the image extracted by AE and RBM is fused in this paper. Edge features and texture features are included in the fusion features. The fusion features can fully express the information in the image, and track fastener defects are more conducive to be detected.Figure 5
Image feature reconstruction of track fasteners. (a) Image feature reconstruction extracted by AE. (b) Image feature reconstruction extracted by RBM.
(a)(b)


#### 4.2.2. Defect Detection Results of Pandrol Track Fasteners Based on Deep Feature Fusion Network (DFFN)

Based on the DFFN-based Pandrol track fastener defect detection method, the image features of the track fastener are extracted through AE and RBM, the extracted full-image features are feature-fused, and then the state of the track fastener is detected by MD.There are 542 normal left track fasteners and 42 abnormal left track fasteners. Among them, 300 normal left track fastener pictures are randomly selected to train the network, and the remaining left track fastener pictures are used for detection. There are 522 normal right track fasteners and 42 abnormal right track fasteners. Among them, 300 normal right track fastener pictures are randomly selected to train the network, and the remaining right track fastener pictures are used for detection.For AE extraction of track fastener image features, two coding layers and two decoding layers are designed in AE network. When encoding, the image dimension is reduced to 256 dimensions. Through the first coding layer, the ReLU activation function is used to output 128 dimensions; through the second coding layer, the ReLU activation function is used to output 64 dimensions; and then the output of the second encoding is used as the input of decoding. The 256-dimensional image features are output by two decoding layers. Training AE model, set the training cycle to 100 and the bit size to 100 and save the model. When RBM is used to extract the image features of track fastener, RBM is called directly, and the training period is set to 100 and the bit size is 100. Then, the track fastener image features extracted by AE and the track fastener image features extracted by RBM are fused in serial, and the covariance matrix, inverse matrix of covariance matrix, and mean vector of training set are calculated, and then MD is calculated.The defects detection results of Pandrol track fasteners based on DFFN are shown in Table1. For Pandrol track fastener, the track fastener defect detection method based on AE or RBM extraction track fastener image features has high error rate and missing alarm rate. The false alarm rate and error rate of track fastener defect detection are effectively reduced by DFFN that combines AE and RBM. It shows that the image features of track fasteners extracted by AE and RBM are merged, and the image edge features and texture features are used at the same time, which is more conducive to the detection of track fastener defects. The performance of track fastener defect detection has been improved.Table 1
Defects detection results of Pandrol track fasteners based on DFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerAE2142838311.577.1410.92RBM2123037512.4011.9012.32DFFN2123042012.40010.56Right track fastenerAE1972639411.269.5211.36RBM1942837512.6111.9012.50DFFN1932942013.06010.98


#### 4.2.3. Verification of Feature Interference of Bolt Subgraph of Pandrol Track Fastener

The abnormal pictures detected in DFFN test pictures are displayed, and some pictures are normal. DFFN detects normal pictures as abnormal pictures. In order to find out the reason, the automatic segmentation algorithm of track fastener image is used to segment the image, and only the track fastener clip subimage is used for detection. The result shows that the same picture of track fasteners is an abnormal picture when the full picture is used for detection. The test result is a normal picture when only the track fasteners clip subimage is used for testing. In fact, the detected picture is a normal track fastener image. The results show that the defect detection of track fasteners is interfered by the features of the bolt subgraphs of track fasteners. The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener is shown in Figure6.Figure 6
The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener.


#### 4.2.4. Experimental Results of Automatic Segmentation of Pandrol Track Fastener Pictures

The input image of track fastener is processed by gray processing, gray histogram equalization, and Gaussian filtering, and the normalized correlation coefficient is used to locate the clip area of the track fastener and segment it.The segmentation results of the Pandrol track fastener pictures are shown in Table2. The experiment was compared and analyzed. Experiments show that it is better to segment the picture after grayscale, histogram equalization, and Gaussian filtering. Pictures that are unsuccessfully segmented are defined as false alarm image in track fastener detection and counted in the detection false alarm rate.Table 2
Track fastener pictures automatic segmentation results.
Image processing methodNO (piece)NS (piece)SR (%)Original1064101295.11Grayscale1064103797.46Grayscale + histogram equalization1064105298.87Grayscale + histogram equalization + Gaussian filtering1064106299.81


#### 4.2.5. Defect Detection Results of Pandrol Track Fasteners Based on LDFFN

For the left and right fasteners, 300 normal track fastener clip subimages were used for training, and the remaining clip subimages were used for detection. The data set is the same as DFFN.The defect detection results of Pandrol track fasteners based on LDFFN are shown in Table3. For the track fastener of Pandrol, the false alarm rate and error rate of track fastener defect detection method based on LDFFN are effectively reduced compared with the track fastener defect detection method based on DFFN. It is proved that the defect detection of track fasteners is interfered by the features of the bolt subimages of track fasteners. The track fastener clip subimage obtained by the automatic segmentation algorithm of track fastener images is used for track fastener defect detection, which can effectively reduce the false alarm rate and error rate of detection. The performance of track fastener defect detection has been improved. The validity of Pandrol track fastener defects based on LDFFN is verified by experiments.Table 3
Defect detection results of Pandrol track fasteners based on LDFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerDFFN2123042012.40010.56LDFFN231114204.5503.87Right track fastenerDFFN1932942013.06010.98LDFFN21394204.0503.41


## 4.1. Data Set

The picture of track fasteners in this article comes from the Shijiazhuang-Taiyuan high-speed rail line, called Pandrol fasteners. The state of common Pandrol track fasteners is shown in Figure4, where (a) is the image of normal track fasteners; (b) is the image of broken track fastener; (c) is the image of loose track fastener.Figure 4
Common track fastener status. (a) The image of normal track fasteners. (b) The image of broken track fastener. (c) The image of loose track fastener.
(a)(b)(c)A total of 1064 pictures and 84 pictures of normal and abnormal track fasteners, respectively, on Pandrol are presented. Among them, there are 542 left fasteners for the normal track, 42 left fasteners for the abnormal track, 522 right fasteners for the normal track, and 42 right fasteners for the abnormal track. In this paper, the detection indexes in reference are cited for comparison of detection efficiency.(1)
Segmentation rate (SR): NO is represented as the number of original images, and NS is represented as the number of pictures successfully segmented [41].(9)SR=NSNO×100%.(2)
False alarm rate (FAR), missed alarm rate (MAR), and error rate (ER): DAA indicates that the abnormal track fastener detection result is abnormal; DAN indicates that the abnormal track fastener test result is normal; DNN indicates that the normal track fastener test result is normal; DNA indicates that the normal track fastener test result is abnormal [41].(10)FAR=DNADNN+DNA×100%,MAR=DANDAA+DAN×100%,ER=DNA+DANDNN+DNA+DAA+DAN×100%.


## 4.2. Experimental Process and Results

### 4.2.1. Reconstruction of Picture Features of Track Fasteners

Feature reconstruction images of track fastener are shown in Figure5. (a) is the reconstruction image of track fastener image extracted by AE, and (b) is the image feature reconstruction map of track fastener extracted by RBM. The edge feature of track fastener contour can be better extracted by AE, and the internal texture feature of track fastener can be better extracted by RBM. Therefore, the image extracted by AE and RBM is fused in this paper. Edge features and texture features are included in the fusion features. The fusion features can fully express the information in the image, and track fastener defects are more conducive to be detected.Figure 5
Image feature reconstruction of track fasteners. (a) Image feature reconstruction extracted by AE. (b) Image feature reconstruction extracted by RBM.
(a)(b)


### 4.2.2. Defect Detection Results of Pandrol Track Fasteners Based on Deep Feature Fusion Network (DFFN)

Based on the DFFN-based Pandrol track fastener defect detection method, the image features of the track fastener are extracted through AE and RBM, the extracted full-image features are feature-fused, and then the state of the track fastener is detected by MD.There are 542 normal left track fasteners and 42 abnormal left track fasteners. Among them, 300 normal left track fastener pictures are randomly selected to train the network, and the remaining left track fastener pictures are used for detection. There are 522 normal right track fasteners and 42 abnormal right track fasteners. Among them, 300 normal right track fastener pictures are randomly selected to train the network, and the remaining right track fastener pictures are used for detection.For AE extraction of track fastener image features, two coding layers and two decoding layers are designed in AE network. When encoding, the image dimension is reduced to 256 dimensions. Through the first coding layer, the ReLU activation function is used to output 128 dimensions; through the second coding layer, the ReLU activation function is used to output 64 dimensions; and then the output of the second encoding is used as the input of decoding. The 256-dimensional image features are output by two decoding layers. Training AE model, set the training cycle to 100 and the bit size to 100 and save the model. When RBM is used to extract the image features of track fastener, RBM is called directly, and the training period is set to 100 and the bit size is 100. Then, the track fastener image features extracted by AE and the track fastener image features extracted by RBM are fused in serial, and the covariance matrix, inverse matrix of covariance matrix, and mean vector of training set are calculated, and then MD is calculated.The defects detection results of Pandrol track fasteners based on DFFN are shown in Table1. For Pandrol track fastener, the track fastener defect detection method based on AE or RBM extraction track fastener image features has high error rate and missing alarm rate. The false alarm rate and error rate of track fastener defect detection are effectively reduced by DFFN that combines AE and RBM. It shows that the image features of track fasteners extracted by AE and RBM are merged, and the image edge features and texture features are used at the same time, which is more conducive to the detection of track fastener defects. The performance of track fastener defect detection has been improved.Table 1
Defects detection results of Pandrol track fasteners based on DFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerAE2142838311.577.1410.92RBM2123037512.4011.9012.32DFFN2123042012.40010.56Right track fastenerAE1972639411.269.5211.36RBM1942837512.6111.9012.50DFFN1932942013.06010.98


### 4.2.3. Verification of Feature Interference of Bolt Subgraph of Pandrol Track Fastener

The abnormal pictures detected in DFFN test pictures are displayed, and some pictures are normal. DFFN detects normal pictures as abnormal pictures. In order to find out the reason, the automatic segmentation algorithm of track fastener image is used to segment the image, and only the track fastener clip subimage is used for detection. The result shows that the same picture of track fasteners is an abnormal picture when the full picture is used for detection. The test result is a normal picture when only the track fasteners clip subimage is used for testing. In fact, the detected picture is a normal track fastener image. The results show that the defect detection of track fasteners is interfered by the features of the bolt subgraphs of track fasteners. The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener is shown in Figure6.Figure 6
The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener.


### 4.2.4. Experimental Results of Automatic Segmentation of Pandrol Track Fastener Pictures

The input image of track fastener is processed by gray processing, gray histogram equalization, and Gaussian filtering, and the normalized correlation coefficient is used to locate the clip area of the track fastener and segment it.The segmentation results of the Pandrol track fastener pictures are shown in Table2. The experiment was compared and analyzed. Experiments show that it is better to segment the picture after grayscale, histogram equalization, and Gaussian filtering. Pictures that are unsuccessfully segmented are defined as false alarm image in track fastener detection and counted in the detection false alarm rate.Table 2
Track fastener pictures automatic segmentation results.
Image processing methodNO (piece)NS (piece)SR (%)Original1064101295.11Grayscale1064103797.46Grayscale + histogram equalization1064105298.87Grayscale + histogram equalization + Gaussian filtering1064106299.81


### 4.2.5. Defect Detection Results of Pandrol Track Fasteners Based on LDFFN

For the left and right fasteners, 300 normal track fastener clip subimages were used for training, and the remaining clip subimages were used for detection. The data set is the same as DFFN.The defect detection results of Pandrol track fasteners based on LDFFN are shown in Table3. For the track fastener of Pandrol, the false alarm rate and error rate of track fastener defect detection method based on LDFFN are effectively reduced compared with the track fastener defect detection method based on DFFN. It is proved that the defect detection of track fasteners is interfered by the features of the bolt subimages of track fasteners. The track fastener clip subimage obtained by the automatic segmentation algorithm of track fastener images is used for track fastener defect detection, which can effectively reduce the false alarm rate and error rate of detection. The performance of track fastener defect detection has been improved. The validity of Pandrol track fastener defects based on LDFFN is verified by experiments.Table 3
Defect detection results of Pandrol track fasteners based on LDFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerDFFN2123042012.40010.56LDFFN231114204.5503.87Right track fastenerDFFN1932942013.06010.98LDFFN21394204.0503.41


## 4.2.1. Reconstruction of Picture Features of Track Fasteners

Feature reconstruction images of track fastener are shown in Figure5. (a) is the reconstruction image of track fastener image extracted by AE, and (b) is the image feature reconstruction map of track fastener extracted by RBM. The edge feature of track fastener contour can be better extracted by AE, and the internal texture feature of track fastener can be better extracted by RBM. Therefore, the image extracted by AE and RBM is fused in this paper. Edge features and texture features are included in the fusion features. The fusion features can fully express the information in the image, and track fastener defects are more conducive to be detected.Figure 5
Image feature reconstruction of track fasteners. (a) Image feature reconstruction extracted by AE. (b) Image feature reconstruction extracted by RBM.
(a)(b)


## 4.2.2. Defect Detection Results of Pandrol Track Fasteners Based on Deep Feature Fusion Network (DFFN)

Based on the DFFN-based Pandrol track fastener defect detection method, the image features of the track fastener are extracted through AE and RBM, the extracted full-image features are feature-fused, and then the state of the track fastener is detected by MD.There are 542 normal left track fasteners and 42 abnormal left track fasteners. Among them, 300 normal left track fastener pictures are randomly selected to train the network, and the remaining left track fastener pictures are used for detection. There are 522 normal right track fasteners and 42 abnormal right track fasteners. Among them, 300 normal right track fastener pictures are randomly selected to train the network, and the remaining right track fastener pictures are used for detection.For AE extraction of track fastener image features, two coding layers and two decoding layers are designed in AE network. When encoding, the image dimension is reduced to 256 dimensions. Through the first coding layer, the ReLU activation function is used to output 128 dimensions; through the second coding layer, the ReLU activation function is used to output 64 dimensions; and then the output of the second encoding is used as the input of decoding. The 256-dimensional image features are output by two decoding layers. Training AE model, set the training cycle to 100 and the bit size to 100 and save the model. When RBM is used to extract the image features of track fastener, RBM is called directly, and the training period is set to 100 and the bit size is 100. Then, the track fastener image features extracted by AE and the track fastener image features extracted by RBM are fused in serial, and the covariance matrix, inverse matrix of covariance matrix, and mean vector of training set are calculated, and then MD is calculated.The defects detection results of Pandrol track fasteners based on DFFN are shown in Table1. For Pandrol track fastener, the track fastener defect detection method based on AE or RBM extraction track fastener image features has high error rate and missing alarm rate. The false alarm rate and error rate of track fastener defect detection are effectively reduced by DFFN that combines AE and RBM. It shows that the image features of track fasteners extracted by AE and RBM are merged, and the image edge features and texture features are used at the same time, which is more conducive to the detection of track fastener defects. The performance of track fastener defect detection has been improved.Table 1
Defects detection results of Pandrol track fasteners based on DFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerAE2142838311.577.1410.92RBM2123037512.4011.9012.32DFFN2123042012.40010.56Right track fastenerAE1972639411.269.5211.36RBM1942837512.6111.9012.50DFFN1932942013.06010.98


## 4.2.3. Verification of Feature Interference of Bolt Subgraph of Pandrol Track Fastener

The abnormal pictures detected in DFFN test pictures are displayed, and some pictures are normal. DFFN detects normal pictures as abnormal pictures. In order to find out the reason, the automatic segmentation algorithm of track fastener image is used to segment the image, and only the track fastener clip subimage is used for detection. The result shows that the same picture of track fasteners is an abnormal picture when the full picture is used for detection. The test result is a normal picture when only the track fasteners clip subimage is used for testing. In fact, the detected picture is a normal track fastener image. The results show that the defect detection of track fasteners is interfered by the features of the bolt subgraphs of track fasteners. The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener is shown in Figure6.Figure 6
The characteristic interference verification diagram of the bolt subgraph of the Pandrol track fastener.


## 4.2.4. Experimental Results of Automatic Segmentation of Pandrol Track Fastener Pictures

The input image of track fastener is processed by gray processing, gray histogram equalization, and Gaussian filtering, and the normalized correlation coefficient is used to locate the clip area of the track fastener and segment it.The segmentation results of the Pandrol track fastener pictures are shown in Table2. The experiment was compared and analyzed. Experiments show that it is better to segment the picture after grayscale, histogram equalization, and Gaussian filtering. Pictures that are unsuccessfully segmented are defined as false alarm image in track fastener detection and counted in the detection false alarm rate.Table 2
Track fastener pictures automatic segmentation results.
Image processing methodNO (piece)NS (piece)SR (%)Original1064101295.11Grayscale1064103797.46Grayscale + histogram equalization1064105298.87Grayscale + histogram equalization + Gaussian filtering1064106299.81


## 4.2.5. Defect Detection Results of Pandrol Track Fasteners Based on LDFFN

For the left and right fasteners, 300 normal track fastener clip subimages were used for training, and the remaining clip subimages were used for detection. The data set is the same as DFFN.The defect detection results of Pandrol track fasteners based on LDFFN are shown in Table3. For the track fastener of Pandrol, the false alarm rate and error rate of track fastener defect detection method based on LDFFN are effectively reduced compared with the track fastener defect detection method based on DFFN. It is proved that the defect detection of track fasteners is interfered by the features of the bolt subimages of track fasteners. The track fastener clip subimage obtained by the automatic segmentation algorithm of track fastener images is used for track fastener defect detection, which can effectively reduce the false alarm rate and error rate of detection. The performance of track fastener defect detection has been improved. The validity of Pandrol track fastener defects based on LDFFN is verified by experiments.Table 3
Defect detection results of Pandrol track fasteners based on LDFFN.
Track fastenerDetection methodDNN (piece)DNA (piece)DAA (piece)DAN (piece)FAR (%)MAR (%)ER (%)Left track fastenerDFFN2123042012.40010.56LDFFN231114204.5503.87Right track fastenerDFFN1932942013.06010.98LDFFN21394204.0503.41


## 5. Conclusion

In this paper, a defect detection method for track fastener of Pandrol based on LDFFN is proposed. Firstly, the clip subgraph of track fasteners is obtained through the automatic segmentation algorithm of track fastener images. Secondly, through AE and RBM, the edge feature and texture feature of the track fastener clip subimage are extracted, respectively, and the extracted track fastener clip subimage features are fused. Finally, the similarity measurement method MD is used to detect the status of track fasteners. The problem that the feature of track fastener bolt subimage interferes with the detection of track fastener defects and the problem of the sparse number of abnormal image samples of track fasteners are solved by the Pandrol track fastener defect detection method based on LDFFN. Moreover, the deep fusion features include both the edge feature and texture feature of the image. The full expression of the image feature makes the track fastener defect more conducive to be detected. The experimental results verify the effectiveness of the proposed LDFFN-based method for detecting defects in Pandrol track fasteners.When the defect detection of track fastener is carried out through the track fastener picture, the information of the bolt subpicture of track fastener is not effectively used, and whether the bolt is loose cannot be detected. In the future, the research will attempt to detect the track fastener bolts by other detection methods, so that the information of the track fastener bolts can be effectively used.



---

*Source: 6687146-2021-03-04.xml*