# Integrating the Symmetry Image and Improved Sparse Representation for Railway Fastener Classification and Defect Recognition

Authors: Jiajia Liu; Bailin Li; Ying Xiong; Biao He; Li Li
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

The detection of fastener defects is an important task for ensuring the safety of railway traffic. The earlier automatic inspection systems based on computer vision can detect effectively the completely missing fasteners, but they have weaker ability to recognize the partially worn ones. In this paper, we propose a method for detecting both partly worn and completely missing fasteners, the proposed algorithm exploits the first and second symmetry sample of original testing fastener image and integrates them for improved representation-based fastener recognition. This scheme is simple and computationally efficient. The underlying rationales of the scheme are as follows: First, the new virtual symmetrical images really reflect some possible appearance of the fastener; then the integration of two judgments of the symmetrical sample for fastener recognition can somewhat overcome the misclassification problem. Second, the improved sparse representation method discarding the training samples that are “far” from the test sample and uses a small number of samples that are “near” to the test sample to represent the test sample, so as to perform classification and it is able to reduce the side-effect of the error identification problem of the original fastener image. The experimental results show that the proposed method outperforms state-of-the-art fastener recognition methods.


---

## Body

## 1. Introduction

Railroads conduct regular inspection of the track is a very critical task for maintaining safe and efficient operation. Traditionally, this task is executed by trained works to periodically walk along railway lines to search for any damage of railway facilities. However, the manual inspection is slow, costly, and even dangerous. With the extension of high-speed railway network, the inspection faces more challenges than ever before. Recently, the researchers all over the world are interested in developing automatic inspection devices, which are installed on track checking train and are able to detect railway defects very efficiently. An automatic railway inspection system is composed of a series of functions such as gauge measurement [1], track profile measurement [2, 3], track-surface defects detection [4], and fastener defects detection [5, 6]. In this paper, our research aims to develop fastener defects detection algorithm and automatically find defective fasteners based on computer vision technologies.The fasteners are used to hold the track on sleepers, as shown in Figure1, while the causes of two common quality problems are partly worn and missing fasteners also shown in Figure 1, which would cause the train derailment and even threaten the safety of train operation; therefore, the detection accuracy is the most important target in fastener classification and defect recognition. In the past decade, some researchers have devoted themselves to exploiting fastener inspection methods. For hook-shaped fasteners, Yang et al. [7] took advantage of direction field as the template of fastener. Similarly, Stella et al. [8] employed wavelet transformation and principal component analysis to preprocess railway images and searched for the missing fasteners using the neural classifier. To achieve real-time performance, De Ruvo et al. [9] applied the error back propagation algorithm to model the fasteners and also introduced a FPGA-based architecture for automatic hexagonal bolts detection using the same algorithm [10]. However, the main work of the researches above aims at searching for missing fasteners, and it is difficult to detect the partly worn fastener. Recently, Mazzeo et al. [5] and Rubinsztejn [11] have successfully applied the Adaboost algorithm to the fastener detection. Specifically, Xia et al. [6] divided the hook-shaped fastener into four parts and each part was independently trained by Adaboost. Similarly, Li et al. [12] used image processing methods to detect the components of fastener. Other technologies that have been used for modeling and detecting fasteners include support vector machine (SVM) [13], Gabor filters [14], and edge detection [15]. The methods mentioned above simply use the part of fastener as an independent object, and they can solve the detection problem of the partly worn fastener to some extent; however, some important factors such as the symmetrical structure of fastener and the relationship between the image feature and classifier are neglected; thus they have weaker ability to identify the partially worn fasteners.Figure 1
Fasteners with various existences form on the real railway line. (a) The fastener in left image is intact fastener. (b) The fastener in middle image is partially worn. (c) The fastener in right image is completely missing.
(a)
(b)
(c)In summary, the majority of earlier methods effectively utilize image feature and take advantage of discriminative classifiers to classify the testing fastener samples, in order to achieve the purpose of defect identification. However, up to now, fastener recognition is still confronted with a number of challenges such as varying illumination, image noise, and fastener poses. Affected by these factors, the interclass distance among the different classes of samples, especially the distance between partially worn fastener and intact fastener, is difficult to distinguish; on the other hand, there is no uniform representation of the defective cases, it seems that more training samples are able to reveal more possible variation of samples in illumination or pose and are consequently beneficial for correct classification of target; however, there are only a limited number of instances to train classifiers, and the training sample is not enough. In order to address these challenges and obtain better recognition result, people have made many efforts to synthesize new samples from the original images. For example, in the face recognition, literature [16–18] used simple geometric transformation or virtual views to generate virtual samples, and Sharma et al. [19] synthesized multiple virtual views of a person under different pose and illumination from a single face image; these methods exploited extended training samples to classify the face. Though previous literatures have made many efforts in making virtual or synthesized object images, the special nature of the detection object is neglected, such as the symmetry of the structure of fastener in fastener recognition.This motivates us to exploit the symmetrical structure of the fastener to generate two new “virtual samples” of every test fastener and devise a representation-based method to perform fastener recognition, and our method is designed for finding both the partially worn and the completely missing instances. This paper has the following main contributions: first, the idea of symmetry is introduced into fastener recognition for the first time, and we use every original testing sample to generate two “symmetrical fastener” samples. Letti, yi, and zi, respectively, stand for the original fastener image and the first and second “symmetrical fastener” image. Then the left half columns of yi are set to the same parts as that of ti and the right half columns of yi are the mirror image of the left half columns of yi; however, the right half columns of zi are set to the same parts as that of ti and the left half columns of zi are the mirror image of the right half columns of zi. Our proposed scheme is to use a simple way to obtain more training and test samples by the symmetrical operation. Second, an improved representation-based classification algorithm was proposed for fastener detection; this method simultaneously uses the first and second “symmetrical fastener” testing samples to perform a two-step classification. This method also takes advantage of decision fusion, which has proven to be very competent and is usually better than the decision level and feature level fusion. Third, it provides a very large number of experiments and the experimental results show that the proposed method is very competitive.The remainder of this paper is organized as follows. In Section2, we describe the proposed method in detail. Section 3 provides the analysis and the computational complexity of our methods. In Section 4, our methods are verified by extensive experiments using fastener databases. Section 5 offers our conclusions.


## 2. The Proposed Method

In this section, we present the main steps of the proposed method in detail. Prior to commencing the method, there are several premise conditions we need to explain.First, the fastener dataset still consists of three states such as missing fasteners, worn fasteners, and intact fasteners after the symmetry operations (as shown in Figure1). Therefore, there are three classes and each class has n training samples in the new training set based on the state of new “symmetrical sample”; let x1,x2,…,xN represent all the N training samples and column vector x(i-1)n+k (k=1,2,…,n) stands for the kth training sample of the ith subject; On the other hand, there are two “virtual symmetry samples” obtained from the original test fastener ti and forming two new sets of testing subsets Y=[y1,y2,…,yM] and Z=[z1,z2,…,zM] based on the original testing set T=[t1,t2,…,tM].Second, all the samples would be converted into one-dimensional column vectors via feature extraction algorithm before the classification algorithm is employed. As shown in Figure2, the appearance of the first and second symmetry sample has a great change relative to original image; and the main difference lies in the fact that the intact fasteners and defective fasteners are the end region of the fastener (regions in four red rectangles); thus the region within four red rectangles is the region of interest (ROI); therefore, Pyramid Histogram of Oriented Gradients (PHOG) [20] of ROI in image is extracted, and we use x~i, y~i, and z~i to denote the one-dimensional column vectors of the fasteners xi, yi, and zi, respectively.Figure 2
The four regions of interest in fastener image; (b) and (c), respectively, the first and second symmetrical fasteners generated from the original sample as shown in (a); we can see that the appearance of two new samples has a great change relative to original image; and the main difference lies in the fact that the intact fasteners and defective fasteners are the end region of the fastener.
(a)
Original fastener
(b)
The first symmetrical fastener
(c)
The second symmetrical fastener


### 2.1. Main Processes of the Proposed Method

The proposed method includes the following two modules. The first module generates “symmetrical fastener” of original samples in the testing datasets. The second module uses the two types of “symmetrical image” in test sets to perform two-step fastener recognition, respectively. The algorithm of two-step fastener recognition will be described in Section2.2. The proposed algorithm in detail is as follows.Step 1.
Use every original testing sample to generate two “symmetrical fastener” samples. Letti be the ith testing sample in the form of image matrix. Let yi and zi, respectively, stand for the first and second “symmetrical fastener” testing samples generated from ti. The left half columns of yi are set to the same parts as that of ti and the right half columns of yi are the mirror image of the left half columns of yi. However, the right half columns of zi are set to the same parts as that of ti and the left half columns of zi are the mirror image of the right half columns of zi. The mirror image S of an arbitrary image R is defined as S(i,j)=R(i,V-j+1), i=1,…,U; j=1,…,V. U and V are the numbers of rows and columns of R, respectively. S(i,j) denotes the pixel located in the ith row and jth column of S [21].Step 2.
Use the first “symmetrical fastener” to perform the improved sparse representation algorithm for fastener recognition. The method proposes representing the symmetrical fasteneryi as a sparse linear combination of all the samples in train sets and identifies which training sample the test sample yi is most similar to. Let the class label Si1 of this training sample denote the ultimate class label of the test sample yi. For the details of algorithm, please see Section 2.2.Step 3.
Use the second “symmetrical fastener” to perform the improved sparse representation algorithm for fastener recognition. LetSi2 denote the result of test sample zi with respect to the ith class. This step shares the same algorithm as Step 2.Step 4.
Integrate the results obtained using Steps2 and 3 to conduct fusion decision. In other words, the judgment result of the original test sample ti is achieved according to the results of the first symmetrical image yi and second symmetrical image zi. The details of the fusion rule are as shown in Table 1.Table 1
The decision rule for the ultimate form of original testing fastener.
The form of the first symmetrical fasteneryi
The form of the second symmetrical fastenerzi
The ultimate form of original testing fastenerti
Intact
Intact
Intact
Partially worn
Partially worn
Missing
Partially worn
Partially worn
Intact
Partially worn
Partially worn
Partially worn
Missing
Partially worn
Missing
Intact
Partially worn
Partially worn
Partially worn
Missing
Missing


### 2.2. The Improved Sparse Representation Algorithm for Symmetrical Image Recognition

Sparse representation is one core issue of the modern theory of compressed sensing, and it is intended to represent a sample using a minimal number of nonzero coefficient terms. A number of works have shown that sparse representation and classification (SRC) algorithm can obtain a very high accuracy for image classification such as object recognition and image resolution [22–25]. However, the conventional SRC algorithm such as the one proposed in [22] has a relatively high computational cost, the improved sparse representation algorithm for fastener recognition is presented in this subsection, and our algorithm is mathematically tractable and computationally efficient. The first step in our algorithm aims at identifying and discarding the training samples that are “far” from the test sample and the second step then exploits the remaining training samples x1,…,xK as candidates for the class label of the test sample, and the second step represents the test sample as a linear combination of the “nearest K neighbors” and uses the representation result to perform classification. For simplicity of presentation, we describe the algorithm only on the first “symmetrical fastener”; the algorithm on the second “symmetrical fastener” is the same except that the first “symmetrical fastener” images are replaced with the second “symmetrical fastener” images.


#### 2.2.1. Decision of theK Nearest Sample to the Test Image

The choice algorithm ofK nearest feature subspace exploits the representation ability of each training sample to determine the “K nearest neighbors” for the test sample. This method uses all of the training samples to represent each test sample [21] and exploits the representation result to identify the K nearest neighbors of the test sample from the set of the training samples. It first assumes that the following equation is approximately satisfied:(1)y~=f1x~1+f2x~2+⋯+fNx~N,where y is the test sample and fi is referred to as the coefficient. Let X=[x~1⋯x~i⋯x~N] and F=[f1⋯f2⋯fN]T. We rewrite (1) into the following equation: (2)y~=XF.F is calculated using F-=(XTX+μI)-1XTy~. And F-=[f-1,…,f-i,…,f-N]T. μ is a small positive constant and I is the identity matrix. Equation (1) implies that the effect on representing the test sample of the kth sample can be evaluated using(3)dk=y~-∑i=1Nf~ix~i2.Equation (1) shows that every training sample makes its own contribution to representing of the test sample y and the contribution that the ith training sample makes is fixi. The contribution in representing the test sample can be also assessed by the residual of the test sample with respect to the ith training sample; that is, dk=y~-f~ix~i2. dk can also be somewhat viewed as a measurement of the distance between the test sample and the ith training sample. We consider that a small dk means that the ith training sample has a great contribution in representing the test sample. We exploit dk to identify the K training samples that have the K greatest contributions; we would like to point out that the effect on representing the test sample of the kth training sample is somewhat similar to the distance between the test sample and the kth training sample.


#### 2.2.2. The Precise Recognition for the Symmetrical Image

The second phase we identify the test sample using all the retainedK training samples, and represents the test sample as a linear combination of all the K nearest neighbors and uses the representation result to perform the precise fastener classification. If dr1≤dr2≤⋯≤drN, then we say that the r1th,r2th⋯rkth training samples are the first K candidate samples of the sample. In other words, we can consider that the ultimate class label of the test sample should be one element of D=cr1,cr2,…,crk, and cr1,cr2,…,crk are the class labels of the first K candidate samples, respectively. As the above steps roughly determine that the test sample is from a small number of classes, we refer to the selection of the first K candidate training samples as coarse classification.The algorithm then uses a linear combination of the firstK candidate training samples to represent the test sample. In other words, if the first K candidate training samples are denoted by x1,x2,…,xk, respectively, then the algorithm assumes that the following equation is approximately satisfied:(4)y~=f1x~1′+f2x~2′+⋯+fkx~k′,where fk is the coefficient. We rewrite (4) as(5)y~=X′F,where F=[f1⋯fk]T and X′=[x~1′,…,x~k′]. F is calculated using(6)F=X′TX′+γI-1X′Ty~,F~=f~1f~2,…,f~kT.γ is a small positive constant and I also denotes the identity matrix. Suppose that x~1′,…,x~k′ stand for all the training samples and the coefficients are f1,…,fk, respectively. The ultimate effect on representing the test sample of the kth training sample can be evaluated using(7)devs=y~-∑i=1Kf~ix~i′2.Ifr=argmindevs, then test sample y~ is ultimately assigned to the rth class, which is also referred to as the result of precise classification.


## 2.1. Main Processes of the Proposed Method

The proposed method includes the following two modules. The first module generates “symmetrical fastener” of original samples in the testing datasets. The second module uses the two types of “symmetrical image” in test sets to perform two-step fastener recognition, respectively. The algorithm of two-step fastener recognition will be described in Section2.2. The proposed algorithm in detail is as follows.Step 1.
Use every original testing sample to generate two “symmetrical fastener” samples. Letti be the ith testing sample in the form of image matrix. Let yi and zi, respectively, stand for the first and second “symmetrical fastener” testing samples generated from ti. The left half columns of yi are set to the same parts as that of ti and the right half columns of yi are the mirror image of the left half columns of yi. However, the right half columns of zi are set to the same parts as that of ti and the left half columns of zi are the mirror image of the right half columns of zi. The mirror image S of an arbitrary image R is defined as S(i,j)=R(i,V-j+1), i=1,…,U; j=1,…,V. U and V are the numbers of rows and columns of R, respectively. S(i,j) denotes the pixel located in the ith row and jth column of S [21].Step 2.
Use the first “symmetrical fastener” to perform the improved sparse representation algorithm for fastener recognition. The method proposes representing the symmetrical fasteneryi as a sparse linear combination of all the samples in train sets and identifies which training sample the test sample yi is most similar to. Let the class label Si1 of this training sample denote the ultimate class label of the test sample yi. For the details of algorithm, please see Section 2.2.Step 3.
Use the second “symmetrical fastener” to perform the improved sparse representation algorithm for fastener recognition. LetSi2 denote the result of test sample zi with respect to the ith class. This step shares the same algorithm as Step 2.Step 4.
Integrate the results obtained using Steps2 and 3 to conduct fusion decision. In other words, the judgment result of the original test sample ti is achieved according to the results of the first symmetrical image yi and second symmetrical image zi. The details of the fusion rule are as shown in Table 1.Table 1
The decision rule for the ultimate form of original testing fastener.
The form of the first symmetrical fasteneryi
The form of the second symmetrical fastenerzi
The ultimate form of original testing fastenerti
Intact
Intact
Intact
Partially worn
Partially worn
Missing
Partially worn
Partially worn
Intact
Partially worn
Partially worn
Partially worn
Missing
Partially worn
Missing
Intact
Partially worn
Partially worn
Partially worn
Missing
Missing


## 2.2. The Improved Sparse Representation Algorithm for Symmetrical Image Recognition

Sparse representation is one core issue of the modern theory of compressed sensing, and it is intended to represent a sample using a minimal number of nonzero coefficient terms. A number of works have shown that sparse representation and classification (SRC) algorithm can obtain a very high accuracy for image classification such as object recognition and image resolution [22–25]. However, the conventional SRC algorithm such as the one proposed in [22] has a relatively high computational cost, the improved sparse representation algorithm for fastener recognition is presented in this subsection, and our algorithm is mathematically tractable and computationally efficient. The first step in our algorithm aims at identifying and discarding the training samples that are “far” from the test sample and the second step then exploits the remaining training samples x1,…,xK as candidates for the class label of the test sample, and the second step represents the test sample as a linear combination of the “nearest K neighbors” and uses the representation result to perform classification. For simplicity of presentation, we describe the algorithm only on the first “symmetrical fastener”; the algorithm on the second “symmetrical fastener” is the same except that the first “symmetrical fastener” images are replaced with the second “symmetrical fastener” images.


### 2.2.1. Decision of theK Nearest Sample to the Test Image

The choice algorithm ofK nearest feature subspace exploits the representation ability of each training sample to determine the “K nearest neighbors” for the test sample. This method uses all of the training samples to represent each test sample [21] and exploits the representation result to identify the K nearest neighbors of the test sample from the set of the training samples. It first assumes that the following equation is approximately satisfied:(1)y~=f1x~1+f2x~2+⋯+fNx~N,where y is the test sample and fi is referred to as the coefficient. Let X=[x~1⋯x~i⋯x~N] and F=[f1⋯f2⋯fN]T. We rewrite (1) into the following equation: (2)y~=XF.F is calculated using F-=(XTX+μI)-1XTy~. And F-=[f-1,…,f-i,…,f-N]T. μ is a small positive constant and I is the identity matrix. Equation (1) implies that the effect on representing the test sample of the kth sample can be evaluated using(3)dk=y~-∑i=1Nf~ix~i2.Equation (1) shows that every training sample makes its own contribution to representing of the test sample y and the contribution that the ith training sample makes is fixi. The contribution in representing the test sample can be also assessed by the residual of the test sample with respect to the ith training sample; that is, dk=y~-f~ix~i2. dk can also be somewhat viewed as a measurement of the distance between the test sample and the ith training sample. We consider that a small dk means that the ith training sample has a great contribution in representing the test sample. We exploit dk to identify the K training samples that have the K greatest contributions; we would like to point out that the effect on representing the test sample of the kth training sample is somewhat similar to the distance between the test sample and the kth training sample.


### 2.2.2. The Precise Recognition for the Symmetrical Image

The second phase we identify the test sample using all the retainedK training samples, and represents the test sample as a linear combination of all the K nearest neighbors and uses the representation result to perform the precise fastener classification. If dr1≤dr2≤⋯≤drN, then we say that the r1th,r2th⋯rkth training samples are the first K candidate samples of the sample. In other words, we can consider that the ultimate class label of the test sample should be one element of D=cr1,cr2,…,crk, and cr1,cr2,…,crk are the class labels of the first K candidate samples, respectively. As the above steps roughly determine that the test sample is from a small number of classes, we refer to the selection of the first K candidate training samples as coarse classification.The algorithm then uses a linear combination of the firstK candidate training samples to represent the test sample. In other words, if the first K candidate training samples are denoted by x1,x2,…,xk, respectively, then the algorithm assumes that the following equation is approximately satisfied:(4)y~=f1x~1′+f2x~2′+⋯+fkx~k′,where fk is the coefficient. We rewrite (4) as(5)y~=X′F,where F=[f1⋯fk]T and X′=[x~1′,…,x~k′]. F is calculated using(6)F=X′TX′+γI-1X′Ty~,F~=f~1f~2,…,f~kT.γ is a small positive constant and I also denotes the identity matrix. Suppose that x~1′,…,x~k′ stand for all the training samples and the coefficients are f1,…,fk, respectively. The ultimate effect on representing the test sample of the kth training sample can be evaluated using(7)devs=y~-∑i=1Kf~ix~i′2.Ifr=argmindevs, then test sample y~ is ultimately assigned to the rth class, which is also referred to as the result of precise classification.


## 2.2.1. Decision of theK Nearest Sample to the Test Image

The choice algorithm ofK nearest feature subspace exploits the representation ability of each training sample to determine the “K nearest neighbors” for the test sample. This method uses all of the training samples to represent each test sample [21] and exploits the representation result to identify the K nearest neighbors of the test sample from the set of the training samples. It first assumes that the following equation is approximately satisfied:(1)y~=f1x~1+f2x~2+⋯+fNx~N,where y is the test sample and fi is referred to as the coefficient. Let X=[x~1⋯x~i⋯x~N] and F=[f1⋯f2⋯fN]T. We rewrite (1) into the following equation: (2)y~=XF.F is calculated using F-=(XTX+μI)-1XTy~. And F-=[f-1,…,f-i,…,f-N]T. μ is a small positive constant and I is the identity matrix. Equation (1) implies that the effect on representing the test sample of the kth sample can be evaluated using(3)dk=y~-∑i=1Nf~ix~i2.Equation (1) shows that every training sample makes its own contribution to representing of the test sample y and the contribution that the ith training sample makes is fixi. The contribution in representing the test sample can be also assessed by the residual of the test sample with respect to the ith training sample; that is, dk=y~-f~ix~i2. dk can also be somewhat viewed as a measurement of the distance between the test sample and the ith training sample. We consider that a small dk means that the ith training sample has a great contribution in representing the test sample. We exploit dk to identify the K training samples that have the K greatest contributions; we would like to point out that the effect on representing the test sample of the kth training sample is somewhat similar to the distance between the test sample and the kth training sample.


## 2.2.2. The Precise Recognition for the Symmetrical Image

The second phase we identify the test sample using all the retainedK training samples, and represents the test sample as a linear combination of all the K nearest neighbors and uses the representation result to perform the precise fastener classification. If dr1≤dr2≤⋯≤drN, then we say that the r1th,r2th⋯rkth training samples are the first K candidate samples of the sample. In other words, we can consider that the ultimate class label of the test sample should be one element of D=cr1,cr2,…,crk, and cr1,cr2,…,crk are the class labels of the first K candidate samples, respectively. As the above steps roughly determine that the test sample is from a small number of classes, we refer to the selection of the first K candidate training samples as coarse classification.The algorithm then uses a linear combination of the firstK candidate training samples to represent the test sample. In other words, if the first K candidate training samples are denoted by x1,x2,…,xk, respectively, then the algorithm assumes that the following equation is approximately satisfied:(4)y~=f1x~1′+f2x~2′+⋯+fkx~k′,where fk is the coefficient. We rewrite (4) as(5)y~=X′F,where F=[f1⋯fk]T and X′=[x~1′,…,x~k′]. F is calculated using(6)F=X′TX′+γI-1X′Ty~,F~=f~1f~2,…,f~kT.γ is a small positive constant and I also denotes the identity matrix. Suppose that x~1′,…,x~k′ stand for all the training samples and the coefficients are f1,…,fk, respectively. The ultimate effect on representing the test sample of the kth training sample can be evaluated using(7)devs=y~-∑i=1Kf~ix~i′2.Ifr=argmindevs, then test sample y~ is ultimately assigned to the rth class, which is also referred to as the result of precise classification.


## 3. Rationale Analysis of the Proposed Method

In this section we show the intuitive rationales and the justification of the proposed method.First, though the “symmetrical fastener” image is simply generated from the original fastener image, it also appears to be a natural image and properly reflects some possible appearance of the fastener image. Figure3 shows some original samples from the fastener database and the “symmetrical fastener” samples generated from the original samples. We can see that the “symmetrical fastener” samples not only seem to be different from the original samples, but also indeed somewhat reflect the possible variation of the fastener in image pose and illumination, which are not shown by the original samples. On the other hand, symmetrical operation enriches the diversity of fastener samples, and that is very useful to overcome the trouble of nonsufficient training samples. Since the “symmetrical fastener” training samples are complementary for the original training samples, the fastener detection captures only a few original training samples and still obtains enough information of the fastener.Figure 3
Some original samples from the database and the corresponding “symmetrical fastener” samples. (a) Showing the original samples; (b) and (c), respectively, showing the first and second “symmetrical fastener” samples generated from the original sample.
(a)
(b)
(c)Second, the symmetrical image is also sufficiently different from the original fastener image in terms of the distance metric; it can somewhat overcome the error identification problem of the fastener in fastener recognition. Identify the probable causes of impacts: the recognition effect on representing the test sample is somewhat similar to the distance between the test sample and the train sample ofkth class; in other words, a test sample is most likely from the classes which are close to it. The Latent Dirichlet Allocation reported in literature [26] achieved a very good test results in fastener classification and recognition; it outperforms the other methods mentioned above and can detect both partially worn and completely missing fasteners, and many other methods listed in the bibliography cannot detect the partially worn fasteners well. However, we found that it also has limited capacity in the detection of partially worn fasteners. For example, Figures 4(a), 4(b), and 4(c) respectively show the original testing sample, its first and second symmetrical sample, among them the original testing sample is erroneously and correctly classified by Latent Dirichlet Allocation and our method, respectively. Therefore, we calculate the mean Chi-square distance of Figures 4(a), 4(b), and 4(c) to the three classes such as missing fasteners, worn fasteners, and intact fasteners in training set and use the these distances as the indicator to determine which class they belong to. For two L-dimensional normalized histograms M and N, the Chi-square distance [27] could be described by the following equation:(8)χ2N,M=12∑i=1Lni-mi2ni+mi,where ni, mi denote the ith element value of feature histograms N and M. The test results in Table 2 show that the original testing sample and its first symmetrical sample have a smaller distance from the intact fasteners, and the second symmetrical sample has a smaller distance from the partly worn fasteners; Figures 4(d), 4(e), and 4(f), respectively, show the original training samples in training sets which are the nearest to them. As the experimental analysis above, the Latent Dirichlet Allocation method exploits only the original training samples and our method uses both the first and second “symmetrical fastener” testing samples; as a consequence, the fastener recognition method using only the original fastener images will be very hard to obtain accurate identification result; however, it is very easy to determine which classes the first and second symmetrical fastener belong to and then combine the recognition results of the first and second fastener to conduct the final discrimination result of original testing sample; the judgment rules are as shown in Table 1.Table 2
The original and symmetrical distances of the samples shown in Figure4.
Number of the subjects
Missing fastener
Worn fastener
Intact fastener
Original distance (×105)
4.72
3.47
3.24
The first symmetrical distance (×105)
4.93
3.55
2.76
The second symmetrical distance (×105)
2.89
3.37
4.58Figure 4
A test sample that is erroneously and correctly classified by the Latent Dirichlet Allocation and our method, respectively. (a) The test sample. (b) The first symmetrical sample of the test sample. (c) The second symmetrical sample of test sample. (d) The training sample which is the nearest to the test sample. (e) The training sample which is the nearest to the first symmetrical sample. (f) The training sample which is the nearest to the second symmetrical sample.
(a)
(b)
(c)
(d)
(e)
(f)The fastener sample in Figure4 is not a special case. we randomly select 60 samples in testing sets that the right branch of fasteners is partially broken and their first and second symmetrical sample are also obtained by image symmetry operation and then measure the PHOG feature distance between every 60 images of each class and intact fastener images in training sets; finally, we can achieve the mean distance of each class to intact training fasteners. As the test result shown in Figure 5, the original distance, the first symmetrical distance, and the second symmetrical distance are 3.59, 2.78, and 4.37, respectively; further, the original distance is much bigger than the first symmetrical distance and also much smaller than the second symmetrical distance; the two classes symmetrical images are two polarized developments relative to the original distance between original testing fasteners and intact training fasteners. Meanwhile, to better illustrate the advantages of the method of fastener image symmetry operation, we display the classification results of the three classes of fasteners by the improved sparse representation algorithm; as the test result shown in Figure 6, the recall rate of 60 original testing images is 78.4%, and the recall rate of the first and second symmetrical images is 96.7% and 100%, respectively. Then we can indirectly obtain that the final accuracy of original testing image is 96.7% according to the judgment results of the first and second symmetrical images. Thus, the correct recognition rate is higher than its previous value 78.4%. We can obtain that the following conclusion is that the change of the symmetrical distance (relative to original distance) is helpful for determining which class the first and second symmetrical fastener belong to, and the simultaneous use of the symmetrical sample of the original testing fastener is helpful for a “distance” based classifier to accurately classify the test sample and is able to reduce the side-effect of the error identification problem of the original fastener image.Figure 5
Demonstration of the idea of the construction of symmetrical sample. The dotted lines indicate the mean distance between the three classes of fastener images and intact training fasteners. The original distance, the first symmetrical distance, and the second symmetrical distance, respectively, express the feature distance between original test fastener, the first symmetrical fastener, the second symmetrical fastener, and intact fastener.Figure 6
The classification results of the three classes of fasteners by the improved sparse representation algorithm.The third rationale of the proposed method is that the algorithm uses the two-step fastener recognition, which is able to enhance the robustness of fastener recognition. The conventional SRC methods usually use the entire set of training samples to produce a representation for the test sample. They compute the distance between the representation of the test sample and that of the training sample and then use the distance as a measurement to indicate how well each class can represent this sample or the probability that the test sample comes from that class. That is to say, the shorter the distance a class has, the greater the probability the test sample comes from this class. Relative to the conventional SRC methods, the algorithm in this paper specifically takes the relationship between the test sample and its neighbors into account and only uses a linear combination of theK nearest neighbors to represent and classify the test fastener. It has the following underlying rationale: the related literature has shown that the test sample is usually not from the classes which have bigger distance from it [28]; therefore, the fine recognition can increase the probability of the test sample being correctly classified by eliminating the training samples from these classes. In the first phase of our algorithm, we compute the distances and exploit the discriminative information from K nearest principle to retrieve the first K nearest subspaces for the test sample and assume that the residual training samples have no effects on the ultimate classification decision. Such a practice does not only eliminate the interference of the residual training samples, but also reduce the scale of the problem. It will be very helpful for the second phase to perform an accurate classification. The second phase of our algorithm performs SRC based on the training samples from the selected k subspaces. Moreover, when determining the K nearest subspaces, the first phase uses only the linear regression in which the hat matrix for each class can be calculated offline. Thus, it has a very low time cost, and the whole method is also computationally much more efficient than the original SRC. Moreover, this method has similarities with Local Projection methods in that they only use a subset of the training samples. For example, Sugiyama integrated the ideas of the LDA (Linear Discriminant Analysis) and locality preserving projection to generate a Local Projection method for a problem where the samples in a class are multimodal [29]. This method is first to exploit the local structure of patterns to produce a representation for test samples and then uses a classifier to classify the test samples; we refer to this method as Local Projection method. However, the proposed method in this paper has some distinctions with Local Projection method: it takes the relationship between the test sample and neighbors into account by setting different coefficients for different neighbors and uses only the K nearest neighbors to represent and classify the test sample; the test sample and its nearest neighbors usually belong to the same class.In a word, the proposed method has the following rationales: in the module of the construction of symmetrical sample, the use of the symmetrical image does enable the fastener recognition method to exploit more available information of the fastener which is not shown by the original samples; since the “symmetrical fastener” training samples are complementary for the original training samples, it is very useful to overcome the trouble of nonsufficient training samples. Moreover, it is a simple way to obtain more training and test samples, and the detection system can capture only a few original training samples and still obtain enough information of the fastener. On the other hand, the symmetrical image is also sufficiently different from the original fastener image in terms of the distance metric, and the original distance is much bigger than the first symmetrical distance and also much smaller than the second symmetrical distance. Two types of symmetrical distance are two polarized developments relative to the original distance between original testing images and original training images; it is helpful for the improved sparse representation algorithm to accurately classify the test sample and overcome the error identification problem of the fastener image in fastener recognition. In the module of the fastener recognition, its first phase identifies a number of training samples that are the most similar to the test sample and takes the class labels of the identified training samples as candidates for the class label of the test sample. As the class labels of the identified training samples are usually a subset of those of all the training samples, ultimate classification becomes a problem of determining the class label from a smaller number of candidates. Under the conditions where the genuine class label of the test sample is really one of those of the identified training samples, this will be very helpful for the second phase to perform an accurate classification.


## 4. Experimental Results

In this section, we evaluated the effectiveness of our methods by performing experiments on railway fasteners databases. This fastener dataset consists of 1500 images from 3 subjects each providing 500 images; we segment the fastener image from the rail image which is collected by the video collection system, and the image size is110∗170 pixels. For each subject, part of 500 images was randomly selected to serve as training samples and the remaining ones were used for testing. All experiments were implemented using the Matlab 2010 on a PC with an Intel i5-2430 Processor. The performance of fastener classification is evaluated by recall rate [30], recall rate indicates the correctness of classification and recognition results, recall rate is widely used criteria for evaluating the performance of classification or detection algorithms, and it is defined as(9)Recallrate=TpN,where Tp is the number of samples that are correctly classified in N testing samples. In other words, recall rate requires detection method to find most of defects on rail images without missing.


### 4.1. Experiment of Image Symmetrical Operation in the Fastener Database

We first verify the effectiveness of image symmetrical operation method by performing experiments on railway fastener databases. In this section, 50, 100, 150, and 250 fastener images of each subject were randomly selected to serve as original training samples and the remaining ones were used for testing, respectively. It should be noted that the fasteners of the same type are counted separately. Hereafter, the integrations of our proposed scheme and support vector machine (SVM), Nearest Neighbor Analysis (NNA), and SRC are referred to as the improvements to SVM, NNA, and SRC, respectively. Table3 shows the recall rate of different methods; we see that our proposed scheme can improve SVM, NNA, and SRC and obtain high recall rate for all types of fasteners, especially for partially worn fastener. For example, when each subject provided 150 training samples, the rates of recall rate of SVM, NNA, and SRC are 83.7%, 87.8%, and 89.8%, respectively; however, the recall rates of the improvements to SVM, NNA, and SRC are 95.7%, 96.8%, and 97.5%, respectively. The experiments show that the proposed scheme not only greatly improves the accuracy of the representation-based classification and recognition methods but also is helpful for improving other fastener recognition methods.Table 3
The recall rate (%) of different methods on the fastener database.
Number of the training samples of each subject
Intact fastener
Partially worn fastener
Missing fastener
50
100
150
250
50
100
150
250
50
100
150
250
SVM
84.3
87.5
89.0
93.5
77.5
81.3
83.7
85.3
92.5
94.8
95.3
98.0
Improvement to SVM
88.8
91.2
95.8
96.3
89.5
91.6
95.7
98.5
94.5
96.8
97.3
98.5
NNA
86.3
88.5
92.8
94.5
81.6
83.3
87.8
92.3
91.5
94.3
95.8
97.3
Improvement to NNA
87.5
89.8
94.5
96.5
89.8
91.3
96.8
99.0
92.5
95.8
96.3
97.8
SRC
87.3
89.3
93.3
95.5
85.5
88.3
89.8
91.3
94.3
94.8
96.3
97.5
Improvement to SRC
89.8
90.3
95.8
96.3
89.3
93.7
97.5
96.8
94.8
95.5
97.2
98.3


### 4.2. Variation of the Performance of Our Method with Different Number of Candidate Samples

In this section, we mainly want to examine howK affects computation time and recognition accuracy. In this section, 250 fastener images of each subject were randomly selected to serve as original training samples and the rest are used for testing. In addition, we set the parameter K to be 10, 20, 60, 100, 150, and 200 in the fastener databases. In order to comprehensively show the performance of our method, we use Table 4 to briefly indicate the variation of the rate of recall rate of our method with different number of candidate samples. We see that when the number of candidate samples is smaller, our method almost always obtains a higher recall rate. This clearly shows that the problem of fastener detection turns into a k-class classification problem from a c-class classification problem by discarding the samples that are “far” from the test sample in the first phase of the improved sparse representation algorithm (of course, K is smaller than c), and it is really beneficial for the improvement of the accuracy of our method. However, this classifier exploits the neighbors of the test sample too coarsely and usually produces a bad classification result when K has a large value. Moreover, compared to the original SRC, though the proposed algorithm adds a procedure to find the K candidate training samples, it can significantly reduce the scale of the precise fastener recognition in the second-phase problem solved in performing SRC, and its own computational complexity is quite low. Therefore, the total computational cost of the improved sparse representation algorithm decreases, which is confirmed in the experimental results. The results of CPU time are also shown in Table 4. When using K=20 our method can obtain around 5 times speedup compared to classical SRC.Table 4
Comparison of recall rate and mean CPU time with different number of candidate samples.
Method
SRC
SRC-KNS
k
=
10
k
=
20
k
=
60
k
=
100
k
=
150
k
=
200
CPU time (s)
1.606
0.314
0.336
0.387
0.462
0.605
0.839
Recall rate (%)
87.3
97.8
98.8
95.3
91.8
88.3
87.8


### 4.3. Performance Comparison with Relative Fastener Detection Methods

In this section, we compare the proposed method with some relative fastener detection methods. We first reimplemented the feature extraction and classification algorithm in [6, 7, 26, 31] and let them operate on original images. Moreover, we also let the methods operate on the symmetrical images, so as to evaluate the effectiveness of the proposed method. When we set the parameter K=20, in other words, the first 20 fastener images of each subject were used as the original training samples and the remaining fastener images were taken as the test samples, the proposed method outperforms the other methods in fastener databases, especially in the subsets of partially worn fastener. As the results shown in Table 5, when the experiment operates on symmetrical images, the recall rates of the methods in literatures [6, 7, 26, 31] and our method are 94.5%, 85.3%, 94.3%, 96.8%, and 98.5%, respectively; however, the recall rates of methods in literatures [6, 7, 26, 31] operating on original images are 86.3%, 84.5%, 87.8%, 96.3%, and 98.5%, respectively. This indicates the virtual symmetrical images and the sparsity are helpful in improving the classification accuracy for fastener recognition. Moreover, the CRC [31] and our algorithm achieve higher classification accuracies than the other methods. The CRC is similar to the first phase of the proposed method. It classifies the test sample into the class corresponding to the least error in our first phase. However, there is misclassification in CRC if the right training sample does not contribute most in the first phase. Our method can avoid such misclassifications by recalculating the coefficient vector using the retained K training samples in the precise recognition phase for the symmetrical image of our method; thus this phase increases the contribution of the right training samples and suppresses those of the others, and this method enhances the sparsity of the K nearest samples in representation. The experimental results show that the proposed method can classify the fastener with a high accuracy and outperforms the state-of-the-art fastener recognition methods.Table 5
The recall rate (%) of different methods on fastener database.
Methods
The recall rate
Intact fastener
Partially worn fastener
Missing fastener
Xia’s method on original image [6]
94.3
86.3
94.8
Xia’s method on symmetrical image [6]
94.0
94.5
95.3
Yang’s method on original image [7]
93.5
84.5
96.3
Yang’s method on symmetrical image [7]
93.8
85.3
95.8
Feng’s method on original image [26]
95.5
87.8
95.5
Feng’s method on symmetrical image [26]
95.8
94.3
96.0
Zhang’s method on original image [31]
95.3
96.3
95.3
Zhang’s method on symmetrical image [31]
95.3
96.8
95.5
The proposed method in this paper
97.5
98.5
98.3


## 4.1. Experiment of Image Symmetrical Operation in the Fastener Database

We first verify the effectiveness of image symmetrical operation method by performing experiments on railway fastener databases. In this section, 50, 100, 150, and 250 fastener images of each subject were randomly selected to serve as original training samples and the remaining ones were used for testing, respectively. It should be noted that the fasteners of the same type are counted separately. Hereafter, the integrations of our proposed scheme and support vector machine (SVM), Nearest Neighbor Analysis (NNA), and SRC are referred to as the improvements to SVM, NNA, and SRC, respectively. Table3 shows the recall rate of different methods; we see that our proposed scheme can improve SVM, NNA, and SRC and obtain high recall rate for all types of fasteners, especially for partially worn fastener. For example, when each subject provided 150 training samples, the rates of recall rate of SVM, NNA, and SRC are 83.7%, 87.8%, and 89.8%, respectively; however, the recall rates of the improvements to SVM, NNA, and SRC are 95.7%, 96.8%, and 97.5%, respectively. The experiments show that the proposed scheme not only greatly improves the accuracy of the representation-based classification and recognition methods but also is helpful for improving other fastener recognition methods.Table 3
The recall rate (%) of different methods on the fastener database.
Number of the training samples of each subject
Intact fastener
Partially worn fastener
Missing fastener
50
100
150
250
50
100
150
250
50
100
150
250
SVM
84.3
87.5
89.0
93.5
77.5
81.3
83.7
85.3
92.5
94.8
95.3
98.0
Improvement to SVM
88.8
91.2
95.8
96.3
89.5
91.6
95.7
98.5
94.5
96.8
97.3
98.5
NNA
86.3
88.5
92.8
94.5
81.6
83.3
87.8
92.3
91.5
94.3
95.8
97.3
Improvement to NNA
87.5
89.8
94.5
96.5
89.8
91.3
96.8
99.0
92.5
95.8
96.3
97.8
SRC
87.3
89.3
93.3
95.5
85.5
88.3
89.8
91.3
94.3
94.8
96.3
97.5
Improvement to SRC
89.8
90.3
95.8
96.3
89.3
93.7
97.5
96.8
94.8
95.5
97.2
98.3


## 4.2. Variation of the Performance of Our Method with Different Number of Candidate Samples

In this section, we mainly want to examine howK affects computation time and recognition accuracy. In this section, 250 fastener images of each subject were randomly selected to serve as original training samples and the rest are used for testing. In addition, we set the parameter K to be 10, 20, 60, 100, 150, and 200 in the fastener databases. In order to comprehensively show the performance of our method, we use Table 4 to briefly indicate the variation of the rate of recall rate of our method with different number of candidate samples. We see that when the number of candidate samples is smaller, our method almost always obtains a higher recall rate. This clearly shows that the problem of fastener detection turns into a k-class classification problem from a c-class classification problem by discarding the samples that are “far” from the test sample in the first phase of the improved sparse representation algorithm (of course, K is smaller than c), and it is really beneficial for the improvement of the accuracy of our method. However, this classifier exploits the neighbors of the test sample too coarsely and usually produces a bad classification result when K has a large value. Moreover, compared to the original SRC, though the proposed algorithm adds a procedure to find the K candidate training samples, it can significantly reduce the scale of the precise fastener recognition in the second-phase problem solved in performing SRC, and its own computational complexity is quite low. Therefore, the total computational cost of the improved sparse representation algorithm decreases, which is confirmed in the experimental results. The results of CPU time are also shown in Table 4. When using K=20 our method can obtain around 5 times speedup compared to classical SRC.Table 4
Comparison of recall rate and mean CPU time with different number of candidate samples.
Method
SRC
SRC-KNS
k
=
10
k
=
20
k
=
60
k
=
100
k
=
150
k
=
200
CPU time (s)
1.606
0.314
0.336
0.387
0.462
0.605
0.839
Recall rate (%)
87.3
97.8
98.8
95.3
91.8
88.3
87.8


## 4.3. Performance Comparison with Relative Fastener Detection Methods

In this section, we compare the proposed method with some relative fastener detection methods. We first reimplemented the feature extraction and classification algorithm in [6, 7, 26, 31] and let them operate on original images. Moreover, we also let the methods operate on the symmetrical images, so as to evaluate the effectiveness of the proposed method. When we set the parameter K=20, in other words, the first 20 fastener images of each subject were used as the original training samples and the remaining fastener images were taken as the test samples, the proposed method outperforms the other methods in fastener databases, especially in the subsets of partially worn fastener. As the results shown in Table 5, when the experiment operates on symmetrical images, the recall rates of the methods in literatures [6, 7, 26, 31] and our method are 94.5%, 85.3%, 94.3%, 96.8%, and 98.5%, respectively; however, the recall rates of methods in literatures [6, 7, 26, 31] operating on original images are 86.3%, 84.5%, 87.8%, 96.3%, and 98.5%, respectively. This indicates the virtual symmetrical images and the sparsity are helpful in improving the classification accuracy for fastener recognition. Moreover, the CRC [31] and our algorithm achieve higher classification accuracies than the other methods. The CRC is similar to the first phase of the proposed method. It classifies the test sample into the class corresponding to the least error in our first phase. However, there is misclassification in CRC if the right training sample does not contribute most in the first phase. Our method can avoid such misclassifications by recalculating the coefficient vector using the retained K training samples in the precise recognition phase for the symmetrical image of our method; thus this phase increases the contribution of the right training samples and suppresses those of the others, and this method enhances the sparsity of the K nearest samples in representation. The experimental results show that the proposed method can classify the fastener with a high accuracy and outperforms the state-of-the-art fastener recognition methods.Table 5
The recall rate (%) of different methods on fastener database.
Methods
The recall rate
Intact fastener
Partially worn fastener
Missing fastener
Xia’s method on original image [6]
94.3
86.3
94.8
Xia’s method on symmetrical image [6]
94.0
94.5
95.3
Yang’s method on original image [7]
93.5
84.5
96.3
Yang’s method on symmetrical image [7]
93.8
85.3
95.8
Feng’s method on original image [26]
95.5
87.8
95.5
Feng’s method on symmetrical image [26]
95.8
94.3
96.0
Zhang’s method on original image [31]
95.3
96.3
95.3
Zhang’s method on symmetrical image [31]
95.3
96.8
95.5
The proposed method in this paper
97.5
98.5
98.3


## 5. Conclusion

The detection of worn and missing fasteners is an important task in railway inspection. However, the traditional manual inspection is of poor efficiency and dangerous to workers. On the other hand, the earlier automatic inspection systems based on computer visions are of low reliability. In this paper, a novel railway inspection method is proposed, which is able to detect the damaged fasteners, especially the fasteners which partially worn. This method has the following three major advantages:(1) The proposed symmetrical image of the original fastener indeed simulates possible variation of the fastener image; thus we capture only a few original training samples and still obtain enough information of the fastener. The proposed scheme is very simple and also helpful for improving fastener recognition methods. (2) We turn the detection problem of the original testing image into the detection of two tests of symmetrical image. Due to the interclass feature distance between the two classes symmetrical images and intact training fasteners are two polarized developments (relative to the original distance between intact training fasteners and intact testing fasteners); it is very easy to judge which class the first and second symmetrical image belong to; then the proposed method can achieve a better recognition result by integrating two classes of symmetrical image for representation-based fastener recognition. (3) The first and second phases of the improved sparse representation algorithm in this paper respectively made a coarse and fine classification decision for the test samples. The first phase exploits the discriminative information of K nearest samples to enhance their sparsity of the test sample and then represent the test sample as a linear combination of the determined K nearest neighbors so as to perform fine classification. It can greatly eliminate the side-effect, on the classification of the test sample, of the classes that are “far” from it (the test sample is usually not from these classes) and greatly improve the accuracy of the representation-based classification methods. The analyses and experimental results sufficiently show the rationales of the proposed scheme.



---

*Source: 462528-2015-11-09.xml*