# Quantitative Nondestructive Testing of Broken Wires for Wire Rope Based on Magnetic and Infrared Information

Authors: Xi Li; Juwei Zhang; Jingzhuo Shi
Journal: Journal of Sensors
 (2020)

---

## Abstract

The lifetime of wire rope is crucial in industry manufacturing, mining, and so on. The damage can be detected by using appropriate nondestructive testing techniques or destructive tests by cutting the part. For broken wires classification problems, this work is aimed at improving the recognition accuracy. Facing the defects at the exterior of the rope, a novel method for recognition of broken wires is firstly developed based on magnetic and infrared information fusion. A denoising method, which is adopted for magnetic signal, is proposed for eliminating baseline signal and wave strand. An image segmentation method is employed for parting the defects of infrared images. Characteristic vectors are extracted from magnetic images and infrared images, then kernel extreme learning machine network is applied to implement recognition of broken wires. Experimental results show that the denoising method and image segmentation are effective and the information fusion can improve the classification accuracy, which can provide useful information for estimating the residual lifetime of wire rope.


---

## Body

## 1. Introduction

Wire ropes play an important role in many fields such as cranes, oil drilling rigs, elevators, and mine hoist. The safety of wire ropes is closely related to people’s life and resources loss as well as the normal operation of industry. Because of the complex structure of wire ropes and the diversity of application environment, it is difficult to evaluate the health of wire ropes in service [1, 2]. Thus, it is necessary to effectively and accurately perform the quantitative nondestructive testing (NDT) of wire rope by adopting proper methods.At present, the NDT methods of wire rope include electromagnetic [3, 4], X-ray [5], acoustic [6–9], and optical [10] method [1]. X-ray apparatus has radioactive contamination; acoustic method detects wire rope by striking, which is simple but one-sided; CCD camera optical testing method can directly show the real defects through imaging, but it is susceptible to oil pollution; because of high sensitivity, high speed, and low cost, electromagnetic NDT method is widely used [11–14]. However, no single nondestructive testing technique can identify all kinds of defects. Infrared nondestructive testing does not contain dangerous radiation and has characteristic of noncontact; thus, it has widely applied in solving real problems in numerous areas [15].In addition, its popular application areas contain building sector [16, 17], aeronautics and astronautics [18], chemical industry [19], food [20], cultural heritage [21], and so on. Munoz et al. [22, 23] determined heat source dissipation from infrared thermographic measurements based on the heat diffusion equation provided by thermodynamics principles and identified damage evolution in carbon fibre reinforced composites combing acoustic emission and infrared thermography.Magnetic flux leakage (MFL) detection of wire rope mainly includes the forward calculation model of MFL detection, pretreatment of MFL signal, and inversion of defect [24]. For example, Yan et al. [25] employed a three-dimensional finite element method (FEM) to analyze MFL signals. This method provided theoretical guidance for detection signal analysis and hardware design. Based on the magnetic dipole model, Yang [2] created the leakage magnetic field analysis models of single wire fracture, surface broken wire, and internal broken wire of wire rope, which provided the theoretical basis for the quantitative analysis of wire rope. Zhao and Zhang [11, 12] made FEM on the distribution of magnetic flux leakage of typical broken wire defects in steel cables, and obtained the relationship between MFL and detection distance, damage size, and internal broken wire. In [13, 14], a magnetic dipole model was established to design the prototype, which provided a theoretical basis for the quantification of defects. Through the FEM model of wire rope and the FEM simulation under different broken wires, DU et al. [26] studied the influence of different broken wires on the safety coefficient of wire rope.Because actual MFL detection signals are polluted by many noise sources, it is necessary to preprocess the signals in order to reconstruct the defects. Zhang et al. [27, 28] utilized wavelet based on compressed sensing to denoise the strand wave, but it restored a lot of noise; then, they combined the Hilbert-Huang Transform (HHT) and Compressed Sensing Wavelet Filtering (CSWF) to reduce various background noises. Zheng and Zhang [29] exploited wavelet soft threshold to inhibit the noise; nevertheless, the denoising effect is poor. Then Zheng and Zhang [30] implemented Variational Mode Decomposition (VMD) and a wavelet transformation to remove noise from the raw MFL signals, which can effectively eliminate noise. Hong et al. [31] proposed an adaptive wavelet threshold denoising method based on a new threshold function, which achieved good denoising effect on the MFL signal of wire rope. To realize the visualization of defects, Zhao [13] utilized an adaptive notch filtering algorithm for suppressing wave noise.To visualize and quantify defects and realize quantitative detection of broken wires, researchers need to implement defect inversion. In order to perform defect inversion, numerous scholars have used various methods. Through adopting the wavelet super-resolution reconstruction technique, the resolution of defect grayscale was improved in [32]. Zhang and Tan [33] proposed a super-resolution (SR) reconstruction method based on Tikhonov regular multiframe, which can effectively remain image features of defects while the axial resolution was reduced and circumferential resolution was increased. In [28, 32], researchers implemented classification of defects by adopting back propagation (BP) neural networks. However, BP was easy to fall into local minimum, which can lead to problems such as network underfitting and insufficient generalization ability. Wan et al. [34] investigated the theory on optimal wavelet packet with the Least Squares Support Vector Machine (LS-SVM) to diagnose elevator faults, which was then validated by the experiment. Zheng and Zhang and Qin et al. [29, 35] took the Support Vector Machine (SVM) with a radial basis function classified to conduct the fault pattern recognition, whereas this method was not very effective.The researchers [15, 36] investigated the failures of steel ropes and defect of ferromagnetic specimens by means of thermovision. In [15], since the measurements required extremely sensitive thermovision technology, the method can detect the tight of ropes at certain conditions. In [34], the researchers developed a new active thermography technique, which can detect the defect in ferromagnetic steel specimens. The fusion of infrared and other information is effective and widely used. Kee and Oh et al. [16] combined air-coupled impact-echo and infrared thermography. It can improve effectiveness of the individual test data. Data fusion of ground-penetrating radar and infrared thermography improved the accuracy of detecting defects [37]. The researchers [38] combined finite element analysis with experimental data from infrared thermography, which provided accurate means to assess quantitatively the size and position of thermal imperfections. According to these, it is demonstrated that data fusion is effective. In this paper, fused data based on infrared thermography and magnetic is utilized to detect the number of broken wires.Electromagnetic NDT for wire rope is susceptible to hardware design and magnetic signal processing. In [13, 14], the location and number of sensors can affect the quality of acquisition signal. Insufficient quantity will lead to the serious loss of MFL signal, while dense placement of sensors can lead to serious signal interference, resulting in difficulty of noise reduction. Meanwhile, the small broken wire defect information may be drowned out by noise. However, thermal infrared is a visualization method, which can intuitively grasp the surface damage state of wire rope and be closer to the actual damage pattern than magnetic data. Meanwhile, it is without the shortcomings of magnetic detection method and it can make up for the loss of small defects in magnetic information. Thus, the combination of the two methods supply more information for the damage and can avoid the loss of defect information.To improve classification accuracy of broken wires and provide a reference for evaluating the service life of wire rope, the combination of infrared information and magnetic information is put forward for the first time to perform quantitative identification of wire rope. To processing magnetic signal, an algorithm based on Wavelet Total Variation (WATV) is proposed to remove noise from the raw MFL signals. The noise from high-frequency magnetic leakage, baseline drift, and strand waves can be suppressed by the proposed algorithm. To separate defects from infrared images, an image processing method based on distance is presented. After extracting statistical texture, invariant moment characteristics, and color moment, a fusion method based on kernel extreme learning machine (KELM) of decision level fusion is proposed to combine magnetic and infrared information. Experiment results show that the information fusion based on magnetic and infrared can improve the recognition rate of broken wires.In the next sections, the platform to get data, the processing for magnetic data, steps for extracting infrared information, and recognition for broken wires after information fusion will be introduced in turn. In this paper, major innovations are as follows: (1) the proposed denoising algorithm based on WATV can eliminate noise generated by channel imbalance, the structure of wire ropes, and so on; (2) an infrared image segmentation algorithm based on distance is presented; and (3) information fusion combined magnetic with infrared to perform classification is firstly adopted.


## 2. Related Knowledge

### 2.1. WATV Denoising

WATV denoising method can suppress pseudo-Gibbs oscillations and spurious noise spikes by estimating all wavelet coefficients simultaneously via the minimization of a single objective function. The method is formulated as an optimization problem incorporating both wavelet sparsity and TV regularization [39]. The wavelet coefficients are calculated by solving the optimization problem shown in formula (1). If ajsatisfies formula (2), F(w) is strictly convex.
(1)ω⌢=argminωFω=12Wy−ω22+∑j,kλjϕωj,k;aj+βDWTω1Wrepresents the wavelet transform; in ωj,k, j and k are the scale and time indices, respectively. λj and βare the regularization parameters. The penalty term DWTω1 is the total variation of the signal estimate x⌢; the function ϕ⋅;aj is a non-convex sparsity-inducing penalty function with
(2)0≤aj<1λjBased on variable splitting and the alternating direction method of multipliers (ADMM), equation (1) can be expressed as a constrained problem:
(3)argminu,ωg1ω+g2usubjecttou=ω(4)g1ω=12Wy−ω22+∑j,kλjϕωj,k;aj(5)g2u=βDWTu1The augmented Lagrangian is given by:(6)Lω,u,μ=g1ω+g2u+μ2u−ω−d22where μ>0. As proven in [39, 40], an iteration algorithm which consists of three steps to solve (1) is obtained:
(7)ω=argminωg1ω+μ2u−ω−d22=argminω∑j,k12pj,k−ωj,k2+λjμ+1ϕωj,k;ajwhere p=Wy+μu−d/μ+1
(8)u=argminug2u+μ2u−ω−d22=v+WtvdWTv,β/μ−WTvwhere v=d+ω, tvd is total variation denoising.
(9)d=d−u−ωWe initializeu=Wy and d=0.The detailed solutions to (7) and (8) are given in [39]. The solution of equation (1) can be calculated by (7), (8), (9).


### 2.2. Bilateral Texture Filtering

The Bilateral Texture Filtering (BFT) effectively removes texture while preserving structure. The computation of guidance image via patch shift is the only additional step over the standard bilateral filter. How patch shift works is illustrated in detail in [41]. The BTF can be expressed as follows:
(10)△Ωq=ImaxΩq−IminΩq(11)mRTVΩq=△Ωqmaxr∈Ωq∂Ir∑r∈Ωq∂Ir+εε=10−9(12)αp=211+exp−σαmRTVΩp−mRTVΩq−0.5(13)Gp′=αpGp+1‐αpBpwhere ImaxΩq and IminΩq are the maximum and the minimum image intensities in patch Ωq that is least likely to contain a prominent structure edge; Ωp is the center patch of average intensity for each pixel p; Bp is the average image intensity; Gp represents the texture signal that is obtained by copying Bq at q that has the smallest △Ωq in the neighborhood of p(Bq is the average intensity in Ωq); σα controls the sharpness of the weight transition from edges to smooth/texture regions; Gp′ is the modified guidance image.


### 2.3. Kernel Extreme Learning Machine

KELM is formulated by applying Mercer’s condition to ELM (extreme learning machine) [42]. It has great generalization and stability. KELM is a single-layer feed forward neural network. The hidden layer output matrix H with L hidden neurons is:
(14)H=hw1Tx1+b1hw2Tx1+b2⋯hwLTx1+bL⋮⋮⋮⋮hw1TxN+b1hw2TxN+b2⋯hwLTxN+bLwhere h· is the activation function of hidden layer, W=w1,w2,⋯,wL∈Rd×L and B=b1,b2,⋯,bLT∈RL are the input weights and biases between randomly generated input layers and hidden layers, respectively. X=x1,x2,⋯,xN∈Rd×N denotes a set of samples with label T=t1,t2,⋯,tN∈Rc×N. The classification problem for the proposed constrained-optimization-based ELM with a single output node can be formulated as [42, 43]:
(15)minβ22+C2∑i=1Nξi2subjecttohxiβ=tiT−ξiT,i=1⋯⋯N⇔Hβ=TT−ξTwhere β∈RL×C is the output weight of hidden layer and output layer, ξ=ξ1,ξ2,⋯,ξL is the prediction error matrix of training data, and C is a penalty factor. The output function of the ELM classifier is:
(16)fx=hxβ′=hxHTHTH+IL×LC‐1TThen, the output of KELM is(17)fx=hxβ′=hxHTHTH+IL×LC‐1T=kx,x1⋮kx,xNTΩ+IN×NC‐1Twhere Ω=HHT∈RN×N, Ωi,j=hxihxjT=kxi,xj, and k· is the kernel function.


## 2.1. WATV Denoising

WATV denoising method can suppress pseudo-Gibbs oscillations and spurious noise spikes by estimating all wavelet coefficients simultaneously via the minimization of a single objective function. The method is formulated as an optimization problem incorporating both wavelet sparsity and TV regularization [39]. The wavelet coefficients are calculated by solving the optimization problem shown in formula (1). If ajsatisfies formula (2), F(w) is strictly convex.
(1)ω⌢=argminωFω=12Wy−ω22+∑j,kλjϕωj,k;aj+βDWTω1Wrepresents the wavelet transform; in ωj,k, j and k are the scale and time indices, respectively. λj and βare the regularization parameters. The penalty term DWTω1 is the total variation of the signal estimate x⌢; the function ϕ⋅;aj is a non-convex sparsity-inducing penalty function with
(2)0≤aj<1λjBased on variable splitting and the alternating direction method of multipliers (ADMM), equation (1) can be expressed as a constrained problem:
(3)argminu,ωg1ω+g2usubjecttou=ω(4)g1ω=12Wy−ω22+∑j,kλjϕωj,k;aj(5)g2u=βDWTu1The augmented Lagrangian is given by:(6)Lω,u,μ=g1ω+g2u+μ2u−ω−d22where μ>0. As proven in [39, 40], an iteration algorithm which consists of three steps to solve (1) is obtained:
(7)ω=argminωg1ω+μ2u−ω−d22=argminω∑j,k12pj,k−ωj,k2+λjμ+1ϕωj,k;ajwhere p=Wy+μu−d/μ+1
(8)u=argminug2u+μ2u−ω−d22=v+WtvdWTv,β/μ−WTvwhere v=d+ω, tvd is total variation denoising.
(9)d=d−u−ωWe initializeu=Wy and d=0.The detailed solutions to (7) and (8) are given in [39]. The solution of equation (1) can be calculated by (7), (8), (9).


## 2.2. Bilateral Texture Filtering

The Bilateral Texture Filtering (BFT) effectively removes texture while preserving structure. The computation of guidance image via patch shift is the only additional step over the standard bilateral filter. How patch shift works is illustrated in detail in [41]. The BTF can be expressed as follows:
(10)△Ωq=ImaxΩq−IminΩq(11)mRTVΩq=△Ωqmaxr∈Ωq∂Ir∑r∈Ωq∂Ir+εε=10−9(12)αp=211+exp−σαmRTVΩp−mRTVΩq−0.5(13)Gp′=αpGp+1‐αpBpwhere ImaxΩq and IminΩq are the maximum and the minimum image intensities in patch Ωq that is least likely to contain a prominent structure edge; Ωp is the center patch of average intensity for each pixel p; Bp is the average image intensity; Gp represents the texture signal that is obtained by copying Bq at q that has the smallest △Ωq in the neighborhood of p(Bq is the average intensity in Ωq); σα controls the sharpness of the weight transition from edges to smooth/texture regions; Gp′ is the modified guidance image.


## 2.3. Kernel Extreme Learning Machine

KELM is formulated by applying Mercer’s condition to ELM (extreme learning machine) [42]. It has great generalization and stability. KELM is a single-layer feed forward neural network. The hidden layer output matrix H with L hidden neurons is:
(14)H=hw1Tx1+b1hw2Tx1+b2⋯hwLTx1+bL⋮⋮⋮⋮hw1TxN+b1hw2TxN+b2⋯hwLTxN+bLwhere h· is the activation function of hidden layer, W=w1,w2,⋯,wL∈Rd×L and B=b1,b2,⋯,bLT∈RL are the input weights and biases between randomly generated input layers and hidden layers, respectively. X=x1,x2,⋯,xN∈Rd×N denotes a set of samples with label T=t1,t2,⋯,tN∈Rc×N. The classification problem for the proposed constrained-optimization-based ELM with a single output node can be formulated as [42, 43]:
(15)minβ22+C2∑i=1Nξi2subjecttohxiβ=tiT−ξiT,i=1⋯⋯N⇔Hβ=TT−ξTwhere β∈RL×C is the output weight of hidden layer and output layer, ξ=ξ1,ξ2,⋯,ξL is the prediction error matrix of training data, and C is a penalty factor. The output function of the ELM classifier is:
(16)fx=hxβ′=hxHTHTH+IL×LC‐1TThen, the output of KELM is(17)fx=hxβ′=hxHTHTH+IL×LC‐1T=kx,x1⋮kx,xNTΩ+IN×NC‐1Twhere Ω=HHT∈RN×N, Ωi,j=hxihxjT=kxi,xj, and k· is the kernel function.


## 3. Experiments

In this part, through processing and fusing magnetic signal and infrared image, the classification for six kinds of broken wires is implemented. In this experiment, the number of broken wires is one, two, three, four, five, and seven. Many wires are wound into a strand, then it is wounded into a wire rope. The damage of the wire rope is related to the geometry and winding mode of the wire rope [1, 2, 13]. As shown in Figure 1, the structure of the wire rope is 6∗36 with a diameter of 28 mm. The length of the wire ropes is 6.5 m. The specimens used are 185, where the number of training samples is 139 and testing number is 46. The number of broken wires is from 1 to 5 and 7 wires, where the number of every samples set of broken wires is 30, 30, 32, 34, 35, and 34. The width of samples contains 2 mm, 5 mm, and 1.5 cm. The depth of defects is 1 mm. The type of defect is shown in Figure 2.Figure 1
Wire rope cross-section drawing.Figure 2
Schematic of defect.
(a)
One broken wire
(b)
One broken wire
(c)
Two broken wires
(d)
Three broken wires
(e)
Three broken wires
(f)
Five broken wires
(g)
Seven broken wires
(h)
Four broken wiresTo implement the recognition of broken wires, the processing steps are shown in Figure3.Figure 3
Recognition flow chart.


### 3.1. Data Collection

When there is no defect on wire rope and materials of the wire rope are uniform and identical, the magnetic flux through the cross-section of the wire rope should be equal in the axial direction. If there is a defect, the permeability at the defect becomes smaller, the magnetic field only passes through the air field and then returns to the inside of the wire rope; thus, magnetic leakage on surface is formed [12–14]. According to this principle, a magnetic flux leakage detection device is designed. Data collection contains magnetic signal acquisition and thermal infrared image acquisition. The specific devices and collecting procedures are as follows: the magnetic data acquisition device adopted contains Unsaturated Magnetic Excitation (UME) source, an array of 18 Giant Magnetoresistance (GMR) sensors, data acquisition unit, data storage, and control system [33].As shown in Figure4, data collection steps are as follows: After loading unsaturated magnetic field on wire rope, the weak MFL signal can be obtained through equal-space sampling. As the acquisition system moves along the axial direction of the wire rope, the photoelectric encoder produces the pulses. Then, the control system collects the defect information from 18 channels according to pulses. And the final magnetic data is stored in the SD card.Figure 4
Schematic of magnetic data acquisition device: (a) magnetic signal experiment platform; (b) magnetic leakage detection system for wire rope.
(a)
(b)Because the rate of infrared radiation from defect location is different from that from nondefect location, the damage of wire rope can be detected. Infrared information acquisition system, as shown in Figure5(a), includes heating unit and data collection. The heating unit is composed of the metal tube and tight wires. The metal tube is 40 mm in diameter and 20 cm in length. Wire is adopted to heat the metal tube. Infrared thermography is adopted to capture the images of defect information. The angle of camera should be adjusted according to the location of the defects to maintain the distance between the defect and the camera lens constant. The camera we adopted is thermal imager FLUKE TIX 660. The thermal resolution of the infrared camera is -20°C-1200°C. The distance between wire rope and camera is 35±2 cm. The specific processes are as follows: after the wire is energized, the wire rope temperature rises by heating the metal tube. When the temperature of fault is maintained at about 40±2(°C), the defect images are taken by the infrared camera. Single images are acquired through the device shown in Figure 5(a). After installing the thermal infrared camera on the tripod, the defective part is heated, and the images of wire rope surface defect are obtained by panning the tripod. The focus of the image is formed by centering the defect and fixing the distance between the defect and the camera. The captured raw infrared picture is shown in Figure 5(b). (The defect is marked by a box.)Figure 5
Infrared image acquisition: (a) schematic of infrared data acquisition device; (b) the raw infrared picture of defect; (c) thermal infrared image capture system; (d) testing platform for wire rope.
(a)
(b)
(c)
(d)


### 3.2. Data Processing

The data analysis covers processes for magnetic data and infrared image, which involves signal processing and image processing. Filtering signal noise and damage visualization are the main procedures for magnetic information analysis. For managing infrared images, texture filtering and defect segmentation are adopted.


#### 3.2.1. Magnetic Data Processing

Using the system mentioned in Figure4, raw UME signals can be obtained. As shown in Figure 6. Raw UME signals including incoherent baseline caused by channel imbalance, system noise, and strand wave noise produced by structure of wire rope should be filtered out to obtain pure defects information.Figure 6
Schematic of original data.The algorithm based on WATV is employed to inhibit the noise. The data from each channel need to be processed as follows:(1)
Note thatyii=1~18 is the each channel data. Select yi and apply WATV to it:
(i)
Initializeu=Wy by wavelet transform, d=0, and aj=1/λj
(ii)
Calculatep=Wy+μu−d/μ+1. Update ω according to formulation (7), (8), (9) until reaching the number of iterations (the number is set to 10 to remove baseline drift).
(iii)
Retain the wavelet scale containing defect information and set others to zero
(iv)
Then signalx is reconstructed by inverse wavelet transform: x=WTω
(2)
Get the denoised signalx′ after repeating step 1 for three times to ensure the effect of noise reduction and not excessive noise reduction. Let x′ as the input signal and repeat i and ii. Use wavelet soft threshold on the wavelet coefficients at different scales:
(i)
Select 5 db wavelet to decompose it into eight layers
(ii)
The coefficients of different scales should be quantified by soft threshold
(iii)
Reconstruct the wavelet coefficients
(3)
Take a small strand wave of 460. Judge the signal segment in the whole signal similar to the selected wave and set the similar band to zero (if the amplitude difference of signal with the same length as the select strand wave is less than 5, then set the signal to zero.)
(4)
Remove channels that do not contain defect information
(5)
The noise reduction data can be obtained by superposing the processed signalsThe single channel data processed by the above algorithm is shown in Figure7. Through adopting the above algorithm, the baseline, system noise, and strand wave are removed so that the defects information are clearer. Figure 8 shows the filtered 3D data graph.Figure 7
Single channel data of raw and denoised: (a) raw data of single channel and (b) single channel data after noise reduction.
(a)
(b)Figure 8
Schematic of data after denoised.To visualize the defects, the images of flaws are presented by utilizing gray level normalization, circumferential interpolation, defect location, and segmentation.To eliminate the effect of uneven excitation on wire ropes and convert all the data with a uniform standard, normalization is necessary. Normalization is the basis of data visualization; hence, equation (18) is adopted to stretch the defects between 0–255.
(18)IMNj,q=255×IMj,q700+0.5IMNj,q and IMj,q are the normalized data and raw data, respectively; q is the number of sensor channel and j is the length of signal. The data after normalizing is shown in Figure 9.Figure 9
Schematic of data after normalization.Because circumferential data is acquired from 18 sensor channels, circumferential resolution is much lower than the axial one. The pixel count in circumferential is 18; however, the pixel count in axial is more than ten thousand. Three spline interpolations is employed to enhance the circumferential resolution, which increases the pixel count from 18 to 300. In addition, the procedure contributes to realize the visualization of defect images. The schematic of data after interpolations is shown in Figure10. Then, we obtain gray image of leakage magnetic by converting the double data to unit 8. Figure 11 shows the grayscale image of a wire rope’s leakage magnetic field.Figure 10
Schematic of data after interpolation.Figure 11
Grayscale image of leakage magnetic field.Inspired by [30], the defects are located and segmented using the modulus maximum method. The flaws are divided into images of resolution with 300∗300.


#### 3.2.2. Infrared Image Processing

Infrared images can be obtained using the device as shown in Figure5. To achieve defect segmentation, the following rules is used to convert the color image to a grayscale image:
(19)IMgray=IMR×0.114+IMG×0.229+IMB×0.587where IMgray is the gray picture; IMR, IMG, and IMB denote the red layer, green layer, and blue layer of the color image, respectively. The parameters in equation (19) are selected according to the amount of defect information. The transformed photograph of infrared image with 640∗480 is shown in Figure 12.Figure 12
Grayscale image after transforming.BTF is utilized to implement texture filtering, which can remove the fine grain of wire rope. The specific algorithm is as follows:(1)
ComputeB, the average image of IMgray, by applying k×k box kernel representing a patch
(2)
For each pixelp∈IMgray, calculate mRTV by equations (10) and (11) and the total range △Ωp in formulation (10).
(3)
Acquire the guidance imageG via patch shift on each pixel. Find q∈Ωp with minimum mRTVq, then copyBq to Gp
(4)
Computeα by using equation (12) for each pixel
(5)
ObtainG′=αG+1‐αB
(6)
Obtain filtered imageJ using G′ as guidanceBTF can filter out texture contained in the infrared image of wire rope. The filtered infrared image is shown in Figure13.Figure 13
Image after texture filtered.The image after texture filtering also exists strand wave, which makes trouble for feature extraction. The distances between strand waves are fixed according to the structure of wire rope, and the defects are located between strand waves. Therefore, an algorithm based on distance is proposed to part the damage. The algorithm can be described as follows:(1)
After binarization of imageJ, locate the maximum and minimum values of the row and column with pixel value of 1 in the image, respectively. Then the image B, as shown in Figure 13, is obtained: B=Jlmin:lmax,cmin:cmax (lmin and lmax are the maximum and minimum of line; cmin and cmax are the maximum and minimum of column).
(2)
For each line of imageB, find locbegin and locend:
(20)locbegin=lociifloci=0andloci+1=1locend=lociifloci=1andloci+1=0i=1,2⋯⋯
(3)
Compute the distances for blocks whose pixel value is 1 bylocendi−locbegini+1
(4)
For each line of the imageB, if the distance is between 10 and 70 and the block is larger than 12 (which can avoid the effect of oil pollution), maintain the line or set the line to zero. (The distance of two strands in wire rope is consist and strand wave shown in the image is also consist. Meanwhile, in order to reduce the effect of oil pollution on the segmentation defect, we choose the distance between 10 and 70 and the block larger than 12.)
(5)
Extract the defects of infrared images by finding the locations fromloc that meet (4).The defect, as shown in Figure14, can be extracted from the raw image successfully using the above method. The defect is marked by a box in Figure 14(a), and Figure 14(b) is a magnification of defect.Figure 14
Defect extraction: (a) the image after binarization and (b) defect extracted by the proposed algorithm.
(a)
(b)Through the proposed algorithms, UME signal implements denoising and visualization, and the infrared defects are extracted. Figure15 shows the defects from magnetic and infrared.Figure 15
Image of broken wires (infrared image, magnetic image, and photo of the tested wire part from left to right): (a) one broken wire; (b) two broken wires; (c) three broken wires; (d) four broken wires; (e) five broken wires; (f) seven broken wires.
(a)
(b)
(c)
(d)
(e)
(f)


### 3.3. Data Fusion

In this section, the decision level fusion based on KELM is adopted to implement data fusion. After image preprocessing, appropriate features are extracted. All features are mixed together and uniformly normalized; then, information acquisition, fusion, and decision-making are performed through KELM neural network. The procedures of data fusion are shown in Figure16.Figure 16
Date fusion flow chart.


#### 3.3.1. Feature Extraction

The defect images from UME and infrared have high dimension, which will reduce the speed of classification. Redundancy between features can also be disastrous for networks. Thus, it is necessary to employ proper features to implement recognition. Tan and Zhang [33] had proven that average contrast, third-order moment, conformance, and entropy were more sensitive than other texture features and odd order invariant moments were more sensitive than other moments. Thus, in this experiment, a part of statistical texture features and odd order invariant moments from the magnetic images and the color moments and areas from the infrared images are selected.Texture features involving smoothness, roughness, and regularity are an important method to describe the images, which contain average brightness, standard deviation, third moment, measure (smoothness), conformance, and entropy. Histogram is core in describing image texture, letz denote a random variable of gray level and pzi,i=0,1,2,⋯,L−1 is the histogram, the n-th moment of the mean is:
(21)μn=∑i=0L−1zi−mnpzim is the average brightness:
(22)m=∑i=0L−1zipziThe standard deviation is:(23)σ=μ2z=σ2It reflects the average change in the image [33].The normalized measure is defined as:(24)Rz=1‐11+σ2zwhich is a descriptor of relative smoothness. The third moment is a measure of histogram skewness, which is defined as:
(25)μ3z=∑i=0L−1zi−m3pziit reflects how symmetric the histograms are, and its positive and negative values reflect whether they lean to the left or right.Conformance is defined as:(26)Uz=∑i=0L−1p2ziU has maximum value for all images with the same grayscale. The average entropy is:
(27)ez=−∑i=0L−1pzilog2pziit is a measure of variability and reflects the degree of randomness in the gray-level values. Its value is a variable and the value is zero if the image is constant.Invariant moment groups are based on the statistical analysis of image gray distribution. They are constant for translation, scaling, and rotation, which can describe the features of defects. Given an imagefx,y, which size is M×N, the odd order invariant moments are:
(28)M1=u20+u02M3=u30−3u122+3u21+u032M5=u30−3u12u12+u30u30+u122−3u21+u032+3u21−u03u21+u033u30+u122−u21+u032M7=3u21−u03u30+u21u30+u122−3u21−u032+3u12−u30u21+u033u30+u122+u21+u032where upq is central moments:
(29)upq=∑x∑yx−x¯py−y¯qfx,yx¯ and y¯ are the center of gravity of image: x¯=m10/m00, and y¯=m01/m00; mpq is the p+q order moment, which is defined as:
(30)mpq=∑x∑yxpyqfx,yThe area of the infrared image is defined as the size of the image captured by the proposed algorithm based on distance:(31)A=L×WL is the length of image, W is the width of image.The color moment is the important feature of color images whose color space is not quantified, and the feature vector dimension is low [29]. The three color moments per color component are defined as:
(32)μ=1N∑j=1Npjσ=1N∑j=1Npj−μ21/2s=1N∑j=1Npj−μ31/3pj is the j-th pixel, N is the number of the pixel. We extract three color moments of R, G, and B channels as part of the infrared features.When completing classification via magnetic features only, a part of statistical texture features and odd order invariant moments is adopted. If the magnetic and infrared information are combined to classify the broken wires, we added the color moments and area of infrared images as features.


#### 3.3.2. Fusion Based on KELM

Infrared data is closer to the actual damage pattern than magnetic data and provides more color information; however, different sizes of same broken wires may lead to low accuracy. Magnetic data with the same broken wires has similar visual image. Thus, the combination of the two methods can supply more information for the damage and improve the classification accuracy.The decision level fusion is the highest level of fusion, is directly aimed at specific decision targets, and makes full use of the initial decision of each image [44]. Neural network method is employed to fuse information. After pretreating the data from different sensors and making an initial judgment, the decision of different information should be processed; then, the final result is obtained by decision level fusion [44].The magnetic features and infrared features are put in a matrix; then, the features are normalized to the range of 1 to -1, which can increase the convergence speed of the network and make preparation for data fusion. The label of the sample is a preliminary judgment on the target. The samples with label are the input of KELM. The hidden layer output can be calculated through the input weights and biases in formula (14), and the result is a kernel matrix. The output of KELM can be calculated through formula (16). The weight is obtained by the kernel function. And the data preliminary decision is implemented by feature mapping of neural network, data learning, and weight determination. Then, the fusion is implemented by comparing output of the neural network and target label. The specific classification results are described in Section 3.4.


#### 3.3.3. Research on Sample Size and Fusion Result

BP neural network [28, 29, 32], radial basis function (RBF) algorithm [27], k-nearest neighbor (KNN) [30], and KELM are applied to three models. Table 1 shows the results of fusion for each method under different models. The results in Table 1 are obtained from the average of 20 random experiments.Table 1
Fusion result of several neural network algorithms.
Accuracy
BP
RBF
KNN
KELM
Model 1
96.8%
94.6%
96.7%
97.8%
Model 2
96.8%
95.2%
96.8%
98.4%
Model 3
97.8%
97.8%
95.8%
100.0%In Table1, the parameters of each network are numbers of hidden layer node in BP network is 30, the spread in RBF is 0.01, k in KNN is 6, the penalty coefficient, and kernel parameter in KELM are 100 and 0.01. The number of samples is 185. According to [25, 26, 28], three models are set to research the relationship between the number of training samples and the quality of the result. The number of training sample set corresponding to model 1 is 92. The number of training sample set corresponding to model 2 is 122. The number of training sample set corresponding to model 3 is 139.Through Table1, it is demonstrated that as the number of test samples increases, the quality of the results gets better. When selecting model 2 to present the speed of classifiers, the result of fusion speed is as shown in Table 2. In Table 1, the fusion results of RBF and KELM are better than BP and KNN. Meanwhile, fusion speed of KELM shown in Table 2 is the best. Thus, through analyzing the result in Tables 1 and 2, it is obvious that KELM is reliable. The research for the performance of recognition algorithms and classification results will be presented in the next section.Table 2
Running time for four neural network algorithms (s).
Time (s)
BP
RBF
KNN
KELM
Magnetic data
79.60
0.17
0.02
0.01
Fused data
4.00
0.15
0.03
0.02


### 3.4. Classification Results

In this part, the classification results are presented using different recognition algorithms. KELM has advantages of high running speed and good generalization, we adopted it to implement the recognition of 6 classes of broken wires. In this section, the defects by magnetic information, infrared data, and combination of the magnetic and infrared information are classified, respectively, which proves that the information fusion is more effective. For KELM, the penalty coefficient C and kernel parameterσ are adjusted from the set C=1,100,10000 and σ=0.001,0.01,0.1. The KELM network is trained by a set of 139 randomly selected specimens, and the others are the testing samples.For UME, the average of two error accuracy of 20 random train/test splits with different parameters are reported in Table3. Different parameters will lead to different identification accuracy. When σ=0.1 and C=100, the recognition accuracy rate is the highest; however, the average training accuracy is only 82.3%. The average training accuracy reaches 96.7% and the recognition accuracy is 91.2% when σ=0.01 and C=100. Table 3 presents the average of two error recognition accuracy of 20 randomly generated train/test splits based on the fusion of magnetic and infrared. When the training accuracy is higher than 90%, the highest accuracy is 98.4%.Table 3
Performance of the designed KELM network under different parameter settings.
Accuracy
σ=0.001C=1
σ=0.001C=100
σ=0.01C=1
σ=0.01C=100
σ=0.01C=10,000
σ=0.1C=1
σ=0.1C=100
σ=0.1C=10,000
Magnetic data
87.9%
87.6%
91.2%
91.2%
90.0%
91.4%
92.2%
87.8%
Infrared data
81.1%
81.3%
82.0%
81.7%
81.7%
82.8%
92.7%
82.3%
Fused data
98.0%
97.9%
98.8%
98.4%
98.4%
98.9%
98.9%
98.0%Figure17 shows the absolute error distribution of one group testing result whenσ=0.01 and C=100. The training accuracy of two methods are all higher than 90%. When the magnetic information only exists in the network, the maximum error is 5. When the infrared information only exists in the network, the maximum error is also 5. And the most errors are concentrated in one and two broken wires. However, when the fusion features contain in the network, the maximum error is 2, and the recognition accuracy is higher. It is obvious that there are fewer errors using the fused features than that adopting magnetic features only and infrared only. Therefore, these testing results demonstrate that the fusion of magnetic and infrared not only is feasible but also can improve the recognition accuracy of broken wires.Figure 17
Recognition distribution of error: (a) distribution of error when the magnetic features are only contained; (b) distribution of error when the infrared features are only contained; (c) distribution of error when the magnetic and infrared are fused.
(a)
(b)
(c)Several recognition algorithms are applied to the MFL data: BP neural network [28, 29, 32], RBF algorithm [27], and KNN algorithm [30]. The data for recognition is the same as that used in the KELM network. Tables 4–6 show the recognition results for each method when the limiting error is 2 wires.Table 4
Recognition accuracy for different numbers of hidden layer node in BP network.
Accuracy
N=15
N=17
N=20
N=21
N=24
N=25
N=27
N=30
Magnetic data
82.7%
82.0%
85.4%
84.4%
84.7%
84.6%
86.0%
85.0%
Infrared data
82.3%
82.7%
84.0%
82.2%
84.8%
82.4%
82.3%
82.3%
Fused data
97.0%
96.6%
97.9%
96.6%
97.1%
97.6%
97.4%
97.7%Table 5
Performance of the various RBF classification networks (S is spread).
Accuracy
S=0.008
S=0.009
S=0.01
S=0.03
S=0.05
S=0.1
S=0.12
S=0.15
Magnetic data
87.7%
87.8%
87.8%
91.3%
91.0%
89.6%
88.7%
87.3%
Infrared data
78.2%
78.3%
80.0%
81.3%
82.3%
82.0%
81.4%
81.6%
Fused data
97.2%
97.0%
97.5%
98.9%
98.7%
97.5%
97.3%
96.6%Table 6
Recognition accuracy for different values ofk in KNN algorithm.
Accuracy
K=3
K=6
K=9
K=13
K=17
K=18
K=20
K=23
Magnetic data
86.3%
83.0%
77.7%
78.2%
77.5%
78.0%
76.8%
76.3%
Infrared data
88.4%
90.3%
88.6%
89.0%
89.6%
88.8%
88.0%
87.2%
Fused data
97.2%
98.9%
98.0%
97.9%
97.7%
97.4%
96.8%
96.0%From Tables3–6, it is obvious that the fused data yielded much better classification results than magnetic data. In this paper, neural networks are considered reliable when the training accuracy is higher than 90%. (Note that KNN has no training process and the algorithm finds k samples in the training set closest to the test sample based on a distance measurement [30, 45].) Therefore, the highest recognition rates are presented in Table 7. Table 7 presents the best results of four classifiers using only magnetic data, infrared data, and fused data. It is obvious that the result for fused data is higher than magnetic and infrared.Table 7
Recognition results for four neural network algorithms.
Accuracy
BP
RBF
KNN
KELM
Magnetic data
85.0%
87.8%
86.3%
91.2%
Infrared data
84.8%
82.3%
90.3%
82.3%
Fused data
97.7%
97.5%
98.9%
98.4%


## 3.1. Data Collection

When there is no defect on wire rope and materials of the wire rope are uniform and identical, the magnetic flux through the cross-section of the wire rope should be equal in the axial direction. If there is a defect, the permeability at the defect becomes smaller, the magnetic field only passes through the air field and then returns to the inside of the wire rope; thus, magnetic leakage on surface is formed [12–14]. According to this principle, a magnetic flux leakage detection device is designed. Data collection contains magnetic signal acquisition and thermal infrared image acquisition. The specific devices and collecting procedures are as follows: the magnetic data acquisition device adopted contains Unsaturated Magnetic Excitation (UME) source, an array of 18 Giant Magnetoresistance (GMR) sensors, data acquisition unit, data storage, and control system [33].As shown in Figure4, data collection steps are as follows: After loading unsaturated magnetic field on wire rope, the weak MFL signal can be obtained through equal-space sampling. As the acquisition system moves along the axial direction of the wire rope, the photoelectric encoder produces the pulses. Then, the control system collects the defect information from 18 channels according to pulses. And the final magnetic data is stored in the SD card.Figure 4
Schematic of magnetic data acquisition device: (a) magnetic signal experiment platform; (b) magnetic leakage detection system for wire rope.
(a)
(b)Because the rate of infrared radiation from defect location is different from that from nondefect location, the damage of wire rope can be detected. Infrared information acquisition system, as shown in Figure5(a), includes heating unit and data collection. The heating unit is composed of the metal tube and tight wires. The metal tube is 40 mm in diameter and 20 cm in length. Wire is adopted to heat the metal tube. Infrared thermography is adopted to capture the images of defect information. The angle of camera should be adjusted according to the location of the defects to maintain the distance between the defect and the camera lens constant. The camera we adopted is thermal imager FLUKE TIX 660. The thermal resolution of the infrared camera is -20°C-1200°C. The distance between wire rope and camera is 35±2 cm. The specific processes are as follows: after the wire is energized, the wire rope temperature rises by heating the metal tube. When the temperature of fault is maintained at about 40±2(°C), the defect images are taken by the infrared camera. Single images are acquired through the device shown in Figure 5(a). After installing the thermal infrared camera on the tripod, the defective part is heated, and the images of wire rope surface defect are obtained by panning the tripod. The focus of the image is formed by centering the defect and fixing the distance between the defect and the camera. The captured raw infrared picture is shown in Figure 5(b). (The defect is marked by a box.)Figure 5
Infrared image acquisition: (a) schematic of infrared data acquisition device; (b) the raw infrared picture of defect; (c) thermal infrared image capture system; (d) testing platform for wire rope.
(a)
(b)
(c)
(d)


## 3.2. Data Processing

The data analysis covers processes for magnetic data and infrared image, which involves signal processing and image processing. Filtering signal noise and damage visualization are the main procedures for magnetic information analysis. For managing infrared images, texture filtering and defect segmentation are adopted.


### 3.2.1. Magnetic Data Processing

Using the system mentioned in Figure4, raw UME signals can be obtained. As shown in Figure 6. Raw UME signals including incoherent baseline caused by channel imbalance, system noise, and strand wave noise produced by structure of wire rope should be filtered out to obtain pure defects information.Figure 6
Schematic of original data.The algorithm based on WATV is employed to inhibit the noise. The data from each channel need to be processed as follows:(1)
Note thatyii=1~18 is the each channel data. Select yi and apply WATV to it:
(i)
Initializeu=Wy by wavelet transform, d=0, and aj=1/λj
(ii)
Calculatep=Wy+μu−d/μ+1. Update ω according to formulation (7), (8), (9) until reaching the number of iterations (the number is set to 10 to remove baseline drift).
(iii)
Retain the wavelet scale containing defect information and set others to zero
(iv)
Then signalx is reconstructed by inverse wavelet transform: x=WTω
(2)
Get the denoised signalx′ after repeating step 1 for three times to ensure the effect of noise reduction and not excessive noise reduction. Let x′ as the input signal and repeat i and ii. Use wavelet soft threshold on the wavelet coefficients at different scales:
(i)
Select 5 db wavelet to decompose it into eight layers
(ii)
The coefficients of different scales should be quantified by soft threshold
(iii)
Reconstruct the wavelet coefficients
(3)
Take a small strand wave of 460. Judge the signal segment in the whole signal similar to the selected wave and set the similar band to zero (if the amplitude difference of signal with the same length as the select strand wave is less than 5, then set the signal to zero.)
(4)
Remove channels that do not contain defect information
(5)
The noise reduction data can be obtained by superposing the processed signalsThe single channel data processed by the above algorithm is shown in Figure7. Through adopting the above algorithm, the baseline, system noise, and strand wave are removed so that the defects information are clearer. Figure 8 shows the filtered 3D data graph.Figure 7
Single channel data of raw and denoised: (a) raw data of single channel and (b) single channel data after noise reduction.
(a)
(b)Figure 8
Schematic of data after denoised.To visualize the defects, the images of flaws are presented by utilizing gray level normalization, circumferential interpolation, defect location, and segmentation.To eliminate the effect of uneven excitation on wire ropes and convert all the data with a uniform standard, normalization is necessary. Normalization is the basis of data visualization; hence, equation (18) is adopted to stretch the defects between 0–255.
(18)IMNj,q=255×IMj,q700+0.5IMNj,q and IMj,q are the normalized data and raw data, respectively; q is the number of sensor channel and j is the length of signal. The data after normalizing is shown in Figure 9.Figure 9
Schematic of data after normalization.Because circumferential data is acquired from 18 sensor channels, circumferential resolution is much lower than the axial one. The pixel count in circumferential is 18; however, the pixel count in axial is more than ten thousand. Three spline interpolations is employed to enhance the circumferential resolution, which increases the pixel count from 18 to 300. In addition, the procedure contributes to realize the visualization of defect images. The schematic of data after interpolations is shown in Figure10. Then, we obtain gray image of leakage magnetic by converting the double data to unit 8. Figure 11 shows the grayscale image of a wire rope’s leakage magnetic field.Figure 10
Schematic of data after interpolation.Figure 11
Grayscale image of leakage magnetic field.Inspired by [30], the defects are located and segmented using the modulus maximum method. The flaws are divided into images of resolution with 300∗300.


### 3.2.2. Infrared Image Processing

Infrared images can be obtained using the device as shown in Figure5. To achieve defect segmentation, the following rules is used to convert the color image to a grayscale image:
(19)IMgray=IMR×0.114+IMG×0.229+IMB×0.587where IMgray is the gray picture; IMR, IMG, and IMB denote the red layer, green layer, and blue layer of the color image, respectively. The parameters in equation (19) are selected according to the amount of defect information. The transformed photograph of infrared image with 640∗480 is shown in Figure 12.Figure 12
Grayscale image after transforming.BTF is utilized to implement texture filtering, which can remove the fine grain of wire rope. The specific algorithm is as follows:(1)
ComputeB, the average image of IMgray, by applying k×k box kernel representing a patch
(2)
For each pixelp∈IMgray, calculate mRTV by equations (10) and (11) and the total range △Ωp in formulation (10).
(3)
Acquire the guidance imageG via patch shift on each pixel. Find q∈Ωp with minimum mRTVq, then copyBq to Gp
(4)
Computeα by using equation (12) for each pixel
(5)
ObtainG′=αG+1‐αB
(6)
Obtain filtered imageJ using G′ as guidanceBTF can filter out texture contained in the infrared image of wire rope. The filtered infrared image is shown in Figure13.Figure 13
Image after texture filtered.The image after texture filtering also exists strand wave, which makes trouble for feature extraction. The distances between strand waves are fixed according to the structure of wire rope, and the defects are located between strand waves. Therefore, an algorithm based on distance is proposed to part the damage. The algorithm can be described as follows:(1)
After binarization of imageJ, locate the maximum and minimum values of the row and column with pixel value of 1 in the image, respectively. Then the image B, as shown in Figure 13, is obtained: B=Jlmin:lmax,cmin:cmax (lmin and lmax are the maximum and minimum of line; cmin and cmax are the maximum and minimum of column).
(2)
For each line of imageB, find locbegin and locend:
(20)locbegin=lociifloci=0andloci+1=1locend=lociifloci=1andloci+1=0i=1,2⋯⋯
(3)
Compute the distances for blocks whose pixel value is 1 bylocendi−locbegini+1
(4)
For each line of the imageB, if the distance is between 10 and 70 and the block is larger than 12 (which can avoid the effect of oil pollution), maintain the line or set the line to zero. (The distance of two strands in wire rope is consist and strand wave shown in the image is also consist. Meanwhile, in order to reduce the effect of oil pollution on the segmentation defect, we choose the distance between 10 and 70 and the block larger than 12.)
(5)
Extract the defects of infrared images by finding the locations fromloc that meet (4).The defect, as shown in Figure14, can be extracted from the raw image successfully using the above method. The defect is marked by a box in Figure 14(a), and Figure 14(b) is a magnification of defect.Figure 14
Defect extraction: (a) the image after binarization and (b) defect extracted by the proposed algorithm.
(a)
(b)Through the proposed algorithms, UME signal implements denoising and visualization, and the infrared defects are extracted. Figure15 shows the defects from magnetic and infrared.Figure 15
Image of broken wires (infrared image, magnetic image, and photo of the tested wire part from left to right): (a) one broken wire; (b) two broken wires; (c) three broken wires; (d) four broken wires; (e) five broken wires; (f) seven broken wires.
(a)
(b)
(c)
(d)
(e)
(f)


## 3.2.1. Magnetic Data Processing

Using the system mentioned in Figure4, raw UME signals can be obtained. As shown in Figure 6. Raw UME signals including incoherent baseline caused by channel imbalance, system noise, and strand wave noise produced by structure of wire rope should be filtered out to obtain pure defects information.Figure 6
Schematic of original data.The algorithm based on WATV is employed to inhibit the noise. The data from each channel need to be processed as follows:(1)
Note thatyii=1~18 is the each channel data. Select yi and apply WATV to it:
(i)
Initializeu=Wy by wavelet transform, d=0, and aj=1/λj
(ii)
Calculatep=Wy+μu−d/μ+1. Update ω according to formulation (7), (8), (9) until reaching the number of iterations (the number is set to 10 to remove baseline drift).
(iii)
Retain the wavelet scale containing defect information and set others to zero
(iv)
Then signalx is reconstructed by inverse wavelet transform: x=WTω
(2)
Get the denoised signalx′ after repeating step 1 for three times to ensure the effect of noise reduction and not excessive noise reduction. Let x′ as the input signal and repeat i and ii. Use wavelet soft threshold on the wavelet coefficients at different scales:
(i)
Select 5 db wavelet to decompose it into eight layers
(ii)
The coefficients of different scales should be quantified by soft threshold
(iii)
Reconstruct the wavelet coefficients
(3)
Take a small strand wave of 460. Judge the signal segment in the whole signal similar to the selected wave and set the similar band to zero (if the amplitude difference of signal with the same length as the select strand wave is less than 5, then set the signal to zero.)
(4)
Remove channels that do not contain defect information
(5)
The noise reduction data can be obtained by superposing the processed signalsThe single channel data processed by the above algorithm is shown in Figure7. Through adopting the above algorithm, the baseline, system noise, and strand wave are removed so that the defects information are clearer. Figure 8 shows the filtered 3D data graph.Figure 7
Single channel data of raw and denoised: (a) raw data of single channel and (b) single channel data after noise reduction.
(a)
(b)Figure 8
Schematic of data after denoised.To visualize the defects, the images of flaws are presented by utilizing gray level normalization, circumferential interpolation, defect location, and segmentation.To eliminate the effect of uneven excitation on wire ropes and convert all the data with a uniform standard, normalization is necessary. Normalization is the basis of data visualization; hence, equation (18) is adopted to stretch the defects between 0–255.
(18)IMNj,q=255×IMj,q700+0.5IMNj,q and IMj,q are the normalized data and raw data, respectively; q is the number of sensor channel and j is the length of signal. The data after normalizing is shown in Figure 9.Figure 9
Schematic of data after normalization.Because circumferential data is acquired from 18 sensor channels, circumferential resolution is much lower than the axial one. The pixel count in circumferential is 18; however, the pixel count in axial is more than ten thousand. Three spline interpolations is employed to enhance the circumferential resolution, which increases the pixel count from 18 to 300. In addition, the procedure contributes to realize the visualization of defect images. The schematic of data after interpolations is shown in Figure10. Then, we obtain gray image of leakage magnetic by converting the double data to unit 8. Figure 11 shows the grayscale image of a wire rope’s leakage magnetic field.Figure 10
Schematic of data after interpolation.Figure 11
Grayscale image of leakage magnetic field.Inspired by [30], the defects are located and segmented using the modulus maximum method. The flaws are divided into images of resolution with 300∗300.


## 3.2.2. Infrared Image Processing

Infrared images can be obtained using the device as shown in Figure5. To achieve defect segmentation, the following rules is used to convert the color image to a grayscale image:
(19)IMgray=IMR×0.114+IMG×0.229+IMB×0.587where IMgray is the gray picture; IMR, IMG, and IMB denote the red layer, green layer, and blue layer of the color image, respectively. The parameters in equation (19) are selected according to the amount of defect information. The transformed photograph of infrared image with 640∗480 is shown in Figure 12.Figure 12
Grayscale image after transforming.BTF is utilized to implement texture filtering, which can remove the fine grain of wire rope. The specific algorithm is as follows:(1)
ComputeB, the average image of IMgray, by applying k×k box kernel representing a patch
(2)
For each pixelp∈IMgray, calculate mRTV by equations (10) and (11) and the total range △Ωp in formulation (10).
(3)
Acquire the guidance imageG via patch shift on each pixel. Find q∈Ωp with minimum mRTVq, then copyBq to Gp
(4)
Computeα by using equation (12) for each pixel
(5)
ObtainG′=αG+1‐αB
(6)
Obtain filtered imageJ using G′ as guidanceBTF can filter out texture contained in the infrared image of wire rope. The filtered infrared image is shown in Figure13.Figure 13
Image after texture filtered.The image after texture filtering also exists strand wave, which makes trouble for feature extraction. The distances between strand waves are fixed according to the structure of wire rope, and the defects are located between strand waves. Therefore, an algorithm based on distance is proposed to part the damage. The algorithm can be described as follows:(1)
After binarization of imageJ, locate the maximum and minimum values of the row and column with pixel value of 1 in the image, respectively. Then the image B, as shown in Figure 13, is obtained: B=Jlmin:lmax,cmin:cmax (lmin and lmax are the maximum and minimum of line; cmin and cmax are the maximum and minimum of column).
(2)
For each line of imageB, find locbegin and locend:
(20)locbegin=lociifloci=0andloci+1=1locend=lociifloci=1andloci+1=0i=1,2⋯⋯
(3)
Compute the distances for blocks whose pixel value is 1 bylocendi−locbegini+1
(4)
For each line of the imageB, if the distance is between 10 and 70 and the block is larger than 12 (which can avoid the effect of oil pollution), maintain the line or set the line to zero. (The distance of two strands in wire rope is consist and strand wave shown in the image is also consist. Meanwhile, in order to reduce the effect of oil pollution on the segmentation defect, we choose the distance between 10 and 70 and the block larger than 12.)
(5)
Extract the defects of infrared images by finding the locations fromloc that meet (4).The defect, as shown in Figure14, can be extracted from the raw image successfully using the above method. The defect is marked by a box in Figure 14(a), and Figure 14(b) is a magnification of defect.Figure 14
Defect extraction: (a) the image after binarization and (b) defect extracted by the proposed algorithm.
(a)
(b)Through the proposed algorithms, UME signal implements denoising and visualization, and the infrared defects are extracted. Figure15 shows the defects from magnetic and infrared.Figure 15
Image of broken wires (infrared image, magnetic image, and photo of the tested wire part from left to right): (a) one broken wire; (b) two broken wires; (c) three broken wires; (d) four broken wires; (e) five broken wires; (f) seven broken wires.
(a)
(b)
(c)
(d)
(e)
(f)


## 3.3. Data Fusion

In this section, the decision level fusion based on KELM is adopted to implement data fusion. After image preprocessing, appropriate features are extracted. All features are mixed together and uniformly normalized; then, information acquisition, fusion, and decision-making are performed through KELM neural network. The procedures of data fusion are shown in Figure16.Figure 16
Date fusion flow chart.


### 3.3.1. Feature Extraction

The defect images from UME and infrared have high dimension, which will reduce the speed of classification. Redundancy between features can also be disastrous for networks. Thus, it is necessary to employ proper features to implement recognition. Tan and Zhang [33] had proven that average contrast, third-order moment, conformance, and entropy were more sensitive than other texture features and odd order invariant moments were more sensitive than other moments. Thus, in this experiment, a part of statistical texture features and odd order invariant moments from the magnetic images and the color moments and areas from the infrared images are selected.Texture features involving smoothness, roughness, and regularity are an important method to describe the images, which contain average brightness, standard deviation, third moment, measure (smoothness), conformance, and entropy. Histogram is core in describing image texture, letz denote a random variable of gray level and pzi,i=0,1,2,⋯,L−1 is the histogram, the n-th moment of the mean is:
(21)μn=∑i=0L−1zi−mnpzim is the average brightness:
(22)m=∑i=0L−1zipziThe standard deviation is:(23)σ=μ2z=σ2It reflects the average change in the image [33].The normalized measure is defined as:(24)Rz=1‐11+σ2zwhich is a descriptor of relative smoothness. The third moment is a measure of histogram skewness, which is defined as:
(25)μ3z=∑i=0L−1zi−m3pziit reflects how symmetric the histograms are, and its positive and negative values reflect whether they lean to the left or right.Conformance is defined as:(26)Uz=∑i=0L−1p2ziU has maximum value for all images with the same grayscale. The average entropy is:
(27)ez=−∑i=0L−1pzilog2pziit is a measure of variability and reflects the degree of randomness in the gray-level values. Its value is a variable and the value is zero if the image is constant.Invariant moment groups are based on the statistical analysis of image gray distribution. They are constant for translation, scaling, and rotation, which can describe the features of defects. Given an imagefx,y, which size is M×N, the odd order invariant moments are:
(28)M1=u20+u02M3=u30−3u122+3u21+u032M5=u30−3u12u12+u30u30+u122−3u21+u032+3u21−u03u21+u033u30+u122−u21+u032M7=3u21−u03u30+u21u30+u122−3u21−u032+3u12−u30u21+u033u30+u122+u21+u032where upq is central moments:
(29)upq=∑x∑yx−x¯py−y¯qfx,yx¯ and y¯ are the center of gravity of image: x¯=m10/m00, and y¯=m01/m00; mpq is the p+q order moment, which is defined as:
(30)mpq=∑x∑yxpyqfx,yThe area of the infrared image is defined as the size of the image captured by the proposed algorithm based on distance:(31)A=L×WL is the length of image, W is the width of image.The color moment is the important feature of color images whose color space is not quantified, and the feature vector dimension is low [29]. The three color moments per color component are defined as:
(32)μ=1N∑j=1Npjσ=1N∑j=1Npj−μ21/2s=1N∑j=1Npj−μ31/3pj is the j-th pixel, N is the number of the pixel. We extract three color moments of R, G, and B channels as part of the infrared features.When completing classification via magnetic features only, a part of statistical texture features and odd order invariant moments is adopted. If the magnetic and infrared information are combined to classify the broken wires, we added the color moments and area of infrared images as features.


### 3.3.2. Fusion Based on KELM

Infrared data is closer to the actual damage pattern than magnetic data and provides more color information; however, different sizes of same broken wires may lead to low accuracy. Magnetic data with the same broken wires has similar visual image. Thus, the combination of the two methods can supply more information for the damage and improve the classification accuracy.The decision level fusion is the highest level of fusion, is directly aimed at specific decision targets, and makes full use of the initial decision of each image [44]. Neural network method is employed to fuse information. After pretreating the data from different sensors and making an initial judgment, the decision of different information should be processed; then, the final result is obtained by decision level fusion [44].The magnetic features and infrared features are put in a matrix; then, the features are normalized to the range of 1 to -1, which can increase the convergence speed of the network and make preparation for data fusion. The label of the sample is a preliminary judgment on the target. The samples with label are the input of KELM. The hidden layer output can be calculated through the input weights and biases in formula (14), and the result is a kernel matrix. The output of KELM can be calculated through formula (16). The weight is obtained by the kernel function. And the data preliminary decision is implemented by feature mapping of neural network, data learning, and weight determination. Then, the fusion is implemented by comparing output of the neural network and target label. The specific classification results are described in Section 3.4.


### 3.3.3. Research on Sample Size and Fusion Result

BP neural network [28, 29, 32], radial basis function (RBF) algorithm [27], k-nearest neighbor (KNN) [30], and KELM are applied to three models. Table 1 shows the results of fusion for each method under different models. The results in Table 1 are obtained from the average of 20 random experiments.Table 1
Fusion result of several neural network algorithms.
Accuracy
BP
RBF
KNN
KELM
Model 1
96.8%
94.6%
96.7%
97.8%
Model 2
96.8%
95.2%
96.8%
98.4%
Model 3
97.8%
97.8%
95.8%
100.0%In Table1, the parameters of each network are numbers of hidden layer node in BP network is 30, the spread in RBF is 0.01, k in KNN is 6, the penalty coefficient, and kernel parameter in KELM are 100 and 0.01. The number of samples is 185. According to [25, 26, 28], three models are set to research the relationship between the number of training samples and the quality of the result. The number of training sample set corresponding to model 1 is 92. The number of training sample set corresponding to model 2 is 122. The number of training sample set corresponding to model 3 is 139.Through Table1, it is demonstrated that as the number of test samples increases, the quality of the results gets better. When selecting model 2 to present the speed of classifiers, the result of fusion speed is as shown in Table 2. In Table 1, the fusion results of RBF and KELM are better than BP and KNN. Meanwhile, fusion speed of KELM shown in Table 2 is the best. Thus, through analyzing the result in Tables 1 and 2, it is obvious that KELM is reliable. The research for the performance of recognition algorithms and classification results will be presented in the next section.Table 2
Running time for four neural network algorithms (s).
Time (s)
BP
RBF
KNN
KELM
Magnetic data
79.60
0.17
0.02
0.01
Fused data
4.00
0.15
0.03
0.02


## 3.3.1. Feature Extraction

The defect images from UME and infrared have high dimension, which will reduce the speed of classification. Redundancy between features can also be disastrous for networks. Thus, it is necessary to employ proper features to implement recognition. Tan and Zhang [33] had proven that average contrast, third-order moment, conformance, and entropy were more sensitive than other texture features and odd order invariant moments were more sensitive than other moments. Thus, in this experiment, a part of statistical texture features and odd order invariant moments from the magnetic images and the color moments and areas from the infrared images are selected.Texture features involving smoothness, roughness, and regularity are an important method to describe the images, which contain average brightness, standard deviation, third moment, measure (smoothness), conformance, and entropy. Histogram is core in describing image texture, letz denote a random variable of gray level and pzi,i=0,1,2,⋯,L−1 is the histogram, the n-th moment of the mean is:
(21)μn=∑i=0L−1zi−mnpzim is the average brightness:
(22)m=∑i=0L−1zipziThe standard deviation is:(23)σ=μ2z=σ2It reflects the average change in the image [33].The normalized measure is defined as:(24)Rz=1‐11+σ2zwhich is a descriptor of relative smoothness. The third moment is a measure of histogram skewness, which is defined as:
(25)μ3z=∑i=0L−1zi−m3pziit reflects how symmetric the histograms are, and its positive and negative values reflect whether they lean to the left or right.Conformance is defined as:(26)Uz=∑i=0L−1p2ziU has maximum value for all images with the same grayscale. The average entropy is:
(27)ez=−∑i=0L−1pzilog2pziit is a measure of variability and reflects the degree of randomness in the gray-level values. Its value is a variable and the value is zero if the image is constant.Invariant moment groups are based on the statistical analysis of image gray distribution. They are constant for translation, scaling, and rotation, which can describe the features of defects. Given an imagefx,y, which size is M×N, the odd order invariant moments are:
(28)M1=u20+u02M3=u30−3u122+3u21+u032M5=u30−3u12u12+u30u30+u122−3u21+u032+3u21−u03u21+u033u30+u122−u21+u032M7=3u21−u03u30+u21u30+u122−3u21−u032+3u12−u30u21+u033u30+u122+u21+u032where upq is central moments:
(29)upq=∑x∑yx−x¯py−y¯qfx,yx¯ and y¯ are the center of gravity of image: x¯=m10/m00, and y¯=m01/m00; mpq is the p+q order moment, which is defined as:
(30)mpq=∑x∑yxpyqfx,yThe area of the infrared image is defined as the size of the image captured by the proposed algorithm based on distance:(31)A=L×WL is the length of image, W is the width of image.The color moment is the important feature of color images whose color space is not quantified, and the feature vector dimension is low [29]. The three color moments per color component are defined as:
(32)μ=1N∑j=1Npjσ=1N∑j=1Npj−μ21/2s=1N∑j=1Npj−μ31/3pj is the j-th pixel, N is the number of the pixel. We extract three color moments of R, G, and B channels as part of the infrared features.When completing classification via magnetic features only, a part of statistical texture features and odd order invariant moments is adopted. If the magnetic and infrared information are combined to classify the broken wires, we added the color moments and area of infrared images as features.


## 3.3.2. Fusion Based on KELM

Infrared data is closer to the actual damage pattern than magnetic data and provides more color information; however, different sizes of same broken wires may lead to low accuracy. Magnetic data with the same broken wires has similar visual image. Thus, the combination of the two methods can supply more information for the damage and improve the classification accuracy.The decision level fusion is the highest level of fusion, is directly aimed at specific decision targets, and makes full use of the initial decision of each image [44]. Neural network method is employed to fuse information. After pretreating the data from different sensors and making an initial judgment, the decision of different information should be processed; then, the final result is obtained by decision level fusion [44].The magnetic features and infrared features are put in a matrix; then, the features are normalized to the range of 1 to -1, which can increase the convergence speed of the network and make preparation for data fusion. The label of the sample is a preliminary judgment on the target. The samples with label are the input of KELM. The hidden layer output can be calculated through the input weights and biases in formula (14), and the result is a kernel matrix. The output of KELM can be calculated through formula (16). The weight is obtained by the kernel function. And the data preliminary decision is implemented by feature mapping of neural network, data learning, and weight determination. Then, the fusion is implemented by comparing output of the neural network and target label. The specific classification results are described in Section 3.4.


## 3.3.3. Research on Sample Size and Fusion Result

BP neural network [28, 29, 32], radial basis function (RBF) algorithm [27], k-nearest neighbor (KNN) [30], and KELM are applied to three models. Table 1 shows the results of fusion for each method under different models. The results in Table 1 are obtained from the average of 20 random experiments.Table 1
Fusion result of several neural network algorithms.
Accuracy
BP
RBF
KNN
KELM
Model 1
96.8%
94.6%
96.7%
97.8%
Model 2
96.8%
95.2%
96.8%
98.4%
Model 3
97.8%
97.8%
95.8%
100.0%In Table1, the parameters of each network are numbers of hidden layer node in BP network is 30, the spread in RBF is 0.01, k in KNN is 6, the penalty coefficient, and kernel parameter in KELM are 100 and 0.01. The number of samples is 185. According to [25, 26, 28], three models are set to research the relationship between the number of training samples and the quality of the result. The number of training sample set corresponding to model 1 is 92. The number of training sample set corresponding to model 2 is 122. The number of training sample set corresponding to model 3 is 139.Through Table1, it is demonstrated that as the number of test samples increases, the quality of the results gets better. When selecting model 2 to present the speed of classifiers, the result of fusion speed is as shown in Table 2. In Table 1, the fusion results of RBF and KELM are better than BP and KNN. Meanwhile, fusion speed of KELM shown in Table 2 is the best. Thus, through analyzing the result in Tables 1 and 2, it is obvious that KELM is reliable. The research for the performance of recognition algorithms and classification results will be presented in the next section.Table 2
Running time for four neural network algorithms (s).
Time (s)
BP
RBF
KNN
KELM
Magnetic data
79.60
0.17
0.02
0.01
Fused data
4.00
0.15
0.03
0.02


## 3.4. Classification Results

In this part, the classification results are presented using different recognition algorithms. KELM has advantages of high running speed and good generalization, we adopted it to implement the recognition of 6 classes of broken wires. In this section, the defects by magnetic information, infrared data, and combination of the magnetic and infrared information are classified, respectively, which proves that the information fusion is more effective. For KELM, the penalty coefficient C and kernel parameterσ are adjusted from the set C=1,100,10000 and σ=0.001,0.01,0.1. The KELM network is trained by a set of 139 randomly selected specimens, and the others are the testing samples.For UME, the average of two error accuracy of 20 random train/test splits with different parameters are reported in Table3. Different parameters will lead to different identification accuracy. When σ=0.1 and C=100, the recognition accuracy rate is the highest; however, the average training accuracy is only 82.3%. The average training accuracy reaches 96.7% and the recognition accuracy is 91.2% when σ=0.01 and C=100. Table 3 presents the average of two error recognition accuracy of 20 randomly generated train/test splits based on the fusion of magnetic and infrared. When the training accuracy is higher than 90%, the highest accuracy is 98.4%.Table 3
Performance of the designed KELM network under different parameter settings.
Accuracy
σ=0.001C=1
σ=0.001C=100
σ=0.01C=1
σ=0.01C=100
σ=0.01C=10,000
σ=0.1C=1
σ=0.1C=100
σ=0.1C=10,000
Magnetic data
87.9%
87.6%
91.2%
91.2%
90.0%
91.4%
92.2%
87.8%
Infrared data
81.1%
81.3%
82.0%
81.7%
81.7%
82.8%
92.7%
82.3%
Fused data
98.0%
97.9%
98.8%
98.4%
98.4%
98.9%
98.9%
98.0%Figure17 shows the absolute error distribution of one group testing result whenσ=0.01 and C=100. The training accuracy of two methods are all higher than 90%. When the magnetic information only exists in the network, the maximum error is 5. When the infrared information only exists in the network, the maximum error is also 5. And the most errors are concentrated in one and two broken wires. However, when the fusion features contain in the network, the maximum error is 2, and the recognition accuracy is higher. It is obvious that there are fewer errors using the fused features than that adopting magnetic features only and infrared only. Therefore, these testing results demonstrate that the fusion of magnetic and infrared not only is feasible but also can improve the recognition accuracy of broken wires.Figure 17
Recognition distribution of error: (a) distribution of error when the magnetic features are only contained; (b) distribution of error when the infrared features are only contained; (c) distribution of error when the magnetic and infrared are fused.
(a)
(b)
(c)Several recognition algorithms are applied to the MFL data: BP neural network [28, 29, 32], RBF algorithm [27], and KNN algorithm [30]. The data for recognition is the same as that used in the KELM network. Tables 4–6 show the recognition results for each method when the limiting error is 2 wires.Table 4
Recognition accuracy for different numbers of hidden layer node in BP network.
Accuracy
N=15
N=17
N=20
N=21
N=24
N=25
N=27
N=30
Magnetic data
82.7%
82.0%
85.4%
84.4%
84.7%
84.6%
86.0%
85.0%
Infrared data
82.3%
82.7%
84.0%
82.2%
84.8%
82.4%
82.3%
82.3%
Fused data
97.0%
96.6%
97.9%
96.6%
97.1%
97.6%
97.4%
97.7%Table 5
Performance of the various RBF classification networks (S is spread).
Accuracy
S=0.008
S=0.009
S=0.01
S=0.03
S=0.05
S=0.1
S=0.12
S=0.15
Magnetic data
87.7%
87.8%
87.8%
91.3%
91.0%
89.6%
88.7%
87.3%
Infrared data
78.2%
78.3%
80.0%
81.3%
82.3%
82.0%
81.4%
81.6%
Fused data
97.2%
97.0%
97.5%
98.9%
98.7%
97.5%
97.3%
96.6%Table 6
Recognition accuracy for different values ofk in KNN algorithm.
Accuracy
K=3
K=6
K=9
K=13
K=17
K=18
K=20
K=23
Magnetic data
86.3%
83.0%
77.7%
78.2%
77.5%
78.0%
76.8%
76.3%
Infrared data
88.4%
90.3%
88.6%
89.0%
89.6%
88.8%
88.0%
87.2%
Fused data
97.2%
98.9%
98.0%
97.9%
97.7%
97.4%
96.8%
96.0%From Tables3–6, it is obvious that the fused data yielded much better classification results than magnetic data. In this paper, neural networks are considered reliable when the training accuracy is higher than 90%. (Note that KNN has no training process and the algorithm finds k samples in the training set closest to the test sample based on a distance measurement [30, 45].) Therefore, the highest recognition rates are presented in Table 7. Table 7 presents the best results of four classifiers using only magnetic data, infrared data, and fused data. It is obvious that the result for fused data is higher than magnetic and infrared.Table 7
Recognition results for four neural network algorithms.
Accuracy
BP
RBF
KNN
KELM
Magnetic data
85.0%
87.8%
86.3%
91.2%
Infrared data
84.8%
82.3%
90.3%
82.3%
Fused data
97.7%
97.5%
98.9%
98.4%


## 4. Conclusions and Discussions

In this report, the algorithm based on WATV has been proposed to suppress the noise from UME, which can eliminate the baseline signal and other noises successfully. Then an image processing algorithm based on distance was proposed, through which the damages in infrared images were successfully parted. Then, the sensitive features to fuse magnetic and infrared information were selected. The recognition accuracies for the two ways under different experimental settings were reported. A number of experimental results has clearly demonstrated that the information fusion based on magnetic and infrared outperform magnetic only based classification.The research promotes recognition rate of broken wires and makes contributions to estimating the residual lifetime of wire rope. The two information can overcome the loss of small defects in magnetic signal noise reduction. The system we utilized have good performance facing the defects at the exterior of the rope. However, the thermal infrared acquisition system needs to be perfected to realize the image information acquisition of the whole wire rope. Furthermore, we have not been able to create defects inside of the wire rope. We will simulate the situation when the defect is inside the wire rope through analysis in future work. Meanwhile, efficient noise reduction algorithm is also one of the focuses of future research.



---

*Source: 6419371-2020-03-11.xml*