# A Partial Hierarchical Model for Online Low-Resolution Wear Particle Images Classification

Authors: Xuxu Guo; Rui Tan; Mingyang Yang; Xinrong He; Jia Guo; Suli Fan; Junnan Hu; Taohong Zhang; Aziguli Wulamu
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Wear particle image analysis is an effective method to detect wear condition of mechanical devices. However, the recognition accuracy and recognition efficiency for online wear particle automatic recognition are always mutual restricted because the online wear particle images have almost no texture information and lack clarity. Especially for confusing fatigue wear particles and sliding wear particles, the online recognition is a challenging task. Based on this requirement, a super-resolution reconstruct technique and partial hierarchical convolutional neural network, SR-PHnet, is proposed to classify wear particles in one step. The structure of this network is composed by three modules, one is super-resolution layer module, the second is convolutional neural network classifier module, and the third is support vector machine (SVM) classifier module. The classification result of the second module is partial input to the third module for precision classification of fatigue and sliding particles. In addition, a new feature of radial edge factor (REF) is put forward to target fatigue and sliding wear particles. The test result shows that the new feature has the capability to distinguish fatigue and sliding particles well and time saving. The comparison experiments of the convolution neural network (CNN) method, support vector machine method (SVM) with and without REF feature, and integrated model of back-propagation (BP) and CNN are produced. The comparison results show that the online recognition speed and online recognition rate of the proposed SR-PHnet model in this paper are both improved markedly, especially for fatigue and sliding wear particles.


---

## Body

## 1. Introduction

In the mechanical system, most failures are caused by severe friction or wear. Wear particle monitoring and analysis has been deemed as a powerful technology for machine wear assessment [1, 2]. It is considered that wear particles imply important clues for wear rate and wear mechanism [3]. Many methods [4–6] are used to recognize the features of wear particles. Among these techniques, image analysis is regarded as a very effective method for its easy acquiring and rapid processing speed [7–9].The processing methods are undergoing through two generation approaches. The first generation is denoted by feature engineering [10–13]. Yuan et al. [11] analyzed the boundary of wear particles, extracted a variety of different boundary parameters, and used four different machine learning models to explore the performance of the model. Xu et al. [13] constructed a three-level classification model including K-means and support vector machine (SVM) by extracting the color and geometric features of wear particles. The model can classify different wear particles on different classifiers. Myshkin et al. [14] analyzed the color characteristics of metallic wear debris to recognize different wear debris. Stachowiak et al. [15] statistically analyzed the surface texture parameters of wear particles and designed an automatic classification system based on the surface texture of wear particles. Wang and Wang [16] combined principal component analysis (PCA) and grey relational analysis (CPGA) to identify different wear particles. CPGA is adopted to solve the complicated interrelationships between the specified characteristics of similar wear particles with features of color, shape factor, texture, and so on. However, there are two problems in the methods based on manual feature extraction. Firstly, the selection of wear particle features requires deep insight into both domain-expert knowledge and the learning algorithms. Secondly, feature extraction is not adaptive. When the dataset is transformed, the original features may reduce the recognition rate of the model.With the development of machine learning, the second generation is denoted by deep learning networks [17–23]. Peng and Wang [21] proposed a small sample wear particle recognition model based on CNN. Compared with the general CNN model, this model can identify overlapping particles in the case of a small number of samples and uses one-dimensional convolution to reduce the computational complexity of the model. Peng et al. [22] proposed a hybrid CNN network to automatically classify four different wear particles. The hybrid model uses the idea of transfer learning to initialize the model parameters by using the learning parameters and weights of ImageNet. Aiming at the lack of enough information in two-dimensional wear particle images, Wang et al. [24] proposed a nonparametric recognition model of three-dimensional similar particles by taking fatigue and sliding wear particles as examples. This model can transform 3D image with rich information into 2D image and uses the CNN model to optimize. Peng and Wang [25] used the Inception-v3 network structure to automatically extract the characteristics of wear particles and then used the artificial neural network to classify the types of wear particles, which can realize the classification of overlapping particles. Liu et al. [26] constructed a deep convolutional neural network with Encoder-ASPP-Decoder architecture, which can classify and segment concurrently five different wear particles in ferrography images. However, in these works, the images are high definition ferrography images which have rich texture information. It is not economical and practical by using these kinds of acquisition equipment for online wear particles monitoring and analysis.Online wear particle images acquiring and analysis make wear condition monitoring real time and straightforward by avoiding frequent oil sampling. Optical techniques using LED or laser beam and a charge-coupled device (CCD) sensor to acquire wear particle images are not new [1, 12, 27]. When particles pass through the view of light beam, their morphology can be captured by using the camera. However, the analysis of the online images is remained on the first-generation approach—feature engineering level, that is, because the online wear particle images are of low resolution and no texture information contained. Figure 1 shows the difference of high definition ferrography images (extracted from Ref. [18]) and online wear particle images. The classification of online low-resolution wear particle images by using the deep learning method is still not adequately addressed.Figure 1
(a) High definition ferrography images [18]; (b) online wear particle images.
(a)(b)This paper introduces a partial hierarchical model of deep learning convolution neural network assembling with the feature engineering method, which can process and classify online low-resolution wear particle images in one-stop manner. The proposed network structure is called SR-PHnet. It includes three parts, super-resolution layer, the first-stage classifier module, and the second-stage classifier module. Firstly, the low-resolution images are transformed into super-resolution images by super-resolution image reconstruction in super-resolution layer. Then, the super-resolution images are input to the classification module. The partial hierarchical mechanism is put forward with different classification modules to precision recognize the easy recognition particles (e.g., cutting, sphere, and nonmetallic particles) and difficult recognition particles (e.g., fatigue and sliding particles). The easily confused fatigue and sliding wear particles are identified as one class on the convolution neural network classification module and then recognized separately on the SVM classifier module of the SR-PHnet model. Firstly, convolutional neural network is adopted for classifying four categories (cutting, sphere, nonmetallic, and fatigue or sliding particles); then, fatigue and sliding particle images are inputted into the second classifier for further recognition. A new feature, called radial edge factor (REF), is proposed, which is combined with area, aspect ratio, and roundness to identify fatigue and sliding particles better. The main contributions of this work are as follows:(1)
The super‐resolution layer, the first-stage classifier module and the second-stage classifier module are integrated together to realize unclear online wear particles images in one step.(2)
The partial hierarchical modeling method is effective for easy recognition particles and difficult recognition particles (e.g., fatigue and sliding particles). The former is recognized by convolution neural network, and the latter is classified by the feature engineering method.(3)
Radial edge factor (REF) feature, which can be decoupled from various features, is put forward specially for fatigue and sliding particles.(4)
The mechanism of partial hierarchical modeling determines that the modeling scale and the number of features are smaller but more effective than recent modeling methods for online wear particle classification. It provides higher recognition rate and less computing time.The detailed procedure of modeling is described in Section2. The experiments and discussion are explained in Section 3 and Section 4.


## 2. The Proposed SR-PHnet for Online Low-Resolution Wear Particle Images Classification

The overall structure of the proposed SR-PHnet is shown in Figure2.Figure 2
The network structure of the proposed SR-PHnet model.


### 2.1. Super-Resolution Layers

Since the online collected wear particle image is with low resolution (LR), many details are lost compared with the ferrography wear particle image. When the low-resolution image is directly used as the input of CNN, the prediction accuracy is relatively low. Therefore, we hope to recover some image details from low-resolution wear particle images, and image super-resolution reconstruction [28] is used to achieve this effect. Image super-resolution reconstruction is a technique to reconstruct a high-resolution (HR) image by using one or more low-resolution images based on certain assumptions or prior information [29]. In this paper, the single image super-resolution reconstruction (SISR) [30] technology is used to achieve the goal of image reconstruction. SISR techniques are generally classified into three categories: interpolation-based, reconstruction-based, and sample-based learning. Bicubic interpolation [31] is one of the most common methods, but it cannot reconstruct better image details. When the magnification factor is large, the image obtained by bicubic interpolation will be blurred at the edge. The reconstruction-based method [32] focuses on reconstructing lost high-frequency signals. This method only uses some prior knowledge to regularize the reconstruction process and does not use training ideas to obtain prior information. The learning-based method [33, 34] uses machine learning or deep learning algorithm to learn the mapping relationship between the LR image and the HR image to predict the missing high-frequency information of the image to reconstruct the HR image. The learning-based SISR method is used in this paper to reconstruct the low-resolution wear particle image.As shown in Figure2, three convolutional layers are used to achieve super-resolution reconstruction. Each convolution layer does not change the size of the image but change the number of channels of the image. After passing three convolutional layers, the original image size and channel number are restored. The convolution kernel sizes of the three convolutional layers are 7 × 7, 5 × 5, and 3 × 3, respectively, and the zero-padding required is 3, 2, and 1, respectively. The kernel stride is 1, and ReLU activation function is used for each layer. The output size of each layer is as follows:(1)o=i−k+2ps+1.Here,i is the size of the input image, k is the convolution kernel size of the layer, p is the zero-padding number, and s is the kernel stride.The input of the super-resolution layer is the original low-resolution (LR) image, and the output is the changes required to convert the low-resolution (LR) image into the super-resolution (SR) image, that is, the pixel difference between the SR image and the LR image. Since the SR image needs to match the input size of the convolutional classification layer, we apply the bicubic interpolation method for each LR image to enlarge it into the matched size. Then, through the super-resolution layer, we obtain the difference image and add it to the corresponding LR image to obtain the transformed SR image. The SR imageISR is solved by the following formula:(2)ISR=ILR+sILR,where ILR represents the amplified low-resolution (LR) image through bicubic interpolation and sILR represents the output of the super-resolution layer.The super-resolution layer learns the mapping function from the low-resolution image to the difference between the super-resolution image and the low-resolution image [35]. In order to learn the mapping function, we propose the cost function JSR for the super-resolution layer as follows:(3)JSR=12∑i=1KIHR−ISR2=12∑i=1KIHR−ILR+sILR2,where IHR is the high-resolution form of image I and ISR is the super-resolution reconstruction image obtained after image I passes through the super-resolution layer.The optimization objective of the super-resolution layer is to minimize the cost function valueJSR. We use ImageNet datasets to train the super-resolution layer. The high-resolution images IHR in the ImageNet dataset are first down-sampled to generate the low-resolution imagesILR and then amplified to the original image size to obtain the difference ΔIHR−ILR between it and the original high-resolution image.


### 2.2. The CNN Classifier Module of SR-PHnet Model

In view of the advantage that no need to extract features manually for images classification by the deep learning method [36–39], the proposed SR-PHnet model in this paper uses convolution neural network as the first classifier to recognize wear particles automatically. There contains convolutional layer, pooling layer, and fully connected layer [36] in first-stage classifier. The images after super-resolution layer are inputted to the convolutional layer as shown in Figure 2. The output is the classification of wear particles. Here, we use four categories, cutting, sphere, nonmetallic, and FoS particles (for convenience of expression, FoS is used to as the abbreviation of fatigue or sliding particles). The new portable convolutional neural network ShuffleNetV2 [40, 41] structure is used to classify the wear particles. This structure uses block convolution instead of traditional convolution layer, which greatly accelerates the training and testing process of the network.


#### 2.2.1. Convolutional Layer

Convolution layer is convoluted by multiple convolution kernels. Convolution kernel is generally composed of a three-dimensional weight tensor and a bias. It is assumed that the input tensor isTin∈RW1∗H1∗L1, where RW1∗H1∗L1represents a three-dimension real number tensor of size W1∗H1∗L1. W1,H1, and L1 are width, height, and depth (also called channel number), respectively. The output of convolution layer is also a three-dimensional tensor, which is recorded as Tout∈RW2∗H2∗L2, where RW2∗H2∗L2 represents a three-dimension real number tensor of size W2∗H2∗L2. Then, the convolutional operation is calculated as follows:(4)yi,j,k=fCWk∗X+bk=fC∑l=1LC∑i′=1WC∑j′=1HCWi′,j′,lkxi−1∗s+i′∗s+j′,l+bk,where yi,j,k is the value of Tout at position i,j,k, Wk is the weight tensor of the kth convolution kernel of the convolution layer, bk is the bias vector of the kth convolution kernel of the convolution layer, and Xis the input tensor, for the first convolution layer, X=Tin. WC, HC, and LC are the width, height, and depth of the kth convolution kernel, respectively. The total parameter of the kth convolution kernel isWC∗HC∗LC+l, Wi′,j′,lk is the value of Wk at position i′,j′,l, xi−1∗s+i′∗s+j′,l is the value of X at position i−1∗s+i′,j−1∗s+j′,l, blk is the value of bk at position l, s is the stride of moving the convolution kernel, i.e., it is calculated by adding across s positions, and fC is a function. ReLU [36] operation is adopted, and its calculation is expressed as follows:(5)fCx=max0,x,where x is the output of the upper layer network.


#### 2.2.2. Pooling Layer

The pooling layer is a layer without parameters. The output of general pool layer is also a three-dimensional tensor. Generally, there are maximum pooling layer and average pooling layer. The maximum pooling operation is generally used in the pooling layer. The maximum pooling operation can be described by a mathematical formula as follows:(6)pp,q,k=maxysp∗p−1+1:sp∗p−1+hp,sq∗q−1+1:sq∗q−1+wp,k,where sp is span and hp and wp are width and height of neighborhood, respectively.


#### 2.2.3. Full Connect Layer

The full connection layer is generally composed of weight matrix and offset for vector input. Suppose the input isx=x1,x2,…,xn. The weight matrix of all connected layer is W∈Rm∗m, and the offset is b∈Rm. The output is A=a1,a2,…,am, and they are described as follows:(7)a1=W11∗x1+W12∗x2+⋯+W1nxn,a2=W21∗x1+W22∗x2+⋯+W2nxn,…,am=Wm1∗x1+Wm2∗x2+⋯+Wmnxn.


### 2.3. The SVM Classifier Module of SR-PHnet Model

After the first classification, cutting, sphere, nonmetallic, and FoS (fatigue or sliding) particles are recognized automatically without the guidance of feature manual extraction. The objective of the SVM classifier module of the SR-PHnet model is to recognize fatigue or sliding particles. The partial hierarchical mechanism can reduce the model scale, save computing time, and enhance the recognition accuracy.To improve the recognition accuracy of fatigue and sliding wear particle images, feature extraction is used for guidance of classification. The normal features for recognizing different wear particles are area, aspect ratio, roundness, shape factor, and so on. The recognition rate for all wear particles by only using the feature engineering method is not high. One of the reasons is that the multiple features been coupled together make it hard to distinguish all kinds of wear particles and make the model complicated. Characteristic feature is beneficial for improving recognition rate and reducing model scale. For just aiming at fatigue or sliding particles, it is possible to make features decoupling from the others and reduce the number of features to complete the identifying task. For fatigue or sliding particles, a new feature, radial edge factor (denoted as REF), is put forward in this paper. The schematic diagram is shown in Figure3. The brown dotted circle in Figure 3 is an analytically equivalent circle (AEC) [11] with equal perimeter. It is the ratio of the characteristic radius to the minimum radius of the inscribed circle. It is denoted as follows:(8)REF=CRr.Figure 3
The representation diagram of wear particle feature REF.Here,CR is the characteristic radius of wear particles and r is the minimum radius of the inscribed circle as shown in Figure 3. The characteristic radius CR is calculated as follows:(9)CR=P2π.Here,P is perimeter of wear particle. Eight neighborhood distances are used as the perimeter of wear particles by accumulating edge pixel points. It is slightly different from the actual perimeter drawn as the line, but it has short computing time and little influence on the value.REF can reflect the irregularity of wear particles. When two wear particles have the same size of the inscribed circle, the longer the perimeter of the wear particles, the severe the degree of irregularity. When the perimeters of wear particles are the same, the larger the inscribed circle is, the less irregular the wear particles are. With the help of REF feature, the model can better judge the sliding and fatigue particles. The larger the value of REF feature, the more serious the irregularity of the particle.Combine this featureREF with the other features, areaA, aspect ratioAR, and roundness α, to build the feature engineering model for recognizing fatigue or sliding particles. The feature definitions are listed in Table 1. A is the area from the number of pixels of wear particle image. AR is the ratio of length to width. It is calculated as follows:(10)AR=LW.Table 1
Feature definitions of wear particles.
FeatureDefinitionArea: AThe envelope pixels of wear particle imageAspect ratio: ARThe ratio of length to widthRoundness:αThe similarity between the shape of wear particles and the circleRadial edge factor: REFThe ratio of characteristic radius of wear particle to the minimum radius of inscribed circleHere,L is the length of the smallest circumscribed rectangle surrounding the wear particle and W is the width of the smallest circumscribed rectangle surrounding the wear particle.Roundness of wear particle is the similarity between the shape of wear particles and the circle. It is denoted as follows:(11)α=4Aπ∗P2.Here,Ais particle area and P is perimeter of wear particles.The SVM algorithm [42, 43] is adopted as the feature engineering modeling, and the outputs are two classifications, fatigue and sliding wear particles. For the two-classifier, the kernel function used is the Gaussian kernel function RBF, and the 3-fold cross validation is used to verify the generalization ability of the model.


## 2.1. Super-Resolution Layers

Since the online collected wear particle image is with low resolution (LR), many details are lost compared with the ferrography wear particle image. When the low-resolution image is directly used as the input of CNN, the prediction accuracy is relatively low. Therefore, we hope to recover some image details from low-resolution wear particle images, and image super-resolution reconstruction [28] is used to achieve this effect. Image super-resolution reconstruction is a technique to reconstruct a high-resolution (HR) image by using one or more low-resolution images based on certain assumptions or prior information [29]. In this paper, the single image super-resolution reconstruction (SISR) [30] technology is used to achieve the goal of image reconstruction. SISR techniques are generally classified into three categories: interpolation-based, reconstruction-based, and sample-based learning. Bicubic interpolation [31] is one of the most common methods, but it cannot reconstruct better image details. When the magnification factor is large, the image obtained by bicubic interpolation will be blurred at the edge. The reconstruction-based method [32] focuses on reconstructing lost high-frequency signals. This method only uses some prior knowledge to regularize the reconstruction process and does not use training ideas to obtain prior information. The learning-based method [33, 34] uses machine learning or deep learning algorithm to learn the mapping relationship between the LR image and the HR image to predict the missing high-frequency information of the image to reconstruct the HR image. The learning-based SISR method is used in this paper to reconstruct the low-resolution wear particle image.As shown in Figure2, three convolutional layers are used to achieve super-resolution reconstruction. Each convolution layer does not change the size of the image but change the number of channels of the image. After passing three convolutional layers, the original image size and channel number are restored. The convolution kernel sizes of the three convolutional layers are 7 × 7, 5 × 5, and 3 × 3, respectively, and the zero-padding required is 3, 2, and 1, respectively. The kernel stride is 1, and ReLU activation function is used for each layer. The output size of each layer is as follows:(1)o=i−k+2ps+1.Here,i is the size of the input image, k is the convolution kernel size of the layer, p is the zero-padding number, and s is the kernel stride.The input of the super-resolution layer is the original low-resolution (LR) image, and the output is the changes required to convert the low-resolution (LR) image into the super-resolution (SR) image, that is, the pixel difference between the SR image and the LR image. Since the SR image needs to match the input size of the convolutional classification layer, we apply the bicubic interpolation method for each LR image to enlarge it into the matched size. Then, through the super-resolution layer, we obtain the difference image and add it to the corresponding LR image to obtain the transformed SR image. The SR imageISR is solved by the following formula:(2)ISR=ILR+sILR,where ILR represents the amplified low-resolution (LR) image through bicubic interpolation and sILR represents the output of the super-resolution layer.The super-resolution layer learns the mapping function from the low-resolution image to the difference between the super-resolution image and the low-resolution image [35]. In order to learn the mapping function, we propose the cost function JSR for the super-resolution layer as follows:(3)JSR=12∑i=1KIHR−ISR2=12∑i=1KIHR−ILR+sILR2,where IHR is the high-resolution form of image I and ISR is the super-resolution reconstruction image obtained after image I passes through the super-resolution layer.The optimization objective of the super-resolution layer is to minimize the cost function valueJSR. We use ImageNet datasets to train the super-resolution layer. The high-resolution images IHR in the ImageNet dataset are first down-sampled to generate the low-resolution imagesILR and then amplified to the original image size to obtain the difference ΔIHR−ILR between it and the original high-resolution image.


## 2.2. The CNN Classifier Module of SR-PHnet Model

In view of the advantage that no need to extract features manually for images classification by the deep learning method [36–39], the proposed SR-PHnet model in this paper uses convolution neural network as the first classifier to recognize wear particles automatically. There contains convolutional layer, pooling layer, and fully connected layer [36] in first-stage classifier. The images after super-resolution layer are inputted to the convolutional layer as shown in Figure 2. The output is the classification of wear particles. Here, we use four categories, cutting, sphere, nonmetallic, and FoS particles (for convenience of expression, FoS is used to as the abbreviation of fatigue or sliding particles). The new portable convolutional neural network ShuffleNetV2 [40, 41] structure is used to classify the wear particles. This structure uses block convolution instead of traditional convolution layer, which greatly accelerates the training and testing process of the network.


### 2.2.1. Convolutional Layer

Convolution layer is convoluted by multiple convolution kernels. Convolution kernel is generally composed of a three-dimensional weight tensor and a bias. It is assumed that the input tensor isTin∈RW1∗H1∗L1, where RW1∗H1∗L1represents a three-dimension real number tensor of size W1∗H1∗L1. W1,H1, and L1 are width, height, and depth (also called channel number), respectively. The output of convolution layer is also a three-dimensional tensor, which is recorded as Tout∈RW2∗H2∗L2, where RW2∗H2∗L2 represents a three-dimension real number tensor of size W2∗H2∗L2. Then, the convolutional operation is calculated as follows:(4)yi,j,k=fCWk∗X+bk=fC∑l=1LC∑i′=1WC∑j′=1HCWi′,j′,lkxi−1∗s+i′∗s+j′,l+bk,where yi,j,k is the value of Tout at position i,j,k, Wk is the weight tensor of the kth convolution kernel of the convolution layer, bk is the bias vector of the kth convolution kernel of the convolution layer, and Xis the input tensor, for the first convolution layer, X=Tin. WC, HC, and LC are the width, height, and depth of the kth convolution kernel, respectively. The total parameter of the kth convolution kernel isWC∗HC∗LC+l, Wi′,j′,lk is the value of Wk at position i′,j′,l, xi−1∗s+i′∗s+j′,l is the value of X at position i−1∗s+i′,j−1∗s+j′,l, blk is the value of bk at position l, s is the stride of moving the convolution kernel, i.e., it is calculated by adding across s positions, and fC is a function. ReLU [36] operation is adopted, and its calculation is expressed as follows:(5)fCx=max0,x,where x is the output of the upper layer network.


### 2.2.2. Pooling Layer

The pooling layer is a layer without parameters. The output of general pool layer is also a three-dimensional tensor. Generally, there are maximum pooling layer and average pooling layer. The maximum pooling operation is generally used in the pooling layer. The maximum pooling operation can be described by a mathematical formula as follows:(6)pp,q,k=maxysp∗p−1+1:sp∗p−1+hp,sq∗q−1+1:sq∗q−1+wp,k,where sp is span and hp and wp are width and height of neighborhood, respectively.


### 2.2.3. Full Connect Layer

The full connection layer is generally composed of weight matrix and offset for vector input. Suppose the input isx=x1,x2,…,xn. The weight matrix of all connected layer is W∈Rm∗m, and the offset is b∈Rm. The output is A=a1,a2,…,am, and they are described as follows:(7)a1=W11∗x1+W12∗x2+⋯+W1nxn,a2=W21∗x1+W22∗x2+⋯+W2nxn,…,am=Wm1∗x1+Wm2∗x2+⋯+Wmnxn.


## 2.2.1. Convolutional Layer

Convolution layer is convoluted by multiple convolution kernels. Convolution kernel is generally composed of a three-dimensional weight tensor and a bias. It is assumed that the input tensor isTin∈RW1∗H1∗L1, where RW1∗H1∗L1represents a three-dimension real number tensor of size W1∗H1∗L1. W1,H1, and L1 are width, height, and depth (also called channel number), respectively. The output of convolution layer is also a three-dimensional tensor, which is recorded as Tout∈RW2∗H2∗L2, where RW2∗H2∗L2 represents a three-dimension real number tensor of size W2∗H2∗L2. Then, the convolutional operation is calculated as follows:(4)yi,j,k=fCWk∗X+bk=fC∑l=1LC∑i′=1WC∑j′=1HCWi′,j′,lkxi−1∗s+i′∗s+j′,l+bk,where yi,j,k is the value of Tout at position i,j,k, Wk is the weight tensor of the kth convolution kernel of the convolution layer, bk is the bias vector of the kth convolution kernel of the convolution layer, and Xis the input tensor, for the first convolution layer, X=Tin. WC, HC, and LC are the width, height, and depth of the kth convolution kernel, respectively. The total parameter of the kth convolution kernel isWC∗HC∗LC+l, Wi′,j′,lk is the value of Wk at position i′,j′,l, xi−1∗s+i′∗s+j′,l is the value of X at position i−1∗s+i′,j−1∗s+j′,l, blk is the value of bk at position l, s is the stride of moving the convolution kernel, i.e., it is calculated by adding across s positions, and fC is a function. ReLU [36] operation is adopted, and its calculation is expressed as follows:(5)fCx=max0,x,where x is the output of the upper layer network.


## 2.2.2. Pooling Layer

The pooling layer is a layer without parameters. The output of general pool layer is also a three-dimensional tensor. Generally, there are maximum pooling layer and average pooling layer. The maximum pooling operation is generally used in the pooling layer. The maximum pooling operation can be described by a mathematical formula as follows:(6)pp,q,k=maxysp∗p−1+1:sp∗p−1+hp,sq∗q−1+1:sq∗q−1+wp,k,where sp is span and hp and wp are width and height of neighborhood, respectively.


## 2.2.3. Full Connect Layer

The full connection layer is generally composed of weight matrix and offset for vector input. Suppose the input isx=x1,x2,…,xn. The weight matrix of all connected layer is W∈Rm∗m, and the offset is b∈Rm. The output is A=a1,a2,…,am, and they are described as follows:(7)a1=W11∗x1+W12∗x2+⋯+W1nxn,a2=W21∗x1+W22∗x2+⋯+W2nxn,…,am=Wm1∗x1+Wm2∗x2+⋯+Wmnxn.


## 2.3. The SVM Classifier Module of SR-PHnet Model

After the first classification, cutting, sphere, nonmetallic, and FoS (fatigue or sliding) particles are recognized automatically without the guidance of feature manual extraction. The objective of the SVM classifier module of the SR-PHnet model is to recognize fatigue or sliding particles. The partial hierarchical mechanism can reduce the model scale, save computing time, and enhance the recognition accuracy.To improve the recognition accuracy of fatigue and sliding wear particle images, feature extraction is used for guidance of classification. The normal features for recognizing different wear particles are area, aspect ratio, roundness, shape factor, and so on. The recognition rate for all wear particles by only using the feature engineering method is not high. One of the reasons is that the multiple features been coupled together make it hard to distinguish all kinds of wear particles and make the model complicated. Characteristic feature is beneficial for improving recognition rate and reducing model scale. For just aiming at fatigue or sliding particles, it is possible to make features decoupling from the others and reduce the number of features to complete the identifying task. For fatigue or sliding particles, a new feature, radial edge factor (denoted as REF), is put forward in this paper. The schematic diagram is shown in Figure3. The brown dotted circle in Figure 3 is an analytically equivalent circle (AEC) [11] with equal perimeter. It is the ratio of the characteristic radius to the minimum radius of the inscribed circle. It is denoted as follows:(8)REF=CRr.Figure 3
The representation diagram of wear particle feature REF.Here,CR is the characteristic radius of wear particles and r is the minimum radius of the inscribed circle as shown in Figure 3. The characteristic radius CR is calculated as follows:(9)CR=P2π.Here,P is perimeter of wear particle. Eight neighborhood distances are used as the perimeter of wear particles by accumulating edge pixel points. It is slightly different from the actual perimeter drawn as the line, but it has short computing time and little influence on the value.REF can reflect the irregularity of wear particles. When two wear particles have the same size of the inscribed circle, the longer the perimeter of the wear particles, the severe the degree of irregularity. When the perimeters of wear particles are the same, the larger the inscribed circle is, the less irregular the wear particles are. With the help of REF feature, the model can better judge the sliding and fatigue particles. The larger the value of REF feature, the more serious the irregularity of the particle.Combine this featureREF with the other features, areaA, aspect ratioAR, and roundness α, to build the feature engineering model for recognizing fatigue or sliding particles. The feature definitions are listed in Table 1. A is the area from the number of pixels of wear particle image. AR is the ratio of length to width. It is calculated as follows:(10)AR=LW.Table 1
Feature definitions of wear particles.
FeatureDefinitionArea: AThe envelope pixels of wear particle imageAspect ratio: ARThe ratio of length to widthRoundness:αThe similarity between the shape of wear particles and the circleRadial edge factor: REFThe ratio of characteristic radius of wear particle to the minimum radius of inscribed circleHere,L is the length of the smallest circumscribed rectangle surrounding the wear particle and W is the width of the smallest circumscribed rectangle surrounding the wear particle.Roundness of wear particle is the similarity between the shape of wear particles and the circle. It is denoted as follows:(11)α=4Aπ∗P2.Here,Ais particle area and P is perimeter of wear particles.The SVM algorithm [42, 43] is adopted as the feature engineering modeling, and the outputs are two classifications, fatigue and sliding wear particles. For the two-classifier, the kernel function used is the Gaussian kernel function RBF, and the 3-fold cross validation is used to verify the generalization ability of the model.


## 3. Experiments

### 3.1. Dataset

Online images acquisition is carried out with the CCD camera. The typical labeled images are shown in Figure4. The collected image dataset is divided into two parts. One is for offline training to develop the SR-PHnet model, and the other is as test dataset for online recognition and classification. 1408 samples are collected, including 281 nonmetallic particles, 486 sliding particles, 439 cutting particles, 459 fatigue particles, and 95 sphere particles.Figure 4
Typical wear particle images acquired in experiments: (a) nonmetallic; (b) sliding; (c) fatigue; (d) cutting; (e) sphere.
(a)(b)(c)(d)(e)


### 3.2. SR-PHnet

For super-resolution layer, the convolution kernel sizes of the three convolutional layers are 7 × 7, 5 × 5, and 3 × 3, respectively, and the zero-padding required is 3, 2, and 1, respectively. The kernel stride is 1. After super-resolution reconstruction, the images are transformed to the resolution of 224 × 224 × 3 and are input into the CNN classifier module of the SR-PHnet model for training. The structure settings of the super-resolution layer and the convolution neural network classifier are listed in Table2. For the convolution neural network classifier, in the training, the batch size is 64, the learning rate is 0.0004, and the iteration number is 2000. The output of the CNN classifier includes four categories, i.e., cutting, sphere, nonmetallic, and fatigue or sliding (FoS) particles. The classification result is listed in Table 3. It can be seen that the recognition accuracy of cutting, sphere, and nonmetallic particles is high. Even for FoS particles, it is high to 92.5%, that is, because the automatic feature extraction in the SR-PHnet model is suitable for distinctive wear particle images.Table 2
The structure setting of all unit layers of super-resolution layer and the CNN classifier of the SR-PHnet model.
StructureLayerOutput sizeKernel sizeStrideRepeatOutput channelsSuper resolutionConv17 × 71Conv25 × 51Conv33 × 31SR image224 × 2243CNN classifierConv1112 × 1123 × 32124MaxPool56 × 563 × 32Block228 × 282111628 × 2813Block314 × 142123214 × 1417Block47 × 7214647 × 713Conv57 × 71 × 1111024GlobalPool1 × 17 × 7Table 3
The four-classification result of wear particles in the CNN classifier of SR-PHnet.
TotalCorrectRecognition rate (%)Cutting948894Sphere1515100Nonmetallic575495FoS18617292.5After CNN classifier module of the SR-PHnet model, fatigue or sliding wear particles are transferred to the SVM classifier module of the SR-PHnet model. The SVM classifier distinguishes FoS particles with four features. There are 941 samples in use, including 483 sliding grains and 458 fatigue grains. Using 3-folded cross validation, 142 sliding wear particles and 141 fatigue wear particles were randomly selected as test set and 341 sliding wear particles and 317 fatigue wear particles as training set. The recognition accuracy is listed in Table4. It shows that just four features, including the new feature REF proposed in this paper, have a good performance for distinguishing fatigue or sliding wear particles. The recognition accuracy of fatigue particles and sliding particles is up to 92.9% and 91.5%, respectively.Table 4
The classification result of FoS wear particles in the SVM classifier of the SR-PHnet model.
TotalCorrectRecognition rate (%)Fatigue14113192.9Sliding14213091.5Finally, the recognition accuracy of the SR-PHnet model for all the five types of wear particles is listed in Table5. From the recognition result, the average recognition accuracy is up to 92.7%, and the accuracy for fatigue and sliding particles of online recognition is 85.9% and 84.6%, respectively. It has great progress in online wear particles recognition. The recognition accuracy is just 85.7% and 80% for fatigue and sliding particles in the offline model of Wang et al.’s [23] with high definition ferrography images. Our model is preferable for its lightweight computing and high recognition accuracy on low-resolution images.Table 5
The five classification results of wear particles with the SR-PHnet model.
TotalCorrectRecognition rate (%)Cutting948894Sphere1515100Nonmetallic575495Sliding986584.6Fatigue887485.9


## 3.1. Dataset

Online images acquisition is carried out with the CCD camera. The typical labeled images are shown in Figure4. The collected image dataset is divided into two parts. One is for offline training to develop the SR-PHnet model, and the other is as test dataset for online recognition and classification. 1408 samples are collected, including 281 nonmetallic particles, 486 sliding particles, 439 cutting particles, 459 fatigue particles, and 95 sphere particles.Figure 4
Typical wear particle images acquired in experiments: (a) nonmetallic; (b) sliding; (c) fatigue; (d) cutting; (e) sphere.
(a)(b)(c)(d)(e)


## 3.2. SR-PHnet

For super-resolution layer, the convolution kernel sizes of the three convolutional layers are 7 × 7, 5 × 5, and 3 × 3, respectively, and the zero-padding required is 3, 2, and 1, respectively. The kernel stride is 1. After super-resolution reconstruction, the images are transformed to the resolution of 224 × 224 × 3 and are input into the CNN classifier module of the SR-PHnet model for training. The structure settings of the super-resolution layer and the convolution neural network classifier are listed in Table2. For the convolution neural network classifier, in the training, the batch size is 64, the learning rate is 0.0004, and the iteration number is 2000. The output of the CNN classifier includes four categories, i.e., cutting, sphere, nonmetallic, and fatigue or sliding (FoS) particles. The classification result is listed in Table 3. It can be seen that the recognition accuracy of cutting, sphere, and nonmetallic particles is high. Even for FoS particles, it is high to 92.5%, that is, because the automatic feature extraction in the SR-PHnet model is suitable for distinctive wear particle images.Table 2
The structure setting of all unit layers of super-resolution layer and the CNN classifier of the SR-PHnet model.
StructureLayerOutput sizeKernel sizeStrideRepeatOutput channelsSuper resolutionConv17 × 71Conv25 × 51Conv33 × 31SR image224 × 2243CNN classifierConv1112 × 1123 × 32124MaxPool56 × 563 × 32Block228 × 282111628 × 2813Block314 × 142123214 × 1417Block47 × 7214647 × 713Conv57 × 71 × 1111024GlobalPool1 × 17 × 7Table 3
The four-classification result of wear particles in the CNN classifier of SR-PHnet.
TotalCorrectRecognition rate (%)Cutting948894Sphere1515100Nonmetallic575495FoS18617292.5After CNN classifier module of the SR-PHnet model, fatigue or sliding wear particles are transferred to the SVM classifier module of the SR-PHnet model. The SVM classifier distinguishes FoS particles with four features. There are 941 samples in use, including 483 sliding grains and 458 fatigue grains. Using 3-folded cross validation, 142 sliding wear particles and 141 fatigue wear particles were randomly selected as test set and 341 sliding wear particles and 317 fatigue wear particles as training set. The recognition accuracy is listed in Table4. It shows that just four features, including the new feature REF proposed in this paper, have a good performance for distinguishing fatigue or sliding wear particles. The recognition accuracy of fatigue particles and sliding particles is up to 92.9% and 91.5%, respectively.Table 4
The classification result of FoS wear particles in the SVM classifier of the SR-PHnet model.
TotalCorrectRecognition rate (%)Fatigue14113192.9Sliding14213091.5Finally, the recognition accuracy of the SR-PHnet model for all the five types of wear particles is listed in Table5. From the recognition result, the average recognition accuracy is up to 92.7%, and the accuracy for fatigue and sliding particles of online recognition is 85.9% and 84.6%, respectively. It has great progress in online wear particles recognition. The recognition accuracy is just 85.7% and 80% for fatigue and sliding particles in the offline model of Wang et al.’s [23] with high definition ferrography images. Our model is preferable for its lightweight computing and high recognition accuracy on low-resolution images.Table 5
The five classification results of wear particles with the SR-PHnet model.
TotalCorrectRecognition rate (%)Cutting948894Sphere1515100Nonmetallic575495Sliding986584.6Fatigue887485.9


## 4. Discussion

For distinguishable wear particles with distinctive features (e.g., cutting, sphere, and nonmetallic wear particles), the convolutional neural network method is preferred. Because it recognizes particles directly without feature engineering, it has high recognition accuracy and short computing time. For specific particles that are difficult to identify (e.g., fatigue and sliding wear particles), feature engineering is a good choice for recognition. So, the partial hierarchic method is the optimal combination by using convolutional neural network as the first classifier module and feature engineering method SVM as the second classifier module.To compare the recognition performance, the method of the baseline standard CNN model (AlexNet, ResNet50, and ShuffleNetV2), SVM without REF feature, SVM with REF feature, and our SR-PHnet model proposed in this paper is executed, respectively. The comparison results with AlexNet, ResNet50, and ShuffleNetV2 are listed in Table6. The comparison of recognition precision with AlexNet, SVM without REF feature, SVM with REF feature, and our SR-PHnet is shown in Figure 5. Not only the total sample recognition precision but the fatigue and sliding particles recognition precision, the method of our SR-PHnet model has the best performance as shown in Figure 5. The SVM algorithm has better performance than the AlexNet model for sliding and fatigue particles, that is, because feature extraction manually can customize the feature parameters and it is better than feature extraction automatically for confusing recognition objects. SVM with the REF feature method proposed in this paper has better performance on recognizing FoS particles than SVM without the REF feature. It shows that REF feature has special good performance in recognizing FoS particles than the other normal features.Table 6
The comparison results between SR-PHnet, AlexNet, ResNet50, and ShuffleNetV2.
ModelClassTPFPFNPrecisionRecallF1-scoreOursNonmetallic55620.9020.9650.932Sliding7818200.8130.7960.804Cutting800141.0000.8510.920Fatigue821860.8200.9320.872Bubble15001.0001.0001.000Total sample31042420.8810.8810.881AlexNetNonmetallic51860.8640.8950.879Sliding6928290.7110.7040.708Cutting764180.9500.8090.874Fatigue6930190.6970.7840.738Bubble15200.8821.0000.938Total sample28072720.7950.7950.795ResNet50Nonmetallic54640.9000.9310.915Sliding7520230.7890.7650.777Cutting811130.9880.8620.920Fatigue7821100.7880.8860.834Bubble15200.8821.0000.938Total sample30350500.8580.8580.858ShuffleNetV2Nonmetallic54730.8850.9470.915Sliding7323250.7600.7450.753Cutting793150.9630.8400.898Fatigue7522130.7730.8520.811Bubble15100.9381.0000.968Total sample29656560.8410.8410.841Figure 5
The recognition rate of AlexNet, SVM (no REF), SVM (REF), and the proposed SR-PHnet model.The SR-PHnet model proposed in this paper combines the respective advantages and avoids the disadvantages of different modeling methods for feature extraction automatically and manually. Regardless of sliding and fatigue particles recognition or the average recognition accuracy, SR-PHnet has the best performance.The comparison of recognition accuracy with iteration times of our SR-PHnet model and AlexNet is shown in Figure6. It shows that the SR-PHnet model needs fewer iteration times to achieve high recognition accuracy and convergence to stability.Figure 6
The recognition rate of the hierarchical model with iteration times.To compare our proposed model with the other method, the same dataset of this paper is executed by the model of Ref. [23] in the comparison experiment. The BP-CNN model in Ref. [23] is constructed. Due to the lack of texture information, the feature parameters adopted in the BP-CNN model are set as the same as the SR-PHnet model. The comparison result is listed in Table 7.Table 7
Comparison result between BP-CNN (Ref. [23]]) and SR-PHnet.
Wear particle typeBP-CNN (%)SR-PHnet (%)Cutting8994Sphere86.05100Nonmetallic90.9195Fatigue82.6184.6Sliding80.2185.9Comparing between the BP-CNN in Ref. [23] and SR-PHnet model, the proposed SR-PHnet model has a much higher recognition accuracy than BP-CNN in the three wear particles: cutting, spherical, and nonmetal. The recognition rate of fatigue and sliding particles is also improved in the SR-PHnet model, which shows the effectiveness of the SR-PHnet model. It can be seen that the SR-PHnet model is more adaptable. Because CNN is the first classifier to automatically extract image features, it is more adaptive than manual feature extraction. In the method of BP-CNN, BP neural network is as the first classifier which needs manual feature extraction. It limits the recognition accuracy due to the error of manual feature extraction in the first step. The SR-PHnet model has higher accuracy for not only cutting, spherical, and nonmetal but also for fatigue and sliding particles.


## 5. Conclusions

This work aims to develop a super-resolution reconstruction partial hierarchical SR-PHnet model for online wear particle recognition and classification. The low-resolution online wear particle images lacking of texture information are firstly reconstructed by super-resolution technique, and then the super-resolution images are input to the classifier of the SR-PHnet model. The types of wear particles are merged into four categories (cutting, sphere, nonmetallic, and fatigue or sliding (FoS) particles) on the first state classifier, and then fatigue and sliding (FoS) wear particle images are transferred to the second-stage classifier. On the first stage, there are characteristic patterns existing in the four categories, so convolution neural network is the best classifier for its high recognition precision and short computing time. The recognition accuracy is up to 94%. On the second stage, the feature engineering method is the best choice for recognizing the easily confused fatigue and sliding wear particles. A new feature REF is put forward to combine with the other three features to obtain a high recognition rate. The final recognition accuracy of fatigue and sliding particles after the SR-PHnet model is up to 85.9% and 84.6%, respectively. The comparison experiment results confirm the effectiveness of the SR-PHnet model in online wear particles recognition and classification for low-resolution wear particle images lacking of texture information.



---

*Source: 6630247-2021-04-14.xml*