# Confirmation of Thermal Images and Vibration Signals for Intelligent Machine Fault Diagnostics

Authors: Achmad Widodo; Djoeli Satrijo; Toni Prahasto; Gang-Min Lim; Byeong-Keun Choi
Journal: International Journal of Rotating Machinery
 (2012)

---

## Abstract

This paper deals with the maintenance technique for industrial machinery using the artificial neural network so-called self-organizing map (SOM). The aim of this work is to develop intelligent maintenance system for machinery based on an alternative way, namely, thermal images instead of vibration signals. SOM is selected due to its simplicity and is categorized as an unsupervised algorithm. Following the SOM training, machine fault diagnostics is performed by using the pattern recognition technique of machine conditions. The data used in this work are thermal images and vibration signals, which were acquired from machine fault simulator (MFS). It is a reliable tool and is able to simulate several conditions of faulty machine such as unbalance, misalignment, looseness, and rolling element bearing faults (outer race, inner race, ball, and cage defects). Data acquisition were conducted simultaneously by infrared thermography camera and vibration sensors installed in the MFS. The experimental data are presented as thermal image and vibration signal in the time domain. Feature extraction was carried out to obtain salient features  sensitive to machine conditions from thermal images and vibration signals. These features are then used to train the SOM for intelligent machine diagnostics process. The results show that SOM can perform intelligent fault diagnostics with plausible accuracies.


---

## Body

## 1. Introduction

Machine condition monitoring and fault diagnostics are very important for machineries in the industry to guarantee high performance of their functionality. High reliability is needed to reduce product and profit loss because of unwanted downtime. Therefore, machine condition monitoring and fault diagnostics become a critical issue in maintenance activity to ensure availability, minimize operator hazard, and reduce economic losses [1]. For this purpose, several sensors such as vibration sensors, acoustic emission sensors, and temperature sensors are used to fulfill the requirement for machine condition monitoring procedure [2]. The assessment of machine components of critical rotating machines, such as bearings (journal or rolling element), using such sensors is significant for early detection of machine condition.The application of vibration sensors as fundamental tools for machine condition monitoring has been used extensively over a period of approximately four decades [3]. The reason to use these sensors was their effectiveness of measurement process and data analysis that can represent the machine conditions. Vibration signal monitoring of installed bearings in rotating machines is still effective to obtain the machine’s behavior. Such monitoring could extract the machine conditions through signals analysis in time and frequency domains. Machine fault diagnostics technique using vibration sensors has been reported by many researchers including application of intelligent system in their proposed method [4–7].Machinery monitoring and diagnostics by means of acoustic emission (AE) sensor have also been conducted by experts with their own proposed techniques [8–10]. Their works were mostly applied to slow speed bearings monitoring and fault diagnostics. According to published results, AE sensor was more sensitive than vibration sensor in monitoring low speed bearings. The use of AE sensor, along with intelligent technique, that is, relevance vector machine (RVM), for help in machine diagnostics process using such sensor was reported by Widodo et al. [11]. Another research in this area was reported by El-Ghamry et al. [12] in which he used the automated pattern-recognition procedure to perform fault diagnostics of reciprocating machines by AE sensor.The use of temperature monitoring for journal bearings has been reported by Glavatskih et al. [13] and Glavatskih [14]. Their work did not employ any intelligent systems for monitoring and diagnostics. Temperature monitoring by means of thermocouple sensor is usually used as a secondary sensors instead of vibration and AE sensor. Therefore, the reports which employ intelligent diagnostics using such sensor are rarely found.Another method that is recently popular in machine condition monitoring and fault diagnostics is the thermography technique. The good performance and simplicity to use have made this technique highly popular in the engineering maintenance area. This technique has been used in wide range area such as the mechanical, electrical, petrochemical, material, medical, and structure and building areas, and so forth [15]. In the case of machine condition monitoring and diagnostics, the use of such method has been reported by Mazioud et al. [16] in which fault detection of rolling element bearing degradation using infrared thermography was performed. The implementation of intelligent system use thermography has been proposed by Fidali [17]. He proposed a continuous monitoring using thermography camera and fault-diagnostics based statistical texture analysis. His work employed Fisher criterion and minimization of error classification for diagnostics procedure. Younus et.al. [18] reported evaluation on thermal data image for machine diagnostics. Image features are extracted by principal component and independent component analysis. Their proposed system was augmented by support vector machine (SVM) for diagnostics process. A recent publication by Younus and Yang [19] implemented feature extraction of images by two-dimensional discrete wavelet transform. The intelligent system used the SVM for machine diagnostics.In this paper, confirmation of thermal images and vibration signals for intelligent machine fault diagnostics is studied. The aim of this work is to find a new alternative source feature and a machine diagnostics technique by utilizing thermal images and SOM. Thermal images are processed by the thresholding method for image segmentation and clustered by means ofk-means algorithm. Salient features are then extracted from binary images after images processing. The extracted features used for SOM training are area, perimeter, and central moment of region of interest (ROI). After SOM diagnostics, the result will be compared with the vibration-based method for validation purposes.


## 2. Basic Theory

### 2.1. Thermal Image

Infrared camera does not measure directly a temperature of surface. The detectors of the camera are sensitive to the luminance emitted by the object being studied. This luminance is naturally dependent on the temperature of the surface. Thermal image is an image which is captured by infrared thermography camera. This device can capture electromagnetic spectrum within infrared bands, 0.78–1000μm. A thermal image is a function of radiated energy on an inspected object [15]. This image can be digitized and represented in matrix form for computational processing as follows:
(1)f(x,y)=[f(0,0)f(0,1)⋯f(0,N-1)f(1,0)f(1,1)⋯f(1,N-1)⋮⋮⋮⋮f(M-1,0)f(M-1,1)⋯f(M-1,N-1)].


### 2.2. Image Thresholding

Image thresholding is mostly used for image segmentation [21, 22]. Thresholding is also common method for defect detection of material by IRT image. Thresholding is a process to separate objects from its background in a digital image. Histogram is the main tool in this separation process. Suppose that the grey level corresponds to an image f(x,y) that is composed of lights objects on a dark background, in such a way that object and background pixels have grey levels grouped into two dominant modes. Extracting the object from the background is performed by selecting a threshold T that separates these modes. A thresholded image g(x,y) is defined as
(2)g(x,y)={1iff(x,y)>T,0iff(x,y)≤T.7
The method of global threshold is selected in this paper where the grey-level histogram is calculated from the entire image [22].


### 2.3. Feature Extraction

#### 2.3.1. Image Features

The perimeter and area are meaningful for binary image. Consider a discrete binary image containing one more object, wheref(j,k)=1 if a pixel is part of the object and f(j,k)=0 for all nonobject or background pixels. The bit-quads [23] method is adopted to determine the perimeter and area of ROI. The moment of order (p+q) of an image [24], denoted as mpq, is defined as
(3)mpq=∑x∑yxpyqg(x,y),
where x and y are coordinates of the pixel, and g(x,y) is the grey level of the pixel. The higher order moments represent spatial distribution of the objects. The previous two characteristics make moments a reasonable and proper feature for the classification task. However, the above equation may be greatly affected by the objects shifting. Therefore, to overcome this problem central moment is used and defined as
(4)μpq=∑x∑y(x-x-)p(y-y-)qg(x,y),
where x-=m10/m00 and y-=m01/m00.The relationship between moments and central moments up to order 3 are listed as follows:(5)μ00=m00μ11=m11-y-m10,μ10=0μ30=m30-3x-m20+2x-2m10,μ01=0μ12=m12-2y-m11-x-m02+2y-2m10,μ20=m20-x-m10μ21=m21-2x-m11-y-m02+2x-2m01,μ02=m02-y-m01μ03=m03-3y-m02+2y-2m01.
Then, central moments could be further normalized by the following formulas:
(6)ηpq=μpqμ00γ,
where
(7)γ=1+(p+q)2,forp+q=2,3,…


#### 2.3.2. Vibration Features

Statistical are extracted from time domain vibration data using formulas presented in (8) and (9) as follows
(8)μ=1N∑n=1NXn,RMS=1N∑n=1NXn2,skew=1/N∑i=1N(Xi-μ)3σ3,kurt=1/N∑i=1NXi4σ3.
where,
(9)σ=∑i=1N1N(Xi-μ)2,CF=XmaxRMS,SF=RMSμ,


### 2.4. Self Organizing Map (SOM)

The SOM is a neural network model that implements a characteristic non-linear projection from a high dimensional space of sensory or other input signals onto a low-dimensional array of neurons. The SOM is an unsupervised learning algorithm which produces a map pattern features on its output layer [20]. Input patterns with similar features are mapped onto neighbouring output nodes. The network consists of an input layer with m neurons and an output layer with n neurons as shown in Figure 1.Figure 1
SOM network [20].The image of the signal space tends to manifest the clusters of input information and their relationship on the feature map. With every nodej, a parametric reference vector wj∈Rn is associated. In an abstract scheme it may be imagined that at time k the input x(k), by means of some parallel computing mechanisms, is compared with all wj(k), and the location of best match in some metric is defined as the location of the response. However, in many practical applications, the Euclidean distances ∥x(k)-wj(k)∥ can be made to define the best matching node (neuron) i which is signified by the index i(x),
(10)i(x)=argmin∥x(k)-wj(k)∥=argmin{∥ej(k)∥},
where j=1,2,…,N; k=1,2,…,M;x(k)={x1,x2,…,xM}T represents the input vectors and wj(k)=[wj1,wj2,…,wjM]T(j=1,2,…,N) represents the weight vectors, of the jth output vectors. ej(k) is the error which is given by ∥x(k)-w,i(x)(k)∥=min{∥ej(k)∥}.


## 2.1. Thermal Image

Infrared camera does not measure directly a temperature of surface. The detectors of the camera are sensitive to the luminance emitted by the object being studied. This luminance is naturally dependent on the temperature of the surface. Thermal image is an image which is captured by infrared thermography camera. This device can capture electromagnetic spectrum within infrared bands, 0.78–1000μm. A thermal image is a function of radiated energy on an inspected object [15]. This image can be digitized and represented in matrix form for computational processing as follows:
(1)f(x,y)=[f(0,0)f(0,1)⋯f(0,N-1)f(1,0)f(1,1)⋯f(1,N-1)⋮⋮⋮⋮f(M-1,0)f(M-1,1)⋯f(M-1,N-1)].


## 2.2. Image Thresholding

Image thresholding is mostly used for image segmentation [21, 22]. Thresholding is also common method for defect detection of material by IRT image. Thresholding is a process to separate objects from its background in a digital image. Histogram is the main tool in this separation process. Suppose that the grey level corresponds to an image f(x,y) that is composed of lights objects on a dark background, in such a way that object and background pixels have grey levels grouped into two dominant modes. Extracting the object from the background is performed by selecting a threshold T that separates these modes. A thresholded image g(x,y) is defined as
(2)g(x,y)={1iff(x,y)>T,0iff(x,y)≤T.7
The method of global threshold is selected in this paper where the grey-level histogram is calculated from the entire image [22].


## 2.3. Feature Extraction

### 2.3.1. Image Features

The perimeter and area are meaningful for binary image. Consider a discrete binary image containing one more object, wheref(j,k)=1 if a pixel is part of the object and f(j,k)=0 for all nonobject or background pixels. The bit-quads [23] method is adopted to determine the perimeter and area of ROI. The moment of order (p+q) of an image [24], denoted as mpq, is defined as
(3)mpq=∑x∑yxpyqg(x,y),
where x and y are coordinates of the pixel, and g(x,y) is the grey level of the pixel. The higher order moments represent spatial distribution of the objects. The previous two characteristics make moments a reasonable and proper feature for the classification task. However, the above equation may be greatly affected by the objects shifting. Therefore, to overcome this problem central moment is used and defined as
(4)μpq=∑x∑y(x-x-)p(y-y-)qg(x,y),
where x-=m10/m00 and y-=m01/m00.The relationship between moments and central moments up to order 3 are listed as follows:(5)μ00=m00μ11=m11-y-m10,μ10=0μ30=m30-3x-m20+2x-2m10,μ01=0μ12=m12-2y-m11-x-m02+2y-2m10,μ20=m20-x-m10μ21=m21-2x-m11-y-m02+2x-2m01,μ02=m02-y-m01μ03=m03-3y-m02+2y-2m01.
Then, central moments could be further normalized by the following formulas:
(6)ηpq=μpqμ00γ,
where
(7)γ=1+(p+q)2,forp+q=2,3,…


### 2.3.2. Vibration Features

Statistical are extracted from time domain vibration data using formulas presented in (8) and (9) as follows
(8)μ=1N∑n=1NXn,RMS=1N∑n=1NXn2,skew=1/N∑i=1N(Xi-μ)3σ3,kurt=1/N∑i=1NXi4σ3.
where,
(9)σ=∑i=1N1N(Xi-μ)2,CF=XmaxRMS,SF=RMSμ,


## 2.3.1. Image Features

The perimeter and area are meaningful for binary image. Consider a discrete binary image containing one more object, wheref(j,k)=1 if a pixel is part of the object and f(j,k)=0 for all nonobject or background pixels. The bit-quads [23] method is adopted to determine the perimeter and area of ROI. The moment of order (p+q) of an image [24], denoted as mpq, is defined as
(3)mpq=∑x∑yxpyqg(x,y),
where x and y are coordinates of the pixel, and g(x,y) is the grey level of the pixel. The higher order moments represent spatial distribution of the objects. The previous two characteristics make moments a reasonable and proper feature for the classification task. However, the above equation may be greatly affected by the objects shifting. Therefore, to overcome this problem central moment is used and defined as
(4)μpq=∑x∑y(x-x-)p(y-y-)qg(x,y),
where x-=m10/m00 and y-=m01/m00.The relationship between moments and central moments up to order 3 are listed as follows:(5)μ00=m00μ11=m11-y-m10,μ10=0μ30=m30-3x-m20+2x-2m10,μ01=0μ12=m12-2y-m11-x-m02+2y-2m10,μ20=m20-x-m10μ21=m21-2x-m11-y-m02+2x-2m01,μ02=m02-y-m01μ03=m03-3y-m02+2y-2m01.
Then, central moments could be further normalized by the following formulas:
(6)ηpq=μpqμ00γ,
where
(7)γ=1+(p+q)2,forp+q=2,3,…


## 2.3.2. Vibration Features

Statistical are extracted from time domain vibration data using formulas presented in (8) and (9) as follows
(8)μ=1N∑n=1NXn,RMS=1N∑n=1NXn2,skew=1/N∑i=1N(Xi-μ)3σ3,kurt=1/N∑i=1NXi4σ3.
where,
(9)σ=∑i=1N1N(Xi-μ)2,CF=XmaxRMS,SF=RMSμ,


## 2.4. Self Organizing Map (SOM)

The SOM is a neural network model that implements a characteristic non-linear projection from a high dimensional space of sensory or other input signals onto a low-dimensional array of neurons. The SOM is an unsupervised learning algorithm which produces a map pattern features on its output layer [20]. Input patterns with similar features are mapped onto neighbouring output nodes. The network consists of an input layer with m neurons and an output layer with n neurons as shown in Figure 1.Figure 1
SOM network [20].The image of the signal space tends to manifest the clusters of input information and their relationship on the feature map. With every nodej, a parametric reference vector wj∈Rn is associated. In an abstract scheme it may be imagined that at time k the input x(k), by means of some parallel computing mechanisms, is compared with all wj(k), and the location of best match in some metric is defined as the location of the response. However, in many practical applications, the Euclidean distances ∥x(k)-wj(k)∥ can be made to define the best matching node (neuron) i which is signified by the index i(x),
(10)i(x)=argmin∥x(k)-wj(k)∥=argmin{∥ej(k)∥},
where j=1,2,…,N; k=1,2,…,M;x(k)={x1,x2,…,xM}T represents the input vectors and wj(k)=[wj1,wj2,…,wjM]T(j=1,2,…,N) represents the weight vectors, of the jth output vectors. ej(k) is the error which is given by ∥x(k)-w,i(x)(k)∥=min{∥ej(k)∥}.


## 3. Method and Material

The flowchart of diagnostics process is presented in Figure2. After image acquisition from the object, thermal image is cropped to select the ROI, that is, rolling element bearing image. Next step, image clustering will be performed by using k-means algorithm to cluster the hotspot area based on color palette. Image segmentation by thresholding is conducted by using the a Otsu method [16]. Image thresholding will produce binary image which is useful for image feature extraction. Perimeter, area, and central moments are then calculated based on resulted binary images. After feature extraction, extracted features from normal and faulty condition are separated for SOM training purpose. The feature of normal condition will be used as baseline of the diagnostics process. Finally, SOM will make decision of whether the machine condition is normal or faulty.Figure 2
Flow chart of SOM diagnostics process.Data acquisition was conducted on MFS which is able to simulate abnormality in rotating machine, that is, rolling element bearing fault, unbalance, misalignment, and looseness. Thermal images and vibration signals were simultaneously acquired from the machine that has been operated for 30 minutes to generate initial heat in the bearings. The image acquisition produced 30 images per second, and the machine was operated at 1800 rpm. Figures3 and 4 show the data acquisition process in the laboratory using infrared thermography camera and accelerometer sensors.Figure 3
Data acquisition process.Figure 4
Accelerometer sensors placement.


## 4. Results and Discussion

Data acquisition of thermal image and vibration signal of faulty bearing is presented in Figures5 and 6. FLIR SC5000 camera was used to capture thermal images and LMS Test. Xpress was used to vibration signal acquisition from MFS. The acquired images were stored in RGB format with size of 256 × 256 pixels, and vibration signals were acquired in 5 kHz sampling frequency.Figure 5
Thermal images of MFS.Figure 6
Vibration signal of faulty bearing.After cropping, thermal images captured from right and left bearings are presented in Figures7 and 8. The ROI was determined by red or pink colors that represent the hottest temperature area generated by bearings. Such colors were then extracted and clustered by k-means algorithm with k=6. Figure 9 depicts the clustering result by means of k-means algorithm.Figure 7
Thermal images of right bearing.Figure 8
Thermal images of left bearing.Clustering of bearing images: (a) normal bearing; (b) faulty bearing.
(a)
(b)Figure9 also shows the colors in ROI from which feature extraction will be performed. The selected ROI will be transformed into binary image by thresholding method. Next step is feature extraction of perimeter, area, and central moments from ROI. This process is depicted in Figure 10.Figure 10
Processing of bearing thermal image.Figures11–13 show image features of area, perimeter, and central moment η01, respectively. The labels U, M, L, and B represent the faults in the machine namely unbalance, misalignment, looseness, and bearing faults, respectively. Figure 11 shows random patterns for the hot area of ROI which are not good patterns because the conditions of the machine cannot be distinguished clearly. The feature representation of Figure 12 shows better patterns than the feature of area, even though some data are out from the cluster of conditions. Observing this figure, temperature is increased between normal condition and faulty condition. This plot could be a reference for determination of bearing condition. Figure 13 also shows random structure of patterns but is better than feature of area. These patterns will be then input data for SOM training for fault diagnostics process.Figure 11
Feature area of thermal image.Figure 12
Feature perimeter of thermal image.Figure 13
Feature central momentη01 of thermal image.Feature extraction was also performed on time domain vibration signal by calculating its statistical features. The results of feature extraction are presented in Figures14, 15, 16. It is clearly observed that the machine conditions are well clustered. Comparing with features from thermal images, vibration features show better performance in clustering of machine conditions.Figure 14
RMS vibration.Figure 15
Skewness vibration.Figure 16
Kurtosis vibration.The fault diagnostics process is started by training SOM algorithm using features from thermal images and vibration signals. SOM is trained by 20 training data which consist of five machine conditions. The results of training process are presented by using U-matrix with hexagon topology to visualize distances between neighboring map units and help in seeing the cluster structure of the map. Map units indicate cluster border between conditions in the machine which are visualized by different color in the map. The U-matrices of SOM training are shown in Figures17 and 18. These maps will be references from which fault diagnostics procedure is performed.Figure 17
Mapping of perimeter image.Figure 18
Mapping of RMS vibration.After SOM training, the performance of SOM diagnostics was evaluated by testing process using 10 data from thermal images of bearing fault and unbalance case (BU label), and 8 data from vibration signals of bearing fault case (B label) respectively. In the case of thermal images data, 8 data successfully activated cluster of BU, and 2 data were misclassified in BU cluster. It means that the accuracy is 80%. In addition, testing process using vibration data gives accuracy of 100% which means that all data tests successfully activate the cluster of B. From the experiment, the performance of testing process using vibration data is better than thermal images data. It was well known that vibration data can produce good accuracy in machine fault diagnostics. However, it could be said that the results of testing process by using thermal images and vibration data show good performance. Testing process of SOM using thermal images and vibration data are visualized by presenting the topography of best matching unit (BMU) as depicted in Figures19 and 20, respectively.Figure 19
SOM test for perimeter data.Figure 20
SOM test for RMS vibration.


## 5. Conclusion

This paper deals with the confirmation of thermal images and vibration signals for intelligent machine fault diagnostics using SOM network. Thermal images and vibration signals were acquired from MFS that is a reliable tool for machinery simulation. Image segmentation technique was performed on thermal images to obtain binary images for image feature extraction process. The features extracted from thermal images were area, perimeter, and central moments. Statistical features, namely, RMS, means, skewness and kurtosis, were also extracted from time-domain vibration signals. SOM were then trained by using extracted features from both thermal images and vibration signals. After training, SOM was tested by feeding some data to evaluate the performance of fault diagnostics. The results show that good accuracy was reached by using both source features thermography and vibration. Moreover, it showed that the thermography method could be an alternative way for conducting intelligent machine fault diagnostics with plausible results.



---

*Source: 847203-2012-10-30.xml*