# Fault Diagnosis Method Based on Information Entropy and Relative Principal Component Analysis

Authors: Xiaoming Xu; Chenglin Wen
Journal: Journal of Control Science and Engineering
 (2017)

---

## Abstract

In traditional principle component analysis (PCA), because of the neglect of the dimensions influence between different variables in the system, the selected principal components (PCs) often fail to be representative. While the relative transformation PCA is able to solve the above problem, it is not easy to calculate the weight for each characteristic variable. In order to solve it, this paper proposes a kind of fault diagnosis method based on information entropy and Relative Principle Component Analysis. Firstly, the algorithm calculates the information entropy for each characteristic variable in the original dataset based on the information gain algorithm. Secondly, it standardizes every variable’s dimension in the dataset. And, then, according to the information entropy, it allocates the weight for each standardized characteristic variable. Finally, it utilizes the relative-principal-components model established for fault diagnosis. Furthermore, the simulation experiments based on Tennessee Eastman process and Wine datasets demonstrate the feasibility and effectiveness of the new method.


---

## Body

## 1. Introduction

In the process of the industry manufacturing, there is a large amount of variables that is highly correlative; these variables contain the essential information that would be helpful to judge the status of the system. As a result, it is an important problem to find and predict the fault through this information to ensure that the equipment always works in a safe and reliable way [1, 2].However, during the industry manufacturing, the collected characteristic variables have different units; this raises a problem that we may come out with the different result only due to the unit difference; hence, we have to standardize the unit. What is more, after the standardization, it is inevitable to lose the diversity among different variables and present the property of distribution uniformity in the perspective of geometry which makes it hard to extract the principle component for compression and diagnosis. As to overcome these problem, some methods have been proposed recently [3–8]. Shi et al. use the Mahalanobis distance for relative transformation to reduce the effect of the dimension standardization [4]. Tang et al. propose a relative transformation principal component analysis to reduce the data noise for the transformation oil breakdown voltage prediction [5]. Yi et al. introduce a relative transformation operator to change the original variables in the spatial distribution and eigenvalues of the covariance matrix in the feature space [6]. Wen et al. propose a method called Relative Principle Component Analysis (RPCA); it introduces weighting for each variable based on the prior information of the system to eliminate the false information due to standardizing the variable units [7, 8], but the shortage of this method is that it needs a large amount of prior information from the system which is hard to gain in real engineering application.In order to solve the problem, this paper introduces the concept of information entropy and proposes a new fault diagnosis method that combines the information entropy and relative transformation PCA which is called information entropy relative transformation PCA (InEnRPCA). The information entropy is put forward by Shannon in 1948 [9]; it indicates that the redundancy exists among any information and can be measured based on the symbol in the information such as the number, alphabet, and words. With the development of the information theory, the information entropy has become an effective method to measure the degree of importance for each feature in the sample and achieved a wide application in many areas. For instance, while using the decision-making tree for classification, Hu et al. use the information entropy to calculate the significance for each feature and then prune the decision-making tree and reduce the false alarm [10]. Y. Y. Chen and Y. M. Chen use the information entropy to measure the uncertainty on the data in the attribute reduction algorithm [11]. Wang et al. use the information entropy to balance the weight of each sememe in the area of natural language processing [12].As to the problem we are facing, we start in the perspective of information theory and use the information gain algorithm to extract information entropy as the heuristic knowledge from the original dataset, then use it to calculate the relative transformation factor and allocate the weight for each standardized characteristic variable, and, finally, use the corresponding RPCA method for fault diagnosis.The rest of this paper is organized as follows: in Section2, we review the information entropy and information gain definition and algorithm. The original relative transformation PCA method is given in Section 3. Our simulation experiments on Tennessee Eastman process and the Wine dataset from UCI are stated in Section 4, where we make a comparisons between PCA, USPCA, and improved InEnRPCA with thirteen datasets to demonstrate the effectiveness of the new method. Finally, Section 5 gives conclusions and some discussions.


## 2. Overview of Our Approach

A brief overview of our fault diagnosis approach is given in this section. As the framework in Figure1 shows, the proposed approach consists of two parts: calculating the relative transformation operators based on information entropy and fault diagnosis based on RPCA-kNN.Figure 1
InEnRPCA algorithm process.In the part for calculating the relative transformation operators, information entropy and information gain algorithm are applied on train data to get the relative transformation operators. Then, the operators are combined with the original data to get the relative transformation matrix.In fault diagnosis part, the RPCA are used to deal with processed data for dimension reduction, andkNN are used for classification training. After the above process, a model is built for further fault diagnosis.


## 3. Information Entropies

In the subject of the information theory, probability, and statistics, entropy is used to describe the uncertainty of random variable and can be used to show the reduced degree of the information uncertainty for setY after getting the character X.


### 3.1. The Definition of the Information Entropy and Information Gain

Theorem 1 (information entropy).
AssumeM is Lebesgue measure set generated by measurable set S with algebra σ and measure μ where μ(M)=1, also M can be shown in the form of the incompatible sets with A=Ai, that is, M=Ui=1nAi and Ai∪Aj=Φ,  ∀i≠j [13]. Then, we can come to a conclusion:(1)HA=-∑i=0nμAilog⁡μAi,where μ(Ai) is the measure for Ai.Theorem 2 (information gain).
Given the train setD and the corresponding feature A, the empirical entropy H(D) means the uncertainty level to classify set D, and the conditional empirical entropy HD∣A means the uncertainty level to classify feature A in the condition of set D. Then, the difference between them is the information gain which stands for the reduced level of the uncertainty to classify the set D by given feature A.
Obviously, as to the datasetD, the information gain depends on the features and different features have different information gain, the bigger the information gain, the stronger the ability it has for classifying.


### 3.2. The Information Gain Algorithm

Assume the train dataset isD; D denotes the sample capacity which is equal to the number of the samples. There are K classes in set D with each name being Ck  (k=1,2,…,K) and |Ck| is equal to the number of the samples which belongs to Ck; that is, ∑k=1K|Ck|=D. Also, assume there are n values in feature A called a1,a2,…,an and then divide set D into n parts with each name being Di  (i=1,2,…,n) and Di is equal to the number of the samples that belongs to Di; that is, ∑i=1n|Di|=D. Based on the above, define the set Dik as the intersection of class Ck and subset Di, that is, Dik=Di∀Dk, and Dik is the number of Dik [14]. Then, the information gain algorithm can be shown as follows: input: train dataset D and feature A; output: the information gain of set D by feature A defined as g(D,A).Step 1.
Calculate the empirical entropy of datasetH(D):(2)HD=-∑k=1KCkDlog2⁡CkD.Step 2.
Calculate the empirical entropy of the featureA in the condition of set D:(3)HD∣A=∑i=1nDiDHDi=-∑i=1nDiD∑k=1KDikDilog2⁡DikDi.Step 3.
Calculate the information gain which is also the relative transformation operatorMA:(4)gD,A=HD-HD∣A,MA=gD,A.Step 4.
Repeat the above processes from Steps1 to 3 for each feature i in the sample and get the relative transformation operator with each feature:(5)Mi=gD,i.


## 3.1. The Definition of the Information Entropy and Information Gain

Theorem 1 (information entropy).
AssumeM is Lebesgue measure set generated by measurable set S with algebra σ and measure μ where μ(M)=1, also M can be shown in the form of the incompatible sets with A=Ai, that is, M=Ui=1nAi and Ai∪Aj=Φ,  ∀i≠j [13]. Then, we can come to a conclusion:(1)HA=-∑i=0nμAilog⁡μAi,where μ(Ai) is the measure for Ai.Theorem 2 (information gain).
Given the train setD and the corresponding feature A, the empirical entropy H(D) means the uncertainty level to classify set D, and the conditional empirical entropy HD∣A means the uncertainty level to classify feature A in the condition of set D. Then, the difference between them is the information gain which stands for the reduced level of the uncertainty to classify the set D by given feature A.
Obviously, as to the datasetD, the information gain depends on the features and different features have different information gain, the bigger the information gain, the stronger the ability it has for classifying.


## 3.2. The Information Gain Algorithm

Assume the train dataset isD; D denotes the sample capacity which is equal to the number of the samples. There are K classes in set D with each name being Ck  (k=1,2,…,K) and |Ck| is equal to the number of the samples which belongs to Ck; that is, ∑k=1K|Ck|=D. Also, assume there are n values in feature A called a1,a2,…,an and then divide set D into n parts with each name being Di  (i=1,2,…,n) and Di is equal to the number of the samples that belongs to Di; that is, ∑i=1n|Di|=D. Based on the above, define the set Dik as the intersection of class Ck and subset Di, that is, Dik=Di∀Dk, and Dik is the number of Dik [14]. Then, the information gain algorithm can be shown as follows: input: train dataset D and feature A; output: the information gain of set D by feature A defined as g(D,A).Step 1.
Calculate the empirical entropy of datasetH(D):(2)HD=-∑k=1KCkDlog2⁡CkD.Step 2.
Calculate the empirical entropy of the featureA in the condition of set D:(3)HD∣A=∑i=1nDiDHDi=-∑i=1nDiD∑k=1KDikDilog2⁡DikDi.Step 3.
Calculate the information gain which is also the relative transformation operatorMA:(4)gD,A=HD-HD∣A,MA=gD,A.Step 4.
Repeat the above processes from Steps1 to 3 for each feature i in the sample and get the relative transformation operator with each feature:(5)Mi=gD,i.


## 4. Relative Transformation Principle Component Analysis

After the standardization for the data dimensions by the traditional PCA method, it may bring in some fake information to the principle element due to the uniform distribution. Therefore, we use the relative transformation PCA method to solve the problem [15]. Below is the brief RPCA process.AssumeX∈Ra×b, where a corresponds to the number of the samples and b is the number of features.Step 1.
Transfer the original data to the standardized data with mean zero and variance one.
The mean for each column is(6)m=1ax0Tla,where la=1,1,…,1T∈Ra; then,(7)X=X0-lamTΣ-1,where Σi  (i=1,…,b) is the standard deviation for each column.
We set the relative transformation:(8)XR=XM=x11x12⋯x1Nx21x22⋯x2N⋮⋮⋱⋮xn1xn2⋯xnN×M10⋯00M2⋯0⋮⋮⋱⋮00⋯Mn=x1R1x1R2⋯x1RNx2R1x2R2⋯x2RN⋮⋮⋱⋮xnR1xnR2⋯xnRN,where XR is relative transformation matrix from X and M is a diagonal matrix in which each of the variables Mi is the relative transformation operator that can be achieved from Section 3.2.Step 2.
Compute the covariance matrixRXR from XR:(9)RXR=XTXa-1.Step 3.
Compute the eigenvalueλ and the corresponding eigenvector of the covariance matrix RXR. Assume all eigenvalues satisfy  λ1≥λ2≥⋯, where λi stands for the i′ eigenvalue and pi is the corresponding eigenvector to λi.(10)λI-RXR=0,λiI-RXRpi=0;i=1,2,…,b.Step 4.
Select the number of the relative principle element based on the accumulative contribution rate:(11)∑1iλi∑1nλi>85%.Step 5.
UsekNN for classifying.


## 5. Application Experiments

In order to verify the universal applicability and effectiveness of the new method, firstly, we test it by thirteen experiments based on Tennessee Eastman process dataset and the Wine dataset [16] which are different in numbers of samples N, dimensions D, and classes C; in addition, we divide all the classes in three or four classes in sequence to avoid the deliberate choosing; secondly, experiments are also performed on original PCA and traditional DPPCA, respectively. The datasets descriptions and experimental settings are shown in Table 1.Table 1
The datasets descriptions and parameter constitution.
Dataset Numbers Dimensions Classes Train Test TE (1–4) 800 400 52 4 TE (5–8) 800 400 52 4 TE (9–12) 800 400 52 4 TE (13–16) 800 400 52 4 TE (17–20) 800 400 52 4 TE (1–3) 600 300 52 3 TE (4–6) 600 300 52 3 TE (7–9) 600 300 52 3 TE (10–12) 600 300 52 3 TE (13–15) 600 300 52 3 TE (16–18) 600 300 52 3 TE (19–21) 600 300 52 3 Wine 148 30 14 3


### 5.1. TE Process Example

We use the Tennessee Eastman process data as the testing samples which are obtained from the document. The dataset includes twenty-one classes and each class in the experiment has two hundred train samples and one hundred test samples; besides, each sample has 52 features. The different types in TE process are overlapped and difficult to classify in observed space.After the data preprocessing, the number of the principle elements is determined by the sum of elements being more than 85%. For the twelve experiments based on Tennessee Eastman process, we choose one of the experiments based on TE classes 5, 6, 7, and 8 labeled by F1, F2, F3, and F4, respectively, for detail introduction and demonstration. Its relative transformation operator for each feature to the TE process can be seen in Table 2 and the experimental results with the three approaches are shown in Figures 2(a)–2(c).Table 2
Weight for each feature in the TE dataset.
Feature Weight 1 1.26 2 0.83 3 1.16 4 1.06 5 1.15 6 0.82 7 1.25 8 0.94 9 0.89 10 1.29 11 1.25 12 0.96 13 1.26 14 1.12 15 0.90 16 1.23 17 0.80 18 1.06 19 1.37 20 1.35 21 1.14 22 1.23 23 1.27 24 0.78 25 1.25 26 0.58 27 0.67 28 1.27 29 1.32 30 0.94 31 1.30 32 0.54 33 0.67 34 1.31 35 1.23 36 1.23 37 0.59 38 1.30 39 0.60 40 0.65 41 0.78 42 1.17 43 1.25 44 1.26 45 1.53 46 1.34 47 1.31 48 0.96 49 0.90 50 1.37 51 1.22 52 1.41Figure 2
Feature extraction to TE process: (a) PCA. (b) DSPCA. (c) InEnRPCA.
(a) (b) (c)As we can see from the figures, PCA and DSPCA have poor performance in pattern classification where four classes are overlapped. In Table4, their classification accuracy is 31.5% and 75.3%, respectively. Compared with them, the information entropy RPCA can help identify each class higher and its classification accuracy is 82.3%. Clearly, the InEnRPCA is a suitable feature extraction and classification method for fault diagnosis.


### 5.2. Wine Dataset

We select the Wine data from the machine learning database UCI, which includes three classes with the number of the each class being 59, 61, and 58; besides, each of the samples has fourteen features. The types in Wine are overlapped and difficult to classify in observed space.The relative transformation operator for each feature in the Wine dataset can be seen in Table3 and the experimental results with the three approaches are shown in Figures 3(a)–3(c).Table 3
Weight for each feature in the Wine dataset.
Feature Weight 1 0.53 2 0.28 3 0.13 4 0.28 5 0.20 6 0.42 7 0.71 8 0.17 9 0.28 10 0.59 11 0.43 12 0.57 13 0.59 14 1.56Table 4
The classifying accuracy rate for each method.
Dataset PCA (%) DSPCA (%) InEnRPCA (%) TE (1–4) 50.5 95.8 97.3 TE (5–8) 31.5 75.3 82.3 TE (9–12) 34.0 41.8 47.5 TE (13–16) 38.3 54.5 53.8 TE (17–20) 47.3 77.0 78.0 TE (1–3) 68.7 100.0 100.0 TE (4–6) 62.0 64.0 59.0 TE (7–9) 47.0 73.3 77.3 TE (10–12) 45.7 58.7 59.3 TE (13–15) 49.0 59.3 38.3 TE (16–18) 62.0 83.3 85.0 TE (19–21) 32.7 55.3 59.3 Wine 66.7 96.7 100Figure 3
Feature extraction to Wine dataset: (a) PCA. (b) DSPCA. (c) InEnRPCA.
(a) (b) (c)From Figure2(b) and Table 2, we can see that PCA and DSPCA do not have a perfect performance in pattern classification with the accuracy rate is 66.7% and 96.7%, respectively. Compared with them, the InEnRPCA can identify each class accurately, and the testing data can also be classified obviously, and its classification accuracy rate is 100%.


### 5.3. Discussion

From the experiments above, it is obvious that the proposed methods can successfully distinguish different fault types and implement fault pattern recognition and diagnosis in most times, via utilizing information entropy to calculate relative transformation operator for each feature and combining with relative PCA approach.Meanwhile, we also find that although the proposed method can improve the fault classification performance in most times (nearly80%), the method is sometimes inferior to DSPCA; the reason may be due to the overfitting for the train set and the dissimilarity between the train set and test set; however, we can make sure that utilizing the information entropy to calculate the weight for each feature to avoid distribution equally between different features is reasonable; we will continue in-depth research for better fault diagnosis.


## 5.1. TE Process Example

We use the Tennessee Eastman process data as the testing samples which are obtained from the document. The dataset includes twenty-one classes and each class in the experiment has two hundred train samples and one hundred test samples; besides, each sample has 52 features. The different types in TE process are overlapped and difficult to classify in observed space.After the data preprocessing, the number of the principle elements is determined by the sum of elements being more than 85%. For the twelve experiments based on Tennessee Eastman process, we choose one of the experiments based on TE classes 5, 6, 7, and 8 labeled by F1, F2, F3, and F4, respectively, for detail introduction and demonstration. Its relative transformation operator for each feature to the TE process can be seen in Table 2 and the experimental results with the three approaches are shown in Figures 2(a)–2(c).Table 2
Weight for each feature in the TE dataset.
Feature Weight 1 1.26 2 0.83 3 1.16 4 1.06 5 1.15 6 0.82 7 1.25 8 0.94 9 0.89 10 1.29 11 1.25 12 0.96 13 1.26 14 1.12 15 0.90 16 1.23 17 0.80 18 1.06 19 1.37 20 1.35 21 1.14 22 1.23 23 1.27 24 0.78 25 1.25 26 0.58 27 0.67 28 1.27 29 1.32 30 0.94 31 1.30 32 0.54 33 0.67 34 1.31 35 1.23 36 1.23 37 0.59 38 1.30 39 0.60 40 0.65 41 0.78 42 1.17 43 1.25 44 1.26 45 1.53 46 1.34 47 1.31 48 0.96 49 0.90 50 1.37 51 1.22 52 1.41Figure 2
Feature extraction to TE process: (a) PCA. (b) DSPCA. (c) InEnRPCA.
(a) (b) (c)As we can see from the figures, PCA and DSPCA have poor performance in pattern classification where four classes are overlapped. In Table4, their classification accuracy is 31.5% and 75.3%, respectively. Compared with them, the information entropy RPCA can help identify each class higher and its classification accuracy is 82.3%. Clearly, the InEnRPCA is a suitable feature extraction and classification method for fault diagnosis.


## 5.2. Wine Dataset

We select the Wine data from the machine learning database UCI, which includes three classes with the number of the each class being 59, 61, and 58; besides, each of the samples has fourteen features. The types in Wine are overlapped and difficult to classify in observed space.The relative transformation operator for each feature in the Wine dataset can be seen in Table3 and the experimental results with the three approaches are shown in Figures 3(a)–3(c).Table 3
Weight for each feature in the Wine dataset.
Feature Weight 1 0.53 2 0.28 3 0.13 4 0.28 5 0.20 6 0.42 7 0.71 8 0.17 9 0.28 10 0.59 11 0.43 12 0.57 13 0.59 14 1.56Table 4
The classifying accuracy rate for each method.
Dataset PCA (%) DSPCA (%) InEnRPCA (%) TE (1–4) 50.5 95.8 97.3 TE (5–8) 31.5 75.3 82.3 TE (9–12) 34.0 41.8 47.5 TE (13–16) 38.3 54.5 53.8 TE (17–20) 47.3 77.0 78.0 TE (1–3) 68.7 100.0 100.0 TE (4–6) 62.0 64.0 59.0 TE (7–9) 47.0 73.3 77.3 TE (10–12) 45.7 58.7 59.3 TE (13–15) 49.0 59.3 38.3 TE (16–18) 62.0 83.3 85.0 TE (19–21) 32.7 55.3 59.3 Wine 66.7 96.7 100Figure 3
Feature extraction to Wine dataset: (a) PCA. (b) DSPCA. (c) InEnRPCA.
(a) (b) (c)From Figure2(b) and Table 2, we can see that PCA and DSPCA do not have a perfect performance in pattern classification with the accuracy rate is 66.7% and 96.7%, respectively. Compared with them, the InEnRPCA can identify each class accurately, and the testing data can also be classified obviously, and its classification accuracy rate is 100%.


## 5.3. Discussion

From the experiments above, it is obvious that the proposed methods can successfully distinguish different fault types and implement fault pattern recognition and diagnosis in most times, via utilizing information entropy to calculate relative transformation operator for each feature and combining with relative PCA approach.Meanwhile, we also find that although the proposed method can improve the fault classification performance in most times (nearly80%), the method is sometimes inferior to DSPCA; the reason may be due to the overfitting for the train set and the dissimilarity between the train set and test set; however, we can make sure that utilizing the information entropy to calculate the weight for each feature to avoid distribution equally between different features is reasonable; we will continue in-depth research for better fault diagnosis.


## 6. Conclusion

This paper has analyzed the PCA method in the perspective of information theory and proposed a kind of fault diagnosis research based on information entropy and Relative Principle Component Analysis. This proposed method can solve the following problems effectively:(1) the different result only due to calculating in different perspective of dimensions; (2) being uniformly distributed between the characteristic variables after dimension standardization; (3) how to calculate the weight for each variable. Both theoretical analysis and simulation experiments on Tennessee Eastman process and the Wine dataset from UCI demonstrate the feasibility and effectiveness of the new approach.It is worth noting that the idea of using the information entropy to determine the certainty of the features is not new. However, the idea of combining information entropy with RPCA for faulty detection, to our knowledge, should have no previous publication. The proposed method should be seen as an alternative fault diagnosis method. It is not superior to the other methods in all cases.There is some interesting future work:(i)
To exploit the way to make the energy conservation the same before and after the relative transformation: in the perspective of the energy conservation, the system energy before and after the relative transformation may usually not be the same. If we can consider the energy conservation during the relative transformation, the effect may be much better.(ii)
To apply the relative transformation by information entropy as data preprocessing in other methods: the proposed method is not limited to RPCA method; it is an effective data preprocessing way that can also be applied to other fault diagnosis methods.



---

*Source: 2697297-2017-02-20.xml*