Graph Low-rank Non-negative Matrix Factorization with Auto-encoders for Fault Detection

Authors: yabing  liu
shanshan  yu
wei  guo
fai  leung
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-07 17:39:51

Abstract: Abstract—Fault detection is the process of detecting and
diagnosing faults or abnormalities in a system by analyzing
its operational data. However, with the complexity of
modern industrial processes, some faults are difficult to
be detected in a timely manner due to various factors
such as noise and data nonlinearity. Therefore, data-driven
Fault Detection (FD) has become a widely used method to
detect abnormal events in functional modules. Non-negative
Matrix Factorization (NMF), as an efficient dimensionality
reduction technique, has not had potential applications
in fault detection (FD) thoroughly explored. In order to
improve the FD methods based on NMF, we have developed
a new approach, named graph low-rank non-negative
matrix Factorization with auto-encoders (GLNMFA). GLNMFA integrates the Laplacian operator effectively identifies the local structure among data points, enhancing
the performance of dimensionality reduction algorithms.
It also introduces the nuclear norm to find a low-rank
approximation to the original matrix, thereby constraining
sparsity. Additionally, auto-encoders are incorporated to
learn a low-dimensional representation of the data and
extract key features, which are subsequently applied for
fault detection purposes. We employ an optimization algorithm based on Alternating Direction Method of Multipliers
(ADMM) to optimize this model. Two test statistics T
2
(Hotelling’s T-squared), SPE (Squared Prediction Error)
are used to evaluate detection efficiency. Kernel Density
Estimation (KDE) are used to estimate control limits for
fault detection. The effectiveness of GLNMFA is validated
on two benchmark datasets.

Main paper: 
 i. introduction: 
 In modern industrial production, it is essential to ensure the stable operation of equipment and the continuity of processes. The application of fault detection technology aims to discover and diagnose potential faults in time through real-time monitoring and analysis of various data in the production process, so as to take preventive measures to avoid production disruption. Fault detection methods can be divided into three categories: model-based method, signal-based method and datadriven method. Non-negative matrix factorization (NMF) has emerged as a powerful tool for data analysis, found applications across various domains. Its fundamental idea is to decompose a given data matrix into the product of non-negative basis vectors and coefficient matrices, enabling efficient representation and dimensionality reduction of the data. In recent years, NMF has shown significant potential in the field of fault detection. Fault detection is a critical task in industrial production and equipment maintenance [1] - [4] . Traditional methods for fault detection often rely on expert knowledge or statistical techniques, which may struggle to handle the variability and high dimensionality of data in complex systems [5] . As a data-driven approach, NMF can automatically learn basic patterns from data, offering a new perspective and tool for fault detection [6] . Fault detection methods are generally divided into three kinds, which are signal driven, model driven and data driven. With the rapid development of data collection and data processing technology, data-driven fault detection has become the mainstream [7] . The most popular datadriven methods include Independent component analysis (ICA) [8] , Principal Component Analysis (PCA) [9] , canonical Component analysis (CCA) [10] , and nonnegative matrix factorization (NMF). NMF has been successfully applied in various applications, including feature extraction, topic modeling, and collaborative filtering. Its ability to discover underlying patterns in data and its interpretability make it a valuable tool in exploratory data analysis and dimensionality reduction tasks [11] . In the classical Non-negative Matrix Factorization (NMF) model, enhancing the model's performance can be achieved by incorporating regularization terms and constraints. This approach facilitates the extraction of more meaningful features and enhances the effectiveness of dimensionality reduction [12] , [13] . For example, Sparse Non-negative Matrix Factorization (SNMF) produces more interpretive decomposition results by introducing sparsity constraints into the NMF model. This ensures that most elements in the generated base and coefficient matrices are zero [14] - [17] . Graph Regularized Non-negative Matrix Factorization (GNMF) introduces graph structure information based on standard NMF, and improves the quality and stability of decomposition results by using the relationship between data samples [18] , [19] , [20] . Orthogonal Non-negative Matrix Factorization (ONMF) introduces orthogonal constraints, i.e., the resulting basis matrix is orthogonal [17] , [21] . The orthogonal constraint makes the basis matrix generated by NMF more sparse and mutually exclusive, thus enhancing the interpretation and generalization ability of decomposition results. The statistical strategy based on NMF was first developed by Lee and Seung [22] . Li et al was the first to apply NMF fault detection in non-Gaussian processes [23] . Two statistical metrics are established for fault detection using the NMF method, namely the squared prediction error (SPE) and squared distance statistic (T 2 ), T 2 can effectively combine information from multiple variables to detect anomalies by monitoring changes in these variables and their correlations, and kernel density estimates (KDE) were used to estimate the control limits [24] , [25] . The data on the benchmark Tennessee Eastman process show that the fault detection method based on NMF has better performance. Later, many new NMF variants appeared and proved to have better fault detection performance [26] , [27] .In this paper, a novel fault detection approach is presented, which is based on Non-negative Matrix Factorization (NMF) and is named Graph Low-rank Nonnegative Matrix Factorization with Auto-encoder (GLN-MFA). This method leverages the graph Laplacian to incorporate the topological relationships between process variables, thereby enhancing the model's ability to utilize information from these variables. Additionally, the method utilizes the nuclear norm to ensure a low-rank matrix approximation, which efficiently reduces model redundancy. This advancement is supported by the cited literature [28] - [30] . Auto-encoders play a crucial role in this framework by mapping high-dimensional data into a lower-dimensional space, simplifying data representation and reducing complexity, while retaining essential information [31] . This unsupervised learning algorithm achieves this by encoding input data into a compact representation and then reconstructing it through a decoder to closely match the original input [32] , [33] . Then the fault detection efficiency of this model is discussed [34] - [37] . Based on analysis of benchmark datasets such as the Tennessee Eastman Process (TEP) and XJTU-SY rolling bearings, this paper demonstrates the advantages and potential applications of GLNMFA in fault detection [38] , [39] . This is expected to improve the accuracy and efficiency of fault detection, providing more reliable support for industrial production and equipment maintenance. The remainder of the paper is outlined below, with Section II reviewing the classic NMF and some representative variants. Section III designs an effective algorithm to solve GLNMFA. Section IV introduces apply the GLNMFA model proposed in this paper for fault detection, Section V highlights the advantages of this algorithm by comparing with other methods, and finally, provides a summary of the paper.For a matrix X ∈ R m×N , where m represents the number of variables and N represents the number of samples, the notation x i represents the i th row of matrix X, and x ij represents the element at the i th row and j th column of matrix X, || • || F is the Frobenius norm of the matrix. X * is the nuclear norm of the matrix, representing the sum of all the singular values of the matrix. X T and X -1 are expressed as their transposed and inverse matrices, respectively. The inner product of two matrix can be given by the following formula :X, Y = tr(X T Y) = m i=1 N j=1 x ij y ij .Mathematically, NMF can be formulated as : X ≈ WH. X is the original data matrix, W ∈ R m×k contains the basis vectors, called the basis matrix, and H ∈ R k×N contains the coefficients, called the coefficient matrix, in which k is the reduced dimension satisfying (m + N) × k < m × N . The loss function of NMF is usually defined as the distance between the original matrix X and the approximately reconstructed matrix WH, and the difference between the two is usually measured using the Euclidean distance or the distance based on the KL divergence. According to Lee and Seung's research, the loss function can be reasonably defined as :EQUATIONIn this case, W and H are greater than or equal to zero, which means that their elements are non-negative. There are many optimization methods for NMF-related problems, such as multiplication update (MU), gradient descent (PGD) [40] , etc. But these algorithms converge too slowly. A new alternating direction multiplier method (ADMM) shows good convergence performance when solving NMF-related problems. In this paper, ADMM algorithm is used to solve the target problem.Adding some constraints or regularization terms to the classical NMF model can improve the performance of the model, for example, graph regularized NMF (GNMF) [18] .EQUATIONIn the model, a graph regularization term is added to the classical NMF, where L is referred to as the Laplacian matrix learned from the original matrix X, defined as (L = D -Z). Here, D represents the adjacency matrix, and Z denotes the degree matrix. The addition of this regularization term can better capture the local structure information among the data points, thus improving the performance of data dimensionality reduction [41] .Another popular variant of NMF is obtained by placing a constraint on the coefficient matrix H, named sparse nonnegative matrix decomposition (SNMF) [14] .EQUATIONCompared with the classical NMF model, the above model adds additional constraints on the coefficient matrix H. Here, the sparsity of H is controlled by the l 0 norm constraint on H, so that H ≤ s to achieve better dimensionality reduction. s here is a parameter that controls the sparsity of the coefficient matrix H.This section introduces the proposed NMF model. In order to obtain better performance, two regularization terms, the nuclear norm term and the auto-encoder term, are added on the basis of GNMF are added on the basis of GNMF, nuclear norm term and auto-encoder term. The nuclear norm regularization term can achieve the low-rank approximation of the matrix, thus achieving the effect of dimensionality reduction and denoising, autoencoding terms can learn a low-dimensional representation of the data to extract key features.EQUATION)H * is the nuclear norm of H, λ 1 , λ 2 , λ 3 is the regularization parameter. In fact, it can be viewed as an extension of Model (2).The main idea of ADMM is to decompose the original problem into multiple sub-problems, and gradually approach the optimal solution of the original problem by optimizing each sub-problem alternatively [42] , [43] . The optimization of nuclear norm is complicated. Meanwhile, in order to simplify the model optimization process, introduce two auxiliary variables Y and U, then equation ( 4) can be reformulated as follows:EQUATIONConstruct an augmented Lagrange of the original function as follows:EQUATIONwhere β 1 , β 2 are penalty parameters, and A, B are Lagrange multipliers. Here also need to consider the nonnegative constraints on W and H. Under the framework of ADMM, the optimal solution is iteratively obtained by updating variables one by one [44] . The iterative steps for solving (6) are as follows:W k+1 = arg min W L(W, H k , U k , Y k , A k , B k ) H k+1 = arg min H L(W k+1 , H, U k , Y k , A k , B k ) U k+1 = arg min U L(W k+1 , H k+1 , U, Y k , A k , B k ) Y k+1 = arg min Y L(W k+1 , H k+1 , U k+1 , Y, A k , B k ) A k+1 = A k -β 1 (W k+1 H k+1 -Y k+1 ) B k+1 = B k -β 2 (H k+1 -U k+1 ).The update process for each variable is discussed in detail below:(1) Update W while fixing other variables.updating W can be simplified as following optimization problems.EQUATIONEquation ( 8) is the result after derivation, which belongs to the Sylvester equation, If an equation form such as:AX + XB = C.A ∈ R m×m ,B ∈ R n×n ,X and B ∈ R m×n , then this equation is called the Sylvester equation, Bartels et al.have given an introduction to solving this equation in detail, and this paper does not go into detail [45] . In equation (8) , letA = 2λ 1 XX T , B = β 1 H k H T k , C = 2λ 1 XH T k + β 1 Y k H T k + A k H T k ,it is easy to see that the solution to W is solved in accordance with the form of the Sylvester matrix.(2) Update H while fixing other variables. updating H can be simplified as following optimization problems.EQUATIONEQUATIONEquation (10) can also be seen as a Sylvester equation through simple calculation, which can also be solved. In this equation, I is the identity matrix , I ∈ R k×k .(3) Update Y while fixing other variables. updating Y can be simplified as following optimization problems.min Y 1 2 X -Y 2 F + β 1 2 Y -W k+1 H k+1 + A k /β 1 2 F . (11)The following solution is obtained through derivation:EQUATION(4) Update U while fixing other variables. updating U can be simplified as following optimization problems.EQUATIONSince the nuclear norm is not differentiable in most cases, and this form of formula has a closed solution, it is solved as follows: a. Singular value decomposition (SVD) [46] : Decomposition of matrix A into the product of three matrices U, Σ, V. A = UΣV T , Σ is a diagonal matrix containing singular values of matrix A. b. Select a threshold: Determine a threshold τ , τ is used to determine the magnitude of the singular value. c. Threshold processing: The elements of the Σ matrix is less than or equal to the threshold value The singular values of τ is set to 0, preserving singular values greater than the threshold.σ i = σ i if σ i > τ 0 if σ i ≤ τ .d. Reframe: using the processed Σ matrix, and reconstruct A matrix to A = UΣ V T . e. A is a simplified or denoised version of the original matrix A.Algorithm 1 ADMM for GLNMFA Input: Given the original data matrix X, Laplace matrix L, parameterλ 1 , λ 2 , λ 3 > 0, penalty parameters β 1 , β 2 > 0.Initialize: (W 0 , H 0 , U 0 , Y 0 , A 0 , B 0 ),=A k -β 1 (W k+1 H k+1 -Y k+1 ). 6: Update B k+1 = B k -β 2 (H k+1 -U k+1 ). End WhileThis section introduces the fault detection process based on GLNMFA and conducts a comparative analysis with PCA, NMF, and SNMF using the TEP and XJTU-SY Bearing Dataset. The aim is to establish that GLN-MFA demonstrates higher detection efficiency than other algorithms across the majority of fault variables. All experiments in this article were conducted on Windows 10, Intel(R) Core(TM) i7-8750H, CPU of 2.21 GHz, RAM of 16.0 GB, using Matlab R2020b. A. Fault Detection Process based on GLNMFA Fig. 2 described the fault detection process based on Graph low-rank non-negative matrix decomposition with auto-encoding(GLNMFA-FD) method, detailed steps are as follows:(1) Initialization: given the Normal Sample (not-fault) data matrix X, builds the Laplace matrix L of the original matrix. W and H are initialized using positive random numbers.(2) Calculate W and H: The GLNMFA model is optimized by ADMM algorithm, and the decomposed radix matrix W and coefficient matrix H are obtained.(3) Calculate T 2 and SP E: The coefficient matrix H can be viewed as a low-rank approximation of the original data matrix. As stated in [23] , H reflects the state of the industrial process. To reconstruct H and X, the formula is as follows:EQUATIONIn the process of fault detection based on non-negative matrix factorization, two fault detection indexes T 2 and SP E are constructed as follows:EQUATION(4) Computational control limit: The upper control limits for the two monitoring metrics T 2 and SP E are calculated using KDE. The KDE equation for univariate kernel estimation is shown in equation (17) :EQUATIONP (x) is the estimate of the probability density function, where N is the number of samples, h is the bandwidth, and K(•) is the kernel function. The requirements for kernel functions are shown in equation (18) .EQUATIONThe corresponding control limits for T 2 statistics and SPE statistics are denoted as J th,T 2 and J th,SP E , respectively.(5) According to the test data, a new coefficient matrix H is obtained. New T 2 and SP E are calculated according to equations ( 16).(6) Determine whether the new T 2 and SP E are faulty by comparing them with the corresponding control limits. If the test statistics are out of control, a fault occurs, otherwise normal. Therefore, the detection logic can be defined as: T 2 < J th,T 2 and SPE < J th,SPE ⇒ fault -free T 2 ≥ J th,T 2 or SPE ≥ J th,SPE ⇒ faulty.(19) (7) Two percentage parameters, MAR (Miss alarm rate) and FAR (False alarm rate), are used as criteria to judge fault detection. This formula means that the smaller MAR and FAR, the better the detection results.EQUATIONThe Benchmark Tennessee Eastman process is a standardized platform for testing process monitoring and fault diagnosis algorithms [47] . Fig. 3 shows the complex chemical process for TEP, including 22 processing units and 12 process variables [17] . TE process is highly nonlinear and multi-variable, so it is widely used in research for monitoring algorithms. Its standardization promotes the performance comparison of algorithms and the development of techniques. TE process data sets are used to evaluate the accuracy and reliability of algorithms, which is of great significance to improve the efficiency and safety of chemical processes. In the PCA based approach [48] , the selection variance contribution is 85%, and the corresponding k value is 20 and the number of iterations is set to 1000. I and Table II , respectively. In addition, the best performances are highlighted in bold. In Table I and Table II , T 2 , SPE represents the percentage of the fault variable that is greater than the control limit in the total sample. In the Fig. 5 to Fig. 9 , T 2 , SPE represents the actual value calculated according to the fault detection strategy. It can be seen from the data in the Table I-II that although GLNMFA is weaker than individual algorithms at individual fault points, GLNMFA has the best fault detection efficiency overall.The XJTU-SY rolling bearing acceleration life test dataset provided by Xi'an Jiaotong University is a valuable resource containing vibration signals and lifespan data of rolling bearings under different operating conditions [49] . It is primarily used for research in bearing life prediction and health monitoring, aiding in the development and evaluation of machine learning and deep learning models. Fig. 4 Experimental platform includes AC motor, motor speed controller, shaft, support bearing, hydraulic loading system and test bearing [50] . Data are collected using a portable dynamic signal acquisition unit at a sampling frequency of 25.6 kHz, with a sampling duration of 1 minute and a duration of 1.28 seconds per sample. The dataset, in CSV format, includes vibration signals for analyzing bearing fault types and characteristics. Fault diagnosis studies based on this dataset employ various algorithms such as standard deviation, FFT spectrum, and envelope spectrum for abnormal detection and fault classification. For instance, envelope spectrum analysis of acceleration signals aids in identifying outer race faults. This dataset not only fuels research in prediction and health management but also facilitates the practical application of intelligent operations and maintenance in the industry. The pictures Fig. 5 to Fig. 9 show detection performance of Bearing 3-1 for XJTU-SY bearing. In this dataset, faults are artificially introduced starting at the 201st data point. The figure illustrates the fault points on the horizontal axis, the calculated sample values on the vertical axis, and the dashed line indicates the control limit. A fault point exceeding the control limit confirms the occurrence of a fault, while a point below the limit suggests that no fault has occurred or the fault has not been accurately detected.As can be seen from Table III and Table IV , although the detection result of GLNMFA is not optimal at individual fault points (for example, MAR Values for Bearing 2-5), considering the average results, the detection efficiency of GLNMFA for XJTU-SY data set is nonetheless remains.In this paper, a novel non-negative matrix decomposition model GLNMFA is proposed for fault detection in industrial systems. The model improves the fault detection efficiency by adding regularization terms to the classical NMF model. The graph regularization term plays a pivotal role in preserving the inherent graph structure characteristics of the data, which is essential for comprehending the interdependencies and influences among various components within a complex system. The incorporation of nuclear norm regularization terms results in a sparser representation, aiding in the elimination of noise and insignificant features, thus enhancing the model's interpretability. Experimental validation on the TEP and XJTU-SY datasets has substantiated the effectiveness of the proposed NMF algorithm in fault detection. When compared to existing fault detection methods, this model exhibits superior performance in terms of detection accuracy, robustness, and computational efficiency.Despite the positive results of this study, there is still room for further improvement. Future work can focus on: parameter selection and tuning strategies to automatically determine optimal regularization parameters. Algorithm performance optimization in real-time fault detection scenarios. Multi-modal data fusion to further improve the accuracy of fault detection.  
 
 Appendix:  



Reviews: review: title: title: Graph Low-rank Non-negative Matrix Factorization with Auto-encoders for Fault Detection
main_review: main_review: In this paper, a novel non-negative matrix decomposition model GLNMFA is proposed for fault detection in industrial systems.
a The language in the article needs further polishing.
b The references should be updated. Some closely related and new references should be added to show to further explain the novelty and innovation of the work.
c For the results presented in the Figures in the simulation, more explanations on them seem necessary and helpful to readers.
title: title: In this paper, a novel non-negative matrix decomposition model GLNMFA is proposed for fault detection in industrial systems.
main_review: main_review: a The language in the article needs further polishing.
b The references should be updated. Some closely related and new references should be added to show to further explain the novelty and innovation of the work.
c For the results presented in the Figures in the simulation, more explanations on them seem necessary and helpful to readers.
title: title: This article is well written and can be accepted.
main_review: main_review: 1.There are many grammatical and typographical errors in the manuscript. Please check the full text carefully and correct them.
2.In simulation section, more analysis and descriptions should be given to show the effectiveness of the developed method.
3.To make it easier for readers to understand the novelties of this paper, it would be better for the authors to add the control frame diagram.
4.The engineering background of the proposed problem should be stated more clearly to readers. The literature review is insufficient. Some recently published papers should be included in the references list.