# Statistical Analysis of Nonlinear Processes Based on Penalty Factor

Authors: Yingwei Zhang; Chuanfang Zhang; Wei Zhang
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

A new process monitoring approach is proposed for handling the nonlinear monitoring problem in the electrofused magnesia furnace (EFMF). Compared to conventional method, the contributions are as follows: (1) a new kernel principal component analysis is proposed based on loss function in the feature space; (2) the model of kernel principal component analysis based on forgetting factor is updated; (3) a new iterative kernel principal component analysis algorithm is proposed based on penalty factor.


---

## Body

## 1. Introduction

In consideration of ensuring the safety of the equipment and quality of product, the monitoring of the process performance has become an indispensable issue. In order to enforce the rationality and effectiveness of monitoring, in the last few decades, multivariate statistical process monitoring (MSPM) has been intensively researched. Particularly, principal component analysis (PCA) and partial least squares (PLS) which are widely applied in the industrial processes have been important approaches for monitoring of the process performance and some improved methods, such as kernel principal component analysis (KPCA) and kernel partial least squares (KPLS), have achieved great success in process monitoring and fault diagnosis [1–5].As the scale of modern industrial processes is expanding and the complexity of process is increasing, how to ensure the safety of process operation and improve product quality are two issues need to be solved in industrial production enterprises [6, 7]. Process monitoring technology is an effective way to solve these two issues. Since the complexity and fluctuation of industrial processes, accurate process models are difficult to build and apply [8]. Therefore, application of traditional process monitoring methods based on qualitative or quantitative models is subject to certain limitation.Because of the developments of intelligent instrumentations and computer technology in industrial process applications, a large number of high dimensional and strongly correlated process data is collected and stored [9–11]. It is difficult to remove redundancy and interference to extract useful information. It is an efficient monitoring technology to deal with the correlation of multivariate statistical process [12, 13].In this paper, the following work focused on the process change caused by aging equipment, process drift, and sensor measurement errors in nonlinear industrial process [14–16].In practical industrial process, outliers are contained in the collected data, while traditional kernel principal component analysis method is based on the assumption that there are no outliers in the sample data [17, 18]. Outliers still exist even after mapping them into feature space. Even if the sample data contains only a small amount of outliers, great negative effect will be applied on process model [19, 20]. Therefore, an advanced kernel principal component analysis method is proposed in this paper, which defines a loss function in feature space in the sense of minimum reconstruction error [21]. Then iteration with penalty will be carried to obtain the principal components, which can eliminate the adverse effects of outliers. Whenever a new sample is available, reconstruct it with the previous transfer matrix and calculate the reconstruct error [22–25]. If the new sample is an outlier, then update model with the reconstructed sample, otherwise update model with the original sample. Simulation results show that the advanced KPCA method can reduce the impact of outliers and improve the accuracy of the process monitoring model as well [26–28].The rest of this paper is organized as follows. Kernel principal component analysis based on loss function in the feature space is proposed in Section2. The model updating of kernel principal component analysis based on forgetting factor is proposed in Section 3. Improved kernel principal component analysis algorithm based on penalty factor is proposed in Section 4. Fault monitoring method is proposed in Section 5. The experiment results are given to show the effectiveness of the proposed method in Section 6. Finally, conclusions are summarized in Section 7.


## 2. Kernel Principal Component Analysis Based on Loss Function in the Feature Space

In practical industrial process, outliers are contained in the collected data, while traditional kernel principal component analysis method is based on the assumption that there are no outliers in the sample data [29–32]. The so-called outliers usually refer to the samples whose reconstruction error is much larger than average values and proportions are very small. Outliers still exist even after mapping them into feature space. Even if the sample data contains only a small amount of outliers, great negative effect will be applied on process model. Therefore, an improved kernel principal component analysis method is proposed in this paper, which defines a loss function in feature space in the sense of minimum reconstruction error.


### 2.1. Kernel Principal Component Analysis

In 1909, Mercer demonstrated the concept of positive definite kernel function and regeneration kernel Hilbert space in terms of the math and listed the necessary and sufficient condition of existence and determination of the positive definite kernel function, which is called “Mercer kernel permit conditions.” Kernel method not only has been widely used in differential geometry, differential equation, group theory, and many other mathematical disciplines and in signal processing, machine learning, the Gaussian process analysis, and many other applications but also had the very big breakthrough. Kernel methods of theoretical research and practical application attract more and more attention of scholars and experts. Scholkopf combined the kernel method with principal component analysis and formed the theory of kernel principal component analysis method.Kernel principal component analysis as the extension of principal component analysis maintains the various mathematical and statistical properties of linear principal component analysis. KPCA uses the nonlinear mapping of data to a high dimensional eigenspace to achieve kernel matrix diagonalization and then carries on the principal component analysis. There is no need to calculate inner product of the sample data of nonlinear transformation and we can easily get the nonlinear principal component of the mapping data by the kernel function value between two data points.


### 2.2. Loss Function in the Feature Space

The sample number isN. X
∈
R
N
×
M is mapped into high dimensional space F
:
X
→
Φ
(
X
), where Φ
(
X
)
=
[
Φ
(
x
1
)
,
Φ
(
x
2
)
,
…
,
Φ
(
x
N
)
]. X is supposed to have been centralized processing. W is the transformation matrix, where ∥
W
∥
=
1. 〈
W
,
Φ
(
x
)
〉
W
=
W
W
T
Φ
(
x
) is reconstruction vector of Φ
(
x
); then the reconstruction error of Φ
(
x
) in the feature space is defined as follows:
(1)
∥
e
(
Φ
(
x
i
)
)
∥
2
=
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
=
Φ
(
x
i
)
·
Φ
(
x
i
)
-
2
W
W
T
Φ
(
x
i
)
Φ
(
x
i
)
+
(
W
W
T
)
(
W
W
T
)
Φ
(
x
i
)
Φ
(
x
i
)
=
k
(
x
i
,
x
i
)
-
2
W
W
T
k
(
x
i
,
x
i
)
+
(
W
W
T
)
2
k
(
x
i
,
x
i
)
.In order to minimize the reconstruction error, here the loss function in feature space is defined as follows:(2)
J
1
(
W
)
=
∑
i
=
1
N
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
.


### 2.3. Kernel Principal Component Analysis Based on Loss Function

Formula (2) is expanded as follows:
(3)
J
1
(
W
)
=
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
∥
W
∥
2
-
2
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
+
∑
i
=
1
N
(
Φ
(
x
i
)
)
2
=
-
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
+
∑
i
=
1
N
∥
Φ
(
x
i
)
∥
2
,
where ∑
i
=
1
N
∥
Φ
(
x
i
)
∥
2 is constant. Therefore J
1
(
W
), the loss function in the feature space, is the minimum, when W
T
Φ
(
x
i
) is the maximum. This is equivalent to solving the following optimization problem:
(4)
max
⁡
w
,
z
1
2
γ
∑
i
=
1
N
(
z
i
)
2
-
1
2
W
T
W
s
.
t
.
z
i
=
W
T
Φ
(
x
i
)
.By Lagrange multiplier method, it is obtained as follows:(5)
L
(
W
,
z
,
α
)
=
1
2
γ
∑
i
=
1
N
z
i
2
-
1
2
W
T
W
-
∑
i
=
1
N
α
i
(
z
i
-
W
T
Φ
(
x
i
)
)
.Then(6)
∂
L
∂
W
=
0
⟹
W
=
∑
i
=
1
N
α
i
Φ
(
x
i
)
∂
L
∂
z
i
=
0
⟹
α
i
=
γ
z
i
∂
L
∂
α
i
=
0
⟹
z
i
-
W
T
Φ
(
x
i
)
=
0
,
i
=
1,2
,
…
,
N
.By eliminatingW and z, it is obtained as follows:
(7)
1
γ
α
i
-
∑
i
=
1
N
α
l
Φ
(
x
l
)
T
Φ
(
x
i
)
=
0
,
i
=
1,2
,
…
,
N
.It is defined asλ
=
1
/
γ and then obtained as follows:
(8)
K
α
=
λ
α
,
where K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
,
i
,
j
=
1,2
,
…
,
N. The result of α is the eigenvector of K in the above formula.Becauseα satisfies the normalization condition λ
k
(
α
k
,
α
k
)
=
1, then it is obtained as follows:
(9)
(
W
k
,
W
k
)
=
∑
i
,
j
=
1
N
α
i
k
α
j
k
(
Φ
(
x
i
)
,
Φ
(
x
j
)
)
=
∑
i
,
j
=
1
N
α
i
k
α
j
k
K
(
x
i
,
x
j
)
=
λ
k
(
α
k
,
α
k
)
=
1
.In this way, the projection ofΦ
(
x
) in the kth principal component W
k is as follows:
(10)
t
k
=
〈
W
k
,
Φ
-
(
x
)
〉
=
∑
i
=
1
N
α
i
k
〈
Φ
-
(
x
i
)
,
Φ
-
(
X
)
〉
=
∑
i
=
1
N
α
i
k
k
(
x
i
,
X
)
.


## 2.1. Kernel Principal Component Analysis

In 1909, Mercer demonstrated the concept of positive definite kernel function and regeneration kernel Hilbert space in terms of the math and listed the necessary and sufficient condition of existence and determination of the positive definite kernel function, which is called “Mercer kernel permit conditions.” Kernel method not only has been widely used in differential geometry, differential equation, group theory, and many other mathematical disciplines and in signal processing, machine learning, the Gaussian process analysis, and many other applications but also had the very big breakthrough. Kernel methods of theoretical research and practical application attract more and more attention of scholars and experts. Scholkopf combined the kernel method with principal component analysis and formed the theory of kernel principal component analysis method.Kernel principal component analysis as the extension of principal component analysis maintains the various mathematical and statistical properties of linear principal component analysis. KPCA uses the nonlinear mapping of data to a high dimensional eigenspace to achieve kernel matrix diagonalization and then carries on the principal component analysis. There is no need to calculate inner product of the sample data of nonlinear transformation and we can easily get the nonlinear principal component of the mapping data by the kernel function value between two data points.


## 2.2. Loss Function in the Feature Space

The sample number isN. X
∈
R
N
×
M is mapped into high dimensional space F
:
X
→
Φ
(
X
), where Φ
(
X
)
=
[
Φ
(
x
1
)
,
Φ
(
x
2
)
,
…
,
Φ
(
x
N
)
]. X is supposed to have been centralized processing. W is the transformation matrix, where ∥
W
∥
=
1. 〈
W
,
Φ
(
x
)
〉
W
=
W
W
T
Φ
(
x
) is reconstruction vector of Φ
(
x
); then the reconstruction error of Φ
(
x
) in the feature space is defined as follows:
(1)
∥
e
(
Φ
(
x
i
)
)
∥
2
=
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
=
Φ
(
x
i
)
·
Φ
(
x
i
)
-
2
W
W
T
Φ
(
x
i
)
Φ
(
x
i
)
+
(
W
W
T
)
(
W
W
T
)
Φ
(
x
i
)
Φ
(
x
i
)
=
k
(
x
i
,
x
i
)
-
2
W
W
T
k
(
x
i
,
x
i
)
+
(
W
W
T
)
2
k
(
x
i
,
x
i
)
.In order to minimize the reconstruction error, here the loss function in feature space is defined as follows:(2)
J
1
(
W
)
=
∑
i
=
1
N
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
.


## 2.3. Kernel Principal Component Analysis Based on Loss Function

Formula (2) is expanded as follows:
(3)
J
1
(
W
)
=
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
∥
W
∥
2
-
2
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
+
∑
i
=
1
N
(
Φ
(
x
i
)
)
2
=
-
∑
i
=
1
N
(
W
T
Φ
(
x
i
)
)
2
+
∑
i
=
1
N
∥
Φ
(
x
i
)
∥
2
,
where ∑
i
=
1
N
∥
Φ
(
x
i
)
∥
2 is constant. Therefore J
1
(
W
), the loss function in the feature space, is the minimum, when W
T
Φ
(
x
i
) is the maximum. This is equivalent to solving the following optimization problem:
(4)
max
⁡
w
,
z
1
2
γ
∑
i
=
1
N
(
z
i
)
2
-
1
2
W
T
W
s
.
t
.
z
i
=
W
T
Φ
(
x
i
)
.By Lagrange multiplier method, it is obtained as follows:(5)
L
(
W
,
z
,
α
)
=
1
2
γ
∑
i
=
1
N
z
i
2
-
1
2
W
T
W
-
∑
i
=
1
N
α
i
(
z
i
-
W
T
Φ
(
x
i
)
)
.Then(6)
∂
L
∂
W
=
0
⟹
W
=
∑
i
=
1
N
α
i
Φ
(
x
i
)
∂
L
∂
z
i
=
0
⟹
α
i
=
γ
z
i
∂
L
∂
α
i
=
0
⟹
z
i
-
W
T
Φ
(
x
i
)
=
0
,
i
=
1,2
,
…
,
N
.By eliminatingW and z, it is obtained as follows:
(7)
1
γ
α
i
-
∑
i
=
1
N
α
l
Φ
(
x
l
)
T
Φ
(
x
i
)
=
0
,
i
=
1,2
,
…
,
N
.It is defined asλ
=
1
/
γ and then obtained as follows:
(8)
K
α
=
λ
α
,
where K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
,
i
,
j
=
1,2
,
…
,
N. The result of α is the eigenvector of K in the above formula.Becauseα satisfies the normalization condition λ
k
(
α
k
,
α
k
)
=
1, then it is obtained as follows:
(9)
(
W
k
,
W
k
)
=
∑
i
,
j
=
1
N
α
i
k
α
j
k
(
Φ
(
x
i
)
,
Φ
(
x
j
)
)
=
∑
i
,
j
=
1
N
α
i
k
α
j
k
K
(
x
i
,
x
j
)
=
λ
k
(
α
k
,
α
k
)
=
1
.In this way, the projection ofΦ
(
x
) in the kth principal component W
k is as follows:
(10)
t
k
=
〈
W
k
,
Φ
-
(
x
)
〉
=
∑
i
=
1
N
α
i
k
〈
Φ
-
(
x
i
)
,
Φ
-
(
X
)
〉
=
∑
i
=
1
N
α
i
k
k
(
x
i
,
X
)
.


## 3. The Model Updating of Kernel Principal Component Analysis Based on Forgetting Factor

When the running state of the process changes in multivariate statistical process monitoring, regardless of the system changing slowly or fast, mean and covariance matrix of the model will change. Therefore, when the system changes, it needs to update mean and covariance of the sample data set.Firstly, the method of updating PCA model based on forgetting factor is introduced. Then KPCA model updating method based on forgetting factor will be got by kernel method. When new samples are collected in the process, mean and covariance will change. These changes depend on the change degree of model structure, namely, the size of the forgetting factor. For time-varying Gaussian process, therefore, mean and covariance of forgetting factor in estimate time can be used. Its formula is as follows:(11)
m
t
=
∑
i
=
1
t
α
t
-
i
x
i
∑
i
=
1
t
α
t
-
i
=
x
t
+
α
x
t
-
1
+
⋯
+
α
t
-
1
x
1
1
+
α
+
⋯
+
α
t
-
1
,
S
t
=
∑
i
=
1
t
β
t
-
1
(
x
i
-
m
i
)
(
x
i
-
m
i
)
T
∑
i
=
1
t
β
t
-
i
,
where α, β are two forgetting factors, m
t and S
t are mean vector and covariance matrix of t. Mean vector and covariance matrix of the sample data can be in a more convenient form, namely, the weighted sum of mean and covariance matrix in t
-
1 moment and sample data in t moment. Its formula is as follows:
(12)
m
t
=
1
-
α
1
-
α
t
x
t
+
α
1
-
α
t
-
1
1
-
α
t
m
t
-
1
,
S
t
=
1
-
β
1
-
β
t
x
-
t
x
-
t
T
+
β
1
-
β
t
-
1
1
-
β
t
S
t
-
1
,
where x
-
t
=
x
t
-
m
t is the new centralized sample of t moment. As t increases gradually, formula (12) can be further simplified as
(13)
m
t
=
(
1
-
α
)
x
t
+
α
m
t
-
1
,
(14)
S
t
=
(
1
-
β
)
x
-
t
x
-
t
T
+
S
t
-
1
.It can be seen from formula (14) that if only considering the sample covariance matrix, then it is obtained as
(15)
D
t
=
(
1
-
β
)
x
-
t
x
-
t
T
+
β
S
t
-
1
,
where D
t is a diagonal matrix, whose diagonal elements are the same as diagonal elements of S
t. The correlation coefficient matrix can also be estimated as follows:
(16)
R
t
=
∑
i
=
1
t
β
t
-
i
D
i
-
1
/
2
x
-
i
x
-
i
T
D
i
-
1
/
2
∑
i
=
1
t
β
t
-
i
.Ast increases gradually, formula (16) can be further simplified as
(17)
R
t
=
(
1
-
β
)
D
t
-
1
/
2
x
-
t
x
-
t
T
D
t
-
1
/
2
+
β
R
t
-
1
.As shown in formula (11), the updating of sample date set’s mean and covariance matrix needs determining two weighting coefficients, which are called the forgetting factors. If both forgetting factors are 1, its mean vector and covariance matrix will have the highest similarity degree with mean vector and covariance matrix calculated by all the sample data. If forgetting factors less than 1, as the process running, the weight of adding on the old data will be smaller and smaller, until being eliminated automatically, without discarding the old data by human. Then old data will gradually reduce the influence on process model and even disappear, which ensures that the model is adaptive to the time-varying system. When forgetting factors are closer to 1, there will be more number of sample data taking effect on the current process model.So far, most of the model updating method is based on the constant value forgetting factor which is acquired by experience. However, the optimal value of forgetting factor depends on the degree of process change. Because process changes at different levels, the optimal value of forgetting factor will have a significant difference. When the process changes rapidly, the update rate of mode should be very large, namely, a few new data have the main influence on process model. When process changes slowly, update rate of the model should be smaller, namely, most of sample data has influence on process model. The basic process information will be in a long time to maintain its effectiveness. But the degree of process changes is changing with time in the actual industrial process, and then forgetting factor should be determined according to the actual situation of process changes. In order to deal with process which has nonconstant change degree, constant forgetting factor will not be used. Forgetting factor which is adjusted to different degree of process changes is being used. Here, Fortescue’s method is used to adjust the forgetting factor. The method adopts the model updating based on previous factor. It has two features:(
1
) forgetting factor can take different values, which bring a degree of flexibility to the model. (
2
) Forgetting factor value is decided by the change of mean and covariance matrix directly and is not dependent on T
2 and SPE statistic values. Apply the same concept to recursive principal component analysis method. Then the calculation method of the sample data set’s mean and covariance update is obtained as
(18)
α
t
=
α
max
⁡
-
(
α
max
⁡
-
α
min
⁡
)
[
1
-
exp
⁡
{
-
k
(
∥
Δ
m
t
-
1
∥
∥
Δ
m
nor
∥
)
n
}
]
,
where α
max
⁡ and α
min
⁡ are, respectively, the forgetting factor’s maximum value and minimum value. k and n are parameters of the function. ∥
Δ
m
∥ is the Euclidean norm of the difference between two consecutive mean vectors. ∥
Δ
m
nor
∥ is mean based on historical data ∥
Δ
m
∥. In the same way, forgetting factor β which is used to update the covariance matrix can be calculated according to the formula as follows:
(19)
β
t
=
β
max
⁡
-
(
β
max
⁡
-
β
min
⁡
)
[
1
-
exp
⁡
{
-
k
(
∥
Δ
R
t
-
1
∥
∥
Δ
R
nor
∥
)
n
}
]
,
where β
max
⁡ and β
min
⁡ are, respectively, the forgetting factor’s maximum value and minimum value. ∥
Δ
R
∥ is the Euclidean norm of the difference between two consecutive correlation coefficient matrices. It can be seen that four parameters (α
max
⁡ (or β
max
⁡), α
min
⁡ (or β
min
⁡), k, and n) in formulae (18) and (19) need to be determined. The default values are α
max
⁡
=
0.99, α
min
⁡
=
0.1, k
=
0.6931, and n
=
1.This method is introduced to the model updating of KPCA, combining with exponential weighting KPCA method. Then the update KPCA method based on forgetting factor is obtained. Let kernel matrix beK
t
-
1 at t
-
1 moment. According to exponential weighting KPCA method, recursive update formula of kernel matrix at t moment is as follows:
(20)
K
t
=
γ
t
K
t
-
1
+
(
1
-
γ
t
)
k
-
t
T
k
-
t
,
where γ
t is the weighting factor; it can be calculated according to the formula as follows:
(21)
γ
t
=
γ
max
⁡
-
(
γ
max
⁡
-
γ
min
⁡
)
[
1
-
exp
⁡
(
-
k
(
∥
Δ
R
∥
)
)
]
,
where γ
min
⁡
=
0.1, γ
max
⁡
=
0.99, and ∥
Δ
R
∥ is the Euclidean norm of the difference between two consecutive correlation coefficient matrices. The sensitivity of model is controlled by k, whose default value is k
=
0.6931.


## 4. Improved Kernel Principal Component Analysis Algorithm Based on Penalty Factor

KPCA algorithm based on eigenvalue decomposition is a batch-mode algorithm, which needs to know all the sample points before modeling. It is not suitable for online monitoring or samples increased gradually. And KPCA is often based on the assumption that sample is not contaminated by outliers. There are outliers in the actual samples, in this paper, an iterative kernel principal component analysis method based on penalty factor is presented to solve the problem of outliers in the sample.


### 4.1. Iterative Kernel Principal Component Analysis Algorithm

For the loss function defined by formula (3), the stochastic gradient descent method is used to solve the optimization problem as follows:
(22)
d
J
1
(
W
)
d
W
=
2
(
W
W
T
K
-
K
)
W
.Then iterative formula is as follows:(23)
W
n
+
1
=
W
n
+
μ
n
(
K
-
W
n
W
n
T
K
)
W
n
,
where μ
n is the iteration step length, 0
<
μ
n
<
1, and W
n is convergence to the first nonlinear principal component.Because the nonlinear principal components are orthogonal to each other, Schmidt orthogonal method is used to calculate thejth principal component W
j:
(24)
W
n
+
1
j
=
W
n
j
+
μ
n
(
K
j
-
W
n
j
W
n
j
T
K
j
)
W
n
j
,
(25)
K
j
=
K
-
∑
i
=
1
j
-
1
W
i
W
i
T
K
.Steps of iterative KPCA algorithm can be summarized as follows.(1)
Input: sample data setX, maximum iterations n
max
⁡, the initial iteration step n
=
1, and principal component j
=
1.
(2)
Calculate the kernel matrixK, where [
K
]
i
j
=
K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
=
k
(
x
i
,
x
j
). Then carry on the centralized processing K
-
=
K
-
1
N
K
-
K
1
N
+
1
N
K
1
N, where
(26)
1
N
=
1
N
[
1
⋯
1
⋮
⋱
⋮
1
⋯
1
]
︸
N
.
(3)
Calculate thejth principal component W
j.
(4)
|
W
n
+
1
j
-
W
n
j
|
>
ε and n
<
n
max
⁡
n
=
n
+
1 and return to step (
3
). |
W
n
+
1
j
-
W
n
j
|
≤
ε output: W
n
+
1
j.
(5)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≥
ξ
,
j
=
j
+
1
. Return to step (
3
) and calculate next principal component.
(6)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≤
ξ. Terminate the iteration and output W.


### 4.2. Iterative Kernel Principal Component Analysis Algorithm Based on Penalty Factor

Although there are little outliers in sample data in KPCA algorithm, it also has great influence on KPCA model. The calculated principal components are towards the direction of outliers in order to reduce the overall square errors in the process of calculating principal components. In order to reduce the influence of the outliers on KPCA model, the penalty factor is added to the square error formula in the feature space.Penalty factorη
(
1
-
C
i
) is added to formula (3):
(27)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
,
where η is the predefined threshold and η
>
0. C
i is defined as follows:
(28)
C
i
=
{
1
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
≤
η
,
0
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
>
η
.By the above formula, after adding penalty factor, points which exceeded predefined thresholdη are seen as outliers. After setting J
2
(
W
) of outliers as η, the influence on KPCA model is reduced. Noticing that C
i is discrete, in order to use the proposed iterative KPCA to calculate principal components, continuous Sigmoid function is adopted to approximate discrete variable C
i.Minimum of error function in formula (25) is calculated, and iterative formula is obtained as follows:
(29)
W
n
+
1
=
W
n
+
μ
n
1
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
(
K
-
W
W
T
K
)
W
n
,
where 0
<
μ
n
<
1 is iteration step length, and 1
/
(
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
) is continuous Sigmoid function, which can adjust parameters according to the current input values and eliminate the influence of the outliers on KPCA model. So the smaller the threshold value η is, the more the sample points will be treated as outliers.Because the nonlinear principal components are orthogonal to each other, Schmidt orthogonal method is used to calculate thejth principal component W
j:
(30)
W
n
+
1
j
=
W
n
j
+
μ
n
1
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
(
K
j
-
W
W
T
K
j
)
W
n
j
,
K
j
=
K
0
-
∑
i
=
1
j
-
1
W
i
W
i
T
K
0
.Steps of iterative KPCA algorithm based on penalty factor can be summarized as follows.(1)
Input: sample data setX, maximum iterations n
max
⁡, the initial iteration step n
=
1, and principal component j
=
1.
(2)
Calculate the kernel matrixK, where [
K
]
i
j
=
K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
=
k
(
x
i
,
x
j
). Then carry on the centralized processing K
-
=
K
-
1
N
K
-
K
1
N
+
1
N
K
1
N, where
(31)
1
N
=
1
N
[
1
⋯
1
⋮
⋱
⋮
1
⋯
1
]
︸
N
.
(3)
Calculate thejth principal component W
j. If j
=
1, formula (29) is used to calculate the first principal component W
1. From the second iteration, sample reconstruction error is calculated in each time. Take points with ratio of r as outliers, determine the threshold η, and calculate penalty factor for iteration. Then the first principal component is obtained.
(4)
Ifj
=
2,3
,
…
,
l, formula (30) is used to calculate the rest of principal components W
2
,
W
3
,
…
,
W
l.
(5)
|
W
n
+
1
j
-
W
n
j
|
>
ε and n
<
n
max
⁡
n
=
n
+
1 and return to step (
3
). |
W
n
+
1
j
-
W
n
j
|
≤
ε output: W
n
+
1
j.
(6)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≥
ξ
,
j
=
j
+
1
. Return to step (
3
) and calculate next principal component.
(7)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≤
ξ. Terminate the iteration and output W.


## 4.1. Iterative Kernel Principal Component Analysis Algorithm

For the loss function defined by formula (3), the stochastic gradient descent method is used to solve the optimization problem as follows:
(22)
d
J
1
(
W
)
d
W
=
2
(
W
W
T
K
-
K
)
W
.Then iterative formula is as follows:(23)
W
n
+
1
=
W
n
+
μ
n
(
K
-
W
n
W
n
T
K
)
W
n
,
where μ
n is the iteration step length, 0
<
μ
n
<
1, and W
n is convergence to the first nonlinear principal component.Because the nonlinear principal components are orthogonal to each other, Schmidt orthogonal method is used to calculate thejth principal component W
j:
(24)
W
n
+
1
j
=
W
n
j
+
μ
n
(
K
j
-
W
n
j
W
n
j
T
K
j
)
W
n
j
,
(25)
K
j
=
K
-
∑
i
=
1
j
-
1
W
i
W
i
T
K
.Steps of iterative KPCA algorithm can be summarized as follows.(1)
Input: sample data setX, maximum iterations n
max
⁡, the initial iteration step n
=
1, and principal component j
=
1.
(2)
Calculate the kernel matrixK, where [
K
]
i
j
=
K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
=
k
(
x
i
,
x
j
). Then carry on the centralized processing K
-
=
K
-
1
N
K
-
K
1
N
+
1
N
K
1
N, where
(26)
1
N
=
1
N
[
1
⋯
1
⋮
⋱
⋮
1
⋯
1
]
︸
N
.
(3)
Calculate thejth principal component W
j.
(4)
|
W
n
+
1
j
-
W
n
j
|
>
ε and n
<
n
max
⁡
n
=
n
+
1 and return to step (
3
). |
W
n
+
1
j
-
W
n
j
|
≤
ε output: W
n
+
1
j.
(5)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≥
ξ
,
j
=
j
+
1
. Return to step (
3
) and calculate next principal component.
(6)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≤
ξ. Terminate the iteration and output W.


## 4.2. Iterative Kernel Principal Component Analysis Algorithm Based on Penalty Factor

Although there are little outliers in sample data in KPCA algorithm, it also has great influence on KPCA model. The calculated principal components are towards the direction of outliers in order to reduce the overall square errors in the process of calculating principal components. In order to reduce the influence of the outliers on KPCA model, the penalty factor is added to the square error formula in the feature space.Penalty factorη
(
1
-
C
i
) is added to formula (3):
(27)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
,
where η is the predefined threshold and η
>
0. C
i is defined as follows:
(28)
C
i
=
{
1
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
≤
η
,
0
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
>
η
.By the above formula, after adding penalty factor, points which exceeded predefined thresholdη are seen as outliers. After setting J
2
(
W
) of outliers as η, the influence on KPCA model is reduced. Noticing that C
i is discrete, in order to use the proposed iterative KPCA to calculate principal components, continuous Sigmoid function is adopted to approximate discrete variable C
i.Minimum of error function in formula (25) is calculated, and iterative formula is obtained as follows:
(29)
W
n
+
1
=
W
n
+
μ
n
1
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
(
K
-
W
W
T
K
)
W
n
,
where 0
<
μ
n
<
1 is iteration step length, and 1
/
(
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
) is continuous Sigmoid function, which can adjust parameters according to the current input values and eliminate the influence of the outliers on KPCA model. So the smaller the threshold value η is, the more the sample points will be treated as outliers.Because the nonlinear principal components are orthogonal to each other, Schmidt orthogonal method is used to calculate thejth principal component W
j:
(30)
W
n
+
1
j
=
W
n
j
+
μ
n
1
1
+
e
β
(
∥
e
n
(
Φ
(
x
)
)
∥
2
-
η
)
(
K
j
-
W
W
T
K
j
)
W
n
j
,
K
j
=
K
0
-
∑
i
=
1
j
-
1
W
i
W
i
T
K
0
.Steps of iterative KPCA algorithm based on penalty factor can be summarized as follows.(1)
Input: sample data setX, maximum iterations n
max
⁡, the initial iteration step n
=
1, and principal component j
=
1.
(2)
Calculate the kernel matrixK, where [
K
]
i
j
=
K
i
j
=
〈
Φ
(
x
i
)
,
Φ
(
x
j
)
〉
=
k
(
x
i
,
x
j
). Then carry on the centralized processing K
-
=
K
-
1
N
K
-
K
1
N
+
1
N
K
1
N, where
(31)
1
N
=
1
N
[
1
⋯
1
⋮
⋱
⋮
1
⋯
1
]
︸
N
.
(3)
Calculate thejth principal component W
j. If j
=
1, formula (29) is used to calculate the first principal component W
1. From the second iteration, sample reconstruction error is calculated in each time. Take points with ratio of r as outliers, determine the threshold η, and calculate penalty factor for iteration. Then the first principal component is obtained.
(4)
Ifj
=
2,3
,
…
,
l, formula (30) is used to calculate the rest of principal components W
2
,
W
3
,
…
,
W
l.
(5)
|
W
n
+
1
j
-
W
n
j
|
>
ε and n
<
n
max
⁡
n
=
n
+
1 and return to step (
3
). |
W
n
+
1
j
-
W
n
j
|
≤
ε output: W
n
+
1
j.
(6)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≥
ξ
,
j
=
j
+
1
. Return to step (
3
) and calculate next principal component.
(7)
J
2
(
W
)
=
∑
i
=
1
N
(
C
i
∥
Φ
(
x
i
)
-
W
W
T
Φ
(
x
i
)
∥
2
+
η
(
1
-
C
i
)
)
≤
ξ. Terminate the iteration and output W.


## 5. Fault Monitoring Method

This section provides fault monitoring method using the proposed iterative kernel principal component analysis algorithm based on penalty factor. It can be broadly divided into offline modeling phase and online monitoring phase.


### 5.1. Offline Modeling Phase

(1)
KPCA model is established based on historical data, and the initial standardization of kernel matrix is obtained.
(2)
Set the ratio of outliersr, determine the threshold η, and calculate penalty factor and principal components W.
(3)
CalculateT
2 and SPE statistics and the corresponding control limits.


### 5.2. Online Monitoring Phase

(1)
Calculate the meanm
t
-
1, covariance matrix S
t
-
1, T
lim
⁡
,
t
-
1
2, and SP
E
lim
⁡
,
t
-
1 at t
-
1 moment.
(2)
Collect new samples data oft moment; calculate forgetting factors α
t, β
t, the mean m
t, and covariance matrix S
t.
(3)
Obtain kernel matrixk
t of new samples and standardize it.
(4)
Use the method of iteration kernel principal component to update KPCA model and calculateT
2 and SPE statistics and the corresponding control limits T
lim
⁡
,
t
2, SP
E
lim
⁡
,
t.
(5)
Collect new sample data and return to step(
3
).The flow chart of improved KPCA algorithm is shown in Figure1.Figure 1
Flow chart of improved kernel principal component analysis method.


## 5.1. Offline Modeling Phase

(1)
KPCA model is established based on historical data, and the initial standardization of kernel matrix is obtained.
(2)
Set the ratio of outliersr, determine the threshold η, and calculate penalty factor and principal components W.
(3)
CalculateT
2 and SPE statistics and the corresponding control limits.


## 5.2. Online Monitoring Phase

(1)
Calculate the meanm
t
-
1, covariance matrix S
t
-
1, T
lim
⁡
,
t
-
1
2, and SP
E
lim
⁡
,
t
-
1 at t
-
1 moment.
(2)
Collect new samples data oft moment; calculate forgetting factors α
t, β
t, the mean m
t, and covariance matrix S
t.
(3)
Obtain kernel matrixk
t of new samples and standardize it.
(4)
Use the method of iteration kernel principal component to update KPCA model and calculateT
2 and SPE statistics and the corresponding control limits T
lim
⁡
,
t
2, SP
E
lim
⁡
,
t.
(5)
Collect new sample data and return to step(
3
).The flow chart of improved KPCA algorithm is shown in Figure1.Figure 1
Flow chart of improved kernel principal component analysis method.


## 6. Experiment and Discussion

With the development of technology of melting, electrofused magnesia furnace has already gotten extensive application in the industry. Electrofused magnesia furnace refining technology can enhance the quality and increase the production variety. The working conditions of the electrofused magnesia furnace are changed frequently and have complex characteristics such as strong nonlinearity and multiple modes. Electrofused magnesia furnace production process is used for fault diagnosis to verify the effectiveness of the proposed statistical analysis of nonlinear processes based on penalty factor. The improved KPCA method is used to monitor the normal and failure condition, respectively. Process fault is introduced from the 700th sample. It is caused by abnormal electrode actuators. Current of electrofused magnesia furnace plunges sharply. Temperatures become abnormal.800 pieces of sample data in normal working condition is used to test the improved KPCA process monitoring method proposed in this paper. ThenT
2 and SPE statistics of improved KPCA method are obtained in normal working condition, as shown in Figure 2. It can be seen from Figure 2 that the changes of T
2 and SPE statistics in improved KPCA method are reduced. This is because penalty factors are used to punish deviation larger samples in the iterative calculation process of principal components. The distance between sample points and original points of principal components is reduced. Therefore, T
2 and SPE statistic fluctuation decrease. But only iterative KPCA is used to model without updating the control limits; fault alarms still exist in the process. If T
2 and SPE control limits are updated at the same time, the statistics will not overrun the limits obviously. Compared with traditional methods, the proposed method has better accuracy and lower fault alarm rate. Simulation results verify the feasibility of this method to eliminate outliers.Statistics ofT
2 and SPE process monitoring using improved KPCA in normal working condition.
(a)
(b)In order to monitor the process of fault condition, faults are added to sample data in the normal working condition. Process faults are introduced from the 700th sample. The parameters start to drift and change faster at this time. Improved KPCA and conventional KPCA are used to monitor the process of fault condition. And process monitoring charts are shown in Figure3. Figure 3 is statistics of T
2 and SPE process monitoring using improved KPCA in fault condition. From Figure 3 you can see under the condition of process parameter drift, when faults do not occur in the process, the improved KPCA method can eliminate the influence of outliers and better describe the process of change. When faults occur in the process, the improved KPCA method can accurately and timely find them. Compared with the traditional method, the improved KPCA method has better accuracy and lower fault alarm rate.Statistics ofT
2 and SPE process monitoring using improved KPCA in fault condition.
(a)
(b)


## 7. Conclusion

In order to solve the problem of outliers in the sample data, an improved KPCA method is proposed in this paper. The method is based on loss function in feature space. And forgetting factor is introduced into recursive update of kernel matrix. Then penalty factor is added to calculate the process monitoring model. Compared with conventional KPCA method, improved KPCA method proposed in this paper does further research on eliminating outliers. Iterative KPCA method is more suitable for online monitoring of the process. Adding the penalty factor has good effect in eliminating outliers. In this paper, MATLAB software is used to do simulation experiments, and the simulation results verify the feasibility of the method. The improvement of KPCA method is more useful in the process of monitoring contained outliers.



---

*Source: 945948-2014-09-30.xml*