# Online Regularized and Kernelized Extreme Learning Machines with Forgetting Mechanism

Authors: Xinran Zhou; Zijian Liu; Congxu Zhu
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

To apply the single hidden-layer feedforward neural networks (SLFN) to identify time-varying system, online regularized extreme learning machine (ELM) with forgetting mechanism (FORELM) and online kernelized ELM with forgetting mechanism (FOKELM) are presented in this paper. The FORELM updates the output weights of SLFN recursively by using Sherman-Morrison formula, and it combines advantages of online sequential ELM with forgetting mechanism (FOS-ELM) and regularized online sequential ELM (ReOS-ELM); that is, it can capture the latest properties of identified system by studying a certain number of the newest samples and also can avoid issue of ill-conditioned matrix inversion by regularization. The FOKELM tackles the problem of matrix expansion of kernel based incremental ELM (KB-IELM) by deleting the oldest sample according to the block matrix inverse formula when samples occur continually. The experimental results show that the proposed FORELM and FOKELM have better stability than FOS-ELM and have higher accuracy than ReOS-ELM in nonstationary environments; moreover, FORELM and FOKELM have time efficiencies superiority over dynamic regression extreme learning machine (DR-ELM) under certain conditions.


---

## Body

## 1. Introduction

Plenty of research work has shown that the single hidden-layer feedforward neural networks (SLFN) can approximate any function and form decision boundaries with arbitrary shapes if the activation function is chosen properly [1–3]. However, most of traditional approaches (such as BP algorithm) for training SLFN are slow due to their iterative steps. To train SLFN fast, Huang et al. proposed a learning algorithm called extreme learning machine (ELM), which randomly assigns the hidden nodes parameters (the input weights and hidden layer biases of additive networks or the centers and impact factors of RBF networks) and then determines the output weights by the Moore-Penrose generalized inverse [4, 5]. The original ELM is batch learning algorithm.For some practical fields where the training data are generated gradually, online sequential learning algorithms are preferred over batch learning algorithms as sequential learning algorithms do not require retraining whenever a new sample is received. Hence, Liang et al. developed a kind of online sequential ELM (OS-ELM) using the recursive least square [6]. OS-ELM for SLFN produced better generalization performance at faster learning speed compared with the previous sequential learning algorithms. Moreover, for time-varying environments, recently several incremental sequential ELMs are presented; they apply constant or adaptive forgetting factor [7, 8] or iteration approach [9] to strengthen new sample’s contribution on model. Speaking theoretically, they cannot eliminate old samples’ effect on model thoroughly. To let ELM study the latest properties of identified object, Zhao et al. developed online sequential ELM with forgetting mechanism (FOS-ELM) [10]. Fixed-memory extreme learning machine (FM-ELM) of Zhang and Wang [11] can be thought of as a special case of FOS-ELM with the parameter p in [10] being 1. Although experimental results show FOS-ELM has higher accuracy [10], it may encounter the matrix singularity problem and run unstably.As a variant of ELM, regularized ELM (RELM) [12–14], which is equivalent to the constrained optimization based ELM [15, 16] mathematically, absorbing the thought of structural risk minimization of statistical learning theory [17], can overcome the overfitting problem of ELM and provides better generalization ability than original ELM when noises or outliers exist in the dataset [12]. Furthermore, the regularized OS-ELM (ReOS-ELM) developed by Huynh and Won [13], which is equivalent to sequential regularized ELM (SRELM) [14] and least square incremental ELM (LS-IELM) [18] essentially, can avoid singularity problem.If the feature mapping in SLFN is unknown to users, the kernel based ELM (KELM) can be constructed [15, 16]. For the application where samples arrive gradually, Guo et al. developed kernel based incremental ELM (KB-IELM) [18].However, in time-varying or nonstationary applications, the newer training data usually carry more information about systems, and the older ones possibly carry less information, even misleading information; that is, the training samples usually have timeliness. The ReOS-ELM and KB-IELM cannot reflect the timeliness of sequential training data well. On the other hand, if a huge number of samples emerge, for KB-IELM, the required storage space for matrix will infinitely increase with learning ongoing and new samples arriving ceaselessly, and at last storage overflow will happen necessarily, so KB-IELM cannot be utilized at all under the circumstances.In this paper, we combine advantages of FOS-ELM and ReOS-ELM and propose online regularized ELM with forgetting mechanism (FORELM) for time-varying applications. FORELM can overcome the potential matrix singularity problem by using regularization and eliminate effections of the outdated data on model by incorporating forgetting mechanism. Like FOS-ELM, the ensemble skill also may be employed in FORELM to enhance its stability; that is, FORELM comprisesp (p≥1) ReOE-ELMs with forgetting mechanism, each of which trains a SLFN; the average of those outputs represents the final output of the ensemble of these SLFN. Additionally, forgetting mechanism also is incorporated into KB-IELM and online kernelized ELM with forgetting mechanism (FOKELM) is presented, which can deal with the problem of matrix expansion of KB-IELM. The designed FORELM and FOKELM update model recursively. The experimental results show the better performance of FORELM and FOKELM approach in nonstationary environments.It should be noted that our methods adjust the output weights of SLFN due to addition and deletion of the samples one by one, namely, learn and forget samples sequentially, and network architecture is fixed. They are completely different from those offline incremental ELMs (I-ELM) [19–21] and incremental RELM [22] seeking optimal network architecture by adding hidden nodes one by one and learning the data in batch mode.The rest of this paper is organized as follows. Section2 gives a brief review of the basic concepts and related works of ReOS-ELM and KB-IELM. Section 3 proposes new online learning algorithms, namely, FORELM and FOKELM. Performance evaluation is conducted in Section 4. Conclusions are drawn in Section 5.


## 2. Brief Review of the ReOS-ELM and KB-IELM

For simplicity, ELM based learning algorithm for SLFN with multiple input single output is discussed.The output of a SLFN withL hidden nodes (additive or RBF nodes) can be represented by
(1)f(x)=∑i=1LβiG(ai,bi,x)=h(x)β,x∈Rn,ai∈Rn,
where ai and bi are the learning parameters of hidden nodes, β=[β1,β2,…,βL]T is the vector of the output weights, and G(ai,bi,x) denotes the output of the ith hidden node with respect to the input x, that is, activation function. h(x)=[G(a1,b1,x),…,G(aL,bL,x)] is a feature mapping from the n-dimensional input space to the L-dimensional hidden-layer feature space. In ELM, ai and bi are randomly determined firstly.For a given set of distinct training data{(xi,yi)}1N⊂Rn×R, where xi is an n-dimensional input vector and yi is the corresponding scalar observation, the RELM, that is, constrained optimization based ELM, can be formulated as(2a)Minimize:LPELM=12∥β∥2+c12∥ξ∥2,(2b)Subject to:Hβ=Y-ξ,where ξ=[ξ1,ξ2,…,ξN]T denotes the training error. Y=[y1,y2,…,yN]T indicates the target value of all the samples. HN×L=[h(x1)T,…,h(xN)T]T is the mapping matrix for the inputs of all the samples. c is the regularization parameter (a positive constant).Based on the KKT theorem, the constrained optimization of (2a) and (2b) can be transferred to the following dual optimization problem:
(3)LDELM=12∥β∥2+c12∥ξ∥2-α(Hβ-Y+ξ),
where α=[α1,α2,…,αN] is the Lagrange multipliers vector. Using KKT optimality conditions, the following equations can be obtained:(4a)∂LDELM∂β=0⟶β=HTa,(4b)∂LDELM∂ξ=0⟶a=cξ,(4c)∂LDELM∂α=0⟶Hβ-Y+ξ=0.Ultimately, β  can be obtained as follows [12, 15, 16]:(5a)β=(c-1I+HTH)-1HTY(5b)orβ=HT(c-1I+HHT)-1Y.In order to reduce computational costs, whenN>L, one may prefer to apply solution (5a), and when N<L, one may prefer to apply solution (5b).If the feature mappingh(x) is unknown, one can apply Mercer’s conditions on RELM. The kernel matrix is defined as Ω=HHT and Ωij=h(xi)h(xj)T=K(xi,xj). Then, the output of SLFN by kernel based RELM can be given as
(6)f(x)=h(x)β=h(x)HT(c-1I+HHT)-1Y=[K(x,x1),K(x,x2),…,K(x,xN)](c-1I+Ω)-1Y.


### 2.1. ReOS-ELM

The ReOS-ELM, that is, SRELM and LS-IELM, can be retold as follows.For timek-1, let hi=h(xi), Hk-1=[hsThs+1T⋯hk-1T]T,  Yk-1=[ysys+1⋯yk-1]T; then, according to (5a), solution of RELM can be expressed as(7a)Pk-1=(c-1I+Hk-1THk-1)-1,(7b)βk-1=Pk-1Hk-1TYk-1.For timek, the new sample (xk,yk) arrives; thus Hk=[Hk-1ThkT]T, Yk=[Yk-1Tyk]T; applying Sherman-Morrison-Woodbury (SMW) formula [23], the current P and β can be computed as(8a)Pk=(c-1I+HkTHk)-1=(c-1I+Hk-1THk-1+hkThk)-1=(Pk-1-1+hkThk)-1=Pk-1-Pk-1hkThkPk-11+hkPk-1hkT,(8b)βk=PkHkTYk=Pk(Hk-1TYk-1+hkTyk)=Pk(Pk-1-1Pk-1Hk-1TYk-1+hkTyk)=Pk((Pk-1-hkThk)βk-1+hkTYk)=βk-1+PkhkT(yk-hkβk-1).


### 2.2. KB-IELM

For timek-1, let
(9)Ak-1=(c-1I+Ωk-1)=[c-1+K(xs,xs)K(xs,xs+1)⋯K(xs,xk-1)K(xs+1,xs)c-1+K(xs+1,xs+1)⋯K(xs+1,xk-1)⋮⋮⋮⋮K(xk-1,xs)K(xk-1,xs+1)⋯c-1+K(xk-1,xk-1)].For timek, the new sample (xk,yk) arrives; thus
(10)Ak=c-1I+Ωk=[Ak-1VkVkTvk],
where Vk=[K(xk,xs),K(xk,xs+1),…,K(xk,xk-1)]T, vk=K(xk,xk)+1/c. Using the block matrix inverse formula [23], Ak-1 can be calculated from Ak-1-1 as
(11)Ak-1=[Ak-1VkVkTvk]-1=[Ak-1-1+Ak-1-1Vkρk-1Vk+1TAk-1-1-Ak-1-1Vtρk-1-ρk-1VkTAk-1-1ρk-1],
where ρk=vk-VkTAk-1-1Vk.


## 2.1. ReOS-ELM

The ReOS-ELM, that is, SRELM and LS-IELM, can be retold as follows.For timek-1, let hi=h(xi), Hk-1=[hsThs+1T⋯hk-1T]T,  Yk-1=[ysys+1⋯yk-1]T; then, according to (5a), solution of RELM can be expressed as(7a)Pk-1=(c-1I+Hk-1THk-1)-1,(7b)βk-1=Pk-1Hk-1TYk-1.For timek, the new sample (xk,yk) arrives; thus Hk=[Hk-1ThkT]T, Yk=[Yk-1Tyk]T; applying Sherman-Morrison-Woodbury (SMW) formula [23], the current P and β can be computed as(8a)Pk=(c-1I+HkTHk)-1=(c-1I+Hk-1THk-1+hkThk)-1=(Pk-1-1+hkThk)-1=Pk-1-Pk-1hkThkPk-11+hkPk-1hkT,(8b)βk=PkHkTYk=Pk(Hk-1TYk-1+hkTyk)=Pk(Pk-1-1Pk-1Hk-1TYk-1+hkTyk)=Pk((Pk-1-hkThk)βk-1+hkTYk)=βk-1+PkhkT(yk-hkβk-1).


## 2.2. KB-IELM

For timek-1, let
(9)Ak-1=(c-1I+Ωk-1)=[c-1+K(xs,xs)K(xs,xs+1)⋯K(xs,xk-1)K(xs+1,xs)c-1+K(xs+1,xs+1)⋯K(xs+1,xk-1)⋮⋮⋮⋮K(xk-1,xs)K(xk-1,xs+1)⋯c-1+K(xk-1,xk-1)].For timek, the new sample (xk,yk) arrives; thus
(10)Ak=c-1I+Ωk=[Ak-1VkVkTvk],
where Vk=[K(xk,xs),K(xk,xs+1),…,K(xk,xk-1)]T, vk=K(xk,xk)+1/c. Using the block matrix inverse formula [23], Ak-1 can be calculated from Ak-1-1 as
(11)Ak-1=[Ak-1VkVkTvk]-1=[Ak-1-1+Ak-1-1Vkρk-1Vk+1TAk-1-1-Ak-1-1Vtρk-1-ρk-1VkTAk-1-1ρk-1],
where ρk=vk-VkTAk-1-1Vk.


## 3. The Proposed FORELM and FOKELM

When SLFN is employed to model online for time-varying system, training samples are not only generated one by one, but also often have the property of timeliness; that is, training data have a period of validity. Therefore, during the learning process by online sequential learning algorithm, the older or outdated training data, whose effectiveness is less or is lost after several unit times, should be abandoned, which is the idea of forgetting mechanism [10]. ReOS-ELM (i.e., SRELM or LS-IELM) and KB-IELM cannot reflect the timeliness of sequential training data. In this section, the forgetting mechanism is added to them to eliminate the outdated data that might have misleading or bad effect on built SLFN. On the other hand, for KB-IELM, to abandon samples can prevent matrix A-1 of (11) from expanding infinitely. The computing procedures of deleting sample are given, and the completed online regularized ELM and kernelized ELM with forgetting mechanism are presented.


### 3.1. Decremental RELM and FORELM

After RELM has studied a given numberz of samples and SLFN has been applied for prediction, RELM would discard the oldest sample (xs,ys) from samples set.LetH-=[hs+1Ths+2T⋯hkT]T,  Y-=[ys+1ys+2⋯yk]T; then Hk=[hsTH-T]T,  Yk=[ysY-T]T, and
(12)P-=(c-1I+H-TH-)-1=(c-1I+HkTHk-hsThs)-1=(Pk-1-hsThs)-1.
Furthermore, using SMW formula, then
(13)P-=Pk+PkhsThsPk1-hsPkhsT.
Moreover,
(14)β-=P-H-TY-=P-(HkTYk-hsTys)=P-(Pk-1PkHkTYk-hsTys)=P-((P--1+hsThs)βk-hsTys)=βk+P-hsT(hsβk-ys).
Next time, Pk+1 and βk+1 can be calculated from P- (viewed as Pk) and β- (viewed as βk) according to (8a) and (8b), respectively.Suppose that FORELM may consist ofp ReOS-ELMs with forgetting mechanism, by which SFLNs trained have the same output of hidden node G(a,b,x) and the same number of hidden nodes L. In the following FORELM algorithm, the variables and parameters with superscript r are relevant to the rth SFLN to be trained by the rth ReOS-ELM with forgetting mechanism. Synthesize ReOS-ELM and the decremental RELM; then we can get FORELM as follows.Step 1.
Initialization:(1)
Choose the hidden output functionG(a,b,x) of SFLN with the certain activation function and the number of hidden nodes L. Set the value of p.(2)
Randomly assign hidden parameters(air,bir), i=1,2,…,L, r=1,2,…,p.(3)
Determinec; set Pk-1r=cIL×L.Step 2.
Incrementally learn initialz-1 samples; that is, repeat the following procedure for z-1 times:(1)
Get current sample(xk,yk).(2)
Calculatehkr and Pkr: hkr=[G(a1r,b1r,xk),…,G(aLr,bLr,xk)]; calculate Pkr by (8a).(3)
Calculateβkr: if sample (xk,yk) is the first one, then βkr=Pkr(hkr)Tyk, else calculate βkr by (8b).Step 3.
Online modeling and prediction: repeat the following procedure during every step:(1)
Acquire currentyk, form new sample (xk,yk), and calculate hkr, Pkr, and βkr by (8a) and (8b).(2)
Prediction: formxk+1, the output of the rth SFLN; that is, prediction y^k+1r of yk+1 can be calculated by (1): y^k+1r=hr(xk+1)βkr, the final prediction y^k+1=∑r=1py^k+1r/p.(3)
Delete the oldest sample(xs,ys): calculate P-r and β-r by (13) and (14), respectively.


### 3.2. Decremental KELM and FOKELM

After KELM has studied the given numberz of samples and SLFN has been applied for prediction, KELM would discard the oldest sample (xs,ys) from samples set.LetΩ-ij=K(xs+i,xs+j),  i,j=1,2,…,k-s.  A-=Ω-+c-1I,  V-=[K(xs,xs+1),…,K(xs,xk)], v-=K(xs,xs)+1/c; then Ak can be written in the following partitioned matrix form:
(15)Ak=Ω+c-1I=[v-V-V-TA-].Moreover, using the block matrix inverse formula, such equation can be obtained:(16)Ak-1=Ω+c-1I=[v-V-V-TA-]-1=[ρ--1-ρ--1V-A--1-A--1VTρ--1A--1+A--1V-Tρ--1V-A--1],
where ρ-=v--V-A--1V-T.RewriteAk-1 in the partitioned matrix form as
(17)Ak-1=[Ak-1(1,1)Ak-1(1,2:end)Ak-1(2:end,1)Ak-1(2:end,2:end)].Compare (16) and (17); A--1 can be calculated as
(18)A--1=Ak-1(2:end,2:end)-A--1V-Tρ--1V-A--1=Ak-1(2:end,2:end)-(-A--1V-Tρ--1)(-ρ--1V-A--1)ρ--1=Ak-1(2:end,2:end)-Ak-1(2:end,1)×Ak-1(1,2:end)Ak-1(1,1).Next time, computeAk+1-1 from A--1 (viewed as Ak-1) according to (11).Integrate KB-IELM with the decremental KELM; further, we can obtain FOKELM as follows.Step 1.
Initialization: choose kernelK(xi,xj) with corresponding parameter values, and determine c.Step 2.
Incrementally learn initialz-1 samples: calculate Ak-1: if there exists a sample only, then Ak-1=1/(K(xk,xk)+c-1), else calculate Ak-1 by (11).Step 3.
Online modeling and prediction:(1)
Acquire new sample(xk,yk) and calculate Ak-1 by (11).(2)
Prediction: formxk+1, and calculate prediction y^k+1  of yk+1 by (6); namely,
(19)y^k+1=[K(xk+1,xk-z+1),…,K(xk+1,xk)]×Ak-1[yk-z+1,…,yk]T.(3)
Delete the oldest sample(xs,ys): calculate A--1 by (18).


## 3.1. Decremental RELM and FORELM

After RELM has studied a given numberz of samples and SLFN has been applied for prediction, RELM would discard the oldest sample (xs,ys) from samples set.LetH-=[hs+1Ths+2T⋯hkT]T,  Y-=[ys+1ys+2⋯yk]T; then Hk=[hsTH-T]T,  Yk=[ysY-T]T, and
(12)P-=(c-1I+H-TH-)-1=(c-1I+HkTHk-hsThs)-1=(Pk-1-hsThs)-1.
Furthermore, using SMW formula, then
(13)P-=Pk+PkhsThsPk1-hsPkhsT.
Moreover,
(14)β-=P-H-TY-=P-(HkTYk-hsTys)=P-(Pk-1PkHkTYk-hsTys)=P-((P--1+hsThs)βk-hsTys)=βk+P-hsT(hsβk-ys).
Next time, Pk+1 and βk+1 can be calculated from P- (viewed as Pk) and β- (viewed as βk) according to (8a) and (8b), respectively.Suppose that FORELM may consist ofp ReOS-ELMs with forgetting mechanism, by which SFLNs trained have the same output of hidden node G(a,b,x) and the same number of hidden nodes L. In the following FORELM algorithm, the variables and parameters with superscript r are relevant to the rth SFLN to be trained by the rth ReOS-ELM with forgetting mechanism. Synthesize ReOS-ELM and the decremental RELM; then we can get FORELM as follows.Step 1.
Initialization:(1)
Choose the hidden output functionG(a,b,x) of SFLN with the certain activation function and the number of hidden nodes L. Set the value of p.(2)
Randomly assign hidden parameters(air,bir), i=1,2,…,L, r=1,2,…,p.(3)
Determinec; set Pk-1r=cIL×L.Step 2.
Incrementally learn initialz-1 samples; that is, repeat the following procedure for z-1 times:(1)
Get current sample(xk,yk).(2)
Calculatehkr and Pkr: hkr=[G(a1r,b1r,xk),…,G(aLr,bLr,xk)]; calculate Pkr by (8a).(3)
Calculateβkr: if sample (xk,yk) is the first one, then βkr=Pkr(hkr)Tyk, else calculate βkr by (8b).Step 3.
Online modeling and prediction: repeat the following procedure during every step:(1)
Acquire currentyk, form new sample (xk,yk), and calculate hkr, Pkr, and βkr by (8a) and (8b).(2)
Prediction: formxk+1, the output of the rth SFLN; that is, prediction y^k+1r of yk+1 can be calculated by (1): y^k+1r=hr(xk+1)βkr, the final prediction y^k+1=∑r=1py^k+1r/p.(3)
Delete the oldest sample(xs,ys): calculate P-r and β-r by (13) and (14), respectively.


## 3.2. Decremental KELM and FOKELM

After KELM has studied the given numberz of samples and SLFN has been applied for prediction, KELM would discard the oldest sample (xs,ys) from samples set.LetΩ-ij=K(xs+i,xs+j),  i,j=1,2,…,k-s.  A-=Ω-+c-1I,  V-=[K(xs,xs+1),…,K(xs,xk)], v-=K(xs,xs)+1/c; then Ak can be written in the following partitioned matrix form:
(15)Ak=Ω+c-1I=[v-V-V-TA-].Moreover, using the block matrix inverse formula, such equation can be obtained:(16)Ak-1=Ω+c-1I=[v-V-V-TA-]-1=[ρ--1-ρ--1V-A--1-A--1VTρ--1A--1+A--1V-Tρ--1V-A--1],
where ρ-=v--V-A--1V-T.RewriteAk-1 in the partitioned matrix form as
(17)Ak-1=[Ak-1(1,1)Ak-1(1,2:end)Ak-1(2:end,1)Ak-1(2:end,2:end)].Compare (16) and (17); A--1 can be calculated as
(18)A--1=Ak-1(2:end,2:end)-A--1V-Tρ--1V-A--1=Ak-1(2:end,2:end)-(-A--1V-Tρ--1)(-ρ--1V-A--1)ρ--1=Ak-1(2:end,2:end)-Ak-1(2:end,1)×Ak-1(1,2:end)Ak-1(1,1).Next time, computeAk+1-1 from A--1 (viewed as Ak-1) according to (11).Integrate KB-IELM with the decremental KELM; further, we can obtain FOKELM as follows.Step 1.
Initialization: choose kernelK(xi,xj) with corresponding parameter values, and determine c.Step 2.
Incrementally learn initialz-1 samples: calculate Ak-1: if there exists a sample only, then Ak-1=1/(K(xk,xk)+c-1), else calculate Ak-1 by (11).Step 3.
Online modeling and prediction:(1)
Acquire new sample(xk,yk) and calculate Ak-1 by (11).(2)
Prediction: formxk+1, and calculate prediction y^k+1  of yk+1 by (6); namely,
(19)y^k+1=[K(xk+1,xk-z+1),…,K(xk+1,xk)]×Ak-1[yk-z+1,…,yk]T.(3)
Delete the oldest sample(xs,ys): calculate A--1 by (18).


## 4. Performance Evaluation

In this section, the performance of the presented FORELM and FOKELM is verified via the time-varying nonlinear process identification simulations. Those simulations are designed from the aspects of accuracy, stability, and computation complexity of the proposed FORELM and FOKELM by comparison with the FOS-ELM, ReOS-ELM (i.e., SRELM or LS-IELM), and dynamic regression extreme learning machine (DR-ELM) [24]. DR-ELM also is a kind of online sequential RELM and is designed by Zhang and Wang using solution (5b) of RELM and the block matrix inverse formula.All the performance evaluations were executed in MATLAB 7.0.1 environment running on Windows XP with Intel Core i3-3220 3.3 GHz CPU and 4 GB RAM.Simulation
1. The unknown identified system is a modified version of the one addressed in [25], by changing the constant and the coefficients of variables, to form a time-varying system, as done in [11]:
(20)y(k+1)={y(k)1+y(k)2+u(k)3k≤1002y(k)2+y(k)2+u(k)3100<k≤300y(k)1+2y(k)2+2u(k)3300<k.The system (20) can be expressed as follows:
(21)y(k)=f(x(k)),
where f(·) is a nonlinear function and x(k) is the regression input data vector
(22)x(k)W=[y(k-1),…,y(k-ny),u(k-nd),…,u(k-nu)],
with nd, nu, and ny being model structure parameters. Apply SLFN to approximate (20); accordingly (x(k),y(k)) is the learning sample (xk,yk) of SLFN.Denotek0=z+max⁡(ny,nu)-nd,  k1=k0+350. The input is set as follows:
(23)u(k)={rand(·)k≤k0sin(2π(k-k0)120)k0<k≤k1sin(2π(k-k1)50)k1<k,
where rand(·) generates random numbers which are uniformly distributed in the interval (0,1).In all experiments, the output of hidden node with respect to the inputx of a SLFN in (1) is set to the sigmoidal additive function; that is, G(a,b,x)=1/(1+exp⁡(a·x+b)), the components of a; that is, the input weights and bias b are randomly chosen from the range [-2,2]. In FOKELM, the Gaussian kernel function is applied; namely, K(xi,xj)=exp⁡(-∥xi-xj∥2/σ).The root-mean-square error (RMSE) of prediction and the maximal absolute prediction error (MAPE) are regarded as measure indices of model accuracy and stability, respectively. Consider(24)RMSE=1t2-t1+1∑i=t1t2(y^i-yi)2,MAPE=max⁡i=t1,…,t2|y^i-yi|,
where t1=k0+1, t2=650. Simulation is carried on for 650 instances. In model (21), nd=1, nu=2, and ny=3. Due to randomness of parameters a,  b, and u(t) during initial stage (k≤k0), the results of simulation must possess variation. For each approach, the results are averaged over 5 trials.ReOS-ELM does not discard any old sample; thus it hasz→∞.In offline modeling, the training samples set is fixed; thus one may search relative optimal values for model parameter of ELMs. Nevertheless, in online modeling for time-varying system, the training samples set is changing; it is difficult to choose optimal values for parameters in practice. Therefore, we set the same parameter of these ELMs with the same value manually; for example, their parametersC all are set to 250; then we compare their performances.RMSE and MAPE of the proposed ELMs and other aforementioned ELMs are listed in Tables1 and 2, respectively, and the corresponding running time (i.e., training time plus predicting time) of these various ELMs is given in Table 3.Table 1
RMSE comparison between the proposed ELMs and other ELMs on identification of process (20) with input (23).
z
50
70
100
150
200
∞
L
=
15
FOS-ELM (p=2)
0.3474
0.0967
0.0731
0.0975
0.0817
—
ReOS-ELM
—
—
—
—
—
0.0961
DR-ELM
0.0681
0.0683
0.0712
0.0833
0.0671
—
FORELM (p=2)
0.0850
0.0708
0.0680
0.0816
0.0718
—
L
=
25
FOS-ELM
0.9070
0.1808
0.1324
0.0821
0.0883
—
ReOS-ELM
—
—
—
—
—
0.0804
DR-ELM
0.0543
0.0670
0.0789
0.0635
0.0656
—
FORELM
0.0578
0.0697
0.0833
0.0653
0.0600
—
L
=
50
FOS-ELM
×
×
×
9.9541
0.4806
—
ReOS-ELM
—
—
—
—
—
0.0529
DR-ELM
0.0377
0.0351
0.0345
0.0425
0.0457
—
FORELM
0.0344
0.0320
0.0324
0.0457
0.0456
—
L
=
100
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0359
DR-ELM
0.0298
0.0306
0.0308
0.0391
0.0393
—
FORELM
0.0312
0.0263
0.0288
0.0405
0.0378
—
L
=
150
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0351
DR-ELM
0.0268
0.0270
0.0281
0.0417
0.0365
—
FORELM
0.0270
0.0276
0.0284
0.0378
0.0330
—
L
=
200
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0344
DR-ELM
0.0259
0.0284
0.0272
0.0363
0.0351
—
FORELM
0.0263
0.0289
0.0274
0.0355
0.0325
—
L
=
400
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0296
DR-ELM
0.0240
0.0255
0.0249
0.0362
0.0327
—
FORELM
0.0231
0.0248
0.0256
0.0372
0.0310
—
L
=
800
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0292
DR-ELM
0.0233
0.0253
0.0270
0.0374
0.0320
—
FORELM
0.0223
0.0256
0.0252
0.0407
0.0318
—
L
=
1000
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.0298
DR-ELM
0.0234
0.0263
0.0278
0.0418
0.0323
—
FORELM
0.0232
0.0238
0.0259
0.0372
0.0310
—
FOKELM
0.0236
0.0251
0.0267
0.0386
0.0331
—
“—” represents nondefinition or inexistence in the case.
“×” represents nullification owing to the too large RMSE or MAPE.Table 2
MAPE comparison between the proposed ELMs and other ELMs on identification of process (20) with input (23).
z
50
70
100
150
200
∞
L
=
15
FOS-ELM (p=2)
5.8327
0.6123
0.5415
1.0993
0.7288
—
ReOS-ELM
—
—
—
—
—
0.3236
DR-ELM
0.3412
0.4318
0.3440
0.7437
0.4007
—
FORELM (p=2)
0.2689
0.3639
0.3135
0.8583
0.4661
—
L
=
25
FOS-ELM
5.0473
1.7010
2.4941
0.9537
0.9272
—
ReOS-ELM
—
—
—
—
—
0.4773
DR-ELM
0.3418
0.3158
0.4577
0.5680
0.3127
—
FORELM
0.3391
0.2568
0.3834
0.6228
0.2908
—
L
=
50
FOS-ELM
×
×
×
209.8850
3.9992
—
ReOS-ELM
—
—
—
—
—
0.2551
DR-ELM
0.3876
0.3109
0.3808
0.5486
0.3029
—
FORELM
0.3478
0.2960
0.3145
0.5504
0.3514
—
L
=
100
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.2622
DR-ELM
0.2473
0.2748
0.2163
0.5471
0.3381
—
FORELM
0.3229
0.2506
0.2074
0.5501
0.2936
—
L
=
150
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.3360
DR-ELM
0.2725
0.2365
0.2133
0.5439
0.3071
—
FORELM
0.2713
0.2418
0.2050
0.5432
0.3133
—
L
=
200
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.2687
DR-ELM
0.2539
0.2454
0.2069
0.5443
0.3161
—
FORELM
0.2643
0.2370
0.2029
0.5443
0.3148
—
L
=
400
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.2657
DR-ELM
0.2520
0.2456
0.2053
0.5420
0.3260
—
FORELM
0.2277
0.2382
0.2285
0.5430
0.3208
—
L
=
800
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.3784
DR-ELM
0.2388
0.2401
0.3112
0.5415
0.3225
—
FORELM
0.2128
0.2325
0.3006
0.5416
0.3247
—
L
=
1000
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.4179
DR-ELM
0.2230
0.2550
0.3412
0.5431
0.3283
—
FORELM
0.2228
0.2525
0.3347
0.5412
0.3293
—
FOKELM
0.2397
0.2345
0.3125
0.5421
0.3178
—Table 3
Running time (second) comparison between the proposed ELMs and other ELMs on identification of process (20) with input (23).
z
50
70
100
150
200
∞
L
=
15
FOS-ELM (p=2)
0.0620
0.0620
0.0621
0.0621
0.0622
—
ReOS-ELM
—
—
—
—
—
0.0160
DR-ELM
0.2650
0.3590
0.5781
1.0790
1.6102
—
FORELM (p=2)
0.0620
0.0620
0.0620
0.0621
0.0622
—
L
=
25
FOS-ELM
0.0621
0.0621
0.0621
0.0623
0.0623
—
ReOS-ELM
—
—
—
—
—
0.0167
DR-ELM
0.2500
0.3750
0.6094
1.0630
1.6250
—
FORELM
0.0621
0.0621
0.0621
0.0622
0.0622
—
L
=
50
FOS-ELM
×
×
×
0.1250
0.1250
—
ReOS-ELM
—
—
—
—
—
0.0470
DR-ELM
0.2650
0.3906
0.6250
1.1100
1.6560
—
FORELM
0.1400
0.1562
0.1560
0.1250
0.1250
—
L
=
100
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.1250
DR-ELM
0.2813
0.4219
0.6719
1.2031
1.7380
—
FORELM
0.3750
0.4063
0.3440
0.3440
0.3280
—
L
=
150
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.2344
DR-ELM
0.3125
0.4540
0.7500
1.2970
1.9220
—
FORELM
0.7810
0.8003
0.6560
0.6720
0.6570
—
L
=
200
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
0.3901
DR-ELM
0.3290
0.5160
0.7810
1.3440
1.9690
—
FORELM
1.2702
1.2750
1.1250
1.0320
0.9375
—
L
=
400
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
1.7350
DR-ELM
0.3906
0.5781
0.8750
1.4680
2.2820
—
FORELM
6.6410
6.4530
6.2030
5.6090
5.1719
—
L
=
800
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
8.7190
DR-ELM
0.5470
0.7970
1.2660
2.0938
2.9840
—
FORELM
32.7820
31.6720
30.1400
27.5780
27.7500
—
L
=
1000
FOS-ELM
×
×
×
×
×
—
ReOS-ELM
—
—
—
—
—
13.6094
DR-ELM
0.6250
1.0000
1.4530
2.4380
3.4060
—
FORELM
51.4530
49.3440
46.8750
43.1880
38.9060
—
FOKELM
0.2811
0.4364
0.6951
1.1720
1.7813
—From Tables1–3, one can see the following results.(1)
RMSE and MAPE of FORELM are smaller than those of FOS-ELM with the samez and L values. The reason for this is that (HTH) in FOS-ELM may be (nearly) singular at some instances; thus (HTH)-1 calculated recursively is nonsense and unbelievable, and when L≥z, FOS-ELM cannot work owing to its too large RMSE or MAPE. Accordingly, “×” represents nullification in Tables 1–3, whereas FORELM does not suffer from such a problem. In addition, RMSE and MAPE of FOKELM also are smaller than those of FOS-ELM with the same z values.(2)
RMSE of FORELM and FOKELM is smaller than that of ReOS-ELM with the sameL and C when parameter z, namely, the length of sliding time windows, is set properly. The reason for this is that ReOS-ELM neglects timeliness of samples of the time-varying process and does not get rid of effects of old samples; contrarily, FORELM and FOKELM stress actions of the newest ones. It should be noticed that z is relevant to characteristics of the specific time-varying system and its inputs. In Table 1, when z=50, 70, or 100, the effect is good.(3)
Whenz is fixed, with L increasing, RMSEs of these ELMs tend to decrease firstly. But changes are not obvious later.(4)
FORELM requires nearly the same time as FOS-ELM, but more time than ReOS-ELM. It is because both FORELM and FOS-ELM involvep incremental and decremental learning procedures, but ReOS-ELM does one incremental learning procedure only.(5)
Both FORELM and DR-ELM use regularization trick, so they should obtain the same or similar prediction effect theoretically, and Tables1 and 2 also show they have similar simulation results statistically. However, their time efficiencies are different. From Table 3, it can be seen that, for the case where L is small or L≤z, FORELM takes less time than DR-ELM. Thus, when L is small and modeling speed is preferred to accuracy, one may try to employ FORELM.(6)
Whenz is fixed, if L is large enough, there are no significant differences between RMSE of FORELM and DR-ELM and RMSE of FOKELM with appropriate parameter values. In Table 3, it is obvious that, with L increasing, DR-ELM costs more and more time. But time cost by FOKELM is irrelevant with L. According to the procedures of DR-ELM and FOKELM, if L is large enough to make calculating h(xi)h(xj)T more complex than calculating K(xi,xj), FOKELM will take less time than DR-ELM.To intuitively observe and compare the accuracy and stability of these ELMs with the samea, b values and initial u(k)(k≤k0) signal, absolute prediction error (APE) curves of one trial of every approach (L=25, z=70) are shown in Figure 1. Clearly, Figure 1(a) shows that, at a few instances, prediction errors of FOS-ELM are much greater, although prediction errors are very small at other instances; thus FOS-ELM is unstable. Comparing Figures 1(b), 1(c), 1(d), and 1(e), we can see that, at most instances, prediction errors of DR-ELM, FORELM, and FOKELM are smaller than those of ReOS-ELM, and prediction effect of FORELM is similar to that of DR-ELM.APE curves of ELMs on process (20) with input (23).
(a)
APE curves of FOS-ELM
(b)
APE curves of ReOS-ELM
(c)
APE curves of DR-ELM
(d)
APE curves of FORELM
(e)
APE curves of FOKELMOn the whole, FORELM and FOKELM have higher accuracy than FOS-ELM and ReOS-ELM.Simulation
2. In this subsection, the proposed methods are tested on modeling for a second-order bioreactor process described by the following differential equations [26]:
(25)c˙1(t)=-c1(t)u(t)+c1(t)(1-c2(t))ec2(t)/γ,c˙2(t)=-c2(t)u(t)+c1(t)(1-c2(t))ec2(t)/γ1+β1+β-c2(t),
where c1(t) is the cell concentration that is considered as the output of the process (y(t)=c1(t)), c2(t) is the amount of nutrients per unit volume, and u(t) represents the flow rate as the control input (the excitation signal for modeling); the c1(t) and c2(t) can take values between zero and one, and u(t) is allowed a magnitude in interval [0,2].In the simulation, with the growth rate parameterβ=0.02, the nutrient inhibition parameter γ is considered as the time-varying parameter; that is,
(26)γ={0.30s<t≤20s0.4820s<t≤60s0.660s<t.LetTs indicate sampling interval. Denote t0=(z+max⁡(ny,nu)-nd)Ts,  t1=t0+350Ts. The input is set below:
(27)u(t)={0.5rand(·)+0.75t≤t00.25sin(2π(t-t0)24)+1t0<t≤t10.25sin(2π(t-t1)10)+1t1<t,
where, at every sampling instance, rand(·) generates random numbers which are uniformly distributed in the interval (0,1).SetTs=0.2 s. With the same a, b, and initial u(t), APE curves of every approach (L=20, z=100) for one trial are drawn in Figure 2. Clearly, on the whole, APE curves of FORELM and FOKELM are smaller than those of FOS-ELM and ReOS-ELM, and FORELM has nearly the same prediction effect as DR-ELM. Further, RMSE of FOS-ELM, ReOS-ELM, DR-ELM, FORELM, and FOKELM are 0.096241, 0.012203, 0.007439, 0.007619, and 0.007102, respectively.APE curves of ELMs on process (25) with time-varying γ in (26).
(a)
APE curves of FOS-ELM
(b)
APE curves of ReOS-ELM
(c)
APE curves of DR-ELM
(d)
APE curves of FORELM
(e)
APE curves of FOKELMThrough many comparative trials, we may attain the same results as the ones in Simulation 1.


## 5. Conclusions

ReOS-ELM (i.e., SRELM or LS-IELM) can yield good generalization models and will not suffer from matrix singularity or ill-posed problems, but it is unsuitable in time-varying applications. On the other hand, FOS-ELM, thanks to its forgetting mechanism, can reflect the timeliness of data and train SLFN in nonstationary environments, but it may encounter the matrix singularity problem and run unstably.In the paper, the forgetting mechanism is incorporated to ReOS-ELM, and we obtain FORELM which blends advantages of ReOS-ELM and FOS-ELM. In addition, the forgetting mechanism also is added to KB-IELM; consequently, FOKELM is obtained, which can overcome matrix expansion problem of KB-IELM.Performance comparison between the proposed ELMs and other ELMs was carried out on identification of time-varying systems in the aspects of accuracy, stability, and computational complexity. The experimental results show that FORELM and FOKELM have better stability than FOS-ELM and have higher accuracy than ReOS-ELM in nonstationary environments statistically. When the numberL of hidden nodes is small or L≤ the length  z of sliding time windows, FORELM has time efficiency superiority over DR-ELM. On the other hand, if L is large enough, FOKELM will be faster than DR-ELM.



---

*Source: 938548-2014-07-10.xml*