# An Improved Method for Stochastic Nonlinear System’s Identification Using Fuzzy-Type Output-Error Autoregressive Hammerstein–Wiener Model Based on Gradient Algorithm, Multi-Innovation, and Data Filtering Techniques

Authors: Donia Ben Halima Abid; Saif Eddine Abouda; Hanane Medhaffar; Mohamed Chtourou
Journal: Complexity
 (2021)

---

## Abstract

This paper proposes an innovative identification approach of nonlinear stochastic systems using Hammerstein–Wiener (HW) model with output-error autoregressive (OEA) noise. Two fuzzy systems are suggested for the identification of the input and output nonlinear blocks of a proposed model from given input-output data measurements. In this work, the need for the commonly used assumptions including well-known structure of input and/or output nonlinearities and/or reversible nonlinear output is eliminated by replacing the intermediate variables and noise with their estimates. Four parametric estimation algorithms to identify the proposed fuzzy-type stochastic output-error autoregressive HW (FSOEAHW) model are derived based on backpropagation algorithm and multi-innovation and data filtering identification techniques. The proposed algorithms are improved backpropagation gradient (IBPG) algorithm, multi-innovation IBPG (MIIBPG) algorithm, a data filtering IBPG (FIBPG) algorithm, and a multi-innovation-based FIBPG (MIFIBPG) algorithm. The convergence of the parameter estimation algorithms is studied. The effectiveness of the proposed algorithms is shown by a given simulation example.


---

## Body

## 1. Introduction

The system identification is an approach of modeling a dynamic system based on data observations [1–9]. It consists of finding a suitable mathematical model which approximates, as well as possible, the system’s behavior by giving an appropriate structure and parameters model, establishing a relation between inputs and outputs system measurements [10–13].All physics systems are nonlinear, and it is natural to use nonlinear model to describe such systems. Therefore, nonlinear identification and control techniques have received, recently, more attention, since linear identification and control-based approaches are widely developed and are becoming mature. One class of such nonlinear modeling system is so-called block-oriented models that can be represented in various configurations where linear dynamic blocks and nonlinear static or dynamic subsystems are cascaded. The Hammerstein (H) model (static nonlinear block followed by a dynamic linear one) and the Wiener (W) system (a linear dynamic subsystem followed by a static nonlinear block) are the basic class of the cascaded systems which are widely used in many industrial practice engineering applications [14–22] and, therefore, the modeling approaches of such class of block-oriented models have received great attention for many years [23–36]. Hammerstein and Wiener systems are combined together to produce more complex subcategories, namely, Hammerstein–Wiener (HW) model (a linear block is cascaded between two nonlinear subsystems) and a Wiener–Hammerstein (WH) system (a nonlinear block is embedded between two linear blocks). Particularly, in many cases, the real system contains both actuator and sensor nonlinearities. Then, it is appropriate to consider the input nonlinear block as actuator nonlinearity and the output nonlinear subsystem as sensor or/and process nonlinearity. The HW model has several advantages: (i) It has a physical view of the nonlinear characteristics of the real system, which is important in the analysis, monitoring, diagnosis, and control of the system. (ii) The HW dynamics are mainly produced by a linear subsystem. Then, developed linear approaches could be used. (iii) When the output nonlinearity has an inverse function, the linear control techniques can be easily applied with desired performances. This block-oriented model form is perfectly used in different fields, such as electrical, mechanical, hydraulic, and chemical fields [37–46]. Consequently, the identification of HW models has been an active research topic nowadays. In literature, different methods are proposed and can be roughly divided into different categories: the recursive, the iterative, the blind, the subspace, the frequency domain, the overparameterization, decomposition methods, and so on. The basic idea of the abovementioned methods is that the model parameters are approximated either by constructing hybrid model of nonlinear and linear parts (e.g., overparameterization, subspace, decomposition, and blind methods [36, 43–46]) or by separated steps where the estimation input and output nonlinear subsystems parameters and the dynamic linear part ones are established based on the unmeasured intermediate variables estimation step (such as frequency domain, iterative, recursive, stochastic, correlation methods, and a special input-based one [35, 47–51]).However, the common representation of the abovementioned works is that the process noise (i.e., noise given between the linear dynamic part and the nonlinear block) or output disturbances (i.e., measurement noise given after nonlinear output block) have not been considered, which is not the case in practical physical process. Then, it is more evident to consider some stochastic disturbances. Along these lines, a very few papers that deal with stochastic H-W model have been proposed in the literature. For example, in [43–49], the authors treated a particular structure of H-W presenting some forms of measurement noise. For H-W with process noise, some papers were proposed in [49–53]. Esmaeilani et al. and Wills et al. [54–56] have proposed identification methods for an H-W system’s class presenting measurement and process noises.It should be noted that the aforementioned approaches’ applications are severely limited due to some problems. In fact, correspondent-cited ideas are restrictive to polynomial forms of the input and/or output nonlinear blocks or well-known input and/or output nonlinear characteristics (like dead zone or backlash) but with unknown parameters. Therefore, if the nonlinearity is not continuous or not in the polynomial form or with unknown characteristics, the algorithms do not give satisfactory performances. Moreover, the redundancy problem producing an oversizing dimension of HW’s matrix parameters is another loophole of previously mentioned methods. For compensation of these shortcomings, artificial intelligent systems such as neural networks and fuzzy systems can be explored to model the H-W model owing to their universal approximation property and their ability to model a given nonlinear function to any arbitrary accuracy. Considering that the H-W structure should be ensured, the identification problem of neural network or fuzzy-type H-W is different (where the modeling of the two nonlinear blocks and the dynamic linear part are altogether needed) from that of traditional neural network and fuzzy system which focuses only on the global data process nonlinear transformation. Nowadays, only scattered works were reported in the identification of H-W model based on neural networks and fuzzy systems. In [57, 58], a multistage approach is proposed to establish a recurrent neural network type H-W model based on an active region boundary initialization, a frequency domain eigensystem realization, and least squares and recursive recurrent learning algorithms. A neuro-fuzzy-type H-W system was presented in [59] using two-stage input signal. For the same form of system, Jia et al. and Li et al. [60, 61] combined special input signal and correlation technique to identify separately and nonrecursively system parameters. However, most of the above-cited algorithms are based on some restrictive conditions, especially the prior knowledge of some parameters and the output nonlinearity’s invertibility assumption to obtain intermediate variable’s estimate, which is not always the case. Even in [62] a backpropagation gradient algorithm is used to estimate jointly unknown parameters and intermediate variables of neural network type H-W system with polynomial form of input and output nonlinear blocks.It is clear that there are some problems that need to be addressed in nonlinear process identification using H-W model to achieve satisfactory results. The first one is how to consider stochastic nonlinear process disturbances in the H-W model description. The second is how to identify H-W model without any prior system knowledge and using, only, input and output measurement. The third is how to acquire the powerful identification aptitude of the H-W model with minimum attractive theoretical assumptions. The last is to identify the learning algorithm that can be used to achieve good performance.To deal with the above-described issues, this paper presents a novel modeling and parameter identification method for the identification of nonlinear stochastic process described by fuzzy-type output-error autoregressive H-W (FSOEAHW) model. Thus, compared with the existent studies, the main contributions of this paper are as follows.The first originality lies in the proposed fuzzy-type H-W scheme. In fact, the linear part is considered as a discrete transfer function and an output-error autoregressive mathematical model describes the process disturbance. Moreover, two fuzzy models are designed to describe the input and output nonlinearities of H-W model with only input and output measurement knowledge. As a result, the proposed model can not only describe the dynamics of such nonlinear system operating in stochastic environment and avoid the encountered input and output nonlinear blocks restrictions based on particular or basis functions forms but also eliminate the reversibility assumption of the static output part based on internal variable estimates without parameter model oversizing problem.The second involvement is related to the proposed parameter estimation algorithms. The major idea is to combine a feedback propagation gradient algorithm with a multi-innovation and data filtering technique for the fuzzy-type H-W model identification. The proposed identification algorithms are based on input/output measurement and the approximation of all internal variables resulting from the preceding corresponding parameter estimates. In this instance, four algorithms are suggested: an improved backpropagation gradient (IBPG) algorithm, a multi-innovation improved backpropagation gradient (MIIBPG) algorithm for improving the convergence rate through the multi-innovation identification technique, a data filtering IBPG (FIBPG) algorithm, and a multi-innovation FIBPG (MIFIBPG).The remainder of this paper is organized as follows. The considered system is described in Section2. In Section 3, the FSOEAHW parameter estimation algorithms, IBPG, MIIBPG, FIBPG, and MIFIBPG, are developed and their convergence analysis is studied. Section 4 presents simulation results. Some conclusions are offered in Section 5.


## 2. System Description

### 2.1. Stochastic Output-Error Autoregressive Hammerstein–Wiener (SOEAHW) System

Consider a stochastic output-error autoregressive Hammerstein–Wiener (SOEAHW) system given by Figure1.Figure 1
SOEAHW model.In that system,uk and yk are system input and output, respectively. yk1, xk, and hk are internal variables. The discrete linear transfer function G. is surrounded by an input static nonlinear block f1. and an output static nonlinear block f2.. It is assumed that the measured output yk contains an unknown additive noise component wk described by an autoregressive mathematical model. vk is a white noise with zero mean and unknown variance σ2.The first unmeasurable intermediate variablehk is the output of input nonlinear block and it is expressed by the following equation:(1)hk=f1uk.The linear dynamic part is given by the following expression:(2)Gz=BzAz=b1z−1+b2z−2+⋯+bmz−nb1+a1z−1+a2z−2+⋯+anz−na.The second unmeasurable intermediate variable can be written as follows:(3)xk=Gzhk=−∑i=1naaixk−i+∑j=1nbbjhk−j.The output of the nonlinear output block is expressed as follows:(4)y1k=f2xk.Noisewk is given by(5)wk=1Czvk=11+c1z−1+⋯+cncz−ncvk=−∑p=1nccpwk−p+vk=cTψk+vk.We havec=c1,c2,…,cncT and ψk=−wk−1,−wk−2,…,−wk−ncT. Given equations (4) and (5), the system’s output is expressed as(6)yk=f2xk+wk=yk1+wk=yk1+cTψk+vk.The objective of this paper is to give the following:(i)
A good description of an unknown nonlinear system operating in a stochastic environment using HW model despite the presence of disturbance and the great model complexity, with minimum attractive theoretical assumptions that are not always valid, especially the rigorous restrictions related to the nonlinear output block’s reversibility and well-known input and/or output nonlinear blocks characteristics(ii)
A suitable approximation of the above-described system such as the sum of square residual term errorsE (given as follows) which should be reduced as possible using adequate algorithm(7)Ef^1,θ^,f^2=∑k=1Myk−y^k2,where y^k is a priori estimated output.To achieve the abovementioned objectives and based on the universal approximation properties of fuzzy systems, we propose using two fuzzy models to describe the input and output nonlinearities.


### 2.2. Fuzzy-Type Stochastic Output-Error Autoregressive Hammerstein–Wiener (FSOEAHW) Model

The crucial challenge for the identification of Hammerstein–Wiener model is to result in effective modeling of two static nonlinear functions. Better modeling requires not only approximating the nonlinear function accurately but also simplifying the identification process. In the literature, several researches are restrictive to a polynomial form of the input and/or output nonlinear blocks or well-known input and/or output nonlinear characteristics (such as dead zone or backlash) but with unknown parameters [41–60, 63–74]. Therefore, if the nonlinearity is not continuous or not in the polynomial form or with unknown characteristic, the algorithms do not give a satisfactory performance. For compensation of the aforementioned shortcomings encountered in the existent structure of Hammerstein–Wiener model, in this paper, we devoted two fuzzy models to describe the input and output nonlinearities of H-W model with only input and output measurement knowledge.In this section, we propose developing a modeling approach of a nonlinear dynamic process operating in the stochastic environment identification based on fuzzy model. According to the previous section, we propose a new fuzzy-type stochastic output-error autoregressive H-W (FSOEAHW) model given in Figure2. It consists of two static nonlinear blocks described by two independent fuzzy systems: a dynamic linear block and autoregressive noise block.Figure 2
Fuzzy-type stochastic output-error autoregressive Hammerstein–Wiener (FSOEAHW) model.Then the first fuzzy system’s outputhk can be formulated as follows:(8)hk=∑n=1N1Wn1gn1uk=W1Tφ1uk.Giving thatWn1n=1,2,…,N1 is the nth fuzzy rule’s consequence and N1 is the total rule number, gn1uk=μn1uk/∑i1=1N1μ1i1uk with μ1nuk=exp−1/2uk−o1n/σ1n2 is nth Gaussian membership function where on1 and σn1 present, respectively, the correspondent center and width. W1=W11,W21,…,W1N1T is a vector parameter, and φ1uk=g11uk,g21uk,…,g1N1ukT is an input information vector.It should be noted that the membership functions can be of several shapes such as triangular, trapezoidal, and Gaussian. The only condition that must be fulfilled is that it must be in the interval [0, 1]. In the literature, Gaussian shape is commonly used because of its simplicity, its smoothness, and nonzero at all points. It is defined by only two parameters (the center and the width) and it is a continuously differentiable function [75–77].The second unmeasurable intermediate variable of the FSOEAHW can be written as follows:(9)xk=−∑i=1naaixk−i+∑j=1nbbjhk−j=θlTφkl,where θl=a1,…,ana,b1,…,bnb and φkl=−xk−1,…,−xk−na,hk−1,…,hk−nb are, respectively, the vector parameters and the observation vector of the linear dynamic part.The output of the second fuzzy systemyk1 is expressed by(10)yk1=∑m=1N2Wm2gm2xk=W2Tφ2xk,where gm2xk=μm2xk/∑i2=1N2μ2i2xk, with μm2xk=exp−1/2xk−om2/σm22m=1,2,…,N2 in which om2 and σm2 are the center and width of the mth membership function μm2xk. W2=W12,W22,…,W2N2T and φ2xk=g12xk,g22xk,…,g2N2xkT are, respectively, the vector parameter and the observation vector of the output nonlinear block. Wm2 is the mth consequence fuzzy rule and N2 is the correspondent total rule number.Noisewk is expressed by (5). Thus, the system output can be rewritten as(11)yk=∑m=1N2Wm2gm2xk+wk=∑m=1N2Wm2gm2xk−∑p=1nccpwk−p+vk,or, equivalently, it can be written in the following matrix form:(12)yk=W2Tφ2xk+cTψk+vk=WTφk+vk.Hence, the parameter vector isW=W12,W22,…,W2N2,c1,c2,…,cncT and the observation vector is equal to φk=φ2xkT,ψkTT=g12xk,g12xk,…,g2N2xk,−wk−1,−wk−2,…,−wk−ncT.Note that all vector parameters and observation vectors (φ1uk, φ2xk, and φk) are unknown. The problem then consists in developing a novel identification algorithm estimating the unknown parameters and variables using the input measurement and output measurement uk and yk that respect the objectives cited in the previous paragraph. Founded on the estimated model parameters, the mathematical model could be built identifying a given practical stochastic nonlinear dynamical system.


## 2.1. Stochastic Output-Error Autoregressive Hammerstein–Wiener (SOEAHW) System

Consider a stochastic output-error autoregressive Hammerstein–Wiener (SOEAHW) system given by Figure1.Figure 1
SOEAHW model.In that system,uk and yk are system input and output, respectively. yk1, xk, and hk are internal variables. The discrete linear transfer function G. is surrounded by an input static nonlinear block f1. and an output static nonlinear block f2.. It is assumed that the measured output yk contains an unknown additive noise component wk described by an autoregressive mathematical model. vk is a white noise with zero mean and unknown variance σ2.The first unmeasurable intermediate variablehk is the output of input nonlinear block and it is expressed by the following equation:(1)hk=f1uk.The linear dynamic part is given by the following expression:(2)Gz=BzAz=b1z−1+b2z−2+⋯+bmz−nb1+a1z−1+a2z−2+⋯+anz−na.The second unmeasurable intermediate variable can be written as follows:(3)xk=Gzhk=−∑i=1naaixk−i+∑j=1nbbjhk−j.The output of the nonlinear output block is expressed as follows:(4)y1k=f2xk.Noisewk is given by(5)wk=1Czvk=11+c1z−1+⋯+cncz−ncvk=−∑p=1nccpwk−p+vk=cTψk+vk.We havec=c1,c2,…,cncT and ψk=−wk−1,−wk−2,…,−wk−ncT. Given equations (4) and (5), the system’s output is expressed as(6)yk=f2xk+wk=yk1+wk=yk1+cTψk+vk.The objective of this paper is to give the following:(i)
A good description of an unknown nonlinear system operating in a stochastic environment using HW model despite the presence of disturbance and the great model complexity, with minimum attractive theoretical assumptions that are not always valid, especially the rigorous restrictions related to the nonlinear output block’s reversibility and well-known input and/or output nonlinear blocks characteristics(ii)
A suitable approximation of the above-described system such as the sum of square residual term errorsE (given as follows) which should be reduced as possible using adequate algorithm(7)Ef^1,θ^,f^2=∑k=1Myk−y^k2,where y^k is a priori estimated output.To achieve the abovementioned objectives and based on the universal approximation properties of fuzzy systems, we propose using two fuzzy models to describe the input and output nonlinearities.


## 2.2. Fuzzy-Type Stochastic Output-Error Autoregressive Hammerstein–Wiener (FSOEAHW) Model

The crucial challenge for the identification of Hammerstein–Wiener model is to result in effective modeling of two static nonlinear functions. Better modeling requires not only approximating the nonlinear function accurately but also simplifying the identification process. In the literature, several researches are restrictive to a polynomial form of the input and/or output nonlinear blocks or well-known input and/or output nonlinear characteristics (such as dead zone or backlash) but with unknown parameters [41–60, 63–74]. Therefore, if the nonlinearity is not continuous or not in the polynomial form or with unknown characteristic, the algorithms do not give a satisfactory performance. For compensation of the aforementioned shortcomings encountered in the existent structure of Hammerstein–Wiener model, in this paper, we devoted two fuzzy models to describe the input and output nonlinearities of H-W model with only input and output measurement knowledge.In this section, we propose developing a modeling approach of a nonlinear dynamic process operating in the stochastic environment identification based on fuzzy model. According to the previous section, we propose a new fuzzy-type stochastic output-error autoregressive H-W (FSOEAHW) model given in Figure2. It consists of two static nonlinear blocks described by two independent fuzzy systems: a dynamic linear block and autoregressive noise block.Figure 2
Fuzzy-type stochastic output-error autoregressive Hammerstein–Wiener (FSOEAHW) model.Then the first fuzzy system’s outputhk can be formulated as follows:(8)hk=∑n=1N1Wn1gn1uk=W1Tφ1uk.Giving thatWn1n=1,2,…,N1 is the nth fuzzy rule’s consequence and N1 is the total rule number, gn1uk=μn1uk/∑i1=1N1μ1i1uk with μ1nuk=exp−1/2uk−o1n/σ1n2 is nth Gaussian membership function where on1 and σn1 present, respectively, the correspondent center and width. W1=W11,W21,…,W1N1T is a vector parameter, and φ1uk=g11uk,g21uk,…,g1N1ukT is an input information vector.It should be noted that the membership functions can be of several shapes such as triangular, trapezoidal, and Gaussian. The only condition that must be fulfilled is that it must be in the interval [0, 1]. In the literature, Gaussian shape is commonly used because of its simplicity, its smoothness, and nonzero at all points. It is defined by only two parameters (the center and the width) and it is a continuously differentiable function [75–77].The second unmeasurable intermediate variable of the FSOEAHW can be written as follows:(9)xk=−∑i=1naaixk−i+∑j=1nbbjhk−j=θlTφkl,where θl=a1,…,ana,b1,…,bnb and φkl=−xk−1,…,−xk−na,hk−1,…,hk−nb are, respectively, the vector parameters and the observation vector of the linear dynamic part.The output of the second fuzzy systemyk1 is expressed by(10)yk1=∑m=1N2Wm2gm2xk=W2Tφ2xk,where gm2xk=μm2xk/∑i2=1N2μ2i2xk, with μm2xk=exp−1/2xk−om2/σm22m=1,2,…,N2 in which om2 and σm2 are the center and width of the mth membership function μm2xk. W2=W12,W22,…,W2N2T and φ2xk=g12xk,g22xk,…,g2N2xkT are, respectively, the vector parameter and the observation vector of the output nonlinear block. Wm2 is the mth consequence fuzzy rule and N2 is the correspondent total rule number.Noisewk is expressed by (5). Thus, the system output can be rewritten as(11)yk=∑m=1N2Wm2gm2xk+wk=∑m=1N2Wm2gm2xk−∑p=1nccpwk−p+vk,or, equivalently, it can be written in the following matrix form:(12)yk=W2Tφ2xk+cTψk+vk=WTφk+vk.Hence, the parameter vector isW=W12,W22,…,W2N2,c1,c2,…,cncT and the observation vector is equal to φk=φ2xkT,ψkTT=g12xk,g12xk,…,g2N2xk,−wk−1,−wk−2,…,−wk−ncT.Note that all vector parameters and observation vectors (φ1uk, φ2xk, and φk) are unknown. The problem then consists in developing a novel identification algorithm estimating the unknown parameters and variables using the input measurement and output measurement uk and yk that respect the objectives cited in the previous paragraph. Founded on the estimated model parameters, the mathematical model could be built identifying a given practical stochastic nonlinear dynamical system.


## 3. FSOEAHW Parameter Estimation

In this section, we suggest using a backpropagation gradient algorithm to estimate unknown parameters and unmeasured variable.The gradient algorithm is an important tool in the linear and nonlinear problems in which a modification of parameter estimates is reached using the negative gradient direction of the criterion function. Today and in the subject of identification and control, different recursive and iterative gradient algorithms are proposed [16, 17, 78–84]. Stochastic gradient algorithm is a basic recursive identification algorithm which is used to study different types of systems such as multivariable systems [85, 86] and nonlinear block-oriented systems [16, 17, 26, 87]. In [88], an iterative gradient parameter estimation for output-error autoregressive systems using hierarchical principle has been presented. Yu et al. [31] used a gradient-based backpropagation algorithm to identify Hammerstein neural network type system. An equal algorithm has been extended by [62] for neural network type H-W system identification.The weakness of the gradient-based methods lies in the fact that estimation accuracy is not good enough for precision control purpose and the convergence rate is very slow. To improve these drawbacks, different approaches have been proposed in the literature. Among them, the multi-innovation technique is one of the most popular techniques which can improve the parameter estimation quality [89]. It consists of parameter estimation based not only on the current data but also on previous finite data at each iteration. Different papers are presented, in the literature, for the identification of different classes of systems like bilinear-in-parameter systems [90], multivariable linear systems [91, 92], block-oriented systems [93–95], and so on.Another meaningful way for improving parameter estimation is manipulating the powerful data filtering technique. The focal idea based on identification algorithm is to generate system parameter estimates using a special filter to filter the measurement data and then identify the filtered system model and the filtered noise model. In these regards, the structure of the system to identify will be changed without eliminating noise from data or changing the relationship between variables. This technique has shown the effectiveness in the identification of different types of disturbed systems such as those considered in [70, 92, 96–104].In pursuit, we present four estimation algorithms to give the unknown parameters of the proposed FSOEAHW model. The first algorithm is the improved backpropagation gradient (IBPG) that utilized the backpropagation-based gradient algorithm. The second algorithm, namely, MIIBPG, employs a multi-innovation technique and IBPG algorithm for improving the convergence rate through the multi-innovation identification technique. Lastly, and for the same purpose, a data filtering-based IBPG (FIBPG) algorithm and a multi-innovation-based FIBPG (MIFIBPG) algorithm are proposed.


### 3.1. Improved Backpropagation Gradient (IBPG) Algorithm

Leth^k,t, x^k,t, and y^1k,t be the estimated variables, respectively, of intermediate variables hk, xk, and y1k at iteration step t as follows:(13)h^k,t=W^1,t−1Tφ^1,t−1uk,(14)x^k,t=−∑i=1naa^i,t−1x^k−i,t−1+∑j=1nbb^j,t−1h^k−j,t=θ^l,t−1Tφ^kl,t−1,(15)y^k1,t=W^2,t−1Tφ^2,t−1x^k,t,where the estimated vector parameters at recursive step t are W^1t=W^11,t,W^21,t,…,W^1N1,tT, θ^l,t=a^1,t,…,a^na,t,b^1,t,…,b^nb,t, and W^2,t=W^12,t,W^22,t,…,W^2N2,tT. The predicted observation vectors are φ^1,t.=g^11,t.,g^21,t.,…,g^1N1,t.T, φ^kl,t=−x^k−1,t,…,−x^k−na,t,h^k−1,t,…,h^k−nb,t, and φ^2,t.=g^12,t.,g^22,t.,…,g^2N2,t.T with gn1,t.=μn1,t./∑i1=1N1μ1i1,t. and g^m2,t.=μ^m2,t./∑i2=1N2μ^2i2,t. in which μ^n1,tuk=exp−1/2uk−o^n1,t/σ^n1,t2, μ^m2,tx^k,t=exp−1/2x^k,t−o^m2,t/σ^m2,t2, and o^n1,t, σ^n1,t, o^m2,t, and σ^m2,t are the center and width estimates of, respectively, the nth and mth parameters of membership functions μ^n1,t. and μ^m2,t., n=1:N1 and m=1:N2.As a result, thea priori estimated output y^k can be expressed by the following equivalent adjustable model:(16)y^k,t=y^k1,t−∑p=1ncc^p,t−1w^k−p,t=W^2,t−1Tφ^2,t−1x^k,t+c^t−1Tψ^k,t=W^t−1Tφ^k,t−1,where c^t=c^1,t,c^2,t,…,c^nc,tT, ψ^k,t=−w^k−1,t,−w^k−2,t,…,−w^k−nc,tT, W^t=W^12,t,W^22,t…,W^2N2,t,c^1,t,c^2,t,…,c^nc,tT, and φ^k,t=φ^2,t.T,ψ^k,tTT=g^11,t.,g^21,t.,…,g^1N1,t.,−w^k−1,−w^k−2,…,−w^k−ncT in which w^ is the estimate of output noise w which can be expressed from equation (6) as wk=yk−y1k.To givew^k, yk1 is replaced by its estimate y^k1,t, which leads to the following expression:(17)w^k,t=yk−W^2,tTφ^2,tθ^l,tTφ^kl,t.Using equation (16), we can define an estimation error term as(18)ek1,t=yk−y^k,t.Using backpropagation gradient algorithm, each unknown parameter in the FSOEHW model is updated according to adjustment formula given by the minimization of the following quadratic error function with respect to this parameter:(19)Jk1,t=12ek1,t2=12yk−y^k,t2.By recursive application of the chain rule, all unmeasurable variables should be calculated first based on the FSOEHW parameter values before adjustment (i.e., the parameters resulting from the previous adjustment stept-1). Then each unknown parameter is adjusted according to the following expression:(20)αt=αt−1+Δαt=αt−1−ηα∂Jk1,t∂αt−1,where ηα is the learning rate of parameter updates. Parameters αt and αt−1 are, respectively, values of parameter α after and before each adjustment step. α represents one of a tuning FSOEAHW’s set of parameters W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2,c^p with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2,p=1,…,nc.Applying IBPG algorithm, the adjustment equations of the first fuzzy model parameters are given as follows:(21)W^n1,t=W^n1,t−1+ηW^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂W^n1,t−1,n=1,2,…,N1,(22)o^n1,t=o^n1,t−1+ηo^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂o^n1,t−1,(23)σ^n1,t=σ^n1,t−1+ηo^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂σ^n1,t−1,where(24)∂y^k,t∂x^k,t=∑s=1N2x^k,t−o^s2,t−1σ^s2,t−12g^s2,t−1x^k,ty^k1,t−W^s2,t−1,(25)∂x^k,t∂W^n1,t−1=∑j=1nbb^j,t−1g^n1,t−1uk−j−∑s=1naa^s,t−1∂x^k−s,t∂W^n1,t−1,(26)∂x^k,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−jW^n1,t−1−h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂o^n1,t−1,(27)∂x^k∂σ^n1,t−1=∑j=1nbb^j,t−1uk−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−jW^n1,t−1−h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂σ^n1,t−1.The linear dynamic part parameters are adjusted according to equations (28) and (29):(28)a^i,t=a^i,t−1+ηa^iek1,t∂y^k,t∂x^k,t∂x^k,t∂a^i,t−1,i=1,2,…,na,(29)b^j,t=b^j,t−1+ηb^jek1,t∂y^k,t∂x^k,t∂x^k,t∂b^j,t−1,j=1,2,…,nb,(30)∂x^k,t∂a^i,t−1=−x^k−i,t−∑s=1naa^s,t−1∂x^k−s,t∂a^i,t−1,(31)∂x^k,t∂b^j,t−1=h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂b^j,t−1.Finally, the parameter adjustment equations of the second fuzzy system and the additive noise can be obtained by the following formulas:(32)W^m2,t=W^m2,t−1+ηW^2mek1,tg^m2,t−1x^k,t,m=1,2,…,N2,(33)o^m2,t=o^m2,t−1+ηo^2mek1,tx^k−o^m2,t−1σ^m2,t−12g^m2,t−1x^k,tW^m2,t−1−y^k1,t,(34)σ^m2,t=σ^m2,t−1+ησ^2mek1,tx^k,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k,tW^m2,t−1−y^k1,t,(35)c^p,t=c^p,t−1+ηcpek1,t−w^k−p,t,p=1,2,…,nc.The difference between the proposed algorithm (IBPG) and a classical backpropagation gradient algorithm (BPG) lies in the second terms of equations (25)–(27), (30), and (31) which are omitted in the BPG case. In fact, our algorithm is inspired by [15, 24]. The unmeasured variables h^k−j,t, j=1,…,nb, and x^k−i,ti=1,…,na are recalculated at each recursive step t based on a^i,t−1, b^j,t−1, σ^n1,t−1, o^n1,t−1, and W^n1,t−1i=1,…,na,j=1,…,nb,n=1,…,N1, resulting from the just previous recursive step because the considered system is supposed to be invariant and the linear part is dynamic. Then, considering the above-cited terms is recommended.The following summarizes the IBPG identification procedure of FSOEAHW model:Step 1: initialize the FSOEAHW parameters randomly. Fix the learning rates;t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: for inputuk−j, calculate h^k−j,tj=1,…,nb using equation (13).Step 3: computex^k−i,t, y^k1,t, y^k,t, and ek1,t based, respectively, on equations (14)–(16) and (18).Step 4: adjust the FSOEAHW parameters using equations (21)–(35).Step 5: calculate noise’s estimate by equation (17).Step 6: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


#### 3.1.1. IBPG Algorithm Convergence Analysis

The adjustment procedures of FSOEAHW parameters are based on the learning rate choice (ηα). Too small ηα guarantees convergence but with slow training speed, whereas too big ηα guides to parametric divergence. In this section, Theorem 1 gives a selecting approach of a convenient ηα.Theorem 1.
The asymptotic convergence of the IBPG algorithm is guaranteed if each learning rateηα of each correspondent adjustable parameter α is chosen to satisfy(36)0≺ηα≺2Pα,max2,where Pα,max=maxk∂y^k,t/∂αt−1.Proof.
See Appendix A.


### 3.2. Multi-Innovation Improved Backpropagation Gradient (MIIBPG) Algorithm

The multi-innovation technique is an effective tool to enhance the convergence rate of estimation methods. That is why it is joined with different estimation algorithms such as gradient algorithm and least-square method in its recursive and iterative form [33, 105–113]. Particularly, it is well known that recursive gradient algorithm presents a slow convergence rate compared to other estimation approaches [113]. Many factors contribute to this disadvantage, principally its dependence on current data uk,yk only. As a matter of fact, at each recursive step, gradient algorithm does not use previous data uk−z,yk−z1;z1=1,2,…. It does not have the capacity to use the available data in the same step.To overcome this problem, we propose in this section using multi-innovation IBPG (MIIBPG) algorithm based on previous finite data; that is, the MIIBPG algorithm uses the current and the previous data at each iterationt, which can improve the parameter estimation accuracy.The elementary idea is to expand the error termek1,t given by (8) and denoted scalar innovation to an innovation vector (called multi-innovation) [109, 111, 113].Define an innovation vectorEk1L,t as(37)Ek1L,t=ek1,tek1−1,t⋮ek1−L+1,t=yk−y^k,tyk−1−y^k−1,t⋮yk−L+1−y^k−L+1,t∈ℜL,where L is positive integer denoted innovation length and ek1−l,t=yk−l−y^k−l,t is the lth error scalar term at time k−ll=0,1,…,L expressed by(38)ek1−l,t=yk−l−y^k1−l,t−∑p=1ncc^p,t−1w^k−l−p,t=yk−l−W^2,t−1Tφ^2x^k−l,t+c^t−1Tψ^k−l,t=yk−l−W^t−1Tφ^k−l.According to the recursive gradient algorithm minimizing the cost function given by equation (39), the MIIBPG technique provides a set of parameters updating equations listed in the following:(39)Jk1L,t=12Ek1L,tTEk1L,t,(40)c^p,t=c^p,t−1−ηcp∑l=0L−1∂Jk1L,t∂ek1−l,t∂ek1−l,t∂c^p,t−1=c^p,t−1−ηcp∑l=0L−1ek1−l,t−w^k−l−p,t,p=1:nc,(41)W^m2,t=W^m2,t−1+ηW^2m∑l=0L−1ek1−l,tg^m2,t−1x^k−l,t,m=1:N2,(42)o^m2,t=o^m2,t−1+ηo2m∑l=0L−1ek1−lx^k−l,t−o^m2,t−1σ^m2,t−12gm2,t−1x^k−l,tW^m2,t−y^k1−l,t,(43)σ^m2,t=σ^m2,t−1+ησ^2m∑l=0L−1ek1−l,tx^k−l,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k−l,tW^m2,t−1−y^k1−l,t,(44)a^i,t=a^i,t−1+ηai∑l=0L−1ek1−l∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂a^i,t−1,i=1:na,where(45)∂y^k1−l,t∂x^k−l,t=∑m=1N2x^k−1,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l,ty^k1−l,t−W^m2,t−1,(46)∂x^k−l,t∂a^i,t−1=−x^k−l−i,t−∑s=1naa^s,t−1∂x^k−l−s,t∂a^i,t−1,(47)b^j,t=b^j,t−1+ηbj∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂b^j,t−1,j=1:nb,(48)∂x^k−l,t∂b^j,t−1=h^k−l−j,t−∑s=1naa^s,t−1∂x^k−l−s,t∂b^j,t−1,(49)W^n1,t=W^n1,t−11+ηW^1n∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂W^n1,t−1=W^n1,t−1+ηW^1n∑l=0L−1ek−l,t∂y^k1−l,t∂x^k−l,t∑j=1nbb^j,t−1g^n1,t−1uk−l−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1g^n1,t−1uk−l−i−j,n=1:N1,(50)o^n1,t=o^n1,t−1+ηo1∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂o^n1,t−1,(51)∂x^k−l,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−l−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−jW^n1,t−1−h^k−l−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−i−jW^n1,t−1−h^k−l−i−j,t,(52)σ^n1,t=σ^n1,t−1+ησ1n∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂σ^n1,t−1,(53)∂x^k−l∂σ^n1,t−1=∑j=1nbb^j,t−1uk−l−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−jW^n1,t−1−h^k−l−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−i−jW^n1,t−1−h^k−l−i−j,t.The MIIBPG identification algorithm can be implemented based on the following steps:Step 1: initialize the parameters of the linear dynamic parts of the two fuzzy systems and of the output noise randomly. Fix the learning rates and the innovation lengthL; t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: forl=0,…,L−1, and for given input uk−j−l, calculate h^k−j−l,t, x^k−i−l,t, y^k1−l,t, y^k−l,t, and ek1−l,tj=1,…,nb using equations (13)–(16) and (18) replacing k with k-l.Step 3: adjust the FSOEAHW parameters using equations (40)–(53).Step 4: calculate noise’s estimate by equation (17) replacing k with k-l.Step 5: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


#### 3.2.1. MIIBPG Algorithm Convergence Analysis

This section surmises the convergence analysis of MIIBPG algorithm which can be applied to estimate a nonlinear stochastic system that can be illustrated by the proposed FSOEAHW model. The convergence properties of the MIIBPG algorithm are introduced by the following theorem.Theorem 2.
The asymptotic convergence of MIIBPG algorithm is guaranteed if the learning rateηα of the corresponding adjustable parameter α is chosen to satisfy(54)0≺ηα≺2Pα,max2,where Pα,max=maxk∑l=0L−1∂y^k−l,t/∂αt−1.Proof.
See Appendix B.


### 3.3. Data Filtering-Based Improved Backpropagation Gradient (FIBPG) Algorithm

The data filtering technique in system identification is used to deal with the parameter estimation issues of systems disturbed by colored noises. Specifically, the elementary idea is to use a linear filter to filter input-output data so that the original systems with colored noises are transformed into new ones with white noises. Then the system’s structure is transformed to be simpler without changing the relationship between the system inputs and outputs. Owing to the advantages of the data filtering technique, it has been widely applied for different system identification and parameter estimation. For example, each of recursive, iterative, and hierarchical least-square algorithms is combined with data filtering technique in the identification problem of output-error autoregressive linear systems [114], two-input single-output controlled autoregressive moving average linear systems [115], Hammerstein finite impulse response systems with moving average output noise [116], and a multivariable box Jenkins-like system [117]. Similarly, a gradient algorithm in its recursive and iterative form is used with data filtering technique for identification of some classes of linear and nonlinear systems such as linear multivariable autoregressive moving average system [101], a state-space linear system in its observability canonical form disturbed by colored noise [118], and a block-oriented system like Hammerstein finite impulse response system with moving average output noise [119].Stirred by the above description, we propose in this section combining IBPG algorithm (given in Section3.1) with a data filtering technique to improve the accuracy rate of IBPG algorithm. In our case, we propose introducing a linear filter Cz to filter a measurement data in the purpose of parameter and intermediate variables estimation based on two criteria.We define, respectively, the filtered intermediate variableyk1f and the filtered output ykf as(55)ykf=Czyk=yk+∑p=1nccpyk−p=∑p=0nccpyk−p,yk1f=Czyk1=∑p=0nccpyk1−p.We havec0=1. Multiplying both sides of (11) by Cz and using (10), we obtain(56)Czyk=Czyk1+vk=Cz∑m=1N2W2mg2mxk+vk.Thus, the filtered output can be written in matrix form as(57)ykf=W2TCzφ2xk+vk.In that, we can write the filtered observation vector as follows:(58)φ2fxk=Czφ2xk.Consequently, the filtered system output can be defined as(59)ykf=yk1f+vk=W2Tφ2fxk+vk.The objective, now, is to develop an FSOEAHW parameters estimation approach based on gradient algorithm, using a filtered data measurement and minimizing a filtered estimation error given by(60)ek1f,t=ykf−y^kf,t,where the filtered estimated output y^kf,t can be expressed by the equivalent adjustable model:(61)y^kf,t=y^k1f,t=W^2,t−1Tφ^2f,t−1x^k,t,where φ^2f,t−1x^k,t will be expressed later. It represents the estimated filtered observation vector in which the estimated intermediate variable x^k,t is given by (14).Nevertheless,ykf in equation (59) depends on unknown parameter cp, p=1,2,…,nc. To neutralize this problem, we submit a two-step gradient algorithm. In the first step, the unknown parameter cp, p=1,2,…,nc, will be estimated minimizing the criteria Jk2,t (given by equation (62)). Subsequently, unknown parameters set β=W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2 with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2 will be estimated minimizing the criteria Jk1f,t given by equation (66).(62)Jk2,t=12wk−c^t−1Tψ^k,t2.In reality, the quadratic criterionJk2,t contains the unmeasurable output noise vector wk, so the estimation of unknown parameter cp, p=1,2,…,nc, cannot be implemented. Then, we propose replacing wk in (62) by its estimated w^k given by equation (64). Afterward, the new quadratic criterion functions will be defined as(63)Jk2,t=12ek2,t2=12w^k,t−c^t−1Tψ^k,t2,where(64)w^k,t=yk−W^2,t−1Tφ^2,t−1x^k,t.Consequently, the estimation equation of parametercp, p=1,2,…,nc, can be inferred using gradient algorithm minimizing the criteria Jk2,t as follows:(65)c^p,t=c^p,t−1+ηc^pek2,t−w^k−p,t,p=1,2,…,nc.The estimation procedure of unknown parameters setβ is derived using the gradient method minimizing the quadratic criterion function defined as follows:(66)Jk1f,t=12ek1f,t2=12ykf,t−y^kf,t2,where ykf,t and y^kf,t are expressed, respectively, by(67)ykf,t=∑p=0ncc^p,tyk−p,(68)y^kf,t=W^2,t−1Tφ^2f,t−1x^k,t,where(69)φ^2f,t−1x^k=C^tzφ^2,t−1x^k,t.We have(70)C^tz=1+c^1tz−1+⋯+c^nctz−nc.Therefore, the rest of parameters are adjusted using the following equations withc^0,t=1:(71)W^n1,t=W^n1,t−1+ηW^1nek1f,t∂y^kf,t∂W^n1,t−1=W^n1,t−1+ηW^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂W^n1,t−1,n=1,…,N1,where(72)∂x^k−p,t∂W^n1,t−1=∑j=1nbb^j,t−1gn1,t−1uk−p−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1gn1,t−1uk−p−i−j,(73)∂y^k−p,t∂x^k−p,t=∑m=1N2x^k−p−o^m2,t−1σ^m2,t−12gm2,t−1x^k−p,ty^k1−p,t−W^m2,t−1,(74)o^n1,t=o^n1,t−1+ηo^1nek1f,t∂y^kf,t∂o^n1,t−1=o^n1,t−1+ηo^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂o^n1,t−1,(75)∂x^k−p,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−p−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−p−jW^n1,t−1−h^k−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−p−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−p−i−jW^n1,t−1−h^k−p−i−j,t,(76)σ^n1,t=σ^n1,t−1+ησ^1nek1f,t∂y^kf,t∂o^n1,t−1=σ^n1,t−1+ησ^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂σ^n1,t−1,(77)∂x^k−p,t∂σ^n1,t−1=∑j=1nbb^j,t−1uk−p−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−p−jW^n1,t−1−h^k−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−p−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−p−i−jW^n1,t−1−h^k−p−i−j,t,(78)a^i,t=a^i,t−1+ηa^iek1f,t∂y^kf,t∂a^i,t−1=a^i,t−1+ηa^iek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂a^i,t−1,i=1,2,…,na,(79)∂x^k−p,t∂a^i,t−1=−x^k−p−i,t−∑s=1naa^s,t−1∂x^k−p−s,t∂a^i,t−1,(80)b^j,t=b^j,t−1+ηb^jek1f,t∂y^kf,t∂b^j,t−1=b^j,t−1+ηb^jek1f,t∑p=0ncc^p,t∂y^k1−p,t∂x^k−p,t∂x^k−p,t∂b^j,t−1,j=1,2,…,nb,(81)∂x^k−p,t∂b^j,t−1=h^k−j−p,t−∑s=1naa^s,t−1∂x^k−p−s,t∂b^j,t−1,(82)W^m2,t=W^m2,t−1+ηW^2mek1f,t∑p=0ncc^p,tg^m2,t−1x^k−p,t,m=1,2,…,N2,(83)o^m2,t=o^m2,t−1+ηo^2mek1f,t∑p=0ncc^p,tx^k−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−p,tW^m2,t−1−y^k1−p,t,(84)σ^m2,t=σ^m2,t−1+ηo^2mek1f,t∑p=0ncc^p,tx^k−p,t−o^m2,t−12σ^m2,t−3g^m2,t−1x^k−p,tW^m2,t−1−y^k1−p,t.To summarize the FIBPG algorithm, we give the following steps:Step 1: initialize the parameters of the linear dynamic parts of the two fuzzy systems and the output noise parameters randomly. Fix the learning rates;t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: for inputuk−p−j, calculate h^k−p−j,tj=1,…,nb,p=0,…,nc, x^k−p−i,t, y^k1−p,t, and y^k−p,t using, respectively, equations (13)–(16) replacing k with k-p.Step 3: computew^k,t using equation (64).Step 4: adjust parameterc^p, p=1,2,…,nc, by equation (65).Step 5: computeykf,t, y^kf,t, and ek1f,t based, respectively, on equations (66)–(68).Step 6: update the rest of parameters using equations (71)–(84).Step 7: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


#### 3.3.1. FIBPG Convergence Analysis

This subsection aims at the convergence analysis of the FIBPG which can be applied to estimate the parameters of a nonlinear system that can be described by the FSEOFHW model. The convergence properties of the FIBPG algorithm are determined in the following theorem.Theorem 3.
The asymptotic convergence of FIBPG is ensured if the learning rateηc^pp=1,…,nc and those of other FSOEAHW parameters satisfy the following conditions:(85)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where(86)Pc,max=maxk∂w^k,t∂c^p,t−1,Pβ,max=maxk∂y^k∂βt−1.β is an element of a parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2 with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2.Proof.
See Appendix C.


### 3.4. Multi-Innovation and Data Filtering-Based Improved Backpropagation Gradient (MIFIBPG) Algorithm

Similar to Section3.2 and to improve the convergence accuracy, we can combine multi-innovation approach with data filtering technique and then we obtain a multi-innovation FIBPG (MIFIBPG) algorithm.DefineE2Lk,t and E1fLk,t as follows:(87)Ek2L,t=ek2,tek2−1,t⋮ek2−L+1,t=w^k,t−c^t−1Tψ^k,tw^k−1,t−c^t−1Tψ^k−1,t⋮w^k−L+1,t−c^t−1Tψ^k−L+1,t,Ek1fL,t=ek1f,tek1f−1,t⋮ek1f−L+1,t=ykf,t−y^kf,tykf−1,t−y^kf−1,t⋮ykf−L+1,t−y^kf−L+1,t,where(88)ek2−l,t=w^k−l,t−c^t−1Tψ^k−l=yk−W^2,t−1Tφ^2,t−1x^k−l,t−c^t−1Tψ^k−l,ek1f−l,t=ykf−l,t−y^kf−l,t=∑p=0ncc^p,tyk−l−p−W^2,t−1Tφ^2fx^k−l,t,l=0:L−1andc^0=1.Like the previous section and referring to MIIBPG algorithm, initially the vector parameterc^ is adjusted in order to calculate the innovation vector error Ek1fL,t. It is updated according to formula (91) by minimizing the quadratic criterion given by the following equation:(89)Jk2L,t=12Ek2L,tTEk2L,t,c^p,t=c^p,t−1+ηc^p∑l=0L−1ek2−l,t−w^k−l−p,t,p=1:nc.The adjustment equations of all other parameters are gotten using the MIFIBPG algorithm minimizing the cost functionJk1fL,t=1/2Ek1fL,tTEk1fL,t. They are listed as follows:(90)W^m2,t=W^m2,t−1+ηW^2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tg^m2,t−1x^k−l−p,m=1,2,…,N2,(91)o^m2,t=o^m2,t−1+ηo2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tx^k−l−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l−p,tW^m2,t−1−y^k1−l−p,t,(92)σ^m2,t=σ^m2,t−1+ηo2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tx^k−l−p,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k−l−p,tW^m2,t−1−y^k1−l−p,t,(93)a^i,t=a^i,t−1+ηa^i∑l=0L−1ek1f−l,t∂y^kf−l,t∂a^i,t−1=a^i,t−1+ηa^i∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂a^i,t−1,i=1,2,…,na,where(94)∂y^k−l−p,t∂x^k−l−p,t=∑m=1N2x^k−l−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l−p,ty^k1−l−p,t−W^m2,t−1,(95)∂x^k−l−p,t∂a^i,t−1=−x^k−l−p−i−∑s=1naa^s,t−1∂x^k−l−p−s,t∂a^i,t−1,(96)b^j,t=b^j,t−1+ηbj∑l=0L−1ek1f,t∂y^kf−l,t∂b^j,t−1=b^j,t−1+ηb^j∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂b^j,t−1=b^j,t−1+ηb^j∑l=0l−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,th^k−l−p−j,t−∑s=1naa^s,t−1∂x^k−l−p−s,t∂b^j,t−1,j=1,2,…,nb,(97)W^n1,t=W^n1,t−1+ηW1n∑l=0L−1ek1f−l,t∂y^kf−l,t∂W^n1,t−1=W^n1,t−1+ηW1∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k1−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂Wn1,t−1,n=1,2,…,N1,where(98)∂x^k−l−p,t∂W^n1,t−1=∑j=1nbb^j,t−1g^n1,t−1uk−l−p−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1g^n1,t−1uk−l−p−i−j,(99)o^n1,t=o^n1,t−1+ηo1n∑l=0L−1ek1f−l,t∂y^kf−l,t∂o^n1,t−1=o^n1,t−1+ηo^1n∑l=0L−1ek1f−l,t∑p=0ncc⌢p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂o^n1,t−1,(100)∂x^k−l−p∂o^n1,t−1=∑j=1nbb^j,t−1uk−l−p−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−p−jW^n1,t−1−h^k−l−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−p−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−p−i−jW^n1,t−1−h^k−l−p−i−j,t,(101)σ^n1,t=σ^n1,t−1+ησ^1n∑l=0L−1ek1f−l∂y^kf−l∂σ^n1,t−1=σ^n1,t−1+ησ^1n∑l=0L−1ek1f−l,t∑p=0ncc⌢p,t∂y^k1−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂σ^n1,t−1,(102)∂x^k−l−p,t∂σ^n1,t−1=∑j=1nbb^j,t−1uk−l−p−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−p−jW^n1,t−1−h^k−l−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−p−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−p−i−jW^n1,t−1−h^k−l−p−i−j,t.


#### 3.4.1. MIFIBPG Algorithm Convergence Study

Taking into account the mentioned theorems, we can demonstrate readily the convergence properties of this MIFIBPG algorithm for the considered FSOEAHW, which is enunciated by the following theorem.Theorem 4.
The convergence of the MIFIBPG algorithm is ensured if the learnings ratesηc^p and ηβ, where β represents each element of the parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2, with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2, satisfy the following inequalities:(103)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where Pc^p,max=maxk∑l=0L−1∂w^k−l,t/∂c^p,t−1,Pβ,max=maxk∑l=0L−1∂y^k−l/∂βt−1.The theorem’s proof is similar to those given in Appendices B and C for the MIIIBPG and FIBPG algorithms convergence study.


## 3.1. Improved Backpropagation Gradient (IBPG) Algorithm

Leth^k,t, x^k,t, and y^1k,t be the estimated variables, respectively, of intermediate variables hk, xk, and y1k at iteration step t as follows:(13)h^k,t=W^1,t−1Tφ^1,t−1uk,(14)x^k,t=−∑i=1naa^i,t−1x^k−i,t−1+∑j=1nbb^j,t−1h^k−j,t=θ^l,t−1Tφ^kl,t−1,(15)y^k1,t=W^2,t−1Tφ^2,t−1x^k,t,where the estimated vector parameters at recursive step t are W^1t=W^11,t,W^21,t,…,W^1N1,tT, θ^l,t=a^1,t,…,a^na,t,b^1,t,…,b^nb,t, and W^2,t=W^12,t,W^22,t,…,W^2N2,tT. The predicted observation vectors are φ^1,t.=g^11,t.,g^21,t.,…,g^1N1,t.T, φ^kl,t=−x^k−1,t,…,−x^k−na,t,h^k−1,t,…,h^k−nb,t, and φ^2,t.=g^12,t.,g^22,t.,…,g^2N2,t.T with gn1,t.=μn1,t./∑i1=1N1μ1i1,t. and g^m2,t.=μ^m2,t./∑i2=1N2μ^2i2,t. in which μ^n1,tuk=exp−1/2uk−o^n1,t/σ^n1,t2, μ^m2,tx^k,t=exp−1/2x^k,t−o^m2,t/σ^m2,t2, and o^n1,t, σ^n1,t, o^m2,t, and σ^m2,t are the center and width estimates of, respectively, the nth and mth parameters of membership functions μ^n1,t. and μ^m2,t., n=1:N1 and m=1:N2.As a result, thea priori estimated output y^k can be expressed by the following equivalent adjustable model:(16)y^k,t=y^k1,t−∑p=1ncc^p,t−1w^k−p,t=W^2,t−1Tφ^2,t−1x^k,t+c^t−1Tψ^k,t=W^t−1Tφ^k,t−1,where c^t=c^1,t,c^2,t,…,c^nc,tT, ψ^k,t=−w^k−1,t,−w^k−2,t,…,−w^k−nc,tT, W^t=W^12,t,W^22,t…,W^2N2,t,c^1,t,c^2,t,…,c^nc,tT, and φ^k,t=φ^2,t.T,ψ^k,tTT=g^11,t.,g^21,t.,…,g^1N1,t.,−w^k−1,−w^k−2,…,−w^k−ncT in which w^ is the estimate of output noise w which can be expressed from equation (6) as wk=yk−y1k.To givew^k, yk1 is replaced by its estimate y^k1,t, which leads to the following expression:(17)w^k,t=yk−W^2,tTφ^2,tθ^l,tTφ^kl,t.Using equation (16), we can define an estimation error term as(18)ek1,t=yk−y^k,t.Using backpropagation gradient algorithm, each unknown parameter in the FSOEHW model is updated according to adjustment formula given by the minimization of the following quadratic error function with respect to this parameter:(19)Jk1,t=12ek1,t2=12yk−y^k,t2.By recursive application of the chain rule, all unmeasurable variables should be calculated first based on the FSOEHW parameter values before adjustment (i.e., the parameters resulting from the previous adjustment stept-1). Then each unknown parameter is adjusted according to the following expression:(20)αt=αt−1+Δαt=αt−1−ηα∂Jk1,t∂αt−1,where ηα is the learning rate of parameter updates. Parameters αt and αt−1 are, respectively, values of parameter α after and before each adjustment step. α represents one of a tuning FSOEAHW’s set of parameters W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2,c^p with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2,p=1,…,nc.Applying IBPG algorithm, the adjustment equations of the first fuzzy model parameters are given as follows:(21)W^n1,t=W^n1,t−1+ηW^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂W^n1,t−1,n=1,2,…,N1,(22)o^n1,t=o^n1,t−1+ηo^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂o^n1,t−1,(23)σ^n1,t=σ^n1,t−1+ηo^1nek1,t∂y^k,t∂x^k,t∂x^k,t∂σ^n1,t−1,where(24)∂y^k,t∂x^k,t=∑s=1N2x^k,t−o^s2,t−1σ^s2,t−12g^s2,t−1x^k,ty^k1,t−W^s2,t−1,(25)∂x^k,t∂W^n1,t−1=∑j=1nbb^j,t−1g^n1,t−1uk−j−∑s=1naa^s,t−1∂x^k−s,t∂W^n1,t−1,(26)∂x^k,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−jW^n1,t−1−h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂o^n1,t−1,(27)∂x^k∂σ^n1,t−1=∑j=1nbb^j,t−1uk−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−jW^n1,t−1−h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂σ^n1,t−1.The linear dynamic part parameters are adjusted according to equations (28) and (29):(28)a^i,t=a^i,t−1+ηa^iek1,t∂y^k,t∂x^k,t∂x^k,t∂a^i,t−1,i=1,2,…,na,(29)b^j,t=b^j,t−1+ηb^jek1,t∂y^k,t∂x^k,t∂x^k,t∂b^j,t−1,j=1,2,…,nb,(30)∂x^k,t∂a^i,t−1=−x^k−i,t−∑s=1naa^s,t−1∂x^k−s,t∂a^i,t−1,(31)∂x^k,t∂b^j,t−1=h^k−j,t−∑s=1naa^s,t−1∂x^k−s,t∂b^j,t−1.Finally, the parameter adjustment equations of the second fuzzy system and the additive noise can be obtained by the following formulas:(32)W^m2,t=W^m2,t−1+ηW^2mek1,tg^m2,t−1x^k,t,m=1,2,…,N2,(33)o^m2,t=o^m2,t−1+ηo^2mek1,tx^k−o^m2,t−1σ^m2,t−12g^m2,t−1x^k,tW^m2,t−1−y^k1,t,(34)σ^m2,t=σ^m2,t−1+ησ^2mek1,tx^k,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k,tW^m2,t−1−y^k1,t,(35)c^p,t=c^p,t−1+ηcpek1,t−w^k−p,t,p=1,2,…,nc.The difference between the proposed algorithm (IBPG) and a classical backpropagation gradient algorithm (BPG) lies in the second terms of equations (25)–(27), (30), and (31) which are omitted in the BPG case. In fact, our algorithm is inspired by [15, 24]. The unmeasured variables h^k−j,t, j=1,…,nb, and x^k−i,ti=1,…,na are recalculated at each recursive step t based on a^i,t−1, b^j,t−1, σ^n1,t−1, o^n1,t−1, and W^n1,t−1i=1,…,na,j=1,…,nb,n=1,…,N1, resulting from the just previous recursive step because the considered system is supposed to be invariant and the linear part is dynamic. Then, considering the above-cited terms is recommended.The following summarizes the IBPG identification procedure of FSOEAHW model:Step 1: initialize the FSOEAHW parameters randomly. Fix the learning rates;t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: for inputuk−j, calculate h^k−j,tj=1,…,nb using equation (13).Step 3: computex^k−i,t, y^k1,t, y^k,t, and ek1,t based, respectively, on equations (14)–(16) and (18).Step 4: adjust the FSOEAHW parameters using equations (21)–(35).Step 5: calculate noise’s estimate by equation (17).Step 6: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


### 3.1.1. IBPG Algorithm Convergence Analysis

The adjustment procedures of FSOEAHW parameters are based on the learning rate choice (ηα). Too small ηα guarantees convergence but with slow training speed, whereas too big ηα guides to parametric divergence. In this section, Theorem 1 gives a selecting approach of a convenient ηα.Theorem 1.
The asymptotic convergence of the IBPG algorithm is guaranteed if each learning rateηα of each correspondent adjustable parameter α is chosen to satisfy(36)0≺ηα≺2Pα,max2,where Pα,max=maxk∂y^k,t/∂αt−1.Proof.
See Appendix A.


## 3.1.1. IBPG Algorithm Convergence Analysis

The adjustment procedures of FSOEAHW parameters are based on the learning rate choice (ηα). Too small ηα guarantees convergence but with slow training speed, whereas too big ηα guides to parametric divergence. In this section, Theorem 1 gives a selecting approach of a convenient ηα.Theorem 1.
The asymptotic convergence of the IBPG algorithm is guaranteed if each learning rateηα of each correspondent adjustable parameter α is chosen to satisfy(36)0≺ηα≺2Pα,max2,where Pα,max=maxk∂y^k,t/∂αt−1.Proof.
See Appendix A.


## 3.2. Multi-Innovation Improved Backpropagation Gradient (MIIBPG) Algorithm

The multi-innovation technique is an effective tool to enhance the convergence rate of estimation methods. That is why it is joined with different estimation algorithms such as gradient algorithm and least-square method in its recursive and iterative form [33, 105–113]. Particularly, it is well known that recursive gradient algorithm presents a slow convergence rate compared to other estimation approaches [113]. Many factors contribute to this disadvantage, principally its dependence on current data uk,yk only. As a matter of fact, at each recursive step, gradient algorithm does not use previous data uk−z,yk−z1;z1=1,2,…. It does not have the capacity to use the available data in the same step.To overcome this problem, we propose in this section using multi-innovation IBPG (MIIBPG) algorithm based on previous finite data; that is, the MIIBPG algorithm uses the current and the previous data at each iterationt, which can improve the parameter estimation accuracy.The elementary idea is to expand the error termek1,t given by (8) and denoted scalar innovation to an innovation vector (called multi-innovation) [109, 111, 113].Define an innovation vectorEk1L,t as(37)Ek1L,t=ek1,tek1−1,t⋮ek1−L+1,t=yk−y^k,tyk−1−y^k−1,t⋮yk−L+1−y^k−L+1,t∈ℜL,where L is positive integer denoted innovation length and ek1−l,t=yk−l−y^k−l,t is the lth error scalar term at time k−ll=0,1,…,L expressed by(38)ek1−l,t=yk−l−y^k1−l,t−∑p=1ncc^p,t−1w^k−l−p,t=yk−l−W^2,t−1Tφ^2x^k−l,t+c^t−1Tψ^k−l,t=yk−l−W^t−1Tφ^k−l.According to the recursive gradient algorithm minimizing the cost function given by equation (39), the MIIBPG technique provides a set of parameters updating equations listed in the following:(39)Jk1L,t=12Ek1L,tTEk1L,t,(40)c^p,t=c^p,t−1−ηcp∑l=0L−1∂Jk1L,t∂ek1−l,t∂ek1−l,t∂c^p,t−1=c^p,t−1−ηcp∑l=0L−1ek1−l,t−w^k−l−p,t,p=1:nc,(41)W^m2,t=W^m2,t−1+ηW^2m∑l=0L−1ek1−l,tg^m2,t−1x^k−l,t,m=1:N2,(42)o^m2,t=o^m2,t−1+ηo2m∑l=0L−1ek1−lx^k−l,t−o^m2,t−1σ^m2,t−12gm2,t−1x^k−l,tW^m2,t−y^k1−l,t,(43)σ^m2,t=σ^m2,t−1+ησ^2m∑l=0L−1ek1−l,tx^k−l,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k−l,tW^m2,t−1−y^k1−l,t,(44)a^i,t=a^i,t−1+ηai∑l=0L−1ek1−l∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂a^i,t−1,i=1:na,where(45)∂y^k1−l,t∂x^k−l,t=∑m=1N2x^k−1,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l,ty^k1−l,t−W^m2,t−1,(46)∂x^k−l,t∂a^i,t−1=−x^k−l−i,t−∑s=1naa^s,t−1∂x^k−l−s,t∂a^i,t−1,(47)b^j,t=b^j,t−1+ηbj∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂b^j,t−1,j=1:nb,(48)∂x^k−l,t∂b^j,t−1=h^k−l−j,t−∑s=1naa^s,t−1∂x^k−l−s,t∂b^j,t−1,(49)W^n1,t=W^n1,t−11+ηW^1n∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂W^n1,t−1=W^n1,t−1+ηW^1n∑l=0L−1ek−l,t∂y^k1−l,t∂x^k−l,t∑j=1nbb^j,t−1g^n1,t−1uk−l−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1g^n1,t−1uk−l−i−j,n=1:N1,(50)o^n1,t=o^n1,t−1+ηo1∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂o^n1,t−1,(51)∂x^k−l,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−l−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−jW^n1,t−1−h^k−l−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−i−jW^n1,t−1−h^k−l−i−j,t,(52)σ^n1,t=σ^n1,t−1+ησ1n∑l=0L−1ek1−l,t∂y^k1−l,t∂x^k−l,t∂x^k−l,t∂σ^n1,t−1,(53)∂x^k−l∂σ^n1,t−1=∑j=1nbb^j,t−1uk−l−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−jW^n1,t−1−h^k−l−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−i−jW^n1,t−1−h^k−l−i−j,t.The MIIBPG identification algorithm can be implemented based on the following steps:Step 1: initialize the parameters of the linear dynamic parts of the two fuzzy systems and of the output noise randomly. Fix the learning rates and the innovation lengthL; t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: forl=0,…,L−1, and for given input uk−j−l, calculate h^k−j−l,t, x^k−i−l,t, y^k1−l,t, y^k−l,t, and ek1−l,tj=1,…,nb using equations (13)–(16) and (18) replacing k with k-l.Step 3: adjust the FSOEAHW parameters using equations (40)–(53).Step 4: calculate noise’s estimate by equation (17) replacing k with k-l.Step 5: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


### 3.2.1. MIIBPG Algorithm Convergence Analysis

This section surmises the convergence analysis of MIIBPG algorithm which can be applied to estimate a nonlinear stochastic system that can be illustrated by the proposed FSOEAHW model. The convergence properties of the MIIBPG algorithm are introduced by the following theorem.Theorem 2.
The asymptotic convergence of MIIBPG algorithm is guaranteed if the learning rateηα of the corresponding adjustable parameter α is chosen to satisfy(54)0≺ηα≺2Pα,max2,where Pα,max=maxk∑l=0L−1∂y^k−l,t/∂αt−1.Proof.
See Appendix B.


## 3.2.1. MIIBPG Algorithm Convergence Analysis

This section surmises the convergence analysis of MIIBPG algorithm which can be applied to estimate a nonlinear stochastic system that can be illustrated by the proposed FSOEAHW model. The convergence properties of the MIIBPG algorithm are introduced by the following theorem.Theorem 2.
The asymptotic convergence of MIIBPG algorithm is guaranteed if the learning rateηα of the corresponding adjustable parameter α is chosen to satisfy(54)0≺ηα≺2Pα,max2,where Pα,max=maxk∑l=0L−1∂y^k−l,t/∂αt−1.Proof.
See Appendix B.


## 3.3. Data Filtering-Based Improved Backpropagation Gradient (FIBPG) Algorithm

The data filtering technique in system identification is used to deal with the parameter estimation issues of systems disturbed by colored noises. Specifically, the elementary idea is to use a linear filter to filter input-output data so that the original systems with colored noises are transformed into new ones with white noises. Then the system’s structure is transformed to be simpler without changing the relationship between the system inputs and outputs. Owing to the advantages of the data filtering technique, it has been widely applied for different system identification and parameter estimation. For example, each of recursive, iterative, and hierarchical least-square algorithms is combined with data filtering technique in the identification problem of output-error autoregressive linear systems [114], two-input single-output controlled autoregressive moving average linear systems [115], Hammerstein finite impulse response systems with moving average output noise [116], and a multivariable box Jenkins-like system [117]. Similarly, a gradient algorithm in its recursive and iterative form is used with data filtering technique for identification of some classes of linear and nonlinear systems such as linear multivariable autoregressive moving average system [101], a state-space linear system in its observability canonical form disturbed by colored noise [118], and a block-oriented system like Hammerstein finite impulse response system with moving average output noise [119].Stirred by the above description, we propose in this section combining IBPG algorithm (given in Section3.1) with a data filtering technique to improve the accuracy rate of IBPG algorithm. In our case, we propose introducing a linear filter Cz to filter a measurement data in the purpose of parameter and intermediate variables estimation based on two criteria.We define, respectively, the filtered intermediate variableyk1f and the filtered output ykf as(55)ykf=Czyk=yk+∑p=1nccpyk−p=∑p=0nccpyk−p,yk1f=Czyk1=∑p=0nccpyk1−p.We havec0=1. Multiplying both sides of (11) by Cz and using (10), we obtain(56)Czyk=Czyk1+vk=Cz∑m=1N2W2mg2mxk+vk.Thus, the filtered output can be written in matrix form as(57)ykf=W2TCzφ2xk+vk.In that, we can write the filtered observation vector as follows:(58)φ2fxk=Czφ2xk.Consequently, the filtered system output can be defined as(59)ykf=yk1f+vk=W2Tφ2fxk+vk.The objective, now, is to develop an FSOEAHW parameters estimation approach based on gradient algorithm, using a filtered data measurement and minimizing a filtered estimation error given by(60)ek1f,t=ykf−y^kf,t,where the filtered estimated output y^kf,t can be expressed by the equivalent adjustable model:(61)y^kf,t=y^k1f,t=W^2,t−1Tφ^2f,t−1x^k,t,where φ^2f,t−1x^k,t will be expressed later. It represents the estimated filtered observation vector in which the estimated intermediate variable x^k,t is given by (14).Nevertheless,ykf in equation (59) depends on unknown parameter cp, p=1,2,…,nc. To neutralize this problem, we submit a two-step gradient algorithm. In the first step, the unknown parameter cp, p=1,2,…,nc, will be estimated minimizing the criteria Jk2,t (given by equation (62)). Subsequently, unknown parameters set β=W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2 with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2 will be estimated minimizing the criteria Jk1f,t given by equation (66).(62)Jk2,t=12wk−c^t−1Tψ^k,t2.In reality, the quadratic criterionJk2,t contains the unmeasurable output noise vector wk, so the estimation of unknown parameter cp, p=1,2,…,nc, cannot be implemented. Then, we propose replacing wk in (62) by its estimated w^k given by equation (64). Afterward, the new quadratic criterion functions will be defined as(63)Jk2,t=12ek2,t2=12w^k,t−c^t−1Tψ^k,t2,where(64)w^k,t=yk−W^2,t−1Tφ^2,t−1x^k,t.Consequently, the estimation equation of parametercp, p=1,2,…,nc, can be inferred using gradient algorithm minimizing the criteria Jk2,t as follows:(65)c^p,t=c^p,t−1+ηc^pek2,t−w^k−p,t,p=1,2,…,nc.The estimation procedure of unknown parameters setβ is derived using the gradient method minimizing the quadratic criterion function defined as follows:(66)Jk1f,t=12ek1f,t2=12ykf,t−y^kf,t2,where ykf,t and y^kf,t are expressed, respectively, by(67)ykf,t=∑p=0ncc^p,tyk−p,(68)y^kf,t=W^2,t−1Tφ^2f,t−1x^k,t,where(69)φ^2f,t−1x^k=C^tzφ^2,t−1x^k,t.We have(70)C^tz=1+c^1tz−1+⋯+c^nctz−nc.Therefore, the rest of parameters are adjusted using the following equations withc^0,t=1:(71)W^n1,t=W^n1,t−1+ηW^1nek1f,t∂y^kf,t∂W^n1,t−1=W^n1,t−1+ηW^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂W^n1,t−1,n=1,…,N1,where(72)∂x^k−p,t∂W^n1,t−1=∑j=1nbb^j,t−1gn1,t−1uk−p−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1gn1,t−1uk−p−i−j,(73)∂y^k−p,t∂x^k−p,t=∑m=1N2x^k−p−o^m2,t−1σ^m2,t−12gm2,t−1x^k−p,ty^k1−p,t−W^m2,t−1,(74)o^n1,t=o^n1,t−1+ηo^1nek1f,t∂y^kf,t∂o^n1,t−1=o^n1,t−1+ηo^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂o^n1,t−1,(75)∂x^k−p,t∂o^n1,t−1=∑j=1nbb^j,t−1uk−p−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−p−jW^n1,t−1−h^k−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−p−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−p−i−jW^n1,t−1−h^k−p−i−j,t,(76)σ^n1,t=σ^n1,t−1+ησ^1nek1f,t∂y^kf,t∂o^n1,t−1=σ^n1,t−1+ησ^1nek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂σ^n1,t−1,(77)∂x^k−p,t∂σ^n1,t−1=∑j=1nbb^j,t−1uk−p−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−p−jW^n1,t−1−h^k−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−p−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−p−i−jW^n1,t−1−h^k−p−i−j,t,(78)a^i,t=a^i,t−1+ηa^iek1f,t∂y^kf,t∂a^i,t−1=a^i,t−1+ηa^iek1f,t∑p=0ncc^p,t∂y^k−p,t∂x^k−p,t∂x^k−p,t∂a^i,t−1,i=1,2,…,na,(79)∂x^k−p,t∂a^i,t−1=−x^k−p−i,t−∑s=1naa^s,t−1∂x^k−p−s,t∂a^i,t−1,(80)b^j,t=b^j,t−1+ηb^jek1f,t∂y^kf,t∂b^j,t−1=b^j,t−1+ηb^jek1f,t∑p=0ncc^p,t∂y^k1−p,t∂x^k−p,t∂x^k−p,t∂b^j,t−1,j=1,2,…,nb,(81)∂x^k−p,t∂b^j,t−1=h^k−j−p,t−∑s=1naa^s,t−1∂x^k−p−s,t∂b^j,t−1,(82)W^m2,t=W^m2,t−1+ηW^2mek1f,t∑p=0ncc^p,tg^m2,t−1x^k−p,t,m=1,2,…,N2,(83)o^m2,t=o^m2,t−1+ηo^2mek1f,t∑p=0ncc^p,tx^k−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−p,tW^m2,t−1−y^k1−p,t,(84)σ^m2,t=σ^m2,t−1+ηo^2mek1f,t∑p=0ncc^p,tx^k−p,t−o^m2,t−12σ^m2,t−3g^m2,t−1x^k−p,tW^m2,t−1−y^k1−p,t.To summarize the FIBPG algorithm, we give the following steps:Step 1: initialize the parameters of the linear dynamic parts of the two fuzzy systems and the output noise parameters randomly. Fix the learning rates;t=1.At each iterationt and for each sample k, repeat the following steps:Step 2: for inputuk−p−j, calculate h^k−p−j,tj=1,…,nb,p=0,…,nc, x^k−p−i,t, y^k1−p,t, and y^k−p,t using, respectively, equations (13)–(16) replacing k with k-p.Step 3: computew^k,t using equation (64).Step 4: adjust parameterc^p, p=1,2,…,nc, by equation (65).Step 5: computeykf,t, y^kf,t, and ek1f,t based, respectively, on equations (66)–(68).Step 6: update the rest of parameters using equations (71)–(84).Step 7: if the stopping criteriaE given by (7) and the parameter variations (αt−αt−1) are less than fixed small values, then stop; else, go to step 2; t=t+1.


### 3.3.1. FIBPG Convergence Analysis

This subsection aims at the convergence analysis of the FIBPG which can be applied to estimate the parameters of a nonlinear system that can be described by the FSEOFHW model. The convergence properties of the FIBPG algorithm are determined in the following theorem.Theorem 3.
The asymptotic convergence of FIBPG is ensured if the learning rateηc^pp=1,…,nc and those of other FSOEAHW parameters satisfy the following conditions:(85)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where(86)Pc,max=maxk∂w^k,t∂c^p,t−1,Pβ,max=maxk∂y^k∂βt−1.β is an element of a parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2 with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2.Proof.
See Appendix C.


## 3.3.1. FIBPG Convergence Analysis

This subsection aims at the convergence analysis of the FIBPG which can be applied to estimate the parameters of a nonlinear system that can be described by the FSEOFHW model. The convergence properties of the FIBPG algorithm are determined in the following theorem.Theorem 3.
The asymptotic convergence of FIBPG is ensured if the learning rateηc^pp=1,…,nc and those of other FSOEAHW parameters satisfy the following conditions:(85)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where(86)Pc,max=maxk∂w^k,t∂c^p,t−1,Pβ,max=maxk∂y^k∂βt−1.β is an element of a parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2 with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2.Proof.
See Appendix C.


## 3.4. Multi-Innovation and Data Filtering-Based Improved Backpropagation Gradient (MIFIBPG) Algorithm

Similar to Section3.2 and to improve the convergence accuracy, we can combine multi-innovation approach with data filtering technique and then we obtain a multi-innovation FIBPG (MIFIBPG) algorithm.DefineE2Lk,t and E1fLk,t as follows:(87)Ek2L,t=ek2,tek2−1,t⋮ek2−L+1,t=w^k,t−c^t−1Tψ^k,tw^k−1,t−c^t−1Tψ^k−1,t⋮w^k−L+1,t−c^t−1Tψ^k−L+1,t,Ek1fL,t=ek1f,tek1f−1,t⋮ek1f−L+1,t=ykf,t−y^kf,tykf−1,t−y^kf−1,t⋮ykf−L+1,t−y^kf−L+1,t,where(88)ek2−l,t=w^k−l,t−c^t−1Tψ^k−l=yk−W^2,t−1Tφ^2,t−1x^k−l,t−c^t−1Tψ^k−l,ek1f−l,t=ykf−l,t−y^kf−l,t=∑p=0ncc^p,tyk−l−p−W^2,t−1Tφ^2fx^k−l,t,l=0:L−1andc^0=1.Like the previous section and referring to MIIBPG algorithm, initially the vector parameterc^ is adjusted in order to calculate the innovation vector error Ek1fL,t. It is updated according to formula (91) by minimizing the quadratic criterion given by the following equation:(89)Jk2L,t=12Ek2L,tTEk2L,t,c^p,t=c^p,t−1+ηc^p∑l=0L−1ek2−l,t−w^k−l−p,t,p=1:nc.The adjustment equations of all other parameters are gotten using the MIFIBPG algorithm minimizing the cost functionJk1fL,t=1/2Ek1fL,tTEk1fL,t. They are listed as follows:(90)W^m2,t=W^m2,t−1+ηW^2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tg^m2,t−1x^k−l−p,m=1,2,…,N2,(91)o^m2,t=o^m2,t−1+ηo2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tx^k−l−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l−p,tW^m2,t−1−y^k1−l−p,t,(92)σ^m2,t=σ^m2,t−1+ηo2m∑l=0L−1ek1f−l,t∑p=0ncc^p,tx^k−l−p,t−o^m2,t−12σ^m2,t−13g^m2,t−1x^k−l−p,tW^m2,t−1−y^k1−l−p,t,(93)a^i,t=a^i,t−1+ηa^i∑l=0L−1ek1f−l,t∂y^kf−l,t∂a^i,t−1=a^i,t−1+ηa^i∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂a^i,t−1,i=1,2,…,na,where(94)∂y^k−l−p,t∂x^k−l−p,t=∑m=1N2x^k−l−p,t−o^m2,t−1σ^m2,t−12g^m2,t−1x^k−l−p,ty^k1−l−p,t−W^m2,t−1,(95)∂x^k−l−p,t∂a^i,t−1=−x^k−l−p−i−∑s=1naa^s,t−1∂x^k−l−p−s,t∂a^i,t−1,(96)b^j,t=b^j,t−1+ηbj∑l=0L−1ek1f,t∂y^kf−l,t∂b^j,t−1=b^j,t−1+ηb^j∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂b^j,t−1=b^j,t−1+ηb^j∑l=0l−1ek1f−l,t∑p=0ncc^p,t∂y^k−l−p,t∂x^k−l−p,th^k−l−p−j,t−∑s=1naa^s,t−1∂x^k−l−p−s,t∂b^j,t−1,j=1,2,…,nb,(97)W^n1,t=W^n1,t−1+ηW1n∑l=0L−1ek1f−l,t∂y^kf−l,t∂W^n1,t−1=W^n1,t−1+ηW1∑l=0L−1ek1f−l,t∑p=0ncc^p,t∂y^k1−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂Wn1,t−1,n=1,2,…,N1,where(98)∂x^k−l−p,t∂W^n1,t−1=∑j=1nbb^j,t−1g^n1,t−1uk−l−p−j−∑i=1naa^i,t−1∑j=1nbb^j,t−1g^n1,t−1uk−l−p−i−j,(99)o^n1,t=o^n1,t−1+ηo1n∑l=0L−1ek1f−l,t∂y^kf−l,t∂o^n1,t−1=o^n1,t−1+ηo^1n∑l=0L−1ek1f−l,t∑p=0ncc⌢p,t∂y^k−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂o^n1,t−1,(100)∂x^k−l−p∂o^n1,t−1=∑j=1nbb^j,t−1uk−l−p−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−p−jW^n1,t−1−h^k−l−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−p−i−j−o^n1,t−1σ^n1,t−12g^n1,t−1uk−l−p−i−jW^n1,t−1−h^k−l−p−i−j,t,(101)σ^n1,t=σ^n1,t−1+ησ^1n∑l=0L−1ek1f−l∂y^kf−l∂σ^n1,t−1=σ^n1,t−1+ησ^1n∑l=0L−1ek1f−l,t∑p=0ncc⌢p,t∂y^k1−l−p,t∂x^k−l−p,t∂x^k−l−p,t∂σ^n1,t−1,(102)∂x^k−l−p,t∂σ^n1,t−1=∑j=1nbb^j,t−1uk−l−p−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−p−jW^n1,t−1−h^k−l−p−j,t−∑i=1naa^i,t−1∑j=1nbb^j,t−1uk−l−p−i−j−o^n1,t−12σ^n1,t−13g^n1,t−1uk−l−p−i−jW^n1,t−1−h^k−l−p−i−j,t.


### 3.4.1. MIFIBPG Algorithm Convergence Study

Taking into account the mentioned theorems, we can demonstrate readily the convergence properties of this MIFIBPG algorithm for the considered FSOEAHW, which is enunciated by the following theorem.Theorem 4.
The convergence of the MIFIBPG algorithm is ensured if the learnings ratesηc^p and ηβ, where β represents each element of the parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2, with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2, satisfy the following inequalities:(103)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where Pc^p,max=maxk∑l=0L−1∂w^k−l,t/∂c^p,t−1,Pβ,max=maxk∑l=0L−1∂y^k−l/∂βt−1.The theorem’s proof is similar to those given in Appendices B and C for the MIIIBPG and FIBPG algorithms convergence study.


## 3.4.1. MIFIBPG Algorithm Convergence Study

Taking into account the mentioned theorems, we can demonstrate readily the convergence properties of this MIFIBPG algorithm for the considered FSOEAHW, which is enunciated by the following theorem.Theorem 4.
The convergence of the MIFIBPG algorithm is ensured if the learnings ratesηc^p and ηβ, where β represents each element of the parameter set W^n1,o^n1,σ^n1,a^i,b^j,W^m2,o^m2,σ^m2, with n=1,…,N1,i=1,…,na,j=1,…,nb,m=1,…,N2, satisfy the following inequalities:(103)0≺ηc^p≺2Pc,max2,0≺ηβ≺2Pβ,max2,where Pc^p,max=maxk∑l=0L−1∂w^k−l,t/∂c^p,t−1,Pβ,max=maxk∑l=0L−1∂y^k−l/∂βt−1.The theorem’s proof is similar to those given in Appendices B and C for the MIIIBPG and FIBPG algorithms convergence study.


## 4. Example

In this section, we achieve an illustration simulation in order to test the performance and the efficiency of the developed algorithms. Therefore, we consider a nonlinear stochastic system given by stochastic output-error Hammerstein–Wiener mathematical model in which its blocks are represented using the following expressions:(104)hk=f1uk=0,ifuk≤0.15,1−exp−uk−0.2exp−2ukcos3uk,if0.15<uk<3.5,0.96,ifuk≥3.5,Gz=BzAz=b1z−1+b2z−21+a1z−1+a2z−2=z−1+0.15z−21−0.2z−1+0.35z−2,yk1=f2xk=0,ifxk≤−0.3,1+tanhxk−1,if−0.3<xk<1.3,0.96,ifxk≥1.3,1Cz=11+c1z−1=11+0.15z−1.A set of 9000 random values between 0 and 4 are used as inputuk. vk is a white noise sequence with zero mean and noise-to-signal ratio as γns=varvk/vary1k×100=13%. The input uk and the corresponding output yk are presented in Figure 3.Figure 3
The system’s input and output versusk.
(a)(b)The parameters intervening in the proposed FSOEAHW are chosen as follows.The input nonlinear block is approximated using a fuzzy system composed by 8 rules in which consequent parts are initialized randomly between 0 and 1 and the membership function parameterso^1n and σ^1nn=1,…,8 are initialized, respectively, in intervals 04 and 0.10.35.The initial values of parametersa^i, b^j, c^p, i=1,2, j=1,2, and p=1 are considered, respectively, as 0.25,−0.25,0.5,0.1and0.26.The output nonlinear part is depicted by 6 fuzzy rules where the consequent parts parameters are initialized randomly between 0 and 1.5 and the membership function parameterso^m2,0 and σ^m2,0 are taken, respectively, in intervals −0.51.5 and 0.150.25. Numerical simulations are realized to estimate the parameters intervening in the discrete time FSOEAHW.


### 4.1. Parametric Estimation Using IBPG Algorithm

We propose estimating the FSOEAHW parameters using the IBPG algorithm described in Section3.1. Thus, we demonstrate the estimation results by giving the evolution curves of the intermediate variables and output estimates h^k,t, x^k,t, y^1k,t, and y^k,t in Figure 4.Figure 4
Estimation results of the FSOEAHW model using IBPG for the first 1000 examples.In addition, Figure5 represents the evolution curves of the estimation error ek1,t and the parametric distance δt=θ−θ^t/θ in which θ=θlT,cT and θ^t=θ^l,tT,c^tT. In Table 1, we present some obtained numerical values of vector parameter estimates θ^t and the corresponding parametric error for t = 0, 2000, 4000, 6000, and 9000. In addition, to evaluate the estimation quality, we depict in Table 2 the estimation errors (eh=h−h^/h, ex=x−x^/x, ey1=y1−y^1/y1, ew=w−w^/w, and ey=y−y^/y).Figure 5
Evolution curves of estimation error (a) and parametric error (b) for the FSOEAHW model using IBPG algorithm.
(a)(b)Table 1
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using IBPG algorithm.
Algorithmt, ka^1a^2b^1b^2c^1δtIBPG0.0000.2500−0.25000.50000.10000.26000.82522000−0.03380.25030.97960.13650.11120.18084000−0.17610.33220.97580.14910.15070.03506000−0.19300.37571.00220.15180.15770.0254Table 2
Estimation errors for the FSOEAHW model using IBPG algorithm.
Algorithmehexey1eweyIBPG0.00210.00610.00070.51020.0017As is noticed in the above-presented simulation results, a satisfactory estimation is achieved with the proposed IBPG algorithm despite the existence of the noise acting on the process output without restrictive assumptions. Furthermore, the parameter estimation errors gradually become, iteratively, smaller. Thus, the parameter estimates are closer to the true values and the fuzzy models generate a satisfactory estimation accuracy of the nonlinear input and output blocks.For comparison purpose, we propose giving simulation results in which the input and output nonlinear parts are estimated by two independent polynomial functions expressed by the following equations:(105)hk=∑n=1N1χnukn,yk1=∑m=1N2βmxkm,where χnn=1:N1 and βmm=1:N2 are constant parameters. Then the corresponding output-error autoregressive HW (OEAHW) system parameters are χn, ai, bj, cp, and βmn=1:3,i=1:2,j=1:2,p=1,m=1:3. The proposed IBPG algorithm is adopted for parameter adjustment based on updating equations given in Appendix D. It should be noticed that OEAHW is inspired from [24], where the IBPG algorithm is used to estimate a polynomial Wiener model.In this sense, Figure6 illustrates the evolution curves of the estimation and parametric errors for the OEAHW and FSOEAHW models. Table 3 presents the OEAHW quality estimation.Figure 6
Evolution curves of estimation error (a) and parametric error (b) for the FSOEAHW and OEAHW models.
(a)(b)Table 3
Estimation errors for the OEAHW model using IBPG algorithm.
Algorithmehexey1eweyIBPG0.01310.02120.024318.64390.0315The obtained results confirm that FSOEAHW model generates a more satisfactory estimation accuracy compared to the OEAHWM model with polynomial form. Due to the ability of the fuzzy systems for providing a good approximation of the two nonlinear parts, the proposed FSOEAHW reaches a good performance with a comparatively small amount of calculation and the resulting estimated model can capture systems dynamics correctly.


### 4.2. Parametric Estimation Using MIIBPG Algorithm

Using the MIIBPG algorithm for FSOEAHW parameter identification, the vector parameter estimatesθ^t and their corresponding parametric errors δtt=0,2000,4000,6000,9000 are given in Table 4 for L = 2, 5, and 7. Furthermore, for same values of L, the evolution curves of δt and estimation error variances σk,t2 are presented, respectively, in Figures 7 and 8 with σk,t2=∑i=1kek1,t−ek1,t¯2/k in which ek1,t¯ is a statistical mean of the estimation error. The estimation errors of system variables are given in Table 5.Table 4
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using MIIBPG algorithm with L = 2, 5, and 7.
Lt, ka^1a^2b^1b^2c^1δt20.0000.2500−0.25000.50000.10000.26000.82522000−0.10360.30090.98970.13800.15140.09754000−0.19010.34090.98000.14870.15140.02136000−0.19000.35421.01250.14950.15140.01729000−0.20010.35121.01310.15130.15140.012150.0000.2500−0.25000.50000.10000.26000.82522000−0.17050.34800.94200.14440.15420.04284000−0.20190.34930.96080.15110.15420.03336000−0.19380.35720.99130.15150.15420.00899000−0.19990.35291.00810.15220.15420.009070.0000.2500−0.25000.50000.10000.26000.82522000−0.20740.36090.97870.14680.15220.02244000−0.21230.34590.99560.15000.15220.01236000−0.19610.35251.00170.14900.15220.00399000−0.20010.35011.00330.14940.15220.0037Figure 7
Evolution curves of parametric error for the FSOEAHW model using MIIBPG algorithm.Figure 8
Evolution curves of estimation error variances for the FSOEAHW model using MIIBPG algorithm.Table 5
Estimation errors for the FSOEAHW model using MIIBPG algorithm withL = 2, 5, and 7.
Lehexey1ewey20.00080.00460.00070.47400.001850.00060.00330.00060.38300.001670.00060.00130.00050.30260.0015By examining Tables4 and 5 and Figures 7 and 8, we can affirm that the parameter and estimation errors gradually become smaller with the iterative variable and innovation length increasing. In addition, the developed MIIBPG algorithm drives to have higher performance than the standard IBPG algorithm. In fact, the system parameter estimates converge to their true values and the estimation errors decrease to smaller values. In this direction, the proposed multi-innovation approach can estimate the parameters effectively and increasing the innovation length can improve parameter and variable estimation accuracy and accelerate the convergence rate because the algorithm uses more information in each iteration. Hence, the proposed algorithm is effective for the FSOEAHW model identification.


### 4.3. Parametric Estimation Using FIBPG Algorithm

In this section, the efficiency of the FIBPG algorithm compared to the IBPG algorithm is well noticed by visualizing the evolutions curves of the estimation error varianceσek1,t2 and a parametric error δt given, respectively, in Figures 9 and 10. In fact, a smooth and a more precise parametric convergence is ensured. This is more approved by examining Tables 6 and 7 that show the estimation accuracy of FSOEAHW model.Figure 9
Evolution curves of estimation error variance for the FSOEAHW model using FIBPG algorithm.Figure 10
Evolution curves of parametric error for the FSOEAHW model using FIBPG algorithm.Table 6
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using FIBPG algorithm.
FIBPG algorithmt, ka^1a^2b^1b^2c^1δt0.0000.2500−0.25000.50000.10000.26000.82522000−0.02920.26341.01850.13050.14450.17484000−0.16750.32681.00570.14220.14450.03846000−0.18010.35581.00460.14380.14450.02039000−0.19950.35481.00460.14570.14450.0088Table 7
Estimation errors for the FSOEAHW model using FIBPG algorithm.
FIBPG algorithmehexey1ewey0.00070.00470.00060.46080.0015


### 4.4. Parametric Estimation Using MIFIBPG Algorithm

In this part, we exploit the MIFIBPG algorithm to estimate the parameters of the FSOEAHW. Thus, Tables8 and 9 present the parameter estimates and their estimation errors for L = 2, 5, and 7. For the same innovation length values, Figures 11 and 12 give the estimation error variance σek1,t2 and the parametric distance δt.Table 8
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using MIFIBPG algorithm with L = 2, 5, and 7.
Lt, ka^1a^2b^1b^2c^1δt20.0000.2500−0.25000.50000.10000.26000.82522000−0.11180.30711.00590.13460.15430.08854000−0.19210.33541.00580.14380.15430.01746000−0.18870.35011.00580.14380.15430.01249000−0.20060.34891.00580.14620.15430.007550.0000.2500−0.25000.50000.10000.26000.82522000−0.19320.34811.00300.14720.14780.00684000−0.21210.34271.00290.15070.14780.01286000−0.19090.35281.00290.14950.14780.00669000−0.19840.35051.00290.15030.14780.003770.0000.2500−0.25000.50000.10000.26000.82522000−0.22390.37740.99990.14890.15010.03324000−0.21650.34291.00000.15050.15010.01446000−0.19440.35311.00000.14950.15010.00349000−0.19990.35061.00000.14990.15010.0006Table 9
Estimation errors for the FSOEAHW model using MIFIBPG algorithm withL = 2, 5, and 7.
MIFIBPG algorithmehexey1ewey20.00050.00380.00050.40760.001650.00050.00160.00050.33250.001570.00060.00220.00050.30460.0015Figure 11
Evolution curves of estimation error variance for the FSOEAHW model using FIBPG algorithm.Figure 12
Evolution curves of parametric error for the FSOEAHW model using MIFIBPG algorithm.The proposed multi-innovation approach combined with data filtering technique can estimate the parameters effectively and smoothly and increasing the innovation length gives a good parameter estimation accuracy and accelerates the convergence rate and then improves the gradient algorithm drawbacks.


## 4.1. Parametric Estimation Using IBPG Algorithm

We propose estimating the FSOEAHW parameters using the IBPG algorithm described in Section3.1. Thus, we demonstrate the estimation results by giving the evolution curves of the intermediate variables and output estimates h^k,t, x^k,t, y^1k,t, and y^k,t in Figure 4.Figure 4
Estimation results of the FSOEAHW model using IBPG for the first 1000 examples.In addition, Figure5 represents the evolution curves of the estimation error ek1,t and the parametric distance δt=θ−θ^t/θ in which θ=θlT,cT and θ^t=θ^l,tT,c^tT. In Table 1, we present some obtained numerical values of vector parameter estimates θ^t and the corresponding parametric error for t = 0, 2000, 4000, 6000, and 9000. In addition, to evaluate the estimation quality, we depict in Table 2 the estimation errors (eh=h−h^/h, ex=x−x^/x, ey1=y1−y^1/y1, ew=w−w^/w, and ey=y−y^/y).Figure 5
Evolution curves of estimation error (a) and parametric error (b) for the FSOEAHW model using IBPG algorithm.
(a)(b)Table 1
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using IBPG algorithm.
Algorithmt, ka^1a^2b^1b^2c^1δtIBPG0.0000.2500−0.25000.50000.10000.26000.82522000−0.03380.25030.97960.13650.11120.18084000−0.17610.33220.97580.14910.15070.03506000−0.19300.37571.00220.15180.15770.0254Table 2
Estimation errors for the FSOEAHW model using IBPG algorithm.
Algorithmehexey1eweyIBPG0.00210.00610.00070.51020.0017As is noticed in the above-presented simulation results, a satisfactory estimation is achieved with the proposed IBPG algorithm despite the existence of the noise acting on the process output without restrictive assumptions. Furthermore, the parameter estimation errors gradually become, iteratively, smaller. Thus, the parameter estimates are closer to the true values and the fuzzy models generate a satisfactory estimation accuracy of the nonlinear input and output blocks.For comparison purpose, we propose giving simulation results in which the input and output nonlinear parts are estimated by two independent polynomial functions expressed by the following equations:(105)hk=∑n=1N1χnukn,yk1=∑m=1N2βmxkm,where χnn=1:N1 and βmm=1:N2 are constant parameters. Then the corresponding output-error autoregressive HW (OEAHW) system parameters are χn, ai, bj, cp, and βmn=1:3,i=1:2,j=1:2,p=1,m=1:3. The proposed IBPG algorithm is adopted for parameter adjustment based on updating equations given in Appendix D. It should be noticed that OEAHW is inspired from [24], where the IBPG algorithm is used to estimate a polynomial Wiener model.In this sense, Figure6 illustrates the evolution curves of the estimation and parametric errors for the OEAHW and FSOEAHW models. Table 3 presents the OEAHW quality estimation.Figure 6
Evolution curves of estimation error (a) and parametric error (b) for the FSOEAHW and OEAHW models.
(a)(b)Table 3
Estimation errors for the OEAHW model using IBPG algorithm.
Algorithmehexey1eweyIBPG0.01310.02120.024318.64390.0315The obtained results confirm that FSOEAHW model generates a more satisfactory estimation accuracy compared to the OEAHWM model with polynomial form. Due to the ability of the fuzzy systems for providing a good approximation of the two nonlinear parts, the proposed FSOEAHW reaches a good performance with a comparatively small amount of calculation and the resulting estimated model can capture systems dynamics correctly.


## 4.2. Parametric Estimation Using MIIBPG Algorithm

Using the MIIBPG algorithm for FSOEAHW parameter identification, the vector parameter estimatesθ^t and their corresponding parametric errors δtt=0,2000,4000,6000,9000 are given in Table 4 for L = 2, 5, and 7. Furthermore, for same values of L, the evolution curves of δt and estimation error variances σk,t2 are presented, respectively, in Figures 7 and 8 with σk,t2=∑i=1kek1,t−ek1,t¯2/k in which ek1,t¯ is a statistical mean of the estimation error. The estimation errors of system variables are given in Table 5.Table 4
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using MIIBPG algorithm with L = 2, 5, and 7.
Lt, ka^1a^2b^1b^2c^1δt20.0000.2500−0.25000.50000.10000.26000.82522000−0.10360.30090.98970.13800.15140.09754000−0.19010.34090.98000.14870.15140.02136000−0.19000.35421.01250.14950.15140.01729000−0.20010.35121.01310.15130.15140.012150.0000.2500−0.25000.50000.10000.26000.82522000−0.17050.34800.94200.14440.15420.04284000−0.20190.34930.96080.15110.15420.03336000−0.19380.35720.99130.15150.15420.00899000−0.19990.35291.00810.15220.15420.009070.0000.2500−0.25000.50000.10000.26000.82522000−0.20740.36090.97870.14680.15220.02244000−0.21230.34590.99560.15000.15220.01236000−0.19610.35251.00170.14900.15220.00399000−0.20010.35011.00330.14940.15220.0037Figure 7
Evolution curves of parametric error for the FSOEAHW model using MIIBPG algorithm.Figure 8
Evolution curves of estimation error variances for the FSOEAHW model using MIIBPG algorithm.Table 5
Estimation errors for the FSOEAHW model using MIIBPG algorithm withL = 2, 5, and 7.
Lehexey1ewey20.00080.00460.00070.47400.001850.00060.00330.00060.38300.001670.00060.00130.00050.30260.0015By examining Tables4 and 5 and Figures 7 and 8, we can affirm that the parameter and estimation errors gradually become smaller with the iterative variable and innovation length increasing. In addition, the developed MIIBPG algorithm drives to have higher performance than the standard IBPG algorithm. In fact, the system parameter estimates converge to their true values and the estimation errors decrease to smaller values. In this direction, the proposed multi-innovation approach can estimate the parameters effectively and increasing the innovation length can improve parameter and variable estimation accuracy and accelerate the convergence rate because the algorithm uses more information in each iteration. Hence, the proposed algorithm is effective for the FSOEAHW model identification.


## 4.3. Parametric Estimation Using FIBPG Algorithm

In this section, the efficiency of the FIBPG algorithm compared to the IBPG algorithm is well noticed by visualizing the evolutions curves of the estimation error varianceσek1,t2 and a parametric error δt given, respectively, in Figures 9 and 10. In fact, a smooth and a more precise parametric convergence is ensured. This is more approved by examining Tables 6 and 7 that show the estimation accuracy of FSOEAHW model.Figure 9
Evolution curves of estimation error variance for the FSOEAHW model using FIBPG algorithm.Figure 10
Evolution curves of parametric error for the FSOEAHW model using FIBPG algorithm.Table 6
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using FIBPG algorithm.
FIBPG algorithmt, ka^1a^2b^1b^2c^1δt0.0000.2500−0.25000.50000.10000.26000.82522000−0.02920.26341.01850.13050.14450.17484000−0.16750.32681.00570.14220.14450.03846000−0.18010.35581.00460.14380.14450.02039000−0.19950.35481.00460.14570.14450.0088Table 7
Estimation errors for the FSOEAHW model using FIBPG algorithm.
FIBPG algorithmehexey1ewey0.00070.00470.00060.46080.0015


## 4.4. Parametric Estimation Using MIFIBPG Algorithm

In this part, we exploit the MIFIBPG algorithm to estimate the parameters of the FSOEAHW. Thus, Tables8 and 9 present the parameter estimates and their estimation errors for L = 2, 5, and 7. For the same innovation length values, Figures 11 and 12 give the estimation error variance σek1,t2 and the parametric distance δt.Table 8
The vector parameter estimatesθ^t and their parametric errors δt for the FSOEAHW model using MIFIBPG algorithm with L = 2, 5, and 7.
Lt, ka^1a^2b^1b^2c^1δt20.0000.2500−0.25000.50000.10000.26000.82522000−0.11180.30711.00590.13460.15430.08854000−0.19210.33541.00580.14380.15430.01746000−0.18870.35011.00580.14380.15430.01249000−0.20060.34891.00580.14620.15430.007550.0000.2500−0.25000.50000.10000.26000.82522000−0.19320.34811.00300.14720.14780.00684000−0.21210.34271.00290.15070.14780.01286000−0.19090.35281.00290.14950.14780.00669000−0.19840.35051.00290.15030.14780.003770.0000.2500−0.25000.50000.10000.26000.82522000−0.22390.37740.99990.14890.15010.03324000−0.21650.34291.00000.15050.15010.01446000−0.19440.35311.00000.14950.15010.00349000−0.19990.35061.00000.14990.15010.0006Table 9
Estimation errors for the FSOEAHW model using MIFIBPG algorithm withL = 2, 5, and 7.
MIFIBPG algorithmehexey1ewey20.00050.00380.00050.40760.001650.00050.00160.00050.33250.001570.00060.00220.00050.30460.0015Figure 11
Evolution curves of estimation error variance for the FSOEAHW model using FIBPG algorithm.Figure 12
Evolution curves of parametric error for the FSOEAHW model using MIFIBPG algorithm.The proposed multi-innovation approach combined with data filtering technique can estimate the parameters effectively and smoothly and increasing the innovation length gives a good parameter estimation accuracy and accelerates the convergence rate and then improves the gradient algorithm drawbacks.


## 5. Conclusion

This paper deals with modeling and identification of SOEAHW systems based on an artificial intelligence technique. Two fuzzy models with adjustable parameters are used to identify the input and output nonlinear blocks. The output nonlinearity may be noninvertible. A fuzzy model having as input the linear dynamic output estimate approximates it. The four given parameter estimation algorithms (IBPG, MIIBPG, FIBPG, and MIFIBPG) are based on recursive gradient algorithm, multi-innovation technique, and data filtering approach. They have convergence property and validity for online implementation. Simulation results validate the effectiveness of the proposed algorithms. However, the drawbacks of the proposed algorithms lie, particularly, in the updated parameter initialization and the learning rate choice. In fact, it is well known that adjustment equations based on gradient algorithm depend directly on these parameter choices and these have a crucial effect on the convergence and accuracy. The work presented in this paper can be extended to other classes of stochastic nonlinear multivariable systems with colored noise. Furthermore, we can suggest proposing an adaptive fuzzy control of nonlinear industrial process (such as practical hydraulic process) based on the presented SOEAHW model. Finally, a more suitable approach for an appropriate learning rate’s research and other optimization techniques can be proposed to give better performance.



---

*Source: 8525090-2021-08-20.xml*