# Least-Squares Based and Gradient Based Iterative Parameter Estimation Algorithms for a Class of Linear-in-Parameters Multiple-Input Single-Output Output Error Systems

Authors: Cheng Wang; Tao Tang; Dewang Chen
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

The identification of a class of linear-in-parameters multiple-input single-output systems is considered. By using the iterative search, a least-squares based iterative algorithm and a gradient based iterative algorithm are proposed. A nonlinear example is used to verify the effectiveness of the algorithms, and the simulation results show that the least-squares based iterative algorithm can produce more accurate parameter estimates than the gradient based iterative algorithm.


---

## Body

## 1. Introduction

Parameter estimation plays an important role in adaptive forecasting [1], system modeling [2–6], system control [7–9], and adaptive control [10–15]. For decades, many identification methods have been developed [16–20], for example, the bias compensation based least-squares methods [21–23] and the iterative identification methods [24–26]. These methods can be used for identifying linear systems and nonlinear systems. In the literature, Ding presented a decomposition based fast least-squares algorithm for output error systems [27]. Recursive algorithms and iterative algorithms are two types of parameter estimation algorithms. The recursive algorithms use the data as it becomes available [28], whereas the iterative algorithms tend to exploit the advantage of processing a complete batch of available data, which can provide highly accurate parameter estimation. Iterative methods can also be used for solving matrix equations [29–31]. In the literature, Ding proposed a two-stage least-squares based iterative parameter estimation algorithm for CARARMA systems using the decomposition technique [32].As a basic class of multivariable systems, multiple-input single-output (MISO) systems have lots of applications in industrial processes. Several works on MISO system identification have been reported [33]. For example, in order to improve the convergence rate, Liu et al. developed a stochastic gradient algorithm for MISO systems using the multi-innovation theory [34]. The least-squares methods can also be found in the literature.Recently, Wang and Tang studied the identification algorithms for a class of linear-in-parameters single-input single-output (SISO) systems with colored noises using the recursive least-squares method [35]. In this work, we extend these results from SISO systems into a class of linear-in-parameters MISO systems with the colored noises shown in Figure 1 [36, 37]. Consider
(1)y(t)=∑j=1rϑjTηj(uj(t))Aj(z)+D(z)v(t),
where y(t)∈R is the system output, {uj(t)∈R,j=1,2,…,r} are the system inputs, and v(t)∈R is the stochastic white noise with zero mean. Aj(z) and D(z) are polynomials, of known orders (nj,nd), in the unit backward shift operator z-1, and defined by
(2)Aj(z):=1+aj1z-1+aj2z-2+⋯+ajnjz-nj∈R,D(z):=1+d1z-1+d2z-2+⋯+dndz-nd∈R,aji, di, and ϑjT are the unknown parameters to be estimated. The superscript T denotes the matrix/vector transpose. It is worth noting that the models in (1) include but are not limited to linear MISO systems; that is, when ϑj and ηj(uj(t)) are defined by
(3)ϑj∶=[bj1,bj2,…,bjnj]T∈Rnj,ηj(uj(t))∶=[uj(t),uj(t-1),…,uj(t-nj)]T∈Rnj,
system (1) denotes an MISO output error moving average system. When ηj(uj(t)) is a nonlinear function of uj(t), for example,
(4)ηj(uj(t))∶=[sin(uj(t-1)),uj3(t),…,2tan(uj(t-nj))]T∈Rnj,
system (1) denotes a nonlinear MISO system.Figure 1
The linear-in-parameters multiple-input single-output output error moving average systems.On the basis of the iterative algorithms for linear-in-parameters SISO systems [37, 38], this paper develops the least-squares based and gradient based iterative identification algorithms to improve the parameter estimation accuracy for a class of linear-in-parameters MISO output error moving average systems. Compared with the gradient based iterative algorithm, the least-squares based iterative algorithm can provide more accurate parameter estimates.The remainder of this paper is organized as follows. Section2 introduces the identification model. Section 3 derives the least-squares based iterative algorithm. Section 4 proposes a gradient based iterative algorithm. Section 5 presents an illustrative example to show the effectiveness of the algorithms. Finally, concluding remarks are offered in Section 6.


## 2. The Identification Model

Let us define some symbols. The symbolIn denotes an identity matrix of order n; 1n denotes an n-dimensional column vector whose elements are 1; λmax⁡[X] and X-1 represent the maximum eigenvalue and the inverse of the square matrix X.To further develop new identification algorithms for estimating the parameter vectorϑj and the parameters of Aj(z) and D(z) by utilizing the input-output measured data {uj(t),y(t):t=1,2,…}, we derive an identification model for system (1). Without loss of generality, assume that uj(t)=0, y(t)=0, and v(t)=0 for t≤0.Define the intermediate variables as follows:(5)kxj(t)∶=ϑjTηj(uj(t))Aj(z)KKKK=-∑i=1na‍ajixj(t-i)+ϑjTηj(uj(t)),Kw(t)∶=D(z)v(t).Define the parameter vectors as follows:(6)θ≔[θsθn]∈Rn0+nd,θs:=[θ1T,θ2,T,…,θrT]T∈Rn0,n0:=2∑j=1rnj,θn:=[d1,d2,…,dnd]T∈Rnd,θj:=[aj1,aj2,…,ajnj,ϑjT]T∈R2nj,and define the information vectors as follows:(7)φ(t)≔[φs(t)φn(t)]∈Rn0+nd,φs(t)∶=[φ1T(t),φ2T(t),…,φrT(t)]T∈Rn0,kkkkkkkkkkkkkkkkkkkkn0=2∑j=1r‍nj,φn(t)∶=[-v(t-1),-v(t-2),…,-v(t-nd)]T∈Rnd,φj(t)∶=[-xj(t-1),-xj(t-2),…,-xj(t-nj),KKKKηjT(uj(t))]T∈R2nj.Then we can express (5) as
(8)xj(t)=φJT(t)θj,w(t)=φnT(t)θn+v(t),and system (1) can be rewritten as
(9)y(t)=∑j=1r‍xj(t)+w(t)=φsT(t)θs+φnT(t)θn+v(t)=φT(t)θ+v(t).
Equation (9) is the identification model of system (1), and parameter vector θ contains all the parameters of the system.


## 3. The Least-Squares Based Iterative Algorithm

Consider the newestp data from t-p+1 to t and define the quadratic criterion function as follows:
(10)J(θ):=∑i=0p-1‍[y(t-i)-φT(t-i)θ]2.
By minimizing J(θ) and letting the derivative of J(θ) with respect to θ be zero, we can obtain the least-squares estimate of θ as
(11)θ^(t)=[∑i=0p-1‍φ(t-i)φT(t-i)]-1[∑i=0p-1‍φ(t-i)y(t-i)].
The above estimate θ^(t) is impossible to implement due to the unknown noise-free outputs xj(t-i) and unmeasurable noise items v(t-i) in φ(t). Here, the difficulties are solved by using the iterative identification technique [38]: let k=1,2,… be the iterative variable, and let θ^j,k(t) and θ^k(t) be the iterative estimates of θj and θ at iteration k, replace the unknown items xj(t-i) and v(t-i) with their iterative estimates x^j,k(t-i) and v^k(t-i) at iteration k, and define the estimated vectors as follows:
(12)φ^k(t):=[φ^s,kT(t),φ^n,kT(t)]T∈Rn0+nd,φ^s,k(t)∶=[φ^1,kT(t),φ^2,kT(t),…,φ^r,kT(t)]T∈Rn0,kkkkkkkkkkkkkkkkkkkkkn0=2∑j=1r‍nj,φ^n,k(t)∶=[v^k(t-1),v^k(t-2),…,v^k(t-nd)]T∈Rnd,φ^j,k(t)∶=[-x^j,k(t-1),-x^j,k(t-2),…,kkk-x^j,k(t-nj),ηjT(uj(t))]T∈R2nj.
Let θ^j,k(t) and θ^k(t) be the estimates of θj and θ at iteration k, let x^j,k(t) and v^k(t) be the estimates of xj(t) and v(t) at iteration k. Replacing φ(t-i) in (11) with its corresponding estimate φ^k(t-i), we can obtain the following least-squares based iterative algorithm for MISO systems in (1) (the MISO-LSI algorithm for short) into:
(13)θ^(t)=[∑i=0p-1‍φ^k(t-i)φ^kT(t-i)]-1[∑i=0p-1‍φ^k(t-i)y(t-i)],(14)φ^k(t)=[φ^s,kT(t),φ^n,kT(t)]T,(15)φ^s,k(t)=[φ^1,kT(t),φ^2,kT(t),…,φ^r,kT(t)]T,(16)φ^j,k(t)=[-x^j,k(t-1),-x^j,k(t-2),…,-x^j,k(t-nj),KKηjT(uj(t))]T,(17)φ^n,k(t)=[v^k(t-1),v^k(t-2),…,v^k(t-nd)]T,(18)x^j,k(t-i)=φ^j,kT(t-i)θ^j,k(t),(19)v^k(t-i)=y(t-i)-φ^kT(t-i)θ^k(t),(20)θ^k(t)=[θ^s,kT(t),θ^n,kT(t)]T,(21)θ^s,k(t)=[θ^1,kT(t),θ^2,kT(t),…,θ^r,kT(t)]T,(22)θ^n,k(t)=[d^1,k(t),d^2,k(t),…,d^nd,k(t)]T,(23)θ^j,k(t)∶=[a^j1,k(t),a^j2,k(t),…,a^jnj,k(t),ϑ^j,kT(t)]T.
The steps of computing θ^k(t) involved in the algorithm are summarized as follows. (1)
Givenp, let t=p and collect the input-output data {uj(i),y(i):i=0,1,…,p-1}.(2)
Collect the present input-output datauj(t) and y(t).(3)
To initialize, letk=1, x^j,0(t-i) = random number, and v^0(t-i) = random number, i=1,2,…,max⁡[nj,nd].(4)
Formφ^s,k(t-i) using (15) and φ^n,k(t-i) using (17), and form φ^j,k(t-i) using (16) and φ^k(t) using (14).(5)
Update the parameter estimateθ^k(t) using (13).(6)
Computex^j,k(t-i) using (18) and v^k(t-i) using (19).(7)
If∥θ^k(t)-θ^k-1(t)∥⩽ε (a given small number), obtain the iterative time k and the parameter estimate θ^k(t); let θ^0(t+1)=θ^k(t), increase t by 1, and go to Step 2; otherwise, increase k by 1 and go to Step 4.


## 4. The Gradient Based Iterative Algorithm

By minimizingJ(θ) through the negative gradient search, we obtain the following recursive relation of computing the estimate of θ at iteration k:
(24)θ^k(t)=θ^k-1(t)+μk(t)2grad[J(θ^k-1(t))]=θ^k-1(t)+μk(t)∑i=0p-1φ(t-i)KKKKKKKKKKKK×[y(t-i)-φT(t-i)θ^k-1(t)],
where μk(t) is the step-size or the convergence factor to be given later. The same difficulties arise in that the noise-free outputs xj(t-i) in φs(t) and the noise items v(t-i) in φn(t) of φ(t) on the right-hand side of (24) are unknown. Here we apply the same scheme used in the previous section, replacing the unknown vectors with their corresponding iterative estimates. Referring to the method in [38], replacing φ(t-i) in (24) with φ^k(t-i), we can summarize the following gradient based iterative algorithm for MISO systems in (1) (the MISO-GI algorithm for short):
(25)θ^k(t)=θ^k-1(t)+μk(t)×∑i=0p-1‍φ^k(t-i)[y(t-i)-φ^kT(t-i)θ^k-1(t)],(26)φ^k(t)=[φ^s,kT(t),φ^n,kT(t)]T,(27)φ^s,k(t)=[φ^1,kT(t),φ^2,kT(t),…,φ^r,kT(t)]T,(28)φ^j,k(t)=[-x^j,k(t-1),-x^j,k(t-2),…,-x^j,k(t-nj),KKKηjT(uj(t))]T,(29)φ^n,k(t)=[v^k(t-1),v^k(t-2),…,v^k(t-nd)]T,(30)x^j,k(t-i)=φ^j,kT(t-i)θ^j,k(t),(31)v^k(t-i)=y(t-i)-φ^kT(t-i)θ^k(t),(32)μk(t)⩽2λmax⁡-1[∑i=0p-1‍φ^k(t-i)φ^kT(t-i)],(33)θ^k(t)=[θ^s,kT(t),θ^n,kT(t)]T,(34)θ^s,k(t)=[θ^1,kT(t),θ^2,kT(t),…,θ^r,kT(t)]T,(35)θ^n,k(t)=[d^1,k(t),d^2,k(t),…,d^nd,k(t)]T,(36)θ^j,k(t):=[a^j1,k(t),a^j2,k(t),…,a^jnj,k(t),ϑ^j,kT(t)]T.
The steps of computing θ^k(t) involved in the algorithm are summarized as follows. (1)
Givenp, let t=p and collect the input-output data {uj(i),y(i):i=0,1,…,p-1} set θ^0(t)=1n0/p0, p0=106.(2)
Collect the present input-output datauj(t) and y(t).(3)
To initialize, letk=1, x^j,0(t-i)=1/p0, and v^0(t-i)=1/p0, i=1,2,…,max⁡[nj,nd].(4)
Formφ^s,k(t-i) using (27) and φ^n,k(t-i) using (29), and form φ^j,k(t-i) using (28).(5)
Choose an appropriate step-sizeμk(t) using (32) and update the parameter estimate θ^k(t) using (25).(6)
Computex^j,k(t-i) using (30) and v^k(t-i) using (31).(7)
If∥θ^k(t)-θ^k-1(t)∥⩽ε (a given small number), obtain the iterative time k and the parameter estimate θ^k(t); let θ^0(t+1)=θ^k(t), increase t by 1, and go to Step 2; otherwise, increase k by 1 and go to Step 4.


## 5. Example

Consider the following nonlinear multiple-input single-output simulation system:(37)y(t)=ϑ1Tη1(u1(t))A1(z)+ϑ2Tη2(u2(t))A2(z)+D(z)v(t),A1(z)=1+a1z-1=1-0.38z-1,A2(z)=1+a2z-1=1-0.44z-1,D(z)=1+d1z-1=1+0.69z-1,η1(u1(t))=u1(t-1),η2(u2(t))=u22(t-1),ϑ1=b1=1.48,ϑ2=b2=1.58,θ=[a1,b1,a2,b2,d1]T=[-0.38,1.48,-0.44,1.58,0.69]T.
Here, the inputs {u1(t)} and {u2(t)} are taken as uncorrelated persistent excitation signal sequences with zero means and unit variances and {v(t)} as a white noise sequence with zero mean.Usingt=p=1000 data and applying the MISO-GI algorithm in (25)–(32) and the MISO-LSI algorithm in (13)–(19) to estimate the parameters of this nonlinear system, the parameter estimates of each algorithm and their errors with noise variance σ2=0.502 are shown in Table 1; the parameter estimation errors δ∶=∥θ^k(t)-θ∥/∥θ∥ versus k of each algorithm are illustrated in Figure 2. We also investigate the performance of two algorithms under a relatively high noise level with noise variance σ2=1.002, and the corresponding simulation results are illustrated in Table 2 and Figure 3.Table 1
The MISO-GI and MISO-LSI estimates and errors (σ2=0.502).
Algorithm
k
a
1
b
1
a
2
b
2
d
1
δ (%)
MISO-GI
1
0.01462
1.27390
0.09679
0.89451
−0.07066
52.82221
2
−0.23411
1.31836
−0.13385
1.15839
0.62397
24.24129
5
−0.33873
1.45005
−0.48982
1.43125
0.77055
7.82671
8
−0.35503
1.47142
−0.47912
1.46594
0.73773
5.64270
10
−0.35781
1.47633
−0.47364
1.48458
0.72409
4.65183
20
−0.36024
1.48089
−0.45659
1.54144
0.69498
1.99007
MISO-LSI
1
0.03460
1.41912
−0.00027
2.25122
−0.02164
49.09578
2
−0.37155
1.46913
−0.44592
1.58467
0.41478
11.75374
5
−0.36086
1.48239
−0.44390
1.58413
0.68642
0.87072
8
−0.36082
1.48242
−0.44387
1.58420
0.68670
0.87077
10
−0.36082
1.48242
−0.44387
1.58420
0.68670
0.87077
20
−0.36082
1.48242
−0.44387
1.58420
0.68670
0.87077
True values
−0.38000
1.48000
−0.44000
1.58000
0.69000Table 2
The MISO-GI and MISO-LSI estimates and errors (σ2=1.002).
Algorithm
k
a
1
b
1
a
2
b
2
d
1
δ (%)
MISO-GI
1
0.02747
1.25718
0.09726
0.89934
−0.06054
52.75453
2
−0.13980
1.28713
−0.06373
1.08128
0.58629
30.02651
5
−0.31243
1.44812
−0.49633
1.42666
0.73252
7.87070
8
−0.33623
1.47067
−0.48609
1.46212
0.68840
5.72438
10
−0.33927
1.47574
−0.48043
1.48105
0.68211
4.89225
20
−0.34152
1.48130
−0.46229
1.54025
0.68242
2.56420
MISO-LSI
1
0.04885
1.40155
0.00051
2.26283
−0.00903
49.33764
2
−0.36198
1.46011
−0.45714
1.57034
0.44031
10.74006
5
−0.34220
1.48370
−0.44808
1.58781
0.68523
1.70082
8
−0.34214
1.48373
−0.44804
1.58792
0.68545
1.70303
10
−0.34214
1.48373
−0.44804
1.58792
0.68545
1.70303
20
−0.34214
1.48373
−0.44804
1.58792
0.68545
1.70303
True values
−0.38000
1.48000
−0.44000
1.58000
0.69000Figure 2
The parameter estimation errorsδ versus t (σ2=0.502).Figure 3
The parameter estimation errorsδ versus t (σ2=1.002).From the simulation results in Tables1 and 2 and Figures 2 and 3, we can draw the following conclusions.(i)
The parameter estimation errors are getting smaller as the iterative variablek increases.(ii)
Both algorithms can produce highly accurate parameter estimates under different noise variances.(iii)
The MISO-LSI algorithm converges faster than the MISO-GI algorithm does; however, due to the use of a batch of data, the MISO-LSI algorithm involves many matrix computations, resulting in the high computational complexity. One possible solution for reducing the computational load of the MISO-LSI algorithm with largep is using the decomposition technique [27], which is widely adopted in the least-squares based iterative algorithms.


## 6. Conclusions

In this work, we have presented two iterative identification algorithms, a least-squares based iterative algorithm and a gradient based iterative algorithm, for a class of linear-in-parameters multiple-input single-output output error moving average systems. The illustrative example shows that both algorithms can provide more accurate parameter estimates. The proposed methods can be extended to study the identification problems of linear multivariable systems [39, 40] or multirate or nonuniformly sampled systems [41, 42]. The methods in this paper can combine the multi-innovation identification methods [43–50], the iterative identification methods [51, 52], and other identification methods [53–56] to present new identification algorithms for nonlinear systems [57–59] and can also be applied in other fields [60–67].



---

*Source: 673197-2014-08-13.xml*