# A Variable Step-Size Leaky LMS Algorithm

Authors: Sihai Zhao; Jiangye Xu; Yuyan Zhang
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

The leaky LMS algorithm has been extensively studied because of its control of parameter drift. This unexpected parameter drift is linked to the inadequacy of excitation in the input sequence. And generally leaky LMS algorithms use fixed step size to force the performance of compromise between the fast convergence rate and small steady-state misalignment. In this paper, variable step-size (VSS) leaky LMS algorithm is proposed. And the variable step-size method combines the time average estimation of the error and the time average estimation of the normalized quantity. Variable step-size method proposed incorporating with leaky LMS algorithm can effectively eliminate noise interference and make the early convergence, and final small misalignments are obtained together. Simulation results demonstrate that the proposed algorithm has better performance than the existing variable step-size algorithms in the unexcited environment. Furthermore, the proposed algorithm is comparable in performance to other variable step-size algorithms under the adequacy of excitation.


---

## Body

## 1. Introduction

The least-mean-square (LMS) algorithm is widely used in many fields, such as system identification [1, 2], echo cancellation [3], adaptive channel equalization [4], adaptive antenna array [5], and adaptive spectral line enhancement [6], due to its good robustness, low computational complexity, and simple structure. However, it is well known that the convergence rate and steady-state error of the algorithm are directly related to the adaptation step size of a conventional LMS algorithm. The convergence rate increases with the increase of the adaption step size, but the steady-state error changes in the opposite direction [7]. Therefore, the adaptive step size is a contradiction in the iterations, and variable step-size (VSS) LMS algorithm, which is adjusted according to the proximity between the actual value and the optimal value in the iterations, is therefore proposed solving this contradiction. In addition, the evaluation criteria for adjusting step size include squared instantaneous error [8, 9], sign changes of successive samples of the gradient [10–12], cross-correlation of input and error [13], and so on.Although VSS-LMS can balance the convergence rate and steady-state error of LMS algorithm to some extent, it may generate unbounded estimation of parameters due to the existence of inadequacy of excitation. In order to solve this problem, the leakage factor is introduced into the iterations [14–16]. The leaky LMS algorithm can improve the robustness of channel equalization transmission [17, 18] and noise reduction [19], all of which depend on its handling of filter drift. Furthermore, some studies have shown that step size has a certain impact on the performance of leaky LMS algorithm [20], and this paper tries to enhance the performance of leaky LMS algorithm by variable step size.In this paper, the convergence condition of the leaky LMS algorithm with variable step size is analyzed, and the quantization range of step size is given. In Section2, based on the analysis of leakage LMS and variable step-size algorithm, a variable step-size leakage LMS algorithm is proposed. Section 3 analyzes that the performance of the proposed algorithm is conducted, and Section 4 details that the simulation is provided to show the validity of the analytical results. The conclusions are presented in Section 5.


## 2. VVS-L-LMS Adaptive Algorithm

In the conventional L-LMS algorithm [14], the coefficients of the adaptive filter are updated as:
(1)wn+1=γwn+μxnen,where γ is leakage factor and 0≪γ<1, μ is adaptive step size, n is discrete time series, wn is weight coefficient vector of the adaptive filter at nth time, and xn is input signal vector with time series n. wn and xn can be described as:
(2)xn=xn,xn−1,⋯,xn−L+1T,wn=w0n,w1n−1,⋯,wL−1n−L+1T,where L is filter order. The instantaneous estimation error en is
(3)en=dn−xTnwn,where the desired signal dn is
(4)dn=xTnw∗n+tn,where tn is independent noise sequence with zero mean and w∗n is time-varying optimal weight coefficient vector.Substituting Equation (3) into Equation (1) and rearranging the terms yields
(5)wn+1=γI−μxnxTnwn+μxnen.The range of step size can be provided by solving the state equation Equation (5) [20]:
(6)∀nγ−1xn2<μ<γ+1xn2,where γ∈0,1, for xn2≥0; therefore, μ could be negative [20]. The L-LMS algorithm that is same as the conventional LMS algorithm uses a fixed step size in its coefficient update recursion thus inheriting the limitation in the full-update algorithm of having to compromise between fast convergence speed and low level of steady state error.This problem above can be tackled by changing fixed step-sizeμ in Equation (1) to time-varying step-size μn, which is adjusted based on a criterion that tries to measure the proximity of the adaptive filter parameters to the optimal ones. Therefore, Equation (1) can be written as:
(7)wn+1=γwn+μnxnen.According to Kwong’s VVS-LMS algorithm, recursive expression of step size is(8)μn+1=αμn+νe2n,where the positive constant parameters α0<α<1 and λλ>0 control the prediction error. But Kwong’s VVS-LMS algorithm cannot exclude the influence of noise on the instantaneous error. And in order to solve this problem, it is proposed to use qn instead of en; μn can be expressed as:
(9)μn+1=αμn+νq2n,where
(10)qn=pnrn,where pn is smoothing value for the cross-correlation function of error en is used to replace the power of error e2n. rn is input signal power, and they can be described as:
(11)pn=βpn−1+1−βenen−1,(12)rn=βrn−1+1−βe2nxTnxn,where β0<β<1 is the weight parameter that controls the convergence time. And the criteria for the step recursion are as follows:
(13)μn+1=μmaxifμn+1>μmax,μminifμn+1<μmin,μn+1otherwise,where μ0=μmax and p0=μ0. According to Equation (6), μmax and μmin should satisfy sufficient conditions which guarantee the stability of leaky LMS algorithm and μmax<γ+1/xn2 and μmin>γ−1/xn2.


## 3. Performance Analysis

In this section, the performance of proposed algorithm in stationary and nonstationary environment is described. And in stationary environment, the optimal weight coefficient which is in Equation (4) can be described as a constant:
(14)w∗n=w∗.Furthermore, in stationary environment,w∗n varies randomly as described:
(15)w∗n+1=aw∗n+zn,where a<1, but a is very close to 1. zn is a sequence of zero means with covariance σz2I, and I is the identity matrix. According to the above description, w∗n in the stationary environment can be viewed as w∗n in the nonstationary environment where a=1 and the σz2=0.


### 3.1. Convergence of the Mean Weight Vector

In order to discuss the convergence problem of mean weight vector, translated weight vectorVn=wn−w∗n needs to be introduced, which can reflect the weight error. In addition, the autocorrelation matrix of the input signal is defined as R=ExTnxn, which can also be described as R=QΛQTwhere QTQ=I. In the previous formula, Λ is the eigenvalue matrix of R where Λ=diagλN,λN−1,⋯,λN−L+2,λN−L+1, λN≤λN−1≤⋯≤λN−L+1, and Q is unitary matrix. Then, the rotation matrix, V´n=QTvn, X´n=QTxn, V´n=QTvn, W´∗n=QTw∗n, Z´n=QTZn, is defined to describe the convergence of the mean weight vector. Substitute Equation (3), Equation (4), and Equation (15) into Equation (7) and the following expression will be found by rewriting:
(16)V´n+1=γI−μnX´TnX´nV´n+μntnX´n+γ−aW´∗n−Z´n.Using assumptions which describes thatX´n and V´n are independent to find expectations on both sides of Equation (16) yields:
(17)EV´n+1=γI−EμnΛEV´n+γ−aEW´∗n.According to Equation (17), the following conditions can be obtained to ensure the convergence of the mean weight vector:
(18)0<Eμn<γ+1λmax,where λmax is the maximum eigenvalue of the input signal autocorrelation matrix R. According to the above, λmax can be rewritten as
(19)λmax≤trR.Therefore,trR is the trace of the input signal matrix R, and the condition Equation (18) for guaranteeing the convergence of weight vectors can be rewritten as
(20)0<Eμn<γ+1trR.But convergence of mean weight vector is not a sufficient condition to guarantee convergence of mean square error.


### 3.2. Mean Square Error Behavior

In this section, the necessary and sufficient conditions for the mean square error to be stable are discussed. MSE is given by [21]:
(21)ϵn=ϵmin+ϵexn=ϵmin+EV´TnΛV´n,where ϵmin is the minimum of the mean square error where ϵmin=Et2n and ϵexn is the excess mean square error (EMSE). In addition, it can be seen from Equation (21) that the convergence of MSE is directly related to diagonal matrix EV´TnΛV´n. Assuming that μn is independent of V´n and X´n, the expression of EX´nX´TnV´nV´TnX´nX´Tn can be simplified into a sum of second-order moments [8] by using Gaussian factoring theorem. So taking the expectation value of EV´n+1V´Tn+1 yeilds:
(22)EV´n+1V´Tn+1=γ2EV´nV´Tn−γΛEμnEV´nV´Tn−γEμnEV´nV´TnΛ+2Eμ2nΛEV´nV´TnΛ+Eμ2nΛtrΛEV´nV´Tn+γγ−aEV´nPT´Λ−1−γ−aΛEμnV´nPT´Λ−1+γγ−aΛ−1P´EV´Tn−γ−aΛ−1P´EV´TnμnΛ+γ−a2Λ−1P´nPT´nΛ−1+Eμ2nϵmiΛ+σz2I,where W´∗n=Λ−1P´ and P´=QTP where P is cross-correlation between the input vector and the expected vector; it can be written as P=Ednxn. A second moment Z1n including EV´nV´Tn is defined, and Z2n=EV´n. The vector can be described as:
(23)Zn+1=Z1n+1Z2n+1.According to Equation (17) and Equation (22), Zn+1 can be rewritten as:
(24)Zn+1=KZn+C,(25)K=ABOD,where O is a 0 matrix. The matrix K can be described as:
(26)A=γ2I−2γΛEμn+2Λ2Eμ2n+Eμ2nΓΓT,(27)B=2γγ−aΛ−1−2γ−aEμnY,(28)D=γI−EμnΛ,where Y=diagp´1,p´2,⋯,p´N; the ith part of P´ is p´i. Γ=λ1,λ2,⋯,λNT. Eμ2∞ is steady state of Eμ2n, and Eμ∞ is also steady state of Eμn and assume that:
(29)Eμ∞=μ¯,Eμ2∞=μ2.¯In Equation (24), C has no effect on our research, so it can be ignored. And making the following definition is to study further:
(30)EV´n+1V´Tn+1=aijEV´nV´Tn,where aij can be described as follows:
(31)aij=γ2−γμ¯λi+λj+2μ2¯λiλj.But,(32)aiiaij−aij2=γ−μ¯λi2+2μ2¯−μ¯2λi2γ−μ¯λj2+2μ2¯−μ¯2λj2−γ−μ¯λiγ−μ¯λj+2μ2¯−μ¯2λiλj2=2μ2¯−μ¯2λiγ−μ¯λj+λjγ−μ¯λi2≥0.Under the condition that can guarantee MSE converges,aij must satisfy the condition as:
(33)aij<1.In order to make sure thatEV´n+1V´Tn+1 converges, we are concerned that we need to make sure that Equation (24) is exponential stable. And aj is
(34)aj=γ2−2γμ¯λj+2μ2¯λj2.And if the above conditions are satisfied, the following must be true:(35)detK−ρI=detD−ρIdetA−ρI,where ρi is the roots of detK−ρI and the detD−ρI can be got from Equation (28):
(36)detD−ρI=∏j=1Nγ−μ¯λj,(37)detA−ρI=detdiaga1−ρ,⋯,an−ρ∙detI+diag1a1−ρ,⋯,1an−ρ∙μ2¯Λ2I=∏j=1Naj−ρ1+∑j=1nλj2aj−ρ.According to Equation (37), the conditions that can guarantee the root of Equation (37) in the unit circle can be described as:
(38)aj<1j=1,2,3,⋯,N,1+μ2¯∑i=1nλj2aj−1>0.Following the same argument in Equation (38). The above conditions can be equivalent to:
(39)1−λi22μ2¯−μ¯2γ−μ¯λi2>1j=1,2,3,⋯,N,(40)∑i=1nμ2¯λj2γ−μ¯λi2+2μ2¯−μ¯2λi2−1>−1.Z1n+1 convergence can be guaranteed if the above conditions Equation (39) and Equation (40) are satisfied. And Equation (21) can be rewritten as:
(41)ϵn=ϵmin+ϵexn=ϵmin+ΓTZ1n,where Yˇ=p´1,p´2,⋯,p´NT and Equation (11) can be rewritten as:
(42)pn=1−β∑i=0n−1βien−ien−i−1.Taking the expectation of both sides of Equation (42) fields:
(43)Epn=2πEp2n,where
(44)Ep2n=1−β2∑i=0n−1∑j=0n−1β2iβ2j×Ee2n−ie2n−i+1e2n−je2n−j+1,Ep2∞=1−βϵmin+ϵexn21+β.Taking the expectation of both sides of Equation (12) fields:
(45)Ern=βErn−1+1−βEe2nxTnxn,where
(46)Ee2nxTnxn=ϵmin+ϵexn∑NN−L+1Exmi2n,Er2n=β2Er2n−1+2β1−βErn−1Ee2nxTnxn+1−β2Ee4nxTnxnxTnxn,where
(47)Ee4nxTnxnxTnxn=Ee4n∑N−L+1N∑N−L+1NExmi2nxni2n,Ee4n=3ϵmin+6ϵmintrΛEV´nV´Tn+3trΛEV´nV´Tn2+6trEΛV´nV´Tn2−6trEI−μn−1Λ4EV´n−1EV´Tn−1Λ2,Er2∞=2β+1Ee4n∑N−L+1N∑N−L+1NExmi2nxni2n,where
(48)Ee4n=3ϵmin+ΓTZ1n+6ΓTZ1nZ1TΓ,where ΓTZ1nZ1TΓ≪ϵmin+ΓTZ1n. According to argument in [8], the sufficient conditions to guarantee MSE convergence can be described as:
(49)Eμ2∞Eμ∞=2ανEq2∞1−α2<6αν1−β1+2βϵmin31−α21+βExx4,where Exx4=∑NN−L+1∑NN−L+1Exmi2nxni2n. If the input signal is Gaussian white noise signal, then Exx4=3Nσx4 where σx4 is the input signal variance.


### 3.3. Steady State Misadjustment

In this section, misadjustment of steady state is studied. And the misadjustment is defined as:(50)M=ϵex∞ϵmin,and the ϵexn can be represented as:
(51)ϵexn=ϵn−ϵmin=ΓTZ1n.Accordingly, the solution ofZ1n converges to Z1∞, and they can be described as:
(52)Z1n=AZ1n+BZ2n+γ−a2Λ−12Yˇ+μ¯ϵminΓ+σz2I.And in the stationary environmentσz2=0, the Z1∞ can be expressed as:
(53)Z1∞=I−A−1BZ2∞+γ−a2Λ−12Yˇ+μ¯ϵminΓ,where Yˇ=p´12,p´22,⋯,p´N2 and I−A−1 can be computed as:
(54)I−A−1=1−γ2I+2γμ¯Λ−2μ2¯Λ2−μ2¯ΓΓT−1=Aˇ+μ2¯AˇΓΓTAˇ1−μ2¯ΓTAˇΓ=Aˇ1−μ2¯∑ni=1λi2/2λi1−λiγμ¯−μ2¯+1−γ2,where
(55)Aˇ=diag12λ11−λ1γμ¯−μ2¯+1−γ2,12λ21−λ2γμ¯−μ2¯+1−γ2⋯,12λn1−λnγμ¯−μ2¯+1−γ2.And using Equation (17) and Equation (27), the BZ2n can be solved:
(56)BZ2∞=2p´j2γ−12γ−μ¯λiλi21−γμ¯λi.Substituting Equation (53)–Equation (56) into Equation (51) yields:
(57)ϵex∞=∑i=1n2p´j2γ−12γ−μ¯λi/2λi21−γμ¯λi1−λiγμ¯−μ2¯+λi1−γμ¯λi1−γ2+∑i=1nγ−12p´i2+μ¯ϵmin/2λi21−λiγμ¯−μ2¯+λi1−γ21−μ2¯∑i=1nλi2/2λi1−λiγμ¯−μ2¯+1−γ2.For a leakage factor approaching 1, the above equation can be approximated as:(58)ϵex∞≈∑i=1nμ¯ϵmin/2λi21−λiγμ¯−μ2¯1−μ2¯∑i=1nλi2/2λi1−λiγμ¯−μ2¯.From Equation (49), the following formula can be got:
(59)g=μ2¯μ¯≈6αν1−β1+2βϵmin31−α21+βExx4.Substituting Equation (59) into Equation (58) yields
(60)ϵex∞≈∑i=1n1/2λi21−λiγ−g1−∑i=1ngλi2/2λi1−λiγ−gϵmin.


## 3.1. Convergence of the Mean Weight Vector

In order to discuss the convergence problem of mean weight vector, translated weight vectorVn=wn−w∗n needs to be introduced, which can reflect the weight error. In addition, the autocorrelation matrix of the input signal is defined as R=ExTnxn, which can also be described as R=QΛQTwhere QTQ=I. In the previous formula, Λ is the eigenvalue matrix of R where Λ=diagλN,λN−1,⋯,λN−L+2,λN−L+1, λN≤λN−1≤⋯≤λN−L+1, and Q is unitary matrix. Then, the rotation matrix, V´n=QTvn, X´n=QTxn, V´n=QTvn, W´∗n=QTw∗n, Z´n=QTZn, is defined to describe the convergence of the mean weight vector. Substitute Equation (3), Equation (4), and Equation (15) into Equation (7) and the following expression will be found by rewriting:
(16)V´n+1=γI−μnX´TnX´nV´n+μntnX´n+γ−aW´∗n−Z´n.Using assumptions which describes thatX´n and V´n are independent to find expectations on both sides of Equation (16) yields:
(17)EV´n+1=γI−EμnΛEV´n+γ−aEW´∗n.According to Equation (17), the following conditions can be obtained to ensure the convergence of the mean weight vector:
(18)0<Eμn<γ+1λmax,where λmax is the maximum eigenvalue of the input signal autocorrelation matrix R. According to the above, λmax can be rewritten as
(19)λmax≤trR.Therefore,trR is the trace of the input signal matrix R, and the condition Equation (18) for guaranteeing the convergence of weight vectors can be rewritten as
(20)0<Eμn<γ+1trR.But convergence of mean weight vector is not a sufficient condition to guarantee convergence of mean square error.


## 3.2. Mean Square Error Behavior

In this section, the necessary and sufficient conditions for the mean square error to be stable are discussed. MSE is given by [21]:
(21)ϵn=ϵmin+ϵexn=ϵmin+EV´TnΛV´n,where ϵmin is the minimum of the mean square error where ϵmin=Et2n and ϵexn is the excess mean square error (EMSE). In addition, it can be seen from Equation (21) that the convergence of MSE is directly related to diagonal matrix EV´TnΛV´n. Assuming that μn is independent of V´n and X´n, the expression of EX´nX´TnV´nV´TnX´nX´Tn can be simplified into a sum of second-order moments [8] by using Gaussian factoring theorem. So taking the expectation value of EV´n+1V´Tn+1 yeilds:
(22)EV´n+1V´Tn+1=γ2EV´nV´Tn−γΛEμnEV´nV´Tn−γEμnEV´nV´TnΛ+2Eμ2nΛEV´nV´TnΛ+Eμ2nΛtrΛEV´nV´Tn+γγ−aEV´nPT´Λ−1−γ−aΛEμnV´nPT´Λ−1+γγ−aΛ−1P´EV´Tn−γ−aΛ−1P´EV´TnμnΛ+γ−a2Λ−1P´nPT´nΛ−1+Eμ2nϵmiΛ+σz2I,where W´∗n=Λ−1P´ and P´=QTP where P is cross-correlation between the input vector and the expected vector; it can be written as P=Ednxn. A second moment Z1n including EV´nV´Tn is defined, and Z2n=EV´n. The vector can be described as:
(23)Zn+1=Z1n+1Z2n+1.According to Equation (17) and Equation (22), Zn+1 can be rewritten as:
(24)Zn+1=KZn+C,(25)K=ABOD,where O is a 0 matrix. The matrix K can be described as:
(26)A=γ2I−2γΛEμn+2Λ2Eμ2n+Eμ2nΓΓT,(27)B=2γγ−aΛ−1−2γ−aEμnY,(28)D=γI−EμnΛ,where Y=diagp´1,p´2,⋯,p´N; the ith part of P´ is p´i. Γ=λ1,λ2,⋯,λNT. Eμ2∞ is steady state of Eμ2n, and Eμ∞ is also steady state of Eμn and assume that:
(29)Eμ∞=μ¯,Eμ2∞=μ2.¯In Equation (24), C has no effect on our research, so it can be ignored. And making the following definition is to study further:
(30)EV´n+1V´Tn+1=aijEV´nV´Tn,where aij can be described as follows:
(31)aij=γ2−γμ¯λi+λj+2μ2¯λiλj.But,(32)aiiaij−aij2=γ−μ¯λi2+2μ2¯−μ¯2λi2γ−μ¯λj2+2μ2¯−μ¯2λj2−γ−μ¯λiγ−μ¯λj+2μ2¯−μ¯2λiλj2=2μ2¯−μ¯2λiγ−μ¯λj+λjγ−μ¯λi2≥0.Under the condition that can guarantee MSE converges,aij must satisfy the condition as:
(33)aij<1.In order to make sure thatEV´n+1V´Tn+1 converges, we are concerned that we need to make sure that Equation (24) is exponential stable. And aj is
(34)aj=γ2−2γμ¯λj+2μ2¯λj2.And if the above conditions are satisfied, the following must be true:(35)detK−ρI=detD−ρIdetA−ρI,where ρi is the roots of detK−ρI and the detD−ρI can be got from Equation (28):
(36)detD−ρI=∏j=1Nγ−μ¯λj,(37)detA−ρI=detdiaga1−ρ,⋯,an−ρ∙detI+diag1a1−ρ,⋯,1an−ρ∙μ2¯Λ2I=∏j=1Naj−ρ1+∑j=1nλj2aj−ρ.According to Equation (37), the conditions that can guarantee the root of Equation (37) in the unit circle can be described as:
(38)aj<1j=1,2,3,⋯,N,1+μ2¯∑i=1nλj2aj−1>0.Following the same argument in Equation (38). The above conditions can be equivalent to:
(39)1−λi22μ2¯−μ¯2γ−μ¯λi2>1j=1,2,3,⋯,N,(40)∑i=1nμ2¯λj2γ−μ¯λi2+2μ2¯−μ¯2λi2−1>−1.Z1n+1 convergence can be guaranteed if the above conditions Equation (39) and Equation (40) are satisfied. And Equation (21) can be rewritten as:
(41)ϵn=ϵmin+ϵexn=ϵmin+ΓTZ1n,where Yˇ=p´1,p´2,⋯,p´NT and Equation (11) can be rewritten as:
(42)pn=1−β∑i=0n−1βien−ien−i−1.Taking the expectation of both sides of Equation (42) fields:
(43)Epn=2πEp2n,where
(44)Ep2n=1−β2∑i=0n−1∑j=0n−1β2iβ2j×Ee2n−ie2n−i+1e2n−je2n−j+1,Ep2∞=1−βϵmin+ϵexn21+β.Taking the expectation of both sides of Equation (12) fields:
(45)Ern=βErn−1+1−βEe2nxTnxn,where
(46)Ee2nxTnxn=ϵmin+ϵexn∑NN−L+1Exmi2n,Er2n=β2Er2n−1+2β1−βErn−1Ee2nxTnxn+1−β2Ee4nxTnxnxTnxn,where
(47)Ee4nxTnxnxTnxn=Ee4n∑N−L+1N∑N−L+1NExmi2nxni2n,Ee4n=3ϵmin+6ϵmintrΛEV´nV´Tn+3trΛEV´nV´Tn2+6trEΛV´nV´Tn2−6trEI−μn−1Λ4EV´n−1EV´Tn−1Λ2,Er2∞=2β+1Ee4n∑N−L+1N∑N−L+1NExmi2nxni2n,where
(48)Ee4n=3ϵmin+ΓTZ1n+6ΓTZ1nZ1TΓ,where ΓTZ1nZ1TΓ≪ϵmin+ΓTZ1n. According to argument in [8], the sufficient conditions to guarantee MSE convergence can be described as:
(49)Eμ2∞Eμ∞=2ανEq2∞1−α2<6αν1−β1+2βϵmin31−α21+βExx4,where Exx4=∑NN−L+1∑NN−L+1Exmi2nxni2n. If the input signal is Gaussian white noise signal, then Exx4=3Nσx4 where σx4 is the input signal variance.


## 3.3. Steady State Misadjustment

In this section, misadjustment of steady state is studied. And the misadjustment is defined as:(50)M=ϵex∞ϵmin,and the ϵexn can be represented as:
(51)ϵexn=ϵn−ϵmin=ΓTZ1n.Accordingly, the solution ofZ1n converges to Z1∞, and they can be described as:
(52)Z1n=AZ1n+BZ2n+γ−a2Λ−12Yˇ+μ¯ϵminΓ+σz2I.And in the stationary environmentσz2=0, the Z1∞ can be expressed as:
(53)Z1∞=I−A−1BZ2∞+γ−a2Λ−12Yˇ+μ¯ϵminΓ,where Yˇ=p´12,p´22,⋯,p´N2 and I−A−1 can be computed as:
(54)I−A−1=1−γ2I+2γμ¯Λ−2μ2¯Λ2−μ2¯ΓΓT−1=Aˇ+μ2¯AˇΓΓTAˇ1−μ2¯ΓTAˇΓ=Aˇ1−μ2¯∑ni=1λi2/2λi1−λiγμ¯−μ2¯+1−γ2,where
(55)Aˇ=diag12λ11−λ1γμ¯−μ2¯+1−γ2,12λ21−λ2γμ¯−μ2¯+1−γ2⋯,12λn1−λnγμ¯−μ2¯+1−γ2.And using Equation (17) and Equation (27), the BZ2n can be solved:
(56)BZ2∞=2p´j2γ−12γ−μ¯λiλi21−γμ¯λi.Substituting Equation (53)–Equation (56) into Equation (51) yields:
(57)ϵex∞=∑i=1n2p´j2γ−12γ−μ¯λi/2λi21−γμ¯λi1−λiγμ¯−μ2¯+λi1−γμ¯λi1−γ2+∑i=1nγ−12p´i2+μ¯ϵmin/2λi21−λiγμ¯−μ2¯+λi1−γ21−μ2¯∑i=1nλi2/2λi1−λiγμ¯−μ2¯+1−γ2.For a leakage factor approaching 1, the above equation can be approximated as:(58)ϵex∞≈∑i=1nμ¯ϵmin/2λi21−λiγμ¯−μ2¯1−μ2¯∑i=1nλi2/2λi1−λiγμ¯−μ2¯.From Equation (49), the following formula can be got:
(59)g=μ2¯μ¯≈6αν1−β1+2βϵmin31−α21+βExx4.Substituting Equation (59) into Equation (58) yields
(60)ϵex∞≈∑i=1n1/2λi21−λiγ−g1−∑i=1ngλi2/2λi1−λiγ−gϵmin.


## 4. Simulation Results

This part analyzes the proposed algorithm in nonstationary environment and focuses on system identification. The performance of the proposed algorithm is compared with MVSS [9], FSS [8], conventional leaky LMS [14] algorithm, and conventional VSS. In order to compare the performance of each algorithm fairly, the algorithms mentioned above are all placed in the same experimental environment, and the parameters of each algorithm are selected to meet the needs of the experiment. Monte Carlo method was used to conduct 200 independent experiments for the following experiments to ensure the reliability of experimental results.


### 4.1. Example 1: White Input, Low SNR

In the Example 1, the unknown moving average system which is excited by a zero-mean uncorrelated white Gaussian signal of unity varianceϵmin has four time-invariant coefficients. And the FIR filter has the same order as the unknown system L=4. The desired signal dn is corrupted by zero-mean, uncorrelated Gaussian noise of variance which is also the sequence of input signals. The VSS, proposed algorithm, and MVSS choose α=0.97 which is stable in both stationary and nonstationary environments. The ν for VSS is set to νVSS=4.8×10−5 according to the literature, and the ν for proposed algorithm, and MVSS be chosen as 1×10−4 and the β=0.98. The fixed step-size (FSS) algorithm and conventional leaky LMS algorithm all use the step size μFSS=3.5×10−3. Variable step-size iteration algorithm boundary lines are μmax=0.1 and μmin=1×10−4. The leakage factor γ=0.999. Figure 1 shows that both the proposed algorithm and MVSS have good convergence and have basically the same misalignment as FSS algorithm at low SNR. In addition, the leakage algorithm misalignment is basically the same as that of FSS with white noisy input [22] and that is the reason for the big gap between the leaky algorithm and the other algorithms. Figure 2 illustrates that the step-size convergence process of the proposed algorithm is similar to MVSS algorithm and better than VVS algorithm at low SNR.Figure 1
Compare the MSE of different algorithms for white input caseSNR=10 dB.Figure 2
Compare the convergence process of variable step-size algorithms for white input caseSNR=10 dB.


### 4.2. Example 2: Correlated Input, Low SNR

In this example, the unknown system to be identified and the filter are set up in the same way as in the Example 1. But the filter input signal is changed from white noise input to correlation input which can be described as:(61)xn=0.9xn−1+cn,where cn is uncorrelated Gaussian white noise with unit variance. And Figure 3 depicts this input signal.Figure 3
Compare the white input and correlated input withn=3000.The step size of FSS algorithm isμFSS=3×10−4. To ensure that the convergence value of conventional leaky LMS is close to that of the FSS algorithm, the step size can be set to μLeaky=3×10−3, and the leakage factor is γ=0.999. The parameters of VSS algorithm with μmax=1×10−3 and μmin=1×10−5 are set to αVSS=0.97 and νVSS=8×10−5, respectively. The proposed algorithm and MVSS algorithm have the same α and ν as VSS algorithm, and β=0.98. The step-size range of the variable step-size algorithm above is μmax=1×10−2 and μmin=1×10−5. Figure 4 shows that the convergence speed of the proposed algorithm is close to that of MVSS under the condition of correlated input, and they are both slightly faster than that of FSS algorithm. Under the action of leakage factor, the same MSE convergence value can be obtained only when the step length of leakage algorithm is larger than that of FSS algorithm. As can be seen from Figure 5, the convergence of MVSS step size is slightly faster than that of the proposed algorithm. Under the effect of leakage factor, the peak step size of the proposed algorithm has a big gap with that of MVSS algorithm.Figure 4
Compare the MSE of different algorithms for correlated input caseSNR=10 dB.Figure 5
Compare the convergence process of variable step-size algorithms for correlated input caseSNR=10 dB.


### 4.3. Example 3: High SNR

Example 3 has the same parameters of filter and unknown system as Examples 1 and 2, except that it has a higher SNR which isSNR=40 dB.The FSS algorithm adopts the same step size which isμFSS=μLeaky=5×10−2 as the leaky LMS algorithm, and the leaage factor of the leaky LMS algorithm is γ=0.99. All variable step-size algorithms, including MVSS, FSS, and proposed algorithm, use the same step-size range which is μmax=0.1 and μmin=1×10−5. And the ν of MVSS, FSS, and proposed algorithm is νFSS=1×10−2, νMVSS=1, and νpro=1×10−5, respectively. The β of MVSS and proposed algorithm are all set to β=0.98. Figure 6 shows that in the case of large SNR, the gap between the convergence the MSE values of FSS and of leaky LMS algorithm becomes larger. And the proposed algorithm converges slightly faster than other variable step-size algorithms. Figure 7 mainly illustrates that the proposed algorithm has advantages in step-size convergence under the condition of high SNR.Figure 6
Compare the MSE of different algorithms for white input caseSNR=40 dB.Figure 7
Compare the convergence process of variable step-size algorithms for white input caseSNR=40 dB.


### 4.4. Example 4: Unexcited Input High Order

The parameters of the system to be identified and the filter in Example 4 are the same as those in the above three examples except the order which higher order numbersL=14 are used in this example. The input signal used in Example 4 is a weak white signal with zero mean and σx2=1×10−4, and the noisy signal which destroys the desired signal dn is zero-mean, uncorrelated Gaussian noise of variance.The FSS algorithm and the leaky LMS algorithm use the same step sizeμFSS=μLeaky=0.01. And the leakage factor of leaky LMS algorithm and proposed algorithm is γ=0.999. The step size of all variable step-size algorithms including FSS, MVSS, and proposed algorithm are set to μmax=0.05 and μmin=1×10−5, and the α of the all variable step-size algorithms is α=0.97. In addition, the ν of all variable step-size algorithms is selected as νFSS=1×10−5, νMVSS=1×10−3, and νPro=1×10−2, respectively. The β of MVSS and proposed algorithm is chosen as β=0.98, as can be seen from Figure 8, under the condition of unexcited input, the MSE which leaky algorithms with leakage factor converges to closer to that of FSS. And the proposed algorithm converges faster and can obtain better MSE compared with other algorithms. The existence of leakage factor can better deal with the parameter drift in the case of underexcitation, and the use of variable step size can make the algorithm convergence faster. Figure 9 shows that the step-size convergence of the proposed algorithm is faster than that of other variable step-size algorithms under the condition of underexcitation.Figure 8
Compare the MSE of different algorithms for unexcited input caseSNR=60 dB.Figure 9
Compare the convergence process of variable step-size algorithms for unexcited input caseSNR=60 dB.


## 4.1. Example 1: White Input, Low SNR

In the Example 1, the unknown moving average system which is excited by a zero-mean uncorrelated white Gaussian signal of unity varianceϵmin has four time-invariant coefficients. And the FIR filter has the same order as the unknown system L=4. The desired signal dn is corrupted by zero-mean, uncorrelated Gaussian noise of variance which is also the sequence of input signals. The VSS, proposed algorithm, and MVSS choose α=0.97 which is stable in both stationary and nonstationary environments. The ν for VSS is set to νVSS=4.8×10−5 according to the literature, and the ν for proposed algorithm, and MVSS be chosen as 1×10−4 and the β=0.98. The fixed step-size (FSS) algorithm and conventional leaky LMS algorithm all use the step size μFSS=3.5×10−3. Variable step-size iteration algorithm boundary lines are μmax=0.1 and μmin=1×10−4. The leakage factor γ=0.999. Figure 1 shows that both the proposed algorithm and MVSS have good convergence and have basically the same misalignment as FSS algorithm at low SNR. In addition, the leakage algorithm misalignment is basically the same as that of FSS with white noisy input [22] and that is the reason for the big gap between the leaky algorithm and the other algorithms. Figure 2 illustrates that the step-size convergence process of the proposed algorithm is similar to MVSS algorithm and better than VVS algorithm at low SNR.Figure 1
Compare the MSE of different algorithms for white input caseSNR=10 dB.Figure 2
Compare the convergence process of variable step-size algorithms for white input caseSNR=10 dB.


## 4.2. Example 2: Correlated Input, Low SNR

In this example, the unknown system to be identified and the filter are set up in the same way as in the Example 1. But the filter input signal is changed from white noise input to correlation input which can be described as:(61)xn=0.9xn−1+cn,where cn is uncorrelated Gaussian white noise with unit variance. And Figure 3 depicts this input signal.Figure 3
Compare the white input and correlated input withn=3000.The step size of FSS algorithm isμFSS=3×10−4. To ensure that the convergence value of conventional leaky LMS is close to that of the FSS algorithm, the step size can be set to μLeaky=3×10−3, and the leakage factor is γ=0.999. The parameters of VSS algorithm with μmax=1×10−3 and μmin=1×10−5 are set to αVSS=0.97 and νVSS=8×10−5, respectively. The proposed algorithm and MVSS algorithm have the same α and ν as VSS algorithm, and β=0.98. The step-size range of the variable step-size algorithm above is μmax=1×10−2 and μmin=1×10−5. Figure 4 shows that the convergence speed of the proposed algorithm is close to that of MVSS under the condition of correlated input, and they are both slightly faster than that of FSS algorithm. Under the action of leakage factor, the same MSE convergence value can be obtained only when the step length of leakage algorithm is larger than that of FSS algorithm. As can be seen from Figure 5, the convergence of MVSS step size is slightly faster than that of the proposed algorithm. Under the effect of leakage factor, the peak step size of the proposed algorithm has a big gap with that of MVSS algorithm.Figure 4
Compare the MSE of different algorithms for correlated input caseSNR=10 dB.Figure 5
Compare the convergence process of variable step-size algorithms for correlated input caseSNR=10 dB.


## 4.3. Example 3: High SNR

Example 3 has the same parameters of filter and unknown system as Examples 1 and 2, except that it has a higher SNR which isSNR=40 dB.The FSS algorithm adopts the same step size which isμFSS=μLeaky=5×10−2 as the leaky LMS algorithm, and the leaage factor of the leaky LMS algorithm is γ=0.99. All variable step-size algorithms, including MVSS, FSS, and proposed algorithm, use the same step-size range which is μmax=0.1 and μmin=1×10−5. And the ν of MVSS, FSS, and proposed algorithm is νFSS=1×10−2, νMVSS=1, and νpro=1×10−5, respectively. The β of MVSS and proposed algorithm are all set to β=0.98. Figure 6 shows that in the case of large SNR, the gap between the convergence the MSE values of FSS and of leaky LMS algorithm becomes larger. And the proposed algorithm converges slightly faster than other variable step-size algorithms. Figure 7 mainly illustrates that the proposed algorithm has advantages in step-size convergence under the condition of high SNR.Figure 6
Compare the MSE of different algorithms for white input caseSNR=40 dB.Figure 7
Compare the convergence process of variable step-size algorithms for white input caseSNR=40 dB.


## 4.4. Example 4: Unexcited Input High Order

The parameters of the system to be identified and the filter in Example 4 are the same as those in the above three examples except the order which higher order numbersL=14 are used in this example. The input signal used in Example 4 is a weak white signal with zero mean and σx2=1×10−4, and the noisy signal which destroys the desired signal dn is zero-mean, uncorrelated Gaussian noise of variance.The FSS algorithm and the leaky LMS algorithm use the same step sizeμFSS=μLeaky=0.01. And the leakage factor of leaky LMS algorithm and proposed algorithm is γ=0.999. The step size of all variable step-size algorithms including FSS, MVSS, and proposed algorithm are set to μmax=0.05 and μmin=1×10−5, and the α of the all variable step-size algorithms is α=0.97. In addition, the ν of all variable step-size algorithms is selected as νFSS=1×10−5, νMVSS=1×10−3, and νPro=1×10−2, respectively. The β of MVSS and proposed algorithm is chosen as β=0.98, as can be seen from Figure 8, under the condition of unexcited input, the MSE which leaky algorithms with leakage factor converges to closer to that of FSS. And the proposed algorithm converges faster and can obtain better MSE compared with other algorithms. The existence of leakage factor can better deal with the parameter drift in the case of underexcitation, and the use of variable step size can make the algorithm convergence faster. Figure 9 shows that the step-size convergence of the proposed algorithm is faster than that of other variable step-size algorithms under the condition of underexcitation.Figure 8
Compare the MSE of different algorithms for unexcited input caseSNR=60 dB.Figure 9
Compare the convergence process of variable step-size algorithms for unexcited input caseSNR=60 dB.


## 5. Conclusion

Based on the studies in this paper, the following conclusions can be drawn: (1) The proposed algorithm combines the time average estimation of error autocorrelation with the time average estimation of normalized quantity. By adjusting the parameters, the proposed algorithm can eliminate uncorrelated noise well. (2) For the same MSE, the convergence speed of the proposed algorithm is close to that of other variable-step algorithms under the condition of low SNR and high SNR. (3) The leakage factor can reduce the parameter drift caused by unexcitation. And in the case of underexcitation, the convergence rate of the proposed algorithm is faster than the other variable step-size algorithms, and the results are closer to wiener solution. In conclusion, the proposed algorithm has better performance in the case of unexcitation.



---

*Source: 7951643-2021-11-09.xml*