# Experimental and Analytical Studies on Improved Feedforward ML Estimation Based on LS-SVR

Authors: Xueqian Liu; Hongyi Yu
Journal: Discrete Dynamics in Nature and Society
 (2013)

---

## Abstract

Maximum likelihood (ML) algorithm is the most common and effective parameter estimation method. However, when dealing with small sample and low signal-to-noise ratio (SNR), threshold effects are resulted and estimation performance degrades greatly. It is proved that support vector machine (SVM) is suitable for small sample. Consequently, we employ the linear relationship between least squares support vector regression (LS-SVR)’s inputs and outputs and regard LS-SVR process as a time-varying linear filter to increase input SNR of received signals and decrease the threshold value of mean square error (MSE) curve. Furthermore, it is verified that by taking single-tone sinusoidal frequency estimation, for example, and integrating data analysis and experimental validation, if LS-SVR’s parameters are set appropriately, not only can the LS-SVR process ensure the single-tone sinusoid and additive white Gaussian noise (AWGN) channel characteristics of original signals well, but it can also improves the frequency estimation performance. During experimental simulations, LS-SVR process is applied to two common and representative single-tone sinusoidal ML frequency estimation algorithms, the DFT-based frequency-domain periodogram (FDP) and phase-based Kay ones. And the threshold values of their MSE curves are decreased by 0.3 dB and 1.2 dB, respectively, which obviously exhibit the advantage of the proposed algorithm.


---

## Body

## 1. Introduction

Maximum likelihood (ML) estimation depends on the asymptotic theory, which means that the statistical characteristics are shown accurately only when the sample size is infinity. However, burst-mode transmissions always bring problems about short data and severe conditions. Therefore, threshold effect is existing. Namely, the mean square error (MSE) of ML estimation can reach Cramer-Rao lower bound (CRLB) if it is higher than a value, or the performance will be deteriorated rapidly.Statistical learning theory (SLT) and structure risk minimization (SRM) rule in it are specialized in small-sample learning [1]. As their concrete implement, support vector machine (SVM) overcomes the over-fitting and local minimum problems currently existing in artificial neural network (ANN). Least squares support vector regression (LS-SVR) has the following improvements: inequality constraint are substituted by equality one; a squared loss function is taken for the error variable. Hence, we introduce LS-SVR to improve ML estimator and take feedforward single-tone sinusoidal frequency estimation for example, in this study.Estimating frequency of a single-tone sinusoid has attracted considerable attention for many decades. Rife and Boorstyn exploited the relationship of maximum likelihood estimator (MLE) to discrete Fourier transform (DFT) and proposed a frequency-domain periodogram (FDP) algorithm having two stages: coarse search and fine search [2]. In order to reduce the calculation cost, a great deal of improved algorithms have erupted mainly from two sides: interpolation-based and phase-based ones.During the former, an iterative binary search for the true signal frequency has been presented, which is particularly suited for digital signal processing (DSP) implementation [3]. In [4], the same authors have proposed a number of hybrid estimators that combine the dichotomous search with various interpolation techniques in order to reduce the computational complexity, at the expense of acquisition range. And, other modified dichotomous search frequency estimators have been addressed in [5–7]. Besides, complex Fourier coefficients have been utilized to interpolate the true signal frequency between the maximum and the second highest bin [8]. However, it has been shown to have a frequency dependent performance [9]. Two improved estimators have been proposed, which were implemented iteratively [10, 11]. Rational combination of three spectrum lines (RCTSL) has been employed as the fine estimation, because of its constant combinational weights in least square approximation [12]. Other methods used for interpolation include Lagrange interpolator [13], L-filter DFT [14], nonlinear filter [15], Kaise window [16], trigonometric polynomial interpolator [17], narrowband approximation interpolator [18], and so on. In the latter, Tretter [19] was the first person to propose a phase-based approach by introducing an approximated and linear model for instantaneous signal phase. Subsequently, a great deal of improvements have erupted mainly in the following three parts: taking differences over one or more delays, which is well-known as Kay and generalized Kay estimators [20–25]; introducing autocorrelations and their different functions, such as Fitz, L&R, and M&M estimators [26–30]; and preprocessing by means of lowpass filter, blocking average, and filter banks to increase signal-to-noise ratio (SNR) [31–34].In this paper, we present an improved feedforward ML estimation based on LS-SVR, taking single-tone sinusoidal frequency estimation, for example. LS-SVR process is regarded as a time-varying linear filter to increase input SNR of received signals, and accordingly, the threshold value of MSE curve is decreased. Reliability and validation of LS-SVR process are verified by integrating data analysis and experimental simulation. It is verified that not only can the LS-SVR ensure the single-tone sinusoid and AWGN channel characteristics of original signals well, but also increases the input SNR of received signals efficiently and improves frequency estimation performance. During experimental simulations, LS-SVR process is applied to two common and representative single-tone sinusoidal frequency estimation algorithms, the DFT-based FDP and phase-based Kay algorithms. The estimation performance of having the LS-SVR process and not are compared, respectively, to exhibit the advantage of the proposed algorithm, if its parameters are set appropriately.The remainder of this paper is organized as follows. Section2 briefly introduces the basic theory of LS-SVR. Section 3 describes the model of single-tone sinusoidal frequency estimation and the classical algorithms including FDP and Kay. In Section 4, the LS-SVR process is concretely explained and analyzed. And Section 5 shows the results of simulations and experiments. The paper is concluded in Section 6 finally.


## 2. Theory of LS-SVR

At first, a linear hyperplanef(x)=(w·ϕ(x))+b insofar as for ε is assumed to fit all elements of the training set S={(x1,y1),…,(xN,yN)}⊂Rn×R, where w is the high-dimensional coefficient of f(x), (·) is an inner product operator, and ϕ(·) is a nonlinear mapping from low to high dimension feature space. Also, ε-insensitive loss function is defined as
(1)L(y,f(x))=|y-f(x)|ε={0,|y-f(x)|≤ε|y-f(x)|-ε,elsed
i denotes the distance from point (xi,yi)∈S to f(x):
(2)di=|(w·ϕ(xi))+b-yi|1+∥w∥2≤ε1+∥w∥2,i=1,…,N.According to (2), we optimize f(x) through maximizing ε/(1+∥w∥2), that is, minimizing ∥w∥2. Thereby, SVR is presented as
(3)minJ(w,b)=12∥w∥2s.t.|(w·ϕ(xi))+b-yi|≤ε,i=1,…,N.Then, we proceed to conquer inseparable condition by introducing error variablesei and least squares (LS) method, and convert (3) into
(4)minJ(w,b)=12∥w∥2+C2∑i=1Nei2,s.t.yi=(w·ϕ(xi))+b+ei,i=1,…,N,
where penalty factor C is a positive constant to take compromise in LS-SVR’s generalization capability and fitting errors, which are denoted by the first and second item of J(w,b), respectively.Next step, we use Lagrange multiplier method and replace(ϕ(xi)·ϕ(xj)) with kernel function K(xi,xj):
(5)[01T1Q+1CI][bα]=[0Y],
where 1=(1,…,1)T︸N, Y=(y1,…,yN)T, α=(α1,…,αN)T, α1,…,αN are Lagrange multipliers, Q is a kernel function matrix, and radius basis function (RBF) is adopted in this study, so:
(6)Qij=K(xi,xj)=exp(-∥xi-xj∥2h2),
where Qij is the (i,j)th element of Q; the width of RBF h is a positive constant.Ultimately, the discriminant function is described as(7)f(x)=(w·ϕ(x))+b=∑i=1NαiK(xi,x)+b.


## 3. Signal Model and Classical Algorithms

### 3.1. Signal Model

The sinusoid signal polluted by noise is modeled as(8)rn=Aexp[j(2πfn+θ)]+wn,n=0,…,N-1.
Here, A>0, f∈[-0.5,0.5), θ∈[-π,π) are the amplitude, deterministic but unknown frequency, and initial phase, respectively; wn is an independent complex additive white Gaussian noise (AWGN) with zero-mean and variance σ2; and N is the sample size.


### 3.2. Classical Algorithms

FDP algorithm extends the number of DFT points toK by appending (K-N) zeros and searches the maximum bin [2]. Now, the frequency step is 1/K. Consider
(9)f^FDP=argmaxfk{|R(fk)|},k=0,…,K-1,
where R(fk)=∑n=0N-1rne-j2πfkn, fk=k/K. For the sake of convenience of fast Fourier transform (FFT) calculation, K is usually set to 2N,4N,8N,….Kay algorithm measures differential phase with a delay of one symbol interval and derives the estimation value of frequency [20]. Consider
(10)f^Kay=12π∑k=1N-1wkarg(rkrk-1*),
where the weights wk=6k(N-k)/N(N2-1), k=1,…,N-1.


## 3.1. Signal Model

The sinusoid signal polluted by noise is modeled as(8)rn=Aexp[j(2πfn+θ)]+wn,n=0,…,N-1.
Here, A>0, f∈[-0.5,0.5), θ∈[-π,π) are the amplitude, deterministic but unknown frequency, and initial phase, respectively; wn is an independent complex additive white Gaussian noise (AWGN) with zero-mean and variance σ2; and N is the sample size.


## 3.2. Classical Algorithms

FDP algorithm extends the number of DFT points toK by appending (K-N) zeros and searches the maximum bin [2]. Now, the frequency step is 1/K. Consider
(9)f^FDP=argmaxfk{|R(fk)|},k=0,…,K-1,
where R(fk)=∑n=0N-1rne-j2πfkn, fk=k/K. For the sake of convenience of fast Fourier transform (FFT) calculation, K is usually set to 2N,4N,8N,….Kay algorithm measures differential phase with a delay of one symbol interval and derives the estimation value of frequency [20]. Consider
(10)f^Kay=12π∑k=1N-1wkarg(rkrk-1*),
where the weights wk=6k(N-k)/N(N2-1), k=1,…,N-1.


## 4. LS-SVR Process and Its Analysis

### 4.1. LS-SVR Process

At first, we construct the training setS={(xn,yn)∣n=1,…,N}, where xn=n-1 is time series, yn=rn-1=Aexp{j[2πf(n-1)+θ]}+wn-1=gn+wn′ is received series, and gn and wn′ are signal and noise components of yn, perspectively.Secondly, we try to fitS by f(x)=(w·ϕ(x))+b, x∈[0,N-1]. However, LS-SVR process is only available for real signal, so yn is divided into real and image parts and preprocessed perspectively. That is,(1)
we employ LS-SVR to fit training setSreal={(xnreal,ynreal)∣n=1,…,N}, where xnreal=xn=n-1, ynreal=Re[yn]=Acos[2πf(n-1)+θ]+wn-1real, wn-1real is the real part of wn-1, so it is an independent real AWGN with zero-mean and variance σ2/2, and derive freal(x),(2)
we employ LS-SVR to fit training setSimag={(xnimag,ynimag)∣n=1,…,N}, where xnimag=xn=n-1, ynimag=Im[yn]=Asin[2πf(n-1)+θ]+wn-1imag, wn-1imag is the image part of wn-1, so it is an independent real AWGN with zero-mean and variance σ2/2, and derive fimag(x),(3)
f(x)=freal(x)+j×fimag(x), x∈[0,N-1].Then, we substitutexn into f(x) and get a new series of received signals.At last, we utilize the classical algorithm to estimate frequency accurately.


### 4.2. Analysis of LS-SVR Process

The real part ofyn is taken, for example, and now Y=(y1real,…,yNreal)T=(Acosθ+w0,…,Acos[2πf(N-1)+θ]+wN-1)T. The solutions of (5) are described as
(11)b=1T(Q+(1/C)I)-1Y1T(Q+(1/C)I)-11α=(Q+1CI)-1(Y-1b).Y
^
=
(
y
^
1
,
…
,
y
^
N
)
T
=
(
f
(
x
1
)
,
…
,
f
(
x
N
)
)
T is defined as outputs of LS-SVR process and (11) and (12) are substituted into (7)
(12)Y^=Qα+1b=Q(Q+1CI)-1Y+[I-Q(Q+1CI)-1]1b={11T(Q+1/CI)-11T(Q+1/CI)-11Q(Q+1CI)-1+[I-Q(Q+1CI)-1]×11T(Q+(1/C)I)-11T(Q+(1/C)I)-11}Y=βY.It is clear that now LS-SVR process can be regarded as a time-varying linear filter, and its coefficients are determined by(13)β=Q(Q+1CI)-1+[I-Q(Q+1CI)-1]×11T(Q+(1/C)I)-11T(Q+(1/C)I)-11.Received signals polluted by noise areY=G+W, where G=(Re[g1],…,Re[gN])T=(Acosθ,…,Acos[2πf(N-1)+θ])T, and W=(Re[w1′],…,Re[wN′])T=(w0real,…,wN-1real)T are signal and noise components of Y, respectively. The outputs of LS-SVR process are
(14)Y^=βY=β(G+W)=βG+βW=G^+W^,
where G^=(g^1,…,g^N)T, W^=(w^1,…,w^N)T are the signal and noise components of Y^=(y^1,…,y^N)T, respectively.Further, the mean, autocorrelation, and covariance function ofY^ are obtained as follows, where E{·} is a mean operator:
(15)E{Y^}=E{βG+βW}=βGE{Y^Y^T}=E{βYYTβT}=βE{YYT}βT=β(E{GGT}+E{WWT})βT=β(GGT+σ2I)βT,
where E{YYT}=E{GGT}+E{WWT}=GGT+σ2I. Consider
(16)E{(Y^-E{Y^})(Y^-E{Y^})T}=E{Y^Y^T}-E{Y^}E{Y^T}=β(E{YYT}-GGT)βT=σ2ββT.Therefore, we conclude with the following.(1)g^n=∑i=1Nβn,igi=A∑i=1Nβn,icos[2πf(i-1)+θ], n=1,…,N, the signal components of outputs of LS-SVR process, do not have single-tone sinusoid characteristics, where βn,i is (n,i)th element of β. But G^ will have the most powerful spetrum component in f by selecting the width of RBF h properly.Firstly, settingf=0.15, θ=0, N=32, SNR=0dB, and the parameter of LS-SVR C=5, the arbitrary amplitude spectrums of G^, while h=1 and h=5 are illustrated in Figure 1, respectively. It is shown that when h=1, the spectrum component of G^ in f is much more powerful than other places. It means that now the output of LS-SVR process still keeps the spectrum characteristics of Acos(2πfx+θ) and can be used to estimate the frequency of Acos(2πfx+θ). However, as h increases, the spectrum component of G^ in f inversely decreases and others gradually increase; hence, now the output of LS-SVR process cannot keep the spectrum characteristics of Acos(2πfx+θ).Amplitude spectrums ofG^.
(a)
h
=
1
(b)
h
=
5Furthermore, everything is as in Figure1, while h=1, h=4, and h=5, the time-domain waveforms of G^ are plotted in Figure 2, perspectively. The conclusion of Figure 2 is consistent with Figure 1, just when h=1, the amplitude of time-domain waveforms of G^ is less than Acos(2πfx+θ).Time-domain waveforms ofG^.
(a)
Original signals
(b)
h
=
1
(c)
h
=
4
(d)
h
=
5The Euclidean distance betweenG^ and G is defined as follows, where max(·) is the operation of taking maximum value:
(17)d=∑n=1N(E{g^n}max(|E{g^n}|)-gnmax(|gn|))2.Everything is as in Figure1, and the number of Monte Carlo experiments is 10000; the values of d with different h are listed in Table 1. Obviously, G^ can be very close to G through proper choice of h. Nevertheless, G^ will gradually deviate from Gas h increases.Table 1
Values ofd with different h.
h
0.5
1
2
3
4
5
d
6
.
1
×
10
-
4
0.0014
0.0132
0.2223
2.1279
8.123Consequently, proper choice ofh ensures that LS-SVR process can be used for frequency estimation of single-tone sinusoidal signals. Integrating the above analyses, the value of h must be less than 3.(2) Through analyzing the covariance function ofY^, it is shown that LS-SVR process is feasible and valid with a proper choice of C and h.From (16), it is obvious that the covariance function is related to σ2 and β. Taking N=4, for example, when C=5, h=1, (16) is calculated as
(18)σ2·[0.868780.084560.005550.041120.084560.827230.082670.005550.005550.082670.827230.084550.041120.005550.084550.86878].We can deduce the following by analyzing (18)(A)
The elements except main diagonal ones denote the correlations betweenw^n in different moments. Compared with the main diagonal elements, these elements are rather small. Hereby, w^n in different moments are approximately uncorrelated with each other. In addition, they are certainly subjected to Gaussian distribution, as the reason that LS-SVR process is linear. As result, w^n in different moments are almost independent.(B)
The main diagonal elements denote the powers ofw^n. w^n in different moments are independent and identically distributed (i.i.d) by reason of their nearly equal values. And also, it is the premise that the classical algorithms of feedforward ML frequency estimation can be still employed after LS-SVR process.(C)
Integrating (12), when C decreases rapidly, Q+(1/C)I≈(1/C)I, β≈(11T/N). Clearly, now y^n in different moments are equal and can not fit Y correctly, which means LS-SVR process is ineffective. On the contrary, when C increases and tends to infinity, now Q+(1/C)I≈Q, β≈I, which means that LS-SVR process can hardly influence Y. We apply LS-SVR process to FDP algorithm and derive the proposed algorithm called LS-SVR for short. Everything is as in Figure 1, and the number of Monte Carlo experiments is 10000; Figure 3 illustrates the impact of C on MSE performance, which is consistent with all analyses above, also C=5 is set in this study.Figure 3
Impact ofC on MSE performance.At the same time, according to(19)Qij=K(xi,xj)=exp[-(i-j)2h2],limh→0Qij={1,i=j0,i≠j,
thus, limh→0Q=I. Integrating (12), limh→0β=(C/(C+1))I+(1/(C+1))(1N1NT)/N≈(C/(C+1))I, while C=5, N=32, which means that LS-SVR process can influence Y in a fixed proportion and barely improves the frequency estimation performance. On the contrary, limh→+∞Qij=1, i≠j. As a result, the correlations between w^n in different moments rise and their powers are not equal. When C=5, h=5, (16) is calculated as
(20)σ2·[0.427480.311880.185280.075360.311880.274180.228660.185280.185280.228660.274180.311880.075360.185280.311880.42748].Everything is as in Figure3 other than that C=5; the impact of h on MSE performance is shown in Figure 4, which is consistent with all analyses above, and h=1 is set in this study.Figure 4
Impact ofh on MSE performance.(3) SettingC and h appropriately, LS-SVR process can increase SNR of Y and improve the performance of feedforward ML frequency estimations under the condition of small sample and low SNR.We assumeλin=(1/LP)GTG/(1/LP)WTW=((GTG)/(WTW)), λout=(((1/LP)G^TG^)/((1/LP)W^TW^))=((G^TG^)/(W^TW^)), and show the SNR variation between before and after LS-SVR process by
(21)λoutλin=G^TG^WTWW^TW^GTG=GTβTβGWTWWTβTβWGTG.10
lg
(
E
{
λ
out
/
λ
in
}
) is defined as the plus of LS-SVR process, where λout/λin is related to β which is determined by C and h. Everything is still as in Figure 1, the pluses of LS-SVR process with different C and h are listed in Table 2, which are consistent with Figures 3 and 4.Table 2
Pluses of LS-SVR process with differentC and h.
Parameter settings
The plus of LS-SVR process (dB)
C
=
5
h
=
1
0.7648
C
=
1000
h
=
1
0.0059
C
=
5
h
=
0.5
−0.0242


## 4.1. LS-SVR Process

At first, we construct the training setS={(xn,yn)∣n=1,…,N}, where xn=n-1 is time series, yn=rn-1=Aexp{j[2πf(n-1)+θ]}+wn-1=gn+wn′ is received series, and gn and wn′ are signal and noise components of yn, perspectively.Secondly, we try to fitS by f(x)=(w·ϕ(x))+b, x∈[0,N-1]. However, LS-SVR process is only available for real signal, so yn is divided into real and image parts and preprocessed perspectively. That is,(1)
we employ LS-SVR to fit training setSreal={(xnreal,ynreal)∣n=1,…,N}, where xnreal=xn=n-1, ynreal=Re[yn]=Acos[2πf(n-1)+θ]+wn-1real, wn-1real is the real part of wn-1, so it is an independent real AWGN with zero-mean and variance σ2/2, and derive freal(x),(2)
we employ LS-SVR to fit training setSimag={(xnimag,ynimag)∣n=1,…,N}, where xnimag=xn=n-1, ynimag=Im[yn]=Asin[2πf(n-1)+θ]+wn-1imag, wn-1imag is the image part of wn-1, so it is an independent real AWGN with zero-mean and variance σ2/2, and derive fimag(x),(3)
f(x)=freal(x)+j×fimag(x), x∈[0,N-1].Then, we substitutexn into f(x) and get a new series of received signals.At last, we utilize the classical algorithm to estimate frequency accurately.


## 4.2. Analysis of LS-SVR Process

The real part ofyn is taken, for example, and now Y=(y1real,…,yNreal)T=(Acosθ+w0,…,Acos[2πf(N-1)+θ]+wN-1)T. The solutions of (5) are described as
(11)b=1T(Q+(1/C)I)-1Y1T(Q+(1/C)I)-11α=(Q+1CI)-1(Y-1b).Y
^
=
(
y
^
1
,
…
,
y
^
N
)
T
=
(
f
(
x
1
)
,
…
,
f
(
x
N
)
)
T is defined as outputs of LS-SVR process and (11) and (12) are substituted into (7)
(12)Y^=Qα+1b=Q(Q+1CI)-1Y+[I-Q(Q+1CI)-1]1b={11T(Q+1/CI)-11T(Q+1/CI)-11Q(Q+1CI)-1+[I-Q(Q+1CI)-1]×11T(Q+(1/C)I)-11T(Q+(1/C)I)-11}Y=βY.It is clear that now LS-SVR process can be regarded as a time-varying linear filter, and its coefficients are determined by(13)β=Q(Q+1CI)-1+[I-Q(Q+1CI)-1]×11T(Q+(1/C)I)-11T(Q+(1/C)I)-11.Received signals polluted by noise areY=G+W, where G=(Re[g1],…,Re[gN])T=(Acosθ,…,Acos[2πf(N-1)+θ])T, and W=(Re[w1′],…,Re[wN′])T=(w0real,…,wN-1real)T are signal and noise components of Y, respectively. The outputs of LS-SVR process are
(14)Y^=βY=β(G+W)=βG+βW=G^+W^,
where G^=(g^1,…,g^N)T, W^=(w^1,…,w^N)T are the signal and noise components of Y^=(y^1,…,y^N)T, respectively.Further, the mean, autocorrelation, and covariance function ofY^ are obtained as follows, where E{·} is a mean operator:
(15)E{Y^}=E{βG+βW}=βGE{Y^Y^T}=E{βYYTβT}=βE{YYT}βT=β(E{GGT}+E{WWT})βT=β(GGT+σ2I)βT,
where E{YYT}=E{GGT}+E{WWT}=GGT+σ2I. Consider
(16)E{(Y^-E{Y^})(Y^-E{Y^})T}=E{Y^Y^T}-E{Y^}E{Y^T}=β(E{YYT}-GGT)βT=σ2ββT.Therefore, we conclude with the following.(1)g^n=∑i=1Nβn,igi=A∑i=1Nβn,icos[2πf(i-1)+θ], n=1,…,N, the signal components of outputs of LS-SVR process, do not have single-tone sinusoid characteristics, where βn,i is (n,i)th element of β. But G^ will have the most powerful spetrum component in f by selecting the width of RBF h properly.Firstly, settingf=0.15, θ=0, N=32, SNR=0dB, and the parameter of LS-SVR C=5, the arbitrary amplitude spectrums of G^, while h=1 and h=5 are illustrated in Figure 1, respectively. It is shown that when h=1, the spectrum component of G^ in f is much more powerful than other places. It means that now the output of LS-SVR process still keeps the spectrum characteristics of Acos(2πfx+θ) and can be used to estimate the frequency of Acos(2πfx+θ). However, as h increases, the spectrum component of G^ in f inversely decreases and others gradually increase; hence, now the output of LS-SVR process cannot keep the spectrum characteristics of Acos(2πfx+θ).Amplitude spectrums ofG^.
(a)
h
=
1
(b)
h
=
5Furthermore, everything is as in Figure1, while h=1, h=4, and h=5, the time-domain waveforms of G^ are plotted in Figure 2, perspectively. The conclusion of Figure 2 is consistent with Figure 1, just when h=1, the amplitude of time-domain waveforms of G^ is less than Acos(2πfx+θ).Time-domain waveforms ofG^.
(a)
Original signals
(b)
h
=
1
(c)
h
=
4
(d)
h
=
5The Euclidean distance betweenG^ and G is defined as follows, where max(·) is the operation of taking maximum value:
(17)d=∑n=1N(E{g^n}max(|E{g^n}|)-gnmax(|gn|))2.Everything is as in Figure1, and the number of Monte Carlo experiments is 10000; the values of d with different h are listed in Table 1. Obviously, G^ can be very close to G through proper choice of h. Nevertheless, G^ will gradually deviate from Gas h increases.Table 1
Values ofd with different h.
h
0.5
1
2
3
4
5
d
6
.
1
×
10
-
4
0.0014
0.0132
0.2223
2.1279
8.123Consequently, proper choice ofh ensures that LS-SVR process can be used for frequency estimation of single-tone sinusoidal signals. Integrating the above analyses, the value of h must be less than 3.(2) Through analyzing the covariance function ofY^, it is shown that LS-SVR process is feasible and valid with a proper choice of C and h.From (16), it is obvious that the covariance function is related to σ2 and β. Taking N=4, for example, when C=5, h=1, (16) is calculated as
(18)σ2·[0.868780.084560.005550.041120.084560.827230.082670.005550.005550.082670.827230.084550.041120.005550.084550.86878].We can deduce the following by analyzing (18)(A)
The elements except main diagonal ones denote the correlations betweenw^n in different moments. Compared with the main diagonal elements, these elements are rather small. Hereby, w^n in different moments are approximately uncorrelated with each other. In addition, they are certainly subjected to Gaussian distribution, as the reason that LS-SVR process is linear. As result, w^n in different moments are almost independent.(B)
The main diagonal elements denote the powers ofw^n. w^n in different moments are independent and identically distributed (i.i.d) by reason of their nearly equal values. And also, it is the premise that the classical algorithms of feedforward ML frequency estimation can be still employed after LS-SVR process.(C)
Integrating (12), when C decreases rapidly, Q+(1/C)I≈(1/C)I, β≈(11T/N). Clearly, now y^n in different moments are equal and can not fit Y correctly, which means LS-SVR process is ineffective. On the contrary, when C increases and tends to infinity, now Q+(1/C)I≈Q, β≈I, which means that LS-SVR process can hardly influence Y. We apply LS-SVR process to FDP algorithm and derive the proposed algorithm called LS-SVR for short. Everything is as in Figure 1, and the number of Monte Carlo experiments is 10000; Figure 3 illustrates the impact of C on MSE performance, which is consistent with all analyses above, also C=5 is set in this study.Figure 3
Impact ofC on MSE performance.At the same time, according to(19)Qij=K(xi,xj)=exp[-(i-j)2h2],limh→0Qij={1,i=j0,i≠j,
thus, limh→0Q=I. Integrating (12), limh→0β=(C/(C+1))I+(1/(C+1))(1N1NT)/N≈(C/(C+1))I, while C=5, N=32, which means that LS-SVR process can influence Y in a fixed proportion and barely improves the frequency estimation performance. On the contrary, limh→+∞Qij=1, i≠j. As a result, the correlations between w^n in different moments rise and their powers are not equal. When C=5, h=5, (16) is calculated as
(20)σ2·[0.427480.311880.185280.075360.311880.274180.228660.185280.185280.228660.274180.311880.075360.185280.311880.42748].Everything is as in Figure3 other than that C=5; the impact of h on MSE performance is shown in Figure 4, which is consistent with all analyses above, and h=1 is set in this study.Figure 4
Impact ofh on MSE performance.(3) SettingC and h appropriately, LS-SVR process can increase SNR of Y and improve the performance of feedforward ML frequency estimations under the condition of small sample and low SNR.We assumeλin=(1/LP)GTG/(1/LP)WTW=((GTG)/(WTW)), λout=(((1/LP)G^TG^)/((1/LP)W^TW^))=((G^TG^)/(W^TW^)), and show the SNR variation between before and after LS-SVR process by
(21)λoutλin=G^TG^WTWW^TW^GTG=GTβTβGWTWWTβTβWGTG.10
lg
(
E
{
λ
out
/
λ
in
}
) is defined as the plus of LS-SVR process, where λout/λin is related to β which is determined by C and h. Everything is still as in Figure 1, the pluses of LS-SVR process with different C and h are listed in Table 2, which are consistent with Figures 3 and 4.Table 2
Pluses of LS-SVR process with differentC and h.
Parameter settings
The plus of LS-SVR process (dB)
C
=
5
h
=
1
0.7648
C
=
1000
h
=
1
0.0059
C
=
5
h
=
0.5
−0.0242


## 5. Simulations and Experiments

We apply LS-SVR process to two common and representative single-tone sinusoidal ML frequency estimation algorithms, the DFT-based FDP and phase-based Kay ones, and derive the proposed algorithm called LS-SVR for short, where the number of DFT points of FDP algorithm isK=32N.


### 5.1. Mean Performance

Everything is as in Figure3 other than C=5; Figures 5 and 6 illustrate the mean of these three algorithms with different SNR. As is shown, whether high or low SNR, LS-SVR process can hardly change the unbiased ranges of FDP and Kay algorithms. Also, the unbiased ranges of all three algorithms will degrade with deterioration of SNR.Mean of FDP and LS-SVR algorithms.
(a)
SNR is −4 dB
(b)
SNR is 0 dBMean of Kay and LS-SVR algorithms.
(a)
SNR is 6 dB
(b)
SNR is 10 dB


### 5.2. MSE Performance

Everything is as in Figure3 except C=5; the MSE curves of these three algorithms are both shown in Figures 7 and 8, where CRLB is defined as (1/SNR)(3/(2π)2N(N-1)(2N-1)) [2]. We can see that LS-SVR process effectively improves the MSE performance of both FDP and Kay algorithm, and their threshold values are decreased by 0.3 dB and 1.2 dB, respectively.MSE curves of FDP and LS-SVR algorithms.
(a)
SNR
∈
[
-
6,3
]
dB
(b)
SNR
∈
[
-
2
,
-
1
]
dBMSE curves of Kay and LS-SVR algorithms.
(a)
SNR
∈
[
4,13
]
dB
(b)
SNR
∈
[
8,10
]
dB


### 5.3. Impact of Sample SizeN

Everything is still as in Figure3 except that C=5, Figure 9 illustrates the impact of N on MSE performance. We can know that the MSE curve of LS-SVR algorithm will decrease as N increases. However, when LS-SVR process is applied to FDP algorithm, its threshold value will increase as N increases; when LS-SVR process is applied to Kay algorithm, its threshold value will keep the same. The reason is related to the concrete frequency estimation algorithm after LS-SVR process.Impact ofN on MSE performance.
(a)
FDP and LS-SVR algorithms
(b)
Kay and LS-SVR algorithms


## 5.1. Mean Performance

Everything is as in Figure3 other than C=5; Figures 5 and 6 illustrate the mean of these three algorithms with different SNR. As is shown, whether high or low SNR, LS-SVR process can hardly change the unbiased ranges of FDP and Kay algorithms. Also, the unbiased ranges of all three algorithms will degrade with deterioration of SNR.Mean of FDP and LS-SVR algorithms.
(a)
SNR is −4 dB
(b)
SNR is 0 dBMean of Kay and LS-SVR algorithms.
(a)
SNR is 6 dB
(b)
SNR is 10 dB


## 5.2. MSE Performance

Everything is as in Figure3 except C=5; the MSE curves of these three algorithms are both shown in Figures 7 and 8, where CRLB is defined as (1/SNR)(3/(2π)2N(N-1)(2N-1)) [2]. We can see that LS-SVR process effectively improves the MSE performance of both FDP and Kay algorithm, and their threshold values are decreased by 0.3 dB and 1.2 dB, respectively.MSE curves of FDP and LS-SVR algorithms.
(a)
SNR
∈
[
-
6,3
]
dB
(b)
SNR
∈
[
-
2
,
-
1
]
dBMSE curves of Kay and LS-SVR algorithms.
(a)
SNR
∈
[
4,13
]
dB
(b)
SNR
∈
[
8,10
]
dB


## 5.3. Impact of Sample SizeN

Everything is still as in Figure3 except that C=5, Figure 9 illustrates the impact of N on MSE performance. We can know that the MSE curve of LS-SVR algorithm will decrease as N increases. However, when LS-SVR process is applied to FDP algorithm, its threshold value will increase as N increases; when LS-SVR process is applied to Kay algorithm, its threshold value will keep the same. The reason is related to the concrete frequency estimation algorithm after LS-SVR process.Impact ofN on MSE performance.
(a)
FDP and LS-SVR algorithms
(b)
Kay and LS-SVR algorithms


## 6. Conclusions

LS-SVR, as a modified and expanded version of SVR, still keeps its good capabilities in generalizing, high-dimensional processing, and nonlinear processing. Moreover, the equality constraint and squared loss function of LS-SVR make its computation and analysis rather simple.In this paper, we take single-tone sinusoidal frequency estimation, for example, and propose an improved feedforward ML algorithm for NDA estimation based on LS-SVR. Also, we demonstrate its feasibility and validity by statistical analyses and experimental simulations, and discuss the proper parameter settings of LS-SVR. Our results suggest that the threshold value of MSE curve of the proposed algorithm is lower than the original one. More importantly, the analyses and applications presented in this paper can be used in other fields of parameter estimation in the same way, which provide a novel thought to the corresponding research.The problems waiting for solution and the next direction are generalized as follows:(1)
increasing LS-SVR’s operating efficiency and improving the real-time characteristics of parameter estimation based on LS-SVR,(2)
making the best choice of LS-SVR’s parameters: the penalty factorC and the width of RBF h, not only the proper setting.



---

*Source: 192021-2013-11-27.xml*