# QML Estimators in Linear Regression Models with Functional Coefficient Autoregressive Processes

Authors: Hongchang Hu
Journal: Mathematical Problems in Engineering
 (2010)

---

## Abstract

This paper studies a linear regression model, whose errors are functional coefficient autoregressive
processes. Firstly, the quasi-maximum likelihood (QML) estimators of some unknown parameters are given. Secondly, under general conditions, the asymptotic properties (existence, consistency, and asymptotic distributions) of the QML estimators are investigated.
These results extend those of Maller (2003), White (1959), Brockwell and Davis (1987), and so on. Lastly, the validity and feasibility of the method are illuminated by a simulation example and a real example.


---

## Body

## 1. Introduction

Consider the following linear regression model:(1.1)yt=xtTβ+ɛt,t=1,2,…,n,
where yt's are scalar response variables, xt's are explanatory variables, β is a d-dimensional unknown parameter, and the ɛt's are functional coefficient autoregressive processes given as(1.2)ɛ1=η1,ɛt=ft(θ)ɛt-1+ηt,t=2,3,…,n,
where ηt's are independent and identically distributed random errors with zero mean and finite variance σ2, θ is a one-dimensional unknown parameter and ft(θ) is a real valued function defined on a compact set Θ which contains the true value θ0 as an inner point and is a subset of R1. The values of θ0 and σ2 are unknown.Model (1.1) includes many special cases, such as an ordinary linear regression model when ft(θ)≡0; see [1–11]. In the sequel, we always assume that ft(θ)≠0, for some θ∈Θ, is a linear regression model with constant coefficient autoregressive processes (when ft(θ)=θ, see Maller [12], Pere [13], and Fuller [14]), time-dependent and functional coefficient autoregressive processes (when β=0, see Kwoun and Yajima [15]), constant coefficient autoregressive processes (when ft(θ)=θandβ=0, see White [16, 17], Hamilton [18], Brockwell and Davis [19], and Abadir and Lucas [20]), time-dependent or time-varying autoregressive processes (when ft(θ)=atandβ=0, see Carsoule and Franses [21], Azrak and Mélard [22], and Dahlhaus [23]), and so forth.Regression analysis is one of the most mature and widely applied branches of statistics. Linear regression analysis is one of the most widely used statistical techniques. Its applications occur in almost every field, including engineering, economics, the physical sciences, management, life and biological sciences, and the social sciences. Linear regression model is the most important and popular model in the statistical literature, which attracts many statisticians to estimate the coefficients of the regression model. For the ordinary linear regression model (when the errors are independent and identically distributed random variables), Bai and Guo [1], Chen [2], Anderson and Taylor [3], Drygas [4], González-Rodríguez et al. [5], Hampel et al. [6], He [7], Cui [8], Durbin [9], Hoerl and Kennard [10], Li and Yang [11], and Zhang et al. [24] used various estimation methods (Least squares estimate method, robust estimation, biased estimation, and Bayes estimation) to obtain estimators of the unknown parameters in (1.1) and discussed some large or small sample properties of these estimators.However, the independence assumption for the errors is not always appropriate in applications, especially for sequentially collected economic and physical data, which often exhibit evident dependence on the errors. Recently, linear regression with serially correlated errors has attracted increasing attention from statisticians. One case of considerable interest is that the errors are autoregressive processes and the asymptotic theory of this estimator was developed by Hannan and Kavalieris [25]. Fox and Taqqu [26] established its asymptotic normality in the case of long-memory stationary Gaussian observations errors. Giraitis and Surgailis [27] extended this result to non-Gaussian linear sequences. The asymptotic distribution of the maximum likelihood estimator was studied by Giraitis and Koul in [28] and Koul in [29] when the errors are nonlinear instantaneous functions of a Gaussian long-memory sequence. Koul and Surgailis [30] established the asymptotic normality of the Whittle estimator in linear regression models with non-Gaussian long-memory moving average errors. When the errors are Gaussian, or a function of Gaussian random variables that are strictly stationary and long range dependent, Koul and Mukherjee [31] investigated the linear model. Shiohama and Taniguchi [32] estimated the regression parameters in a linear regression model with autoregressive process.In addition to (constant or functional or random coefficient) autoregressive model, it has gained much attention and has been applied to many fields, such as economics, physics, geography, geology, biology, and agriculture. Fan and Yao [33], Berk [34], Hannan and Kavalieris [35], Goldenshluger and Zeevi [36], Liebscher [37], An et al. [38], Elsebach [39], Carsoule and Franses [21], Baran et al. [40], Distaso [41], and Harvill and Ray [42] used various estimation methods (the least squares method, the Yule-Walker method, the method of stochastic approximation, and robust estimation method) to obtain some estimators and discussed their asymptotic properties, or investigated hypotheses testing.This paper discusses the model (1.1)-(1.2) including stationary and explosive processes. The organization of the paper is as follows. In Section 2 some estimators of β,θ, and σ2 are given by the quasi-maximum likelihood method. Under general conditions, the existence and consistency the quasi-maximum likelihood estimators are investigated, and asymptotic normality as well, in Section 3. Some preliminary lemmas are presented in Section 4. The main proofs are presented in Section 5, with some examples in Section 6.


## 2. Estimation Method

Write the “true” model as(2.1)yt=xtTβ0+et,t=1,2,…,n,(2.2)e1=η1,et=ft(θ0)et-1+ηt,t=2,3,…,n,
where ft′(θ0)=(dft(θ)/dθ)∣θ=θ0≠0, and ηt's are i.i.d errors with zero mean and finite variance σ02. Define ∏i=0-1ft-i(θ0)=1, and by (2.2) we have(2.3)et=∑j=0t-1(∏i=0j-1ft-i(θ0))ηt-j.
Thus et is measurable with respect to the σ-field H generated by η1,η2,…,ηt, and(2.4)Eet=0,Var(et)=σ02∑j=0t-1(∏i=0j-1ft-i2(θ0)).Assume at first that theηt’s are i.i.d. N(0,σ2). Using similar arguments to those of Fuller [14] or Maller [12], we get the log-likelihood of y2,y3,…,yn conditional on y1:(2.5)Ψn(β,θ,σ2)=logLn=-12(n-1)logσ2-12σ2∑t=2n(ɛt-ft(θ)ɛt-1)2-12(n-1)log(2π).
At this stage we drop the normality assumption, but still maximize (2.5) to obtain QML estimators, denoted by σ̂n2,β̂n,θ̂n (when they exist):(2.6)∂Ψn∂σ2=-n-12σ2+12σ4∑t=2n(ɛt-ft(θ)ɛt-1)2,(2.7)∂Ψn∂θ=1σ2∑t=2nft′(θ)(ɛt-ft(θ)ɛt-1)ɛt-1,(2.8)∂Ψn∂β=1σ2∑t=2n(ɛt-ft(θ)ɛt-1)(xt-ft(θ)xt-1).
Thus σ̂n2,β̂n,θ̂n satisfy the following estimation equations:(2.9)σ̂n2=1n-1∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)2,(2.10)∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)ft′(θ̂n)ɛ̂t-1=0,(2.11)∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(xt-ft(θ̂n)xt-1)=0,
where(2.12)ɛ̂t=yt-xtTβ̂n.Remark 2.1.
Ifft(θ)=θ, then the above equations become the same as Maller's [12]. Therefore, we extend the QML estimators of Maller [12].
To calculate the values of the QML estimators, we may use the grid search method, steepest ascent method, Newton-Raphson method, and modified Newton-Raphson method. In order to calculate in Section6, we introduce the most popular modified Newton-Raphson method proposed by Davidon-Fletcher-Powell (see Hamilton [18]).
Let(d+2)×1 vector θ⃗(m)=(σ(m)2,β(m),θ(m)) denote an estimator of θ⃗=(σ2,β,θ) that has been calculated at the mth iteration, and let A(m) denote an estimation of [H(θ⃗(m))]-1. The new estimator θ⃗(m+1) is given by
(2.13)θ⃗(m+1)=θ⃗(m)+sA(m)g(θ⃗(m))
for s the positive scalar that maximizes Ψn{θ⃗(m)+sA(m)g(θ⃗(m))}, where (d+2)×1 vector
(2.14)g(θ⃗(m))=∂Ψn(θ⃗)∂θ⃗∣θ⃗=θ⃗(m)=(∂Ψn∂σ2∣σ2=σ(m)2∂Ψn∂β∣β=β(m)∂Ψn∂θ∣θ=θ(m))
and (d+2)×(d+2) symmetric matrix
(2.15)H(θ⃗(m))=-∂2Ψn(θ⃗)∂θ⃗∂θ⃗T∣θ⃗=θ⃗(m)=(∂2Ψn∂(σ2)2∂2Ψn∂σ2∂β∂2Ψn∂σ2∂θ*∂2Ψn∂β∂βT∂2Ψn∂β∂θ**∂2Ψn∂θ2)∣θ⃗=θ⃗(m),
where
(2.16)∂2Ψn∂(σ2)2=n-12σ4-1σ6∑t=2n(ɛt-ft(θ)ɛt-1)2,∂2Ψn∂σ2∂β=-1σ4∑t=2n(ɛt-ft(θ)ɛt-1)(xt-ft(θ)xt-1)T,∂2Ψn∂σ2∂θ=-1σ4∑t=2n(ɛt-ft(θ)ɛt-1)ft′(θ),(2.17)∂2Ψn∂β∂βT=-1σ2∑t=2n(xt-ft(θ)xt-1)(xt-ft(θ)xt-1)T,(2.18)∂2Ψn∂β∂θ=-1σ2∑t=2n(ft′(θ)ɛt-1xt+ft′(θ)ɛtxt-1-2ft(θ)ft′(θ)xt-1ɛt-1),∂2Ψn∂θ2=-1σ2∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)ɛtɛt-1).
Onceθ⃗(m+1) and the gradient at θ⃗(m+1) have been calculated, a new estimation A(m+1) is found from
(2.19)A(m+1)=A(m)-A(m)(Δg(m+1))(Δg(m+1))TA(m)(Δg(m+1))TA(m)(Δg(m+1))-(Δθ⃗(m+1))(Δθ⃗(m+1))T(Δg(m+1))T(Δθ⃗(m+1)),
where
(2.20)Δθ⃗(m+1)=θ⃗(m+1)-θ⃗(m),Δg(m+1)=g(θ⃗(m+1))-g(θ⃗(m)).
It is well known that least squares estimators in ordinary linear regression model are very good estimators, so a recursive algorithms procedure is to start the iteration withβ(0),σ(0)2 which are least squares estimators of β and σ2, respectively. Take θ(0) such that ft(θ(0))=0. Iterations are stopped if some termination criterion is reached, for example, if
(2.21)∥θ⃗(m+1)-θ⃗(m)∥∥θ⃗(m)∥<δ,
for some prechosen small number δ>0.
Up to this point, we obtain the values of QML estimators when the functionft(θ)=f(t,θ) is known. However, the function ft(θ) is never the case in practice; we have to estimate it. By (2.12) and (1.2), we obtain
(2.22)f̃(t,θ̂n)=ɛ̂tɛ̂t-1,t=2,3,…,n.
Based on the dataset {f̃(t,θ̂n),t=2,3,…,n}, we may obtain the estimation function f̂(t,θ̂n) of f(t,θ) by some smoothing methods (see Simonff [43], Fan and Yao [33], Green and Silverman [44], Fan and Gijbels [45], etc.)
To obtain our results, the following conditions are sufficient.
(A1)Xn=∑t=2nxtxtT is positive definite for sufficiently large n and
(2.23)limn→∞max1≤t≤nxtTXn-1xt=0,(2.24)lim supn→∞|λ|max(Xn-1/2ZnXn-T/2)<1,
where Zn=(1/2)∑t=2n(xtxt-1T+xt-1xtT) and |λ|max(·) denotes the maximum in absolute value of the eigenvalues of a symmetric matrix.
(A2) There is a constantα>0 such that
(2.25)∑j=1t(∏i=0j-1ft-i2(θ))≤α
for any t∈{1,2,…,n} and θ∈Θ.
(A3) The derivativesft′(θ)=dft(θ)/dθ,ft′′(θ)=dft′(θ)/dθ exist and are bounded for any t and θ∈Θ.Remark 2.2.
Maller [12] applied the condition (A1), and Kwoun and Yajima [15] used the conditions (A2) and (A3). Thus our conditions are general. (A1) delineates the class of xt for which our results hold in the sense required. It is further discussed by Maller in [12]. Kwoun and Yajima [15] call {et} stable if Var(et) is bounded. Thus (A2) implies that {et} is stable. However, {et} is not stationary. In fact, by (2.3), we obtain that
(2.26)Cov(et,et+k)=σ02{∏i=0k-1ft+k-i(θ0)+ft(θ0)∏i=0kft+k-i(θ0)+⋯+∏l=0t-2ft-l(θ0)∏i=0t+k-2ft+k-i(θ0)},
which is dependent of t.
For ease of exposition, we will introduce the following notations which will be used later in the paper.
Define(d+1)-vector φ=(β,θ), and
(2.27)Sn(φ)=σ2∂Ψn∂φ=σ2(∂Ψn∂β,∂Ψn∂θ),Fn(φ)=-σ2∂2Ψn∂φ∂φT.
By (2.7) and (2.8), we get
(2.28)Fn(φ)=(Xn(θ)∑t=2n(ft′(θ)ɛt-1xt+ft′(θ)ɛtxt-1-2ft(θ)ft′(θ)xt-1ɛt-1)*∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)ɛtɛt-1)),
where Xn(θ)=-σ2(∂2Ψn/∂β∂βT) and the * indicates that the element is filled in by symmetry. Thus,
(2.29)Dn=E(Fn(φ0))=(Xn(θ0)0*∑t=2n((ft′2(θ0)+ft(θ0)ft′′(θ0))Eet-12-ft′′(θ0)E(etet-1)))=(Xn(θ0)0*∑t=2nft′2(θ0)Eet-12)=(Xn(θ0)0*Δn(θ0,σ0)),
where
(2.30)Δn(θ0,σ0)=∑t=2nft′2(θ0)Eet-12=σ02∑t=2nft′2(θ0)∑j=0t-2(∏i=0j-1ft-i2(θ))=O(n).


## 3. Statement of Main Results

Theorem 3.1.
Suppose that conditions (A1)–(A3) hold. Then there is a sequenceAn↓0 such that, for each A>0, as n→∞, the probability
(3.1)P{thereareestimatorsφ̂n,σ̂n2withSn(φ̂n)=0,and(φ̂n,σ̂n2)∈Nn′(A)}→1.
Furthermore,
(3.2)(φ̂n,σ̂n2)→p(φ0,σ02),n→∞,
where, for each n=1,2,…,A>0 and An∈(0,σ02); define neighborhoods
(3.3)Nn(A)={φ∈Rd+1:(φ-φ0)TDn(φ-φ0)≤A2},Nn′(A)=Nn(A)∩{σ2∈[σ02-An,σ02+An]}.Theorem 3.2.
Suppose that conditions (A1)–(A3) hold. Then(3.4)1σ̂nFnT/2(φ̂n)(φ̂n-φ0)→DN(0,Id+1),n→∞.Remark 3.3.
Forθ∈Rm,m∈N, our results still hold.
In the following, we will investigate some special cases in the model (1.1)-(1.2). Although the following results are directly obtained from Theorems 3.1 and 3.2, we discuss these results in order to compare with the corresponding results.Corollary 3.4.
Letft(θ)=θ. If condition (A1) holds, then, for |θ|≠1, (3.1), (3.2), and (3.4) hold.Remark 3.5.
These results are the same as the corresponding results of Maller [12].Corollary 3.6.
Ifβ=0 and ft(θ)=θ, then, for ∣θ∣≠1,
(3.5)∑t=2nɛt-12σ̂n(θ̂n-θ0)→DN(0,1),n→∞,
where
(3.6)σ̂n2=1n-1∑t=2n(ɛt-θ̂nɛt-1)2,θ̂n=∑t=2n(ɛtɛt-1)∑t=2nɛt-12.Remark 3.7.
These estimators are the same as the least squares estimators (see White [16]). For |θ|>1, {ɛt} are explosive processes. In the case, the corollary is the same as the results of White [17]. While |θ|<1, notice that σ̂n2→pσ02 and (1/(n-1))∑t=2nɛt-12→pEɛt2=σ02/(1-θ02), and by Corollary 3.6 we obtain
(3.7)n(θ̂n-θ0)→DN(0,1-θ02).
The result was discussed by many authors, such as Fujikoshi and Ochi [46] and Brockwell and Davis [19].Corollary 3.8.
Letβ=0. If conditions (A2) and (A3) hold, then
(3.8)Fn1/2(θ̂n)σ̂n(θ̂n-θ0)→DN(0,1),n→∞,
where
(3.9)Fn(θ̂n)=∑t=2n((ft′2(θ̂n)+ft(θ̂n)ft′′(θ̂n))ɛt-12-ft′′(θ̂n)ɛtɛt-1),σ̂n2=1n-1∑t=2n(ɛt-ft(θ̂n)ɛt-1)2.Corollary 3.9.
Letft(θ)=at. If condition (A1) holds, then
(3.10)1σ̂n{∑t=2n(xt-atxt-1)(xt-atxt-1)T}T/2(β̂n-β0)→DN(0,Id),n→∞.Remark 3.10.
Letat=0. Note that ∑t=2nxtxtT=O(n) and σ̂n2→pσ02; we easily obtain asymptotic normality of the (quasi-)maximum likelihood or least squares estimator in ordinary linear regression models from the corollary.


## 4. Some Lemmas

To prove Theorems3.1 and 3.2, we first introduce the following lemmas.Lemma 4.1.
The matrixDn is positive definite for large enough n with E(Sn(φ0))=0 and Var(Sn(φ0))=σ02Dn.Proof.
It is easy to show that the matrixDn is positive definite for large enough n. By (2.8), we have
(4.1)σ02E(∂Ψn∂β∣β=β0)=∑t=2nE(et-ft(θ0)et-1)(xt-ft(θ0)xt-1)=∑t=2n(xt-ft(θ0)xt-1)Eηt=0.
Note that et-1 and ηt are independent of each other; thus by (2.7) and Eηt=0, we have
(4.2)σ02E(∂Ψn∂θ∣θ=θ0)=∑t=2nE((et-ft(θ0)et-1)ft′(θ0)et-1)=∑t=2nft′(θ0)E(ηtet-1)=0.
Hence, from (4.1) and (4.2),
(4.3)E(Sn(φ0))=σ02E(∂Ψn∂β∣β=β0,∂Ψn∂θ∣θ=θ0)=0.
By (2.8) and (2.17), we have
(4.4)Var(σ02∂Ψn∂β∣β=β0)=Var{∑t=2n(et-ft(θ0)et-1)(xt-ft(θ0)xt-1)}=Var{∑t=2nηt(xt-ft(θ0)xt-1)}=σ02Xn(θ0).
Note that {ft′(θ0)ηtet-1,Ht} is a martingale difference sequence with
(4.5)Var(ft′(θ0)ηtet-1)=ft′2(θ0)Eηt2Eet-12=σ02ft′2(θ0)Eet-12,
so
(4.6)Var(σ02∂Ψn∂θ∣θ=θ0)=Var{∑t=2nηtft′(θ0)et-1}=∑t=2nft′2(θ0)Eet-12=σ02Δn(θ0,σ0).
By (2.7) and (2.8) and noting that et-1 and ηt are independent of each other, we have
(4.7)Cov(σ02∂Ψn∂β∣β=β0,σ02∂Ψn∂θ∣θ=θ0)=E(σ02∂Ψn∂β∣β=β0,σ02∂Ψn∂θ∣θ=θ0)=E(∑t=2nηt2(xt-ft(θ0)xt-1)ft′(θ0)et-1)+E(∑t=3nηt(xt-ft(θ0)xt-1)∑s=2t-1ηsfs′(θ0)es-1)+E(∑s=3nηsfs′(θ0)es-1∑t=2s-1ηt(xt-ft(θ0)xt-1))=0.
From (4.4)–(4.7), it follows that Var(Sn(φ0))=σ02Dn.Lemma 4.2.
If condition (A1) holds, then, for anyθ∈Θ, the matrix Xn(θ) is positive definite for large enough n, and
(4.8)limn→∞max1≤t≤nxtTXn-1(θ)xt=0.Proof.
Letλ1 and λd be the smallest and largest roots of |Zn-λXn|=0. Then from the study of Rao in [47, Ex 22.1],
(4.9)λ1≤uTZnuuTXnu≤λd
for unit vectors u. Thus by (2.24), there are some δ∈(max{0,1-(1+min2≤t≤n|ft2(θ)|)/max2≤t≤n|ft(θ)|},1) and n0(δ) such that n≥N0 implies that
(4.10)|uTZnu|≤(1-δ)uTXnu.
By (4.10), we have
(4.11)uTXnu=∑t=2n(uT(xt-ft(θ)xt-1))2=∑t=2n((uTxt)2+ft2(θ)(uTxt-1)2-ft(θ)uTxt-1xtTu-ft(θ)uTxtxt-1Tu)≥∑t=2n(uTxt)2+min2≤t≤n|ft2(θ)|∑t=2n(uTxt-1)2-max2≤t≤n|ft(θ)|uTZnu≥uTXnu+min2≤t≤n|ft2(θ)|uTXnu-max2≤t≤n|ft(θ)|uTZnu≥(1+min2≤t≤n|ft2(θ)|-max2≤t≤n|ft(θ)|(1-δ))uTXnu=C(θ,δ)uTXnu.
By the study of Rao in [47, page 60] and (2.23), we have
(4.12)(uTxt)2uTXnu→0.
From (4.12) and C(θ,δ)>0,
(4.13)xtTXn-1(θ)=supu((uTxt)2uTXn(θ)u)≤supu((uTxt)2C(θ,δ)uTXnu)→0.Lemma 4.3 (see [48]).
LetWn be a symmetric random matrix with eigenvalues λj(n),1≤j≤d. Then
(4.14)Wn→pI⇔λj(n)→p1,n→∞.Lemma 4.4.
For eachA>0,
(4.15)supφ∈Nn(A)∥Dn-1/2Fn(φ)Dn-T/2-Φn∥→p0,n→∞,
and also
(4.16)Φn→DΦ,(4.17)limc→0lim supA→∞lim supn→∞P{infφ∈Nn(A)λmin(Dn-1/2Fn(φ)Dn-T/2)≤c}=0,
where
(4.18)Φn=(Id00∑t=2nft′2(θ0)et-12Δn(θ0,σ0)),Φ=Id+1.Proof.
LetXn(θ0)=Xn1/2(θ0)XnT/2(θ0) be a square root decomposition of Xn(θ0). Then
(4.19)Dn=(Xn1/2(θ0)0*Δn(θ0,σ0))(XnT/2(θ0)0*Δn(θ0,σ0))=Dn1/2DnT/2.
Let φ∈Nn(A). Then
(4.20)(φ-φ0)TDn(φ-φ0)=(β-β0)TXn(θ0)(β-β0)+(θ-θ0)2Δn(θ0,σ0)≤A2.
From (2.28), (2.29), and (4.18), (4.21)Dn-1/2Fn(φ)Dn-T/2-Φn=(Xn-1/2(θ0)Xn(θ)Xn-T/2(θ0)-IdXn-1/2(θ0)∑t=2n(ft′(θ)ɛt-1xt+ft′(θ)ɛtxt-1-2ft(θ)ft′(θ)ɛt-1xt-1)Δn(θ0,σ0)*∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)ɛtɛt-1)-∑t=2nft′2(θ0)et-12Δn(θ0,σ0)).
Let
(4.22)Nnβ(A)={β:|(β-β0)TXn1/2(θ0)|2≤A2},(4.23)Nnθ(A)={θ:|θ-θ0|≤AΔn(θ0,σ0)}.
In the first step, we will show that, for eachA>0,
(4.24)supθ∈Nnθ(A)∥Xn-1/2(θ0)Xn(θ)Xn-T/2(θ0)-Id∥→0,n→∞.
In fact, note that
(4.25)Xn-1/2(θ0)Xn(θ)Xn-T/2(θ0)-Id=Xn-1/2(θ0)(Xn(θ)-Xn(θ0))Xn-T/2(θ0)=Xn-1/2(θ0)(T1+T2-T3)Xn-T/2(θ0),
where
(4.26)T1=∑t=2n(ft(θ0)-ft(θ))xt-1(xt-ft(θ0)xt-1)T,T2=∑t=2n(xt-ft(θ0)xt-1)xt-1T,T3=∑t=2n(ft(θ0)-ft(θ))2xt-1xt-1T.
Letu,v∈Rd,|u|=|v|=1, and let unT=uTXn-1/2(θ0),vnT=Xn-T/2(θ0)v. By Cauchy-Schwartz inequality, Lemma 4.2, condition (A3), and noting that θ∈Nnθ(A), we have that
(4.27)|unTT1vn|=|∑t=2n(ft(θ0)-ft(θ))unTxt-1(xt-ft(θ0)xt-1)Tvn|≤max2≤t≤n|ft(θ0)-ft(θ)||∑t=2nunTxt-1(xt-ft(θ0)xt-1)Tvn|≤max2≤t≤n|ft(θ0)-ft(θ)|(∑t=2nunTxt-1xt-1Tun)1/2·(∑t=2nvnT(xt-ft(θ0)xt-1)(xt-ft(θ0)xt-1)Tvn)1/2≤max2≤t≤n|ft(θ0)-ft(θ)|(∑t=2nunTxtxtTun)1/2≤max2≤t≤n|ft′(θ̃)||θ0-θ|·nmax1≤t≤n(xtTXn-1(θ0)xt)≤CnΔn(θ0,σ0)o(1)→0.
Here θ̃=aθ+(1-a)θ0 for some 0≤a≤1. Similar to the proof of T1, we easily obtain that
(4.28)|unTT2vn|→0.
By Cauchy-Schwartz inequality, Lemma 4.2, condition (A3), and noting that Nnθ(A), we have that
(4.29)|unTT3vn|=|unT∑t=2n(ft(θ0)-ft(θ))2xt-1xt-1Tvn|≤max2≤t≤n|ft(θ0)-ft(θ)|2(∑t=2nunTxtxtTun∑t=2nvnTxtxtTvn)1/2≤nmax2≤t≤n|ft′(θ̃)|2|θ0-θ|2max1≤t≤n(xtTXn-1(θ0)xt)≤nA2Δn(θ0,σ0)o(1)→0.
Hence, (4.24) follows from (4.25)–(4.29).
In the second step, we will show that(4.30)Xn-1/2(θ0)∑t=2n(ft′(θ)ɛt-1xt+ft′(θ)ɛtxt-1-2ft(θ)ft′(θ)ɛt-1xt-1)Δn(θ0,σ0)→p0.
Note that
(4.31)ɛt=yt-xtTβ=xtT(β0-β)+et,ɛt-ft(θ0)ɛt-1=(xt-ft(θ0)xt-1)T(β0-β)+ηt.
Consider
(4.32)J=∑t=2n(ft′(θ)ɛt-1xt+ft′(θ)ɛtxt-1-2ft(θ)ft′(θ)ɛt-1xt-1)=∑t=2n(ɛt-1ft′(θ)(xt-ft(θ0)xt-1)+ft′(θ)(ɛt-ft(θ0)ɛt-1)xt-1)+2ft(θ)(ft(θ0)-ft(θ))ɛt-1xt-1=T1+T2+T3+T4+2T5+2T6,
where
(4.33)T1=∑t=2nxt-1Tft′(θ)(β0-β)(xt-ft(θ0)xt-1),T2=∑t=2nft′(θ)et-1(xt-ft(θ0)xt-1),T3=∑t=2nft′(θ)(xt-ft(θ0)xt-1)T(β0-β)xt-1,T4=∑t=2nft′(θ)ηtxt-1,T5=∑t=2nft′(θ)(ft(θ0)-ft(θ))xt-1T(β0-β)xt-1,T6=∑t=2nft′(θ)(ft(θ0)-ft(θ))et-1xt-1.
For β∈Nnβ(A) and each A>0, we have
(4.34)|(β0-β)Txt|2=(β0-β)TXn1/2(θ0)Xn-1/2(θ0)xtxtTXn-T/2(θ0)XnT/2(θ0)(β0-β)≤max1≤t≤n(xtTXn-1(θ0)xt)(β0-β)TXn(θ0)(β0-β)≤A2max1≤t≤n(xtTXn-1(θ0)xt).
By (4.34) and Lemma 4.2, we have
(4.35)supβ∈Nnβ(A)max1≤t≤n|(β0-β)Txt|→0,n→∞,A>0.
Using Cauchy-Schwartz inequality, condition (A3), and (4.35), we obtain
(4.36)unTT1=∑t=2nunTxt-1T(β0-β)ft′(θ)(xt-ft(θ0)xt-1)≤{∑t=2n(xt-1T(β0-β))2}1/2×{∑t=2n(ft′2(θ)unT(xt-ft(θ0)xt-1)(xt-ft(θ0)xt-1)Tun)2}1/2≤nmax1≤t≤n|(β0-β)Txt|max1≤t≤n|ft′(θ)|=o(n).
Let
(4.37)atn=unTft′(θ)(xt-ft(θ0)xt-1).
Then from Lemma 4.2,
(4.38)max2≤t≤natn2=max2≤t≤n|ft′2(θ)|×max2≤t≤n{uT(xt-ft(θ0)xt-1)Xn-1(θ0)(xt-ft(θ0)xt-1)Tu}=o(1).
By condition (A2) and (4.38), we have
(4.39)Var(unTT2)=Var(∑t=2natnet-1)=Var(∑t=2natnet-1)=Var{∑j=1n-1ηj(∑t=j+1natn∏i=0t-j-1ft-i(θ0))}=σ02∑j=1n-1(∑t=j+1natn∏i=0t-j-1ft-i(θ0))≤σ02max2≤t≤n|atn|∑j=1n-1(∏i=0t-j-1ft-i(θ0))≤ασ02max2≤t≤n|atn|n=o(n).
Thus by Chebychev inequality and (4.39),
(4.40)unTT2=op(n).
Using the similar argument as T1, we obtain that
(4.41)unTT3=op(n).
Using the similar argument as T2, we obtain that
(4.42)unTT4=op(n),unTT6=op(n).
By Cauchy-Schwartz inequality, (4.35), and (4.27), we get
(4.43)unTT5=∑t=2nft′(θ)(ft(θ0)-ft(θ))xt-1T(β0-β)utTxt-1≤{∑t=2nft′2(θ)(ft(θ0)-ft(θ))2(xt-1T(β0-β))2∑t=2n(utTxt-1)2}1/2≤max2≤t≤n|ft′(θ)|max2≤t≤n|ft′(θ̃)||θ0-θ|·{∑t=2n(xt-1T(β0-β))2∑t=2n(utTxt-1)2}1/2≤CAΔn(θ0,σ0)no(1)=o(n).
Thus (4.30) follows immediately from (4.32), (4.36), and (4.40)–(4.43).
In the third step, we will show that(4.44)∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)ɛtɛt-1)-∑t=2nft′2(θ0)et-12Δn(θ0,σ0)→p0.
Let
(4.45)J=∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)ɛtɛt-1)-∑t=2nft′2(θ0)et-12.
Then
(4.46)J=∑t=2n((ft′2(θ)+ft(θ)ft′′(θ))ɛt-12-ft′′(θ)(ft(θ)ɛt-1+ηt)ɛt-1)-∑t=2nft′2(θ0)et-12=∑t=2n{ft′2(θ)ɛt-12-ft′′(θ)ηtɛt-1-ft′2(θ0)et-12}=∑t=2n{ft′2(xt-1T(β0-β))2+(ft′2(θ)-ft′2(θ0))2et-12}+∑t=2n{2ft′2(θ0)xt-1T(β0-β)et-1-ft′′(θ)xt-1T(β0-β)ηt-1-ft′′(θ0)ηtet-1}=T1+T2+2T3-T4-T5.
By (4.34), it is easy to show that
(4.47)T1=o(n).
From condition (A3), (2.30), and (4.23), we obtain that
(4.48)|ET2|=|(ft′2(θ)-ft′2(θ0))2Eet-12|=|ft′′(θ̃)(θ-θ0)(ft′(θ)+ft′(θ0)ft′2(θ0))ft′2(θ0)Eet-12|≤max2≤t≤n|ft′′(θ̃)(ft′(θ)+ft′(θ0)ft′2(θ0))||θ-θ0|Δn(θ0,σ0)≤max2≤t≤n|ft′′(θ̃)(ft′(θ)+ft′(θ0)ft′2(θ0))|AΔn(θ0,σ0)Δn(θ0,σ0)=o(Δn(θ0,σ0)).
Hence, by Markov inequality,
(4.49)T2=Op(Δn(θ0,σ0)).
Using the similar argument as (4.40), we easily obtain that
(4.50)T3=op(n).
By Markov inequality and noting that
(4.51)Var(T4)≤σ02nmax2≤t≤n|ft′′(θ)|2max2≤t≤n(xtT(β0-β))=o(n),
we have that
(4.52)T4=op(n).
Using the similar argument as (4.6), we easily obtain that
(4.53)T5=Op(Δn(θ0,σ0)).
Hence, (4.44) follows immediately from (4.46), (4.47), and (4.49)–(4.53).
This completes the proof of (4.15) from (4.21), (4.24), (4.30), and (4.44). To prove (4.16), we need to show that
(4.54)∑t=2nft′2(θ0)et-12Δn(θ0,σ0)→p1,n→∞.
This follows immediately from (2.27) and Markov inequality.
Finally, we will prove (4.17). By (4.15) and (4.16), we have
(4.55)Dn-1/2F(φ)Dn-T/2→pI,n→∞,
uniformly in φ∈Nn(A) for each A>0. Thus, by Lemma 4.3,
(4.56)λmin(Dn-1/2F(φ)Dn-T/2)→p1,n→∞.
This implies (4.17).Lemma 4.5 (see [49]).
Let{Sni,Fni,1≤i≤kn,n≥1} be a zero-mean, square-integrable martingale array with differences Xni, and let η2 be an a.s. finite random variable. Suppose that ∑iE{Xni2I(|Xni|>ɛ)∣Fn,i-1}→p0, for all ɛ→0, and ∑iE{Xni2∣Fn,i-1}→pη2. Then
(4.57)Snkn=∑iXni→DZ,
where the r.v. Z has characteristic function E{exp(-(1/2)η2t2)}.


## 5. Proof of Theorems

### 5.1. Proof of Theorem3.1

TakeA>0, let(5.1)Mn(A)={φ∈Rd+1:(φ-φ0)TDn(φ-φ0)=A2}
be the boundary of Nn(A), and let φ∈Mn(A). Using (2.27) and Taylor expansion, for each σ2>0, we have(5.2)Ψn(φ,σ2)=Ψn(φ0,σ2)+(φ-φ0)T∂Ψn(φ0,σ2)∂φ+12(φ-φ0)T∂2Ψn(φ0,σ2)∂φ∂φT(φ-φ0)=1σ2Ψn(φ0,σ2)+(φ-φ0)TSn(φ0)-12σ2(φ-φ0)TFn(φ̃)(φ-φ0),
where φ̃=aφ+(1-a)φ0 for some 0≤a≤1.LetQn(φ)=(1/2)(φ-φ0)TFn(φ̃)(φ-φ0) and vn(φ)=(1/A)DnT/2(φ-φ0). Take c>0 and φ∈Mn(A), and by (5.2) we obtain that(5.3)P{Ψn(φ,σ2)≥Ψn(φ0,σ2)forsomeφ∈Mn(A)}≤P{(φ-φ0)TSn(φ0)≥Qn(φ),Qn(φ)>cA2forsomeφ∈Mn(A)}+P{Qn(φ)≤cA2forsomeφ∈Mn(A)}≤P{vnT(φ)Dn-1/2Sn(φ0)>cAforsomeφ∈Mn(A)}+P{vnT(φ)Dn-1/2Fn(φ̃)Dn-T/2vn(φ)≤cforsomeφ∈Mn(A)}≤P{|Dn-1/2Sn(φ0)|>cA}+P{infφ∈Nn(A)λmin(Dn-1/2Fn(φ̃)Dn-T/2)≤c}.
By Lemma 4.1 and Chebychev inequality, we obtain(5.4)P{|Dn-1/2Sn(φ0)|>cA}≤Var(Dn-1/2Sn(φ0))c2A2=σ02c2A2.
Let A→∞, then c↓0, and using (4.17), we have(5.5)P{infφ∈Nn(A)λmin(Dn-1/2Fn(φ̃)Dn-T/2)≤c}→0.
By (5.3)–(5.5), we have(5.6)limA→∞lim infn→∞P{Ψn(φ,σ2)<Ψn(φ0,σ2)∀φ∈Mn(A)}=1.
By Lemma 4.3, λmin(Xn(θ0))→∞ as n→∞. Hence λmin(Dn)→∞. Moreover, from (4.17), we have (5.7)infφ∈Nn(A)λmin(Fn(φ))→p∞.
This implies that Ψn(φ,σ2) is concave on Nn(A). Noting this fact and (5.6), we get(5.8)limA→∞lim infn→∞P{supφ∈Mn(A)Ψn(φ,σ2)<Ψn(φ0,σ2),Ψn(φ,σ2)isconcaveonNn(A)}=1.
On the event in the brackets, the continuous function Ψn(φ,σ2) has a unique maximum in φ over the compact neighborhood Nn(A). Hence (5.9)limA→∞lim infn→∞P{Sn(φ̂n(A))=0forauniqueφ̂n(A)∈Nn(A)}=1.
Moreover, there is a sequence An→∞ such that φ̂n=φ̂(An) satisfies (5.10)lim infn→∞P{Sn(φ̂n)=0andφ̂nmaximizesΨn(φ,σ2)uniquelyinNn(A)}=1.
Thus the φ̂n=(β̂n,θ̂n) is a QML estimator for φ0. It is clearly consistent, and (5.11)limA→∞lim infn→∞P{φ̂n∈Nn(A)}=1.
Since φ̂n=(β̂n,θ̂n) is a QML estimator for φ0, σ̂n2 is a QML estimator for σ02 from (2.9).To complete the proof, we will show thatσ̂n2→σ02 as n→∞. If φ̂n∈Nn(A), then β̂n∈Nnβ(A) and θ̂n∈Nnθ(A). By (2.12) and (2.1), we have(5.12)ɛ̂t-ft(θ̂n)ɛ̂t-1=(xt-ft(θ̂n)xt-1)T(β0-β̂n)+(et-ft(θ̂net-1)).
By (2.9), (2.11), and (5.12), we have(5.13)(n-1)σ̂n2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1){(xt-ft(θ̂n)xt-1)T(β0-β̂n)+(et-ft(θ̂n)et-1)}=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(xt-ft(θ̂n)xt-1)T(β0-β̂n)+∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(et-ft(θ̂n)et-1)=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(et-ft(θ̂n)et-1).
From (5.12), it follows that(5.14)∑t=2n{(xt-ft(θ̂n)xt-1)T(β0-β̂n)}2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)2-2∑t=2nft(θ̂n)xt-1)T(β0-β̂n)(et-ft(θ̂n)et-1)+∑t=2n(et-ft(θ̂n)et-1)2.
From (2.2),(5.15)∑t=2n(et-ft(θ̂n)et-1)2=∑t=2n(ft(θ0)et-1-ft(θ̂n)et-1)2=∑t=2nηt2+2∑t=2n(ft(θ0)-ft(θ̂n))ηtet-1+∑t=2n(ft(θ0)-ft(θ̂n))2et-12.
By (5.13)–(5.15), we have(5.16)(n-1)σ̂n2=∑t=2n(et-ft(θ̂n)et-1)2-∑t=2n((xt-ft(θ̂n)xt-1)T(β0-β̂n))2=∑t=2nηt2+2∑t=2n(ft(θ0)-ft(θ̂n))ηtet-1+∑t=2n(ft(θ0)-ft(θ̂n))2et-12-∑t=2n((xt-ft(θ̂n)xt-1)T(β0-β̂n))2=T1+2T2+T3-T4.
By the law of large numbers,(5.17)1n-1T1=1n-1∑t=2nηt2→σ02,a.s.(n→∞).
Since {(ft(θ0)-ft(θ̂n))ηtet-1,Ht-1} is a martingale difference sequence with (5.18)Var{(ft(θ0)-ft(θ̂n))ηtet-1}=(ft(θ0)-ft(θ̂n))2σ02Eet-12,Var(T2)=∑t=2nE((ft(θ0)-ft(θ̂n))ηtet-1)2=σ02∑t=2n(ft(θ0)-ft(θ̂n))2Eet-12=σ02∑t=2nft′2(θ̃)ft′2(θ0)(θ0-θ̂n)2ft′2(θ0)Eet-12≤σ02max2≤t≤n|ft′(θ̃)ft′(θ0)|2|(θ0-θ̂n)|2Δn(θ0,σ02)≤CA2.
By Chebychev inequality, we have(5.19)1n-1T2→p0,(n→∞).
By Markov inequality and noting that ET3≤CA2, we obtain that(5.20)1n-1T3→p0,(n→∞).
Write(5.21)T4=∑t=2n((xt-ft(θ0)xt-1)T(β0-β̂n)+(ft(θ0)-ft(θ̂n))xt-1T(β0-β̂n))2=(β0-β̂n)TXn(θ0)(β0-β̂n)+∑t=2n(ft(θ0)-ft(θ̂n))2(xt-1T(β0-β̂n))2+2∑t=2n(xt-ft(θ0)xt-1)T(β0-β̂n)(ft(θ0)-ft(θ̂n))xt-1T(β0-β̂n)=I1+I2+2I3.
Noting that β∈Nnβ(A), we have(5.22)I1=Op(1),(n→∞).
By (4.34) and condition (A3), we have(5.23)|I2|≤max2≤t≤n|xt-1T(β0-β̂n)|2nmax2≤t≤n|ft′(θ̃)|2|θ0-θ̂n|2=o(1),(n→∞).
By (4.34), condition (A3), and Cauchy-Schwartz inequality, we have(5.24)|I3|2≤∑t=2n(β0-β̂n)T(xt-ft(θ0)xt-1)T(xt-ft(θ0)xt-1)(β0-β̂n)·∑t=2n(ft(θ0)-ft(θ̂n))2(xt-1T(β0-β̂n))2≤(β0-β̂n)TXn(θ0)(β0-β̂n)nmax2≤t≤n|ft′(θ̃)|2|θ0-θ̂n|2·max2≤t≤n|xt-1T(β0-β̂n)|2≤Op(1)nA2Δn(θ0,σ02)o(1)=op(1).
By (5.21)–(5.24), we obtain(5.25)1n-1T4→p0,(n→∞).
From (5.16), (5.17), (5.19), (5.20), and (5.25), we have σ̂n2→σ02. We therefore complete the proof of Theorem 3.1.


### 5.2. Proof of Theorem3.2

It is easy to know thatSn(φ̂n)=0 and Fn(φ̂n) is nonsingular from Theorem 3.1. By Taylor's expansion, we have(5.26)0=Sn(φ̂n)=Sn(φ0)-Fn(φ̃n)(φ̂n-φ0).
Since φ̂n∈Nn(A), also φ̃n∈Nn(A). By (4.15), we have(5.27)Fn(φ̃n)=Dn1/2(Φn+Ãn)DnT/2,
where Ãn is a symmetric matrix with Ãn→p0. By (5.26) and (5.27), we have(5.28)DnT/2(φ̂n-φ0)=DnT/2Fn-1(φ̃n)Sn(φ0)=(Φn+Ãn)-1Dn-1/2Sn(φ0).
Similar to (5.27), we have(5.29)Fn(φ̂n)=Dn1/2(Φn+Ân)DnT/2=(Dn1/2(Φn+Ân)1/2)((Φn+Ân)T/2DnT/2)=Fn1/2(φ̂n)FnT/2(φ̂n).
Here Ân→p0. By (5.28), (5.29), and noting that σ̂n2→pσ02 and Dn-1/2Sn(φ0)=Op(1), we obtain that(5.30)FnT/2(φ̂n)(φ̂n-φ0)σ̂n=(Φn+Ân)1/2(Φn+Ãn)-1Dn-1/2Sn(φ0)σ̂n=Φn-1/2Dn-1/2Sn(φ0)σ0+op(1).
From (2.7) and (2.8), we have(5.31)Sn(φ0)σ0=1σ0(∑t=2nηt(xt-ft(θ0)xt-1),∑t=2nft′(θ0)ηtet-1).
From (2.29) and (4.18), we have(5.32)Φn-1/2Dn-1/2=(Id00Δn(θ0,σ0)∑t=2nft′2(θ0)et-12)(Xn-1/2(θ0)001Δn(θ0,σ0))=(Xn-1/2(θ0)001∑t=2nft′2(θ0)et-12).
By (5.30)–(5.32), we have(5.33)Φn-1/2Dn-1/2Sn(φ0)σ0=1σ0(∑t=2nηtXn-1/2(θ0)(xt-ft(θ0)xt-1),∑t=2nft′(θ0)ηtet-1∑t=2nft′2(θ0)et-12).Letu∈Rd with |u|=1, and atn=uXn-1/2(θ0)(xt-ft(θ0)xt-1). Then max2≤t≤natn=o(1), and we will consider the limiting distribution of the following 2-vector:(5.34)1σ0(∑t=2natnηt,∑t=2nft′(θ0)ηtet-1∑t=2nft′2(θ0)et-12).
By Cramer-Wold device, it will suffice to find the asymptotic distribution of the following random:(5.35)∑t=2nηt(u1atnσ0+u2ft′(θ0)et-1σ0Δn(θ0,σ0))=∑t=2nηtmt(n),
where (u1,u2)∈R2 with u12+u22=1. Note that E{ηtmt(n)∣Ht-1}=0, so the sums in (5.35) are partial sums of a martingale triangular array with respect to {Ht}, and we will verify the Lindeberg conditions for their convergence to normality as follows: (5.36)∑t=2nE(ηt2mt2(n)∣Ht-1)=σ02∑t=2nmt2(n)=u12∑t=2natn2+2u1u2Δn(θ0,σ0)∑t=2nft′(θ0)atnet-1+u22Δn(θ0,σ0)∑t=2nft′2(θ0)et-12=u12+op(1)+u22=1+op(1).
Noting that max1≤t≤n(et2/n)=op(1) and max2≤t≤natn=o(1), we obtain that(5.37)max2≤t≤n|mt(n)|≤max2≤t≤n|u1atnσ0|+max2≤t≤n|u2ft′(θ0)et-1σ0Δn(θ0,σ0)|=op(1).
Hence, for given δ>0, there is a set whose probability approaches 1 as n→∞ on which max2≤t≤n|mt(n)|≤δ. On this event, for any c>0,(5.38)∑t=2nE{ηt2mt2(n)I(|ηtmt(n)|>c)∣Ht-1}=∑t=2∞∫c∞y2dp{|ηtmt(n)|≤y∣Ht-1}=∑t=2nmt2(n)∫c/mt(n)∞y2dp{|η1|≤y∣Ht-1}≤∑t=2nmt2(n)∫c/δ∞y2dp{|η1|≤y∣Ht-1}=oδ∑t=2nmt2(n)=oδOp(1)→0,n→∞.
Here oδ→0 as δ→0. This verifies the Lindeberg conditions, and by Lemma 4.5(5.39)∑t=2nηtmt(n)→DN(0,1).
Thus we complete the proof of Theorem 3.2.


## 5.1. Proof of Theorem3.1

TakeA>0, let(5.1)Mn(A)={φ∈Rd+1:(φ-φ0)TDn(φ-φ0)=A2}
be the boundary of Nn(A), and let φ∈Mn(A). Using (2.27) and Taylor expansion, for each σ2>0, we have(5.2)Ψn(φ,σ2)=Ψn(φ0,σ2)+(φ-φ0)T∂Ψn(φ0,σ2)∂φ+12(φ-φ0)T∂2Ψn(φ0,σ2)∂φ∂φT(φ-φ0)=1σ2Ψn(φ0,σ2)+(φ-φ0)TSn(φ0)-12σ2(φ-φ0)TFn(φ̃)(φ-φ0),
where φ̃=aφ+(1-a)φ0 for some 0≤a≤1.LetQn(φ)=(1/2)(φ-φ0)TFn(φ̃)(φ-φ0) and vn(φ)=(1/A)DnT/2(φ-φ0). Take c>0 and φ∈Mn(A), and by (5.2) we obtain that(5.3)P{Ψn(φ,σ2)≥Ψn(φ0,σ2)forsomeφ∈Mn(A)}≤P{(φ-φ0)TSn(φ0)≥Qn(φ),Qn(φ)>cA2forsomeφ∈Mn(A)}+P{Qn(φ)≤cA2forsomeφ∈Mn(A)}≤P{vnT(φ)Dn-1/2Sn(φ0)>cAforsomeφ∈Mn(A)}+P{vnT(φ)Dn-1/2Fn(φ̃)Dn-T/2vn(φ)≤cforsomeφ∈Mn(A)}≤P{|Dn-1/2Sn(φ0)|>cA}+P{infφ∈Nn(A)λmin(Dn-1/2Fn(φ̃)Dn-T/2)≤c}.
By Lemma 4.1 and Chebychev inequality, we obtain(5.4)P{|Dn-1/2Sn(φ0)|>cA}≤Var(Dn-1/2Sn(φ0))c2A2=σ02c2A2.
Let A→∞, then c↓0, and using (4.17), we have(5.5)P{infφ∈Nn(A)λmin(Dn-1/2Fn(φ̃)Dn-T/2)≤c}→0.
By (5.3)–(5.5), we have(5.6)limA→∞lim infn→∞P{Ψn(φ,σ2)<Ψn(φ0,σ2)∀φ∈Mn(A)}=1.
By Lemma 4.3, λmin(Xn(θ0))→∞ as n→∞. Hence λmin(Dn)→∞. Moreover, from (4.17), we have (5.7)infφ∈Nn(A)λmin(Fn(φ))→p∞.
This implies that Ψn(φ,σ2) is concave on Nn(A). Noting this fact and (5.6), we get(5.8)limA→∞lim infn→∞P{supφ∈Mn(A)Ψn(φ,σ2)<Ψn(φ0,σ2),Ψn(φ,σ2)isconcaveonNn(A)}=1.
On the event in the brackets, the continuous function Ψn(φ,σ2) has a unique maximum in φ over the compact neighborhood Nn(A). Hence (5.9)limA→∞lim infn→∞P{Sn(φ̂n(A))=0forauniqueφ̂n(A)∈Nn(A)}=1.
Moreover, there is a sequence An→∞ such that φ̂n=φ̂(An) satisfies (5.10)lim infn→∞P{Sn(φ̂n)=0andφ̂nmaximizesΨn(φ,σ2)uniquelyinNn(A)}=1.
Thus the φ̂n=(β̂n,θ̂n) is a QML estimator for φ0. It is clearly consistent, and (5.11)limA→∞lim infn→∞P{φ̂n∈Nn(A)}=1.
Since φ̂n=(β̂n,θ̂n) is a QML estimator for φ0, σ̂n2 is a QML estimator for σ02 from (2.9).To complete the proof, we will show thatσ̂n2→σ02 as n→∞. If φ̂n∈Nn(A), then β̂n∈Nnβ(A) and θ̂n∈Nnθ(A). By (2.12) and (2.1), we have(5.12)ɛ̂t-ft(θ̂n)ɛ̂t-1=(xt-ft(θ̂n)xt-1)T(β0-β̂n)+(et-ft(θ̂net-1)).
By (2.9), (2.11), and (5.12), we have(5.13)(n-1)σ̂n2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1){(xt-ft(θ̂n)xt-1)T(β0-β̂n)+(et-ft(θ̂n)et-1)}=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(xt-ft(θ̂n)xt-1)T(β0-β̂n)+∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(et-ft(θ̂n)et-1)=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)(et-ft(θ̂n)et-1).
From (5.12), it follows that(5.14)∑t=2n{(xt-ft(θ̂n)xt-1)T(β0-β̂n)}2=∑t=2n(ɛ̂t-ft(θ̂n)ɛ̂t-1)2-2∑t=2nft(θ̂n)xt-1)T(β0-β̂n)(et-ft(θ̂n)et-1)+∑t=2n(et-ft(θ̂n)et-1)2.
From (2.2),(5.15)∑t=2n(et-ft(θ̂n)et-1)2=∑t=2n(ft(θ0)et-1-ft(θ̂n)et-1)2=∑t=2nηt2+2∑t=2n(ft(θ0)-ft(θ̂n))ηtet-1+∑t=2n(ft(θ0)-ft(θ̂n))2et-12.
By (5.13)–(5.15), we have(5.16)(n-1)σ̂n2=∑t=2n(et-ft(θ̂n)et-1)2-∑t=2n((xt-ft(θ̂n)xt-1)T(β0-β̂n))2=∑t=2nηt2+2∑t=2n(ft(θ0)-ft(θ̂n))ηtet-1+∑t=2n(ft(θ0)-ft(θ̂n))2et-12-∑t=2n((xt-ft(θ̂n)xt-1)T(β0-β̂n))2=T1+2T2+T3-T4.
By the law of large numbers,(5.17)1n-1T1=1n-1∑t=2nηt2→σ02,a.s.(n→∞).
Since {(ft(θ0)-ft(θ̂n))ηtet-1,Ht-1} is a martingale difference sequence with (5.18)Var{(ft(θ0)-ft(θ̂n))ηtet-1}=(ft(θ0)-ft(θ̂n))2σ02Eet-12,Var(T2)=∑t=2nE((ft(θ0)-ft(θ̂n))ηtet-1)2=σ02∑t=2n(ft(θ0)-ft(θ̂n))2Eet-12=σ02∑t=2nft′2(θ̃)ft′2(θ0)(θ0-θ̂n)2ft′2(θ0)Eet-12≤σ02max2≤t≤n|ft′(θ̃)ft′(θ0)|2|(θ0-θ̂n)|2Δn(θ0,σ02)≤CA2.
By Chebychev inequality, we have(5.19)1n-1T2→p0,(n→∞).
By Markov inequality and noting that ET3≤CA2, we obtain that(5.20)1n-1T3→p0,(n→∞).
Write(5.21)T4=∑t=2n((xt-ft(θ0)xt-1)T(β0-β̂n)+(ft(θ0)-ft(θ̂n))xt-1T(β0-β̂n))2=(β0-β̂n)TXn(θ0)(β0-β̂n)+∑t=2n(ft(θ0)-ft(θ̂n))2(xt-1T(β0-β̂n))2+2∑t=2n(xt-ft(θ0)xt-1)T(β0-β̂n)(ft(θ0)-ft(θ̂n))xt-1T(β0-β̂n)=I1+I2+2I3.
Noting that β∈Nnβ(A), we have(5.22)I1=Op(1),(n→∞).
By (4.34) and condition (A3), we have(5.23)|I2|≤max2≤t≤n|xt-1T(β0-β̂n)|2nmax2≤t≤n|ft′(θ̃)|2|θ0-θ̂n|2=o(1),(n→∞).
By (4.34), condition (A3), and Cauchy-Schwartz inequality, we have(5.24)|I3|2≤∑t=2n(β0-β̂n)T(xt-ft(θ0)xt-1)T(xt-ft(θ0)xt-1)(β0-β̂n)·∑t=2n(ft(θ0)-ft(θ̂n))2(xt-1T(β0-β̂n))2≤(β0-β̂n)TXn(θ0)(β0-β̂n)nmax2≤t≤n|ft′(θ̃)|2|θ0-θ̂n|2·max2≤t≤n|xt-1T(β0-β̂n)|2≤Op(1)nA2Δn(θ0,σ02)o(1)=op(1).
By (5.21)–(5.24), we obtain(5.25)1n-1T4→p0,(n→∞).
From (5.16), (5.17), (5.19), (5.20), and (5.25), we have σ̂n2→σ02. We therefore complete the proof of Theorem 3.1.


## 5.2. Proof of Theorem3.2

It is easy to know thatSn(φ̂n)=0 and Fn(φ̂n) is nonsingular from Theorem 3.1. By Taylor's expansion, we have(5.26)0=Sn(φ̂n)=Sn(φ0)-Fn(φ̃n)(φ̂n-φ0).
Since φ̂n∈Nn(A), also φ̃n∈Nn(A). By (4.15), we have(5.27)Fn(φ̃n)=Dn1/2(Φn+Ãn)DnT/2,
where Ãn is a symmetric matrix with Ãn→p0. By (5.26) and (5.27), we have(5.28)DnT/2(φ̂n-φ0)=DnT/2Fn-1(φ̃n)Sn(φ0)=(Φn+Ãn)-1Dn-1/2Sn(φ0).
Similar to (5.27), we have(5.29)Fn(φ̂n)=Dn1/2(Φn+Ân)DnT/2=(Dn1/2(Φn+Ân)1/2)((Φn+Ân)T/2DnT/2)=Fn1/2(φ̂n)FnT/2(φ̂n).
Here Ân→p0. By (5.28), (5.29), and noting that σ̂n2→pσ02 and Dn-1/2Sn(φ0)=Op(1), we obtain that(5.30)FnT/2(φ̂n)(φ̂n-φ0)σ̂n=(Φn+Ân)1/2(Φn+Ãn)-1Dn-1/2Sn(φ0)σ̂n=Φn-1/2Dn-1/2Sn(φ0)σ0+op(1).
From (2.7) and (2.8), we have(5.31)Sn(φ0)σ0=1σ0(∑t=2nηt(xt-ft(θ0)xt-1),∑t=2nft′(θ0)ηtet-1).
From (2.29) and (4.18), we have(5.32)Φn-1/2Dn-1/2=(Id00Δn(θ0,σ0)∑t=2nft′2(θ0)et-12)(Xn-1/2(θ0)001Δn(θ0,σ0))=(Xn-1/2(θ0)001∑t=2nft′2(θ0)et-12).
By (5.30)–(5.32), we have(5.33)Φn-1/2Dn-1/2Sn(φ0)σ0=1σ0(∑t=2nηtXn-1/2(θ0)(xt-ft(θ0)xt-1),∑t=2nft′(θ0)ηtet-1∑t=2nft′2(θ0)et-12).Letu∈Rd with |u|=1, and atn=uXn-1/2(θ0)(xt-ft(θ0)xt-1). Then max2≤t≤natn=o(1), and we will consider the limiting distribution of the following 2-vector:(5.34)1σ0(∑t=2natnηt,∑t=2nft′(θ0)ηtet-1∑t=2nft′2(θ0)et-12).
By Cramer-Wold device, it will suffice to find the asymptotic distribution of the following random:(5.35)∑t=2nηt(u1atnσ0+u2ft′(θ0)et-1σ0Δn(θ0,σ0))=∑t=2nηtmt(n),
where (u1,u2)∈R2 with u12+u22=1. Note that E{ηtmt(n)∣Ht-1}=0, so the sums in (5.35) are partial sums of a martingale triangular array with respect to {Ht}, and we will verify the Lindeberg conditions for their convergence to normality as follows: (5.36)∑t=2nE(ηt2mt2(n)∣Ht-1)=σ02∑t=2nmt2(n)=u12∑t=2natn2+2u1u2Δn(θ0,σ0)∑t=2nft′(θ0)atnet-1+u22Δn(θ0,σ0)∑t=2nft′2(θ0)et-12=u12+op(1)+u22=1+op(1).
Noting that max1≤t≤n(et2/n)=op(1) and max2≤t≤natn=o(1), we obtain that(5.37)max2≤t≤n|mt(n)|≤max2≤t≤n|u1atnσ0|+max2≤t≤n|u2ft′(θ0)et-1σ0Δn(θ0,σ0)|=op(1).
Hence, for given δ>0, there is a set whose probability approaches 1 as n→∞ on which max2≤t≤n|mt(n)|≤δ. On this event, for any c>0,(5.38)∑t=2nE{ηt2mt2(n)I(|ηtmt(n)|>c)∣Ht-1}=∑t=2∞∫c∞y2dp{|ηtmt(n)|≤y∣Ht-1}=∑t=2nmt2(n)∫c/mt(n)∞y2dp{|η1|≤y∣Ht-1}≤∑t=2nmt2(n)∫c/δ∞y2dp{|η1|≤y∣Ht-1}=oδ∑t=2nmt2(n)=oδOp(1)→0,n→∞.
Here oδ→0 as δ→0. This verifies the Lindeberg conditions, and by Lemma 4.5(5.39)∑t=2nηtmt(n)→DN(0,1).
Thus we complete the proof of Theorem 3.2.


## 6. Numerical Examples

### 6.1. Simulation Example

We will simulate a regression model (1.1), where xt=t/20,β=3.5,t=1,2,…,100, and the random errors(6.1)ɛt=12sin(π2t+θ)ɛt-1+ηt,
where θ=2,ηt~N(0,1).By the ordinary least squares method, we obtain the least squares estimatorsβ̂LS=3.5136, and σ̂LS2=1.0347. So we take β(0)=3.5136,σ(0)2=1.0347, θ(0)=π/2, and δ=0.01. Therefore, using the iterative computing method, we obtain (6.2)θ⃗(1)=(1.0159,3.5076,1.6573)T,θ⃗(2)=(1.0283,3.5076,1.7599)T.
Since ∥θ⃗(2)-θ⃗(1)∥/∥θ⃗(1)∥<0.01, the QML estimators of β,θ, and σ2 are given by (6.3)σ̂n2=1.0283,β̂n=3.5076,θ̂n=1.7599.These values closely approximate their true values, so our method is successful, especially in estimating the parametersβ and σ2.


### 6.2. Empirical Example

We will use the data studied by Fuller in [14]. The data pertain to the consumption of spirits in the United Kingdom from 1870 to 1983. The dependent variable yt is the annual per capita consumption of spirits in the United Kingdom. The explanatory variables xt1 and xt2 are per capita income and price of spirits, respectively, both deflated by a general price index. All data are in logarithms. The model suggested by Prest can be written as(6.4)yt=β0+β1xt1+β2xt2+β3xt3+β4xt4+ɛt,
where 1869 is the origin for t, xt3=t/100,andxt4=(t-35)2/104, and assuming that ɛt is a stationary time series.Fuller [14] obtained the estimated generalized least squares equation(6.5)ŷt=2.36+0.72xt1-0.80xt2-0.81xt3-0.92xt4,ɛt=0.7633ɛt-1+ηt,
where ηt is a sequence of uncorrelated (0,0.000417) random variables.Using our method, we obtain the following models:(6.6)ŷt=2.3607+0.7437xt1-0.8210xt2-0.7857xt3-0.9178xt4,ɛt=(0.70054+0.00424t)ɛt-1+ηt,
where ηt is a sequence of uncorrelated (0,0.000413) random variables.By the models (6.6), the residual mean square is 0.000413, which is smaller than 0.000417 calculated by the models (6.5).From the above examples, it can be seen that our method is successful and valid. However, a further discussion of fitting the functionft(θ) is needed so that we can find a good method to use in practical applications.


## 6.1. Simulation Example

We will simulate a regression model (1.1), where xt=t/20,β=3.5,t=1,2,…,100, and the random errors(6.1)ɛt=12sin(π2t+θ)ɛt-1+ηt,
where θ=2,ηt~N(0,1).By the ordinary least squares method, we obtain the least squares estimatorsβ̂LS=3.5136, and σ̂LS2=1.0347. So we take β(0)=3.5136,σ(0)2=1.0347, θ(0)=π/2, and δ=0.01. Therefore, using the iterative computing method, we obtain (6.2)θ⃗(1)=(1.0159,3.5076,1.6573)T,θ⃗(2)=(1.0283,3.5076,1.7599)T.
Since ∥θ⃗(2)-θ⃗(1)∥/∥θ⃗(1)∥<0.01, the QML estimators of β,θ, and σ2 are given by (6.3)σ̂n2=1.0283,β̂n=3.5076,θ̂n=1.7599.These values closely approximate their true values, so our method is successful, especially in estimating the parametersβ and σ2.


## 6.2. Empirical Example

We will use the data studied by Fuller in [14]. The data pertain to the consumption of spirits in the United Kingdom from 1870 to 1983. The dependent variable yt is the annual per capita consumption of spirits in the United Kingdom. The explanatory variables xt1 and xt2 are per capita income and price of spirits, respectively, both deflated by a general price index. All data are in logarithms. The model suggested by Prest can be written as(6.4)yt=β0+β1xt1+β2xt2+β3xt3+β4xt4+ɛt,
where 1869 is the origin for t, xt3=t/100,andxt4=(t-35)2/104, and assuming that ɛt is a stationary time series.Fuller [14] obtained the estimated generalized least squares equation(6.5)ŷt=2.36+0.72xt1-0.80xt2-0.81xt3-0.92xt4,ɛt=0.7633ɛt-1+ηt,
where ηt is a sequence of uncorrelated (0,0.000417) random variables.Using our method, we obtain the following models:(6.6)ŷt=2.3607+0.7437xt1-0.8210xt2-0.7857xt3-0.9178xt4,ɛt=(0.70054+0.00424t)ɛt-1+ηt,
where ηt is a sequence of uncorrelated (0,0.000413) random variables.By the models (6.6), the residual mean square is 0.000413, which is smaller than 0.000417 calculated by the models (6.5).From the above examples, it can be seen that our method is successful and valid. However, a further discussion of fitting the functionft(θ) is needed so that we can find a good method to use in practical applications.



---

*Source: 956907-2010-06-28.xml*