# The Law of Iterated Logarithm for Autoregressive Processes

Authors: Yan Wang; Mingzhi Mao; Xiaohua Hu; Tingting He
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

This paper mainly discusses some dynamics asymptotic properties of autoregressive processes. By using them-dependence of random variables, we prove the least squares (LS) estimator of the unknown parameters satisfies the law of iterated logarithm.


---

## Body

## 1. Introduction

Autoregressive process is a basic model in engineering, insurance, and business. It is a representation of a type of random process; as such, it describes certain time-varying processes in nature, engineering, and so forth. The autoregressive model specifies that the output variable depends linearly on its own previous values. It is a special case of the more general autoregressive-moving average (ARMA) model of time series. For example, in engineering one considers a dam, with input of random amounts at random times, and a steady withdrawal of water for irrigation or power usage. This model has a Markovian representation. It is well known that stability problem is an important interest topic, which explains why the stochastic systems always stay in “reasonable values”: the dam does not overflow. The parameters in the process have a direct relation to the system stability, so the estimation of fixed coefficients and its dynamics behaviors are very useful in engineering. There was a rich literature which focused on the research (see Mann and Wald [1], Anderson [2], Menneteau [3], Hwang and Choi [4], Hwang and Baek [5], and Miao and Shen [6]). Recently, some attention has been directed to random coefficient autoregressive models. This way of handling the data allows for large shocks in the dynamic structure of the model and also for some flexibility in the features of the volatility of the series, which are not available in fixed coefficient autoregressive models.For simplicity, in this paper, we are concerned with some dynamics behaviors (in probability language, we call it the law of iterated logarithm) of the parameters estimation for the linear autoregressive model which is of practical importance, by a different technique from the standard method. Define the following first order autoregressive process (AR(1)) {Xt,t≥1} by
(1)Xt=β0+β1Xt-1+ϵt,β1<1,
where the initial value X0 is not necessarily zero and {ϵt} is a sequence of independent and identically distributed (i.i.d.) random errors with mean zero and variance σ2. Based on the observations X1,X2,…,Xn, the least squares (LS) estimators (β^0(n),β^1(n))T of (β0,β1)T are given by
(2)β^0nβ^1nM=n∑t=1nXt-1∑t=1nXt-1∑t=1nXt-12-1∑t=1nXt∑t=1nXtXt-1.
By a simple calculation, we have
(3)β^0n-β0=∑t=1nXt-12∑t=1nϵt-∑t=1nXt-1∑t=1nϵtXt-1n∑t=1nXt-12-∑t=1nXt-12,β^1n-β1=-∑t=1nXt-1∑t=1nϵt+n∑t=1nϵtXt-1n∑t=1nXt-12-∑t=1nXt-12.
It is well known that, in the stable (or, in other words, asymptotically stationary) case, when β1<1, β^1(n) is asymptotically normal under the assumption that {ϵi} is a sequence of i.i.d. random variables with mean zero and variance 1; namely,
(4)nβ^1n-β1⟹N0,1-β1,
where “⇒” denotes the convergence in distribution, although it does not hold uniformly for |β1|<1 (see Anderson [2] and Meyn and Tweedie [7]). In the unstable (or, in other words, unit root) case when β1=1, for the sequence β^1(n), we have
(5)nβ^1n-β1⟹∫01wtdwt∫01wt2dt,
where {w(t):t∈[0,1]} denotes a standard Wiener process. In the explosive case, when |β1|>1, the sequence β^1(n) is again not asymptotically normal. If, for example, ϵ1~N(0,1), then
(6)β1nβ^1n-β1⟹Cauchy0,β12-1.
In general, the limit distribution depends on the distribution of ϵ1 (see Basawa and Brockwell [8]).In this paper, we consider the convergence rates ofβ0(n)→β0 and β1(n)→β1 under some conditions. By constructing an m-dependent random variables sequence and applying an approximation method along with a central limit theorem for random variables, we prove the LS estimators of β0 and β1 satisfy the law of iterated logarithm. Our results can be considered as an embodiment of Miao and Shen [6].In the following statement, we have an assumption as follows:(C.1)
there exists a constantγ>0 such that Eexp⁡{γ|ϵtϵs|}<+∞ for any t≥1 and s≥1.


## 2. Main Results

The main result is as follows.Theorem 1.
Assume that|β1|<1. Then the law of iterated logarithm for (β^0(n)-β0,β^1(n)-β1)T holds; namely,
(7)limsup⁡n→∞n2log⁡log⁡nβ^0n-β0=σ2+β021-β121-β12,a.s.,limsup⁡n→∞n2log⁡log⁡nβ^1n-β1=1-β12,a.s.


## 3. Preliminary Lemmas

In order to prove Theorem1, we need the following lemmas.Lemma 2.
Let{ξn}n≥1 be a sequence of i.i.d. random variables with Eξ1=0 and Eξ12=σ2. Assume that there a positive constant δ>0 such that Eeδ|ξ1|<∞. Then, for any x>0,
(8)lim⁡n→∞12log⁡log⁡nlog⁡P∑i=1nξi2nlog⁡log⁡n≥x=-x22σ2.Proof.
Letλ(n)→∞ and λ(n)/Nn→0 as n→∞. By virtue of the moderate deviation principle for {ξn}n≥1 (see pp 109, Dembo and Zeitouni [9]), we have
(9)limsup⁡n→∞1λ2nlog⁡P∑i=1nξinλn∈A≤-inf⁡x∈AIx,
for any closed set A⊆R, and
(10)liminf⁡n→∞1λ2nlog⁡P∑i=1nξinλn∈B≥-inf⁡x∈BIx,
for any open set B⊆R, where I(x)=1/2x2σ-2. Let A=[x,+∞) and B=(x,+∞). Set λ(n)=2log⁡log⁡n; we easily get the claim by (9) and (10).
The following lemma is about Lévy’s inequality. For completeness, we still give its proof.Lemma 3.
Let{ξn}n≥1 be a sequence of independent random variables taking their values in R. Let Sn=∑i=1nξi. Then for any x>0,
(11)Pmax⁡1≤j≤nSj+mSn-Sj≥x≤2PSn≥x,
where m(ξ) denotes the median of r.v. X.Proof.
LetS0=0, Mk*=max⁡1≤j≤k(Sj+m(Sn-Sj)), Ak={Mk-1*<x,Sk+m(Sn-Sk)≥x}, and Bk={Sn-Sk-m(Sn-Sk)≥0}. From the definition of median, it can be seen that P(Bk)≥1/2 for any 1≤k≤n. Further,
(12)PSn≥x≥P⋃k=1nAkBk=∑k=1nPAkBk=∑k=1nPAkPBk≥12∑k=1nPAn=12PMn*≥x.Remark 4.
A real numberm(ξ) is usually called the median of r.v. ξ, if P(ξ≥m(ξ))≥1/2 and P(ξ≤m(ξ))≥1/2.Lemma 5.
Let{ξn}n≥1 be a sequence of independent random variables with
(13)Eξn=0,Eξn2<∞.
Let Sn=∑i=1nξi and sn2=ESn2. Assume the following conditions hold:
(14)lim⁡n→∞sn=∞,limsup⁡n→∞sn+1sn<∞,
and there exists a constant 0<α≤1 satisfying
(15)∑n=1∞an-2-αEξn2+α<∞.
Then {ξn} satisfies the law of iterated logarithm; that is,
(16)limsup⁡n→∞Sn2sn2log⁡log⁡sn2=1,a.s.Proof.
See Theorem 1.2 of Wittmann [10].Lemma 6.
LetCt,p=∑i=1p+1‍β1i-1ϵtϵt-i, 1≤p≤t-2. Then
(17)limsup⁡p→∞limsup⁡n→∞∑t=1nCt,p2nlog⁡log⁡n=σ211-β12,a.s.,(18)liminf⁡p→∞liminf⁡n→∞∑t=1nCt,p2nlog⁡log⁡n=-σ211-β12,a.s.Proof.
Firstly, note thatEϵt=0 and E[ϵtϵs]=0 for any t≠s. So ECt,p=0. Furthermore, for t≠s,
(19)ECt,pCs,pM=E∑i=1p+1β1i-1ϵtϵt-i∑j=1p+1β1j-1ϵtϵt-jM=∑i=1p+1∑j=1p+1β1i+j-2Eϵtϵsϵt-iϵt-j=0,(20)ECt,p2M=E∑i=1p+1β1i-1ϵtϵt-i2M=∑i=1p+1β12i-2Eϵt2ϵt-i2+∑∑i≠ji,j∈1,…,p+1⁡β1i+j-2Eϵt2ϵt-iϵt-jM=1-β12p+11-β12σ4.
Then it is not difficult to see that {Ct,p}t≥1 is a strictly stationary (p+1)-dependence sequence of random variables. By (20) and Theorem 1 of Chen [11], one can prove the following equation:
(21)limsup⁡p→∞limsup⁡n→∞∑t=1nCt,p2nlog⁡log⁡n=limsup⁡p→∞ECp+2,p2+2∑k=p+32p+3ECp+2,pCk,p1/2=σ211-β12,a.s.
For the proof of (18), the method is similar.Lemma 7.
Let(22)An*=∑i=0n-2∑t=i+2nβ1iϵtϵt-i-1,An,p=∑i=0p∑t=i+2nβ1iϵtϵt-i-1.
Then,
(23)limsup⁡p→∞limsup⁡n→∞12nlog⁡log⁡nAn*-An,p=0,a.s.Proof.
In order to prove (23), we only need to show that, for any δ>0 and enough large p>0,
(24)PAn*-An,p≥δ2nlog⁡log⁡n,i.o.=0,
where i.o. denotes infinitely often occurring. Note that, for any τ>0, there exists an increasing integer sequence {nk} satisfying
(25)nk-1≤1+τk<nk,k∈N.
So nk-nk-1~nkτ/1+τ, where an~bn denotes an/bn→1. Furthermore, for any δ>0, we have the following inequality:
(26)PAn*-An,p≥δ2nlog⁡log⁡n,i.o.≤Pmax⁡nk-1≤n<nkAn*-An,p≥δ2nk-1log⁡log⁡nk-1,i.o.≤Pmax⁡n≤nkAn*-An,p≥δ2nk-1log⁡log⁡nk-1,i.o..
Also note that, for nk enough large,
(27)2nklog⁡log⁡nk2nk-1log⁡log⁡nk-1<1+2τ.
So
(28)PAn*-An,p≥δ2nlog⁡log⁡n,i.o.M≤Pmax⁡n≤nkAn*-An,p≥δ1+2τ2nklog⁡log⁡nk,i.o..
Set r>0 and let τ>0 satisfy δ/1+2τ>r. Thus
(29)PAn*-An,p≥δ2nlog⁡log⁡n,i.o.≤Pmax⁡n≤nkAn*-An,p≥r2nklog⁡log⁡nk,i.o..
Since |β1|<1, one has
(30)K1≔∑i=p+1+∞β1i-pi+1<∞.
Consequently,
(31)Pmax⁡n≤nkAn*-An,p>r2nklog⁡log⁡nk≤P∑i=p+1n-2β1i-pmax⁡n≤nk∑t=i+2nϵtϵt-i-1MMM>∑i=p+1n-2β1i-pi+11-β1r2nklog⁡log⁡nkK1β1p+1≤∑i=p+1n-2Pmax⁡n≤nk∑t=i+2nϵtϵt-i-1MMMMM>i+11-β1r2nklog⁡log⁡nkK1β1p+1.
For any positive natural numbers i, p, and s, define
(32)Ai+1nM=2qi+1+j:q∈N,1≤j≤i+1M∩1,…,n,Bi+1nM=2q+1i+1+j:q∈N,1≤j≤i+1M∩1,…,n.
It is easy to see that {ϵtϵt-i-1}t∈Ai+1(n) and {ϵtϵt-i-1}t∈Bi+1(n) are two sets of i.i.d. random variables. Let {ϵ~tϵ~t-i-1} be an i.i.d. sequence of random variables with the same law as ϵtϵt-i-1. We have
(33)Pmax⁡n≤nkAn*-An,p>r2nklog⁡log⁡nk≤∑i=p+1n-2Pmax⁡n≤nk∑t∈Ai+1nϵ~tϵ~t-i-1MMMMMM>i+11-β1r2nklog⁡log⁡nk2K1β1p+1∑t∈Ai+1nϵ~tϵ~t-i-1M+∑i=p+1n-2Pmax⁡n≤nk∑t∈Bi+1nϵ~tϵ~t-i-1MMMMMMM>i+11-β1r2nklog⁡log⁡nk2K1β1p+1∑t∈Bi+1nϵ~tϵ~t-i-1≤2∑i=p+1n-2Pmax⁡n≤nk∑t=i+2nϵ~tϵ~t-i-1MMMMMM>i+11-β1r2nklog⁡log⁡nk2K1β1p+1∑t∈Bi+1nϵ~tϵ~t-i-1.
Note that for any random variable X with mean zero, |m(X)|≤2Var⁡X. In fact,
(34)PX≥2+ϵVar⁡X≤2+ϵ-1<12.
By Lemma 3,
(35)Pmax⁡n≤nk∑t=i+2nϵ~tϵ~t-i-1MM>i+11-β1r2nklog⁡log⁡nk2K1β1p+1≤2P∑t=i+2nkϵ~tϵ~t-i-1MMM>i+11-β1r2nklog⁡log⁡nk2K1β1p+1-2nkσ4≤2P∑t=i+2nkϵ~tϵ~t-i-1MMM>i+11-β1r12nklog⁡log⁡nk2K1β1p+1,
where the second inequality holds for any fixed constant r1>0 and enough large nk. By Lemma 2, we further have
(36)Pmax⁡n≤nkAn*-An,p>r2nklog⁡log⁡nk≤4∑i=1∞exp⁡-i+11-β12r124K12β12p+2log⁡log⁡nk≤4exp⁡-21-β12r12/4K12β12p+2log⁡log⁡nk1-exp⁡-1-β12r12/4K12β12p+2log⁡log⁡nk~4exp⁡-1-β12r122K12β12p+2log⁡log⁡nk.
From (25), nk~(1+τ)k as k→∞. So
(37)Pmax⁡n≤nkAn*-An,p>r2nklog⁡log⁡nk≤41klog⁡1+τ1-β12r12/2K12β12p+2.
By virtue of |β1|<1 and p→+∞, we easily see that (1-|β1|2)r12/2K12|β1|2p+2>1. Thus
(38)∑k=1∞Pmax⁡n≤nkAn*-An,p>r2nklog⁡log⁡nk<+∞.
Applying Borel-Cantelli Lemma, we get (24). This completes the proof.Lemma 8.
LetBn(1)=∑t=1n‍Xt-1 and Bn(2)=∑t=1n‍Xt-12. Then
(39)lim⁡n→∞Bn1n=β01-β1,a.s.,lim⁡n→∞Bn2n=β021-β12+σ21-β12,a.s.Proof.
From (1), one can easily calculate
(40)Xt-1=β01-β1t-11-β1+β1t-1X0+∑i=1t-1β1t-1-iϵi,t≥2.
Consequently,
(41)Bn1M=X0∑t=1nβ1t-1+β0∑t=2n1-β1t-11-β1MM+ϵ11+β1+⋯+β1n-2MMM+ϵ21+β1+⋯+β1n-3+⋯+ϵn-1.
Now for any n≥k+1,
(42)Eϵk21+β1+⋯+β1n-k-12≤σ21-β12,
so the bracket of (41) divided by n converges a.e. to 0, thanks to Theorem 5.4.1 of Chung [12]. Furthermore,
(43)lim⁡n→∞X01+β1+⋯+β1n-1n=0,a.s.,lim⁡n→∞β0n∑k=2n1-β1k-11-β1=β01-β1.
The former equality of (39) is obtained.
As to the later one,Bn(2) can be written as
(44)β02∑t=2n1-β1t-11-β12+X02∑t=1nβ12t-1M+2X0β0∑t=2nβ1t-11-β1t-11-β1+2X0∑t=2nβ1t-1∑i=1t-1β1t-1-iϵiM+∑t=2n∑i=1t-1β1t-1-iϵi2.
Note that |β1|<1, one can see that
(45)lim⁡n→∞1nβ02∑t=2n1-β1t-11-β12=β021-β12,lim⁡n→∞1nX02∑t=1nβ12t-1+2X0β0∑t=2nβ1t-11-β1t-11-β1=0.
The next to last term of (44) equals 2X0∑i=1n-1ϵi∑j=0n-1-iβ1i+2j, which yields the fact that the mean is zero and variance is bounded by constant times n. Thus
(46)lim⁡n→∞2X0n∑t=2nβ1t-1∑i=1t-1β1t-1-iϵi=0.
The last step is to prove
(47)lim⁡n→∞1n∑t=2n∑i=1t-1β1t-1-iϵi2=σ21-β12,a.s.
By a simple calculation, one has
(48)∑t=2n∑i=1t-1β1t-1-iϵi2M=∑t=1n-1ϵt2∑i=0n-1-tβ12iMM+2∑j=1n-2ϵj∑i=j+1n-1ϵiβ1i-j1-β12n-i1-β12.
Note that {ϵt}t≥1 is a sequence of i.i.d. random variables and
(49)∑i=0n-1-tβ12i=1-β12n-t1-β12.
By virtue of the strong law of large numbers on independent random variables (see Chung [12]), one has
(50)lim⁡n→∞1n∑t=1n-1ϵt2∑i=0n-1-tβ12i=lim⁡n→∞σ2n∑t=1n-1∑i=0n-1-tβ12i=σ21-β12,a.s.
The mean of the third term in (48) is zero and the variance is less than
(51)4σ4∑j=1n-2∑i=j+1n-1β12i-2j1-β122≤4nσ4β121-β123,
which yields
(52)lim⁡n→∞2n∑j=1n-2ϵj∑i=j+1n-1ϵiβ1i-j1-β12n-i1-β12=0,a.s.
So the proof of the lemma is completed.


## 4. Proof of Theorem1

The proof is divided into four steps as follows.Step 1.
Let(53)An=∑t=1nϵtXt-1.
Then An can be calculated by
(54)An=ϵ1X0+∑t=2nϵtβ0+∑t=2nϵtϵt-1+∑t=2nβ1ϵtXt-2=⋯=∑i=0n-2∑t=i+2nβ1iϵtϵt-i-1M+β0∑i=0n-2∑t=i+2nβ1iϵt+X0∑t=1nβ1t-1ϵt≔An*+β0Bn*+X0∑t=1nβ1t-1ϵt.
It is easily seen that
(55)limsup⁡n→∞X0∑t=1nβ1t-1ϵt2nlog⁡log⁡n=0,a.s.
Now we come to prove
(56)limsup⁡n→∞An*2nlog⁡log⁡n=σ21-β12,a.s.
Indeed, set a nature number p and define
(57)An,pM=An,p=An*,ifp>n-2,∑i=0p∑t=i+2nβ1iϵtϵt-i-1,ifp≤n-2.
Note that An* is a linear combination of terms ϵt(j)ϵt-s(t(j)) and E[ϵt(j)ϵt-s(t(j))]=0 for any s<t; we easily obtain E[An*-An,p]=0. Furthermore, from Lemma 7,
(58)limsup⁡p→∞limsup⁡n→∞12nlog⁡log⁡nAn*-An,p=0,a.s.
Next, we let
(59)Ct,p*=∑t=p+2nCt,p=∑t=p+2n∑i=0pβ1iϵtϵt-i-1.
Then the limit distribution of Ct,p*/2nlog⁡log⁡n is the same as that of An,p/2nlog⁡log⁡n. Further,
(60)limsup⁡p→∞limsup⁡n→∞12nlog⁡log⁡nAn,p-Ct,p*=0,a.s.
From (58) and (60), combining Lemma 6, we get (56).Step 2.
We claim that(61)limsup⁡n→∞Bn*-∑t=1nXt-1∑t=1nϵtn2nlog⁡log⁡n=0,a.s.
Note that
(62)Bn*=∑i=0n-2∑t=i+2nβ1iϵt=∑t=2n1-β1t-11-β1ϵt,
and lim⁡n→∞∑t=1n‍Xt-1/n=β0/1-β1; we have
(63)limsup⁡n→∞Bn*-∑t=1nXt-1∑t=1nϵtn2nlog⁡log⁡n=limsup⁡n→∞∑t=2nβ0β1t-1/1-β1ϵt2nlog⁡log⁡n.
Since {ϵt}t≥1 is a sequence of i.i.d. random variables, one easily checks that the sequence {(β0β1t-1/(1-β1))ϵt} satisfies the conditions of Lemma 5. Thus, (61) is obtained.Step 3.
We claim that(64)limsup⁡n→∞-∑t=1nXt-1∑t=1nϵt+n∑t=1nϵtXt-1n2nlog⁡log⁡nM=σ21-β12,a.s.,(65)limsup⁡n→∞∑t=1nXt-12∑t=1nϵt-∑t=1nXt-1∑t=1nϵtXt-1n2nlog⁡log⁡nM=σ61-β122+β01-β12·σ41-β12,a.s.
By (54), (55), and (61), we easily get (64). By virtue of Lemma 8, (65) is equivalent to
(66)limsup⁡n→∞∑t=1nϵtXt-1β021-β12+σ21-β12MMMMM·∑t=1nϵt-β01-β1∑t=1nϵtXt-1MMM0M·2nlog⁡log⁡n-1∑t=1nϵtXt-1M=σ61-β122+β01-β12·σ41-β12,a.s.
From (63), we can see that(67)limsup⁡n→∞β02/1-β12∑t=1nϵt-β02/1-β1∑i=0n-2∑t=i+2nβ1iϵt2nlog⁡log⁡n=0,a.s.So from (58) and (60), (65) is also equivalent to(68)limsup⁡n→∞∑t=1nσ2/1-β12-β0/1-β1∑i=1p+1β1i-1ϵt-iϵt2nlog⁡log⁡nM=σ61-β122+β01-β12·σ41-β12,a.s.Similarly to the proof of Lemma 6, (68) can be easily derived.Step 4.
Since(69)n2log⁡log⁡nβ^0n-β0=∑t=1nXt-12∑t=1nϵt-∑t=1nXt-1∑t=1nϵtXt-1n2nlog⁡log⁡nM·∑t=1nXt-12n-∑t=1nXt-1n2-1,n2log⁡log⁡nβ^1n-β1=-∑t=1nXt-1∑t=1nϵt+n∑t=1nϵtXt-1n2nlog⁡log⁡nM·∑t=1nXt-12n-∑t=1nXt-1n2-1.
By Lemma 8, together with (64) and (65), we obtain (7). This completes the proof of the theorem.



---

*Source: 972712-2014-11-05.xml*