Chapter 1
Time Series Concepts
Updated: January 22, 2015.
This chapter reviews some basic times series concepts that are importantfor describing and modeling financial time series.

1.1

Stochastic Processes

A stochastic process{    1  2        +1    } = { }∞=−∞ 
is a sequence of random variables indexed by time 1  In most applications,the time index is a regularly spaced index representing calendar time (e.g.,days, months, years, etc.). In modeling time series data, the ordering imposed by the time index is important because we often would like to capture
the temporal relationships, if any, between the random variables in the stochastic process. In random sampling from a population, the ordering of therandom variables representing the sample does not matter because they are
independent.
A realization of a stochastic process with  observations is the sequenceof observed data
{1 = 1  2 = 2       =  } = { }=1 
1

To conserve on notation, we will often represent the stochastic process { }∞=−∞
simply as { }

1

2

CHAPTER 1 TIME SERIES CONCEPTS

The goal of time series modeling is to describe the probabilistic behaviorof the underlying stochastic process that is believed to have generated theobserved data in a concise way. In addition, we want to be able to use theobserved sample to estimate important characteristics of a time series modelsuch as measures of time dependence. In order to do this, we need to make anumber of assumptions regarding the joint behavior of the random variablesin the stochastic process such that we may treat the stochastic process inmuch the same way as we treat a random sample from a given population.

1.1.1

Stationary Stochastic Processes

We often describe random sampling from a population as a sequence of independent, and identically distributed (iid) random variables 1  2    suchthat each  is described by the same probability distribution   and write ∼   With time series data, we would like to preserve the identical distribution assumption but we do not want to impose the restriction that eachrandom variable in the sequence is independent of all of the other variables.
In many contexts, we would expect some dependence between random variables close together in time (e.g, 1  and 2 ) but little or no dependence
between random variables far apart in time (e.g., 1 and 100 ). We can allow
for this type of behavior using the concepts of stationarity and ergodicity.
We start with the definition of strict stationarity.
Definition 1 Strict stationarity
A stochastic process { }∞=−∞ is strictly stationary if, for any given finite
integer  and for any set of subscripts 1  2       the joint distribution of(1  2       ) depends only on 1 −  2 −       −  but not on  ¥In simple terms, the joint distribution of random variables in a strictly
stationary stochastic process is time invariant. For example, the joint distribution of (1  5  7 ) is the same as the distribution of (12  16  18 ) Just likein an iid sample, in a strictly stationary process all of the random variables ( = −∞     ∞) have the same marginal distribution   This means
they all have the same mean, variance etc., assuming these quantities exist
However, assuming strict stationarity does not make any assumption about
the correlations between   1       other than that the correlation between  and  only depends on  −  (the time between  and  ) and1.1 STOCHASTIC PROCESSES

3

not on . That is, strict stationarity allows for general temporal dependence
between the random variables in the stochastic process.
A useful property of strict stationarity is that it is preserved under generaltransformations, as summarized in the following proposition.
Proposition 2 let { }∞=−∞ be strictly stationary and let (·) be any func∞tion of the elements in { }∞=−∞  Then {( )}=−∞  is also strictly stationary.
2 ∞∞For example, if { }∞=−∞ is strictly stationary then { }=−∞ and { −1 }=−∞are also strictly stationary.

Example 3 iid sequence
If { }∞=−∞ is an iid sequence, then it is strictly stationary. ¥Example 4 Non iid sequence∞
Let { }∞=−∞ be an iid sequence and let  ∼ (0 1) independent of { }=−∞ ∞Define  =  +  The sequence { }=−∞ is not an independent sequence(because of the common ) but is an identically distributed sequence and isstrictly stationary. ¥If we assume that the stochastic process { }∞=−∞ is strictly stationary
and that [ ] var( ) and all pairwise covariances exist, then we say that{ }∞
=−∞ is a covariance stationary stochastic process.

Definition 5 Covariance stationarity
A stochastic process { }∞=1 is covariance stationary if
1. [ ] =  does not depend on 
2. var( ) =  2 does not depend on 
3. cov(  − ) =   exists, is finite, and depends only on  but not on 
for  = 0 1 2   

4

CHAPTER 1 TIME SERIES CONCEPTS

-2

-1

0

y(t)

1

2

3

Gaussian White Noise Process

0

50

100

150

200

250

time

Figure 1.1: Realization of a GWN(0,1) process.
The term   is called the j order autocovariance. The j order autocorrelation is defined as
cov(  − )= 2 = pvar( )var(− )(1.1)

The autocovariances,    measure the direction of linear dependence between  and −  The autocorrelations,   measure both the direction andstrength of linear dependence between  and − 
The autocovariances and autocorrelations are measures of the linear temporal dependence in a covariance stationary stochastic process. A graphicalsummary of this temporal dependence is given by the plot of  against ,
and is called the autocorrelation function (ACF).
Example 6 Gaussian White NoiseLet  ∼  (0  2 ) Then { }∞=−∞ is called a Gaussian white noise process1.1 STOCHASTIC PROCESSES

5

and is denoted  ∼ GWN(0 2 ). Notice that[ ] = 0 independent of var( ) =  2 independent of cov(  − ) = 0 (for   0) independent of  for all 
so that { }∞=−∞ is covariance stationary. Figure 1.1 shows a realizationof a GWN(0,1) process. The defining characteristic of a GWN process isthe lack of any predictable pattern over time in the realized values of theprocess as illustrated in 1.1. In the electrical engineering literature, white
noise represents the absence of any signal.
Figure 1.1was created with the R commands:> set.seed(123)> y = rnorm(250)> ts.plot(y, main="Gaussian White Noise Process", xlab="time",+
ylab="y(t)", col="blue", lwd=2)> abline(h=0)
The simulated iid N(0,1) values are generated using the rnorm() function.
The command set.seed(123) initializes R’s internal random number generator using the seed 123. Everytime the random number generator seed is
set to a particular value, the random number generator produces the same
set of random numbers. This allows diﬀerent people to create the same set of
random numbers so that results are reproducible. The function ts.plot()creates a time series line plot with a dummy time index. An equivalent plot
can be created using the generic plot() function:> plot(y, main="Gaussian White Noise Process", type="l", xlab="time",+
ylab="y(t)", col="blue", lwd=2)> abline(h=0)¥
Example 7 Gaussian White Noise Model for Continuously Compounded
Returns
Let  denote the continuously compounded monthly return on Microsoft
stock and assume that  ∼ GWN(001 (005)2 ) 60 Simulated values of { }are computed using

6

CHAPTER 1 TIME SERIES CONCEPTS

-0.05

0.00

r(t)

0.05

0.10

GWN Process for Monthly Continuously Compounded Returns

0

10

20

30

40

50

60

time

Figure 1.2: Simuluated returns from GWN(0.01,(0.05)2 )
> set.seed(123)> y = rnorm(60, mean=0.01, sd=0.05)> ts.plot(y,main="GWN Process for Monthly Continuously Compounded Returns",+
xlab="time",ylab="r(t)", col="blue", lwd=2, type="p")> abline(h=c(0,-0.05,0.05), lwd=2, lty=c("solid","dotted","dotted"),+
col=c("black", "red", "red"))and are illustrated in Figure 1.2. Notice that the returns fluctuate aroundthe mean value of 001 and the size of a typically deviation from the mean
is about 005. An implication of the GWN assumption for monthly returns
is that non-overlapping multiperiod returns are also GWN. For example,
consider the two-month return  (2) =  +−1  The non-overlapping process{ (2)} = { −2 (2)  (2) +2 (2) } is GWN with mean [ (2)] = 2 ·(001) = 002 and variance var( (2)) = 2 · (005)2 
Example 8 Independent White NoiseLet  ∼ iid (0  2 ) Then { }∞=−∞ is called an independent white noise1.1 STOCHASTIC PROCESSES

7

process and is denoted  ∼ IWN(0  2 ) The diﬀerence between GWN and
IWN is that with IWN we don’t specify that all random variables are normally distributed. The random variables can have any distribution withmean zero and variance  2 ¥
Example 9 Weak White Noise
Let { }∞=−∞ be a sequence of uncorrelated random variables each with mean
zero and variance  2  Then { }∞=−∞ is called a weak white noise process and2
is denoted  ∼ WN(0  ). With a weak white noise process, the randomvariables are not independent, only uncorrelated. This allows for potential
non-linear temporal dependence between the random variables in the process.
¥

1.1.2

Non-Stationary Processes

In a covariance stationary stochastic process it is assumed that the means,variances and autocovariances are independent of time. In a non-stationary
process, one or more of these assumptions is not true.
Example 10 Deterministically trending process
Suppose { }∞=0 is generated according to the deterministically trending
process =  0 +  1  +    ∼  (0 2 )
 = 0 1 2   Then { }∞=0 is nonstationary because the mean of  depends on  :[ ] =  0 +  1  depends on 
Figure 1.3 shows a realization of this process with  0 = 0  1 = 01 and 2 = 1 created using the R commands:> set.seed(123)> e = rnorm(250)> y.dt = 0.1*seq(1,250) + e> ts.plot(y.dt, lwd=2, col="blue", main="Deterministic Trend + Noise")> abline(a=0, b=0.1)

8

CHAPTER 1 TIME SERIES CONCEPTS

0

5

10

y.dt

15

20

25

Deterministic Trend + Noise

0

50

100

150

200

250

Time

Figure 1.3: Deterministically trending nonstationary process  = 01 +   ∼ (0 1)
Here the non-stationarity is created by the deterministic trend  0 +  1  inthe data. The non-stationary process { }∞=0 can be transformed into astationary process by simply subtracting oﬀ the trend: =  −  0 −  1  =  ∼ WN(0  2 )
The detrended process  ∼WN(0  2 ) ¥Example 11 Random walk
A random walk (RW) process { }∞=1 is defined as
 = −1 +    ∼ IWN(0 2 )
0 is fixed (non-random).

9

1.1 STOCHASTIC PROCESSES
By recursive substitution starting at  = 1 we have
1 = 0 + 1 
2 = 1 + 2 = 0 + 1 + 2 ..
.
 = 0 + 1 + · · · + X
= 0 + 
=1Now, [ ] = 0 which is independent of  However,
!
Ã 

X
X
 =
 2 =  2 × 
var( ) = var
=1

=1

which depends on and so { }∞=1 is not stationary.
Figure 1.4 shows a realization of the RW process with 0 = 0 and  2 = 1
created using the R commands:> set.seed(321)> e = rnorm(250)> y.rw = cumsum(e)> ts.plot(y.rw, lwd=2, col="blue", main="Random Walk")> abline(h=0)
Although { }∞=1 is non-stationary, a simple first-diﬀerencing transformation,
however, yields a covariance stationary process: =  − −1 =  ∼ IWN(0 2 )
¥Example 12 Random Walk Model for log Stock Prices
Let  denote the continuously compounded monthly return on Microsoft
stock and assume that  ∼ GWN(001 (005)2 ) Since  = ln( −1 ) it
follows that ln  = ln −1 +  and so ln  follows a random walk process.
Prices, however, do not follow a random walk since  = ln  = ln −1 + .

10

CHAPTER 1 TIME SERIES CONCEPTS

0

5

y.rw

10

15

Random Walk

0

50

100

150

200

250

Time

Figure 1.4: Random walk process:  = −1 +    ∼ (0 1)

1.1.3

Ergodicity

In a strictly stationary or covariance stationary stochastic process no assumption is made about the strength of dependence between random variables inthe sequence. For example, in a covariance stationary stochastic process
it is possible that 1 = cor(  −1 ) = 100 = cor(  −100 ) = 05 say.
However, in many contexts it is reasonable to assume that the strength ofdependence between random variables in a stochastic process diminishes thefarther apart they become. That is, 1  2 · · · and that eventually  = 0
for  large enough. This diminishing dependence assumption is captured by
the concept of ergodicity.
Definition 13 Ergodicity (intuitive definition)
Intuitively, a stochastic process { }∞=−∞ is ergodic if any two collections ofrandom variables partitioned far apart in the sequence are essentially independent. ¥1.2 MOVING AVERAGE PROCESSES

11

The formal definition of ergodicity is highly technical and requires advanced concepts in probability theory. However, the intuitive definition captures the essence of the concept. The stochastic process { }∞=−∞ is ergodic
if  and − are essentially independent if  is large enough.
If a stochastic process { }∞=−∞ is covariance stationary and ergodic thenstrong restrictions are placed on the joint behavior of the elements in thesequence and on the type of temporal dependence allowed.
Example 14 White noise processes
If { }∞=−∞ is GWN or IWN then it is both covariance stationary and ergodic.
Example 15 Covariance stationary but not ergodic process (White 1984,
pp. xxx)Let  ∼ GWN(0 1) and let  ∼ (0 1) independent of { }∞=−∞  Define∞ =  +  Then { }=−∞ is covariance stationary but not ergodic. To
see why { }∞=−∞ is not ergodic, note that for all   0
var( ) = var( + ) = 1 + 1 = 2  = cov( +  − + ) = cov(  − ) + cov(  ) + cov(−  ) + cov( )= cov( ) = var() = 11 = for all 
2Hence, the correlation between random variables separated far apart does
not eventually go to zero and so { }∞=−∞ cannot be ergodic. ¥The diﬀerent flavors of white noise processes are not very interesting because they do not allow any linear dependence between the observations inthe series. The following sections describe some simple covariance stationary and ergodic time series models that allow for diﬀerent patterns of timedependence captured by autocorrelations.

1.2

Moving Average Processes

Moving average models are simple covariance stationary and ergodic timeseries models that can capture a wide variety of autocorrelation patterns.

12

1.2.1

CHAPTER 1 TIME SERIES CONCEPTS

MA(1) Model

Suppose you want to create a covariance stationary and ergodic stochastic
process { }∞=−∞ in which  and −1 are correlated but  and − are notcorrelated for   1 That is, the time dependence in the process only lasts
for one period. Such a process can be created using the first order moving
average (MA(1)) model: =  +  + −1  − 1    1
 ∼  (0  2 )

(1.2)

The moving average parameter  determines the sign and magnitude of thecorrelation between  and −1  Clearly, if  = 0 then  =  +  so that{ }∞
=−∞ exhibits no time dependence.
To verify that (1.2) process is a covariance stationary process we mustshow that the mean, variance and autocovariances are time invariant. For
the mean, we have[ ] =  + [ ] + [−1 ] = because [ ] = [−1 ] = 0
For the variance, we havevar( ) =  2 = [( − )2 ] = [( + −1 )2 ]= [2 ] + 2[ −1 ] + 2 [2−1 ]=  2 + 0 + 2  2 =  2 (1 + 2 )
The term [ −1 ] = cov(  −1 ) = 0 because { }∞=−∞ is an independent
process.
For  1 = cov(  −1 ) we havecov(  −1 ) = [( − )(−1 − )]= [( + −1 )(−1 + −2 )]= [ −1 ] + [ −2 ]+[2−1 ] + 2 [−1 −2 ]= 0 + 0 +  2 + 0 =  2 Note that the sign of  1 is the same as the sign of  For 1 = cor(  −1 )we have

1
 2

1 = 2 = 2
2 =  (1 +  )(1 + 2 )1.2 MOVING AVERAGE PROCESSES

13

Clearly, 1 = 0 if  = 0; 1  0 if   0; 1  0 if   0 Also, the largest
value for |1 | is 05 which occurs when || = 1For  2 = cov(  −2 ) we havecov(  −2 ) = [( − )(−2 − )]= [( + −1 )(−2 + −3 )]= [ −2 ] + [ −3 ]+[−1 −2 ] + 2 [−1 −3 ]= 0 + 0 + 0 + 0 = 0Similar calculations can be used to show thatcov(  − ) =   = 0 for   1
Hence, for   1 we have  = 0 and there is only time dependence between
 and −1 but no time dependence between  and − for   1 Because
 = 0 for   1 the MA(1) process is ergodic.
Example 16 Simulated values from MA(1) process
The R function arima.sim() can be used to simulate observations froma MA(1) process2 . For example, to simulate 250 observations from (1.2) with
 = 1  = 09 and   = 1 use> ma1.model = list(ma=0.9)> mu = 1> set.seed(123)> ma1.sim = mu + arima.sim(model=ma1.model,n=250)
The ma component of the list object ma1.model specifies the value of  forthe MA(1) model, and is used as an input to the function arima.sim().
By default, arima.sim() sets  = 0 and specifies  ∼ GWN(0 1). Other
distributions for  can be specified using the optional innov argument. For
example, to set  ∼ GWN(0 (01)2 ) use> ma1.sim = mu + arima.sim(model=ma1.model,n=250,+
innov=rnorm(250,mean=0,sd=0.1))
2

The function arima.sim() can be used to simulate observations from the class ofautoregressive integrated moving average (ARIMA) models, of which the MA(1) model isa special case.

14

CHAPTER 1 TIME SERIES CONCEPTS

2
-2

0

y(t)

4

MA(1) Process: mu=1, theta=0.9

0

50

100

150

200

250

8

10

time

rho(j)

0.0

0.4

0.8

ACF for MA(1): theta=0.9

0

2

4

6
lag

Figure 1.5: Simulated values and theoretical ACF from MA(1) process with
 = 1  = 09 and  2 = 1The function ARMAacf() can be used to compute the theoretical autocorrelations,   from the MA(1) model (recall, 1 = (1 + 2 ) and  = 0 for  1) For example, to compute  for  = 1     10 use> ma1.acf = ARMAacf(ar=0, ma=0.9, lag.max=10)> ma1.acf
0
1
2
3
4
5
6
7
8
9
1.0000 0.4972 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
100.0000Figure 1.5 shows the simulated data and the theoretical ACF created using> par(mfrow=c(2,1))> ts.plot(ma1.sim,main="MA(1) Process: mu=1, theta=0.9",+ xlab="time",ylab="y(t)", col="blue", lwd=2)> abline(h=c(0,1))

1.2 MOVING AVERAGE PROCESSES

15> plot(0:10, ma1.acf,type="h", col="blue", lwd=2,+ main="ACF for MA(1): theta=0.9",xlab="lag",ylab="rho(j)")> abline(h=0)> par(mfrow=c(1,1)
Compared to the GWN process in 1.1, the MA(1) process is a bit smootherin its appearance. This is due to the positive one-period time dependence1 = 04972
¥Example 17 MA(1) model for overlapping continuously compounded returns
Let  denote the one-month continuously compounded return and assumethat ∼ iid ( 2 )
Consider creating a monthly time series of two-month continuously compounded returns using (2) =  + −1 
The time series of these two-month returns, observed monthly, overlap by
one month: (2) =  + −1 
−1 (2) = −1 + −2 
−2 (2) = −2 + −3 ..
.
The one-month overlap in the two-month returns implies that { (2)} follows
an MA(1) process. To show this, we need to show that the autocovariancesof { (2)} behave like the autocovariances of an MA(1) process.
To verify that { (2)} follows an MA(1) process, first we have[ (2)] = [ ] + [−1 ] = 2var( (2)) = ( + −1 ) = 2 2
Next, we havecov( (2) −1 (2)) = cov( +−1  −1 +−2 ) = cov(−1  −1 ) = var(−1 ) =  2

16

CHAPTER 1 TIME SERIES CONCEPTS

andcov( (2) −2 (2)) = cov( + −1  −2 + −3 ) = 0
cov( (2) − (2)) = 0 for   1
Hence, the autocovariances of { (2)} are those of an MA(1) process. Notice
that
2
1
1 = 2 = 22What MA(1) process describes { (2)}? Because 1 = 1+
2 = 05 it follows
that  = 1 Hence, the MA(1) process has mean 2 and  = 1 and can be
expressed as (2) = 2 +  + −1  ∼  (0 2 )

1.3

Autoregressive Processes

1.3.1

AR(1) Model

Suppose you want to create a covariance stationary and ergodic stochastic process { }∞=−∞ in which  and −1 are correlated,  and −2 areslightly less correlated,  and −3 are even less correlated and eventually and − are uncorrelated for  large enough. That is, the time dependence
in the process decays to zero as the random variables in the process get farther and farther apart. Such a process can be created using the first order
autoregressive (AR(1)) model: −  = (−1 − ) +   − 1    1
 ∼ iid (0  2 )(1.3)

It can be shown that the AR(1) model is covariance stationary and ergodic
provided −1    1 We will show that the AR(1) process has the following

1.3 AUTOREGRESSIVE PROCESSES

17

properties:[ ] = var( ) =  2 =  2 (1 − 2 )
cov(  −1 ) =  1 =  2 cor(  −1 ) = 1 =  1  2 = cov(  − ) =   =  2  

(1.4)
(1.5)
(1.6)
(1.7)
(1.8)

cor(  − ) =  =    2 =  

(1.9)

Notice that the restriction ||  1 implies thatlim  =  = 0→∞

so that  is essentially independent of − for large  and so { }∞=−∞ isergodic For example, if  = 05 then 10 = (05)10 = 0001; if  = 09 then
10 = (09)10 = 0349. Hence, the closer  is to unity the stronger is the timedependence in the process.
Verifying covariance stationarity for the AR(1) model is more involved
than for the MA(1) model, and establishing the properties (1.4) - (1.9) involves some tricks. First, consider the derivation for (1.4). We have[ ] =  + ([−1 ] − ) + [ ]=  + [−1 ] − If we assume that { }∞=−∞ is covariance stationary then [ ] = [−1 ]
Substituting into the above and solving for [ ] gives (1.4). A similar trick
can be used to derive (1.5):
var( ) = 2 (var(−1 )) + var( ) = 2 (var( )) +  2 which uses the fact that −1 is independent of   and var( ) = var(−1 )provided { } is covariance stationary. Solving for  2 = var( ) gives (1.5).
To determine (1.6), multiply both sides of (1.3) by −1 −  and take expectations to give£¤ 1 =  [( − ) (−1 − )] =  (−1 − )2 +  [ (−1 − )] =  2 

which uses the fact that −1 is independent of   and var( ) = var(−1 ) = 2  Finally, to determine (1.8), multiply both sides of (1.3) by − −  and

18

CHAPTER 1 TIME SERIES CONCEPTS

take expectations to give  =  [( − ) (− − )] =  [(−1 − )(− − )] +  [ (− − )]=  −1 which uses the fact that − is independent of   and  [(−1 − )(− − )] = −1 provided { } is covariance stationary. Using recursive substitution and 0 =  2 gives (1.8).
The AR(1) Model can be re-expressed in the form of a linear regression
model as follows: −  = (−1 − ) +  ⇒ =  −  + −1 + =  + −1 +  
where  = (1−) ⇒  = (1−) This regression model form is convenient
for estimation by ordinary least squares.
Example 18 Simulated values from AR(1) process
The R function arima.sim() can be used to simulate observations from an
AR(1) process, and the function ARMAacf() can be used to compute thetheoretical ACF. For example, to simulate 250 observations from (1.3) with
 = 1  = 09 and   = 1, and compute the theoretical ACF use> ar1.model = list(ar=0.9)> mu = 1> set.seed(123)> ar1.sim = mu + arima.sim(model=ar1.model,n=250)> ar1.acf = ARMAacf(ar=0.9, ma=0, lag.max=10)
These values are shown in Figure 1.6. Compared to the MA(1) process in
1.5, the realizations from the AR(1) process are much smoother. That is,when  wanders high above its mean it tends to stay above the mean for awhile and when it wanders low below the mean it tends to stay below for awhile.

19

1.3 AUTOREGRESSIVE PROCESSES

0 2 4 6-4y(t)AR(1) Process: mu=1, phi=0.9

0

50

100

150

200

250

8

10

time

0.8
0.6
0.4

rho(j)

1.0

ACF for AR(1): phi=0.9

0

2

4

6
lag

Figure 1.6: Simulated values and ACF from AR(1) model with  = 1  = 09and  2 = 1