Introduction to Maximum Likelihood
Estimation

Eric Zivot

July 26, 2012

The Likelihood Function
Let 1      be an iid sample with pdf  (; ) where  is a ( × 1) vectorof parameters that characterize  (; )
Example: Let ˜( 2) thenµ1 (; ) = (2 2)−12 exp − 2 ( − )22 = (  2)0¶The joint density of the sample is, by independence, equal to the product ofthe marginal densities (1     ; ) =  (1; ) · · ·  (; ) =Y (; )

=1

The joint density is an  dimensional function of the data 1      given theparameter vector  and satisfies
Z···Z

 (1     ; ) ≥ 0
 (1     ; )1 · · ·  = 1The likelihood function is defined as the joint density treated as a function ofthe parameters  :(|1     ) =  (1     ; ) =Y (; )

=1

Notice that the likelihood function is a  dimensional function of  given thedata 1     
It is important to keep in mind that the likelihood function, being a functionof  and not the data, is not a proper pdf. It is always positive but
Z···Z

(|1     )1 · · ·  6= 1

To simplify notation, let the vector x = (1     ) denote the observed
sample. Then the joint pdf and likelihood function may be expressed as  (x; )and (|x) respectively.

Example 1 Bernoulli SamplingLet ˜ Bernoulli() That is,The pdf for  is = 1 with probability  = 0 with probability 1 −  (; ) =  (1 − )1−   = 0 1Let 1      be an iid sample with ˜ Bernoulli() The joint density /likelihood function is given by (x; ) = (|x) =Y

=1 (1 − )1− = PP−=1  (1 − )
=1 Since  is a discrete random variable (x; ) = Pr(1 = 1      = )Example 2 Normal SamplingLet 1      be an iid sample with ˜( 2) The pdf for  isµ¶1 (; ) = (22)−12 exp − 2 ( − )2 2 = (  2)0−∞    ∞  2  0 − ∞    ∞The likelihood function is given byYµ1(2 2)−12 exp − 2 ( − )2(|x) =2
=1⎛12−2⎝= (2 )exp −X¶⎞( − )2⎠22 =1

The Maximum Likelihood Estimator
Suppose we have a random sample from the pdf  (; ) and we are interested
in estimating 
The maximum likelihood estimator, denoted ̂ is the value of  that maximizes (|x) That is,̂ = arg max (|x)Alternatively, we say that ̂ solvesmax (|x)It is often quite diﬃcult to directly maximize (|x) It usually much easier
to maximize the log-likelihood function ln (|x) Since ln(·) is a monotonicfunction̂ = arg max ln (|x)With random sampling, the log-likelihood has the particularly simple form⎛ln (|x) = ln ⎝Y=1⎞ (; )⎠ =X

=1ln  (; )Example 3 Bernoulli example continued
Given the likelihood function
(|x) = PP−
=1  (1 − )
=1  

the log-likelihood is³ PP´−
=1 ln (|x) = ln  =1  (1 − )
⎛⎞
⎛⎞


X
X= ⎝⎠ ln() + ⎝ −⎠ ln(1 − )=1
=1Recall the resultsÃ !

ln( · ) = ln() + ln() ln


= ln() − ln() ln( ) =  ln()Example 4 Normal example continued

Given the likelihood function⎛ln (|x) = (2 2)−2 exp ⎝−X⎞

12⎠(−)22 =1the log-likelihood isX
1( − )2ln (|x) = − ln(2) − ln( 2) − 2222 =1

Recall the resultln() = Since the MLE is defined as the maximization problem, we can use the tools ofcalculus to determine its value. That is, we may find the MLE by diﬀerentiatingln (|x) and solving the first order conditions
 ln (̂|x)=0
Since  is ( × 1) the first order conditions define , potentially nonlinear,equations in  unknown values:⎛⎞ ln (̂|x)⎜⎟1⎟
 ln (̂|x) ⎜.
⎜
⎟=0.
=⎜⎟
θ
⎝  ln (̂ |x) ⎠

Review of Optimization Techniques: Unconstrained Optimization
Example: finding the minimum of a univariate function =  () = 2min  =  ()First order conditions for a minimum
 ()
=2 = 2 · ⇒=00=

Second order conditions for a minimum2 ()0
=2·=22• R function optimize()— Use to optimize (maximize or minimize) functions of one variable• Excel solver— General optimizer for unconstrained and constrained optimization problems involving many variables— solver in Oﬃce 2010 is substantially improved and expandedExample: Finding the minimum of a bivariate function =  ( ) = 2 +  2min  =  ( )
First order conditions for a minimum´ ³ 2
 ( )2
 + =2·=0=´ ( )
 ³ 2
20=
 + =2·=⇒  = 0  = 0

Remark:
Second order conditions depend on the properties of the second derivative
Hessian matrix⎡

 2 () 2 ()



 2⎤

 2 ( ) ⎢ 2⎥
( ) == ⎣  2 ()  2⎦ ()• R functions nlminb(), optim()— Use to optimize (maximize or minimize) functions of one or more variables variable— nlminb() uses Newton’s method based on 1st and 2nd derivatives andcan allow for box constraints on parameters— optim() can use 4 types of algorithms (secant method, Newton method,simplex method, simulated annealing)• Excel solver

Example 5 Bernoulli example continued

To find the MLE for  we maximize the log-likelihood functionln (|x) =X

=1⎛

 ln() + ⎝ −X

=1

The derivative of the log-likelihood is
1X

⎛

⎞

⎠ ln(1 − )X⎞ ln (|x)1 ⎝ −−⎠ = 0= =11−
=1The MLE satisfies

 ln (|x)= 0 and solving for  gives1X
̂ = =1

Example 6 Normal example continued
To find the MLE for  = (  2)0 we maximize the log-likelihood function1 X

2ln (|x) = − ln(2) − ln( ) − 2( − )2
222 =1

The derivative of the log-likelihood is a (2 × 1) vector given by⎛⎞ ln (|x) ln (|x) ⎝⎠=  ln x)(|
2

where ln (|x)
1 X( − )
= 2 =1
 ln (|x) 2 −1 1 2 −2 X2=−)+)(−)(( 222
=1Solving

 ln (|x)= 0 gives the normal equations
 ln (̂|x)1 X( − ̂) = 0= 2
̂  =1

 2 −1
 ln (̂|x)=−(̂)2
2

1 2 −2 X+ (̂ )( − ̂)2 = 02
=1

Solving the first equation for ̂ gives1X̂ =
 = ̄
 =1Solving the second equation for ̂ 2 gives
X1
̂2 =( − ̂)2 =1

Notice that ̂2 is not equal to the sample variance.

Invariance Property of Maximum Likelihood Estimators
One of the attractive features of the method of maximum likelihood is its
invariance to one-to-one transformations of the parameters of the log-likelihood.
That is, if ̂ is the MLE of  and  = () is a one-to-one function of then ̂ = (̂) is the mle for 

Example 7 Normal Model Continued
The log-likelihood is parameterized in terms of  and 2 and̂ = ̄X1̂ 2 =( − )2 =1
Suppose we are interested in the MLE for = ( 2) = (2)12
which is a one-to-one function for 2  0
The invariance property says that⎛1̂  = (̂ 2)12 = ⎝X( − ̂ =1⎞12)2⎠

The Precision of the Maximum Likelihood Estimator
Intuitively, the precision of ̂ depends on the curvature of the log-likelihood
function near ̂
If the log-likelihood is very curved or “steep” around ̂ then  will be
precisely estimated. In this case, we say that we have a lot of information
about If the log-likelihood is not curved or “flat” near ̂ then  will not be precisely
estimated. Accordingly, we say that we do not have much information aboutIf the log-likelihood is completely flat in  then the sample contains no information about the true value of  because every value of  produces the same valueof the likelihood function. When this happens we say that  is not identified.

The curvature of the log-likelihood is measured by its second derivative matrix(Hessian)⎡ 2 ln (|x)11 2 ln (|x) ⎢⎢..
=(|x) =⎢⎣ 20 ln (|x) 1···...
···⎤
 2 ln (|x)1⎥⎥..
⎥⎦ 2 ln (|x) Since the Hessian is negative semi-definite, the information in the sample about may be measured by −(|x) If  is a scalar then −(|x) is a positivenumber.
The expected amount of information in the sample about the parameter  isthe information matrix (|x) = −[(|x)]
As we shall see, the Hessian and information matrix are directly related to theprecision of the MLE.

Asymptotic Properties of Maximum Likelihood Estimators
Let 1      be an iid sample with probability density function (pdf)  (; )
where  is a ( × 1) vector of parameters that characterize  (; )
Under general regularity conditions, the ML estimator of  is consistent andasymptotically normally distributed. That is,̂ →  as  → ∞and for  large enough the Central Limit Theorem gives̂ ∼ ( (|x)−1)

Computing MLEs in R: the maxLik package
The R package maxLik has the function maxLik() for computing MLEs for anyuser-defined log-likelihood function• uses the optim() function for maximizing the log-likelihood function• Automatically computes standard errors by inverting the Hessian matrix

Remarks• In practice we don’t know (|x) = −[(|x)] but we can estimate
its value using −(̂|x) Hence, the practically useful asymptotic normality result is̂ ∼ ( −(̂|x)−1)• Estimated standard errors for the elements of ̂ are the square roots ofthe diagonal elements of −(̂|x)−1 :d ̂(rh|x)−1i−(̂) =hi−(̂|x)−1= ( ) element of − (̂|x)−1

Optimality Properties of MLE (or why we care about MLE)

• Recall, a good estimator ̂ has small bias and high precision (small (̂))• The best estimator among all possible estimators has the smallest bias andsmallest (̂)• In many cases, it can be shown that maximum likelihood estimator is thebest estimator among all possible estimators (especially for large samplesizes)

MLE of the CER Model Parameters
Recall, the CER model matrix notation isr = μ + ε ε ∼  (0 Σ)⇒ r ∼  (μ Σ)Given an iid sample r = {r1     r}  the likelihood and log-likelihood functions for θ = (μ Σ) are(θ|r) = (2)−2|Σ|−2 exp

⎧
⎨⎫
⎬


1X−(r − μ)0Σ−1(r − μ)
⎩ 2
⎭
=1


1Xln (θ|r) = − ln(2) − ln |Σ| −(r − μ)0Σ−1(r − μ)222 =1

It can be shown that the MLEs for the elements of μ and Σ are
1 X̂ =  = 1     
 =1

X
1
̂ 2 =( − ̂)2  = 1      =1̂  =r̂ 2  = 1     
1 X( − ̂)( − ̂ )   = 1     ̂  = =1̂̂ =   = 1     ̂  · ̂ 

Remarks• The MLEs for  and  are the same as the plug-in principle estimates• The MLEs for 2    and   are almost equal to the plug-in principle
1 )estimates. They diﬀer by a degrees of freedom adjustment ( 1 vs.  −1
• The plug-in estimates for  2 and   are unbiased; the MLEs have a tiny
bias that disappears in large samples.

• The formulas for the standard errors of the plug-in principle estimates comefrom the formulas for the standard errors of the MLEs