Maximum Likelihood Estimation (MLE) and the Fisher Information | by Xichu Zhang | Towards Data Science

The number of articles on Medium about MLE is enormous, from theory to implementation in different languages. About the Fisher information, there are also quite a few tutorials. However, the connection between the Fisher information and MLE is rarely mentioned. Therefore, I’d like to contribute one post on this topic.
The main idea of MLE is simple. It answers this question:
What parameter will most likely make the model produce the sample we have?
Formally, we consider a sequence of random variables X1, ..., Xn, such that they are identical independently distributed (iid) random variables. All of them come from the same distribution f(x; Θ), where Θ is a vector of parameters (we use this big theta to denote a vector of parameters, which means Θ ∈ Rp, if the model has only one parameter, we will use θ to denote it in this post) and Θ ∈ Ω, where Ω is the sample space of the parameters. This sounds strange, but in MLE, we pick the parameters from sample space, and we want the most likely one. How do we do this? We maximize a likelihood function, which is defined as
The probability of each event can be multiplied together because we know that those observations are independent. In Eq 1.1, each Aj is an event, which can be an interval or a set containing a single point. When there is censoring at a particular value u, the observed event Aj is an interval [u, ∞). Without censoring, Aj contains a single point, and the likelihood of this observation given parameter Θ is
not concerning censoring, the likelihood function can be written as
which we usually see in textbooks.
What do we want to do with L? We want to find out at which value of Θ, L is maximized. For example, if L can be maximized analytically and it is concave, we can calculate the derivative with regard to Θ and let it be zero. At this point, the value of L will be both global and local maximum. Let’s look at an example of multivariate data with normal distribution.
consider the random variable X = (X1, X2, ..., Xn), with mean μ = (μ1, μ2, ..., μn); we assume that the standard variance is a constant σ, this property is also known as the homoscedasticity. The likelihood function is therefore
The indices look a bit confusing, but think about the fact that each observation is arranged into the columns of the matrix X. Eq 1.3 is actually pretty straightforward. xij is just the ith component in the jth observation. And Eq[ex1] is used to estimate each μi. We know that logarithm can turn production into summation, and usually, the summation is easier to deal with. Therefore, we can try to take the logarithm on both sides of Eq 1.3.
(It should be obvious that log refers to the natural logarithm) The rest is easy; we need to do some algebraic manipulation to Eq 1.4. And since σ is a constant, we can factor it out; then we arrive at
Remember that we want to maximize L, which is equivalent to maximizing Eq 1.5 since log increases monotonically. And this is boils downing to minimize the following expression
If you are familiar with ordinary linear models, this should remind you of the least square method. We can see that the least square method is the same as the MLE under the assumption of normality (the error terms have normal distribution).
For the maximum likelihood estimation in practical use, we look at the following example: a dataset of the number of awards earned by students at one high school (available here). If we plot the number of awards
awards <- read.csv2(file='data/Awards_R.csv', header=TRUE, sep=',')summary(awards)awards.num <- awards$num_awardsplot(table(awards.num), main='Awards in math', ylab='Frequency', xlab='Number of awards')
we can see from the graph that it follows a Poisson distribution
The maximum likelihood function of Poisson distribution is defined as
We can implement this as follows
L=function(x){    a=1    for (i in 1:length(awards.num)){        # Poisson probability mass function        a=a*dpois(awards.num[i],x)    }    return(a)}
and optimize this function
# find the value for which L is maximizedsol <- optimize(L, c(0,2), maximum=TRUE)curve(L, from=0, to = 2)x1 <- sol$maximumx2 <- sol$objectivepoints(x1, x2, col="#FF0000", pch=19)abline(v=x1, col="#FF0000")
The interval on which the maximized value is searched is chosen based on the graph of L (see Fig 1.8). The function is maximized numerically by optimize, which searches for the optimized value in the given interval (with a predetermined precision). This gives us
$maximum[1] 0.970013$objective[1] 1.853119e-113
which means the maximum value is 1.853119e-113 and L(0.970013) = 1.853119e-113 — λ = 0.970013 is the optimized parameter. As shown in the graph
The result shows that the sample mean and the value which optimizes L is very close
mean(awards.num)# --> 0.97# sol$maximum = 0.970013
This makes sense since the parameter λ in Poisson distribution is the same as the expected value. To prove this formally, we can take the derivative of the loglikelihood function
setting this derivative to zero, we acquire
where λ with hat denotes the estimator. This tells us, in this example, the maximum likelihood estimator is given by the sample mean.
Suppose the random variable X comes from a distribution f with parameter Θ The Fisher information measures the amount of information about Θ carried by X. Why is this quantification important? It turns out that in both Bayesian and frequentist approaches of statistics, Fisher information is applied. Bayesians use this to find the initial probability, and frequentists construct the confidence interval in MLE. (To read more about the Bayesian and frequentist approach, see here) A concrete example of the importance of Fisher information is talked about in [2]:
The example is tossing a coin ten times in a row, the observation is thus a 10-dimensional array, a possible result looks like X = (1, 1, 1, 1, 1, 0, 0, 0, 0, 0). We want to estimate the probability of getting a head, θ. There are 210 = 1024 possible outcomes of X. We can let another random variable, T be the number of heads in X. T is a function of X, and it is known as a statistic. In some literature, the statistic is described as “a piece of information.” This is true, but to be more precise, it is a function of the observations (the dataset), and it summarizes the data.
In this example, T has the binomial distribution, which is given by the probability density function
In this example, n = 10. X has 1024 possible outcomes, yet T can take only 11 different values. Does it mean that X contains more information than T? Generally speaking, yes, since X takes the order of the coin tossing into account, but T doesn’t. But with regard to θ, no, since the order of the output of the coin-tossing does not influence θ. Also, the statistic T is sufficient, which means given the value of T, the probability of X taking on a particular value doesn’t depend on θ. This means, the conditional probability distribution P(X | T = t, θ) is uniform and is given by
This can also be interpreted in this way: given the value of T, there’s no more information about θ left in X. To quantify the information about the parameter θ in a statistic T and the raw data X, the Fisher information comes into play
where Ω denotes sample space. In case of continuous distribution
the partial derivative of log f(x|θ) is called the score function. We can see that the Fisher information is the variance of the score function. If there are multiple parameters, we have the Fisher information in matrix form with elements
This can also be written as
The equivalence between Def 2.4 and Equation 2.5 is not trivial. This is an important property of Fisher information, and we will prove the one-dimensional case (θ is a single parameter) right now: let’s start with the identity:
which is just the integration of density function f(x;θ) with θ being the parameter. Note that there is a slight difference between f(x|θ) and f(x;θ). The first one denotes a conditional probability — the probability distribution function is under the condition of a given parameter. Yet, the latter means that θ is the parameter of the function, nothing more. Then we take the derivative with regard to θ on both sides.
Now we need to try to make log appear. The trick is as follows
In equation 2.7, we use the multiply by one technique (“multiply by one”, “plus zero” — famous tricks in math), which means we multiply by f(x;θ) and then divide by f(x;θ). The combination of the red parts in Equation 2.8 gives us the derivative of the logarithm of f(x;θ).
Equation 2.9 gives us another important property of Fisher information — the expectation of Fisher information equals zero. (It’s a side note, this property is not used in this post) Get back to the proof of the equivalence between Def 2.4 and Equation 2.5. We retake the derivative of Eq 2.9, with regard to θ
since in Equation 2.10, both the gray and black parts are positive (f(x; θ) is probability measure after all), the only possible scenario is Equation (2.11). Starting from Equation 2.11, we move f(x; θ) from the LHS (left-hand side) to the RHS (right-hand side)
Just like in Equation 2.8, in Equation 2.12, the combination of the red parts again gives us the derivative of the logarithm of f(x; θ).
And the proof is done.
Firstly, we are going to introduce the theorem of the asymptotic distribution of MLE, which tells us the asymptotic distribution of the estimator:
Let X1, ..., Xn be a sample of size n from a distribution given by f(x) with unknown parameter θ. Let the true parameter be θ0, and the MLE of θ0 be θhat, then
which indicates
Since when the sample size approaches infinity, the MLE approaches the true parameter, which is also known as the consistency property of the MLE
we can also argue that Equation 2.8 is also true (refer to Equation 2.5).
(For proof of this theorem, see here, page 5.) Then we can establish the confidence interval from the following
where z is the inverse of the cumulative function, and α is the critical value. The next thing is to find the Fisher information matrix. This is easy since, according to Equation 2,5 and the definition of Hessian, the negative Hessian of the loglikelihood function is the thing we are looking for.
You might question why is the Fisher information matrix in Eq 2.5 the same as the Hessian, though it is an expected value? It’s because the function L is evaluated at the maximum likelihood estimation, which means L is evaluated for a particular θ. And this is the θ that maximizes L. Therefore, the weighted average (we know that the expectation is a weighted average) is not necessary anymore — the observed Fisher information is just the second-order differentiation. “Observed” means that the Fisher information is a function of the observed data. (This topic is also discussed on MathStackExchange).
And we can find the confidence interval using the following code, using the same dataset.
L.log = function(x){    a=1    for (i in 1:length(awards.num)){        # Poisson probability mass function        a=a+dpois(awards.num[i],x, log=TRUE)    }    return(a)}# numerical approachopt.log = optim(par=1, L.log, method="Brent", control=list(fnscale=-1), hessian=TRUE, lower=0, upper=2)opt.logI.log <- opt.log$hessian# since we have only one parameter, there's no inverse of matrix calculatedest.log <- qnorm(1 - alpha/2) / sqrt(n * (-I.log[1,1]))l.est.log <- x1 - est.logh.est.log <- x1 + est.logl.est.logh.est.log# --> CI = [0.9603613, 0.9796647]
Summary:
In this post, the maximum likelihood estimation is quickly introduced, then we look at the Fisher information along with its matrix form. With those two concepts in mind, we then explore how the confidence interval is constructed. Throughout this post, a single example is used: the number of awards in a high school. The analysis is completely implemented in R.
Resources:
[1] Altham, P. M. E. (2005). Introduction to Generalized Linear Modelling in R. Statistical laboratory, giugno.
[2] Ly, A., Marsman, M., Verhagen, J., Grasman, R. P., & Wagenmakers, E. J. (2017). A tutorial on Fisher information. Journal of Mathematical Psychology, 80, 40–55.
[3] Taboga, Marco (2017). “Poisson distribution — Maximum Likelihood Estimation”, Lectures on probability theory and mathematical statistics, Third edition. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/Poisson-distribution-maximum-likelihood.
[4] Klugman, S. A., Panjer, H. H., & Willmot, G. E. (2012). Loss models: from data to decisions (Vol. 715). John Wiley & Sons.
[5] What is censored data? Available at: https://reliability.readthedocs.io/en/latest/What%20is%20censored%20data.html. Accessed on 13 October 2021
Error log
adding the explanation of the notation in Eq 1.1
adding the explanation of the notation in Eq 1.1