The Use of NAG Optimisation Routines forParameter Estimation
Geoff Morgan, NAG Ltd

The NAG Fortran Library contains an increasing number of routines to fit statistical models.
However, for more specialist examples the user has to use the optimisation routines in theNAG E04 chapter. The statistical model will give the probability of observing a data value interms of a set of parameters. The values of the parameters have to be estimated from thedata in such a way that the model best fits the data. For example if we assume that theobservations have a Poisson distribution then the probability of observing the value y iswhere the parameter m is the mean of the distribution. The mean may be related to anotherobserved variable, say x, through the model:

Given a number of observations of y and x we need to find estimates of the parameters b0
and b1. Such a model could be fitted using the NAG routine for generalised linear models
with Poisson errors, G02GCF. However, if the model werethe user would have to fit the model using an E04 routine.
The criterion often used to find estimators of the parameters of a statistical model ismaximum-likelihood. In maximum-likelihood estimation the parameters are chosen so that theobserved data are the most likely outcome. To find such estimators first the likelihood function
has to be found. The likelihood is the probability distribution for the sample considered as afunction of the parameters rather than as a function of the random variable. In the case of asample of independent observations the likelihood is the product of the probability distribution
for each observation. For example, for the Poisson distribution the likelihood isIn many cases the log-likelihood is a simpler function, in the Poisson example

where const is independent of the parameters. In the case of an independent sample from theNormal distribution, the log-likelihood is a sum of squares function and so the maximumlikelihood estimators are the same as the least-squares estimators.
Having found the likelihood or log-likelihood function it can be maximised using anappropriate NAG E04 routine to obtain the parameter estimations. In the case of the Normal
distribution there are specialist non-linear least-squares routines which should be used.
In addition to the parameter estimates the user will generally require the variance-covariancematrix of the parameter estimates. These can be used to test hypotheses about theparameters or to set confidence intervals for the parameters. It is also advisable to examine
the correlation between the parameter estimates. High correlations indicate that either adifferent parameterisation should be considered or, if that is not possible, inferences should
only be made jointly for the correlated parameters. For maximum-likelihood estimators thathave been found at an interior maximum, it can be shown that for large samples the variancecovariance matrix can be approximated by minus the inverse of the expected value of thematrix of second derivatives of the log-likelihood. However, as in many cases the expectedvalues of the derivatives are difficult to compute and the observed values are used instead, ie,
minus the inverse of the Hessian matrix.
Many of the NAG E04 routines return approximations to the Hessian matrix, or to theCholesky factors of the Hessian matrix and many users are misled into attempting to use thisinformation for the variance-covariance calculations. In fact the information returned is fine in
the context of restarting the optimisation process but is generally not good enough for thestatistical calculation. This is particularly so when the routine returns a non-zero value for
IFAIL, such as IFAIL=3 for E04WDF, in which case the routine usually returns with the unit
matrix as an approximation to the Hessian.
Instead the user is recommended to use E04XAF which has been included in the Library togive a more reliable means of estimating the Hessian matrix. The user should also note thatE04YCF has been included to provide estimates of the variance-covariance matrix following
the use of one of the unconstrained non-linear least squares routines.

© The Numerical Algorithms Group Limited
Tel: +44 (0)1865 511245
Email: sales@nag.co.uk