# Generalised Score and Wald Tests

Authors: Paul Rippon; J. C. W. Rayner
Journal: Advances in Decision Sciences
 (2010)

---

## Abstract

The generalised score and Wald tests are described and related to their nongeneralised versions. Two interesting applications are discussed. In the first a new test for the Behrens-Fisher problem is derived. The second is testing homogeneity of variances from multiple univariate normal populations.


---

## Body

## 1. Introduction

This paper is intended to be a tutorial for those wishing to inform themselves about the generalised score and Wald Tests. It extends the content of [1] and has similar objectives; that is, it focuses on the use of these tests rather than their properties. It is intended to be very accessible. Readers need only some prior knowledge of partitioned matrices, score and Wald tests, see, for example, [1] and [2, Chapter 3].The score test is particularly valuable when maximum likelihood (ML) estimation under the full model is not preferred, but ML estimation under the null model is. The converse holds for the Wald test. Thus when ML estimation under one of the null and full models is not preferred, the likelihood ratio test is problematic, but one of the score and Wald tests is not. Here by not preferred we mean that, for example, estimates may be calculated by some iterative scheme with dubious convergence. Other possibilities are that estimates may have a particularly convoluted expression or the finite sample properties (such as large bias) may be inappropriate for the problem of interest.When ML estimation under both the null and full models is not preferred, we need another way forward. This is provided by the generalised score and Wald Tests. These tests are especially valuable when the model may be misspecified, but that will not be the focus here.In Section2 the generalised score and Wald Tests are described. In Section 3 this material is applied to deriving a new test for the Behrens-Fisher problem, while Section 4 looks at testing equality of variances from multiple independent normal samples.


## 2. M-Estimators and Generalised Score Tests

The class of M-estimators includes both ML and method of moments estimators. An M-estimatorγ̃ satisfies
(2.1)∑j=1nΨ(Xj,γ̃)=0p,in whichX1,…,Xn are independent but not necessarily identically distributed, Ψ  is a known p×1 function not depending on j or n, γ is a p-dimensional parameter, and in general 0m denotes an m×1 vector of zeros. The estimating function Ψ must be sufficiently `smooth.’ In particular, its derivatives up to second order, and their expectations, must exist. Hence the matrices A and B defined subsequently are assumed to exist. Also, the expectation of the second-order derivatives must be bounded in probability. More technical details on M-estimators may be found in [3, Chapter 5].In our setting we assume thatγ=(θT,βT)T and that we wish to test H0: θ=0k against the alternative K: θ≠0k with θ being the k×1 vector of primary interest, with β a q×1 vector of nuisance parameters, and with p=k+q. The generalised score test is based on the partial M-estimator that satisfies
(2.2)∑j=1nΨβ(Xj,γ̃0)=0q,whereΨ is partitioned similarly to γ, so that ΨT=(ΨθT,ΨβT), and where γ̃0=(0kT,β̃0T)T in which β̃0 is the M-estimator of β under the null hypothesis. Define
(2.3)Uθ(γ)=∑j=1nΨθ(Xj,γ),A(γ)=-E0[∂Ψ(X,γ)∂γ]=(AθθAθβAβθAββ),B(γ)=E0[ΨΨT]=(BθθBθβBβθBββ),in which E0 denotes expectation under the null hypothesis. Here A(γ) and B(γ) are p×p and Aθθ and Bθθ are k×k. We note that A(γ) is not necessarily symmetric while B(γ) is. This means that the form of the generalised tests given by, for example, [4], needs to be slightly modified. The generalised score test statistic is given by
(2.4)SG=UθT((AT)-1)θθ(A-1B(AT)-1)θθ-1(A-1)θθUθin which, as can readily be shown,(A-1)θθ=(Aθθ-AθβAββ-1Aβθ)-1 and the arguments in Uθ, A, and B are suppressed; here all are γ̃0. Similarly the generalised Wald test statistic is given by
(2.5)WG=θ̃T(A-1B(AT)-1)θθ-1θ̃in which all arguments areγ̃. In the exposition in [4] parameters are estimated by ML but the data do not come from the parametric model: this is ML under misspecification. In [5], Kent’s definitions are given but in place of ML estimators any M-estimators are permitted. It is also noted in [4] that A and B can in practice be replaced by any consistent estimates.An alternative form ofSG that is more convenient for calculation is given in [2], where it is applied to the construction of generalized smooth tests of goodness of fit. This form gives
(2.6)SG=UθT(γ̃0)ΣGS-1(γ̃0)Uθ(γ̃0)in which(2.7)ΣGS(γ)=Bθθ-AθβAββ-1Bβθ-Bθβ(Aββ-1)TAθβT+AθβAββ-1Bββ(Aββ-1)TAθβT.The equivalence of the two forms requires routine but tedious matrix algebra and is omitted here. The asymptotic distribution of both SG and WG under H0 is χk2.IfΨ(X,γ) is the derivative of logarithm of the likelihood, which is the usual score function, then A=B is the usual (symmetric) information matrix, and A-1B(AT)-1=A-1. If ML estimation is used, then WG=θ̂T{Aθθ-AθβAββ-1Aβθ}θ̂, the usual Wald test statistic, and SG=UθT{Aθθ-AθβAββ-1Aβθ}-1Uθ, the usual score test statistic. Both are given in this form in [1]. For more information see [5, 6].In [5, page 328] replacing the inverse of the asymptotic covariance matrix ΣGS(γ̃0) in SG by a generalised inverse of a consistent estimate of ΣGS(γ̃0) is recommended. Although it may sound trivial, when calculating any of the ordinary or generalised score or Wald test statistics, we are finding (X-E[X])TΣ-1(X-E[X]) where X is at least asymptotically multivariate normal and Σ is at least asymptotically the full rank covariance matrix of X. Very occasionally it may be more convenient to find the exact covariance matrix rather than one that is asymptotically equivalent. If so the exact covariance matrix can be used in the above expressions; similarly when appropriate a generalised inverse of the exact or an asymptotically equivalent covariance matrix can be used.


## 3. The Behrens-Fisher Problem

In the Behrens-Fisher problem,Y1,…,Ym is a random sample from an N(μY,σY2) population, and Z1,…,Zn is an independent random sample from an N(μZ,σZ2) population. It is desired to test H: μY=μZ against K: μY≠μZ, with the standard deviations σY and σZ being nuisance parameters. In [2, Example  3.3.2] the likelihood ratio, score, and Wald tests are derived. The score test requires the solution of an inconvenient cubic equation; so this is one situation in which the Wald statistic looks distinctly more appealing than both the likelihood ratio and score test statistics.When the estimating function∑j=1nΨ(Xj,γ) is the usual score function, the generalised score test is the usual score test. To conform to our notation put (Y1,…,Ym,Z1,…,Zn)=XT, μY-μZ=2θ, μY+μZ=2β1, σY2=β2 and σZ2=β2. We test H:θ=0 against K:θ≠0, with nuisance parameters β1, β2 and β3. The logarithm of the likelihood is(3.1)constant-(m2)logβ2-(n2)logβ3-(2β2)-1∑i(yi-β1-θ)2-(2β3)-1∑j(zj-β1+θ)2,and therefore the score function has the following components:(3.2)Sθ(γ)=∑i(yi-β1-θ)β2-∑j(zj-β1+θ)β3,Sβ1(γ)=∑i(yi-β1-θ)β2+∑j(zj-β1+θ)β3,Sβ2(γ)=-m(2β2)+∑i(yi-β1-θ)2(2β22),Sβ3(γ)=-n(2β3)+∑j(zj-β1+θ)2(2β32).These are the partial derivatives of the logarithm of the likelihood. Under the null hypothesis the estimating equations are Sβ1(γ0)=Sβ2(γ0)=Sβ3(γ0)=0. This leads to the inconvenient cubic equation mentioned previously. If we proceed with this model, the cubic must be solved to find β̃10, and hence β̃20=∑i(Yi-β̃10)2/m and β̃30=∑j(Zj-β̃10)2/n. We also find(3.3)Sθ(γ̃0)=2(Y̅-Z̅)β̃20/m+β̃30/n,A=(mβ2+nβ3mβ2-nβ300mβ2-nβ3mβ2+nβ30000m(2β22)0000n(2β32))=B,whence
(3.4)ΣGS=4{β2/m+β3/n},and the generalised score test statistic isSG=(Y̅-Z̅)2/(β̃20/m+β̃30/n). This is just the ordinary score test statistic.While solving the cubic is not a great difficulty, if we modifySβ1(γ) so that it becomes
(3.5)Sβ1(γ)=∑i(yi-β1-θ)+∑j(zj-β1+θ), a possibly less efficient but certainly more convenient estimator of the common mean under the null hypothesis may be found. This estimator is the solution to Sβ1(γ0)=0, namely, β̃10*=(mY̅+nZ̅)/(m+n). If we also modify Sθ(γ) so that
(3.6)Sθ(γ)=∑i(yi-β1-θ)-∑j(zj-β1+θ), while leaving the other two equations unchanged, the generalised score test is based on
(3.7)Sθ(γ̃0)=2mnm+n(Y̅-Z̅).The estimators of β2 and β3 are slightly different from those found previously, being β̃20*=∑i(Yi-β̃10*)2/m and β̃30*=∑j(Zj-β̃10*)2/n. Modifying the previous derivation gives
(3.8)A=(m+nm-n00m-nm+n0000m(2β22)0000n(2β32)),B=(mβ2+nβ3mβ2-nβ300mβ2-nβ3mβ2+nβ30000m(2β22)0000n(2β32)),whence
(3.9)ΣGS=4mn(mβ3+nβ2)(m+n)2,and the generalised score test statistic is(3.10)SG=(Y̅-Z̅)2(β̃20*/m+β̃30*/n).It may be shown that the Wald test statistic is a one-one function of thisSG, so that these two tests are equivalent. However, if using the asymptotic χ12 critical values, the generalised score test has actual test sizes much closer to the nominal sizes than the Wald test. When using simulated critical values that are virtually exact, the generalised score test power is within 1% of the entrenched test due to Welch [7]. So on this criterion the Welch and generalised score tests are virtually indistinguishable.The Welch test is very similar to the Wald test. Using Satterthwaite’s approximation to the null distribution of the Welch test gives excellent agreement between the nominal and actual test sizes. However Satterthwaite’s approximation does not work nearly as well forSG. Hence, in terms of agreement between nominal and actual test sizes using approximations and asymptotic critical values, the Welch test is to be preferred. Support for these assertions and more numerical details are available in [8].


## 4. Testing Equality of Variances

Suppose that we havem independent random samples, with the jth, j=1,…,m, being of size nj and from a normal N(μj,σj2) population. The total sample size is n=n1+⋯+nm. We seek to test equality of variances: H: σ12=⋯=σm2=σ2 say against the alternative K: not H. Popular tests include the likelihood ratio test, frequently referred to as Bartlett’s test, and Levene’s test. The former is known to be nonrobust, while the latter is more robust in that its actual levels are closer to the nominal levels. Levene’s test is less powerful than Bartlett’s when the data are consistent with normality.We now construct a Wald test ofH against K. We could use the generalised Wald test construction with Ψ(X,γ) being the derivative of logarithm of the likelihood, but we leave that as an exercise for the interested reader. We could also calculate one of the forms of the asymptotic covariance matrix, but this is a case where it is simpler to calculate the exact covariance matrix. Moreover the exact covariance matrix involves an inconvenient inverse; so we instead use the Moore-Penrose inverse. This is defined in the appendix, along with some relevant useful results. This approach leads to a simpler test statistic.Throughout this example, since we are calculating the Wald test statistic, all estimation is ML. As a consequence estimators are denoted by hats (⋀) instead of tildes (̃). We also use unbiased versions of the sample variances (with divisors n-1 instead of n). These are asymptotically equivalent to the usual ML estimators, and the corresponding test statistic is asymptotically equivalent to the usual Wald test statistic.Before proceeding with the construction, note that ifS2 is the unbiased sample variance from a random sample of size n from a N(μ,σ2) distribution, then (n-1)S2/σ2 has the χn-12 distribution. As is well known, var{(n-1)S2/σ2}=2(n-1), so that
(4.1)var(S2)=2σ4(n-1)andE[S4]=(n+1)σ4(n-1).From the Rao-Blackwell theorem (n-1)S4/(n+1) is an optimal estimator of σ4, being the unique estimator with minimum variance in the class of unbiased estimators of σ4. This optimality is conferred upon 2S4/(n+1) when estimating var(S2). Writing Sj2 for the unbiased estimator of the jth population variance σj2, j=1,…,m, the optimal estimator of var(Sj2)=2σj4/(nj-1) is dj=2Sj4/(nj+1) for j=1,…,m.Should the null hypothesis be true, an unbiased estimator of the common population varianceσ2 is the pooled sample variance S2=∑jwjSj2 where wj=(nj-1)/(n-m) for j=1,…,m. Note that since ∑j(nj-1)=n-m, ∑jwj=1. Now define
(4.2)σ2=∑jwjσj2,ϕ=(σj2√wj),u=(√wj),C=Im-uuT.Then θ=Cϕ=((σj2-σ2)√wj). This is zero if and only if σj2=σ2 for all j. Hence testing equality of variances is equivalent to testing H: θ=0m against K: θ≠0m. An unbiased estimator of θ is θ̂=((Sj2-S2)√wj) and since C is symmetric, θ̂=Cϕ̂ has covariance matrix estimated by côv(θ̂)=CDC where now D=diag(djwj). Now CDC is not of full rank, and in order to use results on quadratic forms of multivariate normal random variables generalised or pseudoinverses are required. Here we use M+, the Moore-Penrose inverse of the matrix M. See the appendix.BecauseC is idempotent, the Moore-Penrose inverse of CDC is given by
(4.3)(CDC)+=C+D+C+=CD-1C.A Wald test statistic for testingH: θ=0m against K: θ≠0m is
(4.4)θ̂T(côv(θ̂))+θ̂=(Cϕ̂)T(CDC)+(Cϕ̂)=ϕ̂TCCD-1CCϕ̂=ϕ̂TCD-1Cϕ̂=θ̂TD-1θ̂=∑j=1m(Sj2-S2)2dj=TMPsay.Since rank(CDC)=m-1, TMP should be compared with the χm-12 distribution to assess significance. Should the test indicate significance at an appropriate level, rough pairwise comparisons can be made as in the comparison of means in the analysis of variance. To see how to do this first note that, as above, (n-1)S2/σ2 has the χn-12 distribution which, for large n, is approximately N((n-1),2(n-1)). Hence S2 is approximately N(σ2,2σ4/(n-1)) and under the null hypothesis of equality of variances for any i≠j the difference Si2-Sj2 is approximately N(0,2σi4/(ni-1)+2σj4/(nj-1)), and var(Si2-Sj2) can be estimated by di+dj. A least significant difference may be constructed in the usual way.



---

*Source: 292013-2010-02-07.xml*