15
L*(s) = (a9 + a,T7?/?)L(mg) + AMV 2q7/2(e0 + K,TT2) +
= AM2(a48 + aQT7/? + «oT? + «,TT1) +e (30)
where & = ln{mg/(1-mg)}. Equation (30) may be too general a model for the
estimated probabilities of interest; so, in the usual way, coefficients of
the regressors in (30) may be tested to be zero or to satisfy other
constraints. The final response surfaces for the estimated finite sample

powers of D, and f, are

L*(s) - Lima) = AM ?77272(1.16 - .508) (31)
(.23) (.17)
[.24] £.16]
K = 8 R? = .75 69 = 1.02 €,(8) = 10.2 &4(8) = 10.7 1, (8,6) = 1.03
no(1,5) = .02 &3(6) = 6.3 n,(2,4) = .04 &5(2) = .7 SK = 7.5 EK = -.4

ne(3,11) = .24 €,(1) = 1.8 dw = 2.49 no(2,12) = .85

and ,
L*(s) - Ling) = At ?77172(-1.07 = 1.202) (32)
(243) (630)
(.32] (.34]

K= 8 R?2 = .66 8¢= 1.96 &(8) = 6.8 £&1(8) = 26.0 n,(8,6) = .84
n2(1,5) = 1.29 €3,(6) = 23.1 n,(2,4) = .6 &5(2) = .8 SK =-.0 EK = -1.2

ne(3,11) = «70 €7(1) = 1.8 dw = 2.23 n9(2,12) = .01

respectively. .R? is the unadjusted squared multiple correlation
coefficient!7, 62 is the square root of the estimated residual

variance, and (+) and [:] respectively denote conventionally calculated and

17R2 may lie outside the unit interval because (30) has no constant term (see
Schmidt (1976, pp. 3-5)). In particular, R* is less than zero in the
general response surface for D, because L*(s) has a large negative mean.
However, Ge, not R?, is the appropriate measure of the goodness-of-fit

for the response surfaces, so small or negative values of R? per se are not
worrisome.

In this paper, "R?" refers both to the unadjusted squared multiple
correlation coefficient for response surfaces and to the squared population
multiple correlation coefficient for the DGP (8). The uses are distinct
and no confusion should arise from using a common notation.


34

Appendix D: Pesaran's experimental results far the multiple correlation coefficient of the DCP (R?)

being .80 and

Table D.I.

5.

Estimated probabilities far the Cox and F tests when R? = .80.

Probability of
accepting Hy
and

rejecting H,

Probability of
rejecting Hy

and

accepting H,

Probability of
rejecting both
Ho and H,

Probability of
accepting both
Ho and H,

Type I error:
probability of
rejecting Hy

Type II error:
probability of
accepting H,

D test F test

D test F test

D test F test

D test F test

D test F test

D test F test

+78 634
(.018)@ (.022)

600 4o4
(.022) (.022)

T4096
(.017) (.013)
916 +92
(.012) (.013)

+802 .738
(.018) (020)

280.202
(.020) (.018)
+82 BY
(011) (.009)

8 £6
(010) (.010)

456.384
(022) (.022)

050
(.010)

+058
(.010)

+026
(.007)

024
(.007)

032.022
(.008) (.007)

«028.032
(.007) (.008)

132
(.015)

2314
(.021)

“BA
(.019)

318
(.021)

540
(.022)

-82
(.016)

082 048
(.012) (.010)

+086 = .056
(.013) (.010)

182 Bay
(.017) (.021)

372.564
(.022) (.022)

-804 878
(.018) (.015)

032.068
(.008) (.011)

“170.236
(.017)  (.019)

Note: a. Simulation standard errors of estimated probabilities are in parentheses.



30
Having solved for v*® and the lower limit point of the integral in (C2), the
integral itself may be calculated numerically, e.g., with the NAg (1977)
routine GO1BCF, linearly interpolating for P if v* is not integral.
Table C.I compares the approximation given in (C2) with exact values of
the integral in (C1) for the values of v and’) appearing in Patnaik (1949,
p. 207, Table 1). Patnaik's own approximation to the exact probability

involves not only linearly interpolating P for non-integral v*

» but also
interpolating P for the value of the lower limit of the integral in (C2).
(C2) (using GO1BCF) appears to be a better approximation to the exact
probability than Patnaik's approximation, but only marginally so, with both
approximating the exact probability quite well.

The probability of a singly non-central F random variate exceeding

Fy(v,,¥2,0), the 100(1-a) per cent point of the (central) F distribution, ‘is

Pos f GF (v1 5V25A) (c5)
Fav. 5250)

where F(v,,v2,A) is the singly non-central F distribution.with the degrees
of freedom in the numerator and denominator of the F-ratio being v, and v,,
respectively; and A is the non-centrality parameter of the x? random
variate in the numerator, the non-centrality parameter of the x? random
variate in the denominator being zero (see Kendall and Stuart (1973,

Pp. 262, equation (24.105))). Equating the first two moments of the
nor-central x? random variate in the numerator to those of a central y?
random variate (as above), then

°

Pos Sf dF (v*,v,,0) (C6)
Cyr / (vt) Fg (V1 52 50)
where
vs Mt (cT)

Vv, + 2h


39

Pesaran, M.H. (1981) "Comparison of local power of alternative tests of
non-nested regression models", mimeo, Trinity College, Cambridge,
June.

Pesaran, M.H. (1982) "Comparison of local power of alternative tests of
non-nested regression models", Econometrica, 50, 5, 1287-1305.

Phillips, P.C.B. (1977) “Approximations to some finite sample distributions
associated with a first-order stochastic difference equation",
Econometrica, 45, 2, 463-485; and (1982) "Erratum", Econometrica, 50,
1, 274.

Rao, C.R. (1973) Linear Statistical Inference and Its Applications, New
York, John Wiley and Sons, second edition.

Sargan, J.D. (1976) "Econometric estimators and the Edgeworth
approximation", Econometrica, 44, 3, 421-448; and (1977) "Erratum",
Econometrica, 45, 1, 272.

Sargan, J.D. (1980) "Some approximations to the distribution of econometric
criteria which are asymptotically distributed as chi-squared",
Econometrica, 48, 5, 1107-1138.

Schmidt, P. (1976) Econometrics, New York, Marcel Dekker.

Silvey, S.D. (1975) Statistical Inference, London, Chapman and Hall.

Sowey, E.R. (1973) "A classified bibliography of Monte Carlo studies in
econometrics", Journal of Econometrics, 1, 4, 3777-395.

Theil, H. (1971) Principles of Econometrics, New York, John Wiley and Sons.

Wilks, S.S. (1962) Mathematical Statistics, New York, John Wiley and Sons.

White, H. (1980a) "A heteroskedasticity-consistent covariance matrix
estimator and a direct test for heteroskedasticity", Econometrica, 48,
4, 817-838. _

White, H. (1980b) "Using least squares to approximate unknown regression
functions", International Economic Review, 21, 1, 149-170.



lla

Table I. Criteria for evaluating response surfaces

Null Alternative Statistic? Sources
(A) a #1 n2(1,K-m-1) Hendry (1984, p. 952)
(B,) oe? > 1 £,(K-m) Theil (1971, pp. 137-8)
(B,) q invalid parameter ny (q,K~-m-q) Johnston (1963, p. 126)
restrictions
(B,) skewness (SK) and &5(2) Jarque and Bera (1980)
excess kurtosis (EK)
(B,;) heteroscedasticity ne (q,K-m-q) White (1980a, p. 825),
quadratic in regressors Nicholls and Pagan (1983)
(q quadratic terms)
(B,) first-order ARCH &,(1) Engle (1982)
(B,) first-order residual dw Durbin and Watson
autocorrelation ‘ (1950, 1951),
Farebrother (1980)
(B,) ath-order residual Ee(a)s Box and Pierce (1970);
autocorrelation Ne (q,K-m-q) Godfrey (1978), Harvey
(1981, p. 173)
(Bs) H*(+,*) not constant ng (m,K~2m) Kendall
over subsamples . (1946, pp. 242ff),
Chow (1960, pp. 595ff)
(Bs) predictive failure &,(q); Hendry (1979, p. 222);
over a subset of q ny (q,K-m=q) Chow (1960, pp. 594-5)
obser vations>s»¢

Notes: a. The value of q may differ across statistics, as may the number of

b.

regressors m and the number of experiments K across response
surfaces and Monte Carlo studies.

We have labelled the Chow statistic n,(q,K-m-q) to highlight: the
pre-eminence of the issue of constancy. The covariance test;
statistic ng(m,K-2m) is often (and confusingly) referred to as
the "Chow statistic" although Chow (1960, p. 592) was well aware
of its presence in the literature.

Constancy may be tested using Chow's statistic, the covariance
statistic, or the usual x? statistic based upon the forecast:
errors. Often, an even more stringent test may be construct:ed by
substituting unity for the estimated value of Oe” in the re:evant
statistic, thereby testing the "absolute" accuracy of the
response surface. Such statistics are designated as those above,
but with a prime added, e.g., &,(q) becomes E1(q).


19
set [sic] of explanatory variables is large". Pesaran's conclusions may
not depend so much upon the asymptotic nature of the Cox test (vs. the
exact nature of the F test) as upon the comparison of estimated type I and
type II errors for which the density of the statistic (for the type I
error) at its critical values is a small fraction of the density of the
statistic (for the type II error) at one of the critical values.

To see that, consider the extremely simplified example in which the
test statistic t is exactly distributed as N(0,1) under H, and N(y,1) under
H, and the critical value (to be chosen) for a symmetric two-tailed test is
So Or 8, (5 > 6, > 0) with 6 slightly greater than §,. The mean y is
non-zero and is assumed negative without loss of generality. Further,
suppose that, for 6,, power is in an intermediate range, so yu = “So.
Simplifying the example even more (but not appreciably weakening the
argument), suppose that py = ~S, = -1.96 (see Figure 2). At u, the
densities under H, and H, are approximately .058 and .399 respectively.
Noting that the probability of 1 being greater than +1.96 is negligible
under H, and that the test is symmetric, it follows that a test using 6,
would be preferred to one using 6), even if the probability of type I error
were weighted three times as heavily as the probability of type II error.
In the context of the Cox and F tests, because Pesaran was comparing
statistics for which the probability of type I error was small (= .05) and
the power (and hence the probability of type II error) was usually between
.10 and .90, his criterion might well have favoured the test statistic with
the larger probability of type I error (i.e., the Cox statistic), even had
the two test statistics had identical distributions but critical values

corresponding to different probabilities of type I error.


Table D.II.

35

Estimated probabilities far the Cox and F tests when R? = .8.

Probability of
accepting Ho

and

rejecting H,

Probability of
rejecting H,

and

accepting H,

Probability of
rejecting both
Hy and H,

Probability of
accepting both
Hy and H,

Type I error:
probability of

rejecting Hy

Type II error:
probability of
accepting H,

D test F test

D test F test

D test F test

D test F test

D test F test

D test F test

-84 — .768
(016) (.019)

+688
(.021)

-5HO
(.022)

116
(.014)

O44
(.009)

+054
(.010)

+026
(.007)

024
(.007)

O42
(.009)

+034
(008)

062
(.011)

070
(.011)

+022
(.007)

«022
(.007)

+032
(.008)

2024
(007)

060
(.011)

«224
(.019)

674
(021)

+184
(.017)

«086
(.013)

+088
(.013)

+062
(.011)

+074
(.012)

+056
(.010)

O48,
(.010)

+056
(.010)

+104
(.014)

+218
(.020)

210
(.018)

428
(.022)

Note: a. Simulatio standard errors of estimated probabilities are in parentheses.



Hot Y = XoBo + Uo Up ~ D(0,097*Ip) a)
and

Hy: y = X,8, + uy, u, ~ D(0,0,?7*Ip) (2)
and the comprehensive hypothesis

Ha: y = X,B2 + uy u, ~ D(0,02?*Ip) (3)

where the dependent variable y is Tx1, T being the econometric sample size;
Xj is a Txkj matrix of regressors and 8; the corresponding kyx1 vector of
coefficients (i=0,1,2); X, includes all the non-redundant variables in
(X,:X,) with 8, conformable; and uj is a Tx1 vector of disturbances with
mean zer) and variance oj7-Ip (i=0,1,2). Two approaches have been
Suggested for testing H, against H, when X, and X, are predetermined:
following Cox (1961, 1962), Pesaran (1974) proposes evaluating a modified
likelihood ratio statistic for H, and H,; alternatively, because those
hypotheses are nested in H,, the restrictions implied by going from H, to
H, (or H,) can be tested using the F or Wald statistics (cf. Silvey (1975,
pp. 115-116)). Under Hy,

Do g N(0,1) (4)
and
f, 4 F(k,-kKo, T-k2, 0) (5)

where D, is the Cox statistic for which H, is assumed true, f, is the

F statistic for testing H, against the general hypothesis H,, and g

denotes ‘converges in distribution to, as T+ o*", Pesaran (1982) and
Ericsson (1983) derive the asymptotic properties of D, under Hy and Hy;
however, Pesaran (1974) estimates the finite sample rejection frequencies
of D, and D, (the latter being the Cox statistic for which H, is assumed
true) under Hy only. Noting the similarity in those approaches, it follows

from Ericsson (1983) that, under H, as a local alternative to H,,


28

Appendix B: Proof of the asymptotic normality of the empirical logistic function.

It is to be shown that

s - (2N)7? 7
o(s,7) = at/?{in —------ | -1n bog NCO,1). (Bt)
1- s - (2N)7? 1-0

S is the number of "successes" out of a total of N observations (N > 1) on

a binary response variable with probability of "success" equal to 7
(0 < 7 < 1), s (3S/N) is the fraction of "successes", A = S(N-S)/(N-1), and
i cenotes "converges in distribution to, as N+", Results
in Cox (1970, pp. 30-34, 41-42, 78-79), Mann and Wald (1943), and Cramer
(1946, pp. 254, 299-300) are used extensively.

Noting that E(S) = Nn and var(S) = Na(1-1), define U such that

S = Nr + UN. (B2)

Hence E(U) = 0 and var(U) = a(1-1). The term in braces in (B1) is

mw + (U/YN) - (2N)7* 1-9 - (U/YN) - (2N)7?
{-} = 1n - ln
T lr

U/(nCi-m)YN} + Op(N7?) (B3)

(ef. Cox (1970, p. 33, (3.11)) with (his) a=-.5). The variable A in (B1) is
A = Na(t-s) + Op(N®)

= Na(i-m) + Op(Nt/?), (B4)
so
AM? 2 NM 2En(1mn) + Op(NTI/2) 33/2

{Na(1=n)}97? + Op(N®) (B5)
Substituting (B3) and (B5) into (B1),

(Sym) = U/(mCmm) F977 # Op (NT?) (B6)
Since S is the sum of N independent and identically distributed random
variates, each with mean m and variance m(1-m), and U = (S - Na)/YN, then
U/{1(1-1)}#/? converges in distribution to N(O,1) as N > © by the
Lindeberg-Lévy variant of the central limit theorem (Cramér (1946, p. 215)).

Hence (B6) converges in distribution to N(0,1) as N > @. ll


16
White (1¢80a) coefficient standard errors. Tests of (B,) use eight sets of
results for which T and r? take the same values as the fitted results, but
for which the R? of the DGP is 85,18 Noting that the entire discussion
of response surfaces above applies equally well for any estimated
probability (including the estimated probability of type I error), response
surfaces for the estimated probability of type I error for D, are estimated

as well, the final specification of which is

L*(s) - L(.05) = a/?77110.5 (33)
(4.5)
(1.1)
K=8 R2 = .65 Gg = 89 (8) = 6.5 E1(8) = 5.2 14(8,7) = .80
no(1,5) = .00 €3(7) = 5.5 4(2,5) = 1.70 (2) = 1.2 SK = .4 EK = -1.1

ne(l,14) = 2.07 &(1) = 1.0 dw = 1.80 n9(1,14) = .12

The estimated probability of type I error for f, need not be analysed further
because it does not deviate significantly from its asymptotic value.

The summary statistics indicate that the response surfaces for D, and
D, show no signs of mis-specification nor are the restrictions that a, = 1
and x, = 0 (for D,) and a, = 1 and ky = O (for D,) rejected. Given the
data available, those response surfaces support the conjectures that
a) = 1, de? = 1, and that very simple expansions of a and G(e,T7~'/?) are
sufficient to explain the observed Monte Carlo experiments. Further, both

restricted response surfaces predict the eight out-of-sample observations

18see Appendix D for all the results. The results tor {T=80, r?=,90} are not
included in any of the response surfaces although they could have been in
those for Dy. Experiments are ordered as in Table II. .

In general, experiments for prediction could be chosen by random
selection or stratified selection. Cf. Cox (1958), Wilks (1962).

The statistics —,(*), SK, EK, and ng(-,°) use all sixteen experiments
because of the degrees of freedom involved; the others (where appropriate)
use only the eight published in Pesaran (1974) (i.e., for which R® = .80).
However, their values alter only slightly if all sixteen are included.


20a

Teble III. The powers of the F test (f,) with non-oentrality 1, when evaluated
at critical values corresponding to ag = .05 and af = Sp,
ag being the size of the F test and sp the estimated size of the Cox test.
ag = .05 of = Sp estimated

finite

sample
Sample} r?]| A’, |critical| og | asymptotic] critical] of Sp asymptotic | power of

size value power of value power of | D,
f, f,

-90} 8.0] 4.45 | 05003] .76755 3.4175 | 08198] .082 84731 +818
(.012)# (.017)

T=20 | .%] 4.0] 4.45 | .05003| .45798 3.3225 | .08597| 086 -57239 628
(.013) (.022)

-99] .8] 4.45 | .05003| = «12034 3.6800 | .07202| .072 -16823 +196

(.012) (.018)

+90/16.0f 4.10 | .05014}  .98435 3.3325 | .07600} .076 99237 +%8

(.012) (.008)

T=40 |. ] 8.0{ 4.10 | .05014) .79740 3.2400 | .08002}  .080 8635 +830
(.012) (.017)

-99| 1.6 4,10 05014 21897 3.8925 | .05601 056 23383 «306

(.010) (.021)

Note: a. Simulati standard errors of estimated probabilities are in parentheses.



18
Se) than the one for D,, so that issue may be worthwhile investigating
further.

In general, response surfaces appear quite valuable in summarizing
Pesaraa's (1974) experimental results. Simple, well-determined response
surfaces are obtained using only eight experiments for their estimation.
Further, those response surfaces satisfactorily predict the outcomes of
eight additional experiments for which one of the experimental design
parameters (R?) is considerably different from its value in all the
experiments used for estimation, indicating the robust nature of those
response surfaces. Even so, the response surfaces are limited by the
relatively small number of experiments and the restrictiveness of the
parameter space. Such specificity could be reduced markedly by performing
more experiments over a more broadly defined parameter space (e.g., one
including lagged dependent variables, several non-overlapping regressors,
k #k,, and autocorrelated exogenous regressors), thereby allowing more

sophisticated and (hopefully) reliable inferences to be drawn.

6. fdjustments for the estimated probability of type I error

In comparing the properties of the Cox and F statistics, it is only
reasonable to consider both the probability of type I error and the
probability of type II error. The estimated probability of type I error
for D, significantly exceeds .05 in several instances (predominantly for
smaller sample sizes and smaller values of r?), whereas the estimated
probability of type I error for f, never does. To account for both types
of error, Pesaran (1974, p. 163) uses a simple linear loss function in
which type I errors receive the same (or even twice as much) weight as type
II errors, concluding that "the [Cox] test is preferable to the F-test when

the sample size is small (n <$ 40) and the correlation between the competing


29
Appendix C: The calculation of probabilities associated with non-central
x? and F distributions, and with the normal distribution

The discrepancies between the theoretical and estimated finite sample
powers of f, might have arisen from the approximation used to calculate the
probabilities associated with the non-central F distribution. This
appendix, which closely follows the substance and notation of Keridall and
Stuart (1973, pp. 237-239, 241, 262-264), shows that the errors involved do
not appear large enough to explain those discrepancies. For completeness
and clarity in presentation, a related approximation (of the non-central x?
distribution by a central one) is given first. See Johnson (195%) on other
approximations to the non-central x? and F distributions.

The probability of a non-central y? random variate exceeding

Xa(v,0), the 100(1-a) per cent point of the central x? distribution, is

P = f dy?(v,A) (C1)
x5,6¥50)

where x?(v,A) is the non-central x? distribution with degrees of freedom v
and non-centrality parameter ., as given in Kendall and Stuart (1973,

p. 238, equation (24.18)). Equating the first two moments of the
non-central e random variate to those of px?(-,0) (a central y? random

variate multiplied by a factor of proportionality p, to be determined),

then
Pos f dy? (v*¥,0) (c2)
x2(v,0)/p
a
where
*L (v +d)?
vot ee (3)
and
p = ten, (cu)

vty


13

asymptotic theory (cf. Figure 1). Further, for T=80, the three estimated
probabilities are individually and jointly insignificantly differert from .05.

The estimated finite sample power of D, differs significantly from its
asymptotic power in three experiments although, as with the probability of
type I error, the difference between estimated and asymptotic values
decreases as T increases. Also, as with the probability of type I error,
the estimated values are insignificantly different from their asymptotic
values for T=80.13

That the estimated finite sample power for f, differs from its
asymptotic power as markedly as it does in three experiments is a surprise
at first blush, and requires some explanation. As noted in Sections 2 and
3, f, is exactly distributed as F(1, T-k,, a®) but only asymptotically
as F(1, T-k,, A,), where ae and \, are the finite sample and asymptotic

non-centrality parameters in (20) and (17).14 at is not equal to A,

13No test can be made for {T=80, r?=.90} because S = N. See Cox (1970,
pp. 33, 42, 78) on the treatment of experiments with S=0 or S=N.

\4ror pesaran's model,

fy = fy" (W.-W, )y/(ka-k, )}/fy" (Ip-W, y/ (Tok, )}
Noting that (Ip-W,)X, = 0 and (W.-W, )X,8, = (W.-W, )X$8% where pj is the
sub-vector of parameters in B, corresponding to Xj, then, under H,,

fy = {u*"(Wo-W, u*/(k-k, )}/{u®' (Ip-W, )u*/ (T-k, )}

where u* = xtgt + u,. If X, is fixed in repeated samples, or at least if
X, and u, are independent in each sample (and so X, may be conditioned
upon; see Schmidt (1976, pp. 93-94ff)), then, conditional upon X,,

u® ~ N(X$83, oo2*Ip) .

Because (W.-W,) and (I7p-W,) are each idempotent and of ranks (k,~k,) and
(T-k,), respectively,

u®"(Wa-W, u*/og? ~ x2(k2-k,, AZ) and

u*' (Ip-W, u*/ag? ~ x2(Tok2, 0) where

ae = pt xt! (Wo-W, )XEB¢/007
noting that (Ip-W,)Xj = 0. Further, the two x? variates above are
independent because (Ip-W,)(W,-W,) = 0 (see Rao (1973, p. 187)), so

f, ~ F(k,-k,, T-k,, A$) (exactly) .


8
accomplished by replacing + by tg in (24) and comparing the value of
o(S,1q) with the appropriate critical values for the standardized normal
distribution.§
Even when t=1g is not a good approximation, (24) still provides the

basis for analysing the Monte Carlo results. Without loss of generality,

a

1T T.
S| expict(e,7)} (25)

1 = 0p 1- fo,
where 9 is the vector of all the parameters (except T) which define the
model generating the binary random variable of interest, Gt(-,+) is some
appropriate function, a is a parameter which itself may depend upon 6 and
T, and mn is subscripted by T so as to emphasize that m (= E(S/N)) isa
function of the econometric sample size.? Using (25), (24) may be
rewritten as

L¥(s) = a bmg) + al/?-ct(e,T) + € © x N(O,1). (26)

8as indicated by Cox (1970, pp. 27-29), the logistic and normal
distributions only differ slightly, primarily in the tail regions. So,
provided that m is not too close to 0 or 1, the normal distribution should
provide a good approximation ta'the distribution of (24). Furthermore, the
use of the normal approximation is often justified because N is typically
quite "large" in Monte Carlo studies (e.g., N = 500 in Pesaran (1974)).

It is assumed throughout that m and ma each lie strictly within the
unit interval. °

9clearly, (25) is equivalent to (16). However, the functional form of (25)
ensures that predicted powers are within the unit interval. Further, (25)
has the advantage of having split g(6,T) in (16) into two components, an
asymptotic term and a term involving the deviation between the finite
sample and asymptotic distributions. The equivalent partition for (16) is
TM = Ta + (npr). By using asymptotic theory, we are able to simplif’y the
problem of simulating m7 (of 0(1)) directly to one of analytically
calculating mg and simulating only (ny-mg) (of 0(1), and quite possibiy
o(T7!/2)), That is in line with Hammersley and Handscomb's (1964,
pp. 5, 59) precept that one should solve as much of the problem as possible
analytically in order to minimize the imprecision and specificity arising
from simulation. With that in mind, see Hendry (1973) and Nickell (1981)
for two elegant examples in which analytical formulae greatly simplify the
interpretation of previous Monte Carlo studies.

If an analytical approximation to mp better than mg is available
(e.g., an Edgeworth expansion), it could appear in (25) in place of ty,
further reducing the order of the term being simulated.


T7E(X,'X,)

c
4

1
mt

It follows that

My

Mit

My,

Jo

Y

for positive Y.

(1]
["]
[0 vn |

{(8$)?(1-R?)/R?

(r2/(1-r2)} 1/2

rm y4t
Xo XS

x'xt

'
L x1'xs

Xt
Xt

L *t2t

Mio

Moo

yi 2

26

x$'x = -xt'xt
x'x — x'xt
xt'X  xt'xt
Xt Xt 2t
1 Zt,
2 2p?
0
1 . (a22)
(0) 244
01
yY 0
10
o1
(A23)

The four-fold product of matrices for $(;) (appearing in

the parentheses of the formulae for $(;) in (A21)) simplifies to

4
0 7

ie)
(ey f ?

80 (1) = (6*)?(r?)i, is0,..

Wa
Wy)

Ns

Substi

= 8¥ (1-2) 1/2 (14r?)/(209r?)

(ur* + (1-r*) (14?) }/ (ar ®)

(8*)?(1-9?)/ a9?

(a24)

tuting into (A13) and (A14),
, (A25)
» and (A26)
(A27)

Those formulae for y,, w,, and \, were used to calculate the asymptotic

powers of f, and D, under H, from Pesaran's model.


23

the entire set of regressors X, in Pesaran (1974). Further, under Hi,

¥"(Q2-Qo )¥/ (ka-Ky )

* y' (Ip-P,)' (Ip-P, )y/ (Tk, )
ut (Q,-Q, )u/ (kz-Ky ) (a6)
ul (Ip-P2)' (Ip-P, )u/ (Tk, )
where
us = -X{etT + uy,
and
u, ~ N(O, o,7*I7) .
Hence
u' (Q,-Q)U/{ 012 (ka-Ky)} gy X2(Ka~Ko yp )/ (Kako) (aT)
and
ul (Ip-P,)' (Ip-P2)u/{01,2(T-ke)} x2 (Tk, 0)/(THk,) (a8)

where \, = 0o/0,7. These random variates are independent asymptotivally, so
their ratio f, is asymptotically distributed as a singly non-central F,

f2 4 F(k,-ky, T-k,, Ao) (a9)
Assuming X, fixed in repeated samples, or at least assuming X, and uy,
independent and X, to be conditioned upon (see Schmidt (1976,
pp. 93-94ff)), then

uo ~ N(XTBT, o17 Ip) 5 (A10)
and, with Z - X,, the results in (A7), (A8), and (A9) are exact (i.e., with
qa Substituted by ~), with
AZ = BE XT" (Q,-Q, Xt 84/04? (a1)
replacing A, in (A7) and (A9).

In Monte Carlo studies, it is often more convenient (and less
expensive computationally) to evaluate (for instance) both D, and D, under
Ho, than to evaluate D, under H, and H,. Effectively, D, under H, behaves
like D, under H, because each statistic is evaluated under a non-nested

alternative. Using the formulae in (A3)-(A5), it readily follows that


5
(R?=b,?/(b,2+097)), Pesaran (1974): calculates D,, D,, f2, and f,.for 500

replications of each of 400 points defined by all possible combinations of

T = (20, 40, 60, 80)
rc? = (.90, «91, «20, ©99)
2? = (.80, .81, ..., .89) (13)

with a,=100 and b,=2. Following Hendry's (1984, p. 940) notation and
terminology, the Monte Carlo design variables are

8 = (ag,bos7,R?) € © = fo | 2X1, RT} (14)
and

Te T= (T°,T*] (15)
where T is pre-assigned with T°=20 and T'=80.3 Equations (8) and (9)

are the data generation process (DGP); OxT is the parameter space;

equations (10), (11), and (12) are the relationships of interest4; and

the objective of the Monte Carlo study is to determine the finite sample
distributions of the statistics D,, D,, f,, and f, as defined by the
relationships of interest, within the specified parameter space of the DGP.
More modestly, letting t be any of Dy, D,, f,, and f, and 6 be the critical
value associated with a test based on 1, the objective is to find the
finite sample rejection frequency 1 = prob(|t|26). That probability

depends upon 9 and T and can be expressed as a conditional probability

formula:

u

1 prob(|t|26 | 6,T) = g(0,T). (16)

Thus, we wish to know (or obtain a good approximation to) g(6,T) over OxT.5

3In (14), @ could be defined as 6 = (Ce re noting that (b,,R?) maps
one-to-one onto (by,d97).

YWote that: both Hy and H, coincide with the DGP but H, does not:
specifically, H, is a mis~specified model with x, in H, replaced by zt, a
variable correlated with xz.

5Implicitly, g(-,+) is a function of 6 as well. However, because 6 is
constant for each of the two types of statistics in Pesaran (1974), its
presence in g(-,*) is ignored in the analysis below.


Di a NG.) (6)
and

fs a F(ks~k,, T-k,, h3) (7)
where y, and w, are the asymptotic mean and variance of D,, f; is the
F statistic for testing H, against H,, and A, is the asymptotic

non-centrality parameter of that F statistic.!

3. The data generation process and experimental design

Pesaran (1974, p. 160) generates his data in the following manner:
Yt = ao + DoxXt + UgE Upt ~ NID(0,0,7) (8)

=p2)1/2
Xt ~  nrp(o r/(1-r?) )

1
2 fay (epayye  Gepeyd tety...7 (9)

where r is the population correlation coefficient of xX, and 2.2 Given

that process, he considers three specifications:

Hot Yt = ao + DoXt, + uUgt Ugt ~ NID(0,097) (10)
Hit ye = a + diz + UE Ut ~ NID(O,0,7) (11)
Hat Yt = a, + DaxXE + CozeE + UdE Ust ~ NID(O,0,7) . (12)

Letting R be the population multiple correlation coefficient for (8)

lgriesson's (1983) derivation of an equation equivalent to (6) but for Dy
with H, as a local alternative involves an approximation additional to the
standard asymptotic approximation. Pesaran (1982) only requires the usual
asymptotic approximation but must have at least as many regressors (total)
under H, as regressors in H, but not in H,. (Note that the roles of H, and
H, are reversed for D,.) In practice, Ericsson's and Pesaran's
approximations appear numerically similar; and, for the model in Pesaran
(1974), Pesaran's (1982) asymptotic distribution for D,? is just that of f,.
However, the formula in (6) is used throughout. See Appendix A for a
detailed discussion of the statistical Properties of Do, D,, f,, and f,.

For extensive discussions on the role and interpretation of non-nested
hypothesis test statistics, see MacKinnon (1983) (including the comments
thereon), the Journal of Econometrics issue on "non-nested models" (1983,
Vol. 21, No. 1), Mizon (1984), and Mizon and Richard (1986). Godfrey and
Pesaran (1983) suggest and evaluate analytical adjustments for improving
the finite sample properties of the Cox Statistic.

2Pesaran's (1974) notation of ryz has been simplified to r, and n is now T.


14

in general, so f, is not (generally) distributed as an F(1, Tok, A3)-
Further, a’ varies across replications because the x,'s and z's are
generated for each replication. Because the distribution of f, is a
non-linear function of a, which is itself a non-linear function of Xo*,
the expected fraction of rejections by f, need not equal the probability of
rejectior associated with the given critical value for F(1, T-k,, d,).19
However, (7) still holds, so, for large N, o(3,mg) Should be approximately
a standardized normal variate, provided that T is large enough. If X,*
were known for each replication (or if it had been held constant over
replications within experiments and were known for each experiment), the
exact probability of rejection for f, could be calculated for each
replication (and so for each experiment). As it is, X,* is not known, so
only the asymptotic formula in (7) may be used. As noted for D,, the
differenze between the estimated finite sample power of f, and its
asymptotic power shrinks as T increases .16

Because Pesaran's (1974, p. 162) published results contain just nine
experiments (effectively eight for D, and f,), only highly restrictive
response surfaces may be fitted to them. Expanding G(e,T7'/2) and a in
T7/2 only and truncating at the first derivative (all a priori and

arbitrarily) leads to the rather simple response surface

15the discrepancies between the theoretical and estimated finite sample
powers cf f, might have arisen from the approximation used to calculate the
probabilities associated with the non-central F distribution. However,
Appendix C shows that the errors involved are not large enough to explain
those discrepancies.

16, similar analysis applies to the distribution of f, under H,, with
f, ~ F(ky-Ko, T-ka, 43) and f, q F(kK,-ko, Trkz, Xz) where a*® and

A, are defined in an obvious manner. Under H, (i.e., with g{=0),
however ,, a = hz = 0; so the exact and asymptotic distributions of f,
are identical, and not a function of X, at all. The results for f, in
Table IZ are in line with that discussion.


1
error.6 Consider a binary response variable for which the probability
of "success" (or, frequently later, acceptance or rejection by a particular
test) is a (0 < m < 1) and on which there are N observations (N > 1), S

being "successes", Letting

A = [S(N-S)]/(N-1) , . (21)
z
L(e) = at/?in | — o<c<], (22)
1-¢,
and
o> (2N)7?
u*(c) = al21n | ———____ (2N)7? <5 < 1-(2N)7? (23)

1- o- (2N)7*

in a notation similar to that found in Hendry (1984, pp. 957-961) and Mizon
and Hendry (1980, p. 34), then, using results from Cox (1970, pp. 30-34,
41-42, 78-79), it can be shown that

(syn) LF(s) = Lin) gz N(0,1) (24)
where s = S/N and i denotes "converges in distribution to, as N > @ "7
In the context of Pesaran's (1974) Monte Carlo study (and Monte Carlo
studies of powers in general), N is the number of replications ina
particular experiment, S the number of replications for which the value of
the test statistic lies in the critical region, and m the finite sample
(i.e., finite econometric sample T) probability of the test statistic lying
in the critical region.

Typically, 1 is unknown; and, as an initial step in analysing Monte

Carlo results, it is of interest to test whether m equals the (local)

asymptotic (i.e., as T > ») power of the test (ma, say). That is easily

6see Cochran and Cox (1957, pp. 335ff), Cox (1958, pp. 113-128), and the
references in Cochran and Cox (1957, p. 369) on the use of response
surfaces in statistical analyses; cf. Sowey (1973).

TA proof is given in Appendix B.


19a

Figure 2
The density of t under Hg and Hy
Density at Xx

[~ 50

— 4.0 - 3.0 — 2.0 - 1.0 0 1.0 2.0


International Finance Discussion Papers
Number 276

Revised: March 1986

POST~SIMULATION ANALYSIS OF MONTE CARLO EXPERIMENTS:
INTERPRETING PESARAN'S (1974) STUDY
OF NON+NESTED HYPOTHESIS TEST STATISTICS

by

Neil R. Ericsson

NOTE: International Finance Discussion Papers are preliminary materials
circulated to stimulate discussion and critical comment. References in
publications to International Finance Discussion Papers (other than an
acknowledgment by a writer that he has had access to unpublished
material) should be cleared with the author or authors.

al


IFDP
NUMBER
265

264

263

262

261

260

259

258

257

256

255

254

253

252

251

250

41

International Finance Discussion Papers

TITLES

Money Demand in Open Economies:
Substitution Model for Venezuela

A Currency

Comparing Costs of Note Issuance Facilities
and Eurocredits

Some Implications of the President's Tax
Proposals for U.S. Banks with Claims on
Developing Countries

Monetary Stabilization Policy in an Open
Economy

Anticipatory Capital Flows and the Behaviour
of the Dollar

Simulating Exchange Rate Shocks in the MPS
and MCM Models: An Evaluation

Trade Policy for the Multiple Product
Declining Industry

Long Memory Models of Interest Rates, the
Term Structure, and Variance Bounds Tests

Currency Substitution and the New Divisia
Monetary Aggregates: The U.S. Case

The International Transmission of Oil Price
Effects and OPEC's Pricing Policy

U.S. Banks' Lending to Developing Countries:
A Longer-Term View

Conditional Econometric Modelling:
An Application to New House Prices
in the United Kingdom

Low Pushing: Doctrine and Theory

1984
(partial listing)
Postwar Financial Policies in Taiwan, China

Foreign Exchange Constraints and Growth
Possibilities in LDCs

The Determination of Front-end Fees on
Syndicated Eurocurrency Credits

AUTHOR''s)

Jaime Marquez

Rodney H. Mills

Allen B. Frankel

Marcus H. Miller
Arnold Kling
Arnold Kling
Catherine Mann
Gary S. Shea
Jaime Marquez
Jaime Marquez
Henry S. Terrell
Rod Mills

Neil R. Ericsson

David F. Hendry

William Darity, Jr.

Robert F. Emery

Jaime Marquez

Rodney H. Mills
Henry S. Terrell


10

L¥(s) = agb(mg) + oT7}/*L (mg) + 22. + AM ?T7}/2H( 8,777?) + ee (29)

where H(0,T7!/?) is the least squares approximation to G(e,T7?/2) and the
error term e is the combination of e (the error arising from simulation)
and Al 277~1/2(G(+,+)-H(+,+)} (the error from approximating G(-,+) by H(+,°)).

A response surface like (29) summarizes a possibly vast array of Monte
Carlo simulations in a relatively simple formula which may account for much
of the variation in s across experiments and may be useful for predicting
my at points within OxT but not included in the simulations. Further, the
response surface may adequately approximate the underlying finite sample
distribution function. Two types of information are available for
inferring how "good" a response surface like (29) is:

(A) asymptotic theory (i.e., a =1), and

(B) © g NID(O,1)

(ef. Hendry (1984, p. 962)). Although (B) is not directly testable, many
testable implications follow from the null hypothesis that H(-,°)=G(-,°).

(B,) oe? = 1. If H(+,+)#G(+,+), then og? > 1 because e€ is
uncorrelated with A'/?77!/2{G(-,-)-H(+,+)}. The hypothesis og? = 1 may be
tested by noting that, under the null, the residual sum of squares fron
(29) is distributed as a x? random variate with its degrees of freedom
equal to the number of experiments less the number of regressors, provided
N is large. Power under the alternative is directly related to the
magnitude of AT~'{G(-,+)-H(-,*)}? over the experiments.

(B,) The error e does not include any terms involving @, T~'/?, and
In{tg/(1-mg)}. By using OLS, e can not include any of the terms in H(+,°).
However, if H(-+,+)#G(+,+), e contains terms of a higher order than those
included in H(+,+) (cf. Maasoumi and Phillips (1982, p. 198) and Hendry
(1982, p. 210)). By initially specifying a rather general formulation for

H(-+,*) and a and simplifying therefrom, the F statistic comparing the final


9
In practice, a and the functional form of G*(:,*) are unknown although
(e.g., in Pesaran (1974)) they are implicitly defined by the computer
program generating the Monte Carlo data. If a and G*(-,+) were known, the
exact finite sample probability (of "success", rejection) for any
particular value of (@,T) could be calculated directly from (25), obviating
any neec| for conducting Monte Carlo experiments to estimate mp. Even with
a and G*(-,+) unknown, approximations to them may be found; and further,
the accuracy of those approximations may be tested,10

From asymptotic theory, one expects that a * 1 as T + » and that

Gtie,T) = 17'/?G(e,T7!/2) (27)
where G(e,T~!/2) is O(T°) (cf. Phillips (1977, p. 474; 1982), Sargan (1980,
p. 1120)). Thus, a might be expanded in powers of T7'/? about T==:

a = a + a, T7742? + aT + wh, (28)
where a, is expected to be unity; and G(-,-) might be expanded in powers of
77/2 and of the elements of 6.1! Truncating both the series for a and
the series for G(e,T7~'/?), the coefficients of the powers and
cross-products of 6, T~'/, and 1n{g/(1-mg)} may be estimated by least
squares, correcting for heteroscedasticity using the weight ay2, i.e.,

from est-imating

10There are two distinct senses in which a and G(-,+) can be known: for the
particular experiments in which mg is estimated, and for any values of
(6,T) in OxT. Clearly, the latter is far more useful. That distinction
also emphasizes the value of choosing an experimental design which covers a
wide range of @xT.

Aneuryn-Evans and Deaton (1980, pp. 284-285) suggest an alternative
framework for analysing Monte Carlo results on statistics for testing
non-nested hypotheses. However, their approach did not prove fruitful.
See Eriesson (1982a) for details.

11 The parameterisation of @ is not unique, and it may be worthwhile
transforming "natural" parameters of the model into parameters which span
the same range as L*(s) before expanding G(-,.), For instance, a parameter
bounded between -1 and 1 (p, say) might better appear in 6 as p/(1-p7); a
parameter bounded from below by zero (07, say) might better appear in @ as
1n(0?).


37

Ericsson, N.R. (1982b) Testing Non-nested Hypotheses in Systems of Linear
Dynamic Economic Relationships, Ph.D. thesis, University of Loncon.

Ericsson, N.R. (1983) "Asymptotic properties of instrumental variables
statistics for testing non-nested hypotheses", Review of Economic
Studies, 50, 2, 287-304.

Ericsson, N.R. (1986) "Post-simulation analysis of Monte Carlo experiments:
Interpreting Pesaran's (1974) study of non-nested hypothesis test
statistics", Review of Economic Studies, forthcoming.

Farebrother, R.W. (1980) "The Durbin-Watson test for serial correlation
when there is no intercept in the regression", Econometrica, 48, 6,
1553-1563; and (1981) "Erratum", Econometrica, 49, 1, 277.

Godfrey, L.G. (1978) "Testing against general autoregressive and moving
average error models when the regressors include lagged dependert
variables", Econometrica, 46, 6, 1293-1301.

Godfrey, L.G. and M.H. Pesaran (1983) "Tests of non-nested regression
models: Small sample adjustments and Monte Carlo evidence", Journal of
Econometrics, 21, 1, 133-154.

Hammersley, J.M. and D.C. Handscomb (1964) Monte Carlo Methods, London,
Chapman and Hall.

Harvey, A.C. (1981) The Econometric Analysis of Time Series, Oxford, Philip
Allan,

Hendry, D.F. (1973) "On asymptotic theory and finite sample experiments",
Economica, 40, 158, 210-217.

Hendry, D.F. (1979) "Predictive failure and econometric modelling in
macroeconomics: the transactions demand for money" in P. Ormerod (ed.)
Economic Modelling, London, Heinemann Education Books, 217-242.

Hendry, D.F. (1980) "Monte Carlo experimentation in econometrics",
Discussion Paper No. 8035, CORE, Université Catholique de Louvain,
Louvain-la-Neuve, Belgium, November.

Hendry, D.F. (1982) "A reply to Professors Maasoumi and Phillips", Journal
of Econometrics, 19, 2/3, 203-213.

Hendry, D.F. (1984) "Monte Carlo experimentation in econometrics",
Chapter 16 in Z. Griliches and M.D. Intriligator (eds.) Handbook of
Econometrics, Amsterdam, North-Holland Publishing Company, volume II,
937-976.

Hendry, D.F., M. Morgan, and F. Srba (1982) "GIVE guide", mimeo, London
School of Economics.

Hendry, D.F. and F. Srba (1978) "Technical manual for GIVE: General
instrumental variables estimation of linear equations with lagged
dependent variables and first order autoregressive errors", mimeo,
London School of Economics, October.


25
Qi = Py = Wy and Q,xt = NX{ = W,xXt = Xt (i=0,1). The expressions
above for o(i) and ax simplify to

(0) = plim, s*'xt'xte*/T ,

(4) = plimy o**xd' (WW, )imtW xte*/T G21), (a18)
and
My = OETXE'(Ip-W, )X46*/ (0527)
= 8*'Xt'Mxt6*/(0,2T) . (A19)

Adopting the following notation:

my = plim, X$'xt/T ,
my = plim, X,'Xt/T ,
my, = plim, X,'X,/T ,
Mo. = plim, X,'X,/T ,
Moo = plimy Xo'X,/T , (A20)

Moi = Mo’, and my, = m4’, then (i) May be written as

(0) = o* my 48*

C4) = OF! mam, ,7*(m; omg *mormi7*)A-Ims* (LB 1) (nat)
Given the population moments of the regressors, o(i) (is0,...,4) may be
calculated; and hence so can ‘be Wi» 4, and 4,, knowing o,*. From those,
the (approximate) asymptotic powers of the tests based on f,; and D, are
derived. (See Appendix C for details.)

The formulae for ¢(i) and A, may be simplified further by noting two
particular features of the structure of X,: that X, is a Tx3 matrix with
X$, X, and X{ all being Tx1 vectors; and that Xt, X, and xt
satisfy the properties given in (8) and (9) where Xt = (Ky Xoyeee Xp),

a
X = (1,1,...,1), and Xt = (21 ,22,+++,27)'. The relevant moment matrix is


24

Di og NG») (A12)

under H,, where

ty = 7 (G(0)tE(1) / (200%e(2))

Wy = £(0(0)#5(1))?27 (AE (2)) }evar(al*) so? , (A13)
and
var (a0*))/a9? = ¥(C(9)+E(1)75(2))/ (5 (0) #5 (2))?
~ HCE (2)78(3))/1((0)#5(1))5(2)}
+ (E(s)7B(uy)/E (2)? (A14)
with

S(i) = 9(4) 7 (i471) (G20) ,

b(o) = Plimy 6*'X$'Nxge*/T ,

(4) = plimy 6*'X$'(Q,Q,)i-1Q,xt6*/T = (4 2:1) =, and
3* = BtyT (8* a non-zero constant) . (A15)
That is, the role of the parameters and variables in the model H, is
exchanged with that of the parameters and variables in the model H,.
(Superfluous parentheses around subscripts and superscripts denote that
those roles have been exchanged in the definition of the subscripted or
superscripted variable.) Likewise, the distribution of f, under H, is
Ps gq F(k,-k,, T-kK2, d3) (A16)
where A, = 5(0)/007- If Z = X, and X, is fixed, f, is exactly distributed
as an F(k,-k,, T-k,, A%) where
AB = BS'X$"(Q2-Q, )X$B3/ 00?
= 8*'X$"(Q.-Q, )X$6%/ (a9?) (A17)
(and note that A, = plim, A%).
in Pesaran's (1974) model, the formulae above may be simplified
because (i) Z = X, and (ii) X, = (X$ : X: Xf). Letting

Wy = Xy(Xq'Xy)71Xy" and My = Ip-Wy (i=0,1,2), Z = X_ implies that


6
The DGP defined by (8) and (9) and the relationships of interest in
(10)-(12) have certain implications for the properties of the statistics
being examined. First, the formulae for their asymptotic distributions

have explicit representations: A;, ,, and w, may be expressed as

Ag = TDo2(1-r?)/oo?, (17)

wi = 7T/ 2b (1-2) *72(14r2)/(2o9r2), (18)
and

WO, = {ar* + (1-r?)(1+r?)?}/ (urs) . (19)

For r? close to unity (relevant for Pesaran's experimental design; see
(13)), u.? = A, and w, =~ 1, so the distribution of D,? is roughly a
x?(1,4,). Because f, is asymptotically distributed as an F(1, T-k,, A;)
(and hence asymptotically as a x?7(1,4,)), D, should have about the same
power as f, for Pesaran's experiments. Second, for a given local
alternative (i.e., for a constant non-zero Devt in (17) and (18)), ‘she
asymptotic powers of the Cox and F tests tend to the nominal size as r? + 1.
Third, because X, is fixed, f, and f, are exactly distributed as
F(ka7ky, T-k,, 0) and F(k,-k,, T-k,, a*) where the finite sample
non-centrality parameter \* is.

wes bot XS" (Wo-Wy )X$Do/ 097 (20)
with Wy = Xq(Xq'Xq)7'Xy" (i=0,1,2) and X$ = (x,, x2, ..., xp)’. These
analytical results are invaluable in interpreting the Monte Carlo

simulation results, as will become apparent in the following sections.

4, Monte Carlo methodology

Cox (1970, Chapters 3 and 6), in his discussion of the empirical
logistic transform, implicitly provides the basis for developing response
surfaces of estimated finite sample probabilities, including both estimated

finite sample powers and estimated finite sample probabilities of type I


17
with acceptable accuracy, no matter whether the relative or absolute
measure of forecast accuracy is used.'9

The response surface for f, presents more of a puzzle. The

restrictions in (32) are not rejected, and there is no indication of serial
correlation in either the unrestricted or restricted response surfaces (and
hence no indication that higher order terms from the expansions of a and
G(@,T7!/2) ought to be present in those response surfaces). However, og”
is significantly greater than unity in both response surfaces (see &,(6)),
possibly indicative of the problems with f, noted above. For comparison,
Pesaran (1981) obtains the following response surfaces for D, and f, when
using a DGP and relationships of interest similar to (8)-(12) but with the

number of non-overlapping regressors in (10) and (11) varying from one to four ,20

L¥(8) = .967L (nq) + AM 2(,034 ~ 7.61277? + 1.12477?) (34)
(.019) (.025) (.80) (.18)

K = 108 R* = .982 G@ = 1.30

T2(1,104) = 3.02 &3(104) = 176. dw = 1.07

L¥(S) = .977L(mg) + AT/2(-.031 + 1.62277? = 2,182,T7?) (35)
(1023) (2026) (290) (115)

K= 108 R? = .978 Ge = 1.52

n2(1,104) = 1.00 &,(104) = 240. dw = 1.44

These response surfaces concisely summarize results for over one hundred
experiments and satisfy many of the criteria (A) and (B,)-(Bs,). However,

as with (31) and (32), the response surface for f, fits worse (in terns of

191n the response surfaces for D,, it was not necessary to include any "terms
of 0(67!)", i.e., terms resulting from the additional approximation in (6)
(see Ericsson (1983)). That suggests that the normal distribution
approximates the asymptotic distribution of D, quite well.

20pesaran (1982), the published version of Pesaran's (1981) working paper,
unfortunately does not include these response surfaces.


32

Table (.1. A comparison of two approximations to the non-central y?(v,A)
distribution with points on the distribution.
Probability
v d Xav,0) Patnaik's Approximation Exact
approximation from (C2) Probability

4 4 1.765 +0399 +0426 +0500
4 10.000 -7191 -7180 +7118
4 17.309 +9492 -9479 +9500
4 24.000 +9913 -9909 +9925
10 10.000 +3178 +3190 33148

7 1 4,000 -1621 ~1634 +1628
1 16.004 +9499 +9495 +9500
16 10.257 -0430 +0439 +0500
16 24.000 -59NT -59N5 +5898
16 38.970 +9482 -9476 +9500

12 6 24.000 +8187 «8178 «8174
18 24.000 +2936 +2926 -2901

16 8 30.000 +7895 +7895 +7880
8 40.000 +9626 +9626 +9632
32 30.000 +0590 +0594 +0609
32 60.000 +8329 +8326 -8316

24 24 36.000 +1556 +1556 +1567
24 48.000 5333 +5333 +5296
24 72.000 +9656 +9656 -9667



33

Table C.II. A comparison of two approximations to the singly non-central
F(v,,v2,4) distribution with points on the distribution.

Probability ]
——4
Vy Ve A | Falvisv2,0) Patnaik's Approximation Exact
approximation from (C6) probability
3 10 4 3.708 +752 +750 +745
4 6.552 2919 +919 +918
16 3.708 +203 +203 +206
16 6.552 +520 520 517
3] 20 4 3.098 +706 -T07 +700
4 4,938 -889 +888 -887
16 3.098 2119 . 2119 +126
16 4,938 +350 +350 +347
51 10 6 3.326 +731 +733 +731
6 5.636 2913, “914 2914
24 3.326 2157 +156 158
24 5.636 +463 +463 461
5] 20 6 2.711 > 665 -668 664
6 4.103 «869 871 .870
24 2.711 +064 +064 +069
ay 4.103 o24y +244 +245,
8] 10 9 3.072 2715 2715 “714
9 5.057 +909 +909 +908
36 3.072 0117 «117 2119
36 5.057 +409 +409 +408
8} 30 9 2.266 -581 581 -578
9 3.173 815 «815 813
36 2.266 014 014 2017
36 3.173 +085 +085 -088



21
the relationship(s) of interest and DGP considered. To evaluate the
closeness of approximation, this paper presents a framework for inference
about response surfaces. Illustrating this methodology, Pesaran's (1974)
Monte Carlo simulations of the finite sample rejection frequencies of the
Cox and F tests are re-examined. Further, adjustments are made in order
that the estimated finite sample powers of various tests may be compared
with each other on a more equal footing in those instances where the
estimated finite sample probability of type I error for one of the tests
differs significantly from the nominal value. Although this paper is
concerned with tests in particular, the methodology is much more general:
similar techniques have been successful in analysing a wide variety of
estimators, both consistent and inconsistent, linear and non-linear
(ef. Engle, Hendry, and Trumble (1985), Campos (1986), Hendry (1984), and
references therein). Although Monte Carlo experimentation does not replace
analysis, the two can complement each other effectively to provide

convenient formulae for interpreting empirical findings.


2

Pesaran (1974) compares the finite sample properties of the Cox (1961,
1962) and F statistics for testing the mis-specification of a static single
equation model. As a basis for analysing his results, Section 2 describes
the asymptotic properties of those statistics. Section 3 interprets
Pesaran's Monte Carlo design in light of Hendry (1984) and notes further
properties of the statistics when the data generation process is the one
selected by Pesaran. Section 4 explains the valuable roles that a
response surface can have as a convenient summary of Monte Carlo
Simulations and as an approximation to the underlying finite sample
distribution function. Further, the section details how to assess the
adequacy of a response surface. Section 5 compares Pesaran's experimental
results directly with what asymptotic theory would predict, and, to the
extent that significant discrepancies exist, attempts to explain them,
employing the methodology from Section 4, Finally, Section 6 proposes
adjustments such that the estimated finite sample powers of the Cox and F
tests may be compared on a more equal footing in those instances where the
estimated finite sample probability of type I error for the Cox statistic
significantly exceeds its nominal value. Although the approach presented
relates to finite sample rejection frequencies in particular, similar
strategies exist for evaluating most properties of estimators and test
Statistics, including biases, simulation standard errors, and estimated

asymptotic standard errors.

2. Statistics for testing non-nested hypotheses

This section summarizes existing analytical results for the two
statistics examined by Pesaran (1974). Consider the two non-nested

hypotheses


IFDP
NUMBER

276

275

274

273

272

271

270

269

268

267

266

40

International Finance Discussion Papers

TITLES

1986

Post-simulation Analysis of Monte Carlo
Experiments: Interpreting Pesaran's (1974)

Study of Non-nested Hypothesis Test Statistics

A Method for Solving Systems of First Order
Linear Homogeneous Differential Equations
When the Elements of the Forcing Vector
are Modelled as Step Functions

International Comparisons of Fiscal Policy:
The OECD and the IMF Measures of Fiscal
Impulse

An Analysis of the Welfare Implications of

Alternative Exchange Rate Regimes: An

Intertemporal Model with an Application
1985

Expected Fiscal Policy and the Recession
of 1982

Elections and Macroeconomic Policy Cycles

Assertion Without Empirical Basis: An
Econometric Appraisal of Monetary Trends
in... the United Kingdom by Milton Friedman

and Anna J. Schwartz

Canadian Financial Markets:
Proposal for Reform

Was It Real? The Exchange Rate Interest
Differential Relation, 1973-1984

The U.K. Sector of the Federal Reserve's
Multicountry Model: The Effects of
Monetary and Fiscal Policies

Optimal Currency Basket in a World of
Generalized Floating: An Application to
the Nordic Countries

The Government's

AUTHOR(s)

Neil R. Ericsson

Robert A. Johnson

Garry Schinasi

Andrew Feltenstein
David Lebow
Anne Sibert

William H. Branson
Arminio Fraga
Robert A. Johnson

Kenneth Rogoff
Anne Sibert

David F. Hendry
Neil R. Ericsson

Garry J. Schinasi

Richard Meese
Kenneth Rogoff

Hali J. Edison

Hali J. Edison
Erling Vardal

Please address requests for copies to International Finance Discussion
Papers, Division of International Finance, Stop 24, Board of Governors of

the Federal Reserve System, Washington, D.C.

20551.


38

Hendry, D.F. and F. Srba (1980) "AUTOREG: A computer program library for
dynamic econometric models with autoregressive errors", Journal of
Econometrics, 12, 1, 85-102.

Jarque, C.M. and A.K. Bera (1980) "Efficient tests for normality,
nomoscedasticity and serial independence of regression residuals",
Economics Letters, 6, 3, 255-259.

Johnson, N.L. (1959) "On an extension of the connexion between Poisson and

x? distributions", Biometrika, 46, 3 and 4, 352-363.

Johnson, J. (1963) Econometric Methods, New York, McGraw-Hill Book
Company.

Kendall, M.G. (1946) The Advanced Theory of Statistics, Volume II, London,
Charles Griffin.

Kendall, M.G, and A. Stuart (1973) The Advanced Theory of Statistics,
Volume 2, Inference and Relationship, London, Griffin, third edition.

Maasoumi, E. and P.C.B. Phillips (1982) "On the behavior of inconsistent
:.nstrumental variable estimators", Journal of Econometrics, 19, 2/3,
183-201.

MacKinnon, J.G. (1983) "Model specification tests against non-nested
alternatives", Econometric Reviews, 2, 1, 85-110.

Mann, H.B. and A. Wald (1943) "On stochastic limit and order
relationships", Annals of Mathematical Statistics, 14, 3, 217-226.

Mizon, G.E. (1984) "The encompassing approach in econometrics", Chapter 6
in D.F. Hendry and K.F. Wallis (eds.) Econometrics and Quantitative
Economics, Oxford, Basil Blackwell, 135-172.

Mizon, G.E. and D.F,. Hendry (1980) "An empirical application and Monte
Carlo analysis of tests of dynamic specification", Review of Economic
Studies, 47, 1, 21-45.

Mizon, G.E. and J.-F, Richard (1986) "The encompassing principle and its
application to testing non-nested hypotheses", Econometrica,
forthcoming.

Nicholls, D.F. and A.R. Pagan (1983) "Heteroscedasticity in models with
lagged dependent variables", Econometrica, 51, 4, 1233-1242,

Nickell, S. (1981) "Biases in dynamic models with fixed effects",
Econometrica, 49, 6, 1417-1426.

Numerical Algorithms Group (NAg) (1977) NAg FORTRAN Library Manual, Mark 6
(FLM6), Oxford, Numerical Algorithms Group Ltd.

Patnaik, P.B. (1949) "The non-central y?- and F-distributions and their
applications", Biometrika, 36, 1 and 2, 202-232.

Pesaran, M.H. (1974) "On the general problem of model selection", Review of
Economic Studies, 41, 2, 153-171.


12

5. Evaluation of Pesaran's estimated finite sample rejection frequencies

Equation (24) above is the foundation for evaluating Monte Carlo
estimates of finite sample rejection frequencies. This section utilizes
(24) both in assessing how close the simulation results are to what
asymptotic theory would predict and in formulating response surfaces to
explain significant deviations between Monte Carlo results and asymptotic
theory.

The "closeness" of an estimated finite sample rejection frequency of a
test statistic to its asymptotic value can be tested using 9(s,mg) from
(24) above where s is the fraction of replications for which the value of
the statistic lies in the critical region and 1g is the probability of the
statistic lying in the critical region asymptotically. The values of the
estimated finite sample powers and estimated probability of type I error
for the Cox and F statistics, their asymptotic values, and the values of
o(s,1q) are given in Table II. Figure 1 strikingly displays how much of
the inter-experimental variation in the finite sample rejection frequencies
is explained by asymptotic theory. Now consider those results for each
statistic in turn,

Pesaran (1974, p. 161) notes that the estimated probability of type I
error for f, never differs significantly from .05, and the corresponding
values of $(s,.05) in Table II confirm that. Further, the sum of squares
of $(s,.05) over Pesaran's (1974) nine experiments is £,(9) = 6.45,
offering additional support to the hypothesis m = .05.12

The estimated probability of type I error for Dy is significantly
different from (and larger than) .05 in several experiments, although the

magnitude of their difference decreases as T increases, in line with

12Note that £,(9) is distributed as a x?(9) for large N (provided that
mn = .05) and that probly?(9)216.92] = .05.


12b
Figure 1

Asymptotic and estimated finite
sample rejection frequencies for

Do; Dy, and fg
1.0
a
~ 8
o
=
3}
a
2 6
a
=
©
a
@ 4-
=
3
= 2r
£
ug 1 I I 1
0 2 4 6 8 1.0
Asymptotic power (ma)
Oo
fa)
Te
ome) T=20
ac 10 —-— I= ome T=40
ES og} ae
us oO | a | !
0 2 4 6 8 1.0
Asymptotic power (mag)
Legend

Asymptotic power or size

— — Rejection frequency for Dg or Dy (T = 20)
— ie Rejection frequency for Dg or Dy (T = 40)
— — — Rejection frequency for f3 (T = 20)
—— Rejection frequency for. f3 (T = 40)


Post-simulation Analysis of Monte Carlo Experiments:
Interpreting Pesaran's (1974) Study
of Non-nested Hypothesis Test Statistics
by

Neil R. Ericsson*

1. Introduction

Monte Carlo studies in econometrics often have been criticized for
imprecision present in estimating the underlying finite sample properties
investigated and for the specificity of the results from the particular
parameter values and sample sizes chosen, so making any conclusions very
tentative at best. Hendry (1984) presents a methodology reducing both and
which aims to obtain "numerical-analytical formulae which jointly summarize
the experimental findings and known analytical results in order to help
interpret empirical evidence and to compute outcomes at other points within
the relevant parameter space" (p. 944). That methodology affects all
aspects of Monte Carlo experimentation: design, simulation, and
post-simulation analysis. Illustrating such post-simulation analysis, this
paper re-examines Pesaran's (1974) Monte Carlo study of statistics for

testing non-nested hypotheses.

*tThis research was supported in part by a grant from the Social Science
Research Council to the Programme in Methodology, Inference and Modelling

in Econometrics at the London School of Economics (1979-81) while I was at
the LSE. I am grateful for the financial assistance from the

S.S.R.C. although the views expressed in this paper are solely the
responsibility of the author and should not be interpreted as reflecting
those of the S.S.R.C., the Board of Governors of the Federal Reserve

System, or other members of their staffs. I have benefited from valuable
discussions with and comments from Julia Campos, Raymond Chapman, David
Hendry, Grayham Mizon, Hashem Pesaran, Denis Sargan, and Frank Srba. I am
grateful to Hashem Pesaran for detailed information on his 1974 Monte Carlo
experiments (including the results used for forecast tests herein), and for
his permission to quote in equations (34) and (35) unpublished material

from Pesaran (1981). All response surfaces were obtained with the computer
program GIVE (Hendry and Srba (1978, 1980), Hendry, Morgan, and Srba (1982)).
This is a revised and shortened version of Ericsson (1982b, Chapter 3) and is
forthcoming in Ericsson (1986) (but without Appendices A, C, and D).


31
From v,, v2, A, and a, the approximation to P in (C6) may be calculated
numerically, e.g., with the NAg (1977) routine GOIBBF, linearly
interpolating for P in (C6) if v* is not integral.

Table C.II compares the approximation given in (C6) with exact values
of the integral in (C5) for values of Vi» V2, and ) appearing in Patnaik
(1949, p. 222, Table 7).. Patnaik's own approximation to the exacz

-

probability involves both linearly interpolating P for non-integral v* and
interpolating P for the value of the lower limit of the integral in (C6).
The latter interpolation only slightly affects the values obtained, and
both approximations perform well over a wide range of powers.

The probability of a normal variate with mean y and variance o? being
less than a certain critical value z is

*
Zz

(2n)7172 f exp(-u?/2) du (c8)
=o
where z* = (z-y)/o . This integral is calculated directly with the NAg
(1977) routine SI5ABF. The integral corresponding to the upper tail is

calculated in a similar manner with NAg (1977) routine S15ACF.


20

Ideally, one would choose critical values for D, and f, giving
identical probabilities of type I error and use those critical values when
estimating the finite sample powers of D, and f,, but that is not feasible
because the finite sample distribution of D, is not known. As an
alternative, one could solve for the critical value of f, which would
give it the same probability of type I error (ap, say) as the estimated
probability of type I error for D, (sp, say).2! (Table III lists hose
critical values for T = 20 and T = 40.) Then, using those critical values,
the power of f, could be estimated for each Monte Carlo experiment.
However, because that approach would depend upon knowing the values of the
dependent variable and regressors for all replications of all experiments
(information which is not currently available), a somewhat inferior method
is adopted, namely, calculating the power of f, using its asymptotic
non-centrality parameter (A,), evaluated at the adjusted critical value.
Those (adjusted) powers for f, and the estimated powers of D, are presented
in the last two columns of Table III and indicate much smaller differences
between the powers of the tests: based on D, and f, than are apparent in
Pesaran's (1974, p. 162) Table 1. Even so, f, does appear slightly more

powerful than D, at higher powers, and vice versa at lower powers.

Te Concluding remarks

This paper describes and implements an approach for obtaining
numerical-analytical formulae (response surfaces) which integrate existing
analytical knowledge with experimental results. Response surfaces can help
summarize and interpret Monte Carlo simulations, and may reasonably

approximate the unknown finite sample conditional probability formula for

2\cf. sargan (1976, pp. 444ff) who suggests how to improve the efficiency of
the estimated probability by using a control variate.


ABSTRACT

"Monte Carlo experimentation in econometrics helps ‘solve!
deterministic problems by simulating stochastic analogues in which the
analytical unknowns are reformulated as parameters to be estimated."
(Hendry (1980)) With that in mind, Monte Carlo studies may be divided
operationally into three phases: design, simulation, and post-simulation
analysis. This paper provides a guide to the last of those three,
post-simulation analysis, given the design and simulation of a Monte Carlo
study, and uses Pesaran's (1974) study of statistics for testing non=nested
hypotheses to illustrate the techniques described. A statistic is derived
for testing for significant deviations between the asymptotic and
(observed) finite sample properties. Further, that statistic provides the
basis for analysing discrepancies between the finite sample and asymptotic
properties using response surfaces. The results for Pesaran's study
indicate the value of asymptotic theory in interpreting finite sample
properties and certain limitations for doing so. Finally, a method is
proposed for adjusting the finite sample sizes of different test statistics
so that comparisons of their power may be made. Extensions to other finite

sample properties are indicated.


27
It is interesting to note that
wi? = (6%)?(1-r?)(14r?)?/(Yo, 2h") 5 (a28)
so that for r? close to unity,

wi? = (8*)?(1-r?)/o9? = As (a29)
and

wo = 1. (A30)
Hence D,? and f, have nearly identical asymptotic distributions; and, given
a symmetric two-tailed test for D,, D, and f, should have roughly the same
power. Even when r? is not so close to unity, the approximations in (A29)
and (A30) are still reasonable, as can be seen in Table A.I below for

Pesaran's published results.

Table A.I. A comparison of the parameters in the approximate asymptotic
distribution of D, (w, and y,) with the one in the asymptotic
distribution of f, (A,). R? for the DGP is .80.

Parameter
Sample r? w, wi? ds
size
-90 1.23 8.91 8.00
n=20 95 1.11 4.21 4.00
99 1.02 81 -80
+90 1.23 17.83 16.00
n=40 +95 1.11 8.43 8.00
+99 1.02 1.62 1.60
+90 1.23 35.65 32.00
n=80 +95 1.11 16.85 16.00
+99 1.02 3.23 3.20



36

References

Aneuryn-Evans, G. and A. Deaton (1980) "Testing linear versus logarithmic
regression models", Review of Economic Studies, 47, 1, 275-291.

Box, G.E.P. and D.A. Pierce (1970) "Distribution of residual
autocorrelations in autoregressive-integrated moving average time
series models", Journal of the American Statistical Association, 65,
332, 1509-1526.

Campos, J. (1986) "Finite sample properties of the instrumental variables
estimator for dynamic simultaneous equation sub-systems with ARMA
disturbances", Journal of Econometrics, forthcoming.

Chow, G.C. (1960) "Tests of equality between sets of coefficients in two
linear regressions", Econometrica, 28, 3, 591-605.

Cochran, W.G. and G.M. Cox (1957) Experimental Designs, New York, John
Wiley and Sons, second edition.

Cox, D.R. (1958) Planning of Experiments, New York, John Wiley and Sons.

Cox, D.R. (1961) "Tests of separate families of hypotheses" in J. Neyman
(ed.) Proceedings of the Fourth Berkeley Symposium on Mathematical
Statistics and Probability, Berkeley, University of California Press,
volume I, 105-123.

Cox, D.R. (1962) "Further results on tests of separate families of
hypotheses", Journal of the Royal Statistical Society, Series B, 24,
2, 406-424.

Cox, D.R. (1970) The Analysis of Binary Data, London, Methuen.

Cramér, H. (1946) Mathematical Methods of Statistics, Princeton, Princeton
University Press.

Durbin, J. and G.S. Watson (1950) "Testing for serial correlation in least
squares regression. I", Biometrika, 37, 3 and 4, 409-428,

Durbin, J. and G.S. Watson (1951) "Testing for serial correlation in least
squares regression. II", Biometrika, 38, 1 and 2, 159-178.

Engle, R.F. (1982) "Autoregressive conditional heteroscedasticity with
estimates of the variance of United Kingdom inflation", Econometrica,
50, 4, 987-1007.

Engle, R.F., D.F. Hendry, and D. Trumble (1985) "Small-sample properties of
ARCH estimators and tests", Canadian Journal of Economics, 18, 1,
66-93.

Ericsson, N.R. (1982a) "Testing linear versus logarithmic regression
models: a comment", Review of Economic Studies, 49, 3, 477-481.



22
Appendix A: The distributions of D,, f,, D,, and f, under Hy: in general
and for Pesaran's (1974) model in particular

This appendix derives the distributions of the test statistics of
interest, using the formulae in Ericsson (1983). Further, for Pesaran's
(1974) data generation process and relationships of interest, formulae
explicitly in terms of his design parameters are found. The notation is
the same as that in Ericsson (1983) (and that above) except that the
econometric sample size is denoted by T (not n), the regression
coefficients are 8; (not aj), and N is the matrix Z(Z'Z)7'Z' (not the
number of replications).

From Ericsson (1983),

De a N(0,1) (a1)
and

f, 4 F(k,-ky, T-k,, 0) (a2)
under Hy. Further,

Do a Nuvo »Wo) (a3)

under H,, where

Wo = ~ (otti)/(20.v%e2) 5
Wo =  {(Eo+b,)?2/(4C,)}evar(q*)/o,2 , and
var(q¥)/o,? = W(totti~Sa)/ (Cott)? =~ W(Canba)/{(Totlr Ea}
+ (Cs-by)/b27 , (Ada)
with
Ci = 4 7 Oi41 (i200) ,
do = plim, 6'X{'NX}6/T ,
6; = plim, 6'Xf'(Q,Q,)i71Q,xt6/T (Gi2z1) ,
6 = BtvYT (6 a non-zero constant) ,
Qo = NPy = NX4(Xq'NX{)71XG'N (i= 0,1, 2) , and
N = 2(Z'Z)71z' , (A5)

Z being the Txm matrix of observations on m instruments, which is equal to


11
Specification against the general one helps test for the presence of such
factors in the e's of the final specification.

(B;) The error e is normally distributed.

(B,) The e's are serially independent for any ordering of experiments
specified prior to simulation. That follows from the independence of e€
across experiments. If H(-+,+)#G(-,-+) and experiments are ordered to be
(e.g.) increasing in values of @ and T, terms in e involving 6, T7!/?, and
An{ta/(1-1g)} may induce serial correlation and/or heteroscedasticity in
the e's.

(Bs) H(+,*) is constant over regions of the parameter space which
were not included in the estimation of (29).

Table I lists the bulk of the test statistics reported below; the
convention used is that €(q) and nj(q,p) denote statistics which have
central y?(q) and F(q,p) distributions respectively under a common null and
against the ith alternative. Thus, &)(q) and ne(q,K-m-q) both test for
qth-order residual autocorrelation. There are K experiments and m
regressors in the response surface under the null hypothesis.

The extent to which (A) and (B,)-(B;) are not satisfied reflects the
degree of approximation of the response surface to the underlying
conditional probability formula (response function) although the power of
tests of (A) and (B,)-(Bs) depends crucially on the number of replications
per experiment, the experimental design (i.e., the points in oxT examined),
and the choice of DGP and @xT., Finally, even if any of (A) and (B,)-(B,)
are rejected, the response surface still has certain desirable properties
as an approximation to the unknown function G(-,+) (White (1980b,
pp. 155-157)) and it still may account for (and so summarize) much of the

inter~experiment variation.


12a

the estimated type I errors (s) of D, and f,, and, in each case, the value

of a statistic 9(s,°) far testing the closeness of the asymptotic and
finite sample results.

Table II. The asymptotic powers (ng) and estimated finite sample powers (s) of D, and f,,

Dy Do f, f,

Sample | r? 8 ta | 68,7) s? | 9(s,.05) s Ta | $(S,mQ) 3¢ | 9(s,.05)

Size

+9{ 8180 | .822 | -.20¢]  .082 3.18 | 656 [ .768 | 5.85*] ons -.0
(.017)4 (.012) (.021) (.010)

T=20 |. | .628 | .535 | 4.16*| .086 3.58" | 436 | .U58 | -.99 | 056 53
(022) (.013) (.022) (.010)

+9} 1% 49 | 2.91*] 072 2.17% 122 129 | -.52 2052 an
(.018) (.012) (.015) (.010)

+9] .%8 | .979 | -1.58 | .076 2.58" | 932 | .984 | -8.38*] cus ~.30
(008) (.012) (.011) (.010)

T+40 |. | .830 | .815 +9 | .080 2.98" | .764 | .797 | -1.81 052 ab
(.017) (.012) (.019) (.010)

-99{ 306 | .2u8 | 2,98"! “056 53 228 | .219 46 [cue =.
(.021) (.010) (.019) (.009)

-90} 1.000 1.000 - 068 1.76 1.000 1.000 - CHG 51
(.000) (011) (.000) (.co9)

T-80 |. | .980 | .979 -31 | .050 =.09 +968 | .987 | -3.50") 034 | -1.75
(.006) (.010) (.008) (.C08)

-99| 476 436 1.80 +050 -.09 4O4 44 -.01 038 71.34
(.022) (.010) (.022) (.009)

Notes: a. The nominal size of D, and f, is .05.

b. The total number of replications per experiment (N) is 500 for each experiment,

d.

with the multiple correlation coefficient of the DCP (R2) being .8.
c. Under the null hypothesis that =n, (or that 1.05 for Do and f.

Asterisks denote results for which the null hypothesis is rejected at

the 5% level.
Simulation standard errors of estimated probabilities are in parentheses.

2)» (85°) 5 N(O,1).