Reminder about Potentially 

Serious Problems with a Type 

of Blocked ANOVA Analysis

Steve Verrill

David E. Kretschmann

United States Department of Agriculture

Forest

Service

Forest Products

Laboratory

Research Paper

FPL–RP–683

June

2017

0.0 0.2 0.4 0.6 0.8 1.0

Probability of rejection

Non-centrality parameter index

5 10 15 20

th

ps, ancova

ps, 2-way

no ps, 1-way

ps, 1-way, no rho
Abstract

A type of blocked experiment has the potential of being 

poorly designed and/or analyzed. Verrill and coworkers 

have referred to such an experiment as a “predictor sort” 

experiment. David and Gunnink described the procedure 

as “artificial pairing.” In textbooks it is sometimes referred 

to as a “matched pair” or “matched subjects” design. The 

associated design process is also sometimes described as 

“forming blocks via a concomitant variable.” Improperly 

designed or analyzed predictor sort experiments can be 

associated with incorrect or inadequate power calculations and sample sizes, incorrect tests of hypotheses, and 

incorrect confidence intervals. In this paper we review the 

main results in the literature, add a section on multiple 

comparisons, and present results from power and confidence interval coverage simulations that emphasize the 

importance of the proper design and analysis of predictor 

sort experiments.

Keywords: Predictor sort sampling, artificial pairing, 

matched pairs, matched subjects, concomitant variable, 

blocked ANOVA, analysis of covariance, maximum likelihood estimation

June 2017

Verrill, Steve; Kretschmann, David E. 2017. Reminder about potentially 

serious problems with a type of blocked ANOVA analysis. Research 

Paper FPL-RP-683. Madison, WI: U.S. Department of Agriculture, Forest 

Service, Forest Products Laboratory. 117 p.

A limited number of free copies of this publication are available to the 

public from the Forest Products Laboratory, One Gifford Pinchot Drive, 

Madison, WI 53726-2398. This publication is also available online at 

www.fpl.fs.fed.us. Laboratory publications are sent to hundreds of 

libraries in the United States and elsewhere.

The Forest Products Laboratory is maintained in cooperation with the 

University of Wisconsin. 

The use of trade or firm names in this publication is for reader information and does not imply endorsement by the United States Department of 

Agriculture (USDA) of any product or service.

In accordance with Federal civil rights law and U.S. Department of Agriculture (USDA) civil rights regulations and policies, the USDA, its Agencies, 

offices, and employees, and institutions participating in or administering USDA programs are prohibited from discriminating based on race, color, national 

origin, religion, sex, gender identity (including gender expression), sexual orientation, disability, age, marital status, family/parental status, income derived 

from a public assistance program, political beliefs, or reprisal or retaliation for prior civil rights activity, in any program or activity conducted or funded by 

USDA (not all bases apply to all programs). Remedies and complaint filing deadlines vary by program or incident.

Persons with disabilities who require alternative means of communication for program information (e.g., Braille, large print, audiotape, American Sign 

Language, etc.) should contact the responsible Agency or USDA’s TARGET Center at (202) 720–2600 (voice and TTY) or contact USDA through the 

Federal Relay Service at (800) 877–8339. Additionally, program information may be made available in languages other than English.

To file a program discrimination complaint, complete the USDA Program Discrimination Complaint Form, AD-3027, found online at http://www.ascr.usda.

gov/complaint_filing_cust.html and at any USDA office or write a letter addressed to USDA and provide in the letter all of the information requested in 

the form. To request a copy of the complaint form, call (866) 632–9992. Submit your completed form or letter to USDA by: (1) mail: U.S. Department of 

Agriculture, Office of the Assistant Secretary for Civil Rights, 1400 Independence Avenue, SW, Washington, D.C. 20250–9410; (2) fax: (202) 690–7442; or 

(3) email: program.intake@usda.gov.

USDA is an equal opportunity provider, employer, and lender.

Contents

1 Introduction ......................................................................1

2 Hypothesis Tests...............................................................2

3 Confidence Intervals on Means........................................6

4 Confidence Bounds on Quantiles...................................11

5 Scheffé and Tukey Multiple Comparison 

Procedures after a Predictor Sort Allocation......................13

6 Web and R Programs......................................................20

7 Summary.........................................................................21

References..........................................................................22

8 Appendix A—Noncentrality Parameter in the 

Hypothesis Test Simulation ...............................................23

9 Appendix B—An Analysis of Covariance 

Power Calculation..............................................................24

10 Appendix C—Relations between Analysis of 

Covariance and Maximum Likelihood Estimators............28

11 Appendix D—A Maximum Likelihood 

Variance Calculation ..........................................................31

12 Appendix E—Asymptotic Distribution of the 

Analysis of Covariance Estimator of μj after a 

Predictor Sort Allocation....................................................32

13 Appendix F—Proof of the Tukey Multiple 

Comparison Theorem (Theorem 5)....................................34

14 Appendix G—Proof of Theorem 1 in the 

Multi-factor Case ...............................................................36

Tables.................................................................................42

Figures...............................................................................78
Reminder about Potentially Serious Problems

with a Type of Blocked ANOVA Analysis

Steve Verrill, Mathematical Statistician

David E. Kretschmann, Research General Engineer

Forest Products Laboratory, Madison, Wisconsin

1 Introduction

A type of blocked experiment has the potential of being poorly designed and/or analyzed. Verrill

(1993, 1999) and Verrill et al. (2004) referred to such an experiment as a “predictor sort” experiment. David and Gunnink (1997) described the procedure as “artificial pairing.” In textbooks

it is sometimes referred to as a “matched pair” or “matched subjects” design. The associated

design process is also sometimes described as “forming blocks via a concomitant variable.” In a

wood research context, Warren and Madsen (1977) described the specimen allocation procedure as

follows:

One can take steps, however, to ensure that the inherent [initial] strength distributions

of test and control samples are reasonably equivalent. Indeed, failure to do so can only

throw doubt on the results.

Specifically, then, all the boards in the experiment are ordered from weakest to strongest

as nearly as can be judged from their moduli of elasticity, knot size, and slope of grain.

To divide the material into J equivalent groups the first J boards, after ordering, are

taken and randomly allocated one to each group. This is repeated with the second,

third, fourth, etc., sets of J boards. The strength distributions of the resulting groups

should then be essentially the same.

Here the response is lumber strength after a treatment, and the predictor/concomitant used to

form blocks (of size J) would be some combination of lumber stiffness, knot size, and slope of grain

(all of which can be measured nondestructively prior to specimen allocation).

In an agricultural context, the predictor/concomitant variable might be, for example, animal

age, initial animal weight, or plot fertility in a previous trial. In a behavioral or educational context,

the predictor might be, for example, IQ or performance on a pre-test.

In this paper we refer to this type of design as a “predictor sort” design (because we sort

specimens on the basis of a predictor that is correlated with the response and then form blocks

via collections of specimens with adjacent predictor values). Our theory will be established for the

case in which the predictor and the response have a joint bivariate normal distribution.

Verrill (1999) cited discussions of this type of experiment in example 3.3 of Cox (1958), section

8.2 of Steel and Torrie (1960), section 5.1 of Kirk (1968), section 13.17 of Finney (1972), example

11.3 of Ostle and Mensing (1975), chapter 6 of Myers (1979), and example 6.13.1 of Snedecor and

Cochran (1989). A more recent sampling of statistical texts found such experiments discussed in

Kerlinger and Lee (1999), van Zutphen et al. (2001), example 5.1 of Toutenburg (2002), section 4.3

of Ruxton and Colegrave (2006), problem 3.8 of Casella (2008), Cozby and Bates (2011), Tuckman

and Harper (2012), and section 8.1 of Kirk (2013).

Among the variables suggested as predictors/concomitants to be used to form blocks were

age, reaction time, initial weight, concentration of blood constituent, degree of disease, time since

1
college, IQ, scores on a cognitive ability measure, grade point average, prior school performance,

and pretest achievement.

Improperly designed and/or analyzed, predictor sort experiments can be associated with incorrect/inadequate power calculations and sample sizes, incorrect tests of hypotheses, and incorrect

confidence intervals. Verrill (1993) and David and Gunnink (1997) focused on potential problems

with hypothesis tests given a predictor sort design. Verrill (1999) focused on confidence intervals

on treatment means. Verrill et al. (2004) focused on confidence intervals on quantiles. Because

incorrect predictor sort designs and analyses can have serious adverse effects on decision-making,

and because this fact has not yet become common knowledge among statisticians (or at least among

textbook authors), in this paper we review the main results in the literature, add a section on multiple comparisons, and present the results from power and confidence interval coverage simulations

that emphasize the importance of the proper design and analysis of predictor sort experiments.

In Section 2 we focus on tests of hypotheses. In Section 3 we discuss confidence intervals

on treatment means. In Section 4 we discuss confidence intervals on quantiles. In Section 5 we

discuss Scheff ́e and Tukey multiple comparison tests and the corresponding simultaneous confidence

intervals. In Section 6 we describe web and R programs that we have written to aid in the design

and analysis of predictor sort experiments.

In this paper we focus on predictor sort designs, and special predictor sort analyses of data obtained from predictor sort experiments. We contrast the performance of the predictor sort analyses

with standard ANOVA analyses. However, in our simulations and theoretical work, we

also address analysis of covariance and maximum likelihood solutions to the problems.

We find that an unmodified analysis of covariance works well for tests of hypotheses, but not for

confidence intervals on treatment means. We further find that a maximum likelihood approach to

confidence intervals yields correct asymptotic coverages but that predictor sort coverages approach

nominal coverages more rapidly than do maximum likelihood coverages. Further, a maximum

likelihood approach is considerably more complex and does not readily reveal the 1 − ρ

2 + ρ2/J

asymptotic factor (first discussed in Section 3) that is associated with all three (predictor sort,

analysis of covariance, and maximum likelihood) approaches.

2 Hypothesis Tests

We first set some useful notation. Here, for ease of exposition, we restrict ourselves to the onefactor case. Let Yij denote the response for the ith block, i ∈ {1, . . . , I}, of the jth treatment, j ∈

{1, . . . , J}. Let ρ denote the correlation between the predictor/concomitant, X, and the response,

Y . We assume that, prior to treatment, X and Y have a joint bivariate normal distribution.

In a non predictor sort case, the probability model for a blocked ANOVA would be

Yij = μ·j + μi· + σY × ij (1)

where μ·1, . . . , μ·J denote the treatment effects, μ1·, . . . , μI· denote the block effects, and the ij are

i.i.d. N(0,1)’s. In a predictor sort case, we have n = JI specimens. To allocate these specimens,

we order the X’s and randomly assign the J specimens associated with the lowest X values, to the

first block, the specimens associated with the next J lowest values to the next block, and so on. In

this case, the correct probability model is

Yij = μj + σY



ρ



Xk(i,j),n − μX



/σX +

p

1 − ρ

2Pij

(2)

where μ1, . . . , μJ denote the treatment effects; for a fixed i, the J k(i, j) are a randomization of the

2
elements of {(i − 1)J + 1, . . . , iJ}; Xl,n denotes the lth order statistic among the X’s; and the Pij

are i.i.d. N(0,1)’s that are independent of the X’s.

The differences between models (1) and (2) (and, in particular, the fact that Xk(i,j1),n−Xk(i,j2),n

tends to be smaller than an arbitrary X1 − X2 and yet not equal to 0) are the source of both the

advantages and the problems associated with predictor sort experiments. (More detailed heuristic

discussions are provided in Verrill (1993), Verrill and Green (1996), Verrill (1999), and Verrill et

al. (2004).)

Verrill (1993) proved the following theorem on hypothesis testing following a predictor sort

allocation.

Theorem 1

Assume that, prior to treatment, the predictor variable and the variable of interest have a joint

bivariate normal distribution with correlation ρ. Let the allocation of samples be as described in

Section 1. (For the multi-factor case, enough adjacent (in predictor values) experimental units are

chosen at a time to provide one specimen for each cell in a block.) Then, for a factor with J levels,

for 0 ≤ ρ < 1, the asymptotic null distribution of the ANOVA test statistic that treats the groups

of adjacent (in predictor values) experimental units as a block is χ

2

J−1

/(J − 1). The asymptotic

distribution of the ANOVA test statistic that ignores the block structure generated by these groups

is (1 − ρ

2

)χ

2

J−1

/(J − 1).

Proof

See Verrill (1993) for a one-factor proof, and Appendix G for a multi-factor proof.

Because of this asymptotic behavior, if we analyze a predictor sort experiment as a blocked

ANOVA (where the blocks are formed of specimens with similar predictor/concomitant values —

“matched subjects”) and I is sufficiently large, the nominal size of the test will be approximately

equal to the true size. (Actually, for very large ρ values, the true size is reduced from the nominal

size even for fairly large samples. See the web version of Table 1 referenced below.) However, if we

ignore the blocks in our analysis, the actual size can be much lower than nominal size and power

will suffer significantly. (We essentially end up comparing (1−ρ

2

)χ

2

J−1

random variables with χ

2

J−1

critical values.)

To explore these effects, we have performed a large power simulation of the one-factor case.

For all combinations of X,Y correlations 0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, and 0.99; number of

treatments, J, equal to 2, 3, 5, 7, 9, 11, and 20; sample sizes, I, equal to 3, 5, 10, 20, and 40; and

21 noncentrality parameters, we performed 40,000 trials. We created two versions of each of the

resulting data sets. One version was created by allocating the specimens in a data set to the J

treatment conditions via a standard randomization. The second version was created by allocating

the specimens in a data set to the J treatment conditions via a predictor sort.

We then performed performed seven hypothesis tests on the data sets, and two “theoretical

power” calculations:

1. A standard one-way analysis of variance on the non predictor sort version of the data set.

(The estimated power is reported in column 5 of Table 1.)

2. A standard (and thus incorrect) one-way analysis of variance on the predictor sort version of

the data set. (The estimated power is reported in column 6 of Table 1.)

3. A corrected one-way analysis of variance on the predictor sort version of the data set. The

corrected F statistic is the standard one-way statistic divided by 1 − ρˆ

2

. (The estimated

power is reported in column 7 of Table 1.)

3
4. A second corrected one-way analysis of variance on the predictor sort version of the data

set. The corrected F statistic is the standard one-way statistic divided by 1 − ρ

2

true. (The

estimated power is reported in column 8 of Table 1.)

5. A two-way analysis of variance on the predictor sort version of the data set. (The blocks are

formed by specimens with adjacent [randomized within the block] values of the predictor.)

(The estimated power is reported in column 9 of Table 1.)

6. An analysis of covariance on the non predictor sort version of the data set. (The estimated

power is reported in column 10 of Table 1.)

7. An analysis of covariance on the predictor sort version of the data set. (The estimated power

is reported in column 11 of Table 1.)

8. The “theoretical power” for a a corrected one-way analysis of variance on a predictor sort

version of the data set:

Prob NCFJ−1,J(I−1)(γ) > F −1

J−1,J(I−1)(1 − α)



(3)

where NCF denotes a non-central F distribution function, γ =

PJ

j=1 I(μj −μ ̄·)

2/(σ2

Y

(1−ρ

2

))

is the non-centrality parameter of the noncentral F, and F

−1 denotes the inverse of a central

F distribution function. (The calculated “theoretical power” is reported in column 12 of

Table 1.)

9. The “theoretical power” for a two-way ANOVA on a predictor sort version of the data set:

Prob NCFJ−1,(J−1)(I−1)(γ) > F −1

J−1,(J−1)(I−1)(1 − α)



(4)

(The same non-centrality parameter is used in both (3) and (4).) (The calculated “theoretical

power” is reported in column 13 of Table 1.)

The results of these simulations for the J = 2, 5; I = 3, 5, 10, 20, 40; ρ = 0.5, 0.7, 0.9 cases are

presented in Table 1. The results for the remaining cases can be found at

http://www1.fpl.fs.fed.us/ps15 table1.html. Plots that present a portion of the results of

these simulations (J = 2, 5; I = 10, 20; ρ = 0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99) are displayed as

Figures 1-32. In these plots, the “noncentrality parameter index” is the m in column 4 of the

corresponding power table. See Appendix A for a discussion of this index. “th” is the “theoretical

power” calculated by (3) and presented in column 12 of the power table. “ps, ancova” is the power

of an analysis of covariance after a predictor sort allocation (column 11 of the power table). “ps,

2-way” is the power of a blocked analysis of variance after a predictor sort allocation (column 9 of

the power table). “no ps, 1-way” is the power of a one-way analysis of variance after a standard

(non predictor sort) random allocation of specimens (column 5 of the power table). “ps, 1-way, no

rho” is the power of an uncorrected one-way analysis of variance after a predictor sort allocation

(column 6 of the power table).

An analysis of these tables and plots yields the following conclusions:

1. Large increases in statistical power and/or sample size reductions can be gained by performing

a predictor sort allocation and analysis. These improvements become larger as the correlation,

ρ, between the predictor and the response increases. Specifically, if n samples are needed to

achieve a given power when predictor sort allocation is not used, approximately (1 − ρ

2

)n

4
samples are needed to achieve the same power when predictor sort allocation is used. Thus,

for example, a 0.7 correlation yields, roughly, a halving of necessary sample size. (To see

this, consider the ρ = 0.7 entries in Table 1 and at the ps15 table1.html web address

provided above. For m in column 4 of the tables equal to, for example, 7, 12, 16, and 21, and

I = 5, 10, 20, compare the powers listed in columns 8, 9, 11, 12, and 13 with the corresponding

powers in column 5 for I = 10, 20, 40.)

2. It is a statistical blunder to perform a predictor sort allocation and then follow the allocation

with a standard non predictor sort analysis of variance. Such an approach can considerably

reduce power. (To see this, compare the values in column 6 of Table 1 with the values in

columns 9, 11, and 12, or compare the ps, 1-way, no rho lines in Figures 1 through 32 with

the ps, 2-way; ps, ancova; and th lines.)

3. After a predictor sort allocation has been performed, either an analysis of covariance or a

blocked analysis of variance should be performed. For ρ ≤ 0.8 and I ≥ 10, the blocked analysis

of variance performs almost as well as the analysis of covariance. For higher ρ and/or smaller

I, the analysis of covariance performs better. (To see this, compare columns 9 and 11 in

Table 1.)

4. For ρ ≤ 0.8 and I ≥ 10, the power of a blocked ANOVA can be well approximated by (4).

For higher ρ, (4) overestimates the power available from a blocked ANOVA (especially for

lower I). (To see this, compare columns 9 and 13 in Table 1.)

5. It is well known (also see results (47), (49), and (52) in Appendices B and C) that the power

of a one-factor analysis of covariance for testing the hypothesis μ1 = . . . = μJ is given by

Prob(NCFJ−1,IJ−(J+1)(γ) > F −1

J−1,IJ−(J+1)(1 − α))

where, in our case, σ

2

ANCOVA = σ

2

Y

(1−ρ

2

), γ is the noncentrality parameter of the noncentral

F, and

σ

2

ANCOVA × γ =

X

J

j=1

I(μj − μ ̄·)

2 −





X

J

j=1

I(μj − μ ̄·)( ̄x·j − x ̄··)





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

≥

X

J

j=1

I(μj − μ ̄·)

2 −





X

J

j=1

I(μj − μ ̄·)

2X

J

j=1

I( ̄x·j − x ̄··)

2



 /

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

=

X

J

j=1

I(μj − μ ̄·)

2



1 −

X

J

j=1

I( ̄x·j − x ̄··)

2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2



 (5)

(The inequality in (5) is due to the Cauchy-Schwarz theorem.)

The simulations established that for I ≥ 5, this power is well approximated by

Prob(NCFJ−1,J(I−1)(ˆγ) > F −1

J−1,J(I−1)(1 − α))

where ˆγ =

PJ

J=1 I(μj −μ ̄·)

2/(σ2

Y

(1−ρ

2

)) is an approximation to the noncentrality parameter

of the noncentral F. (To see this, compare columns 11 and 12 of Table 1.)

5
A listing of the simulation program that produced the Table 1 power estimates can be obtained

at http://www1.fpl.fs.fed.us/ps15 powersim code.html. A web-based simulation program

that can be run on additional cases (including multi-factor cases) can be found at

http://www1.fpl.fs.fed.us/pspower.html.

It can be argued that in a predictor sort situation a statistician would undoubtedly perform a

blocked analysis (or an analysis of covariance using the predictor/concomitant as the covariate).

However, some authors of statistical texts for non-statisticians appear to treat “matched subject”

allocations as good experimental practice regardless of the subsequent analyses. (We have seen

two recent texts that discussed matching, t-tests, and ANOVAs, but not paired t-tests or blocked

ANOVAs.) Given that very poor power can result if a predictor sort allocation is analyzed via

an unblocked ANOVA, authors of (at least) statistical texts for non-statisticians need to make

this clear. This is especially true for fields in which concomitants might be highly correlated with

responses.

3 Confidence Intervals on Means

Verrill (1999) established the following theorem.

Theorem 2

Assume that, prior to treatment, the predictor variable and the variable of interest, Y , have a

joint bivariate normal distribution with correlation ρ. Denote the variance of Y by σ

2

Y

. Suppose

that there are I blocks and F factors with K1, . . . , KF levels. Let the allocation of samples be

as described in Section 1. (For a multiple factor case, enough adjacent experimental units would

be chosen at a time to provide one specimen for each cell in a block.) Let Y ̄

·j1·...· be the standard

estimate of the mean response for the j1th level of factor 1. Then

p

I × K2 × . . . × KF



Y ̄

·j1·...· − E



Y ̄

·j1·...·

 D

→ N



0, σ2

Y

(1 − ρ

2 + ρ2

/K1)



as I → ∞. The analogous results hold for factors 2, . . . , F.

Why can this result lead to problems?

If the predictor sort nature of an experiment is neglected, then the confidence interval that is

constructed for the mean response associated with level j1 of factor 1 is

y ̄·j1·...· ± t × s/p

I × K2 × . . . × KF (6)

where t is the appropriate critical value and s is the root mean residual sum of squares from the

ANOVA. Verrill (1993) established (see the appendix of the 1993 paper, or Appendix G of the

current paper) that in a predictor sort case, if the problem is treated as a K1 × . . . × KF ANOVA

with I replicates per cell, the mean residual sum of squares, s

2

unblocked, satisfies

s

2

unblocked

p→ σ

2

Y

(7)

as I increases to infinity. If the problem is treated as one involving I blocks with one replicate per

cell, the mean residual sum of squares, s

2

blocked, satisfies

s

2

blocked

p→ (1 − ρ

2

)σ

2

Y

(8)

where ρ is the correlation between the predictor used in the sort and Y .

6
But by Theorem 2 above, the appropriate large sample value for s in (6) is

σY

p

1 − ρ

2 + ρ2/K1

rather than σY or σY

p

1 − ρ

2. This discrepancy is the source of the coverage problems.

Let

Rub(ρ, J) ≡ 1/



(1 − ρ

2 + ρ2

/J)

1/2

and

Rb(ρ, J) ≡



(1 − ρ

2

)/(1 − ρ

2 + ρ2

/J)

1/2

(Notice that we are switching from the “K” treatments notation of the 1999 paper to a “J”

treatments notation here.)

In Figure 33 values of Rub(ρ, J) are plotted. These R values approximate the factor by which

confidence intervals on treatment means are incorrectly inflated when a standard unblocked ANOVA

is performed after a predictor sort allocation.

In Figure 34 values of Rb(ρ, J) are plotted. These values approximate the factor by which

confidence intervals on treatment means are incorrectly deflated when a standard blocked ANOVA

is performed after a predictor sort allocation.

In Figure 35 values of

2 × Φ



Φ

−1

(.975) × Rub(ρ, J)



− 1

are plotted, where Φ denotes the cumulative distribution function of a N(0,1). These values approximate the actual confidence levels that are associated with nominal 95% confidence intervals

on treatment means in the unblocked case.

Finally, in Figure 36 values of

2 × Φ



Φ

−1

(.975) × Rb(ρ, J)



− 1

are plotted. These values approximate the actual confidence levels that are associated with nominal

95% confidence intervals on treatment means in the blocked case.

From these plots it is clear that, given a predictor sort design, for higher ρ values, the widths

and coverages of confidence intervals on treatment means produced by standard ANOVA analyses

can be poor.

Verrill (1999) suggested two possible fixes to incorrect confidence intervals on the μj ’s (onefactor treatment means) in the predictor sort case. First, he noted that the s in (6) could be

“corrected” by multiplying it by an estimate of p

1 − ρ

2 + ρ2/J in the unblocked case or by an

estimate of p1 − ρ

2 + ρ2/J/p

1 − ρ

2 in the blocked case. He then performed simulations that

indicated the number of replications that would be needed to ensure that these asymptotically

correct adjustments would yield good confidence interval coverages. These numbers were reported

in his tables 1 and 2. These tables indicated that for larger ρ’s, fairly large sample sizes would be

needed to ensure good μj confidence interval coverages. This problem appears to be driven by the

sensitivity of the corrections to ˆρ.

We have since realized that we can avoid this problem by making use of results (7) and (8).

Together, they imply that

s

2

unblocked − s

2

blocked

p→ ρ

2σ2

Y

and

s

2

blocked +



s

2

unblocked − s

2

blocked

/J p→ (1 − ρ

2 + ρ2

/J)σ

2

Y

7
so, in the one-factor case, we can take the corrected “ANOVA z” confidence interval on μj to be

y ̄·j ± z

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/J /

√

I (9)

where z = Φ−1(1 − α/2) for a 1 − α confidence level, and the corrected “ANOVA t” confidence

interval on μj to be

y ̄·j ± t

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/J /

√

I (10)

where t = T

−1

J(I−1)(1 − α/2) for a 1 − α confidence level, and TJ(I−1) denotes the cumulative

distribution function of a t distribution with J(I − 1) degrees of freedom (this is an ad hoc choice

for degrees of freedom).

The second solution that was blithely and incorrectly proposed by Verrill (1999) was an analysis

of covariance. Recall (Equation (2)) that in a one-factor predictor sort case, we have

Yij = μj + σY



ρ



Xk(i,j),n − μX



/σX +

p

1 − ρ

2Pij

or

Yij = μj − ρ σY μX/σX + ρ σY Xk(i,j),n/σX +

p

1 − ρ

2 σY Pij

= aj + b Xk(i,j),n +

p

1 − ρ

2 σY Pij (11)

where

aj = μj − ρ σY μX/σX

and

b = ρ σY /σX

Now

aj + b x ̄·· = μj − ρ(σY /σX)μX + ρ(σY /σX) ̄x··

is an approximation to μj and, given the x’s and model (11), it is well known that ˆaj + ˆbx ̄·· has

variance

 

1/I + ( ̄x·j − x ̄··)

2

/

 X

J

k=1

X

I

i=1

(xik − x ̄·k)

2

!! (1 − ρ

2

)σ

2

Y

(12)

For large I, this is of the order

(1 − ρ

2

)σ

2

Y /I

This value is an underestimate of the variance of ˆaj + ˆbx ̄··. Heuristically, this is essentially due to

the fact that we are treating  ̄x·· as a constant when, instead,

Var(aj + b × x ̄··) = b

2σ2

X/(IJ)

= ρ

2σ2

Y /σ2X × σ

2

X/(IJ)

= σ

2

Y ρ

2

/(IJ)

In Appendix C we show that

aˆj + ˆbx ̄·· =  ̄y·j − ˆb( ̄x·j − x ̄··) =  ̄y·j − ρˆ(ˆσY /σˆX)( ̄x·j − x ̄··) = ˆμj (13)

8
where ˆaj and ˆb are the standard analysis of covariance estimators, and ˆρ, ˆσY , ˆσX, and ˆμj are

maximum likelihood estimators obtained in appendix A.2 of Verrill et al. (2004). In Appendix

D we show that under maximum likelihood regularity conditions (which we do not verify at this

point)

√

I(ˆμj − μj )

D

→ N(0, σ2

Y

(1 − ρ

2 + ρ2

/J)) (14)

From results (13) and (14) we have

√

I( ̄y·j − ˆb( ̄x·j − x ̄··) − μj )

D

→ N(0, σ2

Y

(1 − ρ

2 + ρ2

/J)) (15)

Thus, in the one-factor case, we can take the corrected “ANCOVA z” confidence interval on μj

to be

y ̄·j − ˆb( ̄x·j − x ̄··) ± z

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/J/

√

I (16)

where z = Φ−1(1 − α/2) for a 1 − α confidence level, and the corrected “ANCOVA t” confidence

interval on μj to be

y ̄·j − ˆb( ̄x·j − x ̄··) ± t

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/J/

√

I (17)

where t = T

−1

JI−(J+1)(1 − α/2) for a 1 − α confidence level (again, the degrees of freedom are ad

hoc).

Based on result (14), we take the maximum likelihood confidence interval to be

μˆj ± z σˆY

p

1 − ρˆ

2 + ˆρ2/J/√

I (18)

We have performed simulations on one-factor predictor sort ANOVAs and ANCOVAs to evaluate

the resulting confidence interval coverages. For all combinations of X,Y correlations 0.5, 0.6, 0.7,

0.8, 0.9, 0.95, and 0.99; number of factor levels, J, equal to 2, 3, 5, 7, 9, 11, and 20; sample sizes,

I, equal to 3, 5, 10, 20, 40, and 80; and confidence levels 90% and 95%, we performed 10,000

trials. For each of the resulting data sets we calculated standard unblocked and blocked ANOVA

confidence intervals on μ1, corrected z and t ANOVA confidence intervals on μ1, a “standard”

ANCOVA confidence interval on μ1, corrected z and t ANCOVA confidence intervals on μ1, and

the maximum likelihood confidence interval on μ1.

In accord with the reasoning associated with (12), we calculated the “standard” ANCOVA

confidence interval as

y ̄·j − ˆb( ̄x·j − x ̄··) ± tsancova

vuut1/I + ( ̄x·j − x ̄··)

2/

 X

J

k=1

X

I

i=1

(xik − x ̄·k)

2

!

(19)

The results of the 95% confidence interval simulations are presented in Table 2. The results of

both the 90% and 95% confidence interval simulations are available at

http://www1.fpl.fs.fed.us/ps15 table2.html. In these tables, columns 4 and 5 report the

coverages of standard one-way and two-way ANOVAs after a predictor sort allocation. Columns 6

and 7 report the coverages of the confidence intervals given in (9) and (10). Column 8 reports the

coverages of the standard (and thus incorrect) ANCOVA confidence interval given in (19). Columns

9 and 10 report the coverages of the confidence intervals given in (16) and (17). Finally, column 11

reports the coverage of the maximum likelihood confidence interval given by (18).

9
For the 95% confidence interval simulations, we fit the model

coverage − 0.95 = c1/I1/2 + c2/I + c3/I3/2

to the tabled coverages, and then used the resulting fits to estimate the I’s at which the actual

coverages would first fall between 0.94 and 0.96. As an illustration, the data and fits for the J = 3,

ρ = 0.8 case are plotted in Figures 37 (ANOVA) and 38 (ANCOVA). The estimated needed I’s

are provided in Table 3. For blocked ANOVAs, these I’s are much improved over the comparable

values reported in Verrill’s (1999) table 2. For unblocked ANOVAs and J ≥ 5, these I’s are much

improved over the comparable values reported in Verrill’s (1999) table 1.

From the 95% and 90% confidence interval simulation tables it is clear that

1. The confidence interval coverage simulation results are in accord with the large sample results

expressed in Figures 35 and 36. That is, for higher ρ’s, an uncorrected one-factor/unblocked

ANOVA will lead to confidence interval coverages that are larger than the nominal coverages

(see column 4 of Table 2) and a one-factor/blocked ANOVA will lead to coverages that are

lower than nominal coverages (see column 5 of Table 2). Also, as expected from the discussion

in connection with result (12), uncorrected ANCOVA analyses will lead to actual coverages

smaller than nominal coverages (see column 8 of Table 2).

2. Corrected ANOVAs and ANCOVAs yield good coverages for reasonable sample sizes. (See

columns 6, 7, 9, and 10 of Table 2.)

3. For ρ ≤ 0.80, good coverages are obtained most quickly/for the smallest sample sizes by

taking a “uses t” approach. That is, we use the appropriate t critical value rather than the

appropriate z critical value. For ρ ≥ 0.90, corrected ANOVAs yield correct coverages most

quickly if a “uses t” aproach is taken for J = 2 and a “uses z” approach is taken otherwise.

For ρ ≥ 0.90, corrected ANCOVAs yield correct coverages most quickly if a “uses t” aproach

is taken for J = 2, 3 and a “uses z” approach is taken otherwise. (See columns 6, 7, 9, and

10 of Table 2.)

4. The maximum likelihood true coverage is slow to converge to the nominal coverage. (See

column 11 of Table 2.)

A listing of the simulation program that produced the Table 2 coverage estimates can be obtained at http://www1.fpl.fs.fed.us/ps15 confsim code.html. A web-based simulation program that can be run on additional cases (including multi-factor cases) can be found at

http://www1.fpl.fs.fed.us/psconf.html.

We note that under the conditions of Theorem 2, for factor 1, the multi-factor versions of

confidence intervals (9), (10), (16), and (17) have the forms

y ̄·j1·...· ± z ×

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/K1



/

p

I × K2 × . . . × KF (20)

y ̄·j1·...· ± t ×

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/K1



/

p

I × K2 × . . . × KF (21)

y ̄·j1·...·−ˆb( ̄x·j1·...·−x ̄·...·)±z×

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/K1



/

p

I × K2 × . . . × KF (22)

y ̄·j1·...·−ˆb( ̄x·j1·...·−x ̄·...·)±t×

q

s

2

blocked +



s

2

unblocked − s

2

blocked

/K1



/

p

I × K2 × . . . × KF (23)

10
where z = Φ−1

(1−α/2) for a 1−α confidence level, and t = T

−1

K1...KF I−(K1+K2−1+...+KF −1+1)(1−α/2)

for a 1 − α confidence level (again, the degrees of freedom are ad hoc). The analogous results hold

for factors 2, . . . , F.

4 Confidence Bounds on Quantiles

Verrill, Herian, and Green (2004) addressed a more specialized problem associated with predictor

sort allocations.

Designers working with lumber must try to ensure that the strengths of wood structural members exceed the loads to which the members will be subjected. One approach to this problem is to

design so that expected loads do not exceed “allowable strength properties” associated with particular species and grades of lumber (ASTM 2016). An allowable property is commonly obtained

experimentally by taking a sample from the lumber population in question, obtaining a lower onesided confidence bound on the fifth percentile of the strength distribution of the population, and

then dividing by safety and duration-of-load factors.

If a normal strength distribution is assumed, engineers working with solid-sawn lumber (see,

for example, ASTM 2010) can obtain a parametric one-sided lower confidence bound on the fifth

percentile via

Y ̄

I − kI,α,βSI (24)

where we want to cover the α quantile with confidence β × 100% and we have I replicates. Here Y ̄

I

denotes the average of I strength measurements, and SI denotes the sample standard deviation of

the measurements. Guttman (1970, table 4.6) provides kI,α,β values for a range of I, α, and β values.

He credits Owen (1963) for this table. The approach is based on the noncentral t distribution and

is exact if the experimental design involves standard random sampling rather than predictor sort

sampling. A web program that calculates kI,α,β values for the non predictor sort case can be run

at http://www1.fpl.fs.fed.us/tolerance.html.

Scientists in other areas (e.g., composite materials, groundwater monitoring, and soil remediation) also make use of formula (24) to obtain confidence bounds on quantiles. (See, for example,

MIL-HDBK-17-1 (2003), Gibbons (1994), and Michigan DEQ (1994).)

For formula (24) to be valid, the sample of lumber (or composite material, water, soil, . . . ) must

be a standard random sample. However, as noted in the Introduction, wood strength researchers

sometimes replace experimental unit allocation via random sampling with allocation via sorts based

on nondestructive measurements of strength predictors such as modulus of elasticity and specific

gravity. That is, they perform predictor sort experiments.

Verrill et al. (2004) examined the effect of predictor sort sampling on one-sided confidence

bounds for normal quantiles. They found that standard noncentral t theory that ignores the predictor sort nature of the sampling leads to Y ̄ − kS bounds that are too low and thus statistically

conservative (actual confidence levels are greater than nominal levels). On the other hand, maximum likelihood methods yield bounds that are too high and thus statistically nonconservative even

for fairly large sample sizes.

4.1 Main theorem on predictor sort confidence intervals on quantiles

The authors used tools developed in Verrill (1993, 1999) to establish an asymptotic result that

yields the appropriate corrections for the standard noncentral t approach.

11
Theorem 3

Assume that, prior to treatment, the predictor variable and the variable of interest, Y , have a

joint bivariate normal distribution with correlation ρ. Denote the variance of Y by σ

2

Y

. Suppose

that we have a one-factor ANOVA design with I blocks and J treatments, and the allocation of

samples is done via a predictor sort as described in Section 1.

Let Yij be given by (2). Define

Y ̄

·j ≡

X

I

i=1

Yij/I

and

S

2 =

X

J

j=1

X

I

i=1

(Yij − Y ̄

·j )

2

/(IJ − 1)

and let ˆρ be any consistent estimator of ρ. Then,

Prob Y ̄

·j − ˆkI,α,βS ≤ μj + Φ−1

(α) σY



→ β

as I → ∞, where μj is the mean response for the jth treatment, Φ−1 denotes the inverse of a

standard normal cumulative distribution function,

ˆkI,α,β ≡

p

(1 − ρˆ

2 + ˆρ2/J)/I NCT−1

IJ−1,γI (ˆρ)

(β),

NCT−1

IJ−1,γI (ˆρ)

denotes the inverse of a noncentral t distribution with IJ −1 degrees of freedom and

noncentrality parameter γI (ˆρ), and

γI (ˆρ) = −Φ

−1

(α)

√

I



1 − ρˆ

2 + ˆρ2

/J−1/2

Proof

The proof is provided in appendix B of Verrill et al. (2004).

In tables 1 to 24 of Verrill et al. (2004), the authors detail the coverages of four kinds of

confidence interval on a quantile for a variety of combinations of ρ, α, β, number of treatments (J),

and number of replicates (I). The rows of the tables are based on separate 4,000 trial simulations.

The four approaches considered were the incorrect standard approach given in (24), which ignores

the dependencies induced by sorting on the predictor; two versions of the (correct) predictor sort

Y ̄ − kS asymptotic approach described in Theorem 3; and a maximum likelihood approach. The

two versions of the predictor sort approach differ in the estimate used for the correlation between

the predictor and the response. Version 1 uses the consistent estimate

ρˆ ≡

X

J

j=1

X

I

i=1

(Xij − X ̄

··)(Yij − Y ̄·j )/

vuutX

J

j=1

X

I

i=1

(Xij − X ̄

··)

2X

J

j=1

X

I

i=1

(Yij − Y ̄

·j )

2

Version 2 uses the maximum likelihood estimate of ρ. It is clear from the tables that the incorrect

approach is overly conservative, that the problem becomes more severe as the correlation between

the predictor and response variables increases, and that the problem does not vanish as sample sizes

increase. It is also clear from the tables that version 2 of the predictor sort approach dominates the

12
maximum likelihood approach in the sense that the actual coverage always approaches the nominal

coverage more rapidly for the version 2 predictor sort approach than for the maximum likelihood

approach. For smaller J, the version 1 predictor sort approach performs better than the maximum

likelihood approach and the version 2 approach (see Figure 39 for an example). However, for large

J and small I, the version 1 approach does not perform as well.

For smaller I, the asymptotic approaches are non-conservative. The authors have developed a

program that can be run interactively to yield the appropriate k values for small sample sizes. This

program will be described in an upcoming Forest Products Laboratory technical report.

4.2 Reductions in the sample sizes needed to obtain confidence bounds on quantiles given predictor sort sampling

In the course of the development of their asymptotic theory, Verrill et al. (2004) found that the

correct k in the appropriate version of Y ̄ − kS is given by

k ≈ −Φ

−1

(α) + Φ−1(β)I

−1/2p

(Φ−1(α))2/(2J) + 1 − ρ

2 + ρ2/J

where Φ denotes the N(0,1) distribution function. Thus, given higher ρ values, we can have smaller

I values and still have the same k. In fact if we set

I

−1/2p

(Φ−1(α))2/(2J) + 1 − ρ

2 + ρ2/J

equal to a constant, we obtain

I ∝ (Φ−1(α))2/(2J) + 1 − ρ

2 + ρ2

/J

Thus the approximate permissible sample size reduction factor obtained by using a predictor sort

with a correlation of ρ between the predictor and the response is (here the denominator is the

numerator with ρ set equal to 0)



(Φ−1(α))2/(2J) + 1 − ρ

2 + ρ2

/J/



(Φ−1(α))2/(2J) + 1

For a confidence bound on the 0.05 quantile, sample size reduction factors as functions of ρ and J

are plotted in Figure 40. It is clear from the figure that practically significant sample size reductions

(e.g., 30%) are attainable for correlations as low as 0.70.

5 Scheff ́e and Tukey Multiple Comparison Procedures after a

Predictor Sort Allocation

As one would expect given the hypothesis test results established in Verrill (1993), for large sample

sizes, suitably altered versions of the Scheff ́e and Tukey multiple comparison procedures are valid

after a predictor sort allocation. In this section, we describe the alterations and establish the needed

asymptotic results.

We first introduce the notation that we use in this section. Assume that we have F factors,

Kj levels for the jth factor, and I blocks (formed by specimens with adjacent [randomized within

a block] order statistics of the predictor). Let n ≡ IK1 . . . KF , and {Xi, i = 1, . . . , n}, {Zi, i =

1, . . . , n} be i.i.d. N(0,1) random variables. Define Wi ≡ σY



ρXi +

p

1 − ρ

2 Zi



.

We model predictor sort allocation by ordering the X’s (the predictors) and randomly dividing the W’s that correspond to X(i−1)K1...KF +1,n, . . . , XiK1...KF ,n among the K1 × K2 × . . . × KF

treatments. (Here, Xl,n is the lth order statistic among the X’s.)

13
Let Yij1...JF

denote μj1·...· + . . . + μ·...·jF plus the ith W that is assigned to treatment j1 . . . jF ,

where, for example, μj1·...· denotes the effect associated with the j1th level of factor 1, and μ·...·jF

denotes the effect associated with the jF th level of factor F. Then

Yij1...jF = μj1·...· + . . . + μ·...·jF + σY



ρXk(i,j1,...,F ),n +

p

1 − ρ

2 Pij1...jF



(25)

where k(i, j1, . . . , jF ) ∈ {(i−1)K1 . . . KF + 1, . . . , iK1 . . . KF }, and the Pij1...jFare i.i.d. N(0,1) and

are independent of the X’s.

5.1 Scheff ́e’s multiple comparison test/procedure

The result is basically simple, but we need to provide the notation and describe the approach to

express it. This approach is, of course, due to Scheff ́e.

First we essentially restate Scheff ́e’s result in the non predictor sort case and then we identify

those aspects of the proof that need to be altered in the predictor sort case.

Assume that

y ∼ N(Xβ, Iσ

2

) (26)

(We note that the X in equation (25) is distinct from the X in formula (26).) Let c1, . . . , cq

correspond to q linearly independent estimable functions. That is, the c’s are linearly independent

and there exist a1, . . . , aq that lie in the linear span of the columns of the design matrix X such

that

a

T

j X = c

T

j

for j ∈ 1, . . . , q so

E(a

T

j y) = a

T

j Xβ = c

T

j β

for j ∈ 1, . . . , q.

We have

v ≡





a

T

1 y − c

T

1 β

.

.

.

a

T

q y − c

T

q β



 ∼ N



0, Aσ

2



or

v ≡





a

T

1

.

.

.

a

T

q



 (y − Xβ) ∼ N



0, Aσ

2



where

A ≡





a

T

1

.

.

.

a

T

q



 (a1 . . . aq)

Thus,

A−1/2v ∼ N



0, Iq×qσ

2



so

v

TA−1v/σ2 ∼ χ2

q

(27)

Now, by the Cauchy-Schwarz theorem, for all l ∈ R

q

,



l

Tv

2

=



l

TA1/2A−1/2v

2

≤



l

TAl vTA−1v



(28)

14
so by result (27), simultaneously for all l ∈ R

q

(note that the right hand side of (29) does not

depend on l),



l

Tv

2

/



qs2l

TAl

≤



v

TA−1v/q

/s2 ∼ Fq,n−r (29)

where n is the total number of observations, r is the rank of X, and

s

2 =



y

Ty −



(u

T

1 y)

2 + . . . + (uT

r y)

2

 /(n − r)

is the mean residual sum of squares (u1, . . . , ur is an orthonormal basis of the space spanned by

the columns of X).

So how does result (29) change in a predictor sort situation?

For the first factor, we want to test the null hypothesis that μ1·...· = μ2·...· = . . . = μK1·...· or

that c

Tμ = 0 provided that cT1 = 0 (the “contrasts” are all 0). (For ease of presentation, in what

follows, we focus on the first factor, but similar results hold for the other F − 1 factors.)

In this case, in our formulation of the problem, the first K1 columns of X correspond to factor

1. The first column of X contains K2 . . . KF I ones followed by (K1 − 1)K2 . . . KF I zeros, . . . , the

K1th column of X contains (K1 − 1)K2 . . . KF I zeros followed by K2 . . . KF I ones.

The next K2 columns of X correspond to factor 2. The first K3K4 . . . KF I rows of column K1+1

of X contain ones. The next (K2 − 1)K3K4 . . . KF I rows of column K1 + 1 of X contain zeros.

This pattern is repeated K1 −1 more times to complete column K1 + 1. The one’s are shifted down

systematically in the following columns until in column K1 + K2, the first (K2 − 1)K3K4 . . . KF I

rows of column K1 + K2 of X contain zeros, the next K3K4 . . . KF rows of column K1 + K2 of X

contain ones, and this pattern is repeated K1 − 1 more times to complete column K1 + K2.

This process is continued through the final KF columns of X which correspond to the levels of

factor F (or through the final I columns of X if we perform a blocked ANOVA).

For the specific multiple comparison problem that we are considering (we are interested in being

able to test all of the contrasts of the first factor means), c1 contains a 1 in its first element, a −1 in

its second element, and zeros in its remaining elements, . . . , cK1−1 contains a 1 in its first element,

zeros in its next K1 − 2 elements, a −1 in its K1th element, and zeros in its remaining elements.

The aj corresponding to cj has 1/(K2 . . . KF I) in its first K2 . . . KF I elments, −1/(K2 . . . KF I)

in elements jK2 . . . KF I + 1, . . . ,(j + 1)K2 . . . KF I, and zeros elsewhere. If u1, . . . , uK1−1 is an

orthonormal basis of the linear span of a1, . . . , aK1−1, then

v

TA−1v = (y − Xβ)T (a1 . . . aK1−1) A−1





a

T

1

.

.

.

a

T

K1−1



 (y − Xβ)

= (y − Xβ)

T (u1 . . . uK1−1)





u

T

1

.

.

.

u

T

K1−1



 (y − Xβ) (30)

Define n ≡ K1 . . . KF I. It is clear that u1, . . . , uK1−1, 1/

√

n have the same linear span as

w1, . . . , wK1

, where wj contains 1/

√

K2 . . . KF I in rows (j −1)K2 . . . KF I + 1, . . . , jK2 . . . KF I and

15
zeros elsewhere. Thus,

v

TA−1v =

X

K1

j1=1



wT

j1

(y − Xβ)

2

−



1

T(y − Xβ)/

√

n

2

=

X

K1

j1=1



wT

j1

(y − Xβ)

2

− n ( ̄y·...· − (μ−·...· + . . . + μ·...·−))2

=

X

K1

j1=1

K2 . . . KF I ( ̄y·j1·...· − (μj1·...· + μ·−·...· + . . . + μ·...·−))2

− n ( ̄y·...· − (μ−·...· + . . . + μ·...·−))2

=

X

K1

j1=1

K2 . . . KF I [ ̄y·j1·...· − (μj1·...· + μ·−·...· + . . . + μ·...·−)

− ( ̄y·...· − (μ−·...· + . . . + μ·...·−))]2

=

X

K1

j1=1

K2 . . . KF I × σ

2

Y ×



ρ( ̄xk(·,j1,·,...,·),n − x ̄k(·,...,·),n) + p

1 − ρ

2( ̄p·j1·...· − p ̄·...·)

2

D

→ (1 − ρ

2

)σ

2

Y χ

2

K1−1

(31)

as I → ∞ by the material in Section 14.1. Here,

μ−·...· ≡

X

K1

j1=1

μj1·...·/K1

Similar definitions hold for μ·−·...·, . . . , μ·...·−.

Also, in Sections 14.2 and 14.3, we show that in the “unblocked” predictor sort case,

s

2

ub

p→ σ

2

Y

(32)

where

s

2

ub = SSden,unbl/(IK1 . . . KF − (K1 + K2 − 1 + . . . + KF − 1)) (33)

and

SSden,unbl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(yij1...jF − ( ̄y·...· + ( ̄y·j1·...· − y ̄·...·) + . . . + ( ̄y·...·jF − y ̄·...·)))2

whereas, in the “blocked” predictor sort case,

s

2

b

p→ (1 − ρ

2

)σ

2

Y

(34)

where

s

2

b = SSden,bl/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) (35)

and

SSden,bl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(yij1...jF − ( ̄y·...· + ( ̄yi·...· − y ̄·...·) + . . . + ( ̄y·...·jF − y ̄·...·)))2

16
From results (28), (31), (32), and (34), we know that in the factor 1, unblocked predictor sort

case we have, simultaneously for all l ∈ R

K1−1

,



l

Tv

2

/



s

2

ub l

TAl

≤ v

TA−1v/s2

ub

D

→



1 − ρ

2



χ

2

K1−1

and in the blocked predictor sort case we have



l

Tv

2

/



s

2

b

l

TAl

≤ v

TA−1v/s2

b

D

→ χ

2

K1−1

Analogous results hold for factors 2 through F. So, we are led to the following theorem.

Theorem 4

Assume that, prior to treatment, the predictor variable and the variable of interest have a joint

bivariate normal distribution with correlation ρ. Let the allocation of samples be as described in

Section 1. (For the multi-factor case, enough adjacent experimental units are chosen at a time to

provide one specimen for each cell in a block.) Given the notation provided above in association

with results (30), (31), (32), and (34), where the A and v are appropriate for testing the contrasts of

the factor j means, in the unblocked predictor sort case we have, simultaneously for all l ∈ R

Kj−1

,



l

Tv

2

/



s

2

ub l

TAl

≤ v

TA−1v/s2

ub

D

→



1 − ρ

2



χ

2

Kj−1

(36)

and in the blocked predictor sort case we have, simultaneously for all l ∈ R

Kj−1

,



l

Tv

2

/



s

2

b

l

TAl

≤ v

TA−1v/s2

b

D

→ χ

2

Kj−1

(37)

5.2 Simultaneous confidence intervals on contrasts based on Scheff ́e’s multiple

comparison procedure

From result (36), in the unblocked predictor sort case, for the factor j (where A and v will depend

on j), given any  > 0 and α ∈ (0, 1), we can find an N,α such that I > N,α implies that

Prob 



l

Tv

2

≤ s

2

ub (l

TAl)(1 − ρ2

)F

−1

χ

2

Kj−1

(1 − α) for all l ∈ R

Kj−1



> 1 − α − 

where Fχ

2

Kj−1

is the cdf of a χ

2

Kj−1

, or

Prob









l

T





a

T

1

.

.

.

a

T

Kj−1



 y − l

T





c

T

1 β

.

.

.

c

T

Kj−1β









2

≤ s

2

ub(l

TAl)(1 − ρ2

)F

−1

χ

2

Kj−1

(1 − α) for all l ∈ R

Kj−1





> 1 − α − 

That is, for I large enough, the intervals

l

T





a

T

1

.

.

.

a

T

Kj−1



 y ± sub

√

l

TAlp

1 − ρ

2

r

F

−1

χ

2

Kj−1

(1 − α)

are simultaneous (for all l ∈ R

Kj−1

), approximate 1 − α confidence intervals on the contrasts

l

T





c

T

1 β

.

.

.

c

T

Kj−1β



 (38)

17
Similarly, from result (37), in the blocked predictor sort case, for I large enough, the intervals

l

T





a

T

1

.

.

.

a

T

Kj−1



 y ± sb

√

l

TAlr

F

−1

χ

2

Kj−1

(1 − α)

are simultaneous (for all l ∈ R

Kj−1

), approximate 1 − α confidence intervals on the contrasts in

(38).

Note that for a contrast

s

Tβ = (s1 . . . sKj

)

Tβ

(s1 + . . . + sKj = 0) we can obtain the necessary l via



c1 . . . cKj−1



l = s

or





1

−1

0

.

.

.

0

. . .

1

0

.

.

.

0

−1





l = s

so

l =





−s2

−s3

.

.

.

−sKj





5.3 Tukey’s multiple comparison test/procedure

Theorem 5

Assume that the predictor variable and the variable of interest, Y , have a joint bivariate normal

distribution with correlation ρ. Denote the variance of Y by σ

2

Y

. Suppose that there are I blocks

and F factors with K1, . . . , KF levels. Let the allocation of samples be as described in Section 1.

(For a multi-factor case, enough adjacent experimental units would be chosen at a time to provide

one specimen for each cell in a block.) Let Y ̄

·j1·...· be the standard estimate of the mean response

for the j1th level of factor 1. For comparisons of the factor 1 levels, let the numerator of the test

statistic be given by

QI ≡ max

l1,l2∈{1,...,K1}

p

IK2 . . . KF |Y ̄

·l1·...· − Y ̄·l2·...·

|

Let s

2

ub (given by equation (33)) denote the estimate of σ

2

in the unblocked case, and s

2

b

(given

by equation (35)) denote the estimate of σ

2

in the blocked case.

Let FR(K1) denote the distribution of the range of a sample of K1 independent N(0,1)’s.

Then, under the null hypothesis that μ1·...· = . . . = μK1·...·,

QI/



subp

1 − ρ

2



D

→ FR(K1)(39)

and

QI/sb

D

→ FR(K1)(40)

18
Similar results hold for factors 2 through F.

Proof

The proof appears in Appendix F.

We have performed simulations on one-factor predictor sort ANOVAs to evaluate the resulting

sizes of Tukey tests. For all combinations of X,Y correlations 0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, and

0.99, number of treatments, J, equal to 3, 5, 7, 9, 11, and 20, and sample sizes, I, equal to 3, 5,

10, 20, and 40, we performed 100,000 trials. For each of the resulting data sets, we calculated a

standard unblocked Tukey test statistic on the equivalence of the J treatment means, an unblocked

Tukey test statistic that has been corrected by an estimated p

1 − ρ

2 factor, an unblocked Tukey

test statistic that has been corrected by the true p

1 − ρ

2 factor, and a blocked Tukey test statistic.

In Table 4, we report the resulting actual test sizes when the nominal size is 0.05.

We can conclude from this table that

1. For the non-zero ρ’s considered, a standard unblocked Tukey test (one that uses an sub

denominator with no ρ correction) yields test sizes that can be much less than the nominal

test size. (See column 4 of Table 4.)

2. For lower I, an unblocked Tukey test that has been corrected via an estimated ρ yields test

sizes that can be much more than the nominal test size. (See column 5 of Table 4.)

3. For lower I, and ρ = 0.5, 0.6, an unblocked Tukey test that has been corrected via the

true ρ yields test sizes that can be lower than the nominal test size. For lower I, and

ρ = 0.8, 0.9, 0.95, 0.99, an unblocked Tukey test that has been corrected via the true ρ yields

test sizes that can be much higher than the nominal test size. (See column 6 of Table 4.)

4. In general, a blocked Tukey test (one that uses an sb denominator) yields actual test sizes

that closely match the nominal 0.05 test size. The blocked Tukey test does perform somewhat

poorly in the ρ = 0.95, 0.99 cases. (For lower I, actual test sizes fall below the nominal 0.05

value.) (See column 7 of Table 4.)

A listing of the simulation program that produced the size estimates can be found at

http://www1.fpl.fs.fed.us/ps15 tukey size sim code.html.

5.4 Simultaneous confidence intervals based on Tukey’s multiple comparison

test

(Here we work with factor 1. The analogous results hold for factors 2 through F.)

It is clear from the proof of Theorem 5 that, under predictor sort allocation, in the unblocked

case,

max

l1,l2∈{1,...,K1}

p

IK2 . . . KF



Y ̄

·l1·...· − Y ̄·l2·...· − (μl1·...· − μl2·...·)



 /sub

D

→

p

1 − ρ

2FR(K1)

and in the blocked case,

max

l1,l2∈{1,...,K1}

p

IK2 . . . KF



Y ̄

·l1·...· − Y ̄·l2·...· − (μl1·...· − μl2·...·)



 /sb

D

→ FR(K1)

(where, for example, μl1·...·is the effect associated with the l1th level of factor 1). That is, in the

unblocked case,

19
Prob 

max

l1,l2∈{1,...,K1}

p

IK2 . . . KF



Y ̄

·l1·...· − Y ̄·l2·...· − (μl1·...· − μl2·...·)



 /sub ≤

p

1 − ρ

2F

−1

R(K1)

(1 − α)



→ 1−α

as I → ∞.

So, in the unblocked case, given any  > 0, we can find an N such that I > N implies that

Prob Y ̄

·l1·...· − Y ̄·l2·...· − ∆ ≤ μl1·...· − μl2·...· ≤ Y ̄·l1·...· − Y ̄·l2·...· + ∆ for all l1, l2 ∈ {1, . . . , K1}



> 1−α−

where

∆ = sub/

p

IK2 . . . KF

 p

1 − ρ

2F

−1

R(K1)

(1 − α)

so, simultaneously for all l1, l2 pairs in {1, . . . , K1}, for I > N,

Y ̄

·l1·...· − Y ̄·l2·...· ±



sub/

p

IK2 . . . KF

 p

1 − ρ

2F

−1

R(K1)

(1 − α)

is a 1 − α −  confidence interval on μl1·...· − μl2·...·.

Similarly, in the blocked case, given any  > 0, we can find an N such that for I > N,

simultaneously for all l1, l2 pairs in {1, . . . , K1},

Y ̄

·l1·...· − Y ̄·l2·...· ±



sb/

p

IK2 . . . KF



F

−1

R(K1)

(1 − α)

is a 1 − α −  confidence interval on μl1·...· − μl2·...·.

In practical terms this implies that for I large enough, the

Y ̄

·l1·...· − Y ̄·l2·...· ±



sub/

p

IK2 . . . KF

 p

1 − ρ

2F

−1

R(K1)

(1 − α)

intervals are good approximations to 1−α simultaneous confidence intervals on the μl1·...· −μl2·...·’s.

Similarly, for I large enough, the

Y ̄

·l1·...· − Y ̄·l2·...· ±



sb/

p

IK2 . . . KF



F

−1

R(K1)

(1 − α)

intervals are good approximations to 1−α simultaneous confidence intervals on the μl1·...· −μl2·...·’s.

6 Web and R Programs

Forest Products Laboratory scientists have produced predictor sort web programs that help researchers

1. choose sample sizes (perform power calculations) for predictor sort hypothesis tests,

2. allocate specimens via a predictor sort,

3. perform hypothesis tests for a simple one factor, two levels predictor sort experiment, and

4. perform simulations to estimate the coverage of predictor sort confidence intervals on treatment means.

These programs can be accessed at http://www1.fpl.fs.fed.us/predsort.html. This web page

also contains a link to R code that helps users calculate predictor sort confidence intervals on

treatment means.

We have also developed interactive Java code that permits a user to obtain small sample confidence intervals on quantiles in the predictor sort case. This work will appear in a separate Forest

Products Laboratory technical report.

20
7 Summary

We have reminded readers that, properly designed and analyzed, predictor sort experiments (experiments in which the predictor variable is used to form blocks) permit scientists to achieve

considerable increases in statistical power and/or reductions in sample sizes (and thus reductions

in experimental costs). For analysis of variance tests of hypotheses, sample sizes can be reduced

from roughly n to (1 − ρ

2

)n, where ρ is the correlation between the predictor/concomitant and the

variable of interest. For confidence intervals on quantiles, approximate sample size reductions are

illustrated in Figure 40. They can amount to 30% for ρ equal to 0.70, and increase as ρ increases.

Our studies also indicate that the 1 − ρ

2

factor is only approximate, especially for blocked

ANOVAs (as opposed to analyses of covariance). Thus, we have provided a web-based simulation

program that yields estimates of actual powers (in addition to estimates based on large sample

theory).

We have demonstrated that if a scientist performs a predictor sort allocation, but then analyzes

the experiment as an unblocked analysis of variance, their experiment can have extremely low

statistical power (an inability to detect actual differences). This amounts to a serious scientific

blunder.

We have demonstrated theoretically that given a predictor sort allocation, unmodified analyses

of variance (blocked or unblocked) and analyses of covariance yield incorrect confidence intervals

on treatment means. (The confidence intervals are too wide in the unblocked ANOVA case and too

narrow in the blocked ANOVA and analysis of covariance cases.) We have provided a web-based

simulation program that estimates the coverages of incorrect (unmodified) ANOVA confidence

intervals on treatment means, and the coverages of corrected ANOVA and ANCOVA confidence

intervals. We have also provided an R function that helps a scientist calculate corrected confidence

intervals on treatment means estimated from a predictor sort experiment.

We have developed methods that yield correct one-sided lower confidence bounds on quantiles

given a predictor sort allocation.

Finally, we have developed Scheff ́e and Tukey multiple comparison tests and associated simultaneous confidence intervals that are appropriate in the predictor sort case.

All our results have been established under an assumption of a joint bivariate normal relationship

between the predictor and the response.

As noted earlier, it can be argued that in a predictor sort situation a professional statistician would undoubtedly perform a blocked analysis or an analysis of covariance using the predictor/concomitant as the covariate, and thus, we need not exercise special care in identifying,

designing, and analyzing predictor sort experiments. We have a five-fold response. First, identifying a design as a predictor-sort design permits a scientist to perform correct power calculations.

Second, blocked ANOVA hypothesis tests can perform poorly as ρ becomes sufficiently large. Third,

although unmodified blocked ANOVAs (for lower ρ’s) and analyses of covariance yield essentially

correct hypothesis tests, they yield incorrect confidence intervals on treatment means. Fourth,

blocked ANOVAs and analyses of covariance do not help us in the quantile estimation case. Finally, as noted at the end of Section 2, we have seen introductory texts that treat predictor sort

allocation as a good experimental practice independent of the method of analysis. Given the large

decrease in power (especially for larger ρ’s) that can occur if a predictor sort experiment is analyzed

via an unblocked ANOVA, this pedagogy must be corrected.

21
References

ASTM (2010). Standard Practice for Evaluating Allowable Properties for Grades of Structural

Lumber, D 2915-10, Annual Book of ASTM Standards, ASTM International, West Conshohocken, PA.

ASTM (2016). Standard Practice for Establishing Allowable Properties for Visually-Graded Dimension Lumber from In-Grade Tests of Full-Size Specimens, D 1990-16, Annual Book of

ASTM Standards, American Society for Testing and Materials, West Conshohocken, Pennsylvania.

Casella, G. (2008), Statistical Design, Berlin: Springer.

Cox, D.R. (1958), Planning of Experiments, New York: John Wiley.

Cozby, P. and Bates, S. (2011), Methods in Behavioral Research, 11th Edition, New York: McGrawHill.

David, H.A. (1981), Order Statistics, New York: John Wiley & Sons.

David, H.A. and Gunnink, J.L. (1997), “The Paired t Test Under Artificial Pairing,” The American

Statistician, 51, 9–12.

Finney, D.J. (1972), An Introduction to Statistical Science in Agriculture, New York: John Wiley.

Gibbons, R.D. (1994), Statistical Methods for Groundwater Monitoring, New York: John Wiley

and Sons.

Guttman, I. (1970), Statistical Tolerance Regions: Classical and Bayesian, Hafner Publishing

Company, Darien, Connecticut.

Kerlinger, F.N. and Lee, H.B. (1999), Foundations of Behavioral Research, 4th Edition, Cengage

Learning: Boston.

Kirk, R.E. (1968), Experimental Design: Procedures for the Behavioral Sciences, Belmont, CA:

Brooks/Cole.

Kirk, R.E. (2013), Experimental Design: Procedures for the Behavioral Sciences, 4th Edition, Los

Angeles, CA: SAGE Publications.

Lehmann, E.L. (1983), Theory of Point Estimation, New York: John Wiley & Sons.

Michigan DEQ (1994), Guidance Document, Verification of Soil Remediation, Hazardous Waste

Program Section, Waste Management Division, Michigan Department of Environmental Quality, Lansing, Michigan.

MIL-HDBK-17-1 (2003), Guidelines for Characterization of Structural Materials, Department of

Defense Single Stock Point, Philadelphia, Pennsylvania, http://www.dodssp.daps.mil/.

Myers, J.L. (1979), Fundamentals of Experimental Design, Boston: Allyn and Bacon.

Ostle, B., and Mensing, R.W. (1975), Statistics in Research, Ames, IA: The Iowa State University

Press.

22
Owen, D.B. (1963), Factors for one-sided tolerance limits and for variable sampling plans, Sandia

Corporation Monograph No. SCR-607 (19th edn).

Rao, C.R. (1973), Linear Statistical Inference and Its Applications, New York: John Wiley &

Sons.

Ruxton, G.D. and Colgrave, N. (2006), Experimental Design for the Life Sciences, 2nd Edition,

Oxford University Press.

Scheff ́e, H. (1959), The Analysis of Variance, New York: John Wiley & Sons.

Snedecor, G.W., and Cochran, W.G. (1989), Statistical Methods, Ames, IA: The Iowa State University Press.

Steel, R., and Torrie, J. (1960), Principles and Procedures of Statistics, New York: McGraw-Hill.

Toutenburg, H. (2002), Statistical Analysis of Designed Experiments, New York: Springer.

Tuckman, B.W., and Harper, B.E. (2012), Conducting Educational Research, 6th Edition, Lanham, Maryland: Rowman and Littlefield Publishers.

van Zutphen, L.F., Baumans, V., and Beynen, A.C. (2001), Principles of Laboratory Animal

Science, Revised Edition: A contribution to the humane use and care of animals and to the

quality of experimental results, Amsterdam: Elsevier.

Verrill, S.P. (1993), “Predictor Sort Sampling, Tight T’s, and the Analysis of Covariance,” Journal

of the American Statistical Association, 88, 119–124. http://www1.fpl.fs.fed.us/jasa1993.pdf

Verrill, S.P., and Green, D.W. (1996), “ Predictor Sort Sampling, Tight t’s, and the Analysis

of Covariance: Theory, Tables, and Examples,” Research Paper FPL-RP-558, Madison, WI:

U.S. Department of Agriculture, Forest Service, Forest Products Laboratory. 105 pages.

http://www1.fpl.fs.fed.us/fpl rp558.pdf

Verrill, S.P. (1999), “When Good Confidence Intervals Go Bad: Predictor Sort Experiments and

ANOVA,” The American Statistician, 53(1), 38–42. http://www1.fpl.fs.fed.us/amstat1999.pdf

Verrill, S.P., Herian, V.L., and Green, D.W. (2004), “Predictor Sort Sampling and Confidence

Bounds on Quantiles I: Asymptotic Theory,” Research Paper FPL-RP-623, Madison, WI:

U.S. Department of Agriculture, Forest Service, Forest Products Laboratory. 67 pages.

http://www1.fpl.fs.fed.us/fpl rp623.pdf

Warren, W.G., and Madsen, B. (1977), “Computer-Assisted Experimental Design in Forest Products Research: A Case Study Based on Testing the Duration-of-Load Effect,” Forest Products

Journal, 27, Number 3, 45–50.

8 Appendix A — Noncentrality Parameter in the Hypothesis Test

Simulation

To investigate hypothesis test powers, we needed to choose “informative” noncentrality parameters

at which to perform the simulations. When there are J treatments, I “replicates” for each treatment, a correlation ρ between the predictor X and the response Y , and a Y variance of σ

2

Y

, the

23
approximately appropriate noncentrality parameter is (see result (5))

X

J

j=1

I(μj − μ ̄·)

2

/((1 − ρ

2

)σ

2

Y

)

For the purposes of a simulation, σ

2

Y

is just a scale parameter and, without loss of generality, we

set it equal to 1 in the simulations. For a correct predictor sort analysis, and “reasonable” I and

ρ, we wanted power to increase from 0.05 to approximately 1 as PJ

j=1(μj − μ ̄·)

2

increased from 0

to its maximum simulation value. In figure 2 of Verrill and Green (1996), we saw that for J = 2,

I = 24, and ρ = 0.7, power does increase from 0.05 to approximately 1 as |μ2 − μ1|/σY increases

from 0 to 1. For J = 2, we have

X

J

j=1

(μj − μ ̄·)

2 = (μ1 − (μ1 + μ2)/2)2 + (μ2 − (μ1 + μ2)/2)2 = (μ2 − μ1)2

/2

Thus if we want to see behavior similar to Verrill and Green’s figure 2 behavior for ρ = 0.7 and

I ≈ 24 (and σY = 1), we would need, for a general J, and power approximately equal to 1,

X

J

j=1

(μj − μ ̄·)

2 = 1/2 (41)

We can obtain this result in an infinite number of ways. We chose to obtain it by providing that,

for J = 2K, K an integer,

μ1, . . . , μJ = −K∆, . . . , −∆, ∆, . . . , K∆

and for J = 2K + 1, K an integer,

μ1, . . . , μJ = −K∆, . . . , −∆, 0, ∆, . . . , K∆

Thus,

X

J

j=1

(μj − μ ̄·)

2 = 2(K2∆2 + . . . + 12∆2

) = ∆2(K)(K + 1)(2K + 1)/3

Setting this equal to 1/2 as suggested by equation (41), we have

∆ = p

3/(2(K)(K + 1)(2K + 1)) (42)

So, in our simulations we took ∆m = (m − 1)∆/20 for ∆ given by (42) and m ∈ {1, . . . , 21}. (We

refer to m as the “noncentrality parameter index” in Figures 1-32.) This led to powers that varied

roughly between 0.05 and 1 as the noncentrality parameters increased in the I = 20, ρ = 0.7 case.

Powers were, of course, lower for lower I’s and ρ’s. They were higher for higher I’s, and ρ’s.

9 Appendix B — An Analysis of Covariance Power Calculation

Analyses of covariance have been treated in statistical textbooks for many years. See, for example,

Scheff ́e (1959). However, for personal archival purposes, in this appendix we present a derivation

of the appropriate sums of squares (and thus non-centrality parameter) in the one factor case with

24
one concomitant variable. We used the results of this calculation in the power simulation described

in Section 2.

Suppose that we have J “treatments” and I replications for each treatment. We have

y = Xβ + 

where the elements of y are y11, . . . , yI1, . . . , y1J , . . . , yIJ , the first column of X contains I ones

followed by I(J − 1) zeros, . . . , the Jth column of X contains I(J − 1) zeros followed by I ones,

the elements of the J + 1th column of X are x11, . . . , xI1, . . . , x1J , . . . , xIJ where the xij ’s are the

values of the concomitant variable,

β ≡





μ1

.

.

.

μJ

b





where the μj ’s are the treatment effects and b is the slope associated with the concomitant variable,

and

 ∼ N(0, IIJ×IJ )

To proceed with Scheff ́e’s approach, we need aj , j = 1, . . . , J − 1 that lie in the span of the

columns of X, and satisfy

a

T

1 X = ( 1 −1 0 . . . 0 0 )

.

.

.

a

T

J−1X = ( 1 0 . . . 0 −1 0 )

It is easy to check that aj , j ∈ {1, . . . , J − 1}, is the sum of two vectors. The first vector has

1/I as its first I elements, −1/I as elements jI + 1 through (j + 1)I, and zeros elsewhere.

To describe the second vector, we define

x ̄·j ≡

X

I

i=1

xij/I

x ̄·· ≡

X

J

j=1

X

I

i=1

xij/(JI)

and

D ≡

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

Then the second vector is given by





(x11 − x ̄·1)( ̄x·j+1 − x ̄·1)/D

.

.

.

(xI1 − x ̄·1)( ̄x·j+1 − x ̄·1)/D

.

.

.

(x1J − x ̄·J )( ̄x·j+1 − x ̄·1)/D

.

.

.

(xIJ − x ̄·J )( ̄x·j+1 − x ̄·1)/D





25
Now let u1, . . . , uJ−1 denote an orthonormal basis of the linear span of a1, . . . , aJ−1. The

numerator sum of squares of a test of the hypothesis that μ1 = μ2 = . . . = μJ is

SSnum =

J

X−1

j=1

(u

T

j y)

2

(43)

It is clear that the aj ’s are perpendicular to 1, the vector of JI 1’s, and to the J + 1th

column of X. We denote this column by x. Thus it is clear that u1, . . . , uJ−1, 1/

√

JI, (x −

1x ̄··)/

qPJ

j=1

PI

i=1(xij − x ̄··)

2 form an orthonormal basis of the linear span of the columns of X.

Another orthonormal basis of the linear span of the columns of X is w1, . . . , wJ+1 where for

j = 1, . . . , J, wj has the value 1/

√

I for the (j − 1)I + 1, . . . , jI components and 0 elsewhere, and

wJ+1 =





(x11 − x ̄·1)/

√

D

.

.

.

(xI1 − x ̄·1)/

√

D

.

.

.

(x1J − x ̄·J )/

√

D

.

.

.

(xIJ − x ̄·J )/

√

D





Thus,

J

X

+1

j=1

(u

T

j y)

2 =

J

X

+1

j=1

(wT

j y)

2

(44)

From results (43) and (44), we have

SSnum =

J

X

+1

j=1

(wT

j y)

2

−



y

T 1/

√

JI 2

−



y

T

(x − 1x ̄··)/

vuutX

J

j=1

X

I

i=1

(xij − x ̄··)

2





2

=

X

J

j=1

Iy ̄

2

·j +





X

J

j=1

X

I

i=1

(yij − y ̄·j )(xij − x ̄·j )





2

/

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2 − IJy ̄2

··

−





X

J

j=1

X

I

i=1

(yij − y ̄··)(xij − x ̄··)





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

=

X

J

j=1

I( ̄y·j − y ̄··)

2 +





X

J

j=1

X

I

i=1

(yij − y ̄·j )(xij − x ̄·j )





2

/

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

−





X

J

j=1

X

I

i=1

(yij − y ̄··)(xij − x ̄··)





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

(45)

26
Now we know that we can obtain the noncentrality parameter, δ, for performing power calculations by replacing y in the numerator sum of squares by Xβ and dividing the result by σ

2

. That

is, in the current case

yij is replaced by μj + bxij

y ̄·j is replaced by μj + bx ̄·j

y ̄·· is replaced by  ̄μ· + bx ̄··

Applying this to result (45), we have

σ

2 × δ =

X

J

j=1

I (μj − μ ̄· + b( ̄x·j − x ̄··))2 +





X

J

j=1

X

I

i=1

b(xij − x ̄·j )

2





2

/

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

−





X

J

j=1

X

I

i=1

(μj − μ ̄· + b(xij − x ̄··)) (xij − x ̄··)





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

=

X

J

j=1

I(μj − μ ̄·)

2 + 2b

X

J

j=1

I(μj − μ ̄·)( ̄x·j − x ̄··) + b

2X

J

j=1

I( ̄x·j − x ̄··)

2

+ b

2X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

−





X

J

j=1

I(μj − μ ̄·)( ̄x·j − x ̄··) + b

X

J

j=1

X

I

i=1

(xij − x ̄··)

2





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

(46)

Expanding the last term in (46) and cancelling, we obtain

σ

2 × δ =

X

J

j=1

I(μj − μ ̄·)

2 −





X

J

j=1

I(μj − μ ̄·)( ̄x·j − x ̄··)





2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

(47)

So, by the Cauchy-Schwarz theorem,

σ

2 × δ ≥

X

J

j=1

I(μj − μ ̄·)

2 −





X

J

j=1

I(μj − μ ̄·)

2X

J

j=1

I( ̄x·j − x ̄··)

2



 /

X

J

j=1

X

I

i=1

(xij − x ̄··)

2

=

X

J

j=1

I(μj − μ ̄·)

2



1 −

X

J

j=1

I( ̄x·j − x ̄··)

2

/

X

J

j=1

X

I

i=1

(xij − x ̄··)

2





Note that we can make this lower bound on the noncentrality parameter larger by minimizing

X

J

j=1

( ̄x·j − x ̄··)

2

that is, by making the  ̄x·j as similar as possible. A predictor sort allocation helps to do this. Thus,

even if one intends to perform an analysis of covariance, a predictor sort allocation can improve

performance.

27
Because w1, . . . , wJ+1 form an orthonormal basis for the linear span of the columns of X, the

denominator sum of squares for a test of the hypothesis that μ1 = μ2 = . . . = μJ is

SSden = y

T y −

J

X

+1

j=1

(wT

j y)

2

= y

T y −

X

J

j=1

Iy ̄

2

·j

−





X

J

j=1

X

I

i=1

(yij − y ̄·j )(xij − x ̄·j )





2

/

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2 −





X

J

j=1 X

I

i=1

xijyij − Ix ̄·jy ̄·j

!



2

/

X

J

j=1 X

I

i=1

x

2

ij − Ix ̄

2

·j

!

(48)

From equations (45) and (48), we can calculate the F statistic for testing the hypothesis that

μ1 = μ2 = . . . = μJ :

Fancova =



SSnum

J − 1



/



SSden

JI − (J + 1)

The power of this test is

Powerancova = 1 − NCFJ−1,JI−(J+1)(δ)(F

−1

J−1,JI−(J+1)(1 − α)) (49)

where the noncentrality parameter, δ, is given by equation (47), α is the significance level,

NCFJ−1,JI−(J+1)(δ)(s) is the noncentral F distribution function (evaluated at s) with noncentrality parameter δ, J − 1 numerator degrees of freedom, and JI − (J + 1) denominator degrees of

freedom, and F

−1

J−1,JI−(J+1) is the inverse distribution function of a central F distribution with

J − 1 numerator and JI − (J + 1) denominator degrees of freedom.

10 Appendix C — Relations between Analysis of Covariance and

Maximum Likelihood Estimators

Given that least squares and maximum likelihood estimators are essentially equivalent (up to divisors for estimates of variances) for linear models with Gaussian errors, we would expect near

equivalence for analysis of covariance and maximum likelihood estimators. This expectation is

somewhat clouded by the fact that the probability models differ. In the maximum likelihood model

we treat the x values as random variables that are correlated with the response values. In the

analysis of covariance model, we treat the x’s as fixed constants. (As noted in Section 3, this can

lead a statistician/scientist to calculate an analysis of covariance confidence interval on a treatment

effect that is too narrow.)

In this appendix we establish that the two approaches yield essentially equivalent estimates and

this, of course, then yields (MLE) confidence intervals for the analysis of covariance estimators.

Here we restrict our attention to an analysis of covariance with one factor and one predictor. We

assume that there are J treatments and I observations per treatment.

The MLE is based on the following model:

28
yij = μj + σY



ρ(xij − μX)/σX +

p

1 − ρ

2zij

= μj − ρ(σY /σX)μX + ρ(σY /σX) xij +

p

1 − ρ

2 σY zij

= aj + b xij +

p

1 − ρ

2 σY zij

where the x’s are iid N(μX, σ2

X), the z’s are iid N(0, 1), and the x’s and z’s are independent. (This

is in the standard, non predictor sort case.)

Thus, we have the identities

aj = μj − ρ(σY /σX)μX (50)

b = ρ(σY /σX) (51)

and

σ =

p

1 − ρ

2 σY (52)

where a1, . . . , aJ , b, and σ are the parameters of the ANCOVA model, and μ1, . . . , μJ , μX, ρ, σX,

and σY are the parameters of the MLE model.

We know that the ANCOVA estimator of b is y

Tvb where

vb =





(x11 − x ̄·1)/D

.

.

.

(xI1 − x ̄·1)/D

.

.

.

(x1J − x ̄·J )/D

.

.

.

(xIJ − x ̄·J )/D





and

D ≡

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

so

ˆb =

PJ

j=1

PI

i=1(yij − y ̄·j )(xij − x ̄·j )

PJ

j=1

PI

i=1(xij − x ̄·j )

2

(53)

We also know that the ANCOVA estimator of aj is y

Tvj where vj is a sum of two vectors. The

first vector has 1/I in rows (j − 1)I + 1, . . . , jI and zeros elsewhere. The second vector equals

−vbx ̄·j

so

aˆj =  ̄y·j − ˆbx ̄·j (54)

Finally, from (48), we know that we can calculate ˆσ

2 via

(IJ −(J +1))ˆσ

2 =

X

J

j=1

X

I

i=1

y

2

ij −

X

J

j=1

Iy ̄

2

·j −





X

J

j=1

X

I

i=1

(yij − y ̄·j )(xij − x ̄·j )





2

/

X

J

j=1

X

I

i=1

(xij −x ̄·j )

2

(55)

29
Now, formally, there is no μX parameter in the analysis of covariance model. However, given

equations (50) and (51), it is reasonable to take

aˆj + ˆbx ̄··

as the analysis of covariance estimator of μj . Given equation (54) this implies that

μˆancova,j ≡ aˆj + ˆbx ̄·· =  ̄y·j − ˆb( ̄x·j − x ̄··) (56)

is our ANCOVA estimator of μj . From appendix A.2 of Verrill et al. (2004), we know that the

maximum likelihood estimator of μj is

μˆml,j ≡ y ̄·j − ρˆ(ˆσY /σˆX)( ̄x·j − x ̄··) (57)

where ˆρ, ˆσY , and ˆσX are the maximum likelihood estimators of ρ, σY , and σX. Thus, to establish

the equivalence of the ANCOVA and MLE estimators of μj , we need to show that

ˆb = ˆρ(ˆσY /σˆX)

From results (36), (38), and (39) in Verrill et al. (2004) we have

ρˆ(ˆσY /σˆX) =

PJ

j=1

PI

i=1(xij − x ̄·j )(yij − y ̄·j )

PJ

j=1

PI

i=1(xij − x ̄··)

2 − I

PJ

j=1( ̄x·j − x ̄··)

2

=

PJ

j=1

PI

i=1(xij − x ̄·j )(yij − y ̄·j )

PJ

j=1

PI

i=1(xij − x ̄·j )

2

= ˆb

We can also establish the relation between the ANCOVA ˆσ

2 and the MLE ˆσ2

Y

. From results

(26), (28), (36), and (37) in Verrill et al. (2004) we have

σˆ

2

Y =

X

J

j=1

X

I

i=1

(yij − ( ̄y·j − ρˆ(ˆσY /σˆX)( ̄x·j − x ̄··)))2/(IJ)

=

X

J

j=1

X

I

i=1



yij − y ̄·j + ˆb( ̄x·j − x ̄··)

2

/(IJ)

=





X

J

j=1

X

I

i=1

(yij − y ̄·j )

2 + ˆb2X

J

j=1

I( ̄x·j − x ̄··)

2



 /(IJ)

=





X

J

j=1

X

I

i=1

(yij − y ̄·j )

2 +

ρˆ

2σˆ2

Y

σˆ

2

X

X

J

j=1

I( ̄x·j − x ̄··)

2



 /(IJ)

so

IJσˆ

2

Y − IJρˆ

2σˆY2

PI

i=1 I( ̄x·j − x ̄··)

2

PJ

j=1

PI

i=1(xij − x ̄··)

2

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

or

IJσˆ

2

Y

 

1 − ρˆ

2

PI

i=1 I( ̄x·j − x ̄··)

2

PJ

j=1

PI

i=1(xij − x ̄·j )

2 +

PJ

j=1 I( ̄x·j − x ̄··)

2

!

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

30
Thus

IJσˆ

2

Y

 

1 − ρˆ

2 + ˆρ2 − ρˆ2

PI

i=1 I( ̄x·j − x ̄··)

2

PJ

j=1

PI

i=1(xij − x ̄·j )

2 +

PJ

j=1 I( ̄x·j − x ̄··)

2

!

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

and

σˆ

2

Y

(1 − ρˆ

2

) + ˆσ

2

Y ρˆ

2

 

1 −

PI

i=1 I( ̄x·j − x ̄··)

2

PJ

j=1

PI

i=1(xij − x ̄·j )

2 +

PJ

j=1 I( ̄x·j − x ̄··)

2

!

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

/(IJ)

or

σˆ

2

Y

(1 − ρˆ

2

) + ˆσ

2

Y ρˆ

2

 PJ

j=1

PI

i=1(xij − x ̄·j )

2

PJ

j=1

PI

i=1(xij − x ̄·j )

2 +

PJ

j=1 I( ̄x·j − x ̄··)

2

!

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

/(IJ)

so

σˆ

2

Y

(1 − ρˆ

2

) = X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

/(IJ) −

σˆ

2

Y

ρˆ

2

σˆ

2

X

 PJ

j=1

PI

i=1(xij − x ̄·j )

2

IJ !

=

X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

/(IJ) − ˆb

2

 PJ

j=1

PI

i=1(xij − x ̄·j )

2

IJ !

(58)

Thus, by results (53), (55), and (58)

σˆ

2

Y

(1 − ρˆ

2

) = X

J

j=1

X

I

i=1

(yij − y ̄·j )

2

/(IJ) −

 PJ

j=1

PI

i=1(yij − y ̄·j )(xij − x ̄·j )

PJ

j=1

PI

i=1(xij − x ̄·j )

2

!2 PJ

j=1

PI

i=1(xij − x ̄·j )

2

IJ

=





X

J

j=1

X

I

i=1

(yij − y ̄·j )

2 −

PJ

j=1

PI

i=1(yij − y ̄·j )(xij − x ̄·j )

2

PJ

j=1

PI

i=1(xij − x ̄·j )

2



 /(IJ)

= ˆσ

2

(IJ − (J + 1))/(IJ)

11 Appendix D — A Maximum Likelihood Variance Calculation

One can show (see appendix A of Verrill et al. (2004) and, for example, theorem 6.1 of Chapter

6 of Lehmann (1983)) that when everything is independent, and there are J “treatments” and I

bivariate normal Xij , Yij pairs for each treatment such that, for the jth treatment,



Xij

Yij 

∼ N

 μX

μj



,



σ

2

X ρσXσY

ρσXσY σ

2

Y



then the maximum likelihood information matrix for ˆμX,ˆμ1, . . . , μˆJ ,ˆρ,ˆσX, ˆσY is given by



A BT

B C 

where A is a J + 1 by J + 1 matrix, B is a 3 by J + 1 matrix of zeros, and C is a 3 by 3 matrix.

Thus, we can obtain the asymptotic covariance matrix of ˆμX,ˆμ1, . . . , μˆJ by inverting A. Let the

31
ijth element of A be denoted by aij . Verrill et al. (2004) (also see theorem 6.1 of chapter 6 of

Lehmann (1983) for the construction of the information matrix when there are J samples from

distinct distributions) show that

a11 = 1/((1 − ρ

2

)σ

2

X)

and, for j ∈ {2, . . . , J + 1},

ajj = 1/(J(1 − ρ

2

)σ

2

Y

)

and

aj1 = a1j = −ρ/(J(1 − ρ

2

)σXσY )

The remaining elements of A are zeros. Using well-known blockwise inversion formulas (see, for

example, problem 2.7 in section 1b of Rao (1973)), one can obtain the inverse of A. Alternatively,

one can begin with a 3 by 3 A (J = 2), go to a 4 by 4 A (J = 3), and quickly see a pattern.

In fact, one can check that if A has a11 = c, aj1 = a1j = b for j ∈ {2, . . . , J + 1}, ajj = a for

j ∈ {2, . . . , J + 1}, and aij = 0 elsewhere, then

A−1 = B/(ca2 − Jab2)

where b11 = a

2

, bj1 = b1j = −ba for j ∈ {2, . . . , J + 1}, bjj = ca − (J − 1)b

2

for j ∈ {2, . . . , J + 1},

and bij = b

2

elsewhere.

Applying this result to the information matrix, we have (assuming that we can verify the

conditions of Lehmann’s chapter 6 theorem 6.1)

√

IJ(ˆμml,j − μj )

D

→ N(0, Jσ2

Y

(1 − ρ

2 + ρ2

/J))

or

√

I(ˆμml,j − μj )

D

→ N(0, σ2

Y

(1 − ρ

2 + ρ2

/J)) (59)

where ˆμml,j is given by (57) in Appendix C. That is, the asymptotic variance of the maximum

likelihood estimate of μj is the value that we found for Y ̄

·j in the predictor sort case.

12 Appendix E — Asymptotic Distribution of the Analysis of

Covariance Estimator of μj after a Predictor Sort Allocation

As noted in Section 3, Verrill (1999) established that

√

I( ̄y·j − μj )

D

→ N(0, σ2

Y

(1 − ρ

2 + ρ2

/J)) (60)

in the J treatments, I blocks predictor sort case.

Our claim is that in this case, we also have (see result (56))

√

I (ˆμancova,j − μj ) = √I



y ̄·j − ˆb( ̄x·j − x ̄··) − μj



D

→ N



0, σ2

Y

(1 − ρ

2 + ρ2

/J)



(61)

where ˆb is the standard analysis of covariance estimate of b in the model

yij = aj + b × xij + ij

Given (60), we can establish (61) by showing that, given a predictor sort allocation,

√

I

ˆb ( ̄x·j − x ̄··)

p→ 0

32
as I → ∞. In the notation introduced at the beginning of Section 2

√

I ( ̄x·j − x ̄··) = √I



x ̄k(·,j),n − x ̄k(·,·),n

which converges in probability to 0 as I → ∞ by work done in the proof of (A.1) in the appendix

to Verrill (1993). Thus, we need only establish that ˆb = Op(1) to complete the proof of the claim.

We know that, in standard ANCOVA notation,

ˆb =

X

J

j=1

X

I

i=1

(yij − y ̄·j )(xij − x ̄·j )/

X

J

j=1

X

I

i=1

(xij − x ̄·j )

2

In the predictor sort notation of equation (2) this becomes

ˆb = σY ρ/σX + σY

p

1 − ρ

2X

J

j=1

X

I

i=1

(pij − p ̄·j )(xk(i,j),n − x ̄k(·,j),n)/

X

J

j=1

X

I

i=1

(xk(i,j),n − x ̄k(·,j),n)

2

(62)

By the Cauchy-Schwarz theorem we have

X

J

j=1

X

I

i=1

(pij − p ̄·j )(xk(i,j),n − x ̄k(·,j),n)/

X

J

j=1

X

I

i=1

(xk(i,j),n − x ̄k(·,j),n)

2

≤





X

J

j=1

X

I

i=1

(pij − p ̄·j )

2





1/2 



X

J

j=1

X

I

i=1

(xk(i,j),n − x ̄k(·,j),n)

2





1/2

/

X

J

j=1

X

I

i=1

(xk(i,j),n − x ̄k(·,j),n)

2

=

 PJ

j=1

PI

i=1(pij − p ̄·j )

2/(JI)

PJ

j=1

PI

i=1(xk(i,j),n − x ̄k(·,j),n)

2/(JI)

!1/2

≡ (N/D)

1/2

(63)

We know that

N

p→ 1 (64)

as I → ∞. Also,

D =

X

J

j=1

X

I

i=1

x

2

k(i,j),n/(JI) −

X

J

j=1

x ̄

2

k(·,j),n/J (65)

We know that

X

J

j=1

X

I

i=1

x

2

k(i,j),n/(JI) = X

J

j=1

X

I

i=1

x

2

ij/(JI)

p→ μ

2

X + σ

2

X (66)

as I → ∞. Also, for all j ∈ {1, . . . , J},

|x ̄k(·,j),n − x ̄··| = |x ̄k(·,j),n − x ̄k(·,·),n|

= |

X

I

i=1

(xk(i,j),n − x ̄k(i,·),n)/I|

≤

X

I

i=1

|xk(i,j),n − x ̄k(i,·),n|/I

≤

X

I

i=1

(xiJ,n − x(i−1)J+1,n)/I

≤ (xn,n − x1,n)/I p→ 0

33
as I → ∞ by the lemma proven in the appendix to Verrill (1993). Together with the fact that

x ̄··

p→ μX

this yields

X

J

j=1

x ̄

2

k(·,j),n/J p→ μ

2

X (67)

Results (65), (66), and (67) yield

D

p→ σ

2

X (68)

as I → ∞. Results (62), (63), (64), and (68) establish that ˆb = Op(1) which in turn completes the

proof of result (61).

13 Appendix F — Proof of the Tukey Multiple Comparison

Theorem (Theorem 5)

Here we are using the notation introduced at the beginning of Section 5. We work with the first

factor, but the same proof will hold for all F factors.

Consider the Tukey numerator

QI ≡ max

l1,l2∈{1,...,K1}

p

IK2 . . . KF |Y ̄

·l1·...· − Y ̄·l2·...·

|

Under the null hypothesis that μ1·...· = . . . = μK1·...·, we have

QI = max

l1,l2∈{1,...,K1}

p

IK2 . . . KF





ρX ̄

k(·l1·...·),n +

p

1 − ρ

2P ̄

·l1·...· −



ρX ̄

k(·l2·...·),n +

p

1 − ρ

2P ̄

·l2·...·







σY

= max

l1,l2∈{1,...,K1}





ρ

p

IK2 . . . KF



X ̄

k(·l1·...·),n − X ̄k(·l2·...·),n

+

p

1 − ρ

2

p

IK2 . . . KF



P ̄

·l1·...· − P ̄·l2·...·









σY

= max

l1,l2∈{1,...,K1}

|∆I,l1,l2 + (NI,l1 − NI,l2)| σY

p

1 − ρ

2 (69)

where

∆I,l1,l2 ≡ ρ

p

IK2 . . . KF



X ̄

k(·l1·...·),n − X ̄k(·l2·...·),n

/

p

1 − ρ

2

and the NI,l’s are independent N(0,1)’s that are also independent of the ∆I,l1,l2’s.

Now, under predictor sort allocation, for all l1, l2 ∈ {1, . . . , K1},

√

I



X ̄

k(·l1·...·),n − X ̄k(·l2·...·),n



 ≤

X

I

i=1



X ̄

k(il1·...·),n − X ̄k(il2·...·),n



 /

√

I

≤

X

I

i=1

|max X in block i − min X in block i| /

√

I

≤ (max overall X − min overall X)/

√

I

which converges in probability to zero by the lemma proven in the appendix of Verrill (1993).

Thus, for all l1, l2 pairs,

∆I,l1,l2

p→ 0

34
as I → ∞, so

max

l1,l2∈{1,...,K1}

|∆I,l1,l2|

p→ 0 (70)

Next, for all l1, l2 ∈ {1, . . . , K1},

|∆I,l1,l2 + (NI,l1 − NI,l2)| ≤ |∆I,l1,l2| + |NI,l1 − NI,l2|

≤ max

m1,m2∈{1,...,K1}

|∆I,m1,m2| + max

m1,m2∈{1,...,K1}

|NI,m1 − NI,m2|

so

max

m1,m2∈{1,...,K1}

|∆I,m1,m2 + (NI,m1 − NI,m2)| ≤ max

m1,m2∈{1,...,K1}

|∆I,m1,m2|+ max

m1,m2∈{1,...,K1}

|NI,m1 − NI,m2|

(71)

Also, for all l1, l2 ∈ {1, . . . , K1},

|∆I,l1,l2 + (NI,l1 − NI,l2)| ≥ |NI,l1 − NI,l2| − |∆I,l1,l2|

≥ |NI,l1 − NI,l2| − max

m1,m2∈{1,...,K1}

|∆I,m1,m2|

so for all l1, l2 ∈ {1, . . . , K1},

max

m1,m2∈{1,...,K1}

|∆I,m1,m2 + (NI,m1 − NI,m2

)| ≥ |∆I,l1,l2 + (NI,l1 − NI,l2)|

≥ |NI,l1 − NI,l2| − max

m1,m2∈{1,...,K1}

|∆I,m1,m2

|

and thus

max

m1,m2∈{1,...,K1}

|∆I,m1,m2 + (NI,m1 − NI,m2)| ≥ max

m1,m2∈{1,...,K1}

|NI,m1 − NI,m2|− max

m1,m2∈{1,...,K1}

|∆I,m1,m2|

(72)

Results (71) and (72) yield

AI − BI ≤ CI ≤ AI + BI

where

AI ≡ max

m1,m2∈{1,...,K1}

|NI,m1 − NI,m2

|

BI ≡ max

m1,m2∈{1,...,K1}

|∆I,m1,m2|

and

CI ≡ max

m1,m2∈{1,...,K1}

|∆I,m1,m2 + (NI,m1 − NI,m2)|

Thus, for any x,

Prob(AI + BI ≤ x) ≤ Prob(CI ≤ x) ≤ Prob(AI − BI ≤ x) (73)

Now let FR(K1) denote the distribution of the range of K1 independent N(0,1)’s (that is, the

distribution of AI ), and let an arbitrary  > 0 and an arbitrary x be given. Then, since FR(K1)is

a continuous function (see, for example, David (1981)), we can find a δ > 0 such that

FR(K1)(x − δ) > FR(K1)(x) −  (74)

and

FR(K1)(x + δ) < FR(K1)(x) +  (75)

35
Also, by result (70), we can find an N,δ such that I > N,δ implies that

Prob(BI ≤ δ) > 1 −  (76)

Thus, by results (74) and (76), (recall that AI and BI are statistically independent), for I > N,δ,

Prob(AI + BI ≤ x) ≥ Prob(AI ≤ x − δ) × Prob(BI ≤ δ)

= FR(K1)(x − δ) × Prob(BI ≤ δ)

> (FR(K1)(x) − )(1 − ) > FR(K1)(x) − 2 (77)

Similarly,

{AI ≤ x + BI} ⊂ {AI ≤ x + BI and BI ≥ δ} ∪ {AI ≤ x + BI and BI < δ}

so

Prob(AI ≤ x + BI ) ≤ Prob(BI ≥ δ) + Prob(AI ≤ x + δ)

and for I > N,δ, by results (75) and (76),

Prob(AI − BI ≤ x) <  + Prob(AI ≤ x + δ) < FR(K1)(x) + 2 (78)

Results (69), (73), (77), and (78) imply that under a predictor sort allocation, for all x, as I → ∞,

Prob 

QI/



σY

p

1 − ρ

2



≤ x



→ FR(K1)(x)

or

QI/



σY

p

1 − ρ

2



D

→ FR(K1)(79)

As noted in results (32) and (34)

sub

p→ σY (80)

in the unblocked case, and

sb

p→ σY

p

1 − ρ

2 (81)

in the blocked case.

From results (79)–(81), the theorem follows.

14 Appendix G — Proof of Theorem 1 in the Multi-factor Case

14.1 Numerator of the test statistic

We assume that we have F factors, Kj levels in the jth factor, and I replicates per cell (or I blocks

in the blocked case). For a test of the significance of the first factor, the numerator sum of squares

in both the blocked and unblocked cases is

SSnum =

X

K1

j1=1

IK2 . . . KF ( ̄y·j1·...· − y ̄·...·)

2

36
Thus, in the null case, using the notation introduced at the beginning of Section 5, we have

SSnum =

X

K1

j1=1

IK2 . . . KF × σ

2

Y



ρ( ̄xk(·j1·...·),n − x ̄k(·...·),n) + p1 − ρ

2( ̄p·j1·...· − p ̄·...·)

2

=

X

K1

j1=1

IK2 . . . KF × σ

2

Y × ρ

2



x ̄k(·j1·...·),n − x ̄k(·...·),n2

+ 2 X

K1

j1=1

IK2 . . . KF × σ

2

Y × ρ ×

p

1 − ρ

2



x ̄k(·j1·...·),n − x ̄k(·...·),n

( ̄p·j1·...· − p ̄·...·)

+

X

K1

j1=1

IK2 . . . KF × σ

2

Y × (1 − ρ

2

) ( ̄p·j1·...· − p ̄·...·)

2

≡ T1 + T2 + T3 (82)

It is well known that

T3 ∼ σ

2

Y × (1 − ρ

2

) × χ

2

K1−1

(83)

Also, for j1 ∈ {1, . . . , K1},

√

I



x ̄k(·j1·...·),n − x ̄k(·...·),n



 =

√

I











X

I

i=1



x ̄k(ij1·...·),n − x ̄k(i·...·),n/I











≤

X

I

i=1



x ̄k(ij1·...·),n − x ̄k(i·...·),n



 /

√

I

≤

X

I

i=1

(max x in ith block − min x in ith block) /

√

I

≤ (max overall x − min overall x) /

√

I

p→ 0

as I

p→ ∞ by the lemma in the appendix of Verrill (1993). Thus,

T1

p→ 0 (84)

as I

p→ ∞. By results (83) and (84) and the Cauchy-Schwarz theorem,

T2

p→ 0 (85)

Finally, by results (82), (83), (84), and (85), we have

SSnum

D

→ σ

2

Y × (1 − ρ

2

) × χ

2

K1−1

(86)

14.2 Denominator of the test statistic in the predictor sort blocked case

We assume that we have F factors, Kj levels in the jth factor, and I blocks (formed by specimens

with adjacent [randomized within a block] order statistics of the predictor). In this case the

denominator sum of squares is

SSden,bl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(yij1...jF − ( ̄y·...· + ( ̄yi·...· − y ̄·...·) + . . . + ( ̄y·...·jF − y ̄·...·)))2(87)

37
It is easy to show that the μ’s contained in equation (87) cancel and we are left with

SSden,bl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(diffij1...jF)

2

(88)

where, using notation similar to that introduced at the beginning of Section 5,

diffij1...jF = wij1...jF − w ̄i·...·

− (  ̄w·j1·...· − w ̄·...·)

.

.

.

− (  ̄w·...·jF − w ̄·...·)

and

wij1...jF = σY



ρxk(ij1...jF ),n +

p

1 − ρ

2pij1...jF



Now the F-statistic (nominal F) denominator in the blocked predictor sort case is

SSden,bl/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1))

We first claim that

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(  ̄w·j1·...· − w ̄·...·)

2

/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ 0 (89)

(and the similar results for the corresponding j2, . . . , jF terms). It is clear that (89) will follow if

we can show that

X

K1

j1=1

(  ̄w·j1·...· − w ̄·...·)

2 p→ 0

as I → ∞ which, in turn, will follow if

X

K1

j1=1

|w ̄·j1·...· − w ̄·...·|

p→ 0 (90)

as I → ∞. We have

w ̄·j1·...· − w ̄·...· = σY



ρ( ̄xk(·j1·...·),n − x ̄k(·...·),n) + p

1 − ρ

2( ̄p·j1·...· − p ̄·...·)



(91)

We know that

p ̄·j1·...· − p ̄·...·

p→ 0 (92)

as I → ∞ because both  ̄p’s are averages of more than I iid N(0, 1)’s. We also have



x ̄k(·j1·...·),n − x ̄k(·...·),n



 ≤

X

I

i=1



x ̄k(ij1·...·),n − x ̄k(i·...·),n



 /I

≤

X

I

i=1

(max x in block i − min x in block i)/I

≤ (overall max x − overall min x)/I (93)

38
which converges in probability to 0 as I → ∞ by the lemma in the appendix of Verrill (1993). By

results (91), (92), and (93), results (90) and thus (89) follow.

Next we claim that

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(wij1...jF − w ̄i·...·)

2

/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ (1 − ρ

2

)σ

2

Y

(94)

We have

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(wij1...jF − w ̄i·...·)

2

=

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

ρ

2σ2

Y



xk(ij1...jF ),n − x ̄k(i·...·),n2

+

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

2ρ

p

1 − ρ

2σ

2

Y



xk(ij1...jF ),n − x ̄k(i·...·),n(pij1...jF − p ̄i·...·)

+

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(1 − ρ

2

)σ

2

Y

(pij1...jF − p ̄i·...·)

2

≡ T1 + T2 + T3 (95)

First, we claim that

T1/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ 0 (96)

as I → ∞. This follows from a slightly modified version of result (93) above:

X

I

i=1



xk(ij1...jF ),n − x ̄k(i·...·),n



 /

√

I ≤

X

I

i=1

(max x in block i − min x in block i)/

√

I

≤ (overall max x − overall min x)/

√

I

which converges in probability to 0 as I → ∞ by the lemma in the appendix of Verrill (1993), so

X

I

i=1



xk(ij1...jF ),n − x ̄k(i·...·),n2/I p→ 0

as I → ∞.

Next, we claim that

T3/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ (1 − ρ

2

)σ

2

Y

(97)

We know that for i ∈ {1, . . . , I},

X

K1

j1=1

. . . X

KF

jF =1

(pij1...jF − p ̄i·...·)

2

/(K1 . . . KF − 1) ∼ χ

2

K1...KF −1/(K1 . . . KF − 1)

39
which has expectation 1. Thus,

X

I

i=1





X

K1

j1=1

. . . X

KF

jF =1

(pij1...jF − p ̄i·...·)

2

/(K1 . . . KF − 1)



 /I p→ 1

as I → ∞ which establishes result (97).

Results (96) and (97) and the Cauchy-Schwarz theorem yield

T2/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ 0 (98)

Results (95), (96), (97), and (98), yield result (94).

Results (88), (89), and (94), and the Cauchy-Schwarz theorem yield

SSden,bl/(IK1 . . . KF − (I + K1 − 1 + . . . + KF − 1)) p→ (1 − ρ

2

)σ

2

Y

(99)

14.3 Denominator of the test statistic in the predictor sort unblocked case

We assume that we have F factors, Kj levels in the jth factor, and I blocks (formed by specimens

with adjacent [randomized within a block] order statistics of the predictor). However, we further

assume that despite such an allocation, the analysis ignores the “blocking” and acts as if there were

I replicates for each cell. In this case the denominator sum of squares is

SSden,unbl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(yij1...jF − ( ̄y·...· + ( ̄y·j1·...· − y ̄·...·) + . . . + ( ̄y·...·jF − y ̄·...·)))2

(100)

It is easy to show that the μ’s contained in equation (100) cancel and we are left with

SSden,unbl =

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(diffij1...jF

)

2

(101)

where, using notation similar to that introduced at the beginning of Section 5,

diffij1...jF = wij1...jF − w ̄·...·

− (  ̄w·j1·...· − w ̄·...·)

.

.

.

− (  ̄w·...·jF − w ̄·...·)

and

wij1...jF = σY



ρxk(ij1...jF ),n +

p

1 − ρ

2zij1...jF



Now the F-statistic (nominal F) denominator in the unblocked predictor sort case is

SSden,unbl/(IK1 . . . KF − (K1 + K2 − 1 + . . . + KF − 1))

We first claim that

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(  ̄w·j1·...· − w ̄·...·)

2

/(IK1 . . . KF − (K1 + K2 − 1 + . . . + KF − 1)) p→ 0 (102)

40
(and the similar results for the corresponding j2, . . . , jF terms). This follows from the reasoning

used to establish result (89).

Next we claim that

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(wij1...jF − w ̄·...·)

2

/(IK1 . . . KF − (K1 + K2 − 1 + . . . + KF − 1)) p→ σ

2

Y

(103)

This follows immediately from the fact that the summation in (103) is just a reordering of the

summation

X

I

i=1

X

K1

j1=1

. . . X

KF

jF =1

(sij1...jF − s ̄·...·)

2

where the s’s are iid N(0, σ2

Y

)’s.

Results (101), (102), and (103), and the Cauchy-Schwarz theorem yield

SSden,unbl/(IK1 . . . KF − (K1 + K2 − 1 + . . . + KF − 1)) p→ σ

2

Y

(104)

41
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 2 3 1 0.052 0.034 0.124 0.051 0.050 0.052 0.051 0.050 0.050

0.50 2 3 2 0.051 0.034 0.126 0.052 0.050 0.052 0.051 0.050 0.050

0.50 2 3 3 0.052 0.035 0.127 0.053 0.051 0.053 0.052 0.051 0.051

0.50 2 3 4 0.053 0.035 0.130 0.054 0.052 0.054 0.053 0.053 0.052

0.50 2 3 5 0.055 0.037 0.133 0.057 0.054 0.055 0.055 0.056 0.054

0.50 2 3 6 0.058 0.040 0.138 0.060 0.056 0.057 0.058 0.059 0.056

0.50 2 3 7 0.060 0.043 0.143 0.063 0.058 0.060 0.061 0.063 0.058

0.50 2 3 8 0.064 0.045 0.150 0.068 0.060 0.063 0.064 0.067 0.061

0.50 2 3 9 0.068 0.049 0.157 0.073 0.063 0.067 0.069 0.073 0.065

0.50 2 3 10 0.073 0.052 0.165 0.078 0.066 0.072 0.074 0.079 0.069

0.50 2 3 11 0.078 0.057 0.175 0.085 0.070 0.076 0.079 0.086 0.073

0.50 2 3 12 0.084 0.063 0.184 0.093 0.074 0.082 0.085 0.093 0.078

0.50 2 3 13 0.090 0.068 0.195 0.102 0.079 0.088 0.092 0.102 0.083

0.50 2 3 14 0.097 0.075 0.207 0.110 0.085 0.094 0.100 0.111 0.088

0.50 2 3 15 0.103 0.082 0.220 0.120 0.090 0.100 0.108 0.121 0.094

0.50 2 3 16 0.111 0.089 0.233 0.129 0.096 0.107 0.116 0.131 0.101

0.50 2 3 17 0.120 0.097 0.248 0.139 0.102 0.116 0.124 0.143 0.107

0.50 2 3 18 0.129 0.106 0.264 0.151 0.109 0.124 0.134 0.155 0.115

0.50 2 3 19 0.140 0.116 0.279 0.164 0.115 0.133 0.145 0.168 0.122

0.50 2 3 20 0.149 0.125 0.294 0.177 0.123 0.141 0.155 0.181 0.130

0.50 2 3 21 0.160 0.135 0.310 0.190 0.131 0.151 0.165 0.195 0.138

0.50 2 5 1 0.050 0.030 0.074 0.051 0.048 0.050 0.049 0.050 0.050

0.50 2 5 2 0.051 0.030 0.076 0.051 0.049 0.050 0.050 0.051 0.051

0.50 2 5 3 0.052 0.031 0.078 0.054 0.051 0.052 0.053 0.053 0.052

0.50 2 5 4 0.054 0.033 0.082 0.058 0.055 0.055 0.057 0.057 0.055

0.50 2 5 5 0.059 0.037 0.088 0.062 0.059 0.059 0.062 0.062 0.059

0.50 2 5 6 0.064 0.042 0.097 0.070 0.065 0.065 0.069 0.069 0.065

0.50 2 5 7 0.070 0.047 0.107 0.078 0.072 0.073 0.076 0.077 0.071

0.50 2 5 8 0.078 0.054 0.120 0.087 0.080 0.081 0.086 0.087 0.079

0.50 2 5 9 0.087 0.062 0.134 0.099 0.088 0.091 0.098 0.099 0.088

0.50 2 5 10 0.097 0.071 0.149 0.113 0.099 0.102 0.110 0.112 0.099

0.50 2 5 11 0.108 0.082 0.165 0.129 0.110 0.116 0.125 0.127 0.110

0.50 2 5 12 0.120 0.094 0.185 0.146 0.121 0.131 0.140 0.144 0.123

0.50 2 5 13 0.134 0.107 0.207 0.163 0.135 0.146 0.158 0.162 0.137

0.50 2 5 14 0.148 0.122 0.230 0.182 0.150 0.163 0.178 0.182 0.152

0.50 2 5 15 0.163 0.138 0.252 0.204 0.166 0.181 0.198 0.204 0.169

0.50 2 5 16 0.180 0.155 0.278 0.227 0.182 0.201 0.219 0.227 0.186

0.50 2 5 17 0.198 0.173 0.304 0.253 0.200 0.222 0.243 0.252 0.205

0.50 2 5 18 0.218 0.194 0.332 0.277 0.219 0.243 0.268 0.278 0.225

0.50 2 5 19 0.238 0.217 0.359 0.305 0.239 0.266 0.294 0.305 0.246

0.50 2 5 20 0.260 0.241 0.389 0.332 0.260 0.291 0.322 0.333 0.268

0.50 2 5 21 0.283 0.264 0.420 0.361 0.283 0.316 0.352 0.363 0.290

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 2, I = 3, 5

42
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 2 10 1 0.050 0.026 0.059 0.051 0.048 0.052 0.050 0.050 0.050

0.50 2 10 2 0.052 0.028 0.061 0.051 0.051 0.053 0.052 0.052 0.052

0.50 2 10 3 0.056 0.031 0.066 0.056 0.055 0.058 0.057 0.057 0.056

0.50 2 10 4 0.062 0.036 0.076 0.065 0.062 0.065 0.065 0.066 0.064

0.50 2 10 5 0.072 0.044 0.089 0.077 0.073 0.076 0.078 0.078 0.075

0.50 2 10 6 0.081 0.054 0.106 0.092 0.089 0.090 0.094 0.094 0.089

0.50 2 10 7 0.096 0.067 0.127 0.111 0.107 0.110 0.113 0.114 0.107

0.50 2 10 8 0.114 0.083 0.152 0.136 0.127 0.133 0.136 0.137 0.128

0.50 2 10 9 0.134 0.101 0.181 0.163 0.153 0.158 0.163 0.165 0.153

0.50 2 10 10 0.157 0.125 0.215 0.195 0.180 0.188 0.196 0.196 0.180

0.50 2 10 11 0.183 0.152 0.251 0.230 0.210 0.219 0.230 0.231 0.212

0.50 2 10 12 0.212 0.182 0.290 0.268 0.243 0.254 0.268 0.270 0.246

0.50 2 10 13 0.245 0.216 0.333 0.309 0.280 0.294 0.309 0.311 0.284

0.50 2 10 14 0.278 0.252 0.377 0.352 0.319 0.334 0.353 0.356 0.323

0.50 2 10 15 0.315 0.291 0.424 0.398 0.360 0.378 0.398 0.402 0.366

0.50 2 10 16 0.354 0.332 0.472 0.446 0.403 0.425 0.446 0.450 0.409

0.50 2 10 17 0.393 0.378 0.521 0.495 0.449 0.471 0.495 0.498 0.454

0.50 2 10 18 0.433 0.424 0.568 0.542 0.493 0.519 0.544 0.547 0.500

0.50 2 10 19 0.475 0.471 0.617 0.590 0.538 0.566 0.592 0.594 0.545

0.50 2 10 20 0.518 0.518 0.664 0.638 0.583 0.611 0.639 0.641 0.590

0.50 2 10 21 0.561 0.566 0.706 0.683 0.630 0.657 0.684 0.685 0.634

0.50 2 20 1 0.050 0.025 0.054 0.051 0.050 0.051 0.050 0.050 0.050

0.50 2 20 2 0.054 0.027 0.056 0.054 0.053 0.054 0.054 0.054 0.053

0.50 2 20 3 0.062 0.033 0.068 0.064 0.062 0.065 0.065 0.065 0.064

0.50 2 20 4 0.077 0.045 0.087 0.082 0.080 0.084 0.082 0.083 0.082

0.50 2 20 5 0.098 0.061 0.113 0.107 0.104 0.110 0.107 0.110 0.107

0.50 2 20 6 0.123 0.085 0.149 0.140 0.137 0.143 0.142 0.144 0.140

0.50 2 20 7 0.156 0.115 0.194 0.185 0.179 0.184 0.186 0.187 0.180

0.50 2 20 8 0.194 0.152 0.246 0.236 0.225 0.233 0.236 0.238 0.229

0.50 2 20 9 0.238 0.201 0.306 0.295 0.279 0.289 0.296 0.296 0.284

0.50 2 20 10 0.287 0.254 0.370 0.359 0.343 0.352 0.361 0.360 0.345

0.50 2 20 11 0.344 0.314 0.442 0.429 0.411 0.418 0.431 0.429 0.411

0.50 2 20 12 0.402 0.380 0.512 0.499 0.480 0.489 0.502 0.499 0.479

0.50 2 20 13 0.462 0.449 0.582 0.570 0.550 0.560 0.571 0.570 0.548

0.50 2 20 14 0.522 0.519 0.651 0.637 0.619 0.629 0.640 0.638 0.615

0.50 2 20 15 0.582 0.590 0.714 0.702 0.681 0.694 0.705 0.702 0.679

0.50 2 20 16 0.640 0.655 0.769 0.761 0.738 0.753 0.763 0.761 0.738

0.50 2 20 17 0.697 0.718 0.821 0.813 0.791 0.804 0.814 0.812 0.791

0.50 2 20 18 0.748 0.775 0.863 0.857 0.837 0.850 0.858 0.856 0.837

0.50 2 20 19 0.794 0.825 0.899 0.893 0.875 0.888 0.894 0.893 0.876

0.50 2 20 20 0.834 0.867 0.927 0.922 0.907 0.917 0.923 0.922 0.908

0.50 2 20 21 0.870 0.900 0.948 0.945 0.934 0.941 0.945 0.945 0.933

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 2, I = 10, 20

43
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 2 40 1 0.050 0.025 0.053 0.051 0.051 0.050 0.051 0.050 0.050

0.50 2 40 2 0.055 0.030 0.060 0.058 0.058 0.057 0.058 0.057 0.057

0.50 2 40 3 0.073 0.044 0.083 0.080 0.080 0.080 0.080 0.080 0.080

0.50 2 40 4 0.102 0.069 0.123 0.120 0.118 0.119 0.121 0.119 0.118

0.50 2 40 5 0.145 0.109 0.178 0.175 0.171 0.176 0.175 0.175 0.172

0.50 2 40 6 0.199 0.163 0.253 0.248 0.243 0.248 0.248 0.247 0.242

0.50 2 40 7 0.263 0.234 0.340 0.334 0.329 0.333 0.336 0.334 0.327

0.50 2 40 8 0.340 0.317 0.437 0.432 0.423 0.429 0.432 0.431 0.422

0.50 2 40 9 0.423 0.413 0.538 0.534 0.523 0.528 0.534 0.532 0.522

0.50 2 40 10 0.509 0.513 0.636 0.632 0.620 0.628 0.633 0.631 0.620

0.50 2 40 11 0.599 0.614 0.727 0.721 0.710 0.719 0.723 0.722 0.712

0.50 2 40 12 0.682 0.705 0.805 0.801 0.790 0.795 0.801 0.801 0.791

0.50 2 40 13 0.756 0.786 0.868 0.863 0.855 0.859 0.865 0.864 0.856

0.50 2 40 14 0.819 0.852 0.914 0.911 0.904 0.911 0.912 0.912 0.905

0.50 2 40 15 0.870 0.902 0.946 0.945 0.940 0.945 0.945 0.946 0.941

0.50 2 40 16 0.913 0.939 0.968 0.967 0.963 0.969 0.968 0.969 0.965

0.50 2 40 17 0.944 0.963 0.983 0.982 0.980 0.983 0.983 0.983 0.981

0.50 2 40 18 0.964 0.980 0.991 0.991 0.989 0.991 0.991 0.991 0.990

0.50 2 40 19 0.978 0.989 0.995 0.995 0.995 0.996 0.995 0.996 0.995

0.50 2 40 20 0.988 0.995 0.998 0.998 0.997 0.998 0.998 0.998 0.998

0.50 2 40 21 0.994 0.997 0.999 0.999 0.999 0.999 0.999 0.999 0.999

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 2, I = 40

44
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 5 3 1 0.049 0.020 0.092 0.046 0.049 0.051 0.047 0.050 0.050

0.50 5 3 2 0.049 0.020 0.092 0.046 0.049 0.052 0.048 0.050 0.050

0.50 5 3 3 0.049 0.020 0.092 0.047 0.049 0.052 0.049 0.051 0.051

0.50 5 3 4 0.050 0.020 0.092 0.047 0.050 0.053 0.050 0.051 0.051

0.50 5 3 5 0.051 0.021 0.094 0.049 0.051 0.054 0.051 0.053 0.052

0.50 5 3 6 0.052 0.022 0.096 0.050 0.052 0.055 0.053 0.054 0.054

0.50 5 3 7 0.053 0.023 0.098 0.052 0.054 0.057 0.054 0.056 0.055

0.50 5 3 8 0.055 0.023 0.100 0.054 0.055 0.059 0.056 0.058 0.057

0.50 5 3 9 0.057 0.025 0.103 0.056 0.057 0.061 0.059 0.061 0.060

0.50 5 3 10 0.059 0.026 0.107 0.059 0.060 0.064 0.062 0.064 0.062

0.50 5 3 11 0.062 0.028 0.111 0.062 0.062 0.067 0.065 0.067 0.065

0.50 5 3 12 0.065 0.029 0.115 0.065 0.065 0.070 0.068 0.071 0.069

0.50 5 3 13 0.068 0.031 0.119 0.069 0.068 0.074 0.072 0.075 0.073

0.50 5 3 14 0.072 0.034 0.125 0.073 0.072 0.078 0.076 0.079 0.077

0.50 5 3 15 0.076 0.036 0.131 0.077 0.076 0.083 0.081 0.084 0.081

0.50 5 3 16 0.080 0.039 0.138 0.082 0.081 0.088 0.086 0.090 0.086

0.50 5 3 17 0.084 0.042 0.145 0.088 0.086 0.094 0.092 0.096 0.092

0.50 5 3 18 0.089 0.045 0.153 0.095 0.091 0.099 0.098 0.102 0.097

0.50 5 3 19 0.094 0.049 0.161 0.101 0.097 0.105 0.104 0.109 0.104

0.50 5 3 20 0.099 0.053 0.169 0.108 0.104 0.111 0.111 0.117 0.111

0.50 5 3 21 0.105 0.057 0.178 0.115 0.111 0.119 0.119 0.125 0.118

0.50 5 5 1 0.047 0.018 0.066 0.048 0.051 0.049 0.050 0.050 0.050

0.50 5 5 2 0.047 0.018 0.066 0.049 0.051 0.049 0.050 0.050 0.050

0.50 5 5 3 0.048 0.019 0.068 0.050 0.052 0.051 0.051 0.051 0.051

0.50 5 5 4 0.049 0.019 0.070 0.052 0.053 0.052 0.053 0.053 0.053

0.50 5 5 5 0.051 0.020 0.073 0.054 0.055 0.054 0.056 0.055 0.055

0.50 5 5 6 0.054 0.021 0.076 0.057 0.058 0.058 0.059 0.058 0.058

0.50 5 5 7 0.057 0.023 0.080 0.060 0.061 0.061 0.063 0.062 0.062

0.50 5 5 8 0.060 0.025 0.086 0.064 0.065 0.066 0.068 0.067 0.066

0.50 5 5 9 0.065 0.028 0.092 0.069 0.070 0.071 0.072 0.072 0.071

0.50 5 5 10 0.069 0.030 0.099 0.075 0.076 0.078 0.079 0.078 0.077

0.50 5 5 11 0.074 0.033 0.108 0.083 0.084 0.085 0.086 0.086 0.084

0.50 5 5 12 0.080 0.037 0.117 0.090 0.091 0.092 0.095 0.094 0.092

0.50 5 5 13 0.086 0.041 0.127 0.098 0.099 0.100 0.104 0.103 0.100

0.50 5 5 14 0.094 0.046 0.139 0.109 0.109 0.110 0.115 0.113 0.110

0.50 5 5 15 0.102 0.052 0.152 0.119 0.120 0.122 0.126 0.125 0.121

0.50 5 5 16 0.111 0.058 0.165 0.133 0.129 0.133 0.138 0.137 0.132

0.50 5 5 17 0.121 0.066 0.180 0.146 0.142 0.146 0.152 0.151 0.145

0.50 5 5 18 0.131 0.073 0.195 0.160 0.155 0.160 0.167 0.166 0.159

0.50 5 5 19 0.143 0.083 0.212 0.174 0.170 0.175 0.183 0.183 0.175

0.50 5 5 20 0.155 0.093 0.230 0.191 0.185 0.192 0.200 0.200 0.191

0.50 5 5 21 0.169 0.104 0.249 0.210 0.203 0.209 0.218 0.219 0.209

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 5, I = 3, 5

45
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 5 10 1 0.047 0.016 0.057 0.049 0.051 0.046 0.051 0.050 0.050

0.50 5 10 2 0.048 0.016 0.058 0.050 0.052 0.047 0.052 0.051 0.051

0.50 5 10 3 0.050 0.017 0.061 0.053 0.055 0.049 0.054 0.053 0.053

0.50 5 10 4 0.053 0.018 0.065 0.056 0.060 0.053 0.058 0.057 0.057

0.50 5 10 5 0.057 0.021 0.072 0.061 0.064 0.058 0.064 0.062 0.062

0.50 5 10 6 0.062 0.024 0.080 0.068 0.072 0.065 0.071 0.069 0.069

0.50 5 10 7 0.069 0.028 0.089 0.078 0.080 0.075 0.079 0.079 0.078

0.50 5 10 8 0.077 0.033 0.100 0.089 0.091 0.086 0.091 0.090 0.089

0.50 5 10 9 0.087 0.039 0.114 0.103 0.104 0.099 0.105 0.103 0.102

0.50 5 10 10 0.099 0.046 0.131 0.118 0.120 0.115 0.121 0.119 0.117

0.50 5 10 11 0.114 0.056 0.151 0.136 0.137 0.134 0.140 0.138 0.135

0.50 5 10 12 0.129 0.067 0.174 0.158 0.157 0.154 0.161 0.159 0.156

0.50 5 10 13 0.145 0.080 0.198 0.181 0.179 0.178 0.184 0.183 0.179

0.50 5 10 14 0.165 0.095 0.225 0.207 0.206 0.204 0.211 0.210 0.206

0.50 5 10 15 0.187 0.113 0.255 0.237 0.234 0.233 0.242 0.240 0.235

0.50 5 10 16 0.211 0.135 0.288 0.269 0.265 0.267 0.273 0.273 0.267

0.50 5 10 17 0.237 0.158 0.324 0.303 0.300 0.301 0.309 0.308 0.301

0.50 5 10 18 0.265 0.184 0.363 0.340 0.335 0.339 0.346 0.346 0.338

0.50 5 10 19 0.295 0.213 0.403 0.380 0.374 0.378 0.386 0.386 0.377

0.50 5 10 20 0.325 0.246 0.444 0.420 0.413 0.418 0.428 0.428 0.418

0.50 5 10 21 0.359 0.280 0.486 0.463 0.454 0.460 0.472 0.471 0.460

0.50 5 20 1 0.049 0.013 0.051 0.048 0.049 0.048 0.049 0.050 0.050

0.50 5 20 2 0.050 0.014 0.054 0.050 0.051 0.050 0.051 0.052 0.052

0.50 5 20 3 0.054 0.016 0.060 0.056 0.056 0.055 0.056 0.056 0.056

0.50 5 20 4 0.060 0.020 0.068 0.064 0.065 0.064 0.064 0.065 0.064

0.50 5 20 5 0.069 0.025 0.080 0.076 0.077 0.076 0.077 0.077 0.076

0.50 5 20 6 0.081 0.033 0.097 0.091 0.092 0.093 0.093 0.093 0.092

0.50 5 20 7 0.096 0.042 0.119 0.112 0.114 0.114 0.115 0.114 0.113

0.50 5 20 8 0.116 0.056 0.146 0.138 0.139 0.141 0.142 0.141 0.140

0.50 5 20 9 0.139 0.072 0.178 0.171 0.169 0.173 0.172 0.174 0.172

0.50 5 20 10 0.167 0.094 0.215 0.207 0.205 0.212 0.210 0.212 0.210

0.50 5 20 11 0.198 0.120 0.260 0.252 0.249 0.258 0.254 0.257 0.254

0.50 5 20 12 0.236 0.152 0.312 0.302 0.299 0.308 0.304 0.308 0.304

0.50 5 20 13 0.277 0.191 0.368 0.356 0.354 0.363 0.362 0.363 0.359

0.50 5 20 14 0.325 0.237 0.430 0.415 0.415 0.421 0.422 0.423 0.418

0.50 5 20 15 0.376 0.288 0.491 0.477 0.476 0.483 0.483 0.486 0.480

0.50 5 20 16 0.428 0.344 0.555 0.542 0.540 0.545 0.547 0.549 0.543

0.50 5 20 17 0.480 0.403 0.617 0.605 0.602 0.606 0.611 0.612 0.606

0.50 5 20 18 0.533 0.466 0.677 0.668 0.663 0.667 0.671 0.673 0.666

0.50 5 20 19 0.589 0.534 0.734 0.722 0.719 0.724 0.729 0.729 0.723

0.50 5 20 20 0.641 0.599 0.783 0.774 0.770 0.776 0.779 0.781 0.775

0.50 5 20 21 0.694 0.661 0.827 0.820 0.817 0.823 0.824 0.827 0.821

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 5, I = 10, 20

46
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.50 5 40 1 0.049 0.014 0.050 0.049 0.049 0.051 0.050 0.050 0.050

0.50 5 40 2 0.052 0.015 0.054 0.053 0.053 0.055 0.053 0.053 0.053

0.50 5 40 3 0.060 0.019 0.064 0.062 0.063 0.064 0.062 0.063 0.063

0.50 5 40 4 0.074 0.026 0.082 0.080 0.080 0.083 0.080 0.081 0.081

0.50 5 40 5 0.093 0.037 0.110 0.107 0.107 0.108 0.108 0.108 0.108

0.50 5 40 6 0.120 0.055 0.148 0.144 0.145 0.147 0.145 0.146 0.145

0.50 5 40 7 0.156 0.082 0.198 0.193 0.194 0.196 0.195 0.196 0.195

0.50 5 40 8 0.201 0.119 0.260 0.256 0.255 0.258 0.258 0.258 0.257

0.50 5 40 9 0.254 0.168 0.335 0.330 0.329 0.330 0.333 0.333 0.331

0.50 5 40 10 0.316 0.231 0.420 0.415 0.413 0.414 0.416 0.417 0.414

0.50 5 40 11 0.388 0.307 0.512 0.505 0.504 0.504 0.510 0.506 0.503

0.50 5 40 12 0.464 0.393 0.603 0.598 0.596 0.594 0.601 0.597 0.594

0.50 5 40 13 0.543 0.486 0.690 0.684 0.684 0.678 0.687 0.684 0.681

0.50 5 40 14 0.621 0.581 0.766 0.760 0.760 0.758 0.763 0.762 0.759

0.50 5 40 15 0.696 0.671 0.831 0.827 0.827 0.826 0.830 0.829 0.826

0.50 5 40 16 0.762 0.751 0.884 0.881 0.879 0.880 0.883 0.883 0.881

0.50 5 40 17 0.818 0.820 0.924 0.922 0.921 0.921 0.923 0.923 0.922

0.50 5 40 18 0.869 0.875 0.953 0.952 0.951 0.952 0.952 0.953 0.951

0.50 5 40 19 0.908 0.918 0.972 0.971 0.971 0.972 0.971 0.972 0.971

0.50 5 40 20 0.939 0.950 0.983 0.983 0.983 0.984 0.983 0.984 0.984

0.50 5 40 21 0.960 0.970 0.991 0.991 0.991 0.992 0.991 0.992 0.991

Table 1: Power simulation (see Section 2 for details), ρ = 0.5, J = 5, I = 40

47
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 2 3 1 0.050 0.019 0.152 0.056 0.049 0.050 0.050 0.050 0.050

0.70 2 3 2 0.051 0.019 0.154 0.056 0.049 0.050 0.050 0.051 0.050

0.70 2 3 3 0.051 0.020 0.156 0.056 0.050 0.051 0.052 0.052 0.051

0.70 2 3 4 0.053 0.021 0.160 0.059 0.051 0.053 0.053 0.055 0.053

0.70 2 3 5 0.055 0.022 0.165 0.064 0.053 0.056 0.056 0.058 0.055

0.70 2 3 6 0.058 0.024 0.171 0.068 0.056 0.059 0.060 0.063 0.058

0.70 2 3 7 0.061 0.026 0.178 0.073 0.058 0.063 0.065 0.069 0.062

0.70 2 3 8 0.064 0.029 0.187 0.080 0.062 0.067 0.070 0.076 0.067

0.70 2 3 9 0.068 0.032 0.198 0.087 0.066 0.073 0.077 0.084 0.072

0.70 2 3 10 0.073 0.035 0.210 0.096 0.071 0.079 0.085 0.093 0.077

0.70 2 3 11 0.078 0.040 0.223 0.105 0.076 0.086 0.093 0.103 0.083

0.70 2 3 12 0.084 0.045 0.236 0.116 0.081 0.093 0.102 0.114 0.090

0.70 2 3 13 0.089 0.050 0.249 0.128 0.087 0.101 0.112 0.127 0.098

0.70 2 3 14 0.096 0.055 0.265 0.140 0.092 0.110 0.123 0.140 0.106

0.70 2 3 15 0.103 0.062 0.281 0.154 0.100 0.120 0.134 0.154 0.114

0.70 2 3 16 0.110 0.068 0.299 0.168 0.107 0.130 0.146 0.170 0.124

0.70 2 3 17 0.119 0.076 0.317 0.184 0.115 0.141 0.159 0.187 0.133

0.70 2 3 18 0.128 0.084 0.336 0.199 0.123 0.153 0.173 0.204 0.143

0.70 2 3 19 0.137 0.093 0.355 0.216 0.132 0.165 0.187 0.223 0.154

0.70 2 3 20 0.148 0.102 0.375 0.234 0.140 0.178 0.203 0.242 0.165

0.70 2 3 21 0.158 0.113 0.395 0.253 0.149 0.191 0.218 0.263 0.177

0.70 2 5 1 0.050 0.014 0.090 0.057 0.050 0.048 0.050 0.050 0.050

0.70 2 5 2 0.050 0.015 0.090 0.057 0.051 0.050 0.052 0.051 0.051

0.70 2 5 3 0.053 0.016 0.094 0.061 0.054 0.053 0.055 0.054 0.053

0.70 2 5 4 0.056 0.017 0.100 0.066 0.058 0.059 0.061 0.060 0.058

0.70 2 5 5 0.060 0.020 0.108 0.074 0.063 0.065 0.069 0.068 0.064

0.70 2 5 6 0.066 0.023 0.120 0.084 0.070 0.073 0.078 0.078 0.072

0.70 2 5 7 0.074 0.027 0.136 0.097 0.079 0.084 0.091 0.090 0.082

0.70 2 5 8 0.080 0.033 0.153 0.111 0.088 0.098 0.103 0.105 0.093

0.70 2 5 9 0.090 0.039 0.172 0.127 0.099 0.113 0.120 0.123 0.107

0.70 2 5 10 0.099 0.047 0.194 0.148 0.113 0.130 0.139 0.143 0.122

0.70 2 5 11 0.110 0.056 0.219 0.169 0.129 0.149 0.161 0.165 0.139

0.70 2 5 12 0.123 0.066 0.245 0.192 0.146 0.169 0.185 0.190 0.158

0.70 2 5 13 0.136 0.078 0.272 0.217 0.165 0.192 0.210 0.217 0.178

0.70 2 5 14 0.151 0.091 0.302 0.244 0.185 0.217 0.239 0.246 0.200

0.70 2 5 15 0.167 0.106 0.334 0.273 0.206 0.245 0.267 0.277 0.224

0.70 2 5 16 0.186 0.124 0.367 0.304 0.229 0.274 0.299 0.310 0.250

0.70 2 5 17 0.205 0.143 0.400 0.336 0.254 0.305 0.331 0.345 0.277

0.70 2 5 18 0.223 0.162 0.435 0.370 0.279 0.336 0.366 0.381 0.305

0.70 2 5 19 0.244 0.183 0.471 0.405 0.305 0.370 0.401 0.419 0.334

0.70 2 5 20 0.266 0.206 0.506 0.440 0.330 0.401 0.437 0.457 0.364

0.70 2 5 21 0.287 0.231 0.542 0.476 0.359 0.436 0.473 0.495 0.395

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 2, I = 3, 5

48
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 2 10 1 0.050 0.009 0.063 0.053 0.051 0.051 0.050 0.050 0.050

0.70 2 10 2 0.051 0.010 0.066 0.055 0.052 0.053 0.052 0.053 0.052

0.70 2 10 3 0.056 0.012 0.076 0.063 0.057 0.060 0.058 0.060 0.059

0.70 2 10 4 0.062 0.016 0.088 0.074 0.068 0.070 0.071 0.073 0.071

0.70 2 10 5 0.071 0.021 0.107 0.090 0.083 0.087 0.088 0.091 0.087

0.70 2 10 6 0.083 0.029 0.131 0.114 0.102 0.109 0.112 0.115 0.108

0.70 2 10 7 0.098 0.039 0.161 0.141 0.127 0.136 0.140 0.144 0.134

0.70 2 10 8 0.116 0.052 0.197 0.177 0.156 0.169 0.174 0.180 0.166

0.70 2 10 9 0.136 0.068 0.241 0.216 0.190 0.208 0.216 0.220 0.202

0.70 2 10 10 0.159 0.087 0.288 0.262 0.230 0.253 0.263 0.266 0.243

0.70 2 10 11 0.184 0.110 0.339 0.309 0.272 0.301 0.313 0.317 0.288

0.70 2 10 12 0.213 0.140 0.393 0.360 0.320 0.351 0.366 0.371 0.338

0.70 2 10 13 0.245 0.172 0.449 0.416 0.369 0.407 0.421 0.428 0.390

0.70 2 10 14 0.278 0.211 0.506 0.474 0.420 0.463 0.481 0.487 0.444

0.70 2 10 15 0.314 0.253 0.563 0.531 0.473 0.518 0.539 0.545 0.499

0.70 2 10 16 0.353 0.299 0.618 0.589 0.527 0.576 0.595 0.603 0.554

0.70 2 10 17 0.393 0.349 0.673 0.646 0.581 0.631 0.651 0.659 0.608

0.70 2 10 18 0.434 0.402 0.724 0.698 0.634 0.683 0.705 0.711 0.660

0.70 2 10 19 0.477 0.456 0.771 0.746 0.683 0.731 0.754 0.760 0.709

0.70 2 10 20 0.519 0.509 0.813 0.790 0.731 0.775 0.799 0.803 0.754

0.70 2 10 21 0.561 0.564 0.849 0.829 0.775 0.815 0.837 0.841 0.795

0.70 2 20 1 0.050 0.008 0.056 0.050 0.049 0.050 0.050 0.050 0.050

0.70 2 20 2 0.053 0.009 0.061 0.057 0.056 0.055 0.055 0.055 0.055

0.70 2 20 3 0.061 0.013 0.080 0.074 0.071 0.072 0.072 0.072 0.071

0.70 2 20 4 0.073 0.021 0.107 0.100 0.096 0.097 0.101 0.099 0.097

0.70 2 20 5 0.094 0.035 0.149 0.141 0.134 0.135 0.140 0.139 0.134

0.70 2 20 6 0.118 0.054 0.203 0.191 0.182 0.185 0.193 0.190 0.183

0.70 2 20 7 0.151 0.079 0.268 0.255 0.243 0.247 0.257 0.254 0.243

0.70 2 20 8 0.190 0.115 0.341 0.328 0.311 0.320 0.331 0.327 0.313

0.70 2 20 9 0.233 0.160 0.422 0.407 0.386 0.400 0.409 0.408 0.391

0.70 2 20 10 0.283 0.216 0.508 0.490 0.467 0.482 0.494 0.493 0.473

0.70 2 20 11 0.339 0.284 0.591 0.575 0.550 0.568 0.580 0.578 0.556

0.70 2 20 12 0.396 0.357 0.670 0.658 0.629 0.650 0.663 0.660 0.637

0.70 2 20 13 0.457 0.435 0.746 0.731 0.707 0.724 0.740 0.735 0.713

0.70 2 20 14 0.518 0.518 0.809 0.796 0.775 0.791 0.803 0.801 0.779

0.70 2 20 15 0.577 0.601 0.863 0.853 0.833 0.845 0.858 0.856 0.836

0.70 2 20 16 0.634 0.679 0.904 0.897 0.880 0.890 0.900 0.899 0.883

0.70 2 20 17 0.690 0.750 0.936 0.930 0.917 0.926 0.934 0.932 0.919

0.70 2 20 18 0.742 0.811 0.960 0.956 0.944 0.951 0.958 0.956 0.946

0.70 2 20 19 0.791 0.863 0.975 0.972 0.965 0.969 0.975 0.973 0.965

0.70 2 20 20 0.834 0.904 0.985 0.983 0.978 0.981 0.985 0.984 0.979

0.70 2 20 21 0.870 0.935 0.991 0.991 0.987 0.989 0.991 0.991 0.987

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 2, I = 10, 20

49
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 2 40 1 0.049 0.006 0.053 0.049 0.051 0.049 0.050 0.050 0.050

0.70 2 40 2 0.055 0.009 0.064 0.060 0.060 0.062 0.060 0.061 0.061

0.70 2 40 3 0.072 0.018 0.097 0.094 0.092 0.096 0.094 0.095 0.094

0.70 2 40 4 0.102 0.035 0.155 0.150 0.146 0.150 0.151 0.153 0.150

0.70 2 40 5 0.143 0.067 0.239 0.232 0.226 0.232 0.233 0.236 0.231

0.70 2 40 6 0.198 0.116 0.343 0.336 0.329 0.337 0.337 0.340 0.333

0.70 2 40 7 0.265 0.186 0.461 0.455 0.443 0.455 0.453 0.458 0.449

0.70 2 40 8 0.339 0.279 0.584 0.575 0.561 0.575 0.580 0.581 0.571

0.70 2 40 9 0.423 0.389 0.698 0.689 0.680 0.688 0.695 0.696 0.685

0.70 2 40 10 0.510 0.509 0.796 0.790 0.780 0.786 0.794 0.795 0.785

0.70 2 40 11 0.597 0.627 0.870 0.865 0.857 0.866 0.868 0.871 0.863

0.70 2 40 12 0.677 0.735 0.924 0.921 0.914 0.921 0.924 0.925 0.919

0.70 2 40 13 0.751 0.824 0.961 0.959 0.954 0.958 0.960 0.960 0.956

0.70 2 40 14 0.817 0.890 0.982 0.980 0.978 0.978 0.981 0.980 0.978

0.70 2 40 15 0.869 0.938 0.992 0.991 0.990 0.990 0.991 0.991 0.990

0.70 2 40 16 0.910 0.968 0.996 0.996 0.995 0.996 0.996 0.996 0.996

0.70 2 40 17 0.941 0.984 0.999 0.999 0.998 0.999 0.998 0.999 0.998

0.70 2 40 18 0.963 0.993 1.000 1.000 0.999 1.000 1.000 1.000 0.999

0.70 2 40 19 0.977 0.997 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.70 2 40 20 0.987 0.999 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.70 2 40 21 0.993 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 2, I = 40

50
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 5 3 1 0.048 0.006 0.139 0.051 0.050 0.051 0.049 0.050 0.050

0.70 5 3 2 0.048 0.006 0.139 0.052 0.050 0.052 0.049 0.050 0.050

0.70 5 3 3 0.049 0.006 0.139 0.052 0.051 0.052 0.050 0.051 0.051

0.70 5 3 4 0.050 0.006 0.141 0.054 0.052 0.053 0.051 0.052 0.052

0.70 5 3 5 0.050 0.006 0.144 0.055 0.053 0.055 0.053 0.054 0.054

0.70 5 3 6 0.051 0.007 0.146 0.057 0.054 0.057 0.055 0.056 0.056

0.70 5 3 7 0.053 0.007 0.149 0.060 0.056 0.059 0.058 0.059 0.058

0.70 5 3 8 0.055 0.007 0.154 0.063 0.059 0.062 0.061 0.062 0.061

0.70 5 3 9 0.057 0.008 0.158 0.067 0.062 0.066 0.065 0.066 0.065

0.70 5 3 10 0.058 0.009 0.164 0.070 0.065 0.070 0.069 0.070 0.069

0.70 5 3 11 0.061 0.010 0.170 0.075 0.069 0.074 0.074 0.075 0.073

0.70 5 3 12 0.063 0.011 0.177 0.081 0.073 0.079 0.080 0.081 0.078

0.70 5 3 13 0.066 0.011 0.184 0.086 0.078 0.085 0.085 0.087 0.084

0.70 5 3 14 0.069 0.013 0.193 0.092 0.083 0.091 0.092 0.094 0.090

0.70 5 3 15 0.073 0.014 0.202 0.099 0.089 0.097 0.098 0.102 0.097

0.70 5 3 16 0.077 0.015 0.211 0.107 0.095 0.105 0.107 0.111 0.105

0.70 5 3 17 0.081 0.017 0.221 0.116 0.101 0.114 0.116 0.120 0.113

0.70 5 3 18 0.086 0.019 0.233 0.124 0.109 0.122 0.127 0.130 0.122

0.70 5 3 19 0.091 0.021 0.244 0.134 0.116 0.131 0.137 0.141 0.132

0.70 5 3 20 0.097 0.024 0.256 0.144 0.124 0.141 0.147 0.153 0.143

0.70 5 3 21 0.103 0.026 0.269 0.155 0.133 0.151 0.159 0.166 0.154

0.70 5 5 1 0.047 0.003 0.085 0.050 0.049 0.051 0.049 0.050 0.050

0.70 5 5 2 0.047 0.003 0.086 0.051 0.050 0.051 0.050 0.050 0.050

0.70 5 5 3 0.048 0.003 0.087 0.052 0.051 0.053 0.052 0.052 0.052

0.70 5 5 4 0.050 0.004 0.091 0.054 0.053 0.056 0.054 0.054 0.054

0.70 5 5 5 0.052 0.004 0.095 0.057 0.056 0.059 0.057 0.058 0.057

0.70 5 5 6 0.054 0.004 0.101 0.062 0.060 0.063 0.062 0.062 0.062

0.70 5 5 7 0.057 0.005 0.107 0.067 0.064 0.068 0.068 0.068 0.067

0.70 5 5 8 0.060 0.005 0.115 0.074 0.071 0.075 0.075 0.075 0.074

0.70 5 5 9 0.064 0.006 0.126 0.081 0.078 0.083 0.084 0.083 0.082

0.70 5 5 10 0.069 0.007 0.137 0.090 0.085 0.092 0.093 0.093 0.091

0.70 5 5 11 0.075 0.009 0.148 0.100 0.095 0.104 0.103 0.104 0.101

0.70 5 5 12 0.080 0.010 0.162 0.111 0.105 0.116 0.115 0.117 0.113

0.70 5 5 13 0.087 0.012 0.178 0.125 0.117 0.129 0.130 0.132 0.127

0.70 5 5 14 0.095 0.014 0.196 0.139 0.131 0.144 0.146 0.148 0.142

0.70 5 5 15 0.103 0.018 0.217 0.156 0.147 0.162 0.164 0.166 0.159

0.70 5 5 16 0.111 0.021 0.238 0.173 0.163 0.181 0.184 0.186 0.178

0.70 5 5 17 0.122 0.025 0.261 0.193 0.181 0.201 0.206 0.208 0.198

0.70 5 5 18 0.132 0.030 0.284 0.215 0.200 0.223 0.231 0.231 0.221

0.70 5 5 19 0.143 0.035 0.310 0.240 0.221 0.246 0.255 0.257 0.244

0.70 5 5 20 0.157 0.041 0.338 0.266 0.244 0.272 0.283 0.284 0.270

0.70 5 5 21 0.170 0.048 0.367 0.294 0.269 0.298 0.312 0.313 0.297

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 5, I = 3, 5

51
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 5 10 1 0.050 0.002 0.066 0.051 0.051 0.049 0.052 0.050 0.050

0.70 5 10 2 0.050 0.002 0.067 0.052 0.052 0.049 0.053 0.051 0.051

0.70 5 10 3 0.052 0.002 0.070 0.054 0.055 0.053 0.056 0.054 0.054

0.70 5 10 4 0.056 0.002 0.076 0.060 0.060 0.059 0.061 0.060 0.060

0.70 5 10 5 0.059 0.003 0.086 0.068 0.067 0.067 0.069 0.068 0.068

0.70 5 10 6 0.065 0.004 0.098 0.078 0.078 0.078 0.079 0.079 0.078

0.70 5 10 7 0.072 0.005 0.114 0.092 0.091 0.092 0.094 0.093 0.092

0.70 5 10 8 0.082 0.007 0.131 0.108 0.107 0.110 0.111 0.111 0.109

0.70 5 10 9 0.091 0.009 0.152 0.128 0.127 0.131 0.131 0.132 0.130

0.70 5 10 10 0.102 0.012 0.179 0.152 0.148 0.156 0.156 0.157 0.154

0.70 5 10 11 0.116 0.016 0.210 0.181 0.176 0.186 0.185 0.186 0.183

0.70 5 10 12 0.132 0.021 0.244 0.214 0.207 0.218 0.218 0.220 0.215

0.70 5 10 13 0.148 0.028 0.281 0.251 0.241 0.256 0.257 0.258 0.252

0.70 5 10 14 0.168 0.037 0.324 0.290 0.281 0.296 0.297 0.300 0.293

0.70 5 10 15 0.189 0.048 0.370 0.335 0.323 0.342 0.344 0.345 0.337

0.70 5 10 16 0.212 0.062 0.419 0.381 0.367 0.389 0.393 0.394 0.385

0.70 5 10 17 0.238 0.080 0.469 0.430 0.417 0.441 0.445 0.445 0.435

0.70 5 10 18 0.266 0.101 0.519 0.479 0.467 0.493 0.496 0.498 0.486

0.70 5 10 19 0.295 0.126 0.571 0.534 0.517 0.544 0.549 0.551 0.538

0.70 5 10 20 0.327 0.154 0.620 0.586 0.570 0.595 0.601 0.603 0.590

0.70 5 10 21 0.359 0.186 0.671 0.636 0.618 0.646 0.654 0.654 0.641

0.70 5 20 1 0.049 0.001 0.057 0.049 0.050 0.049 0.050 0.050 0.050

0.70 5 20 2 0.051 0.001 0.059 0.052 0.052 0.052 0.052 0.052 0.052

0.70 5 20 3 0.055 0.002 0.066 0.059 0.059 0.058 0.061 0.059 0.059

0.70 5 20 4 0.062 0.003 0.079 0.072 0.071 0.070 0.072 0.072 0.072

0.70 5 20 5 0.069 0.004 0.099 0.088 0.089 0.087 0.090 0.090 0.090

0.70 5 20 6 0.080 0.006 0.124 0.112 0.113 0.112 0.115 0.116 0.115

0.70 5 20 7 0.095 0.009 0.158 0.146 0.145 0.145 0.147 0.149 0.148

0.70 5 20 8 0.113 0.015 0.202 0.187 0.185 0.188 0.190 0.192 0.190

0.70 5 20 9 0.137 0.023 0.256 0.237 0.236 0.239 0.242 0.243 0.240

0.70 5 20 10 0.165 0.035 0.314 0.296 0.293 0.298 0.301 0.303 0.300

0.70 5 20 11 0.197 0.052 0.383 0.364 0.360 0.365 0.369 0.371 0.367

0.70 5 20 12 0.233 0.074 0.457 0.435 0.431 0.438 0.442 0.444 0.439

0.70 5 20 13 0.277 0.105 0.534 0.510 0.508 0.514 0.520 0.521 0.515

0.70 5 20 14 0.323 0.146 0.608 0.588 0.583 0.591 0.597 0.597 0.591

0.70 5 20 15 0.371 0.194 0.681 0.664 0.658 0.663 0.671 0.671 0.665

0.70 5 20 16 0.423 0.254 0.750 0.731 0.727 0.733 0.742 0.740 0.734

0.70 5 20 17 0.479 0.320 0.809 0.793 0.788 0.792 0.801 0.800 0.795

0.70 5 20 18 0.533 0.393 0.858 0.846 0.842 0.846 0.852 0.852 0.847

0.70 5 20 19 0.588 0.471 0.898 0.891 0.884 0.891 0.895 0.894 0.889

0.70 5 20 20 0.640 0.551 0.928 0.923 0.918 0.924 0.926 0.926 0.923

0.70 5 20 21 0.692 0.628 0.952 0.947 0.944 0.949 0.951 0.951 0.948

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 5, I = 10, 20

52
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.70 5 40 1 0.048 0.001 0.053 0.049 0.050 0.049 0.050 0.050 0.050

0.70 5 40 2 0.051 0.001 0.057 0.054 0.054 0.053 0.054 0.055 0.055

0.70 5 40 3 0.058 0.002 0.073 0.069 0.070 0.069 0.069 0.070 0.070

0.70 5 40 4 0.072 0.004 0.101 0.095 0.097 0.097 0.097 0.097 0.097

0.70 5 40 5 0.091 0.008 0.142 0.136 0.137 0.139 0.138 0.139 0.139

0.70 5 40 6 0.117 0.015 0.203 0.195 0.197 0.199 0.197 0.199 0.198

0.70 5 40 7 0.153 0.029 0.284 0.274 0.274 0.277 0.278 0.278 0.276

0.70 5 40 8 0.198 0.050 0.378 0.369 0.368 0.371 0.372 0.373 0.371

0.70 5 40 9 0.251 0.085 0.489 0.477 0.475 0.477 0.481 0.479 0.476

0.70 5 40 10 0.316 0.137 0.597 0.588 0.585 0.588 0.593 0.589 0.586

0.70 5 40 11 0.385 0.209 0.701 0.691 0.690 0.691 0.695 0.694 0.691

0.70 5 40 12 0.462 0.302 0.788 0.782 0.779 0.781 0.786 0.786 0.783

0.70 5 40 13 0.541 0.411 0.864 0.858 0.857 0.854 0.861 0.860 0.858

0.70 5 40 14 0.618 0.526 0.917 0.913 0.912 0.910 0.916 0.915 0.913

0.70 5 40 15 0.691 0.638 0.954 0.951 0.950 0.948 0.953 0.952 0.951

0.70 5 40 16 0.760 0.739 0.976 0.974 0.974 0.973 0.975 0.975 0.974

0.70 5 40 17 0.819 0.826 0.988 0.987 0.987 0.986 0.988 0.988 0.987

0.70 5 40 18 0.870 0.892 0.995 0.995 0.994 0.994 0.995 0.995 0.994

0.70 5 40 19 0.907 0.938 0.998 0.998 0.998 0.997 0.998 0.998 0.998

0.70 5 40 20 0.937 0.966 0.999 0.999 0.999 0.999 0.999 0.999 0.999

0.70 5 40 21 0.959 0.983 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.7, J = 5, I = 40

53
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 2 3 1 0.050 0.003 0.255 0.086 0.046 0.050 0.050 0.050 0.050

0.90 2 3 2 0.050 0.003 0.257 0.087 0.046 0.051 0.052 0.051 0.051

0.90 2 3 3 0.051 0.004 0.261 0.092 0.048 0.054 0.055 0.056 0.054

0.90 2 3 4 0.052 0.004 0.269 0.099 0.050 0.058 0.061 0.063 0.058

0.90 2 3 5 0.054 0.005 0.280 0.109 0.053 0.064 0.068 0.073 0.065

0.90 2 3 6 0.056 0.006 0.291 0.122 0.059 0.073 0.079 0.085 0.073

0.90 2 3 7 0.060 0.007 0.305 0.137 0.064 0.084 0.092 0.101 0.082

0.90 2 3 8 0.063 0.009 0.323 0.156 0.070 0.097 0.108 0.120 0.094

0.90 2 3 9 0.066 0.011 0.343 0.176 0.077 0.111 0.126 0.141 0.107

0.90 2 3 10 0.070 0.013 0.367 0.199 0.085 0.128 0.146 0.166 0.121

0.90 2 3 11 0.074 0.016 0.391 0.222 0.094 0.147 0.167 0.193 0.137

0.90 2 3 12 0.080 0.019 0.413 0.247 0.103 0.168 0.191 0.223 0.154

0.90 2 3 13 0.086 0.023 0.438 0.275 0.112 0.190 0.216 0.256 0.173

0.90 2 3 14 0.092 0.027 0.463 0.303 0.123 0.214 0.244 0.290 0.193

0.90 2 3 15 0.100 0.031 0.491 0.334 0.134 0.239 0.274 0.327 0.213

0.90 2 3 16 0.108 0.036 0.518 0.365 0.147 0.266 0.305 0.365 0.235

0.90 2 3 17 0.117 0.042 0.545 0.396 0.159 0.294 0.336 0.405 0.257

0.90 2 3 18 0.126 0.049 0.573 0.428 0.171 0.322 0.369 0.445 0.281

0.90 2 3 19 0.135 0.057 0.600 0.458 0.184 0.351 0.404 0.486 0.304

0.90 2 3 20 0.146 0.065 0.627 0.490 0.199 0.381 0.439 0.526 0.329

0.90 2 3 21 0.156 0.072 0.653 0.522 0.213 0.410 0.472 0.566 0.354

0.90 2 5 1 0.051 0.001 0.140 0.083 0.046 0.049 0.048 0.050 0.050

0.90 2 5 2 0.052 0.001 0.144 0.086 0.048 0.053 0.050 0.053 0.052

0.90 2 5 3 0.053 0.001 0.155 0.095 0.053 0.060 0.059 0.062 0.059

0.90 2 5 4 0.056 0.002 0.170 0.110 0.060 0.072 0.075 0.077 0.071

0.90 2 5 5 0.060 0.003 0.194 0.131 0.070 0.091 0.096 0.098 0.088

0.90 2 5 6 0.065 0.004 0.223 0.158 0.085 0.116 0.122 0.126 0.109

0.90 2 5 7 0.071 0.006 0.257 0.190 0.103 0.146 0.157 0.161 0.136

0.90 2 5 8 0.078 0.008 0.297 0.228 0.124 0.181 0.196 0.202 0.167

0.90 2 5 9 0.087 0.011 0.339 0.269 0.148 0.221 0.240 0.249 0.203

0.90 2 5 10 0.097 0.015 0.385 0.314 0.174 0.267 0.291 0.302 0.243

0.90 2 5 11 0.109 0.020 0.436 0.360 0.204 0.318 0.346 0.359 0.287

0.90 2 5 12 0.121 0.026 0.487 0.411 0.237 0.371 0.404 0.419 0.335

0.90 2 5 13 0.134 0.034 0.538 0.464 0.274 0.428 0.464 0.482 0.385

0.90 2 5 14 0.149 0.044 0.588 0.515 0.311 0.483 0.526 0.545 0.436

0.90 2 5 15 0.165 0.054 0.639 0.568 0.349 0.542 0.588 0.606 0.489

0.90 2 5 16 0.183 0.069 0.685 0.620 0.390 0.599 0.645 0.665 0.541

0.90 2 5 17 0.203 0.085 0.728 0.669 0.432 0.653 0.699 0.720 0.592

0.90 2 5 18 0.222 0.103 0.768 0.714 0.474 0.704 0.750 0.770 0.642

0.90 2 5 19 0.242 0.125 0.806 0.755 0.515 0.749 0.795 0.815 0.689

0.90 2 5 20 0.263 0.147 0.838 0.793 0.557 0.790 0.835 0.854 0.732

0.90 2 5 21 0.285 0.173 0.866 0.826 0.596 0.827 0.870 0.887 0.773

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 2, I = 3, 5

54
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 2 10 1 0.052 0.000 0.087 0.067 0.048 0.050 0.049 0.050 0.050

0.90 2 10 2 0.052 0.000 0.095 0.074 0.053 0.056 0.057 0.057 0.056

0.90 2 10 3 0.056 0.001 0.119 0.095 0.069 0.076 0.077 0.077 0.075

0.90 2 10 4 0.062 0.001 0.157 0.131 0.095 0.110 0.114 0.113 0.106

0.90 2 10 5 0.071 0.002 0.209 0.179 0.131 0.158 0.163 0.163 0.151

0.90 2 10 6 0.083 0.004 0.275 0.242 0.182 0.220 0.228 0.229 0.210

0.90 2 10 7 0.099 0.006 0.349 0.313 0.240 0.296 0.305 0.308 0.280

0.90 2 10 8 0.116 0.011 0.435 0.397 0.308 0.383 0.396 0.397 0.362

0.90 2 10 9 0.136 0.018 0.523 0.483 0.382 0.473 0.490 0.493 0.449

0.90 2 10 10 0.159 0.028 0.610 0.571 0.465 0.564 0.587 0.589 0.540

0.90 2 10 11 0.186 0.046 0.689 0.654 0.545 0.652 0.677 0.680 0.628

0.90 2 10 12 0.215 0.067 0.762 0.729 0.624 0.736 0.760 0.761 0.710

0.90 2 10 13 0.247 0.096 0.823 0.796 0.697 0.805 0.827 0.829 0.782

0.90 2 10 14 0.279 0.134 0.874 0.852 0.761 0.861 0.882 0.883 0.842

0.90 2 10 15 0.314 0.181 0.914 0.897 0.818 0.907 0.923 0.924 0.890

0.90 2 10 16 0.352 0.235 0.944 0.929 0.865 0.939 0.953 0.953 0.927

0.90 2 10 17 0.392 0.298 0.964 0.955 0.903 0.960 0.972 0.972 0.953

0.90 2 10 18 0.434 0.365 0.978 0.972 0.932 0.976 0.984 0.985 0.971

0.90 2 10 19 0.477 0.437 0.988 0.983 0.956 0.986 0.991 0.992 0.983

0.90 2 10 20 0.519 0.512 0.993 0.990 0.970 0.992 0.996 0.996 0.991

0.90 2 10 21 0.560 0.588 0.996 0.994 0.981 0.996 0.997 0.998 0.995

0.90 2 20 1 0.050 0.000 0.065 0.057 0.050 0.050 0.051 0.050 0.050

0.90 2 20 2 0.052 0.000 0.082 0.073 0.063 0.064 0.066 0.064 0.064

0.90 2 20 3 0.060 0.000 0.130 0.118 0.103 0.108 0.112 0.109 0.106

0.90 2 20 4 0.073 0.001 0.209 0.194 0.169 0.185 0.188 0.186 0.179

0.90 2 20 5 0.092 0.002 0.314 0.295 0.262 0.288 0.294 0.293 0.281

0.90 2 20 6 0.119 0.007 0.440 0.419 0.377 0.415 0.424 0.424 0.406

0.90 2 20 7 0.151 0.017 0.577 0.554 0.509 0.553 0.564 0.564 0.542

0.90 2 20 8 0.191 0.035 0.703 0.683 0.637 0.684 0.698 0.697 0.673

0.90 2 20 9 0.235 0.066 0.807 0.793 0.749 0.797 0.805 0.807 0.786

0.90 2 20 10 0.286 0.118 0.884 0.875 0.840 0.881 0.887 0.889 0.872

0.90 2 20 11 0.342 0.191 0.938 0.930 0.906 0.938 0.943 0.942 0.930

0.90 2 20 12 0.399 0.287 0.968 0.965 0.950 0.970 0.973 0.973 0.966

0.90 2 20 13 0.459 0.398 0.986 0.984 0.974 0.987 0.988 0.989 0.985

0.90 2 20 14 0.522 0.519 0.995 0.994 0.988 0.995 0.996 0.996 0.994

0.90 2 20 15 0.582 0.641 0.998 0.998 0.996 0.998 0.999 0.999 0.998

0.90 2 20 16 0.641 0.747 0.999 0.999 0.999 1.000 1.000 1.000 0.999

0.90 2 20 17 0.695 0.834 1.000 1.000 0.999 1.000 1.000 1.000 1.000

0.90 2 20 18 0.747 0.898 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 20 19 0.795 0.942 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 20 20 0.836 0.969 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 20 21 0.871 0.985 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 2, I = 10, 20

55
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 2 40 1 0.049 0.000 0.058 0.054 0.051 0.050 0.052 0.050 0.050

0.90 2 40 2 0.054 0.000 0.087 0.084 0.078 0.078 0.080 0.080 0.079

0.90 2 40 3 0.071 0.000 0.183 0.176 0.165 0.171 0.172 0.173 0.170

0.90 2 40 4 0.100 0.002 0.341 0.332 0.314 0.329 0.331 0.330 0.323

0.90 2 40 5 0.141 0.010 0.532 0.522 0.501 0.522 0.527 0.527 0.517

0.90 2 40 6 0.195 0.031 0.716 0.706 0.688 0.712 0.714 0.717 0.706

0.90 2 40 7 0.261 0.084 0.858 0.851 0.834 0.855 0.857 0.860 0.851

0.90 2 40 8 0.337 0.183 0.940 0.937 0.928 0.943 0.943 0.944 0.938

0.90 2 40 9 0.419 0.333 0.980 0.979 0.974 0.980 0.981 0.982 0.979

0.90 2 40 10 0.510 0.516 0.994 0.994 0.992 0.995 0.995 0.995 0.994

0.90 2 40 11 0.597 0.692 0.999 0.999 0.998 0.999 0.999 0.999 0.999

0.90 2 40 12 0.680 0.835 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 13 0.756 0.926 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 14 0.822 0.972 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 15 0.873 0.991 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 16 0.911 0.998 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 17 0.941 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 18 0.963 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 19 0.977 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 20 0.987 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 2 40 21 0.993 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 2, I = 40

56
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 5 3 1 0.048 0.000 0.343 0.111 0.046 0.052 0.050 0.050 0.050

0.90 5 3 2 0.048 0.000 0.342 0.113 0.046 0.052 0.051 0.051 0.051

0.90 5 3 3 0.048 0.000 0.344 0.116 0.048 0.054 0.053 0.053 0.052

0.90 5 3 4 0.049 0.000 0.348 0.119 0.049 0.057 0.056 0.056 0.055

0.90 5 3 5 0.049 0.000 0.355 0.124 0.051 0.062 0.060 0.061 0.060

0.90 5 3 6 0.051 0.000 0.361 0.131 0.054 0.067 0.066 0.067 0.065

0.90 5 3 7 0.053 0.000 0.370 0.139 0.058 0.073 0.074 0.074 0.072

0.90 5 3 8 0.054 0.000 0.379 0.150 0.063 0.083 0.082 0.084 0.081

0.90 5 3 9 0.055 0.000 0.391 0.161 0.067 0.093 0.094 0.095 0.091

0.90 5 3 10 0.057 0.000 0.404 0.174 0.073 0.103 0.106 0.109 0.103

0.90 5 3 11 0.060 0.000 0.419 0.190 0.079 0.117 0.121 0.124 0.117

0.90 5 3 12 0.062 0.001 0.434 0.208 0.086 0.133 0.137 0.142 0.132

0.90 5 3 13 0.065 0.001 0.450 0.226 0.094 0.150 0.157 0.161 0.150

0.90 5 3 14 0.068 0.001 0.467 0.246 0.104 0.171 0.177 0.184 0.170

0.90 5 3 15 0.072 0.001 0.485 0.267 0.114 0.193 0.201 0.208 0.192

0.90 5 3 16 0.076 0.001 0.502 0.289 0.125 0.215 0.228 0.235 0.216

0.90 5 3 17 0.080 0.001 0.521 0.314 0.137 0.242 0.256 0.265 0.242

0.90 5 3 18 0.084 0.002 0.540 0.340 0.151 0.271 0.286 0.296 0.270

0.90 5 3 19 0.089 0.002 0.561 0.367 0.165 0.301 0.316 0.330 0.300

0.90 5 3 20 0.095 0.002 0.581 0.394 0.181 0.333 0.349 0.365 0.332

0.90 5 3 21 0.100 0.003 0.603 0.423 0.197 0.365 0.383 0.402 0.365

0.90 5 5 1 0.047 0.000 0.193 0.095 0.049 0.051 0.050 0.050 0.050

0.90 5 5 2 0.047 0.000 0.195 0.097 0.050 0.052 0.052 0.051 0.051

0.90 5 5 3 0.048 0.000 0.202 0.101 0.052 0.056 0.057 0.055 0.055

0.90 5 5 4 0.049 0.000 0.212 0.110 0.057 0.062 0.064 0.062 0.061

0.90 5 5 5 0.050 0.000 0.226 0.121 0.063 0.071 0.074 0.072 0.071

0.90 5 5 6 0.053 0.000 0.244 0.134 0.070 0.084 0.087 0.085 0.083

0.90 5 5 7 0.056 0.000 0.265 0.152 0.081 0.100 0.104 0.102 0.099

0.90 5 5 8 0.060 0.000 0.290 0.173 0.095 0.120 0.124 0.124 0.120

0.90 5 5 9 0.063 0.000 0.318 0.197 0.111 0.144 0.149 0.150 0.144

0.90 5 5 10 0.068 0.000 0.349 0.226 0.129 0.174 0.180 0.181 0.173

0.90 5 5 11 0.073 0.000 0.385 0.258 0.149 0.208 0.216 0.216 0.207

0.90 5 5 12 0.080 0.000 0.423 0.294 0.174 0.247 0.254 0.257 0.245

0.90 5 5 13 0.087 0.000 0.463 0.336 0.202 0.290 0.297 0.303 0.288

0.90 5 5 14 0.095 0.001 0.503 0.377 0.233 0.338 0.345 0.353 0.335

0.90 5 5 15 0.103 0.001 0.546 0.421 0.268 0.390 0.398 0.406 0.386

0.90 5 5 16 0.111 0.001 0.588 0.468 0.305 0.443 0.453 0.462 0.439

0.90 5 5 17 0.120 0.002 0.629 0.516 0.345 0.499 0.511 0.519 0.494

0.90 5 5 18 0.131 0.002 0.671 0.563 0.385 0.555 0.568 0.576 0.549

0.90 5 5 19 0.143 0.004 0.711 0.610 0.431 0.611 0.624 0.632 0.604

0.90 5 5 20 0.155 0.005 0.749 0.657 0.474 0.663 0.677 0.686 0.658

0.90 5 5 21 0.170 0.006 0.784 0.699 0.520 0.714 0.727 0.736 0.708

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 5, I = 3, 5

57
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 5 10 1 0.049 0.000 0.104 0.070 0.048 0.047 0.050 0.050 0.050

0.90 5 10 2 0.049 0.000 0.109 0.074 0.051 0.051 0.052 0.053 0.053

0.90 5 10 3 0.052 0.000 0.121 0.084 0.058 0.059 0.061 0.062 0.062

0.90 5 10 4 0.054 0.000 0.141 0.101 0.071 0.076 0.076 0.078 0.077

0.90 5 10 5 0.059 0.000 0.171 0.126 0.090 0.100 0.101 0.102 0.101

0.90 5 10 6 0.064 0.000 0.210 0.161 0.116 0.132 0.133 0.136 0.134

0.90 5 10 7 0.071 0.000 0.261 0.203 0.150 0.175 0.176 0.181 0.177

0.90 5 10 8 0.078 0.000 0.320 0.256 0.195 0.231 0.234 0.237 0.232

0.90 5 10 9 0.088 0.000 0.388 0.318 0.251 0.299 0.302 0.305 0.297

0.90 5 10 10 0.101 0.000 0.459 0.391 0.316 0.375 0.379 0.381 0.372

0.90 5 10 11 0.114 0.000 0.536 0.466 0.388 0.457 0.463 0.465 0.454

0.90 5 10 12 0.129 0.001 0.612 0.544 0.462 0.543 0.551 0.552 0.540

0.90 5 10 13 0.148 0.001 0.682 0.621 0.539 0.627 0.636 0.637 0.624

0.90 5 10 14 0.168 0.002 0.750 0.696 0.613 0.705 0.715 0.717 0.704

0.90 5 10 15 0.190 0.004 0.808 0.761 0.685 0.778 0.785 0.787 0.775

0.90 5 10 16 0.212 0.006 0.859 0.820 0.752 0.837 0.844 0.847 0.836

0.90 5 10 17 0.237 0.009 0.900 0.869 0.813 0.886 0.892 0.895 0.886

0.90 5 10 18 0.265 0.015 0.931 0.910 0.862 0.924 0.929 0.931 0.924

0.90 5 10 19 0.295 0.024 0.955 0.939 0.900 0.952 0.955 0.956 0.951

0.90 5 10 20 0.327 0.037 0.971 0.960 0.932 0.970 0.973 0.974 0.970

0.90 5 10 21 0.363 0.055 0.982 0.974 0.954 0.983 0.985 0.985 0.983

0.90 5 20 1 0.051 0.000 0.074 0.057 0.049 0.048 0.049 0.050 0.050

0.90 5 20 2 0.052 0.000 0.080 0.063 0.054 0.055 0.055 0.056 0.056

0.90 5 20 3 0.055 0.000 0.104 0.084 0.072 0.077 0.074 0.076 0.076

0.90 5 20 4 0.061 0.000 0.146 0.122 0.106 0.116 0.112 0.113 0.113

0.90 5 20 5 0.070 0.000 0.209 0.178 0.160 0.175 0.169 0.172 0.170

0.90 5 20 6 0.082 0.000 0.292 0.257 0.233 0.256 0.252 0.254 0.251

0.90 5 20 7 0.097 0.000 0.394 0.357 0.327 0.362 0.356 0.359 0.355

0.90 5 20 8 0.116 0.000 0.511 0.473 0.436 0.481 0.475 0.480 0.474

0.90 5 20 9 0.140 0.000 0.631 0.594 0.558 0.604 0.601 0.606 0.599

0.90 5 20 10 0.168 0.001 0.741 0.708 0.677 0.721 0.721 0.723 0.717

0.90 5 20 11 0.200 0.002 0.831 0.806 0.780 0.818 0.819 0.821 0.816

0.90 5 20 12 0.238 0.005 0.899 0.882 0.861 0.890 0.894 0.894 0.890

0.90 5 20 13 0.279 0.011 0.947 0.934 0.920 0.941 0.943 0.943 0.940

0.90 5 20 14 0.325 0.023 0.974 0.967 0.959 0.971 0.972 0.973 0.971

0.90 5 20 15 0.374 0.047 0.988 0.985 0.980 0.988 0.989 0.988 0.987

0.90 5 20 16 0.427 0.084 0.995 0.994 0.992 0.995 0.996 0.995 0.995

0.90 5 20 17 0.480 0.140 0.999 0.998 0.997 0.998 0.999 0.998 0.998

0.90 5 20 18 0.535 0.218 1.000 0.999 0.999 1.000 1.000 0.999 0.999

0.90 5 20 19 0.590 0.318 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 20 20 0.642 0.435 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 20 21 0.695 0.555 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 5, I = 10, 20

58
PS alloc Theoretical

No PS alloc 1-way PS alloc ANCOVA power

ρ J I m 1-way No ρ ρˆ True ρ 2-way No PS alloc PS alloc 1-way 2-way

0.90 5 40 1 0.049 0.000 0.060 0.053 0.050 0.050 0.050 0.050 0.050

0.90 5 40 2 0.052 0.000 0.073 0.066 0.062 0.063 0.063 0.063 0.063

0.90 5 40 3 0.059 0.000 0.121 0.110 0.104 0.107 0.105 0.107 0.107

0.90 5 40 4 0.071 0.000 0.210 0.193 0.185 0.191 0.191 0.194 0.193

0.90 5 40 5 0.089 0.000 0.345 0.326 0.314 0.325 0.327 0.328 0.327

0.90 5 40 6 0.116 0.000 0.514 0.492 0.480 0.495 0.498 0.500 0.497

0.90 5 40 7 0.150 0.000 0.689 0.669 0.657 0.671 0.677 0.677 0.674

0.90 5 40 8 0.194 0.001 0.829 0.817 0.806 0.818 0.823 0.823 0.821

0.90 5 40 9 0.249 0.005 0.922 0.914 0.908 0.917 0.919 0.920 0.918

0.90 5 40 10 0.313 0.016 0.970 0.967 0.964 0.969 0.971 0.970 0.969

0.90 5 40 11 0.385 0.047 0.991 0.990 0.988 0.991 0.991 0.991 0.991

0.90 5 40 12 0.459 0.112 0.998 0.998 0.997 0.998 0.998 0.998 0.998

0.90 5 40 13 0.538 0.230 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 14 0.617 0.390 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 15 0.693 0.573 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 16 0.760 0.749 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 17 0.817 0.873 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 18 0.868 0.945 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 19 0.907 0.981 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 20 0.937 0.995 1.000 1.000 1.000 1.000 1.000 1.000 1.000

0.90 5 40 21 0.960 0.999 1.000 1.000 1.000 1.000 1.000 1.000 1.000

Table 1: Power simulation (see Section 2 for details), ρ = 0.9, J = 5, I = 40

59
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.949 0.949 0.866 0.945 0.952 0.859 0.957 0.767

5 0.949 0.950 0.906 0.942 0.950 0.907 0.950 0.865

2 10 0.949 0.953 0.930 0.948 0.951 0.930 0.947 0.914

20 0.951 0.950 0.942 0.948 0.950 0.940 0.948 0.933

40 0.947 0.944 0.940 0.944 0.947 0.943 0.947 0.938

80 0.953 0.954 0.951 0.953 0.953 0.951 0.952 0.950

3 0.952 0.950 0.896 0.944 0.953 0.895 0.956 0.808

5 0.951 0.951 0.922 0.948 0.953 0.924 0.952 0.881

3 10 0.953 0.952 0.942 0.951 0.954 0.943 0.953 0.925

20 0.954 0.954 0.947 0.953 0.953 0.948 0.953 0.939

40 0.945 0.944 0.941 0.944 0.945 0.942 0.944 0.938

80 0.950 0.951 0.949 0.950 0.950 0.948 0.949 0.947

3 0.949 0.950 0.918 0.948 0.952 0.919 0.951 0.842

5 0.952 0.951 0.935 0.950 0.950 0.935 0.950 0.897

5 10 0.952 0.952 0.945 0.951 0.953 0.946 0.952 0.930

20 0.949 0.948 0.945 0.947 0.949 0.945 0.948 0.937

40 0.949 0.948 0.946 0.947 0.949 0.947 0.949 0.943

80 0.954 0.953 0.953 0.953 0.954 0.954 0.955 0.952

3 0.947 0.948 0.924 0.945 0.947 0.924 0.946 0.852

5 0.949 0.950 0.940 0.948 0.949 0.939 0.949 0.907

0 7 10 0.955 0.954 0.948 0.953 0.955 0.950 0.955 0.934

20 0.951 0.950 0.948 0.950 0.951 0.949 0.951 0.943

40 0.953 0.954 0.952 0.953 0.953 0.952 0.953 0.949

80 0.949 0.949 0.948 0.948 0.948 0.948 0.948 0.946

3 0.956 0.956 0.940 0.955 0.956 0.940 0.955 0.870

5 0.948 0.948 0.938 0.947 0.947 0.938 0.947 0.903

9 10 0.952 0.951 0.948 0.951 0.952 0.949 0.952 0.932

20 0.955 0.955 0.953 0.955 0.955 0.953 0.955 0.946

40 0.950 0.950 0.949 0.950 0.949 0.949 0.949 0.945

80 0.951 0.951 0.950 0.951 0.951 0.950 0.951 0.948

3 0.950 0.950 0.936 0.949 0.949 0.937 0.948 0.868

5 0.951 0.952 0.945 0.951 0.951 0.946 0.951 0.911

11 10 0.952 0.952 0.950 0.952 0.952 0.949 0.952 0.937

20 0.951 0.951 0.950 0.951 0.951 0.950 0.951 0.945

40 0.949 0.948 0.947 0.948 0.949 0.948 0.949 0.946

80 0.946 0.946 0.946 0.946 0.946 0.946 0.946 0.945

3 0.949 0.948 0.941 0.948 0.947 0.940 0.947 0.878

5 0.947 0.948 0.944 0.948 0.948 0.944 0.948 0.911

20 10 0.953 0.952 0.951 0.952 0.953 0.951 0.953 0.935

20 0.948 0.948 0.947 0.948 0.948 0.947 0.948 0.942

40 0.949 0.949 0.949 0.949 0.949 0.949 0.949 0.946

80 0.951 0.951 0.951 0.951 0.951 0.951 0.951 0.950

Table 2: Actual coverage of nominal 95% confidence intervals

60
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.959 0.945 0.875 0.947 0.939 0.859 0.959 0.779

5 0.960 0.937 0.911 0.947 0.931 0.911 0.949 0.864

2 10 0.964 0.937 0.935 0.950 0.934 0.934 0.951 0.911

20 0.960 0.933 0.941 0.948 0.928 0.939 0.948 0.927

40 0.962 0.930 0.944 0.947 0.928 0.945 0.949 0.934

80 0.963 0.929 0.946 0.948 0.927 0.945 0.947 0.937

3 0.965 0.943 0.901 0.949 0.939 0.896 0.958 0.820

5 0.969 0.944 0.930 0.952 0.940 0.928 0.956 0.888

3 10 0.965 0.938 0.939 0.949 0.937 0.940 0.949 0.922

20 0.967 0.937 0.943 0.948 0.938 0.946 0.950 0.935

40 0.969 0.937 0.948 0.950 0.938 0.948 0.951 0.941

80 0.968 0.937 0.948 0.949 0.936 0.948 0.950 0.943

3 0.971 0.944 0.921 0.950 0.942 0.919 0.952 0.844

5 0.971 0.948 0.942 0.955 0.944 0.940 0.953 0.904

5 10 0.969 0.941 0.942 0.949 0.940 0.941 0.949 0.925

20 0.971 0.941 0.946 0.948 0.940 0.945 0.949 0.937

40 0.971 0.941 0.947 0.948 0.940 0.947 0.948 0.941

80 0.973 0.940 0.948 0.949 0.940 0.949 0.949 0.945

3 0.971 0.946 0.930 0.950 0.943 0.928 0.951 0.854

5 0.974 0.947 0.942 0.952 0.948 0.942 0.952 0.906

.50 7 10 0.972 0.945 0.945 0.950 0.944 0.946 0.949 0.930

20 0.974 0.944 0.948 0.950 0.945 0.948 0.950 0.940

40 0.973 0.945 0.949 0.950 0.945 0.949 0.950 0.946

80 0.975 0.944 0.951 0.951 0.945 0.950 0.951 0.948

3 0.974 0.950 0.939 0.953 0.949 0.937 0.955 0.866

5 0.975 0.948 0.946 0.952 0.947 0.944 0.951 0.913

9 10 0.974 0.945 0.946 0.949 0.945 0.946 0.950 0.932

20 0.971 0.944 0.946 0.948 0.943 0.946 0.948 0.939

40 0.973 0.946 0.948 0.949 0.945 0.948 0.949 0.945

80 0.975 0.947 0.951 0.951 0.947 0.950 0.951 0.948

3 0.976 0.949 0.941 0.952 0.948 0.939 0.952 0.872

5 0.976 0.948 0.944 0.951 0.948 0.946 0.951 0.911

11 10 0.975 0.948 0.949 0.952 0.947 0.948 0.951 0.933

20 0.972 0.944 0.946 0.947 0.943 0.946 0.947 0.939

40 0.976 0.949 0.951 0.952 0.948 0.952 0.952 0.948

80 0.976 0.950 0.953 0.954 0.950 0.953 0.954 0.951

3 0.977 0.947 0.942 0.949 0.946 0.941 0.948 0.879

5 0.979 0.955 0.952 0.957 0.954 0.953 0.955 0.921

20 10 0.977 0.950 0.951 0.951 0.950 0.951 0.952 0.936

20 0.976 0.947 0.948 0.949 0.947 0.947 0.948 0.943

40 0.976 0.948 0.950 0.950 0.949 0.951 0.951 0.947

80 0.975 0.951 0.952 0.952 0.950 0.953 0.953 0.950

Table 2 continued: Actual coverage of nominal 95% confidence intervals

61
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.965 0.938 0.885 0.953 0.928 0.864 0.962 0.785

5 0.967 0.932 0.916 0.950 0.924 0.911 0.954 0.863

2 10 0.971 0.928 0.939 0.954 0.923 0.936 0.951 0.911

20 0.969 0.920 0.943 0.950 0.918 0.941 0.948 0.924

40 0.968 0.915 0.945 0.948 0.914 0.944 0.948 0.931

80 0.969 0.916 0.947 0.949 0.916 0.947 0.949 0.936

3 0.971 0.944 0.910 0.954 0.941 0.908 0.960 0.829

5 0.975 0.941 0.935 0.956 0.935 0.932 0.956 0.890

3 10 0.975 0.933 0.941 0.951 0.925 0.938 0.951 0.918

20 0.978 0.931 0.948 0.953 0.931 0.949 0.954 0.936

40 0.974 0.923 0.945 0.948 0.925 0.946 0.950 0.937

80 0.973 0.931 0.950 0.951 0.931 0.950 0.951 0.944

3 0.977 0.942 0.925 0.953 0.939 0.923 0.955 0.848

5 0.979 0.941 0.940 0.953 0.941 0.941 0.955 0.904

5 10 0.979 0.937 0.943 0.950 0.936 0.944 0.950 0.925

20 0.979 0.937 0.949 0.952 0.938 0.948 0.951 0.939

40 0.979 0.937 0.949 0.950 0.937 0.949 0.950 0.944

80 0.980 0.937 0.948 0.949 0.936 0.948 0.949 0.945

3 0.978 0.944 0.932 0.952 0.942 0.931 0.952 0.861

5 0.983 0.942 0.941 0.953 0.943 0.943 0.953 0.910

.60 7 10 0.981 0.941 0.946 0.950 0.942 0.947 0.951 0.931

20 0.981 0.942 0.949 0.950 0.940 0.948 0.949 0.940

40 0.981 0.941 0.950 0.951 0.942 0.950 0.951 0.946

80 0.980 0.940 0.948 0.949 0.939 0.948 0.949 0.946

3 0.982 0.946 0.936 0.951 0.944 0.935 0.952 0.870

5 0.983 0.942 0.942 0.949 0.943 0.943 0.950 0.912

9 10 0.984 0.944 0.948 0.951 0.943 0.948 0.951 0.934

20 0.982 0.943 0.949 0.951 0.943 0.949 0.950 0.942

40 0.982 0.947 0.953 0.954 0.947 0.953 0.954 0.949

80 0.983 0.945 0.952 0.952 0.945 0.951 0.952 0.949

3 0.980 0.943 0.936 0.948 0.939 0.934 0.946 0.869

5 0.983 0.943 0.942 0.950 0.942 0.942 0.951 0.908

11 10 0.984 0.946 0.950 0.953 0.946 0.950 0.953 0.933

20 0.982 0.943 0.948 0.949 0.944 0.948 0.949 0.942

40 0.986 0.942 0.947 0.948 0.944 0.949 0.949 0.945

80 0.983 0.946 0.952 0.952 0.946 0.951 0.951 0.949

3 0.986 0.951 0.948 0.955 0.951 0.948 0.954 0.888

5 0.986 0.950 0.949 0.953 0.949 0.949 0.952 0.918

20 10 0.986 0.950 0.952 0.954 0.951 0.953 0.955 0.940

20 0.987 0.950 0.952 0.953 0.950 0.952 0.953 0.946

40 0.986 0.948 0.950 0.951 0.949 0.952 0.952 0.948

80 0.985 0.945 0.949 0.949 0.946 0.949 0.950 0.948

Table 2 continued: Actual coverage of nominal 95% confidence intervals

62
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.970 0.940 0.890 0.956 0.921 0.872 0.965 0.795

5 0.974 0.922 0.923 0.959 0.909 0.918 0.958 0.867

2 10 0.975 0.911 0.939 0.954 0.900 0.935 0.954 0.906

20 0.978 0.904 0.948 0.956 0.898 0.947 0.954 0.924

40 0.977 0.892 0.949 0.952 0.891 0.949 0.953 0.929

80 0.974 0.893 0.947 0.949 0.891 0.948 0.950 0.931

3 0.978 0.935 0.914 0.958 0.922 0.906 0.963 0.831

5 0.983 0.931 0.936 0.958 0.922 0.934 0.957 0.887

3 10 0.980 0.920 0.943 0.951 0.915 0.941 0.950 0.917

20 0.982 0.913 0.945 0.949 0.910 0.944 0.949 0.928

40 0.985 0.914 0.951 0.953 0.915 0.950 0.952 0.938

80 0.983 0.914 0.950 0.951 0.916 0.950 0.951 0.941

3 0.984 0.941 0.931 0.957 0.933 0.927 0.958 0.855

5 0.987 0.933 0.940 0.953 0.928 0.938 0.952 0.900

5 10 0.987 0.928 0.945 0.951 0.926 0.943 0.951 0.924

20 0.988 0.925 0.948 0.952 0.925 0.946 0.949 0.935

40 0.988 0.929 0.952 0.953 0.930 0.951 0.953 0.945

80 0.988 0.926 0.949 0.950 0.927 0.948 0.949 0.944

3 0.989 0.938 0.932 0.956 0.936 0.934 0.956 0.865

5 0.990 0.942 0.946 0.954 0.939 0.945 0.956 0.913

.70 7 10 0.990 0.936 0.946 0.951 0.933 0.946 0.949 0.929

20 0.990 0.932 0.949 0.951 0.930 0.948 0.950 0.938

40 0.991 0.933 0.949 0.950 0.934 0.948 0.949 0.942

80 0.990 0.935 0.952 0.952 0.936 0.951 0.952 0.948

3 0.989 0.946 0.943 0.957 0.940 0.937 0.954 0.873

5 0.993 0.945 0.949 0.957 0.943 0.948 0.957 0.916

9 10 0.989 0.936 0.944 0.947 0.934 0.944 0.947 0.929

20 0.991 0.937 0.949 0.950 0.937 0.950 0.952 0.941

40 0.989 0.938 0.950 0.951 0.936 0.950 0.950 0.945

80 0.992 0.941 0.952 0.952 0.941 0.952 0.952 0.949

3 0.989 0.942 0.938 0.953 0.942 0.940 0.953 0.874

5 0.992 0.942 0.945 0.953 0.941 0.946 0.953 0.913

11 10 0.992 0.944 0.950 0.952 0.942 0.950 0.953 0.935

20 0.991 0.940 0.950 0.951 0.941 0.950 0.952 0.942

40 0.992 0.941 0.950 0.950 0.940 0.951 0.952 0.946

80 0.991 0.939 0.949 0.949 0.938 0.948 0.948 0.946

3 0.992 0.947 0.945 0.953 0.945 0.945 0.951 0.881

5 0.991 0.943 0.946 0.949 0.943 0.945 0.949 0.912

20 10 0.991 0.943 0.946 0.949 0.941 0.946 0.948 0.933

20 0.991 0.940 0.945 0.946 0.940 0.945 0.945 0.938

40 0.993 0.944 0.950 0.951 0.944 0.949 0.949 0.946

80 0.993 0.943 0.949 0.949 0.944 0.949 0.949 0.947

Table 2 continued: Actual coverage of nominal 95% confidence intervals

63
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.976 0.931 0.898 0.962 0.901 0.873 0.963 0.802

5 0.978 0.908 0.929 0.961 0.871 0.918 0.959 0.866

2 10 0.983 0.881 0.943 0.959 0.864 0.941 0.956 0.906

20 0.983 0.862 0.947 0.956 0.853 0.947 0.954 0.917

40 0.981 0.848 0.947 0.951 0.846 0.946 0.950 0.920

80 0.982 0.850 0.949 0.951 0.847 0.950 0.952 0.929

3 0.986 0.929 0.924 0.966 0.904 0.915 0.969 0.838

5 0.987 0.913 0.937 0.959 0.894 0.934 0.958 0.888

3 10 0.989 0.894 0.945 0.954 0.882 0.943 0.953 0.916

20 0.989 0.884 0.946 0.952 0.877 0.947 0.952 0.928

40 0.991 0.878 0.946 0.949 0.876 0.947 0.950 0.929

80 0.989 0.875 0.946 0.947 0.875 0.946 0.947 0.931

3 0.990 0.933 0.933 0.959 0.914 0.927 0.959 0.861

5 0.994 0.924 0.946 0.960 0.914 0.944 0.958 0.909

5 10 0.993 0.914 0.946 0.953 0.907 0.947 0.953 0.926

20 0.994 0.911 0.949 0.953 0.910 0.950 0.953 0.938

40 0.994 0.903 0.946 0.947 0.904 0.945 0.947 0.937

80 0.994 0.911 0.949 0.949 0.911 0.950 0.950 0.943

3 0.994 0.937 0.940 0.958 0.923 0.938 0.957 0.873

5 0.995 0.928 0.945 0.953 0.923 0.945 0.955 0.910

.80 7 10 0.997 0.929 0.950 0.953 0.923 0.950 0.953 0.932

20 0.997 0.921 0.950 0.952 0.921 0.951 0.953 0.942

40 0.996 0.925 0.956 0.957 0.925 0.955 0.956 0.949

80 0.996 0.918 0.950 0.951 0.919 0.950 0.951 0.945

3 0.996 0.938 0.942 0.957 0.930 0.941 0.956 0.876

5 0.997 0.932 0.946 0.954 0.928 0.944 0.951 0.914

9 10 0.998 0.931 0.951 0.953 0.926 0.949 0.953 0.931

20 0.997 0.930 0.950 0.952 0.929 0.951 0.953 0.943

40 0.997 0.930 0.951 0.952 0.931 0.951 0.952 0.947

80 0.997 0.924 0.949 0.949 0.924 0.949 0.950 0.945

3 0.996 0.940 0.943 0.954 0.932 0.941 0.954 0.882

5 0.998 0.935 0.948 0.954 0.930 0.944 0.951 0.913

11 10 0.998 0.933 0.949 0.952 0.933 0.947 0.950 0.935

20 0.998 0.931 0.948 0.949 0.930 0.948 0.949 0.940

40 0.997 0.931 0.950 0.950 0.931 0.949 0.950 0.946

80 0.998 0.930 0.951 0.951 0.930 0.950 0.950 0.947

3 0.996 0.942 0.944 0.951 0.941 0.947 0.954 0.888

5 0.997 0.944 0.949 0.951 0.937 0.945 0.949 0.918

20 10 0.998 0.942 0.950 0.951 0.941 0.949 0.951 0.937

20 0.998 0.939 0.949 0.949 0.937 0.949 0.950 0.940

40 0.999 0.942 0.952 0.952 0.941 0.950 0.951 0.947

80 0.998 0.941 0.952 0.952 0.942 0.952 0.952 0.950

Table 2 continued: Actual coverage of nominal 95% confidence intervals

64
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.982 0.920 0.904 0.968 0.828 0.875 0.966 0.797

5 0.984 0.867 0.933 0.961 0.782 0.918 0.958 0.860

2 10 0.986 0.799 0.940 0.954 0.747 0.933 0.950 0.885

20 0.987 0.754 0.946 0.952 0.729 0.944 0.951 0.898

40 0.989 0.749 0.950 0.953 0.737 0.948 0.951 0.911

80 0.990 0.743 0.954 0.956 0.736 0.954 0.955 0.918

3 0.991 0.908 0.926 0.967 0.839 0.914 0.967 0.843

5 0.993 0.872 0.939 0.959 0.814 0.933 0.958 0.884

3 10 0.995 0.835 0.948 0.958 0.802 0.948 0.958 0.914

20 0.997 0.812 0.950 0.955 0.798 0.948 0.954 0.921

40 0.995 0.800 0.949 0.951 0.792 0.950 0.952 0.925

80 0.997 0.804 0.952 0.952 0.800 0.953 0.954 0.931

3 0.998 0.921 0.944 0.967 0.870 0.939 0.966 0.880

5 0.998 0.898 0.948 0.960 0.859 0.945 0.957 0.906

5 10 0.998 0.871 0.950 0.955 0.850 0.948 0.954 0.924

20 0.999 0.854 0.947 0.951 0.846 0.947 0.950 0.929

40 0.999 0.853 0.951 0.952 0.846 0.950 0.952 0.939

80 0.999 0.857 0.951 0.951 0.855 0.951 0.951 0.940

3 0.998 0.924 0.943 0.960 0.883 0.939 0.957 0.881

5 0.999 0.907 0.945 0.956 0.882 0.945 0.954 0.911

.90 7 10 0.999 0.888 0.947 0.951 0.878 0.947 0.952 0.928

20 1.000 0.888 0.954 0.956 0.883 0.952 0.954 0.942

40 1.000 0.884 0.949 0.950 0.882 0.949 0.950 0.941

80 1.000 0.879 0.951 0.951 0.877 0.951 0.951 0.944

3 0.999 0.934 0.948 0.961 0.905 0.946 0.959 0.894

5 0.999 0.920 0.952 0.958 0.896 0.950 0.958 0.918

9 10 1.000 0.906 0.951 0.955 0.898 0.953 0.955 0.937

20 1.000 0.901 0.953 0.954 0.893 0.952 0.953 0.943

40 1.000 0.900 0.955 0.956 0.898 0.953 0.953 0.947

80 1.000 0.895 0.950 0.951 0.895 0.950 0.950 0.944

3 0.999 0.935 0.949 0.961 0.908 0.945 0.957 0.892

5 1.000 0.921 0.947 0.953 0.900 0.947 0.953 0.917

11 10 1.000 0.914 0.953 0.956 0.902 0.951 0.954 0.934

20 1.000 0.912 0.952 0.953 0.909 0.953 0.954 0.944

40 1.000 0.910 0.954 0.955 0.910 0.955 0.956 0.950

80 1.000 0.904 0.950 0.950 0.903 0.950 0.951 0.944

3 0.999 0.940 0.947 0.954 0.925 0.947 0.956 0.894

5 1.000 0.936 0.952 0.956 0.924 0.946 0.949 0.921

20 10 1.000 0.932 0.952 0.955 0.925 0.948 0.950 0.935

20 1.000 0.927 0.954 0.955 0.923 0.950 0.951 0.943

40 1.000 0.923 0.949 0.949 0.925 0.948 0.949 0.945

80 1.000 0.928 0.951 0.951 0.925 0.951 0.951 0.948

Table 2 continued: Actual coverage of nominal 95% confidence intervals

65
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.985 0.907 0.911 0.970 0.736 0.873 0.966 0.798

5 0.987 0.821 0.929 0.961 0.648 0.914 0.955 0.852

2 10 0.990 0.715 0.943 0.957 0.614 0.937 0.952 0.883

20 0.992 0.644 0.947 0.954 0.595 0.945 0.952 0.897

40 0.992 0.626 0.951 0.953 0.604 0.949 0.953 0.901

80 0.992 0.606 0.948 0.949 0.594 0.948 0.949 0.902

3 0.995 0.897 0.932 0.973 0.748 0.915 0.967 0.853

5 0.996 0.838 0.941 0.962 0.699 0.933 0.956 0.887

3 10 0.997 0.761 0.950 0.961 0.687 0.946 0.956 0.908

20 0.998 0.706 0.954 0.958 0.668 0.951 0.956 0.918

40 0.998 0.689 0.949 0.951 0.674 0.946 0.949 0.916

80 0.999 0.676 0.948 0.949 0.668 0.948 0.949 0.919

3 0.998 0.902 0.941 0.966 0.781 0.933 0.964 0.883

5 0.999 0.866 0.948 0.961 0.774 0.949 0.960 0.913

5 10 0.999 0.814 0.950 0.956 0.761 0.948 0.953 0.921

20 1.000 0.776 0.950 0.953 0.752 0.947 0.950 0.927

40 1.000 0.776 0.953 0.954 0.763 0.954 0.955 0.937

80 1.000 0.756 0.950 0.951 0.751 0.948 0.949 0.932

3 0.999 0.915 0.948 0.968 0.826 0.949 0.966 0.897

5 1.000 0.881 0.950 0.960 0.809 0.947 0.957 0.914

.95 7 10 1.000 0.848 0.953 0.958 0.807 0.951 0.955 0.931

20 1.000 0.827 0.951 0.953 0.804 0.951 0.954 0.934

40 1.000 0.812 0.948 0.949 0.807 0.949 0.949 0.936

80 1.000 0.804 0.950 0.950 0.800 0.950 0.951 0.938

3 1.000 0.918 0.949 0.962 0.844 0.948 0.964 0.899

5 1.000 0.896 0.953 0.960 0.847 0.950 0.958 0.922

9 10 1.000 0.870 0.955 0.958 0.836 0.953 0.957 0.936

20 1.000 0.853 0.952 0.953 0.832 0.950 0.951 0.941

40 1.000 0.838 0.948 0.949 0.830 0.948 0.949 0.940

80 1.000 0.837 0.952 0.953 0.832 0.952 0.953 0.943

3 1.000 0.926 0.950 0.962 0.862 0.953 0.963 0.908

5 1.000 0.912 0.954 0.960 0.853 0.950 0.957 0.924

11 10 1.000 0.877 0.950 0.953 0.849 0.951 0.953 0.934

20 1.000 0.868 0.954 0.955 0.855 0.951 0.952 0.942

40 1.000 0.857 0.947 0.947 0.851 0.947 0.948 0.940

80 1.000 0.850 0.949 0.949 0.849 0.951 0.951 0.946

3 1.000 0.938 0.952 0.958 0.895 0.949 0.956 0.903

5 1.000 0.927 0.953 0.957 0.897 0.952 0.956 0.926

20 10 1.000 0.914 0.954 0.955 0.899 0.953 0.955 0.941

20 1.000 0.902 0.950 0.951 0.897 0.950 0.951 0.943

40 1.000 0.897 0.949 0.949 0.897 0.948 0.948 0.943

80 1.000 0.901 0.951 0.951 0.899 0.952 0.952 0.949

Table 2 continued: Actual coverage of nominal 95% confidence intervals

66
Incorrect Corrected Corrected

ANOVA ANOVA Incorrect ANCOVA

ρ J I 1-way 2-way (Uses z) (Uses t) ANCOVA (Uses z) (Uses t) MLE

3 0.989 0.890 0.915 0.974 0.433 0.869 0.964 0.799

5 0.991 0.753 0.932 0.964 0.336 0.914 0.955 0.839

2 10 0.992 0.555 0.941 0.955 0.306 0.933 0.950 0.864

20 0.992 0.433 0.948 0.955 0.308 0.947 0.954 0.885

40 0.994 0.367 0.950 0.955 0.305 0.948 0.953 0.890

80 0.994 0.336 0.952 0.954 0.309 0.951 0.953 0.895

3 0.997 0.885 0.940 0.977 0.439 0.918 0.969 0.858

5 0.998 0.765 0.948 0.967 0.392 0.933 0.959 0.879

3 10 0.999 0.610 0.952 0.962 0.376 0.945 0.955 0.899

20 0.999 0.487 0.949 0.954 0.372 0.948 0.952 0.906

40 1.000 0.428 0.949 0.952 0.363 0.948 0.951 0.904

80 0.999 0.392 0.950 0.950 0.365 0.949 0.950 0.909

3 1.000 0.890 0.948 0.972 0.491 0.940 0.966 0.894

5 1.000 0.804 0.949 0.963 0.465 0.948 0.960 0.911

5 10 1.000 0.689 0.952 0.958 0.461 0.948 0.954 0.918

20 1.000 0.573 0.947 0.950 0.448 0.945 0.947 0.918

40 1.000 0.511 0.951 0.953 0.452 0.953 0.955 0.930

80 1.000 0.475 0.949 0.949 0.440 0.948 0.949 0.927

3 1.000 0.895 0.948 0.966 0.537 0.949 0.967 0.908

5 1.000 0.843 0.955 0.963 0.519 0.952 0.961 0.924

.99 7 10 1.000 0.740 0.956 0.959 0.518 0.952 0.954 0.930

20 1.000 0.634 0.954 0.956 0.514 0.952 0.953 0.933

40 1.000 0.577 0.953 0.954 0.518 0.953 0.954 0.938

80 1.000 0.543 0.953 0.954 0.515 0.953 0.954 0.939

3 1.000 0.911 0.951 0.965 0.589 0.955 0.968 0.919

5 1.000 0.859 0.956 0.963 0.569 0.952 0.959 0.922

9 10 1.000 0.770 0.954 0.957 0.564 0.954 0.957 0.934

20 1.000 0.680 0.955 0.957 0.560 0.953 0.955 0.938

40 1.000 0.626 0.951 0.952 0.561 0.950 0.951 0.937

80 1.000 0.594 0.951 0.951 0.563 0.952 0.952 0.940

3 1.000 0.919 0.954 0.965 0.612 0.954 0.966 0.919

5 1.000 0.874 0.953 0.959 0.607 0.953 0.960 0.929

11 10 1.000 0.795 0.957 0.959 0.594 0.951 0.954 0.932

20 1.000 0.713 0.954 0.955 0.595 0.952 0.953 0.940

40 1.000 0.656 0.951 0.951 0.599 0.951 0.951 0.941

80 1.000 0.629 0.952 0.952 0.603 0.951 0.951 0.942

3 1.000 0.930 0.949 0.955 0.716 0.957 0.963 0.931

5 1.000 0.905 0.951 0.954 0.717 0.955 0.958 0.934

20 10 1.000 0.862 0.954 0.957 0.708 0.953 0.955 0.939

20 1.000 0.802 0.951 0.951 0.710 0.949 0.950 0.940

40 1.000 0.764 0.953 0.954 0.715 0.954 0.954 0.946

80 1.000 0.725 0.951 0.951 0.700 0.949 0.949 0.944

Table 2 continued: Actual coverage of nominal 95% confidence intervals

67
Corrected ANOVA Corrected ANCOVA

ρ J Uses z Uses t Uses z Uses t

2 18 3 17 3

3 10 3 9 3

5 6 3 6 3

.5 7 5 3 5 3

9 4 3 4 3

11 3 3 4 3

20 3 3 3 3

2 14 3 15 4

3 8 3 9 3

5 6 3 6 3

.6 7 5 3 5 3

9 5 3 5 3

11 4 3 4 3

20 3 3 3 3

2 10 3 11 5

3 7 3 8 4

5 6 3 7 3

.7 7 4 3 4 3

9 3 3 4 3

11 4 3 3 3

20 3 3 3 3

2 8 8 10 5

3 6 5 7 5

5 4 3 5 3

.8 7 3 3 4 3

9 3 3 3 3

11 3 3 3 3

20 3 3 3 3

Table 3: I’s needed to yield actual coverages that lie between 0.94 and 0.96 for nominal 95%

confidence intervals

68
Corrected ANOVA Corrected ANCOVA

ρ J Uses z Uses t Uses z Uses t

2 8 5 13 4

3 6 6 7 5

5 3 5 4 4

.9 7 3 3 4 3

9 3 4 3 3

11 3 4 3 3

20 3 3 3 3

2 9 7 12 4

3 5 9 7 5

5 3 6 4 5

.95 7 3 6 3 5

9 3 6 3 5

11 3 5 3 4

20 3 3 3 3

2 9 7 13 4

3 3 10 7 5

5 3 7 3 5

.99 7 3 9 3 5

9 3 8 3 6

11 3 6 3 5

20 3 3 3 4

Table 3 continued: I’s needed to yield actual coverages that lie between 0.94 and 0.96 for

nominal 95% confidence intervals

69
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.051 0.089 0.051 0.050

5 0.051 0.065 0.051 0.051

3 10 0.049 0.055 0.049 0.050

20 0.051 0.054 0.051 0.051

40 0.050 0.051 0.050 0.050

3 0.050 0.072 0.050 0.050

5 0.050 0.060 0.050 0.050

5 10 0.050 0.054 0.050 0.049

20 0.049 0.051 0.049 0.050

40 0.050 0.051 0.050 0.050

3 0.049 0.066 0.049 0.049

5 0.050 0.058 0.050 0.050

0.00 7 10 0.050 0.053 0.050 0.049

20 0.050 0.051 0.050 0.050

40 0.050 0.050 0.050 0.049

3 0.050 0.064 0.050 0.050

5 0.050 0.057 0.050 0.051

9 10 0.051 0.054 0.051 0.051

20 0.051 0.053 0.051 0.051

40 0.051 0.051 0.051 0.050

3 0.050 0.061 0.050 0.050

5 0.050 0.056 0.050 0.050

11 10 0.050 0.053 0.050 0.051

20 0.050 0.051 0.050 0.049

40 0.050 0.050 0.050 0.050

3 0.050 0.057 0.050 0.050

5 0.051 0.055 0.051 0.051

20 10 0.049 0.051 0.049 0.049

20 0.050 0.051 0.050 0.050

40 0.050 0.051 0.050 0.050

Table 4: Tukey test size simulation (see Section 5.3 for details), ρ = 0.0

70
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.028 0.110 0.050 0.051

5 0.023 0.068 0.049 0.048

3 10 0.022 0.059 0.051 0.050

20 0.020 0.054 0.051 0.050

40 0.019 0.051 0.049 0.050

3 0.023 0.093 0.049 0.051

5 0.018 0.065 0.049 0.050

5 10 0.016 0.058 0.050 0.052

20 0.015 0.053 0.050 0.050

40 0.014 0.050 0.049 0.049

3 0.018 0.080 0.047 0.050

5 0.015 0.062 0.047 0.050

0.50 7 10 0.013 0.055 0.048 0.050

20 0.013 0.053 0.050 0.050

40 0.012 0.051 0.049 0.050

3 0.015 0.075 0.046 0.050

5 0.013 0.060 0.047 0.050

9 10 0.011 0.054 0.047 0.050

20 0.011 0.052 0.048 0.050

40 0.011 0.052 0.050 0.051

3 0.014 0.071 0.046 0.050

5 0.011 0.059 0.047 0.050

11 10 0.010 0.053 0.047 0.049

20 0.010 0.052 0.049 0.050

40 0.009 0.051 0.049 0.050

3 0.009 0.060 0.044 0.050

5 0.007 0.057 0.046 0.051

20 10 0.007 0.051 0.045 0.048

20 0.006 0.051 0.048 0.050

40 0.006 0.051 0.049 0.050

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.50

71
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.021 0.125 0.051 0.050

5 0.015 0.077 0.050 0.050

3 10 0.012 0.060 0.050 0.049

20 0.011 0.054 0.049 0.049

40 0.010 0.053 0.050 0.051

3 0.013 0.108 0.049 0.049

5 0.010 0.072 0.049 0.049

5 10 0.007 0.060 0.049 0.050

20 0.006 0.054 0.050 0.050

40 0.006 0.052 0.049 0.050

3 0.009 0.097 0.047 0.049

5 0.007 0.070 0.048 0.050

0.60 7 10 0.005 0.057 0.047 0.049

20 0.004 0.053 0.047 0.050

40 0.005 0.052 0.050 0.051

3 0.008 0.090 0.047 0.050

5 0.005 0.067 0.046 0.049

9 10 0.004 0.058 0.048 0.051

20 0.004 0.054 0.050 0.051

40 0.003 0.051 0.049 0.049

3 0.006 0.084 0.046 0.050

5 0.004 0.066 0.046 0.049

11 10 0.003 0.056 0.047 0.049

20 0.004 0.053 0.048 0.050

40 0.003 0.052 0.049 0.051

3 0.003 0.073 0.045 0.051

5 0.002 0.063 0.045 0.050

20 10 0.001 0.057 0.046 0.050

20 0.002 0.053 0.048 0.050

40 0.002 0.052 0.049 0.050

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.60

72
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.012 0.147 0.053 0.050

5 0.007 0.085 0.052 0.049

3 10 0.005 0.065 0.052 0.050

20 0.004 0.056 0.050 0.051

40 0.003 0.051 0.049 0.048

3 0.006 0.136 0.051 0.049

5 0.004 0.084 0.051 0.050

5 10 0.002 0.065 0.051 0.050

20 0.002 0.057 0.051 0.052

40 0.001 0.052 0.049 0.049

3 0.004 0.126 0.050 0.050

5 0.002 0.083 0.050 0.049

0.70 7 10 0.001 0.064 0.050 0.050

20 0.001 0.055 0.049 0.049

40 0.001 0.052 0.048 0.049

3 0.003 0.122 0.050 0.050

5 0.001 0.083 0.050 0.051

9 10 0.001 0.062 0.047 0.049

20 0.001 0.056 0.048 0.049

40 0.001 0.052 0.048 0.049

3 0.002 0.117 0.050 0.049

5 0.001 0.081 0.049 0.049

11 10 0.001 0.062 0.048 0.049

20 0.000 0.056 0.049 0.050

40 0.000 0.053 0.049 0.050

3 0.000 0.107 0.051 0.052

5 0.000 0.081 0.049 0.052

20 10 0.000 0.064 0.048 0.050

20 0.000 0.055 0.048 0.049

40 0.000 0.054 0.049 0.051

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.70

73
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.005 0.191 0.060 0.049

5 0.002 0.107 0.061 0.050

3 10 0.001 0.073 0.056 0.050

20 0.001 0.060 0.052 0.050

40 0.000 0.055 0.051 0.051

3 0.002 0.193 0.063 0.049

5 0.001 0.112 0.058 0.049

5 10 0.000 0.075 0.054 0.051

20 0.000 0.061 0.052 0.051

40 0.000 0.054 0.050 0.050

3 0.001 0.193 0.061 0.049

5 0.000 0.114 0.060 0.050

0.80 7 10 0.000 0.076 0.053 0.050

20 0.000 0.060 0.050 0.049

40 0.000 0.055 0.049 0.050

3 0.000 0.194 0.064 0.049

5 0.000 0.117 0.059 0.051

9 10 0.000 0.076 0.053 0.051

20 0.000 0.061 0.050 0.049

40 0.000 0.055 0.050 0.050

3 0.000 0.192 0.064 0.050

5 0.000 0.119 0.059 0.051

11 10 0.000 0.076 0.052 0.049

20 0.000 0.063 0.050 0.050

40 0.000 0.055 0.049 0.050

3 0.000 0.196 0.073 0.052

5 0.000 0.127 0.060 0.050

20 10 0.000 0.080 0.053 0.050

20 0.000 0.063 0.050 0.050

40 0.000 0.057 0.050 0.051

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.80

74
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.001 0.288 0.095 0.047

5 0.000 0.161 0.087 0.048

3 10 0.000 0.094 0.067 0.049

20 0.000 0.069 0.058 0.050

40 0.000 0.058 0.053 0.049

3 0.000 0.329 0.112 0.047

5 0.000 0.187 0.095 0.048

5 10 0.000 0.104 0.069 0.049

20 0.000 0.074 0.059 0.051

40 0.000 0.060 0.053 0.050

3 0.000 0.360 0.129 0.048

5 0.000 0.209 0.102 0.050

0.90 7 10 0.000 0.112 0.071 0.050

20 0.000 0.075 0.059 0.051

40 0.000 0.060 0.053 0.050

3 0.000 0.393 0.145 0.050

5 0.000 0.226 0.105 0.049

9 10 0.000 0.118 0.073 0.051

20 0.000 0.076 0.059 0.050

40 0.000 0.062 0.053 0.050

3 0.000 0.410 0.157 0.048

5 0.000 0.243 0.114 0.050

11 10 0.000 0.121 0.073 0.050

20 0.000 0.078 0.059 0.050

40 0.000 0.062 0.053 0.050

3 0.000 0.481 0.209 0.052

5 0.000 0.297 0.132 0.052

20 10 0.000 0.139 0.079 0.051

20 0.000 0.083 0.060 0.050

40 0.000 0.063 0.052 0.050

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.90

75
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.000 0.420 0.191 0.045

5 0.000 0.259 0.158 0.046

3 10 0.000 0.140 0.102 0.048

20 0.000 0.087 0.071 0.049

40 0.000 0.065 0.059 0.050

3 0.000 0.520 0.263 0.045

5 0.000 0.329 0.193 0.046

5 10 0.000 0.166 0.113 0.048

20 0.000 0.096 0.075 0.050

40 0.000 0.067 0.059 0.048

3 0.000 0.585 0.323 0.045

5 0.000 0.382 0.222 0.046

0.95 7 10 0.000 0.185 0.122 0.048

20 0.000 0.102 0.077 0.049

40 0.000 0.070 0.059 0.049

3 0.000 0.646 0.379 0.046

5 0.000 0.430 0.249 0.047

9 10 0.000 0.203 0.128 0.049

20 0.000 0.106 0.080 0.050

40 0.000 0.072 0.061 0.050

3 0.000 0.685 0.424 0.046

5 0.000 0.468 0.274 0.047

11 10 0.000 0.217 0.136 0.049

20 0.000 0.112 0.081 0.049

40 0.000 0.072 0.061 0.050

3 0.000 0.816 0.585 0.053

5 0.000 0.603 0.358 0.053

20 10 0.000 0.268 0.161 0.051

20 0.000 0.124 0.087 0.050

40 0.000 0.079 0.064 0.051

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.95

76
1-way ˆs

ρ J I No ρ ρˆ ρtrue 2-way ˆs

3 0.000 0.751 0.654 0.037

5 0.000 0.631 0.556 0.037

3 10 0.000 0.415 0.363 0.041

20 0.000 0.230 0.200 0.045

40 0.000 0.126 0.114 0.049

3 0.000 0.889 0.839 0.038

5 0.000 0.801 0.729 0.038

5 10 0.000 0.554 0.481 0.040

20 0.000 0.291 0.250 0.044

40 0.000 0.147 0.132 0.049

3 0.000 0.946 0.922 0.038

5 0.000 0.888 0.827 0.037

0.99 7 10 0.000 0.645 0.563 0.043

20 0.000 0.343 0.294 0.046

40 0.000 0.163 0.143 0.048

3 0.000 0.976 0.963 0.039

5 0.000 0.937 0.891 0.041

9 10 0.000 0.717 0.635 0.045

20 0.000 0.384 0.326 0.047

40 0.000 0.177 0.154 0.049

3 0.000 0.987 0.981 0.040

5 0.000 0.962 0.926 0.042

11 10 0.000 0.766 0.683 0.045

20 0.000 0.416 0.352 0.048

40 0.000 0.188 0.163 0.049

3 0.000 0.999 0.999 0.052

5 0.000 0.996 0.988 0.056

20 10 0.000 0.902 0.836 0.058

20 0.000 0.539 0.456 0.055

40 0.000 0.231 0.196 0.053

Table 4 continued: Tukey test size simulation (see Section 5.3 for details), ρ = 0.99

77
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

78
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

79
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

80
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

81
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

82
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

83
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

84
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

85
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

86
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

87
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

88
5 10 15 20

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

         

89
5 10 15 20

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

         

90
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

91
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

92
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

  	
 

      

93
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

94
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

95
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

96
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

97
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

98
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

99
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

100
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

101
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

102
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

103
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

104
5 10 15 20

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
         

105
5 10 15 20

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
         

106
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

107
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

108
5 10 15 2

0

0.0 0.2 0.4 0.6 0.8 1.0

Non−centrality parameter index

Probability of rejection

th

ps, ancova

ps, 2−way

no ps, 1−way

ps, 1−way, no rho

 	 
 
      

109
Correlation

Confidence interval inflation factor

0.70 0.75 0.80 0.85 0.90 0.95 1.00

1.0 1.5 2.0 2.5 3.0

J = 10

J = 8

J = 6

J = 4

J = 2

  	

  


  

 


110
Correlation

Confidence interval deflation factor

0.70 0.75 0.80 0.85 0.90 0.95 1.00

0.0 0.2 0.4 0.6 0.8 1.0

J = 10

J = 8

J = 6

J = 4

J = 2

  
  


  

 


111
Correlation

Actual coverage

0.70 0.75 0.80 0.85 0.90 0.95 1.00

0.95 0.96 0.97 0.98 0.99 1.00

J = 10

J = 8

J = 6

J = 4

J = 2

  	

   
 
  

  !" 






112
Correlation

Actual coverage

0.70 0.75 0.80 0.85 0.90 0.95 1.00

0.0 0.2 0.4 0.6 0.8 1.0

J = 10

J = 8

J = 6

J = 4

J = 2

 # 
   
 
  

  !" 






113
0 20 40 60 80

0.85 0.90 0.95 1.00

I

Coverage

ANOVA t

ANOVA z

 $ %

 
& 
   
  

  !"




 &  ' ()(  '  *
 &  (! 
 (!#

114
0 20 40 60 80

0.85 0.90 0.95 1.00

I

Coverage

ANCOVA t

ANCOVA z

 ) %

 
& 
 +  
 + 

 

!" 


 &  ' ()(  '  *
 &  (! 
 (!#

115
I (number of samples per treatment)

Actual confidence interval coverage

0 10 20 30 40

0.60 0.65 0.70 0.75 0.80 0.85

incorrect k

pred. sort 1

pred. sort 2

MLE

 ! ,,
- 
  


  
 
 

  .($/


 & & , &0 && 
 -&  - 


 1


- ,
 
 - &,

& 1& () 2- 
  
  
&   1& 
 2- 


  1& 
 - ((3 4
 .%  5 

  5((/

116
Correlation

Sample size reduction factor

0.70 0.75 0.80 0.85 0.90 0.95 1.00

0.0 0.2 0.4 0.6 0.8 1.0

J = 2

J = 4

J = 6

J = 8

J = 10

J = 30

 ( 
  


 

 

 - (( 4
 & , &0 6



 
 &  


 
 - 


 1
 - ,
 
 -

&,

& 
 - 
  
  
&  

117