Groups Versus Individuals

Type I Error Rates of Four Methods for Analyzing Data Collected in a Groups-VersusIndividuals Design

byStephanie Wehry
University of North Florida
andJames Algina
University of Florida1Groups Versus Individuals

2

ABSTRACT
Using previous work on the Behrens-Fisher problem, two approximate degrees of
freedom tests, that can be used when one treatment is individually administered and one isadministered to groups, were developed. Type I error rates are presented for these tests, anadditional approximate degrees of freedom test developed by Myers, Dicecco, and Lorch (1981),
and a mixed model test. The results indicate that the test that best controls the Type I error ratedepends on the number of groups in the group-administered treatment. The mixed model test
should be avoided.
Keywords: groups-versus-individuals design, approximate degrees of freedom tests, mixed
modelsGroups Versus Individuals

3Type I Error Rates of Four Methods for Analyzing Data Collected in a Groups-VersusIndividuals Design
Introduction
When a groups-versus-individuals design is used to compare two treatments, one
treatment is administered to J groups of n participants (for a total of N G such participants) and
one treatment is individually administered to N I participants or the individual participants may
be in a no-treatment control group. For example, psychotherapy researchers investigating theefficacy of group therapy often use a wait-list control group (Burlingame, Kircher, and Taylor,1994). The therapy is provided to participants in groups because the researchers believe groupprocesses will enhance the effectiveness of the therapy. Group processes do not affect theparticipants in the wait-list control group because they do not receive a treatment, much less meetin groups. According to Clarke (1998) the most common design in psychotherapy research
involves the use of a randomly assigned control condition, which can feature a variety of notreatment control schemes.
The groups-versus-individuals design is also used when the purpose is to compare theeffectiveness of an active treatment delivered to groups to an active treatment delivered
individually. For example Bates, Thompson, and Flanagan (1999) compared the effectiveness ofa mood induction procedure administered to groups to the effectiveness of the same procedureadministered to individuals. Boling and Robinson (1999) investigated the effects of study
environment on a measure of knowledge following a distance-learning lecture. The three levelsof study environment included a printed study guide accessed by individuals, an interactive multimedia study guide accessed by individuals, and a printed study guide accessed by cooperative
study groups.
A possible model for the data collected in a groups-versus-individuals design consists of
two submodels. For participants in the individually administered treatment the submodel isGroups Versus Individuals
Yi:TI = µ I + ε i:TI

4
(1)

where i : T I ( i = 1,…, N I ) denotes the ith participant within the individually-administeredtreatment. For participants in the group-administered treatment
Yi: j:TG = µG + α j:TG + ε i: j:TG

(2)

where i : j : T G ( i = 1,…, n ) denotes the ith participant within the jth group ( j = 1,… , J ) in thegroup-administered treatment. An important question is whether to treat the α j:TG as fixed or
random. When the researcher views the groups in the group-administered treatment as
representative of a larger number of groups, α j:TG should be treated as random. In the remainderof the paper we assume that the groups in the group-administered treatment comprise a random
factor with the groups in the study representing an infinitely large number of groups.
Burlingame, Kircher, and Taylor (1994) reported that the independent samples t test,
ANOVA, and ANCOVA were the most commonly used methods for analyzing data in grouppsychotherapy research. It is well known that these procedures require the scores for individual
to be independently distributed both between and within treatments, an assumption that is likely
to be violated for the participants in the group-administered treatment when α j:TG is random. It isalso well known that these procedures are not robust to violations of the independence
assumption (see, for example, Scheffe, 1958). When the groups-versus-individuals design isused, lack of independence is indicated by a non-zero intraclass correlation coefficient for theparticipants who receive the group-administered treatments. Myers, Dicecco, and Lorch (1981),
using simulated data, showed that the Type I error rates for the independent samples t test is
above the nominal alpha level when the intraclass correlation is positive. Burlingame, Kircher,and Honts (1994) reported similar results. In passing we note that if the researcher believes it isappropriate to treat the α j:TG as fixed, if both error terms are normally distributed, and if the error

Groups Versus Individuals

5

terms have equal variances, the treatments can be compared by using an independent samples
ANOVA and testing the hypothesis

H 0 : µ I = µG

(3)

but generalization of the results to additional groups is not warranted.
Myers et al. (1981) developed two statistical tests of the hypothesis given in equation (3).
These tests take the lack of independence into account and allow generalization of the results tothe population of groups represented by the groups in the group-administered treatment. (In thefollowing, groups will always refer to the groups in the group-administered treatment.) One ofthese procedures used a quasi-F statistic and degrees of freedom approximated by theSatterthwaite (1941) method. Formulated as an approximate degrees of freedom (APDF) t
statistic, the Myers et al. test statistic ist APDF =YI − YG
MS S / TI MSG / TG
+
NI
NG

(4)

NI

where YI = ∑Yi:TI N I is the mean of the criterion scores and
i =1

NI

∑ (Y

MS S / TI = i =1

i:TI

− YI

)

2

(5)

NI −1

is the variance for participants who received the individually administered treatment;Jn

YG = ∑∑ Y i: j:T G N G is the mean of the criterion scores for participants who received the groupj =1 i =1

administered treatment ( i : j : TG ) andJ∑ n (YMSG / TG = j =1

j:TG

− YG

J −1

)

2

(6)

Groups Versus Individuals

6is the between-group mean square for these participants. It can be shown that the squared
denominator of t APDF estimates the sampling variance of the numerator assuming a correct modelfor the data is given by equations (1) and (2) and α j:TG is random. Assuming that

ε i:T ~ N ( 0,σ I2 ) , α j:T ~ N ( 0,τ 2 ) , and ε i: j:T ~ N ( 0,σ G2 ) , the estimated approximate degrees ofIGGfreedom are
2

MSG / TG
 MS S / TI

+


N
N
I
G
fˆ2 = 22

 MS S / TI  MSG / TG
N G 
N I  

+
NI −1
J −1

(7)

It should be noted that in using the Satterthwaite method, the distribution of the square of thedenominator of t APDF is approximated as a multiple of a chi-square distribution with degrees of
freedom estimated by fˆ2 .
Based on simulated data, Myers et al. (1981) reported estimated Type I error rates fortheir APDF test, including results for J = 4 and J = 8 groups in the group-administeredtreatment. For both numbers of groups, estimated Type I error rates were very similar to thenominal level. While these results indicate that the APDF has adequate control of the Type I
error rate when J ≥ 4 , it leaves open the question of how well the test works with a smaller
number of groups and the discussion in Satterthwaite (1941) and results in Scariano and
Davenport (1986) suggest the test may not control the Type I error rate for J ≤ 3 .
The discussion in Satterthwaite (1941) implies that the approximation of the square of thedenominator of t APDF by a multiple of a chi-square distribution improves as J − 1 or N I − 1
increases and as( N I − 1) ( nτ 2 + σ G2 )( J − 1)σ I2(8)Groups Versus Individuals

7

becomes closer 1.0. When there are two groups in the group-administered treatment, J − 1 is as
small as it possibly can be. In addition, calculating the ratio in equation (4) for conditions inwhich σ I2 = τ 2 + σ G2 shows that the ratio can be much larger than 1. Therefore, the discussion in
Satterthwaite would lead one to expect that the APDF t test in Myers et al. (1981) would not work
well when there are just two groups.
Scariano and Davenport (1986) studied Type I error rates for the APDF t test that Welch(1938) proposed as a solution to the Behrens-Fisher problem:t=Ya − Yb
Sa2 Sb2
+
Na Nb

.

(9)

In t, Ya and Yb are means for two individually administered treatments, Sa2 and Sb2 are the sample
variances, and the square of the denominator estimates the sampling variance of the numerator.
The distribution of the Welch t can be approximated by a t distribution with degrees of freedom
approximated the by the Satterthwaite (1941) method. Thus, the Myers et al. (1981) APDF test
and the Welch APDF solution to the Behrens-Fisher problem are both based on the same
theoretical approach to approximating the sampling distribution of the test statistic.
Scariano and Davenport (1986) developed an analytic procedure for calculating the TypeI error rate of the Welch APDF test and showed its Type I error rate can be seriously inflated
when (a) there is a negative relationship between the sampling variances of the means and thedegrees of freedom for the estimated sampling variances and (b) the smaller of the two degrees of
freedom is small. In the Myers et al. (1981) APDF test, the sampling variances of the means are( nτ + σ ) N and σ
2

2
G

G

2
I

N I and the degrees of freedom for estimates of these variance are J − 1

and N I − 1 . When N I = N G and σ I2 = τ 2 + σ G2 , for example, the relationship will be negative
and, when J ≤ 3 , the degrees of freedom will be small. Consequently, the APDF test may notGroups Versus Individuals

8

work well in these conditions. One purpose of the study is to study Type I error rates when J issmall.
Satterthwaite (1941) showed how to approximate the distribution of a sum of two chisquare distributed random variables by another chi-square distribution. He determined thedegrees of freedom for the approximating distribution by equating the mean and variance of thesum with the mean and variance of the approximating chi-square distribution. Thus, theSatterthwaite approach is a two-moment approach to determining the degrees of freedom.
Scariano and Davenport (1986) developed a four-moment approach and showed analytically thatit provides a more conservative test than does the two-moment approach. In the four-moment
approach the estimated approximate degrees of freedom are

fˆ4 =

 u2
1 
+


 J − 1 N I − 1

3

 u3

1

2 +
3 
 ( J − 1) ( N − 1) 
I



2

(10)

where, in the groups-versus individuals design,u=

MSG / Tg N G
MS S / TI N I

.

(11)

A second purpose of the present study was to calculate the actual Type I error rate for the fourmoment approach.
In Scariano and Davenport (1986), the two-moment approach was sometimes liberalwhen the four-moment approach was conservative. As a result, they suggested using an averageof the estimated degrees of freedom produced by the two approaches. Thus, a third purpose wasto analytically evaluate the actual Type I error rate for this averaged degrees of freedom
approach.
An alternative to the preceding approaches is based on a mixed model with a proper
inference space (McLean, Sanders, & Stroup, 1991) and Satterthwaite degrees of freedom.

Groups Versus Individuals

9

When the restricted maximum likelihood estimate (RMLE) of τ 2 is larger than zero and there arean equal number of participants in the groups, the mixed model test is equivalent to the Myers etal. (1981) two-moment test. However, if the RMLE is zero, MSG / TG and MS S / G / TG are pooled
and replace MSG / TG in equation (1). This statistic, which is equivalent to the Welch t test, issmaller than t APDF and may be more conservative than the two-moment test. However, it tends tohave larger degrees of freedom, which may make it more liberal than the two-moment test.
When there are an equal number of participants in the groups, the RMLE of τ 2 is zero
when the method of moments estimate of τ 2 is ≤ 0 (McCulloch & Searle, 2001). The probability
that the method of moments estimate of τ 2 is ≤ 0 is
1 − ρ ICC
prob MSG / TG ≤ MS S / TG / TG = prob  F  J − 1, J ( n − 1) ≤,ρ ICC ( n − 1) + 1 


{

}

(12)

where ρ ICC = τ 2 (τ 2 + σ G2 ) . Figure 1 displays the probability as a function of J, ρ ICC , and n. The
probability can be quite substantial and, in some conditions, we would expect the mixed model
test to perform differently than the two-moment, four-moment, and averaged degrees of freedom
tests. Thus, a fourth purpose of the study is to compare these tests to the mixed model test.

The research was carried out in two studies. In the first study, actual Type I error
rates were calculated for the two-moment approach, the four-moment approach, and theaveraged degrees of freedom approach. In the second study, simulated data were used toestimate the actual Type I error rate for the mixed model approach as well as for the twomoment approach, the four-moment approach, and the averaged degrees of freedom
approach. Taken together, the purposes of the studies were to compare Type I error rates
for the two-moment, four-moment, averaged degrees of freedom, and mixed model
approaches when the number of groups in the group administered treatment is small andGroups Versus Individuals 10
to study the influence of the number of groups, number of participants in a group, andintraclass correlation on the Type I error rates for these methods.

Probability

0.6

ICC=.00, J=2
J=4
J=6
ICC=.15, J=2
J=4
J=6
ICC=.30, J=2
J=4
J=6

0.4

0.2

0.0

4

8

12
Sample Size

16

20

Figure 1. Probability of a Negative Estimate for τ 2

Methodology: Study 1

Actual Type I error rates were calculated for each condition in a 5 (Number ofGroups) × 4 (Intraclass Correlation) × 15 (Number of Participants in a Group)completely crossed factorial design. The levels of the factors were J = 2 to 6 for thenumber of groups; n = 3 and 4, and 6 to 30 in steps of 2 for the number of participants in agroup; and ρ ICC = .00, .20, .40, and .80 for the intraclass correlation. In all conditions,(τ + σ ) σ = 1 and, because the design was balanced across treatments, N = J ( n ) . For all
2

2
G

2
I

I

calculations the nominal alpha level was .05. In the following, when we use the term Type I

error rate without the actual or nominal modifier, we refer to the actual Type I error rate.

Groups Versus Individuals 11
Calculating Type I Error Rates
Scariano and Davenport (1976) developed a method to calculate Type I error rates for the
Welch t test. We applied their method, which we describe below, to the three APDF tests
considered in this paper. It should be noted that although the method we applied was developedin the context of the Behrens-Fisher problem, that is, comparing means of independently
distributed scores for two groups when the variance are not equal for the groups, we did not apply
the method to the Behrens-Fisher problem. Rather we applied the method to comparison of means
for two groups, when scores are not independently distributed within the sub-groups in the groupadministered treatment. Thus, our work is not subject to Sawilowsky’s (2002) criticisms ofresearch on the Behrens-Fisher problem.

The Type I error rate for the APDF t test is∞22> Fα ,1, fˆ  = ∫ Pr  tquasi> Fα ,1, fˆ u  g (u )duPr tquasi





(13)

0

where fˆ is the two-moment, four-moment, or averaged degrees of freedom and α is the2nominal Type I error rate. Cochran (1951) has shown that tquasi
is the ratio of Q to C where

Q ~ F1, m1 + m2 , m1 = J − 1 , m2 = N I − 1 ,C=(1 + u )( m1 + m2 ) ,mu(1 + U )  1 + m2 
 U


(14)

and

( nτ + σ
U=
2

σ

2
I

2
G

NG )

NI

.

(15)

To facilitate numerical integration the variable u can be transformed to m1u ms=  2  m1u 
1 +

m2 


(16)

Groups Versus Individuals 12
and the Type I error rate is found by numerically integrating1∫ Pr Q > C × Fα
0,1, fˆ

s  f ( s ) ds


(17)

where
 m + m2  m2 2
Γ 1
(m −2) 2
Us ( m1 −2 ) 2 (1 − s ) 22 
.
f (s) =
 m1   m2  U (1 − s ) + s  ( m1 + m2 2 )Γ Γ   2   2 (18)

Numerical integration was performed using the trapezoid rule. For J = 2 a singularity occurs at

s = 0 . Therefore, the limits of integration were .0001 and 1. The interval was divided into 1000
segments of equal width. For J = 3 a removable singularity occurs at s = 0 . For J ≥ 3 thelimits of integration were 0 and 1 and this interval was also divided into 1000 segments. As acheck on the calculations, Type I error rates were estimated by using simulated data with 100,000
replications. The results from the simulation were consistent with the results determined bynumerical integration.
Results: Study 1
Figures 2 to 6 contain plots of the Type I error rates against size of groups. The five plots
are for two, three, four, five, and six groups, respectively. Plots within a figure are organized by
the intraclass correlation coefficient. Inspection of Figure 2 indicates that when there are twogroups, the four-moment degrees of freedom should be used, except perhaps when ρ ICC = 0 .
Then the averaged degrees of freedom might be used. When there are three groups (see Figure3), the averaged degrees of freedom might be used at the risk of a slightly liberal test when ρ ICC
is at .20 or greater. The two-moment degrees of freedom results in a test that is too liberal and thefour-moment degrees of freedom results in a test that is too conservative. When there are fourgroups (see Figure 4), the two-moment degrees of freedom provides a test that has a slight liberal
tendency that increases as ρ ICC get larger and as the size of the groups get larger. Use of theGroups Versus Individuals

ICC=0.0

0.10Type I Error Rate

0.10
Type I Error Rate

ICC=0.2

0.12

f2
f4
fa

0.12

13

0.08

0.06

0.06

0.04

0.04

0

5

10

15

20

25

f2
f4
fa

0.08

0

30

5

10

ICC=0.4

0.12

20

25

30

ICC=0.8
0.12

0.10

0.10
Type I Error Rate

Type I Error Rate

15
Size of Groups

Size of Groups

f2
f4
fa

0.08

0.06

f2
f4
fa

0.08

0.06

0.04

0.04

0

5

10

15

20

25

30

Size of Groups

Figure 2. Plots of Type I Error Rates by Size of Group for Two Groups

0

5

10

15
Size of Groups

20

25

30

Groups Versus Individuals

0.08

0.08

ICC=0.2

ICC=0.00.06

0.06
Type I Error Rate

Type I Error Rate

14

0.04

0.04

f2
f4
fa

f2
f4
fa
0.02

0.02
0

5

10

15

20

25

0

30

5

10

15

20

25

30

20

25

30

Size of Groups

Size of Groups

0.08

0.08
ICC=0.4

ICC=0.8

Type I Error Rate

Type I Error Rate

0.06

0.04

0.06

0.04

f2
f4
fa

f2
f4
fa

0.02

0.02
0

5

10

15
20
Size of Groups

25

30

Figure 3. Plots of Type I Error Rates by Size of Group for Three Groups

0

5

10

15
Size of Groups

Groups Versus Individuals

ICC=0.0

ICC=0.2

0.06

0.05

0.05

Type I Error Rate

Type I Error Rate

0.06

0.04

0.03

0.04

0.03

f2
f4
fa

0.02
0

5

10

15

20

25

15

f2
f4
fa

0.02

30

0

5

10

15

Size of Groups

25

30

20

25

30

ICC=0.8

ICC=0.4
0.06

0.06

0.05

0.05
Type I Error Rate

Type I Error Rate

20
n

0.04

0.03

0.04

0.03

f2
f4
fa

f2
f4
fa

0.02

0.02
0

5

10

15

20

25

30

Size of Groups

Figure 4. Plots of Type I Error Rates by Size of Group for Four Groups

0

5

10

15
Size of Groups

Groups Versus Individuals 16
averaged degrees of freedom provides a test that is slightly conservative when ρ ICC is small, but
controls the Type I error rate well as ρ ICC increases. Plots for five or more groups (see Figures 5
and 6) are similar to those for four groups. However, the use of either the two-moment degrees of
freedom or and average degrees of freedom provide reasonable control of the Type I error rate.
Use of the former can result in a slightly liberal test, whereas use of the latter can result in aslightly conservative test.
Methodology: Study 2
As noted in the introduction, simulated data were used to compare the three APDF tests
and the mixed model test. The design had four factors: the four tests, the number of groups, sizeof the groups, and level of the intraclass correlation. There were five levels of the number ofgroups, J = 2, 3, 4, 5, and 6; five levels of group size, n = 4, 8, 12, 16, and 20 subjects nested in
the groups; and seven levels of intraclass correlation, ρ ICC = .00 to .30 in steps of .05.

The simulation was carried out using the random number generation functions ofSAS, Release 8.2. Scores for simulated participants in the individually administeredtreatment level were generated using the equation (1), where µ I was arbitrarily set at 100
and the ε i:T s were pseudorandom standard normal deviates generated using RANNOR.
I

Scores for simulated participants in the group-administered treatment level weregenerated using equation (2), where µG was arbitrarily set at 100, α j:T was aG

pseudorandom normal deviate with mean zero and variance τ 2 and ε i: j ;T was aG

pseudorandom normal deviate with mean zero and variance σ G2 . Each of the conditions
was replicated 5,000 times and the Type I errors of the four tests were counted over the
replications of each condition. The nominal type I error rate was .05 in all conditions.

Groups Versus Individuals

0.06

0.06

ICC=0.0

ICC=0.2

0.05Type I Error Rate

Type I Error Rate

0.05

0.04

0.03

0

5

0.04

0.03

f2
f4
fa

f2
f4
fa

0.02
10

15

20

25

0.02

30

0

Size of groups

5

10

15

20

25

30

20

25

30

Size Of Groups

0.06

0.06

ICC=0.8

ICC=0.4

0.05Type I Error Rate

0.05
Type I Error Rate

17

0.04

0.03

0.04

0.03

f2
f4
fa

f2
f4
fa
0.02

0.02
0

5

10

15

20

25

30

Size of Groups

Figure 5. Plots of Type I Error Rates by Size of Group for Five Groups

0

5

10

15
Size of Groups

Groups Versus Individuals

0.06

0.06

18

ICC=0.2

ICC=0.0

0.05Type I Error Rate

Type I Error Rate

0.05

0.04

0.04

0.03
f2
f4
fa

0.03

f2
f4
fa
0.02

0

5

10

15

20

25

0

30

5

10

15

20

25

30

Size of Groups

Size of Groups

0.06

0.06

ICC=0.8

ICC=0.4

0.05Type I Error Rate

Type I Error Rate

0.05

0.04

f2
f4
fa

0.03

0.04

f2
f4
fa

0.03

0.02
0

5

10

15

20

25

30

Size of Groups

Figure 6. Plots of Type I Error Rates by Size of Group for Six Groups

0

5

10

15
Size of Groups

20

25

30

Groups Versus Individuals 19
The mixed model specified in equations (1) and (2) was implemented by using thefollowing is a SAS program. The individually administered treatment is coded 1 on the
TRT code.
PROC MIXED;CLASS TRT GROUP;MODEL SCORE=TRT/SOLUTION DDFM=SATTERTHWAITE;RANDOM GROUP/GROUP=TRT;REPEATED/GROUP=TRT;PARMS (0) (1) (1) (1)/EQCONS=1
ESTIMATE 'COMP' TRT 1 -1;

The APDF tests are easily carried out in proc iml as the only required statistics are themeans for the two groups, the variance for the treatment administered to individuals, andthe mean squares within and between subgroups for the group-administered treatments.
Results: Study 2
The analytic results showed that, when there were two groups, the APDF test statisticwith the four-moment degrees of freedom provided the best control of the Type I error rate.
Figure 7 compares Type I error rate for the four-moment test and the mixed model test forρ ICC = 0.00 and 0.30. Results for the APDF test statistic and the two-moment degrees of
freedom are also included because the mixed model test is equivalent to the two-moment testwhen the estimate of τ 2 is non-zero. The four-moment degree of freedom test still provides thebest control of the Type I error rate. The mixed model test is more conservative than the twomoment test and is substantially more conservative in conditions in which the probability of azero estimate for τ 2 is large.

Groups Versus Individuals 20

f2, ICC=.00
f2, ICC=.30
f4, ICC=.00
f4, ICC=.30
fm, ICC=.00
fm, ICC=.30

0.13

Type I Error Rate

0.11

f2
fm

0.09

f2

0.07

f4
0.05

f4
fm

0.03

4

8

12
Size of Groups

16

20

Figure 7. Type I Error Rates for Two Groups
When there were three groups, the analytic results showed that the APDF test statisticwith the averaged degrees of freedom provided the best control of the Type I error rate. Type I
error rates for the two-moment test tended to be too large. Figure 8 compares Type I error rates
for the mixed model test and the APDF tests with two-moment and averaged degrees of freedom
when ρ ICC = .00 and .30. The results indicate that the averaged degrees of freedom test still
provides the best control of the Type I error rate.
According to the analytic results, there were four or more groups, both the two-moment
and averaged degrees of freedom tests provided good control over the Type I error rate, with theformer test being slightly more liberal. Type I error rates are depicted in Figure 9 for the twomoment, four-moment test and the mixed model tests for ρ ICC = .00 and .30. The results indicatethat the mixed-model test is conservative and less adequate than the other tests whenGroups Versus Individuals 21
ρ ICC is zero. Inspection of the results for other values of ρ ICC indicate that when ρ ICC = .10 theperformance of the averaged degrees of freedom and the mixed model tests is very similar andas ρ ICC increases the Type I error rates for the mixed model test become slightly larger than thosefor the averaged degrees of freedom test. A similar pattern of results emerged for five or six
groups. In particular, when ρ ICC was near zero the mixed model test was too conservative.
f2, ICC=.00
f2, ICC=.30
fa, ICC=.00
fa, ICC=.30
fm, ICC=.00
fm, ICC=.30

0.09

f2
fm

0.07

fa
f2
fa

0.05

fm
0.03

4

8

12
Size of Groups

16

20

Figure 8. Type I Error Rates for Three Groups
Conclusion
Myers et al. (1981) presented a two-moment approximate degrees of freedom test for use
when one treatment is delivered to individual participants and one is delivered to groups ofparticipants. The test was based on results in Satterthwaite (1941). Simulation results indicatedthat the test provided good control of the Type I error rate for both four groups and eight groupsof participants. Satterthwaite (1941) and Scariano and Davenport (1986) studied a two-moment
approximate degrees of freedom test for a design in which both treatments are delivered

Groups Versus Individuals 22
0.08
f2, ICC=.00
f2, ICC=.30
fa, ICC=.00
fa, ICC=.30
fm, ICC=.00
fm, ICC=.30

Type I Error Rate

0.07

f2
fm

0.06

f2
fa

0.05

fa

0.04
fm

0.03

4

8

12
Size of Groups

16

20

Figure 9. Type I Error Rates for Four Groups
individually. Discussion in Satterthwaite and results in Scariano and Davenport suggest that theMyers et al. test may not perform well when the number of groups is smaller than four. Using ananalytic procedure developed by Scariano and Davenport, we showed that the Myers et al. test
could provide relatively poor control of the Type I error rate when there are two or three groups.
Using results presented in Scariano and Davenport, we developed two alternatives to the Myers etal. (1981) test, a four-moment approximate degrees of freedom test and an averaged degrees of
freedom test. Using the analytic procedure developed by Scariano and Davenport, Type I error
rates were calculated for all three test in a wide range of conditions in which the design wasbalanced across the individually administered treatment and the group-administered treatment andacross the groups in the group-administered treatment. We also estimated Type I error rates forthe mixed model test and the three APDF tests. The results indicated that the four-moment test
should be used when the group-administered treatments are delivered to two groups and theaveraged degrees of freedom test should be used when the group-administered treatments areGroups Versus Individuals 23
delivered to three groups. When there are between four and six groups, we recommend using the
averaged degrees of freedom test. However, because (a) this test is slightly conservative, with aType I error rate between 0.045 and 0.050, and (b) the two-moment test is slightly liberal but
tends to keep the Type I error rate below 0.06, some may prefer the two-moment test. Even when
there are four or more groups, we do not recommend the mixed model test because of its
conservative tendency when the intraclass correlation coefficient is small. These
recommendations are summarized in the Table 1
Table 1
Recommended Tests by the Number of Groups in the Group-Administered Treatment
Number of Groups

Recommended Test

2

Four-Moment Test

3

Averaged Degrees of Freedom Test

4-6

Averaged Degrees of FreedomOr
Two Moment Test

When there are two groups in the group-administered treatment, the four-moment test
provides better control of the Type I error rate than do the other tests. Nevertheless researchers
should be cautious about using a groups-versus-individuals design with two groups because suchdesigns will provide relatively low power. The true degrees of freedom for the four-moment test
is

f4 =

 U2
1 
+


 J − 1 N I − 1

3

 U3

1

2 +
3 
 ( J − 1) ( N − 1) 
I



2

(19)

where U is defined in equation (15). Calculations show that f 4 approaches 1.0 from above as U
increases. Thus in many situations the degrees of freedom for the four-moment test will be very

Groups Versus Individuals 24
small and this will have a negative impact on power. In addition, substituting population
parameters for sample statistics in the Myers et al. (1981) t statistic, we haveµ I − µG
σ

2
I

NI

+

σ G2
NG

+

τ2

.

(20)

J

Therefore even as the two sample sizes increase power will not go to 1.0 if τ 2 ≠ 0 . Finally, thefact that the Type I error rate for the four-moment test declines as n increases suggests power will
decline as n increases because the test becomes more conservative. The predicted low power anddecline in power as n increases were borne out by simulation studies. For example
when σ I2 = σ G2 + τ 2 = 1 , ρ ICC = .2 , and µG − µ I = .8 , estimated power was .23, .21 and .19 as nincreased from 6 to 18 in steps of 6. Comparison of these results to the power of an independent
samples t test with the same overall sample size indicates how much lower power is when agroups-versus-individuals design is used. Note that because σ I2 = σ G2 + τ 2 = 1 , µG − µ I = .8
corresponds to Cohen’s large effect size. Also as n increases from 6 to 18 the sample size in atreatment increases from 12 to 36 in steps of 12. For an independent sample t test with an effectsize equal to .8, power is .47, .77, and .92 as n increase from 12 to 36 by 12.
When there are three groups and the averaged degrees of freedom approach is used,
power does not decline as n increases, but power can still be quit low and does not increase
quickly as n increase. As n increased from 4 to 12, so that the overall sample size remained thesame as in the conditions on which power results were reported for J = 2 , estimated power was.29, .36, and .40 when J was 3.
As suggested by equation (20), power continues to increase as J increases. For example
with J = 6 , as n increased from 2 to 6 in steps of 2 estimated power was .41, .58, and .68 using
the averaged degrees of freedom test. Thus when the groups-versus-individuals test is used, it isimportant to have as many groups as possible and may be more important to have more groups
than to have more participants per group.

Groups Versus Individuals 25
At least four lines of additional research are attractive. First, the performance of the tests
under non-normality should be investigated and if performance is poor developing the test
statistic and degrees of freedom using robust estimates of the means and mean squares is ofinterest. Second, performance of the four tests when the design is unbalanced across theindividually administered treatment and the group-administered treatment, but balanced acrossgroups in the group-administered treatment might be investigated. Third, calculating the
averaged degrees of freedom by differentially weighting the two-moment and four-moment
degrees of freedom might be investigated when there are four or more groups. Weighting thetwo-moment degrees of freedom more heavily will reduce the slight conservative tendency of theaveraged degrees of freedom test. In general, more extensive studies of power than we haveconducted would be worthwhile. Fourth, the three APDF tests should be generalized for use when
the design is not balanced across groups in the group-administered treatment and Type I error
rates for these tests and the mixed model test should be investigated.

Groups Versus Individuals 26
References
Bates, G. W., Thompson, J. C., & Flanagan, C. (1999). The effectiveness of individual
versus group induction of depressed mood. The Journal of Psychology, 33, 245-252.
Boling, N. C., & Robinson, D. H. (1999). Individual study, interactive multimedia, or
cooperative learning: Which activity best supplements lecture-based distance education? Journalof Educational Psychology, 91, 169-174.
Bradley, J.V. (1978). Robustness? British Journal of Mathematical and Statistical
Psychology, 31, 144-152.
Burlingame, G. M., Kircher, J. C., & Honts, C. R. (1994). Analysis of variance versus
bootstrap procedures for analyzing dependent observations in small group research. Small Group
Research, 25, 486-501.
Burlingame, G. M., Kircher, J. C., & Taylor, S. (1994). Methodological considerations ingroup psychotherapy research: Past, present, and future practices. In A. Fuhriman & G.
Burlingame (Eds.), Handbook of group psychotherapy and counseling: An empirical and clinicalsynthesis. (pp. 41-80). New York: Wiley.
Clarke, G. N. (1998). Improving the transition from basic efficacy research toeffectiveness studies: Methodological issues and procedures. In A. E. Kazdin (Ed.),
Methodological issues and strategies in clinical research, (2nd ed.) (pp. 541-559). New York:
Wiley.
Cochran, W. G. (1951). Testing a linear relationship among variances. Biometrics, 7, 1732.
McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New
York: Wiley.
McLean, R. A., Sanders, W. L., & Stroup, W. W. (1991). A unified approach to mixed
linear models. American Statistician, 45, 54-64.

Groups Versus Individuals 27
Myers, J., Dicecco, J., & Lorch, Jr., J. (1981). Group dynamics and individual
performances: Pseudogroup and Quasi-F analyses. Journal of Personality and Social Psychology,
40, 86-98.
Satterthwaite, F. W. (1941). Synthesis of variance. Psychometrika, 6, 309-316.
Scariano, S. M., & Davenport, J. M. (1986). A four-moment approach and other practicalsolutions to the Behrens-Fisher problem. Communications in Statistics: Theory and Methods, 15,1467-1504.
Sawilowsky. S. (2002). The probable difference between means when σ 1 ≠ σ 2 : The
Behrens-Fisher problem. Journal of Modern Applied Statistical Methods, 1, 461-472.
Scheffe, H. (1959). The analysis of variance. New York: Wiley.
Welch, B. L. (1938). On the comparison of several mean values: An alternative
approach. Biometrika, 38, 330-336.

Groups Versus Individuals 28
Footnote
1

Tables containing Type I error rates for all conditions in Study 1 and Study 2 are available at

http://plaza.ufl.edu/algina/index.programs.html