CONFIDENCE INTERVALS FOR SUB-DOMAIN PARAMETERS

WHEN THE SUB-DOMAIN SAMPLE SIZE IS RANDOM

ROBERT J. CASADY, ALAN H. DORFMAN1, and SUOJIN WANG2

ABSTRACT

Let A be a population sub-domain of interest and assume that the elements of A cannot be

identified on the sampling frame and the number of elements in A is not known. Further

assume that a sample of fixed size (say n) is selected from the entire frame and the

resulting sub-domain sample size (say nA ) is random. The problem addressed is the

construction of a confidence interval for a sub-domain parameter such as the sub-domain

aggregate T x A i i A = ÂŒ . The usual approach to this problem is to redefine xi , by setting

x iA i = œ 0 if . Thus, the construction of a confidence interval for the sub-domain total is

recast as the construction of a confidence interval for a population total which can be

addressed (at least asymptotically in n) by normal theory. As an alternative, we condition

on nA and construct confidence intervals which have approximately nominal coverage

under certain assumptions regarding the sub-domain population. We evaluate the new

approach empirically using data from the Bureau of Labor Statistics (BLS) Occupational

Compensation Survey.

KEY WORDS: Bayes Method, Conditioning, Establishment Surveys, Simple Random

 Sampling, Stratification, Survey Methods

1. INTRODUCTION

Let xi be the value of the characteristic of interest for the ii N th a f = 1 2,, , K element of

the population and let A be a sub-domain of interest. The elements of A cannot be

identified on the frame and the number of elements in A (say NA ) is not known; however,

it is assumed that any element of A included in a sample can be identified. The problem is

to construct a confidence interval for the sub-domain total, T x A i i A = ÂŒ , based on a

sample of n elements selected from the entire frame.

The usual approach to this problem is to redefine xi , by setting x iA i = œ 0 if , which

forces the population total T xi i

N

= Â=1 to be equal to TA. Thus, the construction of a

 

1 Robert J. Casady and Alan H. Dorfman, U.S. Bureau of Labor Statistics, 2 Massachusetts Ave.

N.E.,Washington D.C., 20212-0001.

2 Suojin Wang, Department of Statistics, Texas A&M University, College Station, TX, 77843.
2

confidence interval for the sub-domain total is recast as the construction of a confidence

interval for a population total. In what follows it is assumed that the xi 's have been

redefined as above. An overview of domain estimation can be found in Chapter 10 of

Sarndal et.al. (1992). If we assume a simple random sample with replacement, the

standard approach to this problem is along the following line:

Define the additional population parameters,

X TN = = population mean,

S xX N i i

N 2 2

1 = - Â= c h = population variance, and

p NN A A = = proportion of population in A.

Then

(1) $

T Nn xi i

n

= Â= a f 1 , x x n TN i i

n

= = Â =1

$ , s xx n i i

n 2 2

1 = --1 Â = b ga f, and 

p nn $ A A = (where nA is the number of sample elements in A) are unbiased for 

the corresponding population parameters,

(2) ET T TA $ d i = = , so, we define the sub-domain estimator $ $ T T A ∫ ,

(3) var $T NS n A d i = 2 2 ,

(4) n T T NS N A A

d $ d ia f a f - æÆæ 0 1, , and

(5) s2 is consistent for S2.

It follows that n T T Ns N A A

d $ d ia f a f - æÆæ 0 1, , so, when n is "sufficiently large",

appropriate values from the normal distribution can be used to construct confidence

intervals for TA.

However, the proportion of the xi

' s that are equal to zero is at least 1- pA , therefore,

when pA is small and the values of the xi' s for i A Œ are concentrated away from zero,

the convergence in distribution in (4) can be extremely slow. Consequently, the

distribution of 

nT T

Ns

A A

$ d i - can be far from normal even for what are usually considered

to be moderate to large values of n. Dorfman and Valliant (1993) noted this problem in
3

their study of wage distributions for sub-domains consisting of workers in specific

occupational groups. Preliminary empirical work by the authors indicated that supposed

95% confidence intervals for total workers and total wages for occupation based

sub-domains typically provided only 75% to 85% coverage even for a large total sample

size (n=353 establishments). Furthermore, this work indicated that the distribution of

$

T T A A - was strongly dependent on the realized value of nA , which suggested that some

type of "conditional" confidence interval should be considered. Thus, the formal goal of

our research was to establish methodology for the construction of conditional (on nA or

equivalently p$ A ) confidence intervals for TA, which provide nominal, or near nominal,

coverage regardless of the realized value of the sub-domain sample size.

In this paper we propose several methods of conditional confidence interval

construction. These methods result from a Bayes based analysis of the conditional

distribution of a random variable of the form $ $ q = - TTs A A T$

A

d i , where s

TA

$ is a

standardizing random variable. The cases of simple random sampling and stratified

random sampling are considered in Sections 2 and 3, respectively. The results of an

empirical evaluation of the methods are discussed in Section 4. Section 5 provides a

summary and concluding remarks.

2. THE CASE OF SIMPLE RANDOM SAMPLING

2.1 Definitions and Notation

We define the following parameters and estimators:

Sub-domain parameters:

mA AA = T N = sub-domain mean,

s m A iA i A A x N 2 2= - ÂŒ b g = variance of population elements in A.
4

Sub-domain estimators:

$

N pN $ A A = ,

m$ $ $ A i i

n

A AA xn T N A

= = Â=1 (only defined for nA ≥ 1), and

s m $ $ A iA i

n

A x n

2 A 2

1 =- -1 Â = b gb g (only defined for nA ≥ 2).

In what follows it is understood that nA ≥ 2 (or equivalently p n $ A ≥ 2 ) unless specifically

stated otherwise. The relationships given below follow directly from the definitions:

T Np A AA = m and $T Np$ $ A AA = m ,

X p = A A m and x p = A A $ m$ ,

Sp p p A A A AA

2 22 =- + b g 1 m s

and

s

n

n

p p np

n

A AA

A

A

2 22

1

1 1

1 = - - + -- $ $ $ $ b gm s$ . (1)

Also, it is straightforward to verify that

nN T T n p p p

p p

np p

p p

p

np

p pZ p Z

A A AA A A A A

AA A

A A

A A

A A

AA A

A

A A A AA

c hd i b g b g c h

b g b g

b g

b g

b g

$ $ $ $

$ $ $ $

$ ,

-= -+ -

= - -

- + -

= -+

m mm

m s

m m

s

m s

1

1

1 1 2

(2)

where Z np p

p p

A A

A A

1

1 = --

b g $

b g , Z npAA A

A

2 = $ m m $ -

s

b g

and

n s p Z np np

A

A AA A

A

A

- =- + + -1 1 1

2

2 2

2 2

2

a f bg b g

s

g

s

s $ $$ $ , where g ms A AA = .
5

2.2 General Methodology for Confidence Intervals

Let $ $

$

q = T T -

s

A A

TA

d i and assume that the conditional (on p$ A ) distribution function of $q ,

say H pp ( ) AA A A ◊ $ ;,, m s 2 , is known. In order to construct a conditional equal tailed

a f 1 100 - ¥ a % CI for TA, we define an upper critical value

c c p p xH xp p H p p u u AA AA AA ∫ =- ≥ =- ( ) { } ( ) ( ) - a aa , $ , inf $ ; , 2 2 $ ; 1

where pA is considered fixed and the dependence on m A and s A

2 is temporarily

suppressed; a lower critical value, say cl , is defined in a similar manner. A conditional,

equal tailed a f 1 100 - ¥ a % CI for TA is then given by CI u a fa f 1- = a l, , where

u T cs

T cs

A u T

A T

A

A

= +

= +

$

$ .

$

$

 and

l l

 (3)

At this point the obvious practical problem is that the critical values cu and cl depend

not only on p$ A but also on the unknown parameter pA . One approach to this problem is

to take a Bayesian tack and assume the parameter pA is the realization of a random

variable. Adjusting the notation for this assumption, we have H xp p H xp p ( ) $ AA AA , ∫ ( ) $ ; ,

and it follows that 

Pr $ $ $

$ $ , $ ,

q £ =

= z

xp F xp

h p H x p p f p p g p dp

A A

A

A A AA A A

o t c h

b g c hc hb g 1 (4)

where h p f p p g p dp A AA A A b g c hb g $ $ = z and g pA b g is the pdf of pA . It should be noted that as

a consequence of our sampling scheme the distribution of npA$ , conditional on pA , is
6

Binomial n pA b g , so that fpp A A c h $ is known. Under the Bayesian approach, the critical

values are cc p F p uu A A ( ) ( ) * * ∫ =- , $ $ - a a1 2 and cc p F p l l ( )A A ( ) * * ∫ =- - , $ $ - a a 1 1 2 so the

upper and lower limits for a conditional a f 1 100 - ¥ a % CI for TA are

u T cs

T cs

A u T

A T

A

A

= +

= +

$

$ .

*

$

*

$

 and

l l

 (5)

For the purposes of our current research, we are assuming that the prior distribution

g pA b g is N p p A A m s, 2 d i with mpA and s pA

2 to be specified. For an empirical Bayes approach,

we used mp A A = p$ and considered several possible alternatives for s pA

2 which we discuss in

detail below. Our experience indicates that the normality assumption is not crucial, rather,

it is primarily a matter of convenience. On the other hand, the choice of values for the

mean and variance is relatively more important. This will also be discussed in more detail

at later point.

2.3 Confidence Intervals Under Normal Assumptions

Assume that within the sub-domain A the xi are distributed N m s A A , 2 c h. Then

(a) nT T Np p A A AA

$ d i - $ , is distributed Nn p p p m s A A A AA c h b g $ - , $ 2 ,

(b) np p p A

A

A

A A $ $ - $ , L

N

M O

Q

P 1

2

2 b gs

s

 is distributed c 2 np 1 A b g $ - ,

(c) Z np p p 2 AA A A

2

+ L

NM OQP d i $ $ g , is distributed non-central c l 2 1; * c h with l g * = np$ A A

2

, and

(d) the conditional random variable in (b) is stochastically independent of the

conditional random variables in (a) and (c).
7

Consider ( )

( ) $ $

$ $ q

s 1 = T T -N pn

A A

A A

 which utilizes the conditional variance of $TA as the

standardizing term. It follows immediately from (a), (b) and (d) that, conditional on

p p $ , A A b g, the random variable $q1 is distributed as a non-central t with npA$ -1 degrees of

freedom and non-centrality parameter l g = - n

p p

p

A

A A

A

$

$

b g . Thus, we have specified the

conditional distribution function H pp A A c h ◊ $ , of $q1 . As fpp A A c h $ and g pA b g have been

previously specified, it follows that F pA c h ◊ $ in (4) is well-defined although extremely

cumbersome to calculate. The dependence on m A and s A

2 , through g A , should be noted.

Although F pA c h ◊ $ as given above can be used to determine the critical values, they are

extremely difficult to calculate. A relatively simple approach, given in the next paragraph,

provides a close approximation to the critical values. We have verified the closeness of

the approximation by computing the exact values for selected cases using large scale

simulations.

Under the assumption that pA is distributed as a N pA pA

$ ,s 2 d i, it follows from Appendix

A that l p$ A is distributed approximately as a normal with mean zero and variance

g

y

A A

A

p

2 1

1

-

+

b g $ , where y

s

A

A A

p

p pn

A

= $ $ 1-

2

b g . It then follows from the result in Appendix B

that, conditional on p$ A ,

$

$ $ $

T T

N p

n

p

A A

AA A A

A

-

-

+

+

d i

s g b g

y

2 1

1

1

is distributed as a central t with npA$ -1 degrees of freedom. The upper confidence limit
8

u , defined in (5), is given (approximately) by

u T N p

n

p

t A

A A AA A

A

nA = +

- ++

+ - - $ $ $ $

,

s g y

y a

2 1 2

12 1 2 1

1 1

1

c h b g

b g . (6)

As s$ A

2 is conditionally unbiased for s A2 and m s $ $ A AA n 2 2 - is conditionally unbiased for

m A

2 , we use ( ) g ms s $ $$ $ A A AA A n 2 22 2 = - to estimate g A

2 and approximate u with

( ) ( )

( )

( )

( )

~ $ $ $ $ $

$

$ $

,

,

u T N p

n

p

t

T Ns

n

n

n

n

n

p n

s

t

A

A A AA A

A

nA

A

AA A

A

nA

= +

- ++

+

= + Ê -

Ë

Á ˆ

 ̄

 ̃

+ -

Ê

Ë

Á ˆ

 ̄

 ̃ Ê +

Ë

Á ˆ

 ̄

 ̃

+

- -

- -

s g y

y

s y

y

a

a

2 1 2

12 1 2 1

1 2

2

2

1 2

1 2 12 1

1 1

1

1

1

1

1

1

.

A necessary condition for the results in Appendix A to hold is that n be large enough to

ignore the terms with order O n-1 c h, in which case

( ) ~ $ $ $ u T ,

Ns

n

p

s

t A

AAA

A nA @+ + Ê

Ë

Á ˆ

 ̄

 ̃ + Ê

Ë

Á ˆ

 ̄ 1 1  ̃ - -

2

2

1 2

12 1

s y y a . (7)

It should be noted that ~u is strictly decreasing as y A increases and

~ $ ~ u T ,

Ns

n A n t u Æ+ = 12 1 1 - - a A as y A becomes small,

( ) ~ $ $ $ $ ~ , u T Ns

n n

p p

s

t u

n A

A

A AA

nA A

A

@+ -

+

Ê -

Ë

Á ˆ

 ̄

 ̃ Ê

Ë

Á ˆ

 ̄

 ̃ 1 - -

4

4

1 4 2

2

1 2

12 1 2

m a = , for = (8) y

~ $ $ $ ~ u T ,

Ns

n

p s A t u A A

nA = + Ê +

Ë

Á ˆ

 ̄

 ̃ - -

1

2

2 2 1 2

12 1 3

s a = for y A = 1, and

~ $ $ $ ~ , u T Ns

n

p

s

t u A

A A Æ + nA

Ê

Ë

Á

Á

ˆ

 ̄

 ̃

 ̃ - - = s 12 1 4 a as y A becomes large.
9

In each case the lower critical value can be dealt with in an analogous manner resulting in

four competing confidence intervals; namely, CI u i i ii ( ) 1 14 -= = a ( ) ~ ,~ l L , , with , , %

li

defined similarly to u%i in (8) with t nA 12 1 - - a , replaced by t n a A 2 1 , - . In general the competing

confidence intervals are labeled in order by decreasing length, except that the length of

CI3 is longer than CI2 for nA =2 or 3.

The first case is equivalent to assuming that s pA

2 is “large” relative to var( ) p$ A and

leads to using the usual unconditional variance but with degrees of freedom equal to

nA -1. In most practical problems this seems reasonable since s pA

2 is an unknown

constant and var( ) p$ A is Op n ( ) A . It is interesting to note that the heuristic development

in Appendix C also yields CIs with critical values dependent on s. The second case is

motivated by empirical evidence for its advantage at nA =2 and 3, and the consistency to

the standard normal method as nA becomes large. For this case,

s p

A A A A

A A

n p p

n

p p 2

2

4

1

4 = 1 - = - $ $ $ $ b g b g. The third interval is based on the standard empirical

Bayes assumption regarding the prior variance, namely, s pA A A p pn 2 = - $ $ b g 1 . The last

confidence interval is based on the assumption that pA is essentially degenerate at p$ A .

A small empirical study, using an artificial population, suggested that

1. Standard confidence intervals using the usual variance estimate and normal 

quantiles can give very low coverage. The worst cases occurred when γ was a half, 

for several values of p.

2. The strictly conditional intervals (i.e., CI4 ) using the conditional variance can 

give abominable coverage, especially when γ is large. That is, confidence intervals 

based on “large” values of y A gave very poor results.
10

3. The use of the standard variance estimate with degrees of freedom based on the 

number of sample units in the domain (i.e., CI1 ) give conservative coverage.

4. The Empirical Bayes estimates corresponding to y A = 1 and y A

A n = 4 give 

conservative coverage with narrower mean interval length than the CI1 . However, 

the differences were not very great and in proceeding to the more complex stratified

random sampling case, we focus our attention on variants of CI1 .

3. THE CASE OF STRATIFIED RANDOM SAMPLING

3.1 Definitions and Notation

Assume there are K strata and, where appropriate, terms previously defined have

corresponding stratum level definitions. For example, nk is the sample size and nAk is the

number of sample elements in A for the kth stratum. Thus, a natural estimator for the subdomain total T x Np A ki k i A

K

k Ak k

K

= = Œ Ak Â Â = = 1 1 Â m is

$ $ T T Np$ $ A Ak k

K

k Ak k

K

= = Ak Â Â = = 1 1 m . (9)

It is straightforward to verify that

ET T N p p A A A A k Ak Ak Ak A k

K $ - = -∫ $ , $ % Â = di b g p p m m 1

 and

var $T T Np $ , $ $ % nN p

n

A A A A k Ak

Ak

Ak

k B k Ak

Ak

k

k B -= = ∫ Â Â Œ Œ A d i p p 2 2

2

2

2

2

1 1

s s

s ,

where p$ $$ $ A A A AK = pp p 1 2 L , pA A A AK = pp p 1 2 L and B kn k K 1 Ak = ≥ ££ m r 1 1 and .
11

3.2 General Methodology for Confidence Intervals

As before, $

$

$

q =

T T -

s

A A

TA

d i

 has a distribution function H( ) AA A A p p $ ;,, m s 2 , where p$A

is the known realization of a random vector and pA , m A and s A

2 are unknown vectors of

parameters. The upper critical value required to construct a conditional equal tailed

a1 100 - · a f % CI for TA, is denoted by c c u u AA ” ab,p p $ , g and the lower critical value is

denoted by c c l l ” a A A b,p p $ , g; the dependence on m A and s A

2 being suppressed. As in the

previous section, the problem is that the critical values depend on the unknown vector of

parameters pA so the situation is basically the same as for simple random sampling,

although as we shall note, there are complications of a new sort in dealing with the

stratified case.

Analogous to the approach in Section 2.2, if it is assumed that pA is the realization of

a random vector, then we can write Hx x AA AA $ , Pr cpp pp h= £ oq$ $ , t and

f f A A AA bp p pp $ ; g c= $ h. It follows that

Pr $ $ ;$

$ ; $ , $ ,

q£ =

= zz

x Fx

h Hx f g d

A A

A

A A AA A A

p p

p

p p pp p p

o t b g

b g

1

L b gc hb g

where h f gd A AA A A bp pp p p $ $ g c = zzL hbg and gbpA g is the joint pdf of pA . The upper and

lower critical values for an equal tailed a1 100 - · a f % conditional (on p$A ) CI for TA are

cc F uu A A ( ) ( ) * * ” =- , $ $ - a a p p 1 2 and cc F l l( )A A ( ) * * ” =- - , $ $ - a a p p 1 1 2.

Because the samples are selected independently from each stratum we have

f fp p A A k Ak Ak k

K

cp p $ h= =1c$ h and, as a consequence of our within stratum sampling

scheme, n pk Ak $ has a binomial distribution Bn p k Ak b, g. It is reasonable to assume that the

p kK Ak l 1 £ £ q are jointly independent so that g gp A k Ak k

K

bp g= =1b g which implies

f g fp p gp A A A k Ak Ak k Ak k

K

cpp p $ hbg c = =1 $ hb g and h f p p g p dp A k Ak Ak k Ak Ak k

K

bp$ g c = =1z$ hb g .
12

In what follows, we assume that the prior distribution of pAk is N p p Ak Ak m s, d 2 i and for the

empirical Bayes approach, we used mp Ak Ak = p

$ and, analogously to the case of simple

random sampling, we define y

s

Ak

Ak Ak k

p

p pn

Ak

= $ $ 1-

2

b g .

3.3 Confidence Intervals Under Normal Assumptions

Assume that within sub-domain A for the kth stratum the xki are distributed

N m s Ak Ak , c 2 h and the xki are distributed independently from stratum to stratum, then

T T $ $ , dA A AA - ip p is distributed N A A

m s % , % c 2h. It follows that T T $ % $ , dA A AA A - is p p is

distributed N A A bm s % % ,1g. Furthermore, based on the empirical Bayes approach specified

in the previous section, it is straightforward to extend the result in Appendix A to the case

of stratified random sampling and it then follows that m s % %$ A AA p is distributed

N 0 AA A

2 d, varcm s %$ % p h i, where var % $ $ $ m m

y AA k k B Ak

Ak Ak

k Ak

N p p

n

c hp

b g b g = -

+  ̨

2 2

1

1

1 .

Let B kn k K 2 Ak = ‡ ££ m 2 1 and r and assume that B2 „  ̆ . Then, for k B  ̨ 2 , the

terms xki Ak

i

n

Ak A A

Ak

- L

N

M O

Q

P =

b gm s $ $ , 2

1

2 p p are distributed independently as c 2 n 1 bAk - g. Next,

let mwk B k  ̨ 2r be non-negative constants with wk k B ̨ > 20 and $

$

s

m

Ak

ki Ak

i

n

Ak

x

n

Ak

2

2

1

1 =

-

-

=

b g

b g .

Then, based on the usual Satterthwaite (1946) two moment approximation, the

conditional random variable

w n

c

k Ak Ak Ak k B

A A

L -

N

M

M

O

Q

P

P  ̨ 1 2 2

2

b gc$ h

$ , s s

p p

is distributed approximately as a c n2

af, where c wn wn k Ak k B k Ak k B =- -  ̨  ̨

2 1 1 2 2

b g b g

and n =- -  ̨  ̨ wn wn k Ak k B k Ak k B

1 1 2 2

2

e jb b g 2 g. It follows that, conditional on p$A , the

random variable
13

$

$ var % $ %

$

$ var % $ %

$

q

m s

ss n

m s

s s

= - +

-

= - +

- -  ̨  ̨ ̨

T T

wn c

T T

wn wn

A A AA A

k Ak Ak Ak k B

A A AA A

k Ak Ak Ak k B k Ak k B

d i c h

b gc h

d i c h

b gch b g

p p 2

2 2

2

2 2 1 11 2 2 2

is distributed as a central t with n degrees of freedom. Analogous to the conditional

upper confidence limit defined in (6), we define

( )

( )( ) ( )

( )

( ) u T

w n N p

n

p

w n

t A

k Ak Ak Ak

k B

k Ak Ak

k

Ak Ak Ak

Ak k B

k B k Ak

w = +

- - ++

Ł ł +

-

Ł ł

 ̨  ̨

 ̨

-

$

$ $ $

,

1

1 1

1

1

2 2

2

2 2 2

1

2

1 2

1 2

s s

s g y

y

a n (10)

with the lower confidence limit defined in a similar manner. When the y Ak are near zero

we have (approximately)

( )

( )( ) ( ) ( )

( ) u T

w n N p

n

p

w n

t A

k Ak Ak Ak

k B

k Ak Ak

k

Ak Ak

k B

k B k Ak

w @ +

- -+

-

Ł ł

 ̨  ̨

 ̨

-

$

$ $ $

,

1 11

1

2 2

2

2 2

2

1

2

1 2

1 2

s s

s

g

a n (11)

We temporarily let Q= - + ( ) ( )  ̨

N p

n

p k Ak Ak

k

Ak Ak

k B

2 2

2

1

1 1 $ $

s

g and consider two alternatives for

specifying the wk :

Alternative 1 Let 

( )( ) ( ) ( ) w ( ) ( ) N p

n n

p N p

n n

p k

k Ak Ak

k Ak

Ak Ak

k Ak

k Ak

= Ak Ak Ak

- - += - - +

2 2

2

2

2 2

1

1 1

1

1 $ $ $ $

s

g ms .

Then, from equation (11),

uTt 1( ) A

1 2

1 2 1 w = + -

$ Q a n, . (12)
14

where the parameter n1 is referred to as the weighted degrees of freedom.

For k B  ̨ 2 , we use s$Ak

2 to estimate s Ak2 , m s $ $ Ak Ak Ak n 2 2 - to estimate m Ak2 and

( ) $ ( ) ( )( ) $

w $ $ $ $ N p

n n

n p k

k Ak

k Ak

= Ak Ak Ak Ak Ak

- - -+

2

22 2

1

ms s 1 to estimate wk . We can then estimate the

weighted degrees of freedom with n$ $ $ 1

2

2 1 1 2 2

=- -  ̨  ̨ wn wn k Ak k B k Ak e jb b g k B g. We will

delay dealing with Q until after specifying the second alternative.

Alternative 2 Let wk =1 then from equation (11)

( )

( )( )

( ) u T

n

n

t A

Ak Ak Ak

k B

k B Ak

2

1 2

2 2

2

2

1 2

1 2 2

1

1

w = +

-

- Ł ł

 ̨

 ̨

-

$

$

Q ,

s s

a n (13)

with unweighted degrees of freedom n2 = nAk k B -  ̨ 1 2

b g. If we estimate s Ak

2 with s$Ak2

then equation (13) simplifies to

uTt 2( ) A

1 2 w 1 2 2 = + -

$ Q a n, (14)

It is straightforward to show that n2 1 ‡n$, hence, for any specified value of Q , the length

of the confidence intervals under alternative 2 is less than or equal the length under

alternative 1. Using a different approach, Kott (1994) also recommends lowering degrees

of freedom using the Satterthwaite approximation. Similarly, Johnson and Rust (1993)

use the Satterthwaite approximation to get degrees of freedom corresponding to a

resampling variance estimator.
15

Addressing the problem of estimating Q , we have

( ) ( )

( ) ( )

( ) ( ) ( ) ( )

Q= - +

= -+

= -++ -+

 ̨

 ̨

 ̨ -  ̨

N p

n

p

N p

n

p

N p

n

p N p

n

p

k Ak Ak

k

Ak Ak

k B

k Ak

k

Ak Ak Ak

k B

k Ak

k

Ak Ak Ak

k Ak

k

Ak Ak Ak

kB B k B

2 2

2

1

2

2 2

1

2

2 2

2

2 2

1 2 2

1 1

1

1 1

$ $

$ $

$ $ $ $

s

g

m s

ms ms .

For kB B  ̨ -1 2 the estimator s$Ak

2 is not defined, however, it is straightforward to verify

that 1 1 2 22 2 b- £+ - £ pE n p E n $Ak Ak Ak Ak Ak Ak Ak Ak gm sm m $ b $ g $ . Therefore,

s

Np p

n

N p

n

pn p

N p

n

p N p

n

n n

a

k Ak Ak Ak

kB B k

k Ak

k

Ak Ak Ak Ak Ak

k B

k Ak

k

Ak Ak

k B

k Ak

k

Ak k Ak

k B

2

2 2 2

2 2

2

2

2

2

1

11 1

1 11 1

12 2

1 2

= - + -- + -

= - + +-

 ̨-  ̨

 ̨  ̨

$ $ $ $ $ $ $ $

$ $ $ $ $

b g d c hb b g gi

bg b g

m

s m

m s

 (15)

is an "under-estimate" for Q and

s N p

n

N p

n

p N p

n

n n b

k Ak

kB B k

Ak

k Ak

k

Ak Ak

k Ak

k

Ak k Ak

k B k B

2

2 2

2

2

2

2

2

1 2 1 2

= + - + +- 1 11 1  ̨-  ̨  ̨

$ $ $ $ $ $ m ms bg b$ g (16)

is an "over-estimate". Clearly, s s a b

2 2 £ with equality only when B B 1 2 = .

It can also be verified that in the case of stratified sampling, the standard variance

estimator for estimated population totals is

( ) ( )

s N s

n

N p

n

p N p

n

n

std k

k

k k B

k Ak

k k B

Ak Ak

k Ak

k

Ak Ak

k B

2 2

2

2

2

2

2

1

1 2 1

1

1

1 1

=

= - - + - -

 ̨

 ̨  ̨

$ $ $ $ m s$ .

 (17)
16

Note that sstd

2 and sa2

 are equal to terms of order nk

-1

, however, sa

2

 will tend to be smaller in

most practical situations.

These results imply that CIs of the form ( ) $T st A b – 1 2 - , $ a n1 will provide the highest level

of coverage while CIs of the form ( ) $T st A a – 1 2 -a n, 2 will provide the lowest level. Also, CIs

of the form ( ) $T st A std – 1 2 - , $ a n1 and (especially) ( ) $T st A std – 1 2 -a n, 2 have obvious computational

advantages. Several of these competing forms of CI were evaluated in an empirical study

which is reported in detail in Section 3.5.

3.4 The Case of Population Means and Medians

The results in the preceding sections can easily be extended to ratio estimators by the

standard linearization approach. By way of example, suppose we are interested in

estimating the average wage for workers in a particular occupation via a sample of

business establishments; here we let A consist of establishments with employees in the

occupation of interest. Let T W $( ) A and T M $( ) A be estimators of total wages (W) for

employees in occupation A and total number of employees (M) in occupation A as in

equation (9). We can write these estimators as T W cy $( ) A ii i

nA

= =1 and T M cm $( ) A ii i

nA

= =1 ,

where for the i

th establishment, ci is the sampling “weight”, mi is the total number of

employees in occupation A and yi is total wages for employees in occupation A. The ratio

estimator w TW T M AA A = $( ) $( ) is used to estimate the average wage (W WM A = ) for

employees in occupation A. It is straightforward to verify that the usual linear

approximation for the difference ( ) w W A A - is given by

l( ) w W cz AA i i

n

i

A

- = =1 , (18)

where z M y Wx i i Ai = - ( ) -1 . For stratified random sampling, we can re-label the sample
17

establishments to reflect stratification and then equation (18) can be written as

l( ) w W Nn z Z A A kk k ( ) K

ki i

n

k Ak

Ak K

-= = = == 1 11

$ .

This is of the same form as equation (9), so the results of Section 3.3 apply to l( ) w W A A -

under the appropriate normality assumptions on the zki .

Preliminary analytical investigation indicates that these results can also be extended

to the construction of CIs for population medians, or other percentiles, by the use of either

the Woodruff (1952) or the Francisco and Fuller (1991) techniques. Detailed

investigation of this extension is beyond the scope of the current paper.

3.5 The Empirical Study For Stratified Random Sampling

Results on coverage and mean interval length, from two simulation studies, both on

populations derived from a test sample of the Occupational Compensation Survey

Program (OCSP) conducted in 1991, are included in Tables 1-4. One population (the

"Small Population") took the sample itself as the population, with six non-certainty strata,

and one certainty stratum of 12 establishments. Repeated samples were taken from this

population at sizes n=36 and 60, corresponding to the choices nk =4 and nk =8. The

second population (the "Large Population") was constructed by expanding the sample data

through replication of establishments to achieve a population the size of the original

population; again there were six noncertainty and 1 certainty strata; for each stratum

samples were of the size of the actual sample. Domains are defined by the different

occupations of interest; only a fraction of establishments have workers in a particular

occupation, and lie in the corresponding domain.

In both cases sampling was without replacement, so finite population correction

factors were included (as appropriate) in the construction of the CIs. Also, the study was

limited to a concern with 95% coverage.
18

SMALL POPULATION: Table 1 for total wages and Table 2 for mean wages give

coverage and interval length, at two sample sizes nk =4 and nk =8, for 8 occupations, and

4 variance-degrees of freedom combinations: the standard variance estimator, sstd

2 , with

the standard normal z-quantile, and with the unweighted and weighted degrees of

freedom. Results are based on 500 runs. Occupations are ordered by increasing values of

the average value over runs of n2 . We note:

(1) Almost universally, coverage using the standard variance estimator and the standard

normal quantiles (infinite df) is poor.

(2) Coverage for the other interval types is far more satisfactory, in the main matching

nominal or conservative for the weighted degrees of freedom; as expected the unweighted

degrees of freedom tends to yield coverage a few points below the weighted degrees of

freedom coverage.

(3) Confidence intervals for means are better behaved on the whole than for totals. Two

occupations (1122, 4021) yield seriously low coverage for totals even with the improved

procedures; only 4021 gives poor coverage for means.

Interval lengths are taken relative to 2 4 · » 975 z. times the root mean square error of T

$

A

calculated over runs; when the distribution of T

$ is actually normal this ratio is 1.

(4) The relative interval length of the standard interval tends to be too small, that is, less

than 1.

(5) Interval length among the other variance-degrees of freedom combinations is largest

for sstd

2 with n$1 , and smallest for sstd2 with n2 . These differences can be appreciable; there

is a tradeoff between coverage and interval size.

(6) For a given interval type, the relative interval length tends to 1 as n2 increases.

LARGE POPULATION. Tables 3-4 give coverage and interval length for totals workers

and mean wage for the same four interval types, and a wider range of occupations,
19

ordered by average n2 . Results are based on 5000 runs. Here the interval lengths are

taken relative to the median interval length for the standard normal confidence interval.

We note:

(1) Results are consistent with those on the Small Population, in terms of the relative

coverage and interval sizes of the several interval types. The standard normal is

unsatisfactory for many occupations.

(2) Coverage using n$1 is less than 90% only in a small fraction of cases.

(3) There can be marked differences in interval length for the different interval types;

however, all ratios of interval length to 4·root mean square error tend to 1, as n2 gets

large.

(4) There are some differences in problem occupations from the Small Population Study;

for example, the coverage for 4021 is much improved, but 2911 has poor coverage,

especially for the mean, even with the non-standard intervals. These differences are

probably due to some differences in the way the populations were structured; in particular,

all certainty establishments in the original sample were treated as certainties in the Large

Population; this was not the case in the Small Population.

(5) In the main, coverage is better for means than for totals, but there are some obvious

exceptions, especially at low values of n2 .

4. SUMMARY AND CONCLUSIONS

From our theoretical investigation and the two simulation studies relying on OCSP data,

we draw the following conclusions:

1. Standard 95% confidence intervals for domain means or totals, when based on the

standard normal distribution and standard methods of variance estimation, tend to yield

less than actual 95% coverage. The extent of the deviance will vary with domain
20

(occupation in the simulation study), but can be quite considerable even when the sample

is large.

2. New nonstandard methods offer a sharp improvement, giving intervals with better

coverage, typically at or close to the nominal 95% coverage. These intervals tend to be

longer than the standard intervals. The increase in length will vary with domain, and will

depend on the particular method for CI construction that is adopted among those we have

considered. "Asymptotically", that is for "large sample domains", there will be little

difference from standard intervals.

3. The basic ideas behind these intervals are (1) conditioning on the amount of

information on the particular occupation, which, roughly speaking, is measured in terms of

the number of units in the sample that belong to the domain, and (2) An important

unknown is the fraction within each stratum of such units, and to handle this we put a

prior distribution on this unknown, reflective of the degree of our ignorance of it, an idea

we borrow from the Bayesians. However, the bottom line here is coverage probabilities.

4. The principal effect of these ideas is the abandonment, for purposes of CI construction,

of the standard normal quantiles (±1.96 for 95% coverage). These are replaced by

quantiles from the Student's t-distribution, with degrees of freedom determined from the

sample and varying with domain. If because of publication requirements or for other

reasons, there is need to report standard deviations rather than confidence intervals, then

we recommend reporting an effective standard deviation given by the length of the 95%

interval divided by twice 1.96.

5. The most likely candidate for estimate of variance, accompanying the new t-quantile, is

the standard estimate of variance. In most instances this should be quite satisfactory, so
21

that the only change will be in the introduction of the new degrees of freedom

methodology. However, we have considered alternatives to the standard variance

estimator, which in some instances may improve coverage or reduce the length of

confidence intervals.

6. An open question concerns what degree and type of collapsing of strata (if any) should

be used in the estimation of variances and of the degrees of freedom for the purpose of

confidence interval construction. In general, there will be a tradeoff: as strata are reduced

in number, the estimate of variance will tend to increase, but so will the degrees of

freedom (reducing the size of tn2 or tn$1 .) The answer to this question may well be

population specific, and experience of the population from past surveys useful.

REFERENCES

DORFMAN, A. and VALLIANT, R. (1993). Quantile Variance Estimators in Complex 

Surveys, American Statistical Association 1993 Proceedings of the Section of Survey 

Research Methods, to appear.

FRANCISCO, C.A. and FULLER, W. A. (1991). Quantile Estimation with a Complex 

Survey Design, Annals of Statistics, 39, 454-469.

JOHNSON, E.G. and RUST, K.F. (1993). Effective Degrees of Freedom for Variance 

Estimates from a Complex Sample Survey, American Statistical Association 1993 

Proceedings of the Section of Survey Research Methods, to appear.

KOTT, P. S. (1994). A Hypothesis Test of Linear Regression Coefficients with Survey 

Data, Survey Methodology, to appear.
22

SARNDAL, C-E, SWENSSON, B. and WRETMAN, J. (1992). Model Assisted Survey 

Sampling, New York: Springer-Verlag.

SATTERTHWAITE, F. (1946). An approximate Distribution of Estimates of Variance 

Components, Biometrics, 2, 110-114.

WOODRUFF, R.S. (1952). Confidence Intervals for Medians and other Position 

Measures, Journal of the American Statistical Association, 47, 635-646.
23

Appendix A

From the discussion in Section 2.2 we know that npA$ has a binomial distribution

Bi n pA b, g, hence, for p$ , A = 0 1 n, 2 n, , 1 L

( ) ( )

( )( ) ( ) ( )( ) fpp

n

np n p

p p A A

A A

A

npA

A

n pA $

$ $

$ $

= +

+ -+ - G -

G G

1

11 1

1 1

( )

( )

( )

( )( ) ( )

( ) ( )( ) ( )

( )

= +

+

+

+ -+ -

= +

G + - - +-

G

G

G G

n

n

n

np n p

p p

n

k p

A A

A

npA

A

n pA

pA A

1

2

2

11 1

1

1

1

1 1 1 11

$ $

$ $

$ .

For each (fixed) value of p$A , the function k p p A A$ bg is the pdf of a Beta distribution with

parameters w1 = + np 1 A$ and w 2 =- + n p 1 1 A b $ g . As both w1 and w 2 will be larger than

unity with high probability (at least in most real world situations), it is reasonable to

approximate k p p A A$ bg with a normal pdf having equivalent mean and variance. For the

Beta distribution in question the mean and variance are

np

n

p A

A

$ $

+

+

@ 1

2

 and

np n p

n n

p p

n

A A A A $ $ + -+ $ $

+ +

@

11 1 -

2 3

1

2

b gcb g h

a fa f

b g.

Thus, the approximation is k p

p pn

e p A

A A

p p

p pn

A

A A

A A

$

$

$ $

$ $ b g b g

b g b g @ -

- -

1 -

2 1

2

1

p

1

2 .

Assuming that pA is distributed N m s, c 2h it follows from the Bayes formula that the

posterior distribution is
24

( ) ( ) ( ) ( ) ( )

( )

( )

( )

h p p f p p g p f p p g p dp

ce

AA AA A AA A A

pA pA

pA pA n

pA

$$ $

$

$ $

=

@

- -

- + -

Ł ł

0

1

2

1

2

2

1

2

 

 ,

m

s

where 

( )

( )

( ) ( ) ( )

( )

( )

( ) ( )

c e dp p pn

p pn

e

pA pA

pA pA n

pA

A

A A

A A

pA

pA pA n

=

Ø

o

Œ

Œ

Œ

ø

ß

œ

œ

œ

@

- +

-

- -

- + -

Ł ł

- -

Ł ł - +

1

2

 

1

2

 $

$ $

$

$ $ $ $

$ $

2

1

2

2

0

1

1

2

2

1 1 2

2 1

m

s

m

s s

s p

 is the

normalizing constant.

Under the "empirical Bayes" assumption that m = p

$A and s 2 = - p pn $ $ 1 A A b g we have

hp p

p pn

e A A

A A

p p

p pn

A A

A A $

$ $

$

$ $ c h b g

b g

b g @ -

- -

-

F

H

G

I

K 1 J

2 12

2

1 2

p

1

2

.

If we drop the specific assumption regarding s 2, and let y = - p pn $ $ A A 1 s cb gh 2 then

p p A A$ is distributed normal with mean equal p$A and variance p p $ $

n

A A 1

1

-

+

b g b gy

. Under the

empirical Bayes variance assumption we have y =1.
25

Appendix B

Result: Assume W is distributed N c 0 2 c, h and, conditional on W w= , the random

variable T is distributed as a non-central t with n degrees of freedom and non-centrality

parameter w. Then, the unconditional distribution of T c2 +1 is central t with n degrees

of freedom.

Proof: First notice that T can be written as

T X W

S = +

2 n ,

where X is distributed as N( ) 0 1, , S2 is distributed as cn

2 , and XW S , , and 2

 are

mutually independent. Therefore, X XW c ¢=+ + ( ) 1 2 is distributed as N( ) 0 1, . As

X ¢ and S2 are independent, it follows by definition that

¢= + = ¢ T T

c

X

1 S 2 2 n

is distributed as tn .
26

Appendix C

We here make some observations on the use of na −1 degrees of freedom with the

standard statistic 

nTT

Ns

A A

1 2/ d$ − i

, corresponding to the empirical Bayes format with Y A = 0,

from a frequentist standpoint.

The question is why the t distribution with nA -1 degrees of freedom, which would

seem appropriate for use with a conditional distribution based on the domain size nA and

the within domain variance estimate s$A

2 , yields generally sound confidence intervals in

conjunction with the above unconditional statistic based on sample size n and sample

variance s2. We focus on 95% coverage and let t tnA * ;. = − b 1 975g. Also, assume m A is

positive and, to abbreviate notation, let m m ” A , s s 2 2 ” A , g g” A , p p ” A and p p $ $ ” A .

Consider Figure 1, which, for 500 samples of size 300 selected with replacement from

a population of size 3000 having p=0.03 and γ = 9, graphs the ratio R=

(T T $

A A − )/ Nst n * / m 1 2/ ragainst nA ; if the ratio R is less than 1, then TA is in the

corresponding interval estimate. We note (i) the intervals are well behaved, in fact

conservative in that more than 95% of R-values are less than 1, (ii) that as nA increases, R

increases from relatively large negative to relatively large positive, seeming to reflect

changes in the bias of T T $

A A − as nA changes. Furthermore, there is a "within nA "

variation, which in the main seems to increase as nA decreases.

We proceed to analyze R in light of this figure, attempting to get a handle on the bias

and spread, for each nA ≥ 2 (or, equivalently, the "across nA " and "within nA " variation

respectively.). Conditional on nA , the bias of the numerator ET T N p p A A d$ −= − i a$ fμ ≡ bp$.

Then R=

$

* $ $

$ $ $

/ /

/ TTb

t Np n

p

s

A A p − −

12 12

1 2

σ

σ

+

b

t Ns n

p$

/ * / 1 2 . We suggest that, for given nA , the first term

reflects the spread of R , the second the bias.
27

For moderate or large n, we have by (1) that sp p

n

n

2 22 A 1 1

=− + − $ $ a fμ σ $ $ , and

substituting this in both terms, we derive R

TTb

t Np n f A A p =

$ − −

* $ $

$

12 12 / / σ

+n pn

t n

g A

A

−

* 1 2/ , where

fp n = − +− A

−

1 11 2 1 2 $ $ / /m r a fγ b g and g

p nA

= − +−

R

S

T

U

V

W

%

$ $ /

/

γ

γ

2

2

1 2

a f 1 11 , for γ μσ %= $ and

γ μσ $ $$ = .

Consider the first factor in each term. By Section 2.3, 

$

$ $

$

/ /

TTb

Np n

A A p − −

12 12 σ

 is a standard

studentized statistic and so, divided by t*, will lie between -1 and 1, 95% of the time (this

would not be so if we used the conventional z-statistic, in place of t* or t with larger

degrees of freedom). The first factor of the second term is fixed for given nA . For given

n,p we ask how often nA will be such that this value will be large. The expression

n pn

n

A

A

−

1 2/ = $ $

p p p

n

a f- is bounded above by the conventional binomial statistic, which for

np moderate has an approximate z-distribution. But in the situation we are concerned

with, np is typically small, and the division by t* instead of z.975 is an important safety

factor. Values of n pn

t n

A

A

−

* 1 2/ and n pnz n

A

A

−

.

/

975

1 2 and the probabilities with which they arise, were

tabulated for n = 50, 100, 500, and 1000 and for a range of values of p, for example, for

n=50, p= 0.01, .02, .04, .06, .12, .18. It was found that large (absolute) values of n pn

t n

A

A

−

* 1 2/

(say bigger than 0.8) have extremely small probability, with the most vulnerable situation

being the possibility of getting nA =3, 4, or 5, when np is about 12 or 15. In particular, the

case when nA =2 is not worrisome because of the large value of t*. By contrast, large

values of n pn

z n

A

A

−

.

/

975

1 2 were not so improbable.

These considerations suggest the bias will be less than g (or 0.8g) with extremely high

probability, and the within nA variation will be less than f with probability 95%. Note that

for given nA , f and g are functions of γ γ $, % both estimating γ. For the moment consider

them as functions of γ itself, ignoring their stochastic variation. Then it is easy to see that

f is monotonic down, with a maximum at γ=0 of 1 1 1 2− −/ /n b A g and asymptoting to zero as 
28

γ increases [for example at γ=3 (the approximate practical lower bound we found in wage

data), for nA =2 (worst case) and p$ negligibly small, f=0.32]. On the other hand, g is

monotonic increasing, with a value of 0 at γ=0 and asymptoting to 1 1 1 2 − ≈ − $ / a pf . Thus

the larger γ, the more pronounced the across nA variation and the less the within nA

variation, and vice versa. At γ=0, the bias term is zero. Clearly f+g is bounded by a small

number, achieved when γ is between 0 and 1. In fact, squaring f+g and taking the

derivative, one finds that to maximize f+g, we have g» -1 1/ nA . Table 1 gives values of 

γ yielding maximum value of f+g, and the values so yielded, for small values of nA . Table

C-1 suggests the worst case for coverage occurs for γ about 0.65.

Table C-1. Maximum Values of f+g, for given γ

nA 2 3 4 5 6 7 8 9 10

γ .5 .67 .75 .8 .83 .86 .88 .89 .9

f 1.15 .95 .87 .83 .81 .79 .78 .77 .76

g .58 .63 .65 .67 .67 .68 .68 .69 .69

f+g 1.73 1.58 1.53 1.50 1.48 1.47 1.46 1.46 1.45

Note: As nA increases, both f and g approach 2 2/

This discussion ignored the fact that not γ, but $ ~ g g and appear in f and g. Note that

were it not for the fact that there are two estimators of γ, the above argument goes

through, with same bounds on f+g, etc., with γ

$ for example replacing γ. However, γ

% and

γ

$ should typically be close. In particular, it is easy to see that γ

%-γ

$ is distributed as

t nA / 1 2/ , where t has a t-distribution with nA -1 degrees of freedom.
29

An incidental observation is that for large γ the within nA distribution is clearly skewed

downward (see Figure 2). The distribution of the ratio γ

$ is skewed positive, so that f is

skewed negatively, accounting for the effect.