A COMPARISON OF ESTIMATORS FOR THE MEAN OF A FINITE POPULATION, BASED ON A

SYSTEMATIC SAMPLE

Larry Huff, Diem-Tran Kratzke, Gordon Mikkelson, Sandra West

Diem-Tran Kratzke, 2 Massachusetts Ave, N.E. Washington, DC 20212

KEY WORDS: Response Analysis Survey, Systematic

Sampling, Model Based Estimator

1. Introduction

 In this paper we empirically compare several mean

estimators for a finite population based on a systematic

sample. This research began with a quality improvement

project for two Bureau of Labor Statistics’ establishment

programs that collect monthly employment data: the

Covered Employment and Wages (ES-202) program and

the Current Employment Statistics (CES) survey. The

ES-202 program is compiled using data from quarterly

reports of business establishments that are covered under

the Unemployment Insurance laws in the United States.

The CES survey collects monthly employment from a

voluntary sample of business firms and uses the ES-202

universe employment data to annually adjust its industry

employment totals. A Response Analysis Survey (RAS)

was conducted in order to determine the comparability

and accuracy of employment data reported to these two

programs. Each sample unit was asked several questions

pertaining to their response practices for both of these

programs. The samples were selected from among the

CES reporters of ten participating states. The sample

consisted of four panels that were selected approximately

three months apart. The panel samples were selected

with probability proportional to a measure of size based

on size of firm and percent difference between reported

employment to the CES and ES-202 programs.

 Various characteristics of the population are estimated

based on their responses to four specific RAS questions.

Alternative mean estimators and their estimated standard

errors are compared in order to determine the most

appropriate estimation techniques. We compare the

estimators that treat all four panels separately, as well as

estimators that combine the four panels.

 The general sampling design issues are discussed in

Section 2. Background information about the RAS,

including a description of the population and sample

design, is given in Section 3. The estimation techniques

and specific estimators tested are presented in Section 4.

Section 5 provides the empirical results. The conclusions

of the study are summarized in Section 6.

2. General Description

j n = 1,..., . Let Ni j, denote the number of po

elements that are in the (i, j) cell, where N

i

m

j

n

= = =

∑∑

1

 An element in the (i, j) cell has a probability 

selected proportional to ri times cj . That

probability of the selection of element k, given it 

(i, j) cell, is z rc N rc kij i j ij i j

i

m

j

n

,, , =⋅ ⋅⋅ = =

∑∑

1 1

.

 A sample of size nt elements is drawn, w

subscript t denotes the time period the sample i

Let π kij , , denote the probability that the k th elem

the sample. A systematic sample is drawn 

method that satisfies π kij t kij n z ,, ,, = . Thus

π kij t i j ij i j

i

m

j

n

nr c N r c ,, , = ⋅ ⋅⋅ = =

∑∑

1 1

.

 Let Y denote the variable of interest, wh

binomial variable that takes on the values 0 and

unknown probabilities 1- p and p respectivel

objective of the study is to estimate p . Let Yk i, ,

the value of Y for the k th element in the (i, j) cel

there are N elements, the value of p is:

p = Ykij

k

N

i

m

j

n i j

, , N

,

= = =

∑∑∑

1 1 1

.

 Since we only know the value of Y for the

elements we need an estimator for p . One possib

$ ,, ,,

,

p wy N kij kij

k

n

i

m

j

n i j

= = = =

∑∑∑

1 1 1

, (1)

where ni j, is a random variable denoting the nu

sample elements in the (i, j) cell, and wki j , , d

weight, for which there are a number of possibiliti

possibility is the sampling weight; that is,

wki j , , = 1

1 1

π kij ij i j

i

m

j

n

,, , N rc n rc ti j = ⋅⋅ ⋅⋅ [ = =

∑∑

resulting in the estimator:

$ ( ) , ,

,

p

K

N n rc y s

t

i j ki j

k

n

i

m

j

n i j

= = = =

∑∑ 1 ∑

1 1 1

, (2)

h K N

n m

∑∑
 Another possibility is a weight of 1, which results in:

$ , ,

,

p yn ki j

k

n

i

m

j

n

t

i j

1

1 1 1

= = = =

∑∑∑ . (4)

 Next consider estimators from a model based approach.

Let Z rc kij , , i j

* = and consider the model:

Y Z kij , , i j, ki j ki j , ,

* = + β ε , , , where ε σ δ

ki j ki j ,, ,, N Z* ~ (, ( )) 0 2 .

 Note that if there is a relationship between Zki j , ,

* and Y

(or some function of Y ) then the population could be

ordered on Zki j , ,

* prior to sample selection.

 We have $ $ ,, , ,,

* Y Z kij ij kij = β as the fitted model. One way

to estimate the total is by taking the sum of the responses

from the sampled elements plus the sum of the responses

from the non-sampled elements, which is estimated by

$

Ykij , , . Thus the mean estimator is:

$ $ *

,, ,, p y yN ki j ki j

i k S k S

m

j

n

= + 



 



 



 



 = = ∈ ∉

∑∑ ∑ ∑

1 1

, (5)

where S denotes the sample.

 The choice of δ and the portion of the data the model is

applied to will determine βi j, . First consider δ =1 and

fit the model separately for each (i, j) cell, then

$ ,

, ,

, ,

*

, ,

,

βi j

ki j

k S

ki j

k S

kij

k S

ij i j

y

Z

y

n rc = = ∈

∈

∈

∑

∑

∑ ,

and the inner sum becomes:

y y y N n rc

N

n

y

kij kij

k S k S

kij ij ij ij i j

k S

i j

i j

kij

k S

,, ,, ,, , , ,

,

,

, ,

$ ( )($ + ) 



 



 = +−

=

∈ ∈ ∉

∈

∑ ∑ ∑

∑

β

.

 In this situation $ * p becomes

$ ( ) *

, , ,, p Nn y N i j i j ki j

i k S

m

j

n

= 



 



 



 



 = = ∈

∑∑ ∑

1 1

, (6)

which is the same as p$ p .

 The same result, p$ p , is obtained under the model with

any value for δ . Note that within an (i, j) cell, Zki j , ,

* is the

same for all k within the cell.

 Now consider the situation where all the terms are

estimated; that is,

$ $ *

, ,

,

p yN kij

k

N

i

m

j

n i j

= = = =

∑∑∑

1 1 1

,

and use the model with δ = 2 and β = β for all (i j)

which is the same as p$s.

 If the model considered is :

Yki j ki j ,, ,, = + α ε , where ε σ ki j , , ~ (, ) N 0 2

then $ , ,

,

α = = = =

∑∑∑y n kij

k

n

i

m

j

n

t

i j

1 1 1

 and hence,

$ $ *

, ,

,

p yN kij

k

N

i

m

j

n i j

= = = =

∑∑∑

1 1 1

 = NN yki j

k

n

i

m

j

n i j

$ , ,

,

α = = = =

∑∑∑

1 1 1

which is the same as p$1.

 All the above estimators of p assume that N is

In practice this is not necessarily true.

 Consider the following model where it is assum

δ = 0 , and the independent variable is equal to on

cases. That is,

Ykij ij kij ,, , ,, = + β ε , where ε σ ki j , , ~ (, N 0 )

2

$ $ Ykij ij ,, , = β , where $ , ,, ,,

, ,

βij kij kij

k

n

k

n

w y

ij i j

= = =

∑ ∑

1

 The weights wki j , , serve the purpose of makin

design consistent estimator of the population re

coefficient βi j, .

 When the total is estimated by:

$ $ $ ,, , ,

, ,

Ty N kij

k

N

i

m

j

n

i j

k

N

i

m

j

n

i j

i

m

j

ij ij n

= == = = = = = = = =

∑ ∑ ∑∑ ∑ ∑ ∑ ∑

1 1 1 1 1 1 1 1

β

then the estimator of the mean becomes:

$ $ * $

, ,

, ,, ,,

,

p

T

N N

N

N wy

N ij ij

i

m

j

n ij kij ki j

k

n

i

m

j

n i j

== = = =

= = =

∑∑

∑∑ ∑ 1

1 1

1 1 1 β

 If the (i, j) cells are formed into h adjustment c

above formulae is the same as 2c and 3c in Se

Instead of modeling in each cell, as above, 

modeling over the entire data set. In this case 

which is estimated by:

$ ,, ,, ,,

, ,

β = = = = = = =

∑ ∑ ∑∑wy w kij kij ∑∑

k

n

i

m

j

n

ki j

k

n

i

m

j

ij ij n

1 1 1 1 1 1

,

which leads to the estimator

$ * $ $

, ,, ,,

,

p N N wy i j

i

m

j

n

kij kij

k

n

i

m

j

n

i

m

j

n i j

= == = == = = =

∑ ∑ ∑ ∑∑ ∑∑ 1

1 11 1 1 1

β β

 If the (i, j) cells are formed into h adjustment c

above formula is the same as 2a and 3a in Se

Note that the estimators in (9) and (10) will be a n n i, j i, j
programs. The Employment and Wages (ES-202)

program is compiled using Unemployment Insurance (UI)

reports which virtually all businesses must file quarterly

with their State. The Current Employment Statistics

(CES) program is a monthly survey of nearly 400,000

employers. The CES employment estimates are available

approximately one month after the reference month while

the ES-202 universe employment data are available five

months after the reference quarter. The ES-202 universe

employment counts are used to adjust the CES

employment estimates on an annual basis.

 The employment data collected in these two programs

are conceptually the same for all but a few employers

which have employees who are exempt from the State UI

laws. Many of the reported employment to the two

programs are different, and some are substantially

different. These reported employment differences were

the motivation for a Response Analysis Survey (RAS) of

about 8,000 CES reporters in ten cooperating states. The

RAS was designed to determine the causes of reporting

differences in the two programs and to evaluate the

overall quality of collected employment..

3.2 Population and Sample Design

 Conceptually, the population under study consisted of

all sample units reporting to the CES survey, since they

are all covered under the UI laws. Practically, the

population excluded delinquent ES-202 reporters and

other categories of CES reporters such as those with

special reporting arrangements.

 The requirements for the RAS sample design included

both reporters with reporting differences in the two

programs and reporters with no reporting differences.

Since most of the reported employment from the two

programs are equal or nearly equal and most of the CES

sample units are of small and moderate sizes, an equal

probability selection design would result in selecting very

few CES reporters with reporting differences and mostly

small and moderately sized employers. One method of

satisfying the requirements of oversampling large CES

reporters and those with reported employment differences

was to assign to each unit a measure of size which

increases as the reporting size increases and also as the

reported employment difference increases. Each

population unit was assigned an employment size class

(from 1 to 9) and an employment difference size class

(from 1 to 6) based on the percentage difference between

the reported CES and UI employment. This percentage

difference was calculated by dividing the average absolute

employment difference by the average CES employment

ordered by this measure of size and 

systematically.

 Since it takes each state about a year to interv

sample units, the sample was selected in four pane

three months apart to insure that we have the mo

data available at the time of interview. The mea

size were recalculated using updated info

immediately prior to the selection of each panel.

4. Estimation

 Due to the movement of establishments into an

the CES survey, the population at each panel is n

The inference population is established across 

panels by taking the union of all four panel pop

Estimation for the inference population could 

separately for each panel and averaged across

Another alternative is to combine the four pan

consider that all sample units were from one p

base the estimation on the combined panel.

 Our objective is to determine an estimator

inference population mean, Y (which was referr

p in section 2), whose population size is N .

4.1 Non Response Adjustment

 The weighting approach, in which the sampling

for responding units are inflated by dividing 

estimates of the probability of response is used t

unit nonresponse. Every population unit is ass

have a non-zero probability of responding if 

The simplest nonresponse model assumes 

population units have the same probability of res

and that data are missing at random through

population. Another approach is to classify sam

into adjustment cells. This approach assumes 

response rate is different from cell to cell and that

missing at random within the adjustment cells.

We allow that large establishments and establ

with large differences between the reported emp

in the two programs have a different response r

smaller establishments and establishments wit

differences. The adjustment cell will be percent d

(PD) by size where:

PD Class Percent Diff. Size Class Employment

1 [0,5) 1 0 - 09

2 [5,20) 2 10 - 19

3 [20,∞) 3 20 - 49

4 50 - 99

5 ≥100

 Let,

N be the inference population size
nht be the sample size in adjustment cell h in panel t ,

πkht be the probability of selection for unit k in

adjustment cell h in panel t ,

θkht be the response probability for unit k in adjustment

cell h in panel t ,

ykht be the characteristic measured for unit k in

adjustment cell h in panel t .

 Since θ is not known, it will be estimated by $θ , the

observed response rate. In the adjustment cell model, the

response rate is the same for each establishment in an

adjustment cell, so the subscript k will be dropped. In

the simple model, the response rate is computed for the

whole population and both subscripts k and h will be

dropped. When the summation is not over adjustment

cells, the subscript h will be dropped from π and y .

4.2 Estimators

1. Unweighted Estimator (corresponding to p$1 )

 a. Simple Model: At each panel t , the unweighted

estimator for the simple model is

$ ( ) $ Y y n noob t kt

k

= − ∑ t tt θ ,

where the summation is over the respondents in panel t .

 b. Adjustment Cell Model: At each panel t , the

unweighted estimator for the adjustment cell model is

$ (

$ Y y n noob )( ) t

h k ht

= ⋅− ∑∑ kht t t

1

θ ,

where the outside summation is over adjustment cells and

the inside summation is over respondents in adjustment

cell h in panel t .

2. Weighted Estimator by Separate Panel

 a. Simple Model (corresponding to $ * p equation (10)):

At each panel t , the ratio mean estimator for the simple

model is

$

Y ( )() y t kt

kt k kt k

= ∑ ∑ π π

1 ,

where the summation is over the respondents in panel t .

 b. Adjustment Cell Model: At each panel t , the ratio

mean estimator for the adjustment cell model is:

$ ( $ ) ( $ Y ) y t

h k ht

kht

kht h k ht kht

=⋅ ⋅ ∑ ∑ ∑ ∑ 1 11

θ π θ π ,

where the outside summation is over the adjustment cells

and the inside summation is over the respondents within

adjustment cell h in panel t .

 c. Post-stratified: (corresponding to $ * p equation (9)):

At each panel t , the post-stratified estimator for the

simple model is

3. Weighted Estimator by Combined Panel

 Alternatively, the four panels of sample u

combined in one single panel and the expansion e

is applied to this single panel. When the units

together in one panel, the inverse of the proba

selection is no longer appropriate. The expansio

will be used. The subscript t will be dropped

mean in the following formulas.

 a. Simple Model (corresponding to $ * p equatio

The combined-panel estimator for the simple mod

$

Y ( ) () N

n

y

N

n

h

h k h

kht

h

h k h

= ⋅ ∑ ∑ ∑ ∑ ,

where the outside summation is over adjustment 

the inside summation is over respondents

adjustment cell h in panel t .

 b. Adjustment Cell Model: The combin

estimator for the adjustment cell model is

$ ( $ ) ( $ Y ) N

n

y

N

n

h

h k h h

kht

h

h k h h

= ⋅⋅ ⋅ ∑ ∑ ∑ ∑ 1 1

θ θ

where the outside summation is over adjustment 

the inside summation is over respondents

adjustment cell h in the combined panel.

 c. Post-stratified: (corresponding to $ * p equat

The post-stratified estimator for the combined pan

$

Y

N

N

N

n

y

N

n

h

h

h

kht

k

h

k h

h

= ⋅

∑ ⋅

∑ ∑ 1 ,

where the outside summation is over the adjustm

and the inside summation is over respondent

adjustment cell h in the combined panel.

4.3 Variance Estimator

 The data analysis software, SUDAAN, was 

facilitate the estimation of variances whic

calculated via Taylor Series Linearization method

population correction factors were ignored..

5. Empirical Results

 The RAS study involved ten states with appro

800 sample units being interviewed in each state.

of the interviews were completed by the time th

began. We decided to use data from the three st

had the highest number of completed interviews 

first two panels: Florida with 334 interviews, Ne

with 316, and Oregon with 307. The RAS quest

consists of more than 30 questions for each of 

and ES-202 respondents. Responses to four
time period is the pay period that includes the 12th of the

month. The correct method of counting employment

involves an unduplicated count of individuals who worked

or received a check or other form of payment during the

pay period. The content component refers to the kind of

employees to be included in the employment count. For

the ES-202 program, it is based on who is covered by the

state Unemployment Insurance laws. The content

component may vary slightly between the CES and ES202 programs. The following four questions were used in

the empirical comparisons:

• Q10: Do you use the same pay period for all your

employees? (timing)

• Q14: Is the employment figure you use for the monthly

BLS report obtained from your payroll system?

(method)

• Q29: Can [the employment count] include a person

more than once? (content)

• Q31: What time period does this employment count

represent? (timing)

 Estimates for Q10 and Q14 represent the proportions of

firms that answer ‘yes’ to these two questions. Estimates

for Q29 represent the proportions of firms that answer

‘no’ to this question. Estimates for Q31 represent the

proportions of firms that report employment for the pay

period that includes the 12th of the month.

 Results from the three states are similar. Only results

from Florida will be presented. The results for Florida are

shown in Tables 1-3 in the Appendix. The estimates and

their standard errors are given in Table 1. The five types

of estimates by panel are computed by taking the average

of the two panel estimates. The associated standard error

is the standard error of the average.

 For each question and each estimation method, there

was not much difference in whether the nonresponse

correction was done globally or by strata. That is, for

each question, the estimates by adjustment cells or by

simple non response methods are similar. Thus, the

global nonresponse correction is acceptable.

 The differences between the CES and ES-202 estimates

are shown in Table 2. For Q10, Q29, and Q31, all

estimation methods produced positive differences.

Similarly, all estimation methods produced negative

differences for Q14. The unweighted estimates produced

the largest differences for all four questions. The

weighted estimates by combined panel produced the

smallest differences for three of the four questions. These

differences are similar across estimation methods.

The maximum and minimum values from among the

the combined panel adjustment cell model each p

three of the maximum estimates. The only es

procedure that did not produce any maxim

minimum estimate was the combined panel sim

method. The difference between the maxim

minimum values for each question is relatively sm

6. Conclusion

 All methods produced similar standard err

general, the unweighted methods produced small

estimates than the other methods. However, du

potential bias inherent in these methods, we 

recommend them. The remaining methods p

very similar results, whether or not we used th

nonresponse correction and whether or not we c

estimates by panel. As Table 2 and Table 3 sh

differences between the program responses are

across methods, and the difference betwe

maximum and the minimum estimates within a r

is relatively small. Based on the simpl

implementation, the simple ratio method by 

recommended. The simplicity comes from the 

the probability of selection is readily available 

nonresponse adjustment factor need not be calcul

addition, the underlying model for this estim

intuitive. The model and resulting estimator, 

given in Section 2 equation (10), is very sim

basically assumes that the value of the dependent

for a particular unit is equal to the mean over th

data set plus a random noise. By modeling over t

data set, an outlier in any one cell will not have 

influence on the estimate. The collected data sh

evidence that units in different cells have 

response rates.

References

Cochran, W.G. (1977), Sampling Technique, Joh

& Sons, Inc.

Cox, B.G., Binder, D.A., Chinnappa, B.N., Chris

A., Colledge, M.J., and Kott, P.S. (1995), B

Survey Methods, John Wiley & Sons, Inc.

Deville, J.C., Sarndal, C.E., and Sautory, O. 

“Generalized Raking Procedures in Survey Sam

Journal of the American Statistical Association, 

no. 423, Theory and Methods.

Kalton, G., and Maligalig, D.S. (1991), “A Com

of Methods of Weighting Adjustment for Nonres

Proceedings of the Bureau of Census Annual R

Conference.

Lessler, J.T. and Kalsbeek, W.D. (1992), Nons

Error in Surveys John Wiley & Sons Inc
Force Survey,” Social Survey Methods Division,

Statistics Canada, Working Paper SSMD # 009E.
Appendix - Empirical Results Using Florida Data

Table 1. Estimates And Their Standard Errors (in parentheses)

 Timing Method Content Timing

Q10 Q14 Q29 Q31

CES ES-202 CES ES-202 CES ES-202 CES ES-202

By Panel -- (Average of Panels 1 and 2)

Unweighted - Simple 82.84

(2.15)

79.52

(2.29)

89.77

(1.74)

95.69

(1.18)

90.02

(1.69)

68.21

(2.64)

85.01

(2.02)

60.52

(2.80)

Unweighted - Adj. Cell 82.92

(2.15)

79.46

(2.30)

89.69

(1.76)

95.60

(1.21)

90.21

(1.67)

68.30

(2.67)

84.90

(2.03)

60.21

(2.84)

Ratio - Simple 84.66

(2.57)

82.01

(2.67)

90.45

(1.83)

95.69

(1.34)

92.41

(1.57)

75.64

(2.61)

86.40

(2.23)

66.17

(3.38)

Ratio - Adj. Cell 84.70

(2.53)

81.95

(2.65)

90.35

(1.85)

95.61

(1.36)

92.49

(1.57)

75.42

(2.65)

86.33

(2.24)

65.82

(3.42)

Post- Stratified 84.59

(2.45)

81.79

(2.57)

90.94

(1.71)

95.74

(1.30)

92.49

(1.57)

75.18

(2.68)

86.70

(2.25)

65.22

(3.51)

Combined Panel

Ratio - Simple 85.30

(2.38)

82.49

(2.53)

89.91

(2.03)

95.59

(1.27)

90.93

(1.86)

73.44

(2.86)

82.45

(2.77)

63.79

(3.30)

Ratio - Adj. Cell 85.47

(2.39)

82.67

(2.55)

89.75

(2.06)

95.54

(1.29)

90.85

(1.90)

73.40

(2.90)

82.19

(2.82)

63.78

(3.34)

Post-Stratified 85.31

(2.44)

82.57

(2.58)

89.81

(2.07)

95.65

(1.26)

90.73

(1.94)

73.29

(2.92)

82.12

(.286)

63.62

(3.37)

Table 2. Differences Between Program Responses (CES and ES-202)

Timing Method Content Timing

Q10 Q14 Q29 Q31

By Panel -- (Average of Panels 1 and 2)

Unweighted - Simple 3.32 -5.92 21.81 24.49

Unweighted - Adj. Cell 3.46 -5.91 21.91 24.69

Ratio - Simple 2.65 -5.24 16.77 20.24

Ratio - Adj. Cell 2.76 -5.26 17.07 20.51

Post-Stratified 2.81 -4.80 17.32 21.48

Combined Panel

Ratio - Simple 2.81 -5.68 17.49 18.66

Ratio - Adj. Cell 2.80 -5.79 17.45 18.41

Post-Stratified 2.74 -5.84 17.44 18.50

Table 3. Differences Between Minimum And Maximum Estimates Across All 8 Estimation Techniques

Q10 Q14 Q29 Q31