MEASUREMENT OF DIFFERENT DESIGN EFFECTS

Shail Butani, Bureau of Labor Statistics; Lawrence Cahoon, Robert Fay, Donna Kostanich, Bureau of the Census

Shail Butani, 2 Massachusetts Ave., NE Room 4985, Washington, D.C. 20212

Key Words: Ratio Adjustment, Compositing, Seasonal

Adjustment

I. Introduction

The Current Population Survey (CPS), the monthly

labor force survey that the U.S. Bureau of the Census

conducts for the U.S. Bureau of Labor Statistics (BLS),

uses state probability samples (state-based design)

totaling about 60,000 households. In January 1994,

CPS interviewers began collecting data in an entirely

computer-assisted environment, using a new

questionnaire. Before introducing the new data

collection method, the Census Bureau and BLS tested

it on a separate national probability sample of 12,000

households. The results of this parallel survey (PS),

which ran from July 1992 through December 1993,

indicated that the CPS annual average unemployment

rate would have been 0.45 percentage point higher in

1993 had the new data collection method been used.

For a general discussion of the results, see the February

1994 Employment and Earnings (Bureau of Labor

Statistics, 1994, Vol. 41, No. 2); for a more detailed

discussion, see Polivka (1994).

The above difference of 0.45, however, provides an

estimate of only the overall effect on a 1993 annual

average basis. In this paper, we discuss the

measurement of design differences between the PS and

the CPS in January 1994 and subsequent months to

better understand the transition effects that were not

accounted for. We concentrate our analysis on the

effects of: 1) differences in the proportion of

households interviewed from centralized facilities; 2)

use of 1990-based population controls, adjusted for

census undercount, rather than 1980-based controls

formerly used; 3) differences in ratio adjustment

methods; 4) use of composite estimation to reduce

variance; 5) seasonal adjustment; and 6) 1990-based

population controls on variances.

In this paper, we first give a brief background on the

redesign of the CPS; discuss the design differences

between PS and CPS; present estimates of various

design differences; compare the expected estimates to

actual estimates for the early months of 1994, where

possible; summarize our results; and state a topic for

immediate future research. Due to space limitations,

we give only estimates of effects for design differences

on the national unemployment rate. For estimates of

differences in design effects on several demographic

groups and on employment-to-population ratio and

civilian labor force estimates, see Kostanich and

Cahoon (1994).

II. Redesign of the CPS

The main objective of the CPS is to provide a "reliable"

monthly estimate of the unemployment rate for the

nation. Other goals include producing national

monthly estimates on the number of persons employed,

unemployed, and not in the labor force by various

demographic groups and some sub-national estimates

for regions and states. The CPS began in 1940. The

questionnaire has remained essentially unchanged

since the last redesign in 1967. The current design

was undertaken to reflect changes which have taken

place in the workforce and structure of American

society, as well as to take advantage of changes and

improvements in survey methods, such as computerassisted data collection and questionnaire design that

utilizes the theories of cognitive psychology. Further

discussion of the questionnaire and mode of collection

changes incorporated into the CPS in January 1994 can

be found in three articles of the September 1993,

Monthly Labor Review, "Overhauling the Current

Population Survey."

III. Design Differences Between CPS and PS

General--The sample design of the PS was based on

the National Crime Victimization Survey. Like the

CPS, the PS was a multistage probability sample in

which large metropolitan areas are selected with

certainty, and smaller primary sampling units are

selected with probability proportional to size within

strata. For the PS, strata were defined within the four

regions, unlike the CPS strata that are defined within

state. The sample size of the PS is about one-fifth

(12,000 households) the size of the CPS. The

reference period for the PS was the same as that of the

CPS, that is, the week containing the 12th of the

month and the data collection occurred the same week

as the CPS. In the CPS, the households are

interviewed for 4 consecutive months, drop out of the

sample for the next 8 months, and are interviewed

again for 4 consecutive months. Although the PS had

the same rotation pattern as the CPS, it did not

completely achieve this pattern until September 1993

due to the short phase-in period (Miller 1994).

Consequently, we could not measure month-in-sample

effect in the PS. In the following paragraphs, we state

the design differences which we were able to measure

and Table 1 highlights these differences.

Centralization--During the testing period, data for

approximately 18 percent of the PS sample was

collected from one of the two centralized facilities

located in Hagerstown, Maryland, and Tucson,

Arizona, as compared to 9 percent for the CPS. All
data collected from the centralized facilities is

through computer assisted telephone interviewing

(CATI). In January 1994, the new CPS was to have

only 12.5 percent of the sample collected from the

centralized facilities. As described below, test results

indicated that centralization yields a higher

unemployment rate. Thus, we measured what the

effect would have been if we had only 12.5 percent of

the PS sample interviewed from centralized facilities

rather than 18 percent, as was the case with the PS.

Current plans are to increase the centrally interviewed

sample for the new CPS to about 18 percent during

May and June of 1994.

Even though centralized interviewing has been used in

the CPS since January 1989, it was only at the end of

1993 that some statistical evidence regarding the effect

of only centralization became available. This is

because in the old CPS the old paper and pencil

questionnaire was just not administered from a

centralized location, it was converted to and

administered by a computerized version. In addition,

the wording of the lead-in question "What were you

doing most of last week?" was modified. In the

facilities, the interviewers were instructed to offer only

the response option "working or something else?". In

the field, the interviewers were instructed to vary the

response options "working or something else?",

"keeping house or something else?", and "going to

school or something else?", according to the

appearance, age, and sex of the respondent (Rothgeb

1994). In the old CPS, therefore, it was not possible to

distinguish between the effect of centralization,

computerization, and questionnaire.

In order to minimize any potential effects on official

CPS estimates, the percent of sample cases interviewed

from centralized facilities was purposely kept small.

Over a period of 5 years, the percent of the CPS sample

interviewed from centralized facilities gradually

increased to the 9 percent level used in the 1993 CPS.

An evaluation was conducted by randomly assigning

sample housing units to test and control panels. The

test panels were eligible to be interviewed from

centralized facilities while the control panels were not

eligible for centralized facilities (i.e., they were to be

interviewed in the field or otherwise referred to as

decentralized interviewing). Findings based on CPS

data collected between January 1991 through

December 1992 showed that centralized interviewing

with a modified lead-in question produced an

unemployment rate 0.8 percentage point higher than

for the control or decentralized group (Shoemaker

1993). This measure is a combined effect of

centralization, computerization, and questionnaire.

Based on the above results, the possibility of a

centralization effect existed. The PS was, therefore,

designed to facilitate the comparison of centralized

versus decentralized interviewing. In the PS, many of

the confounding effects were eliminated. First, the

exact same questionnaire was administered by the field

interviewers (decentralized interviewing) and

interviewers at the centralized facilities. Second, the

comparisons of results were based only on data for

month-in-sample two through four and six through

eight households because data for them in field are

mainly collected through computer-assisted

interviewing by telephone, as is done from the

centralized facilities; whereas, the data for month-insample one and five households are usually collected

by personal visits (always decentralized). Still, one big

confounding effect remained. This effect concerns

those households that were in the test panel to be

interviewed from the facilities but were sent to be

interviewed in the field because the interviewers from

the centralized facilities were unable to reach the

respondents or households that were without

telephones (these are called "recycles").

The sample sizes to measure the centralization effect in

the PS were extremely small. The sample size for the

centralized facilities (test group) had about 5,500

housing units per month and the sample size in the

field (control group) had only 600 housing units per

month. Because of these small sample sizes, the

variability was too large to draw any conclusions until

the test was complete in December 1993. In the PS,

the unemployment rate for the test group (centralized

facilities) was 1.1 percentage point higher than for the

control group. This difference is marginally

significant at the 90 percent confidence level

(Kostanich 1994).

1990 Population Controls--In both the CPS and the

PS, a second-stage ratio adjustment procedure was

used. This procedure substantially reduces the

variability of the estimates which are correlated with

the population--particularly estimates of employment

and civilian labor force. Additionally, the adjustment

partially corrects for undercoverage for certain

demographic groups, most notably black males. Under

this procedure, the sample weights are adjusted to

ensure that sample-based estimates of population

match independent population controls for various

groups of population described in the next paragraph.

The overall difference in the unemployment rate

between CPS and PS of 0.45 percentage point used

independent population controls as projected forward

from the 1980 decennial census. Beginning with the

January 1994 CPS estimates, however, the independent

population controls are projected from the 1990

decennial census, adjusted for undercount by the Post

Enumeration Survey (PES). This effect was an

important one to measure because of the differential

change in the size of the population of various

demographic groups.

Differences in second-stage ratio adjustment--The

CPS population estimates are consistent with
independent sets of controls for: 1) 50 States and the

District of Columbia; 2) 14 Hispanic and 5 nonHispanic age-sex groups; and 3) 66 white, 42 black,

and 10 other race-age-sex groups. As mentioned under

general design differences, the PS was a national

sample. It was, therefore, not possible to apply the

controls at the state level. Additionally, the smaller

sample size of the PS resulted in 559 national poststratification cells (control categories 2 and 3) as

compared to 600 for CPS. We examined the effect due

to the lack of state controls and fewer national poststratification cells on the estimates.

Composite estimation--In the CPS estimation process,

a last step before seasonal adjustment is the use of a

composite estimation procedure. The main reason for

applying this procedure is to further reduce the

variance, especially on estimates of month-to-month

change. This reduction is achieved by taking

advantage of the seventy-five percent overlap in the

sample from one month to the next.

Additionally, the current composite estimator adjusts

for month-in-sample bias of the old CPS. In the old

CPS, the expected value of an estimate of the

unemployment rate is higher for new participants

(month-in-sample one) than the average

unemployment rate for the entire sample. Similarly,

those reentering the survey, month-in-sample five,

have a higher unemployment rate than the average of

participants for month-in-sample six, seven, and eight.

It has been shown (Bailar 1975) that slight changes in

the questionnaire can lead to significantly different

month-in-sample bias. Because many changes were

made to the questionnaire, the month-in-sample bias of

the new CPS is likely to be different than the old CPS.

It will, however, take a few years to determine the

pattern of bias for the new CPS.

Because the composite estimator differentially weights

the month-in-sample data, it changes the expected

value of estimates relative to uncomposited estimates

(after post-stratification or the application of the

second-stage ratio adjustment). Because all

comparisons made between the CPS and the PS are

based on uncomposited estimates, we also examined

the effect of compositing. In the new CPS, the

compositing started with June 1994 data.

Seasonal adjustment--Because most users are

interested in comparing seasonally adjusted

unemployment rates, we also applied the CPS seasonal

adjustment procedures (described in the next

paragraph) and factors to the composited estimates

from the PS and CPS. Thus, the final comparison to

assess transitional effects between the CPS and PS was

based on composited and seasonally adjusted estimates.

A seasonally adjusted national unemployment rate for

CPS is produced by aggregating 12 independently

adjusted series. The component series are:

agricultural employment, nonagricultural employment,

and unemployment by four sex-age groups (men 20

years and over, women 20 years and over, men 16 to

19 years, and women 16 to 19 years). Eight of these

series are seasonally adjusted using multiplicative

factors; the other four use additive factors. The four

additive series are nonagricultural men and women of

age 16 to 19 years, and unemployed men and women

of age 16 to 19 years. The seasonal adjustment factors

are generated using X-11 ARIMA software and are

given in the January 1994 issue of Employment and

Earnings (Bureau of Labor Statistics, Vol. 41, No. 1).

Each of these 12 series is separately adjusted for

seasonal variation, and the resulting estimates are then

aggregated to derive seasonally adjusted totals. The

seasonally adjusted figure for the labor force is a sum

of eight seasonally adjusted civilian employment

components and four seasonally adjusted unemployment components. The overall unemployment rate is

derived by dividing the resulting estimate of total

unemployment by the estimate of the labor force.

Variances--Our final step was to examine the effect on

variances for 1994 CPS estimates. In general, a

standard error of an estimated level increases as the

size of the estimate increases. Because the 1990-based

population controls tend to increase the level of

estimates, we compared standard errors on estimates

based on 1980 population controls with those based on

1990 population controls. We did not, however,

examine the effect from the redesign on variances.

IV. Estimates of various design differences

General--The 1993 annual average estimate of

national unemployment rate using 1980-based

population controls for PS was 7.26 percent, while that

for CPS was 6.81 percent. Both of these are

uncomposited estimates. This is an increase of 0.45

percentage point (Bureau of Labor Statistics, 1994,

Employment and Earnings, Vol. 41, No. 2; Polivka

1994) under the new methodology. The estimated

standard error on this increase is 0.17.

Centralization--To derive an estimate of this effect,

we reweighted the PS data to reflect the proportion of

centralized interviewing that was to occur in the

January 1994 CPS (12.5 percent). The reweighting of

the PS involved adjusting base weights within test

(eligible for centralized interviewing) and control (not

eligible for centralized interviewing) panels, and then

performing the usual post-stratification using 1990-

based population controls. Annual average estimates

for 1993 from the reweighted PS were compared to

those from the CPS which also used 1990-based

population controls.

The reweighted PS gives an estimated unemployment

rate of 7.29 compared to the original estimate of 7.37.
The difference between these unemployment rates is

close to being significant at the 90 percent confidence

level. Thus, the expected effect of 12.5 percent

centralized interviewing rather than 18 percent in

January 1994 is to reduce the measured difference in

the national unemployment rate by 0.08 percentage

point. In May and June 1994, the proportion of

interviewing from centralized facilities in CPS will

increase to about the percent in PS. By the end of

June, we expect the centralization effect for the new

CPS to be the same as measured in the PS.

1990 Population controls--The 1990-based population

controls, adjusted for undercount, were applied at the

second-stage ratio adjustment step to the monthly CPS

and PS data for 1993. The 1990-based CPS estimate

was 6.90 percent, yielding an increase of 0.10

percentage point from the 1980-based CPS estimate

(6.90 vs. 6.81, the difference rounded to 0.10). The

1990-based PS estimate is 7.37 percent. The overall

increase between PS and CPS based on 1990

population controls is, therefore, 0.47 (7.37 minus

6.90) percentage point as compared to 0.45 using 1980

population controls. Again, all of these estimates are

uncomposited. Additional informa-tion on the effect of

1990-based population controls is given in the

February 1994 Employment and Earnings (Bureau of

Labor Statistics, 1994, Vol. 41, No. 2).

Second-stage ratio adjustment--To measure the effect

of not having state controls and fewer national

age/sex/race, and age/sex/ethnicity controls in PS, we

reweighted the CPS estimates using the PS post

stratification controls. The 1993 annual average

estimate of national unemployment rate based on 1990

population controls for reweighted CPS was 6.89

percent as compared to 6.90 percent for CPS. This

estimated difference of + 0.01 percentage point, is not

statistically significant.

Compositing--To estimate the effect of compositing on

the national unemployment rate based on 1990

population controls, both the PS and CPS estimates

were composited with two different initial months. In

one case, we initialized the composite with October

1992 data, in the other, with February 1993 data. In

both instances, the composited estimate from PS was

0.06 percentage point lower on a 1993 annual average

basis. For CPS, however, the composited estimate was

0.02 and 0.03 percentage point higher on a 1993

annual average basis. In either case, the difference

from compositing is not statistically significant.

Because the PS sample did not achieve 100 percent

true month-in-sample until September 1993, we could

not say in December whether the compositing in the

new CPS would have the effect similar to PS or to the

old CPS. As mentioned above, it will take a few years

to determine the month-in-sample bias of the new CPS.

In any event, the composite estimate has very little

effect on the expected value of the national monthly

unemployment rate. This was examined extensively

under a variety of bias and percentage change

assumptions (Cantwell and Ernst 1993).

Seasonal Adjustment--As part of our research, we

applied the same seasonal adjustment methods and

factors to both the CPS and PS estimates based on

1990 population controls for each month of January

1993 through December 1993 data. The estimates for

January 1993 were uncomposited while those from

February 1993 onwards were composited. Our results

indicate that the monthly difference in the estimates of

unemployment rate between PS and CPS, after

applying the seasonal adjustments was about the same

as before the seasonal adjustment. We support this by

noting that the seasonal adjustment moved the two

series in the same direction by about the same absolute

amount. As with rotation group bias, it will take

several years before the seasonal pattern of the new

CPS is formed.

Variances--Finally, we compared the standard errors

based on 1990 population controls with those based on

1980 population controls for the CPS. The 1980-based

standard errors were estimated using generalized

variance functions that were derived from 1987 CPS

replicate variances. The 1990-based standard errors

were estimated using replication from 1993 CPS data

post-stratified with 1990-based population controls.

For the national monthly unemployment rate, the

estimated standard error is about 6 percent larger

(0.116 vs. 0.110) when 1990-based population controls

were used. As mentioned earlier, we did not measure

the increase in variance from the redesign.

V. Comparison of Estimated and Actual Effects

1990-Based population controls--In January 1994, we

applied both the 1990-based and the 1980-based

controls to the CPS second stage ratio adjustment

process. The difference was, as expected, 0.10

percentage point higher with the 1990 controls. That

is 7.34 percent with 1990 controls vs. 7.23 with 1980

controls (the difference rounded to 0.10). It should be

noted both of these estimates are uncomposited and not

seasonally adjusted.

Compositing--Although the composite estimation

procedure officially started with June 1994 data, we

examined this effect with February through April 1994

data. The composite estimate was 0.06 percentage

point lower than the uncomposited estimate for

February. Similarly, it was lower by 0.05 and 0.04

percentage point if we had started it, respectively, with

March and April data. In all three instances, both the

uncomposited and composited estimates, not seasonally

adjusted, rounded to the same tenth of a percentage

point. For example, in February, the uncomposited

unemployment rate was 7.14 percent while the

composited estimate was 7.08 percent. For June 1994,
the composited estimate was lower by 0.02 percentage

point.

VI. Summary

January through April--Assuming additivity of

effects and no change in the economy, the expected

overall increase in the national unemployment rate for

January through April 1994 was 0.50 percentage point.

The components are:

Redesign (PS90 - CPS90) = + 0.47

Centralization (12.5 % vs. 18 % sample) = - 0.08

1990 population controls (CPS90 - CPS80) = + 0.10

Fewer post-stratification cells = + 0.01

Total = + 0.50

Because these estimates are uncomposited and all the

comparisons are based on uncomposited data, we need

not account for the effect of compositing at this time.

June and beyond--The composite estimation

procedure resumed with June 1994 data. Since the

remaining five percent of the sample cases were added

in the centralized facilities in May and June, the

centralization adjustment is not necessary. Thus, the

expected overall increase in the national

unemployment rate, again assuming no change in the

economy, from June and beyond is expected to be 0.52

percentage point. The components are:

Redesign (PS90 - CPS90) = + 0.47

1990 population controls (CPS90 - CPS80) = + 0.10

Fewer post-stratification cells = + 0.01

Compositing effect = - 0.06 

 to +0.02

Total = + 0.52 

 to +0.60

In summary, we expected the overall difference, which

takes into account design differences including 1990-

based population controls, to be about 0.5 percentage

point between the "old" and "new" CPS.

VII. Future Research

At present, our biggest area of concern is the

appropriateness of current seasonal adjustment factors.

In the near future, staff at BLS and the U.S. Bureau of

the Census will be investigating and researching

alternative approaches to seasonally adjust CPS data.

Acknowledgement: The authors gratefully

acknowledge the contributions of the following people:

Athar Malik, who provided computer programming

support; and Tamela Elliott, who typed the paper.

References

Bailar, B. (1975), The Effects of Rotation Group Bias

on Estimates from Panel Surveys, Journal of the

American Statistical Association, 70, 23-30.

U.S. Bureau of Labor Statistics (1993), Overhauling

the Current Population Survey, The Monthly Labor

Review, Vol. 116, No. 9, Washington, D.C., U.S.

Department of Labor.

U.S. Bureau of Labor Statistics (1994), Employment

and Earnings, Vol. 41, No. 1, Washington, D.C.,

U.S. Department of Labor.

U.S. Bureau of Labor Statistics (1994), Employment

and Earnings, Vol. 41, No. 2, Washington, D.C.,

U.S. Department of Labor.

Cantwell, Patrick J. and Ernst, Lawrence R. (1993),

Short-Term Changes to the CPS Composite

Estimator in January 1994, Proceedings of the

Section on Survey Research Methods, American

Statistical Association.

Kostanich, Donna L. (1994), Mode Effects Analysis -

Highlights. Mode Effects Analysis CPS Overlap

Analysis Team Technical Report 3. U.S. Bureau of

the Census, Washington, D.C. (unpublished)

Kostanich, Donna L. and Cahoon, Lawrence (1994),

Effect of Design Differences Between the Parallel

Survey and New CPS, CPS Bridge Team Technical

Report 3, U.S. Bureau of Labor Statistics,

Washington, D.C. (unpublished)

Miller, Stephen M. (1994), What Would the

Unemployment Rate Have Been Had the Redesigned

Current Population Survey Been in Place from

September 1992 to December 1993?: A

Measurement Error Analysis. CPS Bridge Team

Technical Report 1. U.S. Bureau of Labor Statistics,

Washington, D.C. (unpublished)

Polivka, Anne E. (1994), Comparisons of Labor Force

Estimates for the Parallel Survey and the Current

CPS. Overlap Analysis Team Technical Report 1.

U.S. Bureau of Labor Statistics, Washington, D.C.

(unpublished)

Rothgeb, Jennifer M. (1994), Revision to the CPS

Questionnaire Effect on Data Quality, CPS Overlap

Analysis Team Technical Report 2, U.S. Bureau of

the Census, Washington, D.C. (unpublished)

Shoemaker, Harland H. (1993), Results from the

Current Population Survey CATI Phase-In Project,

Proceedings of the Section on Survey Research

Methods, American Statistical Association.

Tiller, Richard and Welch, Michael (1994), Predicting

The National Unemployment Rate That The Old

CPS Would Have Produced, CPS Bridge Team

Technical Report 2, U.S. Bureau of Labor Statistics,

Washington, D.C. (unpublished)
Table 1 Sample Design for CPS and PS

Component

Old

CPS-93 PS-93

New

CPS-94

Estimated Effect

(New CPS vs PS)

General:

Sample size 60,000 12,000 60,000

Primary strata within state within region within state

Data Collection

Procedures:

 Centralization (CATI) 9%1 18% Jan-April 12.5%

May 16%

June 18%

-.08

-.03

-.00

 Decentralization paper and

pencil

CATI2 & CAPI3 CATI & CAPI

Post-stratification:

 Population controls 1980 census 1980 census 1990 census (PES

adjusted)

+ .10

 Number of controls 600 National

51 States

559 National

No States

600 National

51 States

+.01

Composite estimation Jan-May--

uncomposited

June-onward--

composited

-.06 to +0.02

Seasonal adjustment 12 series 12 series 12 series None4

 

1

 Wording of the lead-in question was modified (See Rothgeb).

2

 CATI - Computer-Assisted Telephone Interviewing

3

 CAPI - Computer-Assisted Personal Interviewing

4

 Seasonal factors of the old CPS may not be appropriate.