Effects of Methodological Change in the Current Population Survey

Cathryn Dippo, Donna Kostanich, Anne Polivka

In January 1994, a new measurement system was introduced into the Current Population Survey

(CPS). The introduction of a new questionnaire and data collection procedures was the result of

a research and development process that began in 1986. Two important goals of the redesign

were to improve data quality and to increase the availability of information on labor market

activity on a monthly basis.

By modifying question wording and sequencing to better operationalize existing definitions,

reduce the reliance on volunteered information, and embed explicit and implicit recall strategies,

data quality was improved. Changes in the definition of discouraged workers recommended by

the National Commission on Employment and Unemployment Statistics were implemented, along

with increasing the availability of the data by moving the questions from the quarter-sample being

interviewed for the fourth and eighth times to the full sample. Questions on multiple job holding

were also added.

The new questionnaire was developed for a fully-automated collection environment with field

representatives using laptop computers for both their personal visit and telephone interviews

(CAPI) and with a centralized staff of interviewers from a telephone, facility (CATI). Thus, it

was possible to use complex skip patterns to tailor questions to respondents' situations and

information from prior interviews to reduce respondent burden.

To measure the effects of the new measurement system, a Parallel Survey (PS) of 12,000

occupied housing units was conducted from July 1992 through December 1993. While the PS

was designed to replicate the collection methodology and procedures as they would be

incorporated into CPS in January 1994, it was only one-fifth the size of CPS, had a national rather

than a state-based sample as in CPS, and used a simpler post-stratification estimation procedure.

Since the percentage of sample units to be interviewed from the centralized telephone facilities

was to double in January 1994, the PS had about twice as many sample units being assigned to

CATI. Continuing the practice of CPS to embed subpanels to test for the effects of different

modes of collection, the PS (and CPS) contained panels for testing the following hypotheses:

HI: No centralized telephone interviewing effect

H2: No centralized & CATI effect

H3: No questionnaire effect given centralized telephone interviewing

H4: No questionnaire designed for CAI effect.

An important factor in the measurement process which could not be controlled in an experimental

design sense was the interviewing staff. While we attempted to limit interviewer effects by

moving some experienced CPS interviewers to the PS and allowing for up to six months for field

experience with the new questionnaire and collection methodology, there remains the possibility

of the results being affected by interviewers. Interviewers' knowledge and behavior could have

been affected by their prior experience.
Based on the 1993 annual average estimates from CPS and the PS, the new measurement system

was expected to yield .5 percentage point higher estimates of the overall unemployment rate and

.7 percentage point higher estimates of the unemployment rate for women. While no significant

difference was found between estimates of the overall employment-to-population ratio from the

PS and CPS, the estimate from the PS for men was .6 percentage point lower than CPS and .7

percentage point higher for women. The PS also yielded 21 percent lower estimates of the

number of part-time workers for economic reasons and 9.7 percent higher estimates of the

number of self-employed. Among the unemployed, there were no significant differences in the

number on layoff between the two surveys, but the estimated number of new entrants from the PS

was 34 percent lower than from CPS.

When the embedded panels were used to test the above hypothesis with respect to the

unemployment rate (UER) overall and for various demographic subpopulations, evidence of

possible effects were found for the following.

Questionnaire given centralized telephone interviewing: Total female UER (marginally), total

black and black female UER

Questionnaire designed for CAI: Total, total male, & total female UER; total white, white

male, & white female UER; total black, black male, & black female UER; total female & white

female LFPR

Centralized interviewing: Total unemployment rate (marginally); total black & black male

unemployment rate

Centralized & CATI: Total, total female, & total male UER; total white, white female, & white

male UER; black female UER.

In order to determine if various sample design differences between the PS and CPS could be

affecting the results, the CPS was reweighted using PS post-stratification procedures (no

significant difference), and the PS was reweighted to reflect the percent centralized interviewing

to be used in January 1994 (.08 percentage point lower). Also, since the CPS would begin using

1990-census-based population controls in January 1994, it was reweighted to determine the effect

of switching from 1980 to 1990 controls (.10 percentage point higher). It was not possible to

evaluate the effects of not using the composite estimator in January 1994 or the effects of using

seasonal adjustment factors based on old CPS with the new.

Having found no evidence that the observed differences in estimates from the two surveys were

due to sampling design, the decision was made to continue parallel measurement, only with the

new measurement system being used in CPS and the old measurement system being used in the

PS. While we would have preferred to keep at least some portion of the measurement process

constant from December to January, this was not possible due to the differences in the sample

designs. Namely, to provide monthly estimates for the 11 largest states and annual average

estimates for the remaining states with the required reliability, the entire state-based CPS sample

had to use the same collection methodology, and the PS with its national versus state-based

sample could not be used.
In preparation for the conversion of CPS from old to new methodologies in January 1994, CPS

interviewers began training on the new methodology in September 1993. They were given home

studies, attended classroom training, and had several months of dummy assignments to practice in

the field. Moreover, provisions were made for moving some PS interviewers to CPS and others

to other Census-collected current surveys. Thus, when the decision was made to continue the PS,

almost 65% of the required staff had to be hired and trained in less than two months. In both

surveys, the respondents in the overlapping panels between December and January (75%)

experienced a shift from one measurement system to another.

The monthly unemployment rate estimates from the two measurement systems a-re presented in

figure 1. (The PS was discontinued after May for budgetary reasons.) While the estimated

unemployment rate from CPS was higher in January than December, it does not appear as if the

expected annual average increase of .6 percentage points (.5 for methodology change and .1 for

change in population controls) will be observed. Moreover, while we knew there could be

problems with the PS in the first few months of 1994 due to new interviewers and switching

respondents from new to old methodology, the monthly estimates continued to be higher than

those from CPS. On the other hand, the expected effects of changing measurement systems on

other labor force characteristics appear to have occurred.

To help users of CPS time series data bridge the gap between the old and new methodologies, the

parameters of a linear model with main effects for survey and collection method were estimated

by GLS under three sets of restrictions, necessitated by the model not being fully identifiable.

While a primary goal of this modeling effort was to develop adjustment factors for analysts (see

tables 1 and 2), significance tests for main effects were also conducted. Under the restriction that

the old and new collection method effects were the same both before and after the switch in

January, it was possible to estimate a new method effect and a PS effect. For some labor force

items, the estimated new method effect is similar to that estimated by comparing 1993 annual

averages from the two surveys. In other cases, most importantly, the overall unemployment rate,

this is not true.

There are a number of lessons that can be learned from this experience of parallel testing. Most

importantly, parallel testing should be done if major changes a-re to be made in a measurement

process. It is just as important to remember, especially when analyzing the results for an ongoing

rotating panel survey like CPS, that you only have one observation for each of two treatments.

Either one or both of the estimates could be from the tail of the sampling distribution. While one

can estimate the sampling variance, nonsampling error is really the focus, since, presumably, the

change in measurement process is to reduce nonsampling error, Therefore, it is necessary to

address all of the factors which could affect both sampling and nonsampling errors. With the

exception of the methodological changes being tested, the parallel survey should mimic is much as

possible the entire survey design environment of the main survey. This includes interviewer

training and experience, field management procedures, and sample design. If multiple changes are

being tested, a sample design which allows for the testing of separate effects should be attempted

in order to assess which of the changes might be affecting the nonsampling error structure.
For further details on the results of the parallel testing, see the following reports which can be

obtained by contacting the Office of Survey Methods Research, Bureau of Labor Statistics, at

202-606-7370 or fax 202-606-7426.

Kostanich, Donna L. and Cahoon, Lawrence S. (I 994), Effect of Design Differences Between the

Parallel Survey and the New CPS, CPS Bridge Team Technical Report 3, March 4.

Miller, Stephen M. (I 994), What Would the Unemployment Rate Have Been Had the Redesigned

Current Population Survey Been in Place from September 1 992 to December 1993?: A

Measurement Error Analysis, CPS Bridge Team Technical Report 1, March 3.

Polivka, Anne E. (1994), Comparisons of Labor Force Estimates from the Parallel Survey and

the CPS During 19,93: Major Labor Force Estimates, CPS Overlap Analysis Team

Technical Report 1, March 18.

Rothgeb, Jennifer M. (1994), Revisions to the CPS Questionnaire: Effects on Data Quality, CPS

Overlap Analysis Team Technical Report 2, April 6.

Thompson, Jenny (1994), Mode Effects Analysis of Major Labor Force Estimates, CPS Overlap

Analysis Team Technical Report 3, April 14.

Table 1. Parameter Estimates for Employmentto-Population Ratio (Restriction 3)

Multiply Additive '93 Average

Total 1.0053* 0.33* 61.6

Men 16+ 0.9964 -0.25 69.9

Women 16+ 1.0156* 0.84* 54.1

White Men 0.9967 -0.23 71.3

White Wm. 1.0169* 0.92* 54.7

Black Men 0.9831 - 1.02* 59.1

Black Wm. 1.0093 0.48 50.5

Teenagers 1.005 0.21 41.7

55-64 Years 1.0124 0.65 53.8
Table 2. Parameter Estimates for

 Unemployment Rate (Restriction 3)

 Multiply Additive '93 Average

Total 1.009 0.079 6.8

Men 16+ 1.012 0.10 7.1

Women 16+ 1.007 0.07 6.5

White Men 1.029 0.19 6.2

White Wm. 1.025 0.15 5.7

Black Men 0.971 -0.38 12.4

Black Wm. 0.965 -0.48 11.0

Men 16-19 1.029 0.71 20.4

Men 25-55 0.983 -0.08

Men 65+ 1.69* 1.93* 3.2

Wm. 25-55 0.992 -0.04