EVALUATING THE DISPLACED-WORKER/JOB-TENURE

SUPPLEMENT TO THE CPS: AN ILLUSTRATION OF

MULTIMETHOD QUALITY ASSESSMENT RESEARCH 1

James L. Esposito, Bureau of Labor Statistics

Key Words: Questionnaire evaluation, behavior coding, interviewer debriefing, respondent debriefing, probe

questions, displacement, specification error, measurement error.

I. Background

In the early 1980s, the American economy was staggered by two recessions that were especially

hard on manufacturing industries—particularly steel and automobile production. Manufacturing

plants were closed, shifts were eliminated, and workers lost good-paying jobs. In an effort to

assess the effects of these developments on the labor force, a small group of economists at the

Bureau of Labor Statistics set about to develop a questionnaire that would estimate the number of

workers displaced from jobs. This survey, which is known as the Displaced Worker Survey

(DWS), was administered as a supplement to the Current Population Survey (CPS) in 1984. And

even though the DWS was intended to be a one-time survey, the data it generated had utility for

both internal and external users and, as a result, it has been administered biennially ever since.

Displaced workers (some prefer the term “dislocated workers”) have been defined as follows:

“While there never has been a precise definition for [displaced workers], the term is generally applied to

persons who have lost jobs in which they had a considerable investment in terms of tenure and skill

development and for whom the prospects of reemployment in similar jobs are rather dim.” (Flaim and

Sehgal, 1985)

“Dislocated workers are individuals with established work histories who have lost their jobs through no fault

of their own and who are likely to encounter considerable difficulty finding comparable employment. Such

individuals are thought to have lost their jobs because the industries or occupations in which they worked are

in long-term decline.” (Browne, 1985)

This research effort expands on an earlier assessment of the DWS conducted in 1996 (Esposito

and Fisher, 1998). In the following section, I describe the methods used in the second phase of

this multiphase evaluation of the DWS.2 In section three, I review some of the more important

findings relevant to the measurement of displacement in the U.S. labor force—the discussion will

be limited to a review of information/data for the first two supplement questions (SD1 and SD2,

see Table 1). In section four, I utilize a framework delineated by Stanley Freedman (Federal

Committee on Statistical Methodology, 1988) in an effort to suggest how our interdisciplinary

research team might minimize response and measurement error in redesigning the DWS.

 

1 The views expressed in this paper are those of the author and do not reflect the policies of BLS. It would not

have been possible, however, to conduct this research without the contributions and insights of subject-matter

specialists at the Bureau of Labor Statistics (Thomas Nardone, Fran Horvath, Jay Meisenheimer, Steven Hipple,

Jennifer Martel, Alan Eck) and without the hard work and cooperation of the Census Bureau’s CPS field staff. I

wish to thank all those individuals who contributed to the success of this research effort.

2

 This multiphase research effort reflects the Bureau’s commitment to survey evaluation research as a means

towards the goal of collecting accurate and reliable labor force data. This commitment to collecting high-quality

survey data was made explicit in 1996 when Katharine Abraham issued Commissioner’s Order No. 2-96,
II. Research Methodology

In evaluating the DWS, we relied on information/data generated by three evaluation methods:

interviewer debriefing, interaction coding (i.e., behavior coding), and respondent debriefing.

Interviewer Debriefing. While there are a variety of ways to gather evaluative information from

interviewers (Converse and Schuman, 1974; DeMaio, 1983; DeMaio et al., 1993), we debriefed

interviewers using a focus-group format and a target-question rating form (Esposito, 1999).

Focus groups generally consist of a moderator and 8-12 participants who come together to

discuss a particular topic. The role of the moderator is to raise specific topics for discussion and

to regulate interactions among the various participants. We conducted three debriefing sessions,

one at each of the Census Bureau’s three telephone centers. The purpose of these debriefing

sessions was to obtain feedback from CPS interviewers regarding several aspects of the

supplement. More specifically, interviewers were asked: (a) to identify any question-specific

problems that manifested themselves during administration of the supplement and to rate the

difficulty level of target questions they had identified as problematic; (b) to respond to a series of

structured questions regarding the effectiveness and comprehensibility of twelve DWS target

questions; (c) to provide opinions/perceptions (and written comments) about the functioning of

these twelve target questions and about the flow, sequence, and structure of the supplement; and

(d) to provide feedback on any additional concerns about the DWS they may have had. An

extensive protocol was used to guide the focus group discussion and stimulate interviewer

feedback. All three sessions were audiotaped and written summaries were prepared from these

tapes. Senior staff supervisors from the telephone centers were responsible for selecting the

interviewers who participated in these debriefing sessions. Twenty-six of the thirty-four

participants were female, and, as requested by the moderators, they differed in terms of

interviewing experience.

Behavior Coding. Behavior coding involves a set of procedures which have been found useful in

identifying problematic questionnaire items (e.g., Cannell and Oksenberg, 1988; Oksenberg,

Cannell and Kalton, 1991). These procedures include developing a coding form, monitoring

interviews, coding interviewer-respondent exchanges, transferring coded data to a database, and

summarizing coded data for individual questionnaire items. The coding form used in this research

effort included six interviewer codes [exact question reading, minor change in wording, major

change in wording (i.e., meaning of question affected by change in wording), probe, verify (i.e., a

previous answer), and feedback] and eight respondent codes [adequate answer, qualified answer,

inadequate answer (i.e., answer does not meet question objectives), request for clarification,

interruption, don’t know, refused to answer, other (i.e., a miscellaneous category)].

Behavior coding was conducted during the first four days of CPS survey week at two telephone

centers (Hagerstown, MD, and Tucson, AZ) and was done on-line, that is, while CPS/DWS

interviews were in progress. One researcher (the author) monitored CPS interviews, selected

cases to code, and coded interactions between interviewers and respondents during DWS

administration. Only data from the first exchange between interviewer and respondent was

analyzed, and, at either end of the exchange, a maximum of two behavior codes was assigned

 

“Ensuring Quality in the Data Collection Process.” This research is also consistent with the Bureau’s Quality

Measurement Model (1994) and with the pretesting policy of the U.S. Bureau of the Census (1998).
(e.g., two possible codes assigned for the respondent’s end of the exchange might be “adequate

answer” and “interruption”). The researcher/coder monitored a CPS interview until it was time

for the DWS to be administered; when the first supplement question appeared (SD1), he started

coding the case using a paper-and-pencil coding form. Sixty-three (63) household interviews were

monitored and a total of 145 person-interviews were coded; sixty (60) different interviewers

conducted the 63 CPS/DWS person interviews.

Telephone-center interviewers are very efficient at reading questionnaire items as worded within a

computer-based interviewing environment. With regard to coded interviewer behaviors, previous

work has led us to expect very high percentages of exact question readings (Esposito and

Rothgeb, 1997). When the percentage of exact question readings falls below 90%, we flag the

item as having potentially problematic wording. With regard to coded respondent behaviors,

diagnostic procedures are not quite as straightforward. While it may be comforting to find that

respondents provide adequate answers over 85% of the time, researchers tend to focus on other

codes to gain insights into the types of problems that may exist. For example, a high percentage

of interruptions (10% or more) often suggests that a questionnaire item is too wordy and/or that

the item is not relevant for the target person. A high percentage of requests for clarification

(10% or more) suggests that there may be problems with various terms/words used in the survey

item. We should note here that while behavior coding is useful in isolating problematic survey

questions, it is necessary and prudent to use other evaluation methods to identify the potential

causes of those problems and to provide insights as to what types of modifications might be made

to improve data quality.

Respondent Debriefing. While there are various techniques available for gathering evaluative

information/data from survey respondents (Forsyth and Lessler, 1991), we used the follow-up

probe question approach pioneered by Belson (1981) for this purpose. A small interdisciplinary

team of questionnaire-design and subject-matter specialists drafted the respondent debriefing

questionnaire. A total of 28 debriefing items were developed that specifically targeted DWS

questions (and concepts). More specifically, the debriefing items were designed: (a) to gather

job-related information that was relevant to the displacement concept, and (b) to determine

whether question-specific problems existed that might jeopardize an accurate count/estimate of

displaced workers. Each debriefing question was designed with a specific objective in mind. For

example, for persons who separated from an employer during the reference period (i.e., “yes”

response to SD1), the research team wanted to know: (a) if the target person lost or left that job;

(b) why she/he lost or left that job; (c) if the job was temporary—that is, “if the job was supposed

to last a limited time or until the completion of a project”; (d) if the job leaver had received

written advance notice prior to separation; (e) if the target person had lost or left more than one

job during the reference period and, (f) if so, how many jobs did the person lose/leave.

Answers to debriefing questions of this sort were useful in helping the research team to detect

potential sources of response error. In an effort to minimize costs and respondent burden, the

research team restricted respondent debriefing to month-in-sample four and eight households,

which together constitute approximately 25% of the full CPS sample. The sequencing of

questions went as follows: respondents were first asked the basic CPS questions for all eligible

household members, then supplement questions for all eligible household members, and then the

debriefing questions. Certain demographic and/or labor force criteria determined which
displacement questions the respondent was eligible to be asked. These criteria (and responses to

specific supplement items) determined which debriefing questions the respondent was asked.
III. Findings

Before summarizing findings, let us review how respondents enter and leave the displaced-worker

question stream. First, the CPS labor force questions are asked for all eligible persons in the

household. Then, after a very brief transition statement, the respondent is asked SD1. If the

respondent believes the target person has lost or left a job for one of the reasons stated in the

question—or if the phrase “or another similar reason” seems appropriate—the respondent

answers “yes” and then she/he is asked SD2. Responses of “no” to SD1 are skipped out of the

displacement series. In SD2, the interviewer reads the question and all six response options; the

respondent is asked to identify the reason that most closely reflects why the target person lost or

left her/his job during the three-year reference period. Responses coded 1-2-or-3 remain in the

displacement question stream; only these persons can be categorized as displaced workers.

Responses coded 4-5-or-6 are channeled out of the displacement question stream; these persons

can not be categorized as displaced workers. As shown in Table 1 (SD1: “no” equals 92.6%;

SD2: options 4-6 equals 36%), most persons are not classified as displaced.

Questionnaire design experts will recognize almost immediately that there are various content and

design problems associated with SD1 and SD2. However, unless it can be demonstrated that such

problems are responsible for significant response error, survey sponsors are generally loathe to

make changes in a questionnaire—this is especially true if a time series is being jeopardized. As

specialists in questionnaire design and evaluation, our responsibility is to provide survey sponsors

with information/data they can use to make informed decisions as to whether changes in the target

questionnaire are warranted. A summary of findings with respect to SD1 and SD2 can be found

in Tables 2 and 3. Behavior coding data suggest that both questions pose challenges for

interviewers and respondents. Because of their length, interviewers have difficulty reading these

questions exactly as worded. The phrase “or another similar reason” apparently causes problems

for some respondents. [Note: If respondents interpret this phrase literally, that is, if the reason

the target person lost/left a job is similar to one of the three reasons listed in SD1 (i.e., if

SD1=yes)—but not similar enough to trigger an exact match in the respondent’s mind—then it

follows that at least some job losers/leavers will be missed as displaced workers when skipped out

of the displacement question stream via SD2/option 6.] Some respondents appear to interpret the

phrase “lose a job or leave one” too broadly (e.g., left for better job). Difficulty ratings for SD1

and SD2 (see Table 4) suggest that a potentially large minority of respondents experience

problems in their attempts to provide adequate answers to these questions. Though suggestive,

findings from behavior coding and interviewer debriefing do not provide “hard data” regarding

response error; uncertainty remains as to how serious these problems with SD1 and SD2 might

be. Fortunately, the method of respondent debriefing generated quantitative data that were useful

in producing estimates of response error. For respondents who answered “yes” to SD1 and

“some other reason” to SD2, a debriefing question [SDB3] was asked to determine which target

persons might be classified as false negatives (i.e., persons not categorized as displaced who

really should have been). Recall that “some-other-reason” responses are skipped out of the DWS

question stream and, as a result, cannot be categorized as displacements. A false negative would

be a person for whom a displacement reason is given to SDB3 (see Table 5, middle column).

Summing these five response options, it appears that, of the “some-other-reason” responses to

SD2, about 16% may actually be false negatives. The good news is that the other 84% probably

were not cases of displacement—though one has to wonder what was going through minds of
respondents when they encoded the phrase “or another similar reason” in supplement item SD1. A

second debriefing question [SDB17] was asked of respondents who answered “no” to SD1 to

determine if the target person had left any job during the reference period. Those that had (about

13%) were then asked why the target person was no longer working at her/his former job

[SDB20]. The results appear in Table 5—and the same reasoning applies with respect to false

negatives. Summing the appropriate response options (third column), it appears that, of the

persons who had lost or left a job during the reference period but who had answered “no” to SD1,

approximately 6% may actually be false negatives. While this percentage is smaller than that

noted above for the some-other-reason group (16%), the larger numbers of persons answering

“no” to SD1 actually mean that there is more response error associated with the SD1=no group.

IV. Discussion

When the DWS was administered in 1998, data for the first supplement item (SD1) was collected

for over 79000 target persons. Data for the second item (SD2) was collected for over 5800

persons; of this number, approximately 3700 responses (64%) were coded into one of the three

displacement response categories and another 1700-plus responses (30.5%) were coded as “some

other reason” (see Table 1). These two questions carry an enormous classification burden and the

tasks we impose on survey participants are indeed challenging; for example, respondents and

interviewers collaborated to map/match over 5800 survey responses into six response categories.

Given the myriad manifestations that job separations can assume, it would be truly amazing if

there were not some measurement error associated with this process. The quality assessment

research described herein was an attempt to provide some insights as to the potential sources of

response and measurement error and, with respect to SD1 and SD2 only, some rough estimates of

potential misclassification. For some, the collection and subsequent analysis of respondent

debriefing data probably would have been quite sufficient to set in motion plans to redesign the

DWS—in fact, some may wonder what the utility of the other two methods might be. But issues

raised by interviewers and their supervisors—corroborated by behavior coding data—point to

potential root causes for the misclassified cases (i.e., false negatives) detected by the respondent

debriefing method. Their comments have prompted a reexamination of the conceptual

foundations of the DWS and raised a lot of very interesting questions, like: What counts as a job?

What does it mean to be displaced from a job? Can persons who work at temporary jobs be

considered displaced? What are the principal causes of displacement in today’s economy? What

should we do about persons who leave a job in anticipation of being displaced? Once these

conceptual issues have been resolved, the chances improve that we will be able to construct a

questionnaire for collecting displacement data that minimizes response and measurement error.

A useful analytical framework for addressing measurement issues, originally proposed for

establishment surveys but readily adaptable for household surveys, has been delineated by

Freedman (Federal Committee on Statistical Methodology, 1988). Freedman defines specification

error as: “...the error that occurs in the planning stage of a survey because data specification is

inadequate or inconsistent with respect to the objectives of the survey” (p. 34). In Table 6, we

utilize his specification-error framework as a means of addressing some of the measurement and

conceptual issues associated with the DWS. It is our hope that recommendations offered in Table

6 might prove useful in the redesign of the DWS.
References

Abraham, K.A (1996). “Ensuring Quality in the Data Collection Process,” Commissioner’s Order No. 2-96, Washington, DC:

Bureau of Labor Statistics.

Belson, W.R. (1981). The Design and Understanding of Survey Questions, Aldershot, England: Gower.

Browne, L. (1985). “Structural Change and Dislocated Workers,” New England Economic Review, January/February, pp. 15-

30.

Cannell, C. and Oksenberg, L. (1988). “Observation of Behavior in Telephone Interviews”, R.M. Groves, P.P. Biemer, L.E.

Lyberg, J.T. Massey, W.L. Nicolls, II, and J. Waksberg (eds.), Telephone Survey Methodology, New York: Wiley, pp. 475-

495.

Converse, J.M. and Schuman, H. (1974). Conversations at Random, New York: Wiley.

DeMaio, T., Mathiowetz, N., Rothgeb, J., Beach, M.E., and Durant, S. (1993). Protocol for Pretesting Demographic Surveys

at the Census Bureau, Census Bureau Monograph, Washington, DC: Bureau of the Census.

DeMaio, T.J. (1983). “Learning from Interviewers,” in T.J. DeMaio (ed.), Approaches to Developing Questionnaires,

Statistical Policy Working Paper 10, Washington, DC: Office of Management and Budget, pp. 119-136.

Esposito, J.L. (1999). “Quality Assessment Research on the Displaced-Worker/Job-Tenure Supplement to the CPS: Results

from the Debriefing of Interviewers, Unpublished Report (BLS).

Esposito, J.L., and Fisher, S. (1998). “A Summary of Quality Assessment Research Conducted on the 1996 DisplacedWorker/Job-Tenure/Occupational-Mobility Supplement (Statistical Note Number 43),” BLS Statistical Note Series, Bureau of

Labor Statistics: Washington, DC.

Esposito, J.L., and Rothgeb, J.M. (1997). “Evaluating Survey Data: Making the Transition from Pretesting to Quality

Assessment”, in Lyberg, Biemer, Collins, deLeeuw, Dippo, Schwarz, and Trewin (eds.), Survey Measurement and Process

Quality, New York: Wiley, pp. 541-571.

Federal Committee on Statistical Methodology (1988). “Measurement of Quality in Establishment Surveys,” Statistical Policy

Working Paper 15, Washington, DC: Statistical Policy Office, U.S. Office of Management and Budget, pp. 33-42.

Flaim, P.O. and Sehgal E. (1985). “Displaced Workers of 1979-83: How Well Have They Fared?,” Monthly Labor Review,

108 (6), pp. 3-16.

Forsyth, B.H. and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy,” In P.P Biemer, R.M. Groves, L.E.

Lyberg, N.A. Mathiowetz, and S. Sudman (eds.), Measurement Errors in Surveys, New York: Wiley, pp. 393-418.

Oksenberg, L., Cannell, C., and Kalton, G. (1991). “New Strategies for Pretesting Questionnaires, Journal of Official

Statistics, 7, pp. 349-365.

U.S. Bureau of the Census (1998). “Pretesting Policy and Options: Demographic Surveys at the Census Bureau,” Washington,

DC: U.S. Department of Commerce.

U.S. Department of Labor, Bureau of Labor Statistics (1994). “The Bureau of Labor Statistics Quality Measurement Model,”

Washington, DC: Bureau of Labor Statistics.

Table 1: DWS Items SD1 and SD2 [Edited Data, 1998]

SD1

[N=79503]

During the last 3 calendar years, that is January 1995 through December 1997, did (you/fill name) lose

a job or leave one because: (Your/His/Her) plant or company closed or moved, (your/her/his) position or

shift was abolished, insufficient work, or another similar reason?

7.4% <1> Yes (Go to SD2)

92.6% <2> No (Go to Tenure Series)

SD2 Which of these specific reasons describes why (fill name is) (you are) no longer working at that job?

[N=5838] READ IF NECESSARY: If (you/she/he) lost or left more than one job in the last 3 years, refer to the

job (you/she/he) had the longest when answering this question and the ones to follow.

[Note: Interviewers are instructed to read all six response options to the respondent.]

25.3% <1> Plant or company closed down or moved (Go to SD3)

 Plant or company still operating but lost or left job because of:

22.4% <2> Insufficient work (Go to SD3)

16.3% <3> Position or shift abolished (Go to SD3)

4.1% <4> Seasonal job completed (Go to Tenure Series)
1.4% <5> Self-operated business failed (Go to Tenure Series)

30.5% <6> Some other reason (Go to Tenure Series)
Table 2. A Summary of Quality Assessment Results for DWS Item SD1

Behavior Coding: We have data for 135 administrations of SD1. These data suggest that interviewers had

difficulty reading this question exactly as worded (i.e., 71% exact readings). The percentages of minor and

major changes in question wording, 16% and 13% respectively, were quite high relative to other supplement

items. On the other side of the interaction, respondents interrupted with an answer to SD1 (almost always a

“no” answer) 25% of the time and provided an inadequate/uncodable answer 10% of the time. Most of the

inadequate answers were the results of respondents reporting a particular situation that did not reflect the intent

of the question (e.g., person said she moved) or that was ambiguous as to cause. For example, one respondent

reported that he did not have sufficient work at his two part-time jobs, so he found a full-time job that provided

sufficient work. What makes this response inadequate is uncertainly as to the cause of the insufficient work.

And while 88% of responses to SD1 could be coded as adequate, in about 6% of these cases the respondent

simply reported her/his situation and left it for the interviewer to make the correct entry. [Note: Due to the fact

that two unique codes were permitted on either side of the exchange, the sum of interviewer and respondent

codes can, and usually does, exceed 100%.] Part of the problem with SD1 is the length of this question and the

propensity of some respondents to interrupt with an answer after they have heard the first part of the question:

“During the last 3 calendar years, that is January 1995 through December 1997, did you lose a job or leave

one?” When this happens, the interviewer is less likely to read the second part of the question; the result is a

relatively high percentage of major changes in question wording (13%) and an increased potential for heading to

SD2 rather than to the next section of the supplement (i.e., tenure series).

Interviewer Debriefing: Interviewers in all three focus groups mentioned problems that respondents appeared

to have in interpreting various features of this question. Particularly problematic was the meaning of the phrase

“or another similar reason”. Given their answers to subsequent questions (i.e., SD2), some respondents clearly

interpreted the question more broadly than intended—to include jobs that may have been lost or left during the

reference period for any reason (e.g., to take a better job; to go back to school; to start a business). Interviewers

themselves were not completely sure what this phrase encompassed. Most apparently assumed that it meant for

a reason similar to one of the reasons that was specifically mentioned in the body of the question. In fact, when

interviewers realized (at SD2) that the reason was not similar (e.g., the target person had left a job voluntarily to

take something better with a different employer), some interviewers felt obligated to skip back to SD1 and

change the entry from “yes” to “no”. Moving on to other problems, one interviewer mentioned an interesting

case that involved a person who the respondent said had lost her job; in actuality, the company changed her

status from full-time to part-time. She had the same duties, but all the full-time jobs like hers were made into

part-time jobs. Another interviewer noted: “I had difficulty coding answers such as: (1) husband was transferred

with the military; (2) company was sold—buyout; (3) self-employed business went bankrupt; and (4) respondent

was fired.” Interviewers also mentioned a variety of pragmatic issues. For example, parents sometimes found it

difficult to report for their children. One respondent stated that his “... son may have had 50 jobs during that

time [the three-year reference period] that he’s left. One for a few days, one for two weeks, then he would quit

a job and he wouldn’t tell me his reason.” The respondent had no idea at which job his son had stayed at the

longest. Interviewers in all three focus groups asked if it was really necessary to ask these questions of retired

persons—especially those individuals who were very old, disabled, or long-term retired. Several interviewers

mentioned being interrupted as they tried to read SD1 in its entirety. Another mentioned that she had to repeat

the reference period approximately half the time she read this question. Difficulty Ratings: SD1 was identified

as problematic by interviewers in all three focus groups. The mean difficulty rating for this question was 2.18

(see Table 4).

Respondent Debriefing: As noted in the text and in Table 5, respondent debriefing data indicate that, of the

persons for whom a “no” response was given to SD1, about 13% separated from an employer during the threeyear reference period. Of that number, approximately 6% may have been displaced from jobs (false negatives).

Other respondent debriefing data will be useful in making decisions on how to re-conceptualize displacement

when the DWS is redesigned. For example, we know from a crosstab involving debriefing item SDB1 that, of

the persons for whom a displacement reason was provided in supplement item SD1, about 20% left their jobs.

This is significant because, in the redesigned DWS, job-leavers (but not job-losers) will have to satisfy additional

conditions to be classified as displaced (e.g., written advance notice that their jobs will be ending).
Table 3. A Summary of Quality Assessment Results for DWS Item SD2

Behavior Coding: As noted in the text and in Table 1, to be asked SD2 the target person has to have lost or left

a job during the three-year reference period, and this did not happen with great frequency in our sample of 145

person interviews (about 12% of the time; 18 persons total). As a result, we have relatively little behavior

coding data for SD2; and so, our findings for SD2 need to be interpreted cautiously. Interviewers had a great

deal of difficulty reading SD2 as worded (0% exact readings); in fact, SD2 was read with major wording

changes 72% of the time this question was asked (13 persons). On the other side of the interaction, in 28% of

the cases (5 persons), the initial response provided by the respondent was coded as inadequate (i.e., these

responses could not be coded into one of the available response options). Like SD1, part of the problem with

SD2 is the length of the question and the propensity of some respondents to interrupt with an answer when they

believe they have heard a response option that fits—or seems to fit—the target person’s situation. Interruptions

occurred about 40% of the time SD2 was asked (7 persons). It is very difficult for some interviewers to continue

reading a list of response options when the respondent has identified one of these options—perhaps the first or

second—as the correct answer. Data-quality problems arise when an option lower on the list of response options

better describes the target person’s situation than the earlier option.

Interviewer Debriefing: Interviewers found this question very difficult to read as worded—respondents often

interrupted with an answer before the interviewer got through the list of response categories. One group

of interviewers was asked if they recalled having read the qualifying information that precedes the second

response option: “Plant or company operating but lost or left job because of:” Not all interviewers read this

qualifying information; they just read the options. One interviewer said she did read this material, but added that

this is where the wording became awkward. The read-as-necessary statement was problematic in that it was not

always clear to interviewers when it was applicable and, if nothing else, it was intrusive if read inappropriately.

Most interviewers seemed to understand that the first three response options to SD2 continued down the

displacement path and that the fourth and fifth options did not; however, few knew for sure what happened

when the sixth option (“some other reason”) was selected. So, when it was “clear” to them that the target

person was not actually displaced from a job (e.g., the person took a better paying job), some interviewers would

back up and change SD1 rather than take their chances with SD2/option-6. Sometimes, the explanation an

employer shares with a soon-to-be-dismissed employee is not completely accurate, and this tends to complicate

the coding process. For example, one interviewer reported that she had a respondent who was told that his

position was being abolished, and the person added: “But I know they filled it with somebody else.” Respondents

did not always know how to categorize the target person’s situation (i.e., they had difficulty mapping particular

situations on to one of the available response options). And, so, when they had an unusual job-loss story to tell,

respondents simply described the situation to the interviewer and left it for him/her to select the correct response

category—as in the case where, say, Company A bought Company B and the target person lost her/his job. The

problem, in part, is that interviewers are not really sure what to do with some of these cases either—the survey

instrument provides them with only one escape route (SD2/option-6), and that route sometimes produces the

wrong classification outcome (i.e., a false negative). Difficulty Ratings: SD2 was identified as problematic by

interviewers in two of three focus groups. The mean difficulty rating for this question was 2.55 (see Table 4).

Respondent Debriefing: As noted in the text and in Table 5, respondent debriefing data indicate that, of the

persons for whom a “yes” response was given to SD1 and “some other reason” was given to SD2, about 16%

may have been displaced from jobs (false negatives). Other respondent debriefing data will be useful in making

decisions on how to re-conceptualize displacement when the DWS is redesigned. For example, we know from a

crosstab involving debriefing item SDB2 that, of the persons for whom a displacement reason was provided in

supplement item SD1, about 13% had held temporary jobs (i.e., a job that was supposed to last for only a limited

time or until the completion of a project). This is significant because, in the redesigned DWS, persons who lost

“temporary jobs” may have to satisfy tenure criteria to be classified as displaced (e.g., they may have to have

worked for their employer for, say, two years or more).
 Table 4: Interviewer Ratings for DWS Items SD1 and SD2

TC Center Ratings Mean SD

TTC SD1: 3 1 2 2 3 1 1 1 1 1 1 3 1.67 0.89

SD2: - - - - - - - - - - - - - -

HTC SD1: 3 2 2 2 3 1 2 2 1 4 2.20 0.92

SD2: 3 3 1 1 4 1 1 2 2 2 2.00 1.05

JTC SD1: 4 2 2 2 4 4 4 1 3 2 2 2 2.67 1.07

SD2: 3 2 3 2 4 1 3 4 5 3 3 3 3.00 1.04

Note: Interviewers were asked to rate problematic supplement items using the following evaluation scale:

Based on your experiences this past week, how frequently have respondents had difficulty providing an

adequate answer to [the target question] when asked?

A. [1] Never or Very Rarely (0 to 5% of the time)

B. [2] Occasionally (some % in between A and C)

C. [3] About Half of the Time (approximately 45-55% of the time)

D. [4] A Good Deal of the Time (some % in between C and E)

E. [5] Always or Almost Always (95 to 100% of the time)

Table 5. Debriefing Questions Used to Identify Potential False Negatives

Selected Response Options (see SDB3 and SDB20)2,3

SD2= “some

other reason”

[SDB3]1,2

SD1= “no”

[SDB20]1,3

N % N %

357 100% 2103 100%

Displacement Reasons (five options)

Employer closed down business 8 2.2% 20 1.0%

Employer moved away 2 0.6% 13 0.6%

Employer was downsizing or restructuring 36 10.1% 48 2.3%

Employer had insufficient work 8 2.2% 36 1.7%

Worker’s position/shift was abolished 3 0.8% 12 0.6%

Aggregated Personal Reasons (14 options; e.g., did not like

job, better job, not enough pay, own illness/injury, fired, moved

away, school/training, child-care problems)

223 62.5% 1608 76.5%

Other (i.e., verbatim responses) 75 21.0% 329 15.6%

Note 1: All cases represent target persons whose jobs were reported NOT to be temporary. Also, some response

options do not appear in column one (e.g., seasonal job completed, don’t know); as a result, row percentages will

not sum to 100% and row entries will not sum to the total N.

Note 2: Respondents who answered “yes” to SD1 and “some other reason” to SD2 (see Table 1), were asked the

following debriefing question (SDB3): Some people leave jobs for personal reasons, such as to further their

education or to care for children. Others lose or leave jobs for economic reasons, such as insufficient work or

downsizing. What is the MAIN reason (you are) (she/he is) no longer working at that job?

Note 3: Respondents who answered “no” to SD1 were asked the following debriefing question (SDB20): What is

the MAIN reason (you are) (she/he is) no longer working at that job?
Table 6: Addressing DWS Measurement and Conceptual Issues [Freedman’s Framework]

Sources/Causes of

Specification Error

DWS-Relevant Comments and Recommendations for Minimizing Response

and Measurement Error

Inadequately specified uses

and needs. Causes: (1)

poorly stated uses and needs;

(2) changing uses and needs.

Three observations regarding the DWS should be noted. First, the DWS was

designed in the early 1980s as a one-time survey. Second, most national

economic surveys developed prior to 1984 probably were not subjected to the

rigorous domain-specific conceptualizing and evaluative pretesting that is

considered almost mandatory today. And third, the structure of the U.S. economy

has changed dramatically over the past 15 years. The survey sponsor should state

explicitly what it considers the appropriate needs and uses of DWS data and

consider the views of internal and external data users when doing so. The sponsor

should monitor economic conditions and user needs on an ongoing basis and

consider making modifications to the survey when such changes seem justified. To

the extent possible, every effort should be made to preserve an existing time

series.

Inadequately specified

concepts. Causes: (1) poorly

defined concepts; (2) using

an existing concept that does

not really fit.

Existing definitions of displaced/dislocated workers are vague and/or dated. It is

very difficult to develop specific objectives for survey questions under such

conditions. For example, the definition provided by Flaim and Sehgal (1985)

refer to “persons who have lost jobs in which they had considerable investment in

terms of tenure and skill development...” While there is a specific question on the

DWS that collects displacement-relevant tenure data, there are no questions that

address skill development. Moreover, what constitutes a “considerable investment

in terms of tenure and skill development” probably differs according to one’s

occupation or employment situation. The survey sponsor should define the

concept of displacement in clear and unambiguous terms. Insofar as they will

serve as the basis for preparing question objections and developing actual survey

questions, all of the terms/concepts used in that definition also need to be clearly

defined.

Inadequately specified data

elements. Causes: (1)

ambiguous definitions; (2)

elements not reflecting

survey concepts; (3)

ambiguous question wording

and constructs.

As noted, SD1 and SD2 carry a heavy burden in classifying job losers and job

leavers as either displaced or not displaced. And, as findings from the three

evaluation methods suggest, respondents and interviewers often struggle with

these questions. Potentially ambiguous terms (e.g., “job”, “insufficient work”) are

not defined for respondents; other constructions (“or another similar reason”)

actually contribute to response and measurement error by channeling potential

cases of displacement out of the appropriate question stream. Also, with regard to

multiple separations, of persons for whom a displacement reason was given in

SD2, about 21% lost or left more than one job during the reference period.

However, the read-as-necessary instructions provided in SD2 are transparent to

the respondent—even when the interviewer does read these instructions, it is not

clear that they would help the respondent select the correct job on which to report.

To reduce burden, the sponsor should consider developing a simple screener item

that removes persons from the question stream who did not lose or leave a job

during the reference period. Each survey question should have an explicit

question objective that relates to the overall goal of the survey. All key terms

should be defined, and those that might be ambiguous for the respondent should

be defined within the question itself.