EVAUJATING INTERVIEWER STYLE FOR C~G 

INDUSTRY AND OCCUPATION INFORMATION 

David Cantor, Westat, inc. 

James L. Esposlto, Bureau of Lmdimr Statistics 

Key Words: Probing, Cognitive Methods, Coding 

1. lntroductlem 

The Current Population Survey (CPS) is one 

of the primary sources of industry and occupation 

(1/0) data for the general population. It is the 

only source of these data that can provide current 

labor force and employment status information for 

particular I/O groups. Similarly, it is one of the 

few micro-data files that often analysts repeated 

cross-sectional estimates of I/O distributions over 

both very short (e.g., monthly) and relatively long 

(one or more years) periods of time. 

The variab'dity associated with survey-b~ 

measures of I/O has been found to be quite high. 

This variability is the cause of some concern among 

analysts of not only thc CPS, but also to 

analysts/policymakers using survey-based measures 

of I/O (BLS-NCES, 1979). For example, Collins 

(1975) found that 17% of the CPS sample changed 

3-digit industry categories between consecutive 

monthly interviews and 32% changed 3-digit 

occupations. 1 This rate of change was much higher 

than those indicated by a direct report of change 

provided by the respondent at an interview 

administered one month later (about 2.5% gross 

change for both industry and occupation). A 

detailed analysis of the verbatim responses given at 

the two interviews attributed about 70% of the 

changes in the 3-digit codes to reasons other than 

an "actual" change (e&. wording or coder 

problems). 

The purpose of the research reported below 

was to focus on I/O error related to interviewer 

style. 2 Specifically, this research had two goals: 

To better understand how interviewers 

affect accuracy of the I/O data on the 

CPS; and 

2. To suggest guidelines that can be used 

for training interviewers on the I/O 

items. 

Much of the variance in the responses to the I/O 

items stem from problems interviewers have with 

probing and recording information necessary to 

classify responses to a 3-digit category (Gacrtner, 

et al., 1989). The focus of this research was to 

better understand the scope and magnitude of 

these problems with the goal of developing better 

training materials for the interviewers. 

As tmexl in this paper, two specific interviewer 

behaviors related to the collection Wocess will be 

d~usse.d (E,sposito, 1990): (1)the record/rig of 

/nfonnat/m provided by the respondent in reply to 

specific questions posed by the interviewer, and 

(2)pmb/ng by the interviewer when inadequate 

informatim is provided by the ~dcnt. These 

two aspects of the process can have substantial 

impacts on an stages of the collection of I/{3 data. 

For czamplc, if the interviewer does not probe 

properly, the respondent may provide inadequate 

or erroneous information. Similarly, if the 

interviewer does not record the information 

properly, the coder will be unable to correctly 

classify the case. 

2. Research Questions and Deslp 

In order: to address both of these aspects of 

interviewer behavior, individual ~ns with nine 

Census coders were conducted. In these sessions 

the coder was instructed to: 

Listen to tape recordings of CPS 3 

interviews containing the I/O quc.qions; 

Write down information from the 

responses that he/she felt would be 

necessary to code the case; and 

Critique the interviewer's questioning 

strategy. For example, was there 

information that the interviewer did not 

ask for? What kiads of prol~ could 

have been inserted to unprove the 

information providcA by the R? 

Individual sessions with the coders lasted 

apprmimately one hour. During this time, each 

coder was asked to listen to six to eight separate 

reports of I/O. Three forms were filled out by the 

coder. The first form was used to record 

infmmation from the interview that the coder 

thought was necessary for dassif3~ the case. The 

second and third forms were used to administer 

questions to the coder on the quality of the 

interview (e.g., effectiveness of probes). 

Two sex~ns were conducted simultaneously 

by two different monitors. The monitors controlled 

the tape recorder, administered debriefing 

questions and answered any questions the coders 

had during the process. Two monitors were used 

to expedit© data coll~n. 

Every effort was made to have at least two 

coders listen to the same interview. This allowed 

for calculating mcasurcs d rcliabmty across codc~ 

Some cffort was made to overlap households across 

the two monitors as welL This was less successful, 

because of time conmxaintL 

661 
Additional information aglccted during the 

original CPS imcrvicw was obtained from tbe CPS 

data-tape. This included: 

Information written down by the CPS 

interviewer, and 

The codes assigned immediately after the 

CPS interview took place. After the CPS 

interview, the data are sent to Census's 

data preparation facility in Jcffcrsonvillc, 

Indiana. The I/0 data arc axle, d at this 

facility within a short period of time. 

For proposes of discussion below, the code 

assigned from listening to the ~pe will" be referred 

to as TRC Crape Record Code). The code 

assigned during the CPS interview will be referred 

to as PIC (Phase I Code). 

3. Analysis and Results 

To investigate issues related to interviewer 

style, two research questions were addrcsseA: 

A. What is the difference between the 

interviewer and the coder dcs~ptions 

and classifications? 

B@ What comments did the coders have on 

the interviewer's probes and questioning 

stytc? 

In this section, the results that directly address each 

of these quc~ons are reported. Prior to 

these results, a general description of the sample is 

provided. 

3.1 Sample 

A total of 41 interviews were available for 

administration to the coders. Of the~ 41, 33 were 

completed with at least one coder. These 33 were 

selected out of the 41 by running through the list of 

eligible interviews in sequential order and finishing 

as many as possil~ in the one day allotted for 

collecting this information. The se, qucnce of the 41 

interviews was not, as far as we can tell, a function 

of anything related to q~nnaire version. 

Of the 33 interviews, six were listened to by 

one coder, 20 were listened to by two coders, and 7 

were listened to by three coders. 

Missing data resulted from several sources. 

F'trst, several of the tapes wcrc difficult to hear or 

inauch'ble to pm'dcular coders. Individual coders 

differed in their ability to understand and/or hear 

~~e sequences. When the coder could not 

tape, tbc data were coded as missing. 

Second, there were three household ID's that could 

not be located on the CPS data ilk. 

Supplementary information could therefore not be 

rctricveA for tbcsc interviews. We suspect this was 

~usc of transcription of the tD. And third, 

tl~rc was a single interview where the rcspoadcnt 

rchmed to provide indust~ information. 

W' h mtcr -w thc d n'butioa 

u~dO~ nd occupa~om does not span m'ouall 

NcvcrO~~ it do~ Wmide 

souse variation for a munber of dJffcrcntI/O 

situations. As one indication of this, thc overall 

rcfcrral rate for thcsc codes was ~ what 

occurred during the ~ interviews as a whole 

(17%). This would sccm to indicate that tbc 

content of tbc descriptions prescnted ~ the 

same difficulty as an average CPS interview. 

3.2 AlFeement hemeea Coders aml CPS 

Int~lew~u 

In orde~ to measure the agreement between 

the information heard by thc I/O coder and written 

by the interviewer, two codes were compared: 

(1) the code ~gncd by the I/O coder lifter 

• " to the tape (TRC), and (2)the code 

'as~at the time of the CPS interview (PIC). 

This compm'ison should provide an overall measure 

of thc loss of information that is attributable to 

transaiption of information by the interviewer. 

CompaHsoa of Detailed Codes. There are 

several ways to measure the difference between the 

~ the t~C. One such comparison is by 

up total number of diHcrcnccs 

between each TRC-PIC comkdnatioa. This 

excludes data that are missing for the reasons 

descn'lwA ~. Out of the remaining 125 

comparisons, there were 19 (15%) that disagreed 

(Table 1). Disagreement here refers to e~r:. 

(a) the codes not matching at the 3-digit level (four 

cases), .or (b)one of the codes being codeA as a 

refcrraP case (15 cases). The distribution by 

industry and occupation is relatively even. 

One way to judge this rate is to compare it to 

the verification rate derived from the CPS 

production This rate is calculated 

having two ~ coders classify a sample 

cases. The rate at which the CPS coder agrees with 

at ~ one of these _~_ ~,~,' coders ~cs 

the "verification rate." This rate of 95% is a 

measure of the overall reliability of coding for the 

UO data. Compared to this standard, the rate of 

85% is slightly smiler. 

Including "referral" as a legitimate code overestimates the rate of disagreement, since these 

represent the most dif~ficuk cases to code. s To 

extent, coders have disaetioa in ".ud~. 

cases to refer and which to try to ~. It m 

probably the case, therefore, that if these codes 

were ultimately assianed a 3-disit code, as in 

many 

aSrcc. Tim ~~m ~ mul~ b~. 

Deleting those ca~ ~4~erc the TRC 

"referred;" and 

662 
mc rode rode, dur dcodi 

process. 

When this is done, the rate of disagreement drops 

by 50%. Out of the 103 3-digit TRC-PIC 

comparisons that remain, 8% disagree with the 

tp•. Unlike the wcvious comparison, however, 

rate differs ~iQcantly by industry and 

occupation. Only I of the 8 differences arc for the 

former, while 7 are for the latter. This is consistent 

with what one would ~ since there arc fewer 

industry codes than occupation codes. 

A second way to judge thesc data is to 

compare the agreement among the TRC and PICs 

for the same interview. This controls, to some 

extent, for coding reliability. The extent to which 

the PIC is in the minority v/s a ~ what one would 

expect from the reli~ility rate of 95% is an 

indication of the effect of listening to the tape and 

collecting information that the interviewer did not 

write down. 

This comparison is rcmicted to the I/O 

reports where at least two coders listene~ to the 

tape. In this case, the PIC is in the minority 9% of 

the time (5 out of 53 pms~lc comparisons). ° This 

implies a verification rate (Le. 91%) that is only 

slightly smaller than the production verification 

rate (95%). 7 

Overall, therefore, these data do not indicate 

a large portion of error that can be directly 

attn'butable to the information transcribed by the 

I/O coder. 

Comparison d Transcribed Informatiom. 

The TRC and PIC seemed to produce 

approximately the same set of codes, at least within 

error L~ociated with coding reliability. To further 

c~lore differences, a detailed comparison of the 

trans~ptiom and debriefing information was made 

for those 19 cases where the TRC and PIC 

differed. This comparison was completed by a 

w~ I/0 coder who mcmlxcd to pinpoint if the 

difference in the TRC and PIC was due to what the 

I/O coder heard or simply to an individual decisim 

by one of the coders. To the extent that it is due to 

what was heard, one can attribute the difference to 

problems with what the interviewer had originally 

transcn'tw~ when mndncti'~ the interview. 

tr Lhis~ In com~ it is assumed that the 

provided by the I/O coder adequately 

.~prescnts what the I/O ct~r thought was 

portant to dassify the case. If information 

imporumt to classifying the case was not wrincn 

down by the coder, this analysis will mistakenly 

~utc the difference to an individual decision by 

the coder, rather than to something in the interview 

itself. 

Of the 19 differences, 12 can be attn'buted to 

differences in the two tr~ptiom. For cxamplc, 

in one case the occupation code differed between 

TRC (387) and PIC (468). The wcrdi~ between 

the two rewals that the dilferem~ in line 23C: 

"school day care msbtmst" for TRC 

"afar school day care msis[tant]" for PIC 

lead to the discrepancy. 'fine TRC wording implies 

a connection to a teaching position to ms 

elementary school teacher. The PIC wording 

s " in day carc. 

are not always due to more information in the 

TRC. This is indicated by the fact that the average 

of the number of wm'ds in each of the two sets of 

descriptiom are identical (133 words)? Subde 

chanses in wording or the addition of a sin0e word 

can account for the difference, as illustrated above. 

This emphasizes the importance of the interviewer 

writing down exactly what the respondent reports, 

with as much of the original detail as pmsible. 

3.3 Comments by Coders om In~ St34e 

When listening to tapes, coders were asked to 

comment on the interviewers' probes. Xhc 

of these comments were to provide an indication of 

problems interviewers currently have with 

coilcaing information necessary to ~ the lob. 

Coders were mke£ to provide both lX~JVe 

and negative comments, i=~mini-a these data for 

the 33 intervicws played to the coders revealed that 

23 of the 33 had at le, ast one comment on either 

industry or occupation. ~ data indicate, 

therefore, that the coders conunented at least once 

for about 7O% of the interviews, Disaarqpdng by 

the amber of positive vs. negative comments, the 

former outnumbered the latter by slightly less than 

2 to .1.(m vs. 11). 9 This last result is somewhat 

surpmmg. It w~ o~imdly m~'l~~ tluU fl~ 

on the probm and interviewing sqne for these 

items. 

The alnence of many negative comments may 

have simply been a funokm of the smafi and 

sel~ nature, of the sample of interviews. We 

suspect, however, it is also related to the difference 

between the coder's desire to easily dmsify the case 

and a concern with trying to come up with the 

"corrcct" codc. This difference is relatcd to the 

quality control procedures used by the Census 

when assigning I/O codes. These procedures 

consist of having a sample of cases classified by 

three coors. One of these coders classifi~ the 

case as part ofthe normal production ofthe Cir. 

The other two coders classify the case after the 

production coding is completed. To monitor 

qual~., the three codes are compared. Coders arc 

penalized ~41enever a code dkagrees with the 

majority. There is no che~ on whmher the 

majority code is actmdly "corre, cL" 

This process conditions the coders to classify 

the case into a category that is most lilly to agree 

with what other coders may aui~ Conflio~ 

663 
infonnatim (e.g., between the job name and major 

activities) makes the coder's job more difficult, 

since it le, avcs dm~ as to how another coder would 

•mify the case. For this mason, the coders would 

to discourage the interviewers from probing 

too much. The more information provided by the 

respondent, the greater the chance that there will 

be conflicting information. When listening to the 

tape, therefore, coders were usually content not to 

comment, as long as the information provid~ was 

enough to classify the case under eJ~Jn8 rules, 

There was a clear tendency for the coders to 

discourage probing any responses that may have 

been ambiguous with regard to what the 

respondent was saying, but still "codcable." 

Notwlth~andlne this last ob~rvaticm, these 

data provide some indication that much of the 

variability observed in the I[O data cannot easily 

be corrected by trying to imwove interviewer's 

prob if 

rs cannot provide useful probes, then it 

would seem unr~ to ~ interviewers to do 

so. The interviewers arc not familiar with the 

e!~tifi~tion system and the task of generating 

pr~ is, apparently, quite difficult even for those 

familiar with the cla~sificstion system. 

At tbc very least, these results indicate that it 

would require an c~cmcly skillful interviewer, 

well acquainted with the I/0 system, to elicit both 

reliable and valid information. S~tially 

reducing error variability in the I/0 has to come. 

from .introducing structured questions that arc 

consistent with how respondents think about and 

dcsm~ their jobs. 

lk~ general comments and the specific ones 

related to the audio tapes can be divided into two 

groups. The first consist of comments where the 

interviewer should have tried to collect more 

specific information. Examples indude: 

Did not ask whether the company was 

whok.~e, retail or something else; 

Did not probe for specific information 

on the type of ind~ 

Information providexl on occupation type 

(the "C" line) was not sufficient for 

coding pmpmes. The rcsponsc citer 

refene4 to a general type of occupatim 

or to aaivities within the occupation; and 

TIM: major activities were not specific 

cnou~ 

The second comment refers to situations 

where the information wovidcd by the respondent 

is missing something that is ~ to 

unambiguously classif3dns the case. In most of 

these cases, ~ver, the interviewer is not in a 

position to know what information is required. For 

example, in one instance, the respondent described 

an industry as "kitchen ranscs- n:nufact.rins." In 

order to code the case, however, information ms 

whether the ranscs are commercial or household 

was n~ 

With regard to the third comment, several 

interviewers said that inadequate occupation names 

a major problem. This frustration stems from 

the ~~.~ fact ~. that the ~r~. rides give this ~ the 

are instructed to try to 

code the occupation from this line and use the 

information on thcaa~cs line when necessary. 

If no clear occupation name is wescnt, however, it 

is difF~alt for the coder to find a proper code. In 

response to this frustration, several coders 

suggested that interviewers be instructed to ask the 

respondent for an occupational title. This would 

ovide the coder with aname that could be 

ed among the eligible codc~ 

The fourth comment refers to descriptions of 

activities that do not provide the coder with 

adequate information to code a case. When this 

comment was made by a coder, it typically 

consisted of asking for more specific information 

about the content of the job (similar to the second 

comment ~ on industry). 

The second type of problem relates to the 

interviewer asking for too much informatioa. 

Examples include: 

Conflicting information on the major 

activity line was not resolved. 

The interviewer asked too many 

questions. She did not stop when there 

was enough information to code the case. 

An extreme example of the coders objection 

to too much information is illustrated in their 

general comments that claimed that asking for 

major activities hurts as much as it assists in the 

coding proce~ ~ ruder, thoesht that since 

most of the information required to code a case 

comes from the name of the oc~pafio~ 

interviewers should coaccntratc on getting good 

information here. The activity line commonly 

co~ with the infomatioa on ocmpatim type 

and makes it more difficult to code the case. 

4. Discussion and Recommendations 

In this seaioa, we discuss the r~ of our 

research rclativc to the two goals stated in the r~ 

seaim of the paper (i.e., to better eaderuaad how 

interviewers affect the accuracy of IlO data 

collected in the CPS, and to susscst guidelines that 

can be used for traini~ intcr~wcrs m the I/O 

items). 

4.1 Tnd,lng l.t~=~i,,~~ to Ask I/¢3 

The vast majority of the data-quality and 

coding problems mentioned by the coders during 

this research can be traced tothe interviewer's hck 

of faminarity with the coding rute. used by mden 

664 
mul with the inhcrcnt comldcsitics of the Ccnsm 

I/O el~ttiFu.~tlon system. H interviewers were 

aware of tbc coding rulcs, perha~ they could 

eliminate some of these m~ m ~ptiom. 

Five specific recommendations were made to 

accomplish this. 

R1. Train interviewers to filter as little of the 

resvome as ~sss'olc. 

Record as much tnfomsa~ as po~:e 

(~ reason). This indudcs the exact 

words useA and, when posu'ble, the order 

fig words arc spoken. Techniques that 

might assist the interviewer arc: 

Be rcady to ~ in the answer m 

soon as the respondent speaks, and 

To slow the respondent down when 

he/she is going too fast. 

Given the fact that, on average, interviewers 

and coders recorded approximately 13 words in 

response to the two occupation questions, this does 

not appear to bc an unreasonable task. 

Repeat back what/s~ dow. for each 

This' .dcs vcrify the 

spelling of namcs or technical terms used 

in the desaipfion. 

This re, commendation has to be balanced 

against several competing constraints. Fu~ 

interviewers have very little space in which to write. 

Sccos~d, the use of dependent interviewing places a 

demand on co~ctin S coherent dcscriptiom that 

can be easily read by the interviewer at future 

interviews. Both of these comtraints require the 

interviewer to filter information when recording 

I/O data. Nonetheless, both of the 

procedural recommendations should improve the 

R's ~d~ility to capture ~ information as the 

respondent dcscnl~s his/her job. 

arc: 

R2. Train intcl~ewers on the imoortancc of 

obtal-l-a an occunation name. 

Probes that /nay be tiscfid for this purpose 

"What do you cart your job?" 

"What is the name of your occupation?" 

(proxy) "What does (he/she) generally 

can (his/her) job?" 

R3. Give interviewers molgs to use when 

IkJir~ multin!c a~es. 

In the last six mon_tk~ which of these 

activities did you do most of the time? 

whi arc your prima activitics? 

R4. Train interviewers as coders. 

A more effcaivc, but ~~~~1 

acquaint interviewers with 

be to tndn them as axlcrs. This was actually 

suncstcd by several 1/¢3 coders during this 

research. The training would not SO into thc more 

derailed rules I/0 coders have to Icam (c4~ whcn 

to refer a case, vcrificmion procedures, using 

amiiiary lists). It would simply wovidc them with 

the besk procedures coders go through and then 

have them classify a number of cases using these 

inks. 

RS. Provide interviewers a reference 

dncnment to *ttltt. in _nrobina_ 

The primry concern here wou~ be that it 

would make the interviewees task more 

complic~ed. Looking up industry and occupation 

names while ~ on the phone is not 

straightforward, tt If a list is constructed, it would 

have to be relatively short and easy to use. 

4.2 Understanding Interviewer Errtw 

While this re,w.~ch was based on a relatively 

small number of cases, it does provide additional 

• " " problems related to collecting 1]¢3 

it was "" At the outset, havcan °dsimdlyumbcr of that coders would 

comments mt relcvam probes that should bc use~ 

This did not turn out to be the case. In fact, there 

were more pos~vc remarks abom the interviewing 

s~dc than nc~Mivc. In suppoN of this qu~ta~vc 

observation was the fact that a ~ majority of the 

identical to tho6c tmigncd at the 

interview. 

The small number of comments by the coders 

reflects, in park tl~ difference between coding 

reliability and validity. This high agreement 

between axlcrs and interviewers, however, also 

re~ the fact that scncrating the correct wobc is 

a very difficult coSnkivc task. The I/O system is 

very comp~ and cvcn though the coders are 

with the system, supplyi~, the carrect 

probe requires complex comparmm between 

different types of axles. The implication is that it 

is probably asking too much of interviewers to 

[generate pr~ without providing more structure 

m the questionnaire. As the questionnaire 

~es automated, it is posrdl~ to accommodate 

this structure by developing more complicated 

brtnching systems. This system, for cnmpk, may 

use key words to provide structured probes to the 

interviewer. 

665 
ltefm, emc~ 

BI~NCES (1g79). Feasibility Study For a Proieft 

on ImDrovement of Occupational Information. 

Nation~ Occupatimud ~nfomafiou CoorcUnat~ 

Committee. Performed under Contract Number 

AO-A01-788-00-~ in the U~. Department of 

Commerce. Published jointly by the Bureau of 

Statistics and the National Center for 

Education Statistics. 

Cantor, D and K. Levin (1992). Summary of 

Activities to Evaluate the Depemdeut imtervtm.l~ 

Preeedure em the C~t Populatlem Survey. 

F'mal Report submitted to the Bureau of Labor 

statistics, Contract No. J-9-j-S-00S3. 

C. (1975). "Comparima of Month-toMonth Chanses in Industry and Occupation C.x)de.s 

with Respondent's Report of Change: CPS 

Mobility Study." Response Research Staff Report 

75-5, May 15, 1975, U.S. Bureau of Census. 

Esposito, Jim (1990) "Resolving the Gross Flow 6 

Problem (R~ch on the I/O Series)." Memo 

dated December 7, 1990 to Alan Tupek, Mike 

Welch and David Cantor. 

Gaertner, G.H. Cantor, D. and N.L Gay. 

Imeareh oa ludum? and Oeeupaam Quemiom 

In the Current Populatlem Survey. F'mal Report 

submitted to the Bureau of Labor Statistics, 

Contract No. J-9-j.8-0083. June, 1989. 

Rothgeb, J., Polivlm, A., Creighton, IL and S.R. 

Cohany (1991). "Development of the Proposed 

Revised Current Population Survey" pp. 56-65 in 

1991 Proeeedlnp of the Sectloa em Survey 

P.emmr~ Methods, American Statistical 

Association, Alexandria, VA. 

Footmetes 

1 This research was conducted as part of a 

larger research project to axxist in the redesign of the Ce'S. (Camor and Le~ 1992.) 

2 Changes in the major I/O groups were about 

half these rates. 

3 The CPS interviews were conducted as part of 

Phase I of the CPS redesiga (Rothseb, et aL, 

1991). Three differeat venk~ of the CPS 

were _admint~cred as t~rta~ 

three were used in 

below. 

4 Referral cases are those interviews where the 

coder could not classify the case. These cases 

are then "referred" to a senior coder when 

arriving at a fml code. 

5 The verification rate for codes referred is 

around 85%. 

There was one case where the PIC asreed 

with one ~This~ 3 TRCs assigned to the 

interview, was not include4 in the 5 

minority code~ 

7 A second way to view this is by comparing all 

TRC's that were coded for the same case. 

Out of a total 104 such comparimm, there 

were 12 disagreements (11%). 

8 When there was a refusal to answer the 

question or there stmply was not any 

information, nothing was counted as a word. 

9 Of the 23 interviews that had at least one 

comment, 8 had both positive and negative 

comments. 

10 This is to be dLei" .,ujnld~l from structured 

probes that are written into the q n~_~nalre. 

11 This was evident from our ex~rience trying to 

train V O coders to das~ respondents while 

talking to them over the phone. 

Table 1. Number of Cx)des Assisned from the Tape Recording by Industry/(k, cupation and Agreement 

with Codes Assigned from the Phase I Interview* 

Am'ccd with Phase I Code? 

No Yes Total 

Industry 8 54 62 

(13%) (87%) (100~) 

Occupation U 52 63 

(17%) (83%) (100~) 

Total 19 106 125 

(15%) (85%) (100~) 

* The tape recordin8 code disagreed with the Phase I code when ~ (1) one was referred and the other was not, or 

(2) the tape recording code differed from the non-referred Phase 1 code. 

666