A Propensity-Adjusted Interviewer Performance Indicator

Fair evaluation of interviewers based on their cooperation rates is complicated in most surveys. depending on the mode(s) of data collection and the methods used to assign sampled cases to interviewers, different interviewers often work cases that are more or less difficult to interview. Further, in centralized telephone facilities, interviewers are given cases affected by the results of earlier attempts by other interviewers. This paper proposes and evaluates an interviewer performance indicator that attempts to repair these weaknesses and can be used in all data collection modes involving interviewers. For each contact made by an interviewer, the deviation between the outcome of that contact (1 = successful interview, 0 = other) and the predicted probability of a successful interview for that contact (according to auxiliary information and survey paradata) is recorded. each interviewer then receives a score that is the mean of these deviations across all contacts. This performance indicator gives larger credit to the interviewer who obtains success on difficult cases versus relatively easy cases. The indicator also gives smaller penalties to failures on very difficult cases. We analyze face-to-face and telephone contacts from three different surveys to illustrate the computation Brady T. West is a research assistant Professor in the survey Methodology Program (sMP), survey research Center (srC), Institute for social research (Isr), University of Michigan, ann arbor, MI, Usa. robert M. Groves is the Provost at Georgetown University, Washington, dC, Usa. Work on this paper by robert Groves was completed while he was director of the survey research Center at the University of Michigan–ann arbor. The authors thank James Wagner, richard Curtin, Grant Benson, and Joe Matuzak for access to and assistance with the survey of Consumer attitudes data, the National Center for Health statistics for access to the National survey of Family Growth data [NCHs Contract 200-2000-07001], and their colleagues at the Institute for employment research (IaB) in Nuremberg, Germany (especially dr. Mark Trappmann and daniela Hochfellner) for access to and assistance with the Pass data (made possible through the IaB research data Center established at the University of Michigan–ann arbor). The authors also thank the National science Foundation for grant support to pursue this idea [ses0207435 to r.M. G. , Principal Investigator]. The authors do not have any conflicts of interest. *address correspondence to Brady T. West, survey Methodology Program, P.o. Box 1248, 426 Thompson street, ann arbor, MI 48106, Usa; e-mail: bwest@umich.edu. Public Opinion Quarterly, vol. 77, No. 1, spring 2013, pp. 352–374 doi:10.1093/poq/nft002 advance access publication 11 april 2013 at Jrdan U niersity of Sence nd T ecology on July 6, 2016 http://poqrdjournals.org/ D ow nladed from of this indicator and its properties. We find that calling-history paradata are the strongest predictors of obtaining interviews in both modes (especially for face-to-face contacts), and produce propensity-adjusted performance indicators that more effectively distinguish among interviewers.