How Training Affects Interviewer 

Performance Over time: A Field 

Experiment with A Large-scale 

National Representative Survey

Hanyu Sun, Angie Kistler, Ryan Hubbard, Brad Edwards, 

Marcia Swinson-Vick, Westat

1
Disclaimer

The statements in this presentation are those of the presenter and do not 

necessarily represent the views of the Agency for Healthcare Research 

and Quality (AHRQ) or the Department of Health and Human Services 

(DHHS).

2
Background

❯ Interviewer contributes to all types of error arising from the survey 

process (e.g. West and Blom 2017).

❯ Interviewer training is often conducted with the intent of reducing the 

interviewer related error in standardized interviews.

❯ Findings from previous research on how training affect interviewer 

performance are mixed:

• Gaining cooperation (e.g., Groves and McGonagle 2001, O’Brien, 

Mayer, Groves and O’Neill 2002, Schnell and Trappman 2006).

• Collecting high quality data from respondents (e.g., Cannell, Miller 

and Oksenberg 1981, Billiet and Loosveldt 1988, Fowler and Mangione

1990).

3
Medical Expenditure Panel Survey (MEPS)

❯ MEPS is the nation’s primary source of nationally-representative 

information on medical expenditures, health care utilization, and health 

insurance coverage, conducted since 1996.

❯ Design:

• Longitudinal overlapping panel survey that interviews US civilian, 

noninstitutionalized population approximately every 6 months for 5 

rounds.

• Subsample of 15,000 households each year participating in the 

previous year’s National Health Interview Survey:

-Most interviews are done in-person, with some small proportion by 

telephone (in non-Pandemic years).

• One person reports on personal characteristics and health care use for 

all other family members and receives $50 for completing each round.

4
Field Interviewer Training Experiment

❯ Conducted a field interviewer training experiment in fall 2019: 

• 250 experienced MEPS-HC field interviewers.

• The purpose of the training was to refresh experienced interviewers 

on two skill sets: 

-Gaining cooperation

-Collecting high quality data

• Three training modes: 

-In-person training

-Videoconferencing training

-Self-administered and self-paced training 

5
Interviewer Assignment

6

Training Mode

Data Quality

Gaining 

Cooperation High Mid Low

In-person 

Training 

(n=105)

High 10 13 5

Mid 13 25 14

Low 4 14 7

WebEx Training 

(n=58)

High 6 7 2

Mid 7 14 8

Low 2 10 2

LMS Training 

(n=79)

High 9 11 1

Mid 8 20 13

Low 1 11 5

Total 60 125 57

Table 1. Interviewer Assignment by Pre-identified 

Performance Groups

❯ For each interviewer, 

computed two baseline 

performance measures using 

data from the preceding 

round of data collection: 

• A composite score on gaining 

cooperation

• A composite score on data 

quality
Training Mode: In-person

❯ In-person training: 

• A two-and-a-half day training including 24 modules. 

• A combination of long and short lectures, large group discussion,

small group exercise, and Computer-Assisted Personal Interviewing 

(CAPI) hands-on practice. 

• Assigned interviewers into small groups to complete exercises: 

-Each group was a mixture of pre-identified high-, mid-, and lowperformers to prompt peer learning. 

• After training, collected feedback from interviewers via a web survey.

7
Training mode: Videoconferencing 

❯ Videoconferencing training: 

• A single two-hour WebEx session that covered 2 modules: 

-Provider Search (data quality)

-Hard Copy Collection (gaining cooperation)

• Each session had about 10 interviewers with varied pre-identified 

performance.

• Training staff kept interviewers engaged during the session.

• After training, collected feedback from interviewers via a web survey.

8
Training Mode: Self-administered LMS 

❯ Learning Management System (LMS) training:

• The LMS manages and delivers assigned electronic training and 

documentation in a browser environment.

• Self-administered and self-paced training that covered 2 modules: 

-Provider Search (data quality)

-Hard Copy Collection (gaining cooperation)

• No interaction with training staff.

• After training, collected feedback from interviewers via a web survey.

9
Statistical Approach

❯ Outcome measure:

• Provider match rate = the number of matched providers / the number 

of eligible providers

❯ For the same interviewer, the provider match rate was computed both 

before the training and after the training (i.e. at the end of the filed 

period).

❯ Fit marginal linear models to examine how provider match rates change 

over time.

10
Results: All interviewers

11

Figure 1. The LS Means and 95% confidence intervals for the 

adjusted differences before and after training by training mode

❯ Fit a marginal linear model to 

predict the provider match rate 

by time (after vs. before 

training), training modes, and 

the two-way interactions 

between time and training 

modes for all interviewers.

❯ Significant improvement on the 

provider match rate before and 

after for interviewers trained in 

WebEx.
Results: High-performers

❯ Fit a marginal linear model for 

high-performers only.

❯ For pre-identified highperformers, there was 

significant improvement on 

the provider match rate before 

and after the training if they 

were trained in WebEx.

❯ No significant improvement 

before and after training for 

high performers trained either 

in person or in LMS.

12

Figure 2a. The LS Means and 95% confidence intervals for the 

adjusted differences before and after training for high-performers
Results: Mid-performers

13

Figure 2b. The LS Means and 95% confidence intervals for the 

adjusted differences before and after training for mid-performers

❯ Fit a marginal linear model for 

mid-performers only.

❯ Marginally significant 

improvement before and after 

the LMS training.

❯ Neither in-person nor WebEx 

training had significant effects 

on performance for midperformers. 
Results: Low-performers

14

Figure 2c. The LS Means and 95% confidence intervals for the 

adjusted differences before and after training for low-performers

❯ Fit a marginal linear model for 

low-performers only.

❯ Marginally significant 

improvement on provider 

match rate before and after 

the training for those trained 

in WebEx.

❯ No significant improvement on 

the provide match rate for 

low-performers trained either 

in person or in LMS. 
Results: Interviewer Feedback

Training Mode 

Debriefing Questions InpersonWebEx LMS

Overall experience as excellent, very 

good or good (%) 100 95.6 97.6

Learned a lot of new information on 

collecting high quality data (%) 48.9 33.3 36.6

A lot of the materials can be applied to 

cases to get better data quality from 

the respondents (%)

55.6 60.0 68.3

Very confident in collecting high 

quality data in more challenging 

situations after training (%)

48.9 55.6 41.5

n 90 45 41

15

❯ Regardless of training mode, 

the vast majority of 

interviewers rated their 

overall experience as 

excellent, very good or good.

❯ No significant differences on 

responses to debriefing items 

between training modes. 

Table 2. Interviewer’s responses to the debriefing 

items by training mode. 
Summary and Discussion 

❯ Improvement on the provider match rate for interviewers trained in 

WebEx:

• Significant improvement for interviewers pre-identified as high 

performers, and marginally significant improvement for interviewers 

pre-identified as low performers. 

❯ Interviewers trained in the three modes provided similar feedback about 

their training experiences.

❯ Training interviewers via videoconferencing is a promising method that 

deserves further consideration.

• Maintains the interaction between the training staff and interviewers

• Requires extensive preparation

16
Thank you!

Hanyu Sun

HanyuSun@westat.com

17