An Automated Refusal Conversion

Strategy for Web Surveys

Taylor Lewis1 and Mark Gorsak1, U.S. Office of Personnel Management

Naomi Yount, Westat

1

Federal Committee on Statistical Methodology (FCSM)

Research and Policy Conference

Washington, DC

March 7, 2018

1The opinions, findings, and conclusions expressed in this presentation are those of the authors and do not

necessarily reflect those of the U.S. Office of Personnel Management.

2The authors would like to thank Laurie Schwede, Jennifer Childs, Gerson Morales, and Paul Beatty from the U.S.

Census Bureau and Andrea Contratto from the U.S. Economic Development Administration for their input and 

feedback in the development of the opt out survey.
I. Background

– Federal Employee Viewpoint Survey (FEVS)

– Refusal conversion

II. 2017 FEVS Opt Out Experiment

– Motivation and Experimental Design

– Key Findings

III. Summary and Ideas for Further Research

Outline

2
Background on the FEVS

3

• The Federal Employee Viewpoint Survey (FEVS) is an annual, Webbased survey of full- and part-time, permanent federal employees 

administered by the U.S. Office of Personnel Management (OPM)

• As of 2017 FEVS: stratified, single-stage sample design of ~1.1M 

individuals from over 80 agencies → response rate around 45%

• Personalized survey link sent via email, with five weekly reminders 

sent to nonrespondents – six week field period in all

• Instrument consists mainly of attitudinal items (e.g., perceptions of 

leadership, job satisfaction) on a Likert-type scale, but also captures 

about a dozen demographics
Background on Refusal Conversion

4

• Ample evidence that refusal rates to surveys are increasing, in turn increasing 

risk of nonresponse bias:

– Groves and Couper (1998)

– Atrostic et al. (2001)

– de Leeuw and de Heer (2002)

– Curtin et al. (2005)

– Brick and Williams (2013)

– Dutwin et al. (2014)

• Survey organizations typically attempt refusal conversion for intervieweradマiミisteヴed suヴ┗e┞s ふofteミ e┝Iludiミg さhaヴdざ ヴefusalsぶ, ┘ith suIIess ヴates of 

10 – 30%

• Not as straightforward to do in self-administered surveys because of 

ambiguity differentiating a refusal from other forms of nonresponse
Offering a Way to Opt Out

5

• Argument in literature (e.g., Sudman, 1985; Mullen et al., 1987) that 

offering the respondent a way to opt out engenders trust and 

empathy with researcher, has potential to increase likelihood of 

participating

• In similar vein, Anderson (2015) argues administrators of online panels 

should abide by CAN-SPAM Act or 2003 statute requiring unsolicited 

emails to contain a visible unsubscribe link

• Our idea: use the opportunity to opt out of Web-based FEVS via link in 

email invitation that launches a short survey with two purposes:

1. Ascertain why the individual has chosen not to respond

2. Attempt a last-moment appeal (i.e., refusal conversion) based on the 

nonresponse reason cited
Opt Out Experimental Design

6

• Approximately 10% of 2017 FEVS sample (small/independent agencies 

excluded) was designated for opt out, with a link in initial invitation 

aミd ヴeマiミdeヴs laHeled さCliIk heヴe if ┞ou aヴe Ioミsideヴiミg ミot 

paヴtiIipatiミg iミ the FEV“ざ

• Opt out link not present for those not designated for experiment
Opt Out Experimental Design (2)

7

• Upon clicking on the opt out link, respondent is taken to following 

landing page:

• Purpose of this question: gauge nonresponse conviction level (i.e., a 

proxy for hard vs. soft refusal)

• Regaヴdless of aミs┘eヴ, all iミdi┗iduals see the follo┘iミg ケuestioミ...
Opt Out Experimental Design (3)

8

• After answer this question, a predetermined 25% of individuals 

receive a confirmatory message that official FEVS emails will stop

• Complementary 75% of individuals given last-moment appeal tailored 

to the response given

• Foヴ e┝aマple, if さI aマ too Hus┞ to take the suヴ┗e┞ざ is Ihoseミ, the 

iミdi┗idual sees the follo┘iミg...
Opt Out Experimental Design (4)

9

• さOtheヴざ ヴespoミses received generic appeal, and write-ins were 

independently coded by two team members; 128 differences reconciled

• Research objective: quantify the effectiveness of last-moment appeal (i.e., 

conversion rate) by nonresponse conviction level and primary reason cited
Opt Out Experiment Results

10

Comments:

• Surprisingly low rate of 

individuals clicking on opt 

out link (~1.5%)

• Individuals who launched the 

opt out survey were about 

twice as likely to respond to 

FEVS than opt out

• Ultimately, response rate 9 

percentage points higher for 

those who clicked on the opt 

out link relative to those who 

did not: 54.2% vs. 45.2%.
Conversion Rates by Nonrespondent Type

11

Comments:

• To be expected, conversion rate was much higher for individuals unsure about participating

• Varied success amongst the various nonresponse reasons
Conversion Rates by Appeal Type

12

Comments:

• Only shown are results where counts for both appeal types at least 20

• For all combinations, tailored appeal increased the conversion rate

• Effect higher for individuals unsure about responding relative to individuals not wanting to 

participate: 14 percentage point increase vs. 8.5

• Biggest increase for those indicating concerns over confidentiality; smallest for those believing 

results not used to change anything (only increase not statistically significant)
• Including opportunity to opt out was a net positive feature: led to 

increased response rate and a glimpse into distribution of reasons 

for nonresponse

• Surprisingly low rate (~1.5%) of individuals clicking link to opt out 

→ to capture more people, future study could consider:

– Place opt out opportunity in separate email or mode

– Different wording or location within email body

• Conversion rates varied notably by tailored appeal type,

suggesting more room for improvement in wording; focus groups 

could lead to more persuasive messaging in bullet points

Summary and Ideas for Further Research

13
Anderson, B. 2015. Why email bounces and opt-outs are good for your research panel.

https://www.qualtrics.com/blog/why-email-bounces-and-opt-outs-are-good-for-your-research-panel/ (accessed February 26, 2018).

Atrostic, B., N. Bates, G. Burt, and A. Silberstein. 2001. Nonresponse in US government household surveys: Consistent measures, recent 

trends, and new insights. Journal of Official Statistics 17:209–26.

Brick, M., and D. Williams. 2013. Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of 

Political and Social Science 645:36–59.

Curtin, R., S. Presser, and E. Singer. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion 

Quarterly 69:87–98.

de Leeuw, E., and W. de Heer. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey 

nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little, 41–54. New York, NY: Wiley.

Dutwin, D., J. Loft, J. Darling, A. Holbrook, T. Johnson, R. Langley, P. Lavrakas, K. Olson, E. Peytcheva, J. Stec, T. Triplett, and A. Zukerberg. 

2014. Current knowledge and considerations regarding survey refusals. Report prepared for the American Association of Public Opinion 

Research Council by the Task Force on Survey Refusals.

Groves, R., and M. Couper. 1998. Nonresponse in Household Interview Surveys. New York, NY: Wiley.

Mullen, P., I. Easling, S. Nixon, and D. Koester. 1987. The cost-effectiveness of randomized incentive and follow-up contacts in a national 

マail suヴ┗e┞ of faマil┞ ph┞siIiaミs.ざ Evaluation & The Health Professions 10:232–245.

Sudman, S. 1985. Mail surveys of reluctant professionals. Evaluation Review 9:349-360.

References

14