© 2021 Ipsos 1

Blind to the Consequences of Measurement?: 

Response Format Effects on Self-Reported Disability

Megan A. Hendrich, Randall K. Thomas, & Frances M. Barlas

Ipsos Public Affairs

2021 FCSM Research & Policy Conference

November 2nd – November 4th, 2021
© 2021 Ipsos 2

When measuring the occurrence of a series of events or behaviors, two 

common response formats have been used to indicate event occurrence: 

Multiple Response Formats (MRF) and Yes-No Formats (YNF).

Study Background

Do you like any of the following 

flavors of ice cream? Please select 

ALL that apply.

❑ Vanilla

❑ Chocolate

❑ Strawberry

❑ Rocky-Road

❑ Mint Chocolate Chip

❑ None of the above

Do you like any of the following 

flavors of ice cream?

Yes No

Vanilla ○ ○

Chocolate ○ ○

Strawberry ○ ○

Rocky-Road ○ ○

Mint Chocolate Chip ○ ○
© 2021 Ipsos 3

With paper-pencil surveys, YNF have been found to result in higher 

endorsement rates than MRF (Ericson & Nelson, 2007; Rasinski et al., 

1994).

Similarly, in a series of five online experiments utilizing both U.S. and 

international samples of adults, Thomas and Klein (2006) found 

consistent evidence that YNF led to higher endorsement rates than MRF 

for a variety of topics:

▪ Protest behaviors against companies

▪ Eating different food products 

▪ Purchase of products at convenience stores

▪ Product ownership

Study Background
© 2021 Ipsos 4

In another study with one mail and two online surveys, Smyth et al. 

(2006) compared MRF and YNF using student samples at Washington 

State University.

Smyth et al. assessed attitudinal, factual, and behavioral issues 

regarding students’ experiences with the university and replicated the 

response format effects, with YNF yielding higher endorsement rates 

than MRF.

Study Background
© 2021 Ipsos 5

Disabilities and health-related impairments have been conceptualized 

and measured differently in national surveys, both at individual and 

household levels (e.g., National Health Interview Survey, Survey of 

Income and Program Participation, Census). 

The prevalence of impairment as established by self-report surveys can 

often significantly impact policy and resource allocation decisions. 

Therefore, we need to develop measurement formats that are both 

highly reliable and valid.

Study Background
© 2021 Ipsos 6

Since we often need to screen for impairment in health-related surveys, 

we were interested in how response formats might influence the 

prevalence of disabilities and impairment by type of functioning and 

severity of impairment.

Specifically, many online surveys use MRF to quickly assess a wide range 

of characteristics, often to identify people of interest for future surveys, 

such as people with specific health-related conditions. 

We suspected that the findings demonstrating higher endorsement rates 

for YNF would carry over to the study of disability prevalence.

Study Background
© 2021 Ipsos 7

Method
© 2021 Ipsos 8

Study 1:

Respondents included 40,717 U.S. adults recruited from Harris Poll 

Online, a non-probability opt-in panel provider. Respondents were sent 

an email invitation to participate in a web-based questionnaire. This 

study was fielded from November 2005 to June 2007.

Study 2:

Respondents included 2,342 adults from Ipsos’s KnowledgePanel®, the 

largest probability-based online panel in the U.S. This study was fielded 

from October 2019 to November 2019.

Method
© 2021 Ipsos 9

In this experiment, respondents were randomly assigned to one of two 

response formats: 

▪ A Yes-No Format (YNF)

▪ A Multiple Response Format (MRF)

Method
© 2021 Ipsos 10

The YNF and MRF both used the following question stem and items:

Do you have any of the following conditions?

1) Blindness or a severe visual impairment in either eye

2) Deafness or hard of hearing in either ear

3) A long-lasting condition that substantially limits one or more basic 

physical activities, such as walking, climbing stairs, reaching, 

lifting, or carrying

4) A long-lasting physical, mental, or emotional condition that 

increases the difficulty of learning, remembering, or concentrating

Method
© 2021 Ipsos 11

Study 1 utilized a grid format for the YNF, presenting the item to the left, 

with radio buttons to the right in columns with ‘Yes’ and ‘No’ labels.

Study 2 utilized a banked format for the YNF, which included ‘Yes’ and 

‘No’ buttons horizontally arrayed underneath each item.

In both studies, the MRF had an additional respondent instruction 

following the question stem: “Please select ALL that apply.”

Additionally, the MRF had a final response of “I have none of these” for 

respondents who did not have any of the listed impairments.

Method
© 2021 Ipsos 12

After answering the initial impairment prevalence items, if respondents 

indicated that they had an impairment, they were asked follow-up questions 

to assess the extent of their impairment (for visual or auditory impairment, 

we also assessed the extent of impairment for left/right eye/ear).

Each impairment severity question had a four-category response format:

Method

1) Not at all impaired

2) Some visual impairment

3) Severe visual impairment

4) Blind

1) No hearing loss

2) Some hearing loss

3) Severe hearing loss

4) Totally deaf

1) Not at all impaired

2) Some physical impairment

3) Severe physical impairment

4) Total loss of function of the affected area(s)

1) No difficulty at all

2) Slight/Mild difficulty learning, remembering, or concentrating

3) Moderate difficulty learning, remembering, or concentrating

4) Severe difficulty learning, remembering, or concentrating
© 2021 Ipsos 13

Results
© 2021 Ipsos 14

Results – Time to Complete

In both studies,

those in the YNF

took more time to

complete than

those in the MRF.

0

5

10

15

20

25

30

Study 1 Study 2

Seconds

Yes-No Format Multiple Response Format
© 2021 Ipsos 15

Results – Prevalence The YNF led

to higher prevalence

rates across all four

types of impairment

and with at least

one type of

impairment.

0%

5%

10%

15%

20%

25%

30%

Visual Hearing Physical Cognitive At least 1

Prevalence

0%

5%

10%

15%

20%

25%

30%

Visual Hearing Physical Cognitive At least 1

Prevalence

Study 1 Study 2
© 2021 Ipsos 16

Results – Severity of Impairment (Study 1)

0%

10%

20%

30%

40%

50%

60%

Left Eye Right Eye Left Ear Right Ear Physical Cognitive

Severity

Yes-No Format Multiple Response Format

Respondents were

somewhat more likely

to indicate a severe

impairment when using

the MRF compared

to the YNF.
© 2021 Ipsos 17

Results – Severity of Impairment (Study 2)

0%

10%

20%

30%

40%

50%

Left Eye Right Eye Left Ear Right Ear Physical Cognitive

Severity

Yes-No Format Multiple Response Format

We found minor

differences in severity

due to response format,

but the pattern did

not fully replicate

Study 1.
© 2021 Ipsos 18

Discussion
© 2021 Ipsos 19

We replicated past research results for topics as diverse as consumer 

purchases, product ownership, and employment. We found consistent 

and significant effects for response format: a YNF led to higher 

prevalence estimates than a MRF.

The key experimental findings were replicated in both an opt-in sample 

and a probability-based sample.

The effects of response format were quite dramatic for the lowest 

prevalence impairment—for visual impairment, a YNF yielded a 83% 

higher prevalence (5.3% vs. 2.9%) in Study 1 and a 63% higher 

prevalence in Study 2 (4.4% vs. 2.7%).

Discussion
© 2021 Ipsos 20

One caveat of this study is that disability prevalence can be affected by 

mode of survey administration:

▪ Those with visual disabilities may be less likely to complete 

visually administered questionnaires (online or paper)

▪ Those with auditory disabilities may be less likely to complete 

orally administered questionnaires (telephone or in-person)

Our future research is also comparing the use of filter questions with 

YNF versus going straight to a graded impairment scale without a filter.

Discussion
Megan A. Hendrich

Megan.Hendrich@ipsos.com

Thank you!