© 2022 Ipsos 1

O. Mina Muller, Randall K. Thomas, 

Frances M. Barlas, & Megan A. Hendrich

Ipsos Public Affairs

It’s a Trap!: 

Use of Trap Questions and Data Quality

2022 FCSM Research & Policy Conference

October 25th – October 27th, 2022

Washington D.C.
© 2022 Ipsos 2

Researchers often have concerns over data quality due to inattentive or unmotivated 

respondents. Various measures have been developed to assess whether respondents are 

providing accurate responses. Besides speeding and response non-differentiation, 

researchers sometimes imbed trap questions that can be used to detect when someone is 

not paying attention. A compliance trap directs a respondent to select a particular response 

(e.g., “Select ‘Somewhat agree’ for this item”) regardless of the question. We studied 

whether cleaning out respondents who fail such traps would improve data quality. In a study 

with over 3,500 completes from an online probability-based panel, respondents were 

randomly assigned to experience two compliance traps or not. We examined respondent 

reactions following their presentation and we also looked at any reduction in bias using 10 

demographic items for which we had benchmark values. We also looked at how trap failure 

was related to speed to complete (another indicator of sub-optimal behavior). For both trap 

conditions, we found that higher trap failure had a modest association with faster 

completion times. We found that there was no difference in average bias between the full 

samples versus the samples that had eliminated participants due to trap failures. It appears 

that trap failures are not as closely related to data quality as many have believed.

Abstract
© 2022 Ipsos 3

• Concerns over data quality due to unmotivated or inattentive 

respondents. 

• Optimal responding vs. sub-optimal responding.

• Several methods to identify suboptimal responses that might impact 

data quality:

• One form of sub- optimal response may not be strongly predictive of 

other forms of sub optimal response simultaneously and may be 

influenced by very different causes (Thomas, 2011). 

• Speeding

• Item nonresponse 

• Grid non-differentiation or straightlining

• Consistency checks 

• Compliance trap questions, attention checks, red-herrings

Introduction
© 2022 Ipsos

4

• Our study uses an experimental 

design to assess the relationship 

between data quality and use of trap 

questions. 

• We tested different types of trap 

questions as well as respondents’ 

reactions to trap questions. 

• Furthermore, we also tested whether 

order of response options or asking 

respondents to select a negative or 

positive response made a difference. 

Study Design
© 2022 Ipsos 5

 We conducted a study between January 14 and February 11, 2022, 

utilizing the KnowledgePanel, one of the largest online probability-based 

panels in the U.S.

 Mode: Web-based

 Sample Type: General Population 

 n = 3,539 completes.

Study Design

Straightforward

Disguised 

No Trap 

Experimental Study: 3x2x2 Design 

TRAP TYPE ORDER OF RESPONSES POSITIVITY

1

2

3

1 A-E

E-A 2

Agree/Disagree 

Sometimes True/ 

Very often true

1

2
© 2022 Ipsos 6

• Overall reaction (rated: Positive, Neutral, Negative).

• Confusion about the purpose of the question.

• Asked two similar questions to check the correlation between 

consistency and trap failure. 

• We removed the sub-optimal responses and looked at their impact to 

determine the extent of bias in the sample. 

Study Design-Continued 

REACTIONS TO TRAPS

CONSISTENCY CHECK

BENCHMARK COMPARISON
© 2022 Ipsos 7

Trap Conditions 

Straightforward Disguised 

1. To show that you are still reading these 

questions, please select the [randomly 

assigned: “Disagree” or ”Agree”] response 

below.

Strongly disagree

Disagree

Neither agree nor disagree

Agree

Strongly agree

1. Political issues are important to many people.

To show that you are still reading these 

questions, please select the [randomly 

assigned: “Disagree” or ”Agree”] response 

below.

Strongly disagree

Disagree

Neither agree nor disagree

Agree

Strongly agree

2. This is an open item to confirm correct 

response presentation – please select the 

["Sometimes true“ or “Very often true”] 

response below.

Not at all true

Sometimes true

Often true

Very often true

Always true

2. People’s experiences in the U.S. vary. This is 

an open item to confirm correct response 

presentation – please select the ["Sometimes 

true“ or “Very often true”] response below.

Not at all true

Sometimes true

Often true

Very often true

Always true

TRAP 1 

TRAP 2 
© 2022 Ipsos 8

Assessment by Trap Type 

Traps Straightforward Disguised

Failed First 7% 13%

Failed Second 3% 8%

Failed at least 

one 5% 9%

Failed both 2% 6%

More people failed the first trap compared to the second trap. 

More people failed the disguised trap compared to the 

straightforward trap. 
© 2022 Ipsos 9

Trap Type and Positive vs. Negative Response Experiment 10%10% 5%5%

DISAGREE AGREE SOMETIMES TRUE VERY OFTEN TRUE

FIRST TRAP SECOND TRAP 
© 2022 Ipsos 10

Reactions to Trap Questions –Trap 1

First Trap Question

[Political issues are 

important to many 

people.] To show that 

you are still reading 

these questions, please 

select the 

“Disagree”/”Agree” 

response below.

Strongly disagree

Disagree

Neither agree nor 

disagree

Agree

Strongly agree

4%

70%

26%

8%

72%

20%

NEGATIVE NEUTRAL POSITIVE

WHAT WAS YOU REACTION TO THE 

QUESTION?

Failed Passed

60%

31%

9%

84%

14%

2%

NOT 

CONFUSED

SOMEWHAT VERY 

WERE YOU AT ALL CONFUSED 

ABOUT THE PURPOSE OF THIS 

QUESTION?

Failed Passed
© 2022 Ipsos 11

Reactions to Trap Questions –Trap 2

Second Trap Question

[People’s experiences in 

the U.S. vary.] This is an 

open item to confirm 

correct response 

presentation – please 

select the "Sometimes 

true“/ “Very often true” 

response below.

Not at all true

Sometimes true

Often true

Very often true

Always true

10%

61%

24%

7%

77%

15%

NEGATIVE NEUTRAL POSITIVE

WHAT WAS YOUR REACTION TO 

THE QUESTION?

Failed Passed

44%44%

7%

75%

19%

5%

NOT CONFUSED SOMEWHAT VERY 

WERE YOU AT ALL CONFUSED 

ABOUT THE PURPOSE OF THIS 

QUESTION?

Failed Passed
© 2022 Ipsos 12

o Bottom 2.5% 

o Next 2.5% 

o Next 5 % 

o Next 10 % 

o Slowest 80%

Is speeding correlated with failure?

We broke down the speeders by:

Fastest

FAILED 

NONE 

FAILED AT 

LEAST ONE

FAILED 

BOTH

FASTEST 2.5% 78% 10% 12%

NEXT 2.5 % 93% 5% 2%

NEXT 2.5% 95% 3% 3%

NEXT 10% 95% 3% 1%

SLOWEST 80% 88% 8% 4%
© 2022 Ipsos 13

Consistency Check and Trap Failure 

Consistency Check 1 

All things considered, how 

satisfied are you with your life 

overall?

A. Very dissatisfied

B. Somewhat dissatisfied 

C. Neither satisfied nor dissatisfied

D. Somewhat satisfied 

E. Very satisfied

Consistency Check 2

How would you rate your overall 

life satisfaction?

A. Very poor

B. Poor

C. Fair

D. Good

E. Very Good

Only 10% not consistent

(> 0.05) Failed Trap Passed Trap

Not consistent 10.3% 11.6%

Consistent 89.7% 88.4%
© 2022 Ipsos 14

Failure by Demographics- Age, Gender, Race/Ethnicity

9%

11%

9%

14%

11% 12%

8%

16%

24%

17%

18-29 30-44 45-59 60+ Male Female White Black Latino Other

p<0.05 p>0.05 p<0.001

Among those who failed....
© 2022 Ipsos 15

Failure by Demographics- Education, Income 

23%

15%

11%

7%

18%

15%

13%

11%

8% 7%

LT HS HS Some

college

College

Plus

LT 25k 25k to 50k 50k to 75k 75k to

100k

100k to

150k

150k +

p<0.001 p<0.001

Among those who failed...
© 2022 Ipsos 16

If we remove sub optimal respondents, how much bias we are 

introducing?

• We have 4 groups of sub-optimal respondents: 

o Those who failed 2 compliance traps only 

o Those who failed 1 or 2 compliance traps 

o Those who failed the consistency trap only

o Those who either failed at least 1 compliance trap or failed consistency trap

• Removed them from the dataset one group at a time and reweighted 

the data. 

Benchmark Comparison 

Research 

Question:
© 2022 Ipsos 17

• We asked a number of questions that had national 

benchmark values: 

Benchmark Comparison 

o Right Direction

o Excellent or very good health

o Married

o Citizen

o 2 or more people living in the household

o At least 1 child under 18 in HH

o Own home

o 3 bedrooms or more in House

o Moved in house more than 5 years ago

o 2 or more vehicles one ton or less

o Has landline phone

o Speaks other than English at home
© 2022 Ipsos 18

To assess bias, we calculated the average absolute deviation:

• Reweighted the data after removing each sub optimal 

group. 

• Calculated the difference between the benchmark and 

weighted benchmark distributions for each subset. 

• Took the absolute value for each difference. 

• Averaged the total absolute difference for each of the 12 

benchmarks to obtain the average absolute deviation 

against the national benchmarks. 

Benchmark Comparison - Average Absolute Deviation 
© 2022 Ipsos 19

Benchmark Comparison - Results

Benchmark Questions 

National 

Benchmark

ALL 

Weighted

No Trap 

Presented

Failed 

both trap 

Passed at 

least 1 trap

Passed 

both traps

Passed 

consistency 

check

Passed both 

consistency 

and 

compliance

Right Direction 29.2% 0.3% 2.6% 0.9% 1.5% 2.0% 0.0% 0.5%

Excellent or very good health 57.4% 14.0% 13.0% 14.5% 14.7% 14.4% 11.0% 11.4%

Married 54.7% 4.2% 6.1% 3.3% 3.3% 2.9% 3.7% 2.8%

Citizen 92.0% 3.5% 3.8% 3.4% 3.4% 3.9% 3.3% 4.0%

2 or more in HH 85.6% 0.9% 0.6% 1.1% 0.7% 0.6% 0.9% 0.8%

At least 1 child under 18 in 

HH 33.2% 0.6% 3.5% 2.5% 2.7% 2.8% 2.5% 3.5%

Own home 70.1% 3.5% 3.5% 3.5% 3.7% 3.8% 4.0% 4.8%

3 bedrooms or more in HH 69.6% 3.3% 4.9% 2.5% 2.7% 2.9% 3.3% 3.6%

Moved in house more than 5 

years ago 60.8% 5.2% 6.2% 4.8% 4.6% 5.1% 5.3% 5.5%

2 or more vehicles one ton or 

less 70.1% 1.4% 1.8% 1.2% 1.6% 1.8% 1.9% 2.1%

Has landline phone 37.3% 2.0% 2.3% 1.8% 1.3% 1.2% 1.7% 0.6%

Speaks other than English at 

home 21.9% 1.5% 1.8% 1.4% 1.9% 2.5% 1.6% 2.8%

Average Deviation 3.4% 4.2% 3.4% 3.5% 3.7% 3.3% 3.5%
© 2022 Ipsos 20

Takeaways
© 2022 Ipsos 21

Conclusions and Discussion 

• Disguised trap question caught more sub optimal 

respondents. 

• Failing trap may not always indicate sub optimal response.

• Sub optimal identifications may not be correlated. 

• Removing suboptimal responses did not impact the data 

quality. 

• We will repeat this research with a longer questionnaire. 
Mina Muller

Mina.muller@ipsos.com

Thank you!