© 2021 Ipsos 1

Coming Clean: Does Data Cleaning Reduce or 

Increase Bias in Sub-groups? 

Randall K. Thomas, Frances M. Barlas, 

and Megan A. Hendrich

Ipsos Public Affairs

2021 FCSM Research & Policy Conference

November 2nd – November 4th, 2021
© 2021 Ipsos 2

Background

• Many researchers believe that it is necessary to clean survey data 

before analysis in order to improve accuracy. 

• One specific concern has been about sub-optimal, or less-thanaccurate, response.

• Sub-optimal response is seen as a source of lower quality data:

• a dishonest or mistaken response (a bias)

• an inattentive response (error), or

• aミ approxiマate respoミse rather thaミ the respoミdeミt’s true 

answer (some good measurement plus some error)
© 2021 Ipsos 3

Background

• Researchers have developed many measures of sub-optimal response, 

including :

• Speeding through the survey

• Grid non-differentiation or straightlining

• Item nonresponse – skipping items

• Extreme or exaggerated responding on numeric entry

• Failure at trap questions (e.g., compliance traps)

• Consistency checks
© 2021 Ipsos 4

Research Questions

• While there is little research on data cleaning and suboptimal response, 

what does exist seems to start with the assumption that data cleaning is 

necessary to improve the accuracy of survey results. 

• With Study 1, we set out to test that assumption and sought to address: 

• How much data cleaning is necessary?

• How should we clean our data?

• Is it possible to do too much data cleaning? 

• In Study 2, we sought to examine how cleaning affects survey results for 

specific subgroups.
© 2021 Ipsos 5

Study 1 - Method
© 2021 Ipsos 6

In October 2015, we conducted parallel studies using two online 

sample sources:

Ipsos KnowledgePanel

• The largest probability-based panel in the U.S. Sample obtained primarily 

through ABS recruitment. 

• Obtained 1,297 completes

Non-probability sample – opt-in sample

• To obtain a demographically balanced sample, we set up interlocking quotas by 

gender, age, race/ethnicity, and education

• Obtained 2,564 completes

Study 1 Method
© 2021 Ipsos 7

Sample Cleaning Criteria and Size

After each round of cleaning, remaining cases were weighted to Current 

Population Survey demographic benchmarks. 
© 2021 Ipsos 8

Benchmarks for Bias Evaluation

American Community 

Survey (2014)

• Home with 2 or fewer 

bedrooms

• Own 2 or more vehicles

• Married

• Household size 2+

• Employed

• Own home

CPS–Civic Engagement 

(2012)

• Discuss politics with family 

and friends

• Voting in local elections

• Contact with friends/family

• Trust people in neighborhood

National Health Interview 

Survey (2013)

• Working landline

• General health – good or 

better

• More than 1 year since 

doctor visit

• Doctor not taking new 

patients

• Lifetime drinker

• Current smoker

• Sleep 7+ hours

CPS – Volunteer 

Supplement (2014)

• Volunteer in last year

• Donate $25 or more

CPS – Food Security (2014)

• Need to spend more on 

food

General Social Survey 

(2014)

• Favor death penalty

• Most people can be 

trusted

• Women less likely to be 

promoted

• Religiosity

CPS – Public Participation 

in the Arts (2012)

• Visited art museum or 

gallery

• Visited park or monument

• Read a book in last year

• Went to the movies

• Went to sporting event

• Worked with plants/ 

gardened
© 2021 Ipsos 9

Average Absolute Deviation

To assess bias among the cleaned subsets – calculated the average 

absolute deviation:

• Calculated difference between the benchmark and the

estimate from each cleaned subset

• Took the absolute value for each difference

• Averaged across absolute values for each benchmark to 

obtain average absolute deviation with each cleaned

subset 
© 2021 Ipsos 10

Study 1 Results
© 2021 Ipsos 11

Hypothetical Distribution

**Not real data**

We initially hypothesized that minimal data cleaning, around 5% to 10%, would reduce 

bias, but extensive cleaning would do more harm than good. 

0

2

4

6

8

10

12

14

0% 2.5% 5% 10% 20% 30% 40% 50%

Average Absolute Deviation

Percent Excluded

KnowledgePanel

Non-probability
© 2021 Ipsos 12

Study 1 Results – Multiple Cleaning Criteria

With KnowledgePanel, no effect on bias with increasingly rigorous exclusion criteria.

0

2

4

6

8

10

12

14

0% 2.5% 5% 10% 20% 30% 40% 50%

Average Absolute Deviation

Percent Excluded

KnowledgePanel
© 2021 Ipsos 13

Study 1 Results – Multiple Cleaning Criteria

Similarly with non-probability sample, no effect on bias with increasingly rigorous 

exclusion criteria.

0

2

4

6

8

10

12

14

0% 2.5% 5% 10% 20% 30% 40% 50%

Average Absolute Deviation

Percent Excluded

KnowledgePanel

Non-probability
© 2021 Ipsos 14

Study 1 Sample Cleaning Criteria and Size

Next, we used time to complete the survey (‘speeders’) as the sole criterion to 

use to clean the data. As before, after each round of cleaning, remaining cases 

were weighted to Current Population Survey demographic benchmarks. 
© 2021 Ipsos 15

Study 1 Replicated Excluding Fastest Completes

Rather than using multiple criteria to clean, using speed to complete as the consistent 

cleaning criterion, we again found no effect on bias with increasingly rigorous 

exclusion.

0

2

4

6

8

10

12

14

0% 2.5% 5% 10% 20% 30% 40% 50%

Average Absolute Deviation

Percent Excluded

KnowledgePanel

Non-probability
© 2021 Ipsos 16

Study 2 Research Questions

• Surprisingly, we found no real change in the overall results or reduction of 

bias as more and more sample was deleted for both sample types. This was 

true using both multiple criteria or a single criterion of speed to complete.

• One issue that we have noticed in other studies, sometimes deletions due to 

suboptimal response or due to speed sometimes appeared to impact some 

groups more that proportionally smaller – including those who are younger, 

more male, and more people of color. 

• Our interest in Study 2 was to compare how cleaning might affect the bias 

we find for specific, smaller groups – White, Black, and Latino.
© 2021 Ipsos 17

Study 2 - Method
© 2021 Ipsos 18

Study 2 Methods

In October 2020, we conducted parallel studies using two online 

sample sources:

• Ipsos KnowledgePanel (KP) – N=3,344

• Non-probability sample – 2 types:

− Opt-in with demographic quotas – using quotas to more 

adequately demographically balanced the sample, with gender, 

age, race/ethnicity, and education quotas (N=2,677)

− Opt-in with no demographic quotas (N=3,293)
© 2021 Ipsos 19

Sample 2 Cleaning Criteria and Size

For this analysis, we used speed of completion as the primary criterion 

for cleaning and created groups within each sample type that eliminated 

0%, then the fastest 2.5%, 5.0%, 10.0%, 20.0%, 30.0%, 40.0%, and 50.0%. 

After each round of cleaning, for each sample source, remaining cases 

were weighted to Current Population Survey demographic benchmarks.

We had 25 benchmarks that we were able to obtain national estimates 

overall for 18+, as well as for White, Black, and Latino groups.
© 2021 Ipsos 20

Study 2 Results
© 2021 Ipsos 21

Study 2 Results – Cleaning for Speed and Bias

Among all the sample sources, KnowledgePanel again had the lowest bias overall and 

there was no effect on bias with increasingly rigorous exclusion criteria. Both opt-in 

samples had higher bias, the no quota opt-in sample showed the highest bias. Increased 

deselection based on speed may have slightly reduced bias for the no quota opt-in sample.

0.0%

5.0%

10.0%

15.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Average Benchmark Divergence - General Population

KP Opt-in - quota Opt-in - no quota
© 2021 Ipsos 22

Study 2 Results – Effects of Cleaning on Group Proportions

We examined how unweighted group proportions were affected by data cleaning. We 

found little effect on the relative group proportions – just a very slight increase in 

Black and Latino under heaviest cleaning for KnowledgePanel and Opt-in with Quotas.

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

KnowledgePanel

Group Proportions

White Black Latino

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Opt-in No Quotas

Group Proportions

White Black Latino

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Opt-in with Quotas

Group Proportions 

White Black Latino
© 2021 Ipsos 23

Study 2 Results – Cleaning for Speed and Bias – Race-ethnicity

Comparing all sample sources, KnowledgePanel, bias was lowest for each raceethnicity group. Across samples we found that the White and Black groups showed the 

lowest bias, while the Latino group had the highest bias. More cleaning did not 

generally reduce or increase bias for the race-ethnicity, though there may be some 

slight increase of bias with more extreme cleaning.

0.0%

5.0%

10.0%

15.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

KnowledgePanel - Bias for Sub-groups

White Black Latino

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Opt-in with Quotas - Bias for Sub-groups

White Black Latino

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Opt-in No Quotas - Bias for Sub-groups

White Black Latino
© 2021 Ipsos 24

Study 2 Results –

Effects of Cleaning 

on Weighting 

Efficiency -

Untrimmed

Next, examined sample cleaning on 

weighting efficiency (which affects 

effective sample size) for each sample 

source using untrimmed weights (no 

constraints on low or high weights).

The KnowledgePanel probability-based 

panel had the highest weighting 

efficiency while the Opt-in No Quotas 

group had the lowest efficiency.

Cleaning did NOT improve weighting 

efficiency.

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Weighting Efficiency - Untrimmed

KP sample Opt-in - quota Opt-in - no quota
© 2021 Ipsos 25

Study 2 Results –

Effects of Cleaning 

on Weighting 

Efficiency - Trimmed

We then examined the weighting 

efficiency for trimmed weights, 

trimming weights to a range from .2 

and 5. 

KnowledgePanel weights did not 

need trimming (all weights were 

between .2 and 5) but trimming 

improved the efficiency of both optin samples.

Cleaning did NOT improve weighting 

efficiency for trimmed weights.

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

0.0% 2.5% 5.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Percent Eliminated

Weighting Efficiency - Trimmed

KP sample Opt-in - quota Opt-in - no quota
© 2021 Ipsos 26

Discussion
© 2021 Ipsos 27

Conclusions

• We anticipated a slight decrease in bias with minimal data cleaning and an 

increase in bias when removing larger proportions of respondents. However, we 

did not find any decrease or increase in bias with more rigorous cleaning.

• Results were confirmed in two studies with both KnowledgePanel and nonprobability samples, using both a multiple cleaning criteria approach as well as 

just using speeding as the criterion to remove cases.

• Data cleanliness is NOT necessarily next to godliness – overall, and within 

specific race-ethnicity groups:

• Data cleaning did not reduce bias

• Cleaning did not change results 

• Cleaning does not appear to affect estimates for smaller groups –

race-ethnicity.
© 2021 Ipsos 28

Discussion

Why is the average bias/error unchanging no matter how many you 

eliminated?

• First, the fastest 1 or 2% or most egregious sub-optimal respondents (often, but 

not always the one and the same) do provide somewhat different responses 

thaミ the other respoミdeミts. But eliマiミatiミg the fastest ヱ or ヲ% doesミ’t chaミge 

any overall point estimates (especially if the fastest are generally random 

responses or are similar to the other respondents' responses).

• Beyond the fastest 2% or most sub-optimal respondents, respondents who are 

faster do not significantly differ from slower respondents. Therefore, eliminating 

respondents based on speed beyond the fastest 1 to 2% eliminates people just 

like people who take longer to respond, again leading to little or real change in 

average estimates or bias, even if one eliminates the fastest 50% of respondents 

(though you do lose statistical power due to loss of respondents and increases in 

weight variance).
© 2021 Ipsos 29

Next Steps

• We are conducting additional research to assess how increased data 

cleaning could impact on variance and covariance of our benchmark 

measures – focus to date has been on point estimates (means and 

proportions). 

• We will look to replicate these findings with additional groups (e.g., gender 

or age).
Randall K. Thomas

Randall.Thomas@Ipsos.com

Thank you!