RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

Benefits and Drawbacks of Using 

Crowdsourcing Techniques for 

Cognitive Testing

Sarah Cook, RTI International

Rachel E. Morgan, Bureau of Justice Statistics

Christopher Krebs, RTI International

Lynn Langton, Bureau of Justice Statistics

Presented at the Federal Committee on Statistical Methodology

2018 Research and Policy Conference

Washington, DC

March 7, 2018
Cognitive Testing

▪ Cognitive testing

– Process of understanding the cognitive process of survey respondents and how 

they think about questions

– Identifying issues with question wording, comprehension, or measurement

– Important to get feedback from your target population

– Improves data quality and measurement

▪ 2 primary types of cognitive testing:

1. In-person interviewing

2. Crowdsourcing
In-person interviewing

▪ In-person interviews

– Traditional method of cognitive testing

– Cognitive understanding is learned through observation and both scripted and 

unscripted probes

– Administration of survey questions in interview setting, with follow up questions to 

probe on respondent understanding of items and thought process when 

responding

▪ Volunteers either complete online screener 

survey or call to be screened

▪ Researchers call and set up appointments with 

eligible respondents

▪ Respondents come into RTI office (or 

convenient location) for in-person interview

▪ ~1 hour interview

▪ Paid cash
Crowdsourcing

▪ Crowdsourcing

– Used to obtain information from a large 

number of people typically using the 

internet

– Pre-registered panel members who are 

looking to complete menial tasks for 

minimal compensation 

– Recruit participants based on 

demographics characteristics

▪ Race, Hispanic origin, age, gender, 

citizenship

– The online platform directs participants to a survey of select items and follow-up 

questions

▪ Can ask the respondent to provide open-ended comments about the questions, including 

any difficulty understanding specific terms or recommendations for improvements

▪ Also, can ask respondents to provide an open-ended narrative to determine if questions 

are capturing phenomena/measuring concepts (which is helpful when classifying different 

types of crime)
Crowdsourcing as an effective method of testing

▪ Crowdsourcing techniques have been used with success in the 

development of multiple BJS collections, including the Campus Climate 

Survey Validation Study and National Crime Victimization Survey (NCVS) 

Supplemental Fraud Survey

▪ Crowdsourcing findings resulted in useful improvements and clarifications 

of survey questions 

▪ Examples:

– Where drop-down boxes were needed instead of openended fields

– Response option revisions due to a lot of “don’t know” 

responses

– Places where “don’t know/uncertain” response options 

were needed

– Revisions and reorganization of screening questions

– Refinement of question wording for clarity
Unique benefits of each method

Traditional in-person interviewing

Build rapport for questions on sensitive 

topics

Can probe at any time

Not likely to yield a representative sample 

of respondents

Able to respond to non-verbal cues

Can ask more questions

Crowdsourcing

Speed – quick and efficient

Low cost or no cost

Diversity of respondents/can select 

specific demographic characteristics

Receive information on measurement

using open-ended questions

No transcribing – responses are provided 

in open-ended text boxes
Example comparison (NCVS SFS cognitive testing)

Crowdsourcing Traditional in-person interviewing

N = 300 N = 18

$1.66/interview $40/interview

~ 5 minutes ~ 60 minutes

Completed in a few days Completed in a few weeks

Recruitment included in cost Recruitment must be done by researchers

Cognitive data gathered through closedand open-ended web survey questions

Cognitive data gathered through probes; 

either scripted or spontaneous based on 

participant actions and responses
Quality of crowdsourced information

▪ Crowdsourcing allows researchers to gather a lot of information in a 

short amount of time but is it quality information?

▪ What did we learn about crowdsourcing data quality when testing 

for the NCVS Supplemental Fraud Survey (SFS)?

– Information from open-ended responses

▪ Did open-ended responses contradict closed-ended survey items?

▪ Responses provided further clarification on experience which helped 

to revise the closed-ended survey items

– Comparison of in-person interviewing and crowdsourcing
Topics better suited for one method or another

Traditional in-person interviewing

Complex topics or topics with many steps 

(e.g., fraud victimization; home buying 

process) 

Sensitive topics 

(e.g. physical/sexual assault victimization; 

women who have had miscarriages)

Conceptually difficult topics

(e.g., anything that is not “common 

knowledge”; attitudes towards GMOs)

Crowdsourcing

Topics with a variety of respondent 

characteristics 

(e.g., a survey of non-US born citizens; 

patient studies)

Topics that apply to the general 

population 

(e.g., demographic questions; accessibility 

of services)

Hard to reach populations

(e.g., Spanish-only speakers; people with 

disabilities)
Using both methods effectively

• If/where there are Crowdsourcing problems

• Why there are 

problems In-Person

Crowdsourcing and in-person methods of cognitive testing can be used 

together to maximize efficiency and information when pretesting
More Information

Sarah Cook

Research Public Health Analyst

RTI International

919.514.1236

scook@rti.org

Rachel E. Morgan

Victimization Statistics Unit 

Bureau of Justice Statistics (BJS) 

202.616.1707

rachel.morgan@usdoj.gov