A Methodological Framework for Crowdsourcing in Research 

Michael Keatinga and Robert D. Furbergb 

aRTI International, Research Triangle Park, NC, mkeating@rti.org 

bRTI International, Research Triangle Park, NC, rfurberg@rti.org 

Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference 

Abstract 

The adaptation of crowdsourcing from commercial marketing in the private sector for use to support the research 

process is increasing, providing investigators a wealth of case studies from which to discern emerging best 

practices.  The successes and failures of crowdsourced research have not yet been analyzed to provide guidance for 

those interested in these new methods, nor have these data been synthesized to yield a methodological framework. 

This paper provides an evidence­informed methodological framework that describes a set of concepts, assumptions, 

and practices to support investigators with an interest in conducting crowdsourced research. 

Our case studies cover the different phases of the research lifecycle, beginning in the research design phase by 

examining open innovation challenges, moving to the implementation phase of crowdsourced cognitive interview 

data collection, and concluding with supplemental data collection.  Successful implementations of crowdsourcing 

require that researchers consider a number of dimensions, including clearly articulated research goals, determination 

of the type of crowdsourcing to use (e.g. open innovation challenge or microtask labor), identification of the target 

audience, an assessment of the factors that motivate an individual within a crowd, and the determination of where to 

apply crowdsourcing results in the overall research lifecycle. Without a guiding framework and process, 

investigators risk unsuccessful implementation of crowdsourced research activities. The purpose of this paper is to 

provide recommendations toward a more standardized use of crowdsourcing methods to support conducting 

research. 

Introduction 

Crowdsourcing has rapidly grown with the proliferation of the internet which has enabled likeminded individuals in 

society to become increasingly connected with one another.  The term originated in a Wired Magazine article by Jeff 

Howe (2006) in which he described the emerging phenomenon as, “outsourcing to a large crowd of people.”  Since 

then the term has evolved and been defined in a variety of ways.  King (2009) described crowdsourcing as, “tapping 

into the collective intelligence of the public to complete a task.”  Key characteristics of crowdsourcing are that it is a 

voluntary and participative online activity, crowdsourcing tasks can be of variable complexity and modularity, and it 

must be mutually beneficial to the crowd and the activity sponsors (Estellés­Arolas and González­Ladrón­deGuevara, 2012). 

In recent years, researchers have increasingly used crowdsourcing methods throughout the research lifecycle with 

varying degrees of success (a. Keating et al., 2013; b. Keating et al., 2013).  Such innovative approaches have 

yielded encouraging results; however, there is a paucity of literature on crowdsourcing methods, which may limit 

investigators’ ability to replicate successful interventions.  In the absence of a guiding framework to help researchers 

design successful approaches to crowdsourcing in research, the methods represent a higher­risk, trial and error 

proposition.  This paper aims to address this gap by providing a modifiable framework for crowdsourcing research 

and provides a model for inducing individual participation in such activities.  We use a variety of case studies, 

specific to survey research, to illustrate the application of this framework.   
 

 

 

           

               

                     

            

             

               

                  

             

 

                   

             

              

           

                

               

                

               

 

               

                  

                                   

                                       

                                    

                                 

                                     

                     

 

                                 

                                  

                                

                                        

                             

           

 

                                 

                            

                             

                                 

                           

                             

                   

Alignment in the Components of Crowdsourcing 

Before beginning any crowdsourcing activity, researchers 

need to consider the components of crowdsourcing, each 

of which plays an integral part in the success of the 

approach.  These components include understanding the 

research goal, the audience, the engagement mechanism, 

the platform, and the sensemaking approach, shown in 

Figure 1.  There is a flow to developing these components 

that begins with the research goal. 

Establish the Goal of the Research 

The research goal is the first component that should be 

established by an investigator as they consider 

crowdsourcing.  Clearly articulating the aim of the 

crowdsourcing initiative before beginning should be 

considered best practice.  As with any goal, we 

recommend that the goal be concrete, specific, and 

measurable.  By crafting goals with these attributes, the 

investigator can determine whether or not the approach 

was successful. 

Define the Target Audience 

Once the research goal is established, the audience 

required to achieve this goal can be defined.  Researchers 

must know their crowd.  Depending upon the research goal, specific audience segments may need to be targeted. 

For example, if the goal is to determine the mail return rates from the United States Census, then researchers will 

need to target individuals capable of contributing data required to conduct this sort of analysis (e.g., data scientists). 

Alternatively, if the goal is to collect photographic data on tobacco product placement in retail locations, then 

researchers will need to target individuals who are willing to do such tasks (e.g., city dwellers who own smartphones 

and are engaged adequately to visit a variety of stores). 

Identify Suitable Engagement Mechanisms 

Knowing the audience will help the research team target recruiting and participation in the crowdsourcing event and 

inform the crafting of an effective engagement mechanism.  This mechanism should be designed to appeal to the 

likely motivations of individuals within the crowd, encouraging them to take part in the crowdsourcing activities. 

This is one of the most important steps in the planning process.  We have further developed a method for crafting 

effective engagement mechanisms by extending a simple yet powerful behavioral framework which we discuss in 

the next section of the paper. 

Determine a Technical Platform to Support Activities 

A suitable platform to support the crowdsourcing activities is identified after the researcher has defined the target 

audience and designed an engagement mechanism.  This platform provides a forum for communicating and 

exchanging value with participants. Selection criteria for a crowdsourcing platform should include the availability of 

the resource to members of the target audience, the ability to integrate relevant engagement mechanisms to drive 

ongoing participation, and the means to distribute incentives after completion of the activities. 

The diversity of crowdsourcing platforms available on the market is growing exponentially, giving researchers a 

tremendous amount of options (visit http://www.crowdsourcing.org/directory for information about potential 

Figure 1. Components of Crowdsourcing 

in Research 
                            

                                     

 

                               

                                  

                               

 

 

 

                              

                                 

                                

                               

                            

                          

 

                               

                            

                                         

                                    

                                 

   

                           

                                      

                            

                         

                               

                              

                                 

                                

                             

crowdsourcing platform options).  Depending upon the crowdsourcing event, researchers can even create their own 

platform for their crowdsourcing event and we discuss a couple of examples below in Case Studies 1 and 2. 

Inventory Data Quality Standards 

Finally, investigators should define standards to characterize useable data, or other crowdsourced returns, that can be 

applied toward satisfying the research goal.  Ensuring that there is alignment between the incoming data from a 

target audience and the research goals will increase the likelihood that these goals will be achieved. 

The Motive-Incentive-Activation-Behavior Model of Crowdsourcing 

“Most of economics can be summarized in four words: ‘People respond to incentives.’ 

The rest is commentary.” – Steven E. Landsburg 

Participation in crowdsourcing activities is driven by the motives of the individual.  In particular situations, 

incentives can and should be used to activate an individual’s motivations, enabling a specific behavior, such as 

taking the time required to contribute to a crowdsourced research activity.  Understanding how motivations can be 

influenced and activated through intrinsic and extrinsic incentive pathways is a critical aspect of designing effective 

crowdsourced interventions.  Rosenstiel (2007) provides a simple model to describe the activation of human 

behavior on the basis of motive­incentive­activation­behavior, or MIAB, which we show in Figure 2. 

Figure 2. The Motive-Incentive-Activation-Behavior Model of Crowdsourcing 

The components of Motivation and Incentive are tightly linked and represent the two fundamental mechanisms for 

engaging prospective participants in crowdsourcing.  The term “motivation” is defined in the Oxford English 

Dictionary as the reason or reasons one has for acting or behaving in a particular way while an “incentive” is defined 

as a thing that motivates or encourages one to do something.  Both intrinsic and extrinsic motivational factors may 

play a role in an individual’s decision to participate and are important considerations in the intervention design 

phase. 

Self­Determination Theory (Deci & Ryan, 1985) distinguishes a difference between two different types of 

motivation that result in a given action.  Intrinsic motivation refers to doing something on the merit of pleasure or 

fulfillment that is initiated without obvious external incentives.  External motivation is activated by external 

incentives, such as direct or indirect monetary compensation or recognition by others. 

Several authors have revealed motives that explain the motivation of participation in open source projects (Hars, 

2002; Hertel, 2003; Lakhani, 2003; Lerner, 2002).  Applied to crowdsourcing, intrinsic motivators could stem from 

an individual’s inborn desire and feelings of competence, satisfaction, and enjoyment while the potential to win a 

prize for participation may act as an external incentive (Leimeister, 2009).  Extrinsic activated motivates can be 

further divided into two classes: direct compensation and social motives (Vallerand, 1998).  Monetary or non-
 

 

                              

                                   

                                

                              

                               

                           

           

                                    

                                    

                         

                        

                                

                           

                   

                                        

                               

                             

                 

                           

                                   

                                  

                                            

        

 

                   

              

               

             

              

           

                   

                

                    

                   

              

                   

               

               

                  

             

               

                        

                       

             

                  

             

                    

                                    

                                 

monetary awards, including trophies, medals, or other prizes are examples of direct compensation.  Social motives 

include the expected reaction of individuals whose opinion is valued by the participant, such as friends, partners, or 

audience members.  Motivation to participate in competitions is greater if members of an individual’s social network 

indicate the importance of participating in an event.  Within survey research, leverage­salience theory argues that 

community involvement in a study has positive effect on individual cooperation and participation (Groves et al., 

2000). As applied to crowdsourcing, participants may expect positive reactions from other participants, organizers, 

or beneficiaries of the activity. 

The concept of Activation addresses an individual’s decision to initiate a behavior.  Of concern in the area of 

activation is the persistence of this state, or the application of continued effort toward achieving a specific goal. 

Thus, consideration of incorporating activation­supporting components is an additional factor for consideration in 

the design of crowdsourcing activities.  Examples of activation­supporting components may include providing 

participants with access to the knowledge of experts for inspiration and reference throughout the activation phase. 

Alternatives to expert knowledge include providing mentorship to participants or supporting an open community 

and exchange of knowledge within the network of participants. 

The term “behavior” is defined by the Oxford English Dictionary as the way in which one acts or conducts oneself. 

In the context of crowdsourcing activities, there are various behaviors that may be considered desirable outcomes, 

including original content generation, providing support or collaboration within a network of contributors, or other 

constitutive components required to devise a specific solution. 

The MIAB model provides investigators considering the use of crowdsourcing methods to support research 

activities with a high­level roadmap of considerations that must be addressed in the design phase of the intervention 

to ensure successful engagement with participants.  Additionally, if things do not go as planned this model can 

provide a way to diagnose the causes of the problem and to craft the solution.  We will discuss two such examples in 

the case studies below. 

Case Study 1: RTI’s 2012 Research Challenge 

In 2012 RTI International was in the process of planning 

for an upcoming omnibus survey.  During the 

questionnaire design phase the decision was made to 

experiment with the potentials of creating a 

crowdsourced instrument.  To complete this task we 

launched the RTI 2012 Research Challenge. 

The goal of this challenge was to create high quality 

survey questions for the upcoming study.  We targeted 

the researcher crowd at large, asking for all ideas.  To 

engage the crowd we used a combination of incentives to 

appeal to extrinsic motivations.  First, participants were 

told that if they won the competition then they would 

receive the response data to their survey questions, 

demographic data and exclusive publishing rights for one 

year after data delivery.  Second, the judges were well 

known in the research community, particularly survey 

research, giving young researchers the opportunity to get 

their work in front of leaders in the field.  RTI created its 

own platform for the event, and used a web form on its 

SurveyPost blog to receive entries and publish 

information about the event.  RTI staff also did heavy 

marketing on professional listservs, like AAPORnet, to 

get the word out about the event.  We created specific 

submission criteria for the event to ensure that sensemaking would be relatively easy for the judges.  We asked 

researchers to submit a two page synopsis of their research idea and up to ten survey questions. 

Figure 3: Crowdsourcing Components of 

RTI’s 2012 Research Challenge 
 

 

                                    

                                

                                  

of material with which to create the questionnaire. 

 

                                      

                                    

                                   

              

                 

                

                     

   

 

 

               

                   

                   

                

                   

               

                      

               

                    

                   

                      

                   

                      

                     

                    

                 

      

This event was very successful and RTI received 76 entries in 23 days.  Topics ranged from questions about 

emerging tobacco products to astrological theories about how people meet their mates.  Participants in the event 

ranged from undergraduate students to deans of well­known universities.  In the end, the omnibus survey had plenty 

Figure 4: Applying the MIAB model to the 2012 Research Challenge 

The MIAB model can help identify some of the reasons why this event may have been successful.  Advancing a 

research agenda through professional exposure is a motive for many researchers.  Gathering data that can be used for 

conference presentations and publications can help to advance a research agenda and thus acts as a good incentive 

for these researchers.  After considering the possibilities, 

researchers activated and submitted a proposal to the research 

challenge.  By doing this, they did the intended behavior, 

giving RTI’s study a good set of survey questions for the 

upcoming study. 

Case Study 2: Crowdsourcing Cognitive Interviewing 

Recruitment on Facebook 

Not everything always goes according to plan when 

crowdsourcing, and we have found value in using the MIAB 

model to help diagnose the sources of problems in our 

crowdsourcing approaches.  Such a case arose recently as 

RTI pilot tested the use of crowdsourcing methods to collect 

large quantities of cognitive interview response data on 

Facebook.  The goal of this study was to determine if we 

could collect large quantities of useful cognitive interview 

data using Facebook as a recruitment tool.  We used targeted 

advertisements that were shown to a crowd of Facebook users 

who liked music.  To engage the crowd we offered a $5 

music gift card to appeal to these users’ extrinsic motivations 

to acquire more music.  RTI created its own platform for the 

event, and used Facebook to broadcast the event and a Web 

survey to receive the data.  The Web survey was highly 

structured to ensure that sensemaking could be done in 

datasets by researchers. 

Figure 5. Crowdsourcing Components of 

Cognitive Interview Recruitment on Facebook 
                                 

                                        

                                  

                     

                                      

                                        

                                        

                                      

                             

 

                                     

                                    

                                        

                                          

                           

 

 

Figure 6. Applying the MIAB Model to Cognitive Interview Recruitment on Facebook 

One of our assumptions in designing this recruitment approach was that music lovers were motivated to acquire 

more music.  As a result, we offered the $5 music gift card incentive to appeal to these motivations and encourage 

activation and participation in the event.  Unfortunately, this incentive was not great enough to lead to participation 

and after a few days we focused on an alternative approach. 

Figure 7: Applying the MIAB Model to a Modified Cognitive Interview Recruitment Approach 

In our modified approach we targeted a crowd of Facebook users who liked the American Red Cross.  Our thought 

process was that people who like the American Red Cross will be motivated by altruistic causes.  As a result, we 

used a $5 donation to the American Red Cross as an incentive for people to participate.  This was successful in 

leading these Facebook users to make the decision to activate and take the Web survey.  This was our intended 

behavior and led to the creation of a large volume of useful cognitive interview data. 

Case Study 3: Collecting Tobacco Retailer Data on Snus in Chicago 

One of the most exciting aspects of crowdsourcing is the ability to collect large quantities of data quickly and 

relatively cheaply.  This can be done in the data collection phase of a project by collecting supplemental datasets 

that add depth to traditional survey data.  For example, if a survey is collecting data on the demand side dynamics 

snus tobacco, then it would also be useful to also know where the local snus supply is.  In 2012 RTI experimented 

with an approach to collect fine level tobacco supply data from retailers in Chicago. 
The goal of this study was to rapidly collect large quantities of local retailer data about snus tobacco. Given the large 

quantities of retail locations that needed to be called and the short duration of these tasks, we used the crowd of 

microtask workers on Amazon Mechanical Turk.  To engage the crowd we offered a $0.75 reward if the worker 

called a retail location to ask if it sold snus tobacco.  We used the Amazon Mechanical Turk crowdsourcing platform 

to manage the data collection and pay workers.  

Sensemaking was simple since the response data was neatly 

organized in a spreadsheet and could be easily matched to 

retailer address data. 

The MIAB model can help to identify some of the reasons 

why this approach may have been successful.  Workers who 

use Amazon Mechanical Turk are generally motivated to 

make money by working jobs posted on the platform.  

When compared to other jobs posted on the platform, our 

research team offered a relatively generous incentive for the 

workers who chose to complete our tasks.  This led to fast 

activation.  We made data entry easy and straightforward in 

our job postings, so that data was captured accurately. 

During the initial phases of data collection we found that 

some workers were not doing the intended behavior when 

they collected tobacco data from retailers.  These workers 

were including other smokeless tobacco products, like dip 

or chewing tobacco, in their results.  We did not want data 

about these other tobacco products and clarified the 

instructions in our job postings.  This proved to be a quick

solution to the problem, and it is one more example of how 

the MIAB model offers a quick method for researchers to

identify where tweaks may be needed to their 

crowdsourcing approach.  

Figure 8: Crowdsourcing Components of 

Tobacco Retailer Data Collection 

Figure 9: Applying the MIAB Model to Tobacco Retailer Data Collection 

In the end we collected data from retailers quickly and effectively, which holds promise for implementing this sort 

of rapid supplemental data collection approach on outgoing studies. 

Conclusions

Crowdsourcing holds promise as a means for researchers to achieve new and ambitious research goals; however, 

without a guiding methodological framework, researchers run the risk of being unsuccessful in their crowdsourcing 

endeavors.   
                             

                                       

                                      

                              

                                    

                                  

                                    

                         

                                      

                       

                         

                                 

               

                 

                            

         

                                

               

                                 

                          

                       

                               

  

                         

                                

                   

                                 

                               

                

                           

                                 

                       

                               

                         

                       

               

                                   

                         

                     

                             

                               

             

Before taking a crowdsourced approach, we encourage researchers to consider and define the components of 

crowdsourcing for the project, all of which flow somewhat naturally from one another.  Begin with a goal.  This goal 

will help the researcher determine who can help achieve that goal.  Knowing who will participate will help to choose 

the appropriate mechanism to encourage engagement.  This mechanism may dictate what platform should be used 

for the crowdsourcing event.  This platform will guide what sort of data will come back from the crowd. 

As the researcher plans their engagement mechanism, they should use the MIAB model to guide this process. 

Incentives should appeal to the motives of the crowd.  Only when motives and incentives are aligned will activation 

occur.  Once activation occurs, researchers should consider activation supporting mechanisms.  Monitoring the 

behavior of participants is also encouraged.  As our case studies showed, sometimes tweaks will need to be made to 

ensure the researcher gets the proper data back to achieve their goals. 

References 

1.   Behavior [Def. 1]. (n.d.). In Oxford English Dictionary, Retrieved February 2, 2014, from 

http://www.oxforddictionaries.com/us/definition/american_english/behavior. 

2.   Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self­determination in human behavior. New York: 

Plenum. 

3.   Estellés­Arolas, Enrique; González­Ladrón­de­Guevara, Fernando (2012), "Towards an Integrated 

Crowdsourcing Definition,” Journal of Information Science 38 (2): 189–200. 

4.   Groves, R. M., Singer, E., Corning, A. (2000).  Leverage­salience theory of survey participation. Public 

Opinion Quarterly 64 (3): 299­308. 

5.   Hars, A., and Ou, S. (2002) Working for free? Motivations for participating in open­source projects.  (2002) 

International Journal of Electronic Commerce, (6) 3, 25–39. 

6.   Hertel, G., Niedner S. & Herrmann, S. (2003), Motivation of software developers in Open Source projects: an 

internet­based survey of contributors to the Linux kernel. Research Policy, 32 pp. 1159—1177. 

7.   Heider, F. (1958) The Psychology of Interpersonal Relations. Mahwah, NJ: Lawrence Erlbaum. 

8.   Howe, J. (2006, June). The rise of crowdsourcing. Wired, (14.06). Retrieved on January 23, 2013 from: 

http://www.wired.com/wired/archive/14.06/crowds.html. 

9.   Incentive [Def. 1]. (n.d.). In Oxford English Dictionary, Retrieved February 2, 2014, from 

http://www.oxforddictionaries.com/us/definition/american_english/incentive. 

10.  Keating, M. D., Rhodes, B. B., & Richards, A. K. (2013, March). Applying crowdsourcing methods in 

social science research. Presented at Federal CASIC Workshops, Washington, DC. 

11.  Keating, M. D., Rhodes, B. B., & Richards, A. K. (2013). Crowdsourcing: A flexible method for innovation, 

data collection, and analysis in social science research. In Social media, sociality, and survey research. (pp. 

179–201). Hoboken, NJ: John Wiley & Sons, Inc. 

12.  King, S. (2009). Using Social Media and Crowd­Sourcing for Quick and Simple Market Research. 

http://money.usnews.com/money/blogs/outside­voices­small­business/2009/01/27/using­social­media­andcrowd­sourcing­for­quick­and­simple­market­research. 

13.  Lakhani, K.R., and Wolf, R.G. (2003). Why hackers do what they do: Understanding motivation and effort in 

free/open source software projects. MIT Sloan Working Paper no. 4425–03, Cambridge, MA. 

14.  Landsburg, S. 2012. The Armchair Economist: Economics and Everyday Life. Free Press, New York, NY. Page 

3. 

15.  Leimeister, J. M.; Huber, M.; Bretschneider, U. & Krcmar, H. (2009): Leveraging Crowdsourcing: ActivationSupporting components for IT­based ideas competition. Journal of Management Information Systems (JMIS), 

Ausgabe/Number: 1, Vol. 26, Erscheinungsjahr/Year: 2009. Seiten/Pages: 197­224. 

16.  Lerner, J., and Tirole, J. (2002) Some simple economics of open source. Journal of Industrial Economics, 50, 2, 

197–234. 

17.  Motivation [Def. 1]. (n.d.). In Oxford English Dictionary, Retrieved February 2, 2014, from 

http://www.oxforddictionaries.com/us/definition/american_english/motivation. 

18.  Rosenstiel, L. von. (2007) Basics of Organizational Psychology. Stuttgart, Germany: Schäffer­Poeschel. 

19.  Vallerand, R.J., and Fortier, M.S. (1998)Measures of intrinsic and extrinsic motivation in sport and physical 

activity: A review and critique. In J.L. Duda (ed.), Advances in Sport and Exercise Psychology Measurement. 

Morgantown, WV: Fitness Information Technology, pp. 81–101.