Using Mechanical Turk and PsiTurk for Dynamic Web Experiments

Using Mechanical Turk and PsiTurk for Dynamic Web Experiments Anna Coenen (anna.coenen@nyu.edu) Doug Markant (doug.markant@nyu.edu) Jay B. Martin (jbmartin@nyu.edu) John McDonnell (john.mcdonnell@nyu.edu) Department of Psychology, New York University New York, NY 10003 through a simple example of how to post a HIT, oversee the data collection, and reimburse participants on the AMT web- site. Keywords: Amazon Mechanical Turk; PsiTurk; Online Ex- periments; Crowdsourcing Objectives Benefits and drawbacks of online experiments This half-day workshop will demonstrate how to build cus- tom web-based experiments that rely on participants from Amazon Mechanical Turk (AMT). Attendees will learn how to deploy web-based experiments using PsiTurk, a Python- based platform that simplifies the process of setting up exper- iments and interacting with AMT. Workshops discussing the AMT marketplace have been of- fered at previous Cognitive Science Society meetings (e.g., Mason & Suri, 2011). This workshop will complement those by stepping through a working demo that attendees can use to follow along and run on their personal computers. Impor- tantly, the demo will illustrate how AMT can be used with dynamic, externally-hosted experiments, rather than the ba- sic survey templates currently offered on AMT. The workshop will have two parts. First, we will outline some of the general advantages and principles of using AMT for online behavioral experiments, including a basic intro- duction to the AMT website and the data collection process more generally. Second, we will show participants how to use the PsiTurk platform to run any web-based experiment on AMT. This portion of the workshop will emphasize “hands- on” training in AMT and PsiTurk that will teach attendees how to deploy their own web-based experiments. Next, we will cover some of the advantages and pitfalls asso- ciated with using AMT for behavioral research. For cognitive psychologists the appeal of using AMT lies in running computer experiments that would otherwise be completed in the lab, typically by undergraduate students. Online experiments have several advantages: 1. Data from a large number of participants can be collected quickly and at low costs. A few hours are typically suffi- cient for recruiting a full set of participants in a standard cognition or perception experiment. 2. Since the data collection is anonymous, using AMT min- imizes experimenter effects and problems with contami- nated subject pools at research departments. 3. For the same reason, experimental results become more replicable. Because subjects do not interact with an exper- imenter, there is no possibility for experimenter confound. If one researcher runs the code for another’s experiment, it is, in principle, a pure replication: there is no source of systematic experimental deviation. 4. In general, web-based experiments are easier to share with other researchers since they are designed to run in standard web browsers and do not require any additional software. This facilitates the re-use of experimental code either for the purposes of direct replication or the design of new ex- periments. Outline of the Workshop Throughout the workshop we will use both slides and live demonstrations of how to use AMT and PsiTurk for running web experiments. Introduction to Mechanical Turk Potential disadvantages of the method concern the quality of the data, including the possibility that comparatively low reimbursement might lower incentives to engage in a task. To address these questions, several authors have used AMT to replicate classic findings in their field. Paolacci, Chan- dler, and Ipeirotis (2010), for example, replicated a number of well-known cognitive biases using AMT data. Germine et al. (2012) found no systematic differences in the results of some widely-used perceptual paradigms using laboratory and online data. Rand (2012) also conducted an extensive study into the reliability of AMT workers’ demographic data and verified that self-reported demographic information is highly We will start by introducing the basic structure behind AMT and demonstrate how to run a simple project. AMT is the largest online service in the US that offers a marketplace for tasks that need to be solved by human rather than machine intelligence. Human Intelligence Tasks (HITs) are submitted by requesters, such as corporations, re- searchers, organizations, or individuals in need for human participants. They can be completed by workers in exchange for a reimbursement that is set by the requester. Workers can also be awarded bonuses or have their payment rejected based on how they completed a HIT. We will walk attendants