Practical Advice on How to Run Human Behavioral Studies

Practical Advice on How to Run Human Behavioral Studies Frank E. Ritter (frank.ritter@psu.edu) Jong W. Kim (Jong.Kim@ucf.edu) College of IST, Penn State Psychology, U. of Central Florida Orlando, FL 32816 USA University Park, PA 16802 USA Keywords: Psychology experimental method; HCI usability studies. An early draft (approximately half the current length) of the material is available in a tech. report at acs.ist.psu.edu/reports/ritterKM09.pdf, and published copies are available from Sage. (I) Objectives and scope of the tutorial The lack of materials on the details of running human experiments can lead to a gap between theory and practice, which is particularly acute in cognitive science experiments done outside of psychology departments. The details about how to run the studies themselves, how to interact with subjects and other tacit knowledge about how to run a study, are often not available and either learned through trial and error, or available only through apprenticeship in a psychology or HCI lab. Researchers in psychology thus often end up appalled by the lack of this common but undocumented sense when behavioral research is performed and reported by researchers outside of psychology. This tutorial provides practical advice on how to run studies for beginning students and researchers coming starting to run studies. This tutorial will provide participants with an overview of how to run studies with human participants, that is, not how to design or analyze studies but the practicalities of how to setup, debug, and run studies. It will help people running experiments to run them more effectively safely, and comfortably. Our purpose is to provide hands-on knowledge about experimental procedure. The tutorial will cover the major topics noted in Figure 1. In particular, the tutorial will cover the role of identifying the research problem and reading in the general area; prepa- ration for running a study, including piloting and IRB pro- posals; preparing to run a formal study, including adver- tising and recruiting subjects; running study sessions; and wrapping up a study. The tutorial can be done as a half-day tutorial as it was at Cognitive Science 2012 in Japan, with around 50 participants, at the Chinese Academies in 2012 (20), at TU/Chemnitz in 2013 (20), and at an industrial site in 2013 (20) or it can be done as a full-day tutorial, as it was done at the BRIMS 2012 conference (12). I have a slight preference for a half-day, but realize that supporting both lengths may be useful to program organizers to have access to flexibility in organizing a program. Figure 1. A pictorial summary of the research process with respect to running a human behavioral study. This is similar to, but developed separately from Bethel and Murphy’s (2010) figure for human-robotic studies A copy of the 121 page tech. report as a printout will be provided (assuming that participant numbers can be speci- fied well enough in advance or copied by the conference locally). In addition, a 30% discount flyer or discounted copy of the book will be available; copies are available online through SageResearchMethods; and complimentary copies are available through Sage’s web site to instructors. (III) Why the presenter and authors are well suited to give a tutorial in the proposed area (II) How the tutorial will be delivered The tutorial will cover the topics in Figure 1 using a lecture/discussion format. The only AV needs are for a projector and screen. The topics will be introduced using a presentation and discussion will follow each section using scenarios and questions included in the book and developed for the Cognitive Science Conference. The presenter is well qualified to prepare and present a tuto- rial in this area. Along with colleagues, Ritter has recently written a book for Sage on this topic (Ritter, Kim, Morgan, & Carlson, under contract from April 2011).