Psychological Research Online : Opportunities and Challenges

As the Internet has changed communication, commerce, and the distribution of information, so too it is changing psychological research. Psychologists can observe new or rare phenomena online and can do research on traditional psychological topics more efficiently, enabling them to expand the scale and scope of their research. Yet these opportunities entail risk both to research quality and to human subjects. Internet research is inherently no more risky than traditional observational, survey or experimental methods. Yet the rapidly changing nature of technology, norms, and online behavior means that the risks and safeguards against them will differ from those characterizing traditional research and will themselves change over time. This paper describes some benefits and challenges of conducting psychological research via the Internet and offers recommendations to both researchers and Institutional Review Boards for dealing with the challenges. Send comments and editorial correspondence to: Robert Kraut HCI Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh PA 15213 robert.kraut@cmu.edu 412 268-7694 Psychological Research Online: Opportunities and Challenges Robert Kraut, Judith Olson, Mahzarin Banaji, Amy Bruckman, Jeffrey Cohen, Mick Couper, The Internet as a research vehicle presents both opportunities and challenges for psychological research. In 1985, only 8.2% of US households had a personal computer, and the Internet as we now know it, with its rich array of communication, information, entertainment, and commercial services, did not exist. Since then, this exotic technology has become domesticated and is now used by the majority of Americans for personal and economic reasons (Cummings & Kraut, 2002). By September of 2001, 66% of the US population used a computer at home, work, or school, and the vast majority of these, 56% of the US population, also used the Internet (U. S. Department of Commerce, 2002). The Internet and the widespread diffusion of personal computing have the potential for unparalleled impact on the conduct of psychological research. For example, the Internet has changed the way scientists collaborate, by increasing the ease with which they can work with geographically distant partners (Walsh & Maloney, 2002) or share information (e.g., http://www.socialpsychology.org/). In this article we will focus on the way the Internet is changing the process of empirical research. The Internet presents empirical researchers with opportunities. It lowers many of the costs associated with collecting data on human behavior, can host online experiments and surveys, allows observers to watch online behavior, and offers the mining of archival data sources. For example, online experiments can collect data from thousands of participants with minimal intervention on the part of experimenters (B. A. Nosek, M. Banaji, & A. G. Greenwald, 2002a). Internet chat rooms and bulletin boards provide a rich sample of human behavior that can be mined for studies of communication (Nardi & Whittaker, 2001), prejudice (Glaser, Dixit, & Green, 2002), organizational behavior (Orlikowski, 2000), or diffusion of innovation (Kraut, Rice, Cool, & Fish, 1998), among other topics. The Internet is also a crucible for observing new social phenomena, such as the behavior of very large social groups (Sproull, 1995), distributed collaboration (Hinds, 2002), and identity-switching (Turkle, 1997), which are interesting in their own right and have the potential to challenge traditional theories of human behavior. At the same time, the Internet raises substantial challenges in terms of quality of data and the treatment of research participants. For example, researchers often lose control over the context in which data are procured when subjects participate in experiments online. Insuring informed consent, explaining instructions, and conducting effective debriefings may be more difficult than in the traditional laboratory experiment. Observations in chat rooms and bulletin boards raise difficult questions about risks to participants, including privacy and lack of informed consent. This article will discuss both the advantages of this new mode for psychological research as well as the challenges that it poses to data quality and the protection of research participants. APA-Internet Version 3.3 9/30/2003 Page 3 After discussing the opportunities and challenges of conducting online research, we close with recommendations in light of these challenges, directed toward both the researcher and the Institutional Review Boards that oversee the protection of human research subjects. We focus our attention primarily on online experiments, surveys, and observation of naturally occurring online behavior, because these are the major types of research conducted currently by psychologists who use the Internet. Furthermore, these methods have obvious parallels in the off-line (non-Internet) world that can be used as yardsticks by which to compare the online methods. Opportunities of Internet research The Internet can have positive impact on the conduct of psychological research, both by changing the costs of data collection and by making visible interesting psychological phenomena that do not exist in traditional settings or are difficult to study there. Making empirical research easier Compared to other modes of collecting data, the Internet can make observational research, self-report surveys, and random-assignment experiments easier to conduct. This ease derives largely from two properties of Internet research: economy and access. Subject recruitment. Use of the Internet decreases the cost of recruiting large, diverse, or specialized samples of research participants for either surveys or online experiments. Many researchers attract volunteers by posting announcements at relevant web sites and distribution lists. This technique can provide a large a diverse sample at low cost. For example, in four years, Nosek, Banaji, and Greenwald (2002b) collected a data set of over 1.5 million completed responses in tests of implicit attitudes. (See Sidebar 2). A survey on online behavior collected data from 40,000 respondents from many countries (Wellman, Quan Haase, Witte, & Hampton, 2001), simply by putting a link to the survey on a National Geographic website. On a smaller scale, the research reported in Sidebar 4 (Williams, Cheung, & Choi, 2000) conducted a pair of online experiments about ostracism, with over 1,500 participants from over 60 countries. And those conducting usability tests of websites can merely post “try this new page and give us your reactions” on a busy website and get thousands of responses within hours. One can post a research opportunity at service sites that specialize in advertising the availability of such opportunities, such as the one hosted by the Social Psychology Network (http://www.socialpsychology.org/expts.htm) or the American Psychological Society (http://psych.hanover.edu/APS/exponnet.html). Commercial services, such as Survey Sampling, Inc. (http://www.surveysampling.com) are available to aid in selecting a sample. Alternately, one can invite participation by sending personalized electronic mail messages to active participants in either specialized or more general online communities (See Couper, Traugott, & Lamias, 2001 for a review of sampling approaches for Internet surveys.) In one sense, the Internet has democratized data collection. Researchers do not need access to introductory psychology classes to recruit research subjects and often do not need grant money to pay them. The Internet has opened research to those with fewer APA-Internet Version 3.3 9/30/2003 Page 4 resources. One consequence is that faculty at small schools, independent scholars, graduate students, and undergraduates can all potentially contribute to psychological research. For example, an undergraduate psychology major, Nicholas Yee, published findings about the psychology of playing online multi-player games, based on 19 surveys he directed to players of the Internet game EverQuest between September 2000 and April 2001, collecting over 18,500 responses from approximately 3,300 players. However, a corollary of this open access is that those with minimal training and supervision can conduct and publish research, some of which might be of low quality. Yee’s research results, for example, are available on his own website (www.nickyee.com) but have not been published in any peer-reviewed venue. Regardless of the quality of this research, his intense polling of a single population has polluted this data source for researchers who may be more qualified. In this sense, the tragedy of the commons has now threatens psychological research (Hardin, 1968). In an another case, an undergraduate, Martin Rimm, published a study in the Georgetown Law Review (Rimm, 1995) reporting on the prevalence of pornography, using research methods that have been heavily disputed (Thomas, 1996). Observing social behavior. The Internet provides scientists interested in social behavior with many archives of communication, from online groups in discussing topics as diverse as medical support, hobbies, popular culture, and technical information (e.g., see the newsgroups archives at http://groups.google.com/groups or the collections of email-based distribution lists at http://tile.net/lists/). Researchers have used these online groups to study such social processes as personal influence (Cummings, Sproull, & Kiesler, 2002), negotiation (Biesenbach-Lucas & Weasenforth, 2002), and identity formation (McKenna & Bargh, 1998). Many online forums make visible psychological phenomena that would be much more difficult to study in traditional settings. Some phenomena, like the evolution of groups or long-term learning, are ordinarily difficult to study in controlled settings because of the difficulty of bringing subjects back to the laboratory many times. Research in social psyc