Assessment in Instructional Games 1 Running head: ASSESSMENT IN INSTRUCTIONAL GAMES The Assessment of Learning in Instructional Games and Simulations

Games and simulations are of great interest to educators, in part because of their proven ability to engage and motivate players to recognize and solve difficult problems in situated contexts. They are compatible with many contemporary theories of learning and related methods of assessment. This review summarizes the research on the methods and difficulties of assessing learning outcomes from the use of games and simulations for instruction. Games and simulations are defined and their salient characteristics are described. They are then considered in terms of their purposes and functions within an educational context, followed by an analysis of the expected learning outcomes and methods of assessment. The review ends by addressing gaps in the literature and proposing beneficial areas for research. Assessment in Instructional Games 3 The Assessment of Learning in Instructional Games and Simulations The field of instructional systems technology has its origins in the visual instruction movement of the early twentieth century, and for many decades it was defined in terms of instructional media. One enduring hallmark of the field is the willingness of designers to experiment with new media and technologies and to study their effectiveness for teaching and learning. Each new medium—from film to radio and sound recording to television to computers and other digital devices—presents it own possibilities and challenges to the instructional designer, and each influences the theories and practices in the field. Digital games are the latest mass medium of popular culture, and interest in their study and use for learning has visibly increased in the last decade. The digital game industry now has annual revenues greater than the movie industry, leading some to identify a Games Generation (Prensky, 2001). The majority of today’s learners (K-12 and traditional first-time college students) have grown up with computers, video games, mobile phones, and portable media players. As a result, they are used to a variety of stimuli, they are adept multi-taskers, and they are enthusiastic adopters of new technologies. This generation’s cognitive style differs from previous generations in its preference for speed, multiple tasks, graphics over text, immediacy, exploration, and connection with others (Prensky, 2001). In 2003 the Pew Internet and American Life Project released the results of a survey of college students. Everyone surveyed had played a video, computer, or online game. Seventy percent played at least once in a while, and 65% played occasionally or regularly. About one in ten admitted that playing was a way of avoiding studying. One third admitted to playing games that were not part of instructional activities during class. A majority (69%) reported no exposure to gaming for educational purposes in the classroom (Jones, 2003). Assessment in Instructional Games 4 Teaching and learning in schools has changed little in the past one hundred years. As a result, learning in the classroom bears little resemblance to learning outside the classroom, leading to a greater degree of disengagement among learners. According to the most recent High School Survey of Student Engagement, 2 out of 3 students report that they are bored in class every day, and 17 percent say they are bored in every class (Yazzie-Mintz, 2007). Educators need to utilize the “cultural tools” of contemporary society to engage students in ways that are familiar to them (Strommen, 1992). The purpose of this review is to summarize the research on the methods and difficulties of assessing learning outcomes from the use of games and simulations for instruction. To provide a foundation, games and simulations are defined and their salient characteristics are described. Games and simulations are then considered in terms of their purposes and functions within an educational context, followed by an analysis of the expected learning outcomes and methods of assessment. The review ends by addressing gaps in the literature and proposing beneficial areas for research. Background and Definitions A Brief History The educational use of games and simulations dates back to war games in the seventeenth century and military training in the eighteenth century (Egenfeldt, 2005; Gredler, 2004). In the 1950s the practice was adapted for business management training, and in 1956 the American Management Association produced the first business game (Leemkuil, de Jong, & Ootes, 2000). As the capabilities of computing technologies have grown, increasingly complex and sophisticated games and simulations have been used for instruction in a variety of content areas, including medical education, the natural and social sciences, and corporate training. Cruickshank Assessment in Instructional Games 5 (1988) described several media-based (audio-visual) simulations from the 1960s and 1970s designed for preservice teachers. He noted that in the 1980s computer-based simulations became more prevalent and also more specialized with regard to content or focus. He cited as an example William Harless of the National Library of Medicine, who developed “an interactive videodiskbased simulation designed to teach clinical problem solving by enabling voice input to ask questions [and] order lab tests” (p. 151). In 1987, Faria (cited in Dempsey, Lucassen, Gilley, & Rasmussen, 1993-1994) reported that a survey revealed that 8,755 instructors in 1,900 business schools used business games in their courses. Research on the use of simulations and games for learning seems to be increasing. Rutter and Bryce (2006) compared the periods of 1995-1999 and 2000-2004 and found nearly twice as many peer-reviewed papers on digital games during the latter period. Bragge and Storgards (2007) used the ISI Web of Science to find 2,100 studies in more than 170 categories related to digital games between 1986 and 2006, with a significant increase beginning in 2003. However, much of the reporting on the use of games for learning is anecdotal, descriptive, or judgmental and not tied to theory or rigorous research (Gredler, 2004; Kirriemuir & McFarlane, 2003; Leemkuil et al., 2000; Washbush & Gosen, 2001; Wideman et al., 2007). A review of the literature by Dempsey, Rasmussen, and Lucassen (1996) consisted of 99 sources from the past 12 years. Building on an earlier article (Dempsey et al., 1993-94), the authors defined five categories of gaming articles: discussion (n=51), research (n=38), reviews (n=12), theory (n=11), and development (n=2). There have been relatively few studies of the use of games for learning in K-12 settings, and these have been primarily case studies, often involving students’ and teachers’ perceptions of learning (McFarlane, Sparrowhawk, & Heald, 2002; Wideman et al, 2007). Based on their Assessment in Instructional Games 6 review of the literature, Wideman et al. (2007) concluded that disciplines with the most research in educational gaming are medical education and business management studies. Bragge and Storgards (2007) combined the 170 categories found in their review into larger domains to find that the three most prominent areas were social sciences, health sciences, and information and communication technologies and mathematics. Definitions and Characteristics Games and simulations. A variety of definitions for “game” and “simulation” are presented in the literature. Wolfe and Crookall (1998) note that despite several decades as a field, researchers and practitioners in simulation and gaming are still grappling to create a generally accepted taxonomy. Gredler (2004) defines games as “competitive exercises in which the objective is to win and players must apply subject matter or other relevant knowledge in an effort to advance in the exercise and win,” while simulations are “open-ended evolving situations with many interacting variables . in which the participants take on bona fide roles with well-defined responsibilities and constraints” (p. 571). For Garris, Ahlers, and Driskell (2002), the key distinction is that a simulation represents reality and a game does not. However, a simulation may contain game-like features and may become a game if a performance goal is set. For example, SimCity is an endless simulation of a city. But if the player sets a particular goal, the simulation turns into a game. Salen and Zimmerman (2004) reviewed many of the major writers on games and simulations and synthesized their definitions: “A game is a system in which players engage in an artificial conflict, defined by rules, that results in a quantifiable outcome” (p. 80) and “[a] simulation is a procedural representation of aspects of ‘reality’” (p. 423). They contend that some simulations are not games but that all games are some form of simulation. Assessment in Instructional Games 7 Similarly, in their proposed taxonomy of simulations, Maier and Grossler (2000) distinguish between modeling-oriented and gaming-oriented simulations, the latter being singleuser simulators or multi-user planning games. However, others have offered different schemas. Leemkuil et al. (2000) categorize simulations based on underlying models: conceptual models based on principles, concepts, and facts related to a system and operational models based on sequences of procedures applied to a system. Gredler (2004) distinguishes types of simulations based on participant roles. Experiential simulations are social microcosms in which learners take on roles and responsibilities and participate in authentic scenarios and tasks. In symbolic simulations the learner is not a functional component of the system but operates on the system, testing her conceptual model of the relationships among the variables of the system. Games are often categorized by their genres, which include action, adventure, strategy, role-playing, racing, sports, shooting, word games, and puzzles. However, games may also be grouped by medium, such as board games, card games, video g