Testing theories of skill learning using a very large sample of online game players

Testing theories of skill learning using a very large sample of online game players Tom Stafford (t.stafford@sheffield.ac.uk) Department of Psychology, University of Sheffield Western Bank, Sheffield, S10 2TP, UK Michael Dewar (michael.dewar@nytimes.com) The New York Times RD learning; game. Introduction The investigation of skill learning suffers from a dilemma. One horn of the dilemma is this: experts in real-world skills can be brought into the lab and their performance tested, but it is difficult to reliably recover comprehensive details of their training. This makes it impossible to be certain of exactly how features of the history of their practice are related to the skilled performance you can observe. The other horn of the dilemma is this: you can test different training regimes rig- orously, but you are restricted to measuring performance on trivial or unnatural skills, and often without extended training of the order that experts in complex real-world skills engage in. Computer games offer a partial resolution to this dilemma. Even simple computer games are not trivial in terms of the cognitive abilities which they test. In fact, these abilities are often the staples of cognitive science: perception, decision making and motor responses. Computer game playing is a real-world skill in which many people choose to become ex- pert, devoting hundreds of hours of practice. Unlike most skills, computer games allow a potential record every action in the history of that practice — allowing for the first time detailed investigation of the connection between features of practice and level of final performance. This is what the cur- rent investigation sets out to do. We take detailed records of practice activity from an online game and relate amount of practice and features of practice to levels of eventual per- formance. In doing this we are able to confirm and quan- tify established findings from experimental studies of learn- ing. In addition we provide a confirmation of a recent result based on the theoretical framework of reinforcement learn- ing (Stafford et al., 2012). Use of online games to collect very large samples offers a new method for the investigation of skill acquisition, we argue, and the work here showcases just some of the possibilities opened up by this approach. Practice amount and spacing We first consider two well established results against which we will validate our data set as a model of skill acquisi- tion: the effects of practice amount and of practice spacing on performance. Studies of learning have shown a lawful re- lation between practice amount and performance. If perfor- mance is gauged in terms of some measure of efficiency (e.g. time taken to make cigars by experienced cigar manufactur- ers Crossman, 1959), then it is possible to express the relation between practice extent and performance in a power law of learning (Newell & Rosenbloom, 1981; Ritter & Schooler, For practical reasons studies of the effect of extensive prac- tice have typically looked at different learners possessing dif- fering amounts of practice rather than the same learners at different stages (i.e. cross-sectional rather than longitudinal designs). Experimental studies of learning which do follow learners longitudinally have predominantly focussed on lab- based tasks which can be mastered in one or a small number of sessions (although there are, of course, honourable excep- tions such as the work looking at the automatisation of visual search performance (e.g. Neisser, Novick, & Lazar, 1963; Czerwinski, Lightfoot, & Shiffrin, 1992). Highlighting the importance of practice quantity in skill development, Ericsson and colleagues stress that the highest levels of performance are never reached without an amount of practice on the order of ten thousand hours (Ericsson, 2006; Ericsson, Krampe, & Tesch-Rmer, 1993). Additionally, they report that the nature of that practice matters — effortful, di- rected, ‘deliberate’ practice is what distinguishes elite per- formers, even among those who appear to have performed similar quantities of practice. Experimental studies of learning have focussed on another factor which defines the nature of practice — spacing. The distributed practice effect denotes the finding that if time de- voted to practice is separated out rather than massed, or if the spacing is larger rather than smaller, retention improves