Interpreting Eye-Movement Protocols

Interpreting Eye-Movement Protocols Dario D. Salvucci (dario@cbr.com) John R. Anderson (ja+@cmu.edu) Cambridge Basic Research; Four Cambridge Center Cambridge, MA 02142 Department of Psychology; Carnegie Mellon University Pittsburgh, PA 15213 Eye movements reveal a great deal about our thoughts and intentions. Exploiting this benefit, researchers have utilized eye movements increasingly as a tool for understanding human behavior at a fine-grained level. However, the popularity of eye-movement data has been tempered by the difficulty of analyzing these data, which typically contain a great deal of individual variability and equipment noise. Researchers must often choose between analyzing a small number of protocols by hand or analyzing a larger number of protocols with very coarse, aggregate measures. We have developed a class of methods that automate the analysis of eye-movement protocols (Salvucci & Anderson, 1998; Salvucci, 1999). The methods analyze, or interpret, these protocols by means of tracing N mapping the observed sequence of eye movements to the sequential predictions of a cognitive model. The tracing process begins by running the cognitive model and generating sequence(s) of predicted thoughts and actions. The tracing process then determines the correspondence between an observed protocol and the predicted sequence that best matches the protocol. Our tracing methodology includes three methods of varying complexity and accuracy. The simplest method, target tracing, performs tracing using a sequence-matching algorithm popularized for user protocol studies (Card, Moran, & Newell, 1983). The two more sophisticated methods, fixation and point tracing, utilize hidden Markov models, powerful statistical tools that have been applied with great success in speech and handwriting recognition (see Rabiner, 1989). All three methods provide fast and accurate interpretations and are robust in the presence of noise and variability. The tracing methods have been implemented into a working system, EyeTracer, that provides an interactive environment for manipulating, replaying, viewing, and analyzing protocols. 1 We have rigorously tested the tracing methods in three illustrative domains: equation solving, reading, and Oeye typingO. In the equation-solving domain, we collected protocols from students solving equations of a particular form and compared the interpretations of the tracing methods to those of expert human coders. Results showed that the tracing methods interpreted the protocols as accurately as the human experts in significantly less time (at least an order of magnitude difference). We also applied the tracing methods with a Otrace-based methodologyO (Ritter & Larkin, 1994) to develop a cognitive model of student behavior in the task. The tracing methods facilitated both exploratory and EyeTracer is publicly available on the World Wide Web at . confirmatory analysis of the protocols and resulted in a successful model of student behavior. In the reading domain, we evaluated the ability of the tracing methods to compare cognitive models with respect to their sequential predictions. For this purpose, we used two competing models of eye-movement control in reading, E-Z Reader 3 and E-Z Reader 5 (Reichle et al., 1998); these models produced similar predictions of non-sequential measures, but E-Z Reader 5 produced qualitatively better predictions of sequential measures. By tracing a reading data set using these models, the tracing methods provided quantitative evidence that E-Z Reader 5 was indeed the better model. The tracing methods also significantly cleaned up the data and facilitated analysis of aggregate duration and fixation probability measures. In the Oeye-typingO domain, computer users typed words by looking at letters on an on-screen keyboard. Unlike earlier eye-typing interfaces, our interface had no restrictions on how long users needed to fixate letters; this feature facilitated fast input but complicated interpretation of user eye movements. Provided with a model of user input, the tracing methods greatly facilitated data analysis and resulted in faster, more accurate user input than was possible using earlier analysis methods. Acknowledgments This work is based on the first authorOs dissertation and was funded in part by Office of Naval Research grant N00014-95- 10223 awarded to John R. Anderson. References Card, S. , Moran, T. , & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Erlbaum. Rabiner, L.ER. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77, 257-286. Reichle, E. D. , Pollatsek, A. , Fisher, D. L. , & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105, 125-157. Ritter, F.EE. , & Larkin, J.EH. (1994). Developing process models as summaries of HCI action sequences. Human- Computer Interaction, 9, 345-383. Salvucci, D. D. (1999). Mapping eye movements to cognitive processes. Doctoral Dissertation, Department of Computer Science, Carnegie Mellon University. Salvucci, D. D. , & Anderson, J. R. (1998). Tracing eye movement protocols with cognitive process models. In Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society. Hillsdale, NJ: Erlbaum.