Comparative Analysis of Semantic Models and Corpora Choice when using Semantic Fields to Predict Eye Movement on Webpages

Comparative Analysis of Semantic Models and Corpora Choice when using Semantic Fields to Predict Eye Movement on Webpages Benjamin P. Stone (bpstone@psychology.adelaide.edu.au) School of Psychology, Level 4, Hughes Building, The University of Adelaide Adelaide, SA 5005 Australia Simon J. Dennis (dennis.210@osu.edu) Department of Psychology, Ohio State University, 1827 Neil Ave Columbus, OH 43210 USA Abstract Nine models are compared in their ability to predict eye- tracking data that was collected from 49 participants’ goal- oriented search tasks on a total of 1809 webpages. Forming the basis of six of these models, three semantic models and two corpus types are compared as components for the Seman- tic Fields model (Stone and Dennis, 2007) that estimates the semantic saliency of different areas displayed on webpages. Latent Semantic Analysis, Sparse Non-Negative Matrix Fac- torization, and Vectorspace were used to generate similarity comparisons of goal and webpage text in the semantic compo- nent of the Semantic Fields model. Surprisingly, Vectorspace was consistently the best performing model in this study. Two types of corpora or knowledge-bases were used to inform the semantic models, the well known TASA corpus and other cor- pora that were constructed from the Wikipedia encyclopedia. In all cases the Wikipedia corpora out performed the TASA corpora. Three other baseline models: Flat, Non-Flat, and No-Model were included as a point of comparison to evaluate the effectiveness of the Semantic Fields models. In all cases the Semantic Fields models outperformed the baseline models when predicting the participants’ eye-tracking data. Keywords: LSA; Vectorspace; SpNMF; Semantic Fields; web pages; eye-tracking, goal-directed visual search Introduction The exponential increase in Internet usage over the last decade has motivated psychological researchers to examine web users’ behavior in this virtual environment. Research fo- cussing on user behavior in web page environments can gen- erally be delineated into two main streams: display-based and semantics-based research. While both methods to some de- gree attempt to predict the area on a webpage that a user will focus their attention on, they approach this task in different ways. Display-based research has focused on perceptual as- pects of the webpage, components such as element and menu position, color usage, and font style. Alternatively, when attempting to predict user’s webpage navigation, semantic- based research matches web user’s information needs to the concepts displayed within the textual content of webpages. Display-based and semantics-based research into web user’s visual search of web page hyperlinks has indicated that the user’s search processes are influenced by: text semantics, el- ement position, aesthetic qualities of elements, and environ- mental learning (Faraday, 2000, 2001; Ling & Van Schaik, 2002, 2004; Chi et al., 2003; Cox & Young, 2004; McCarthy, Sasse, & Rigelsberger, 2003; Pearson & Van Schaik, 2003; Pirolli & Fu, 2003; Rigutti & Gerbino, 2004; Blackmon, Ki- tajima, & Polson, 2005; Kaur & Hornof, 2005; Pirolli, 2005; Stone & Dennis, 2007). Several researchers have highlighted the importance of combining display-based and semantic information when modelling user’s navigation through web sites (Blackmon, Polson, Kitajima, & Lewis, 2002; Chi et al., 2003; Pirolli & Fu, 2003; Kaur & Hornof, 2005; Stone & Dennis, 2007). Research that has combined display and semantic informa- tion when predicting web users’ behavior include the Cogni- tive Walkthrough for the Web (CWW, Blackmon, Kitajima & Polson, 2005), the Bloodhound Project (Chi et al., 2003), and the Latent Semantic Analysis - Semantic Fields model (LSA- SF, Stone & Dennis, 2007). For a detailed description of the CWW and Bloodhound Project the reader is directed to Stone and Dennis (2007). One major unanswered question when integrating display- based and semantic information is the nature of the seman- tic model itself, and the data upon which it should rely. As discussed previously, the computational semantics literature provides multiple options for semantic models, and similarly there are several approaches one might take in specifiying a corpus relevant to the task (Stone, Dennis, & Kwantes, sub- mitted). In this paper, we aim to shed light on these issues, by comparing different models and corpora on their ability to predict human eye movements during web browsing. Semantic Fields (SF) In a previous article, we presented the LSA-SF model (Stone & Dennis, 2007), which was used to predict the eye move- ments of 49 participants recorded during goal-oriented search tasks on three websites. The LSA-SF model used LSA to cal- culate the similarity between a textual representation of the users’ goal and each of the textual elements displayed on a webpage. Using a decay function, these LSA estimates of similarity were then distributed and summed over each pixel position for all of the textual elements contained on a web- page (see Equation 1). Combining the semantic information (L) with distance (d i(x,y) ) from its display position using a de- cay function, enabled the production of maps of information density for each web page in our study (see Figure 1). SF ( x, y) = ∑ L i e −λd i(x,y) i Initially, the TASA corpus was used as a knowledge based