# Implicit Predictive Indicators: Mouse Activity and Dwell Time

Stephen Akuma, Chrisina Jayne, Rahat Iqbal, Faiyaz Doctor

<figure><img src="image_1.png" /></figure>

## To cite this version:

Stephen Akuma, Chrisina Jayne, Rahat Iqbal, Faiyaz Doctor. Implicit Predictive Indicators: Mouse Activity and Dwell Time. 10th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2014, Rhodes, Greece. pp.162-171, 10.1007/978-3-662-44654-6_16.hal-01391311

HAL Id: hal-01391311

https://inria.hal.science/hal-01391311

Submitted on 3 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Implicit Predictive Indicators: Mouse Activity and Dwell Time

Stephen Akuma, Chrisina Jayne, Rahat Iqbal, Faiyaz Doctor

Department of Computing and the Digital Environment, Coventry University, Coventry, UK
akumas@uni.coventry.ac.uk, {ab1527, aa0535, aa9536}@coventry.ac.uk

**Abstract.** Humans vary in their learning behaviour. It is difficult to predict the actual needs of learners through their search activity. It is also difficult to predict accurately the level of satisfaction after the learner finds a perceived relevant document. This research is a preliminary study to examine the predictive strength of some implicit indicators on web documents. An automated study was carried out and 13 participants were given 15 short documents to read and rate according to their perception of relevance to a given topic area. An investigation was carried out to examine if there exists a correlation between user generated implicit indicators and the explicit ratings. The findings show that there is a positive correlation between the dwell time and user explicit ratings. Although there was no significant correlation between mouse movement/distance and user explicit rating, there was a relationship between the homogeneous clusters of the implicit indicators and the user ratings.

**Keywords:** Implicit indicators, explicit rating, mouse activity, dwell time

## 1 Introduction

Information is constantly increasing on the internet leading to information overload. The process of accessing relevant information is not only difficult and time consuming but also cognitively demanding [2]. In order to support users in their search activities, search engine optimisation is required [6]. Implicit feedback and explicit feedback are broadly used to achieve this [6]. Both approaches focus on understanding users' behaviour and interest. In order for users to have recommended information based on their interest, a system must rate every actions of the user. In explicit feedback approach, the most common rating criterion is explicit rating - where users suggest to the system what they think about a given document or information [5]. Predicting users' interest explicitly can be done by users' preference information [19]. Although explicit rating is said to be the most consistent approach in information retrieval, however, its limitation is that it can alter reading and browsing patterns [5]. The implicit approach is therefore more feasible and objective measure. Implicit feedback systems understand the users' interest through the user behaviour. This saves the user the cost of providing feedback [21]. If the implementation of an implicit feedback
system is successful, it can be as good as explicit feedback. The advantages of implic-
it feedback over explicit feedback include:

- Large amount of data can be gathered with no extra cost to the searcher.

- The interaction between a user and the system can be captured at any time.

- With implicit feedback, users need not examine and rate items.

The adaptive engine of most feedback systems uses server-side data like page visited and link selected as its source of implicit data. The challenge in this approach is that the relevancy of a document to be recommended is simply judged by user visit and not by his/her interest on the documents. Client-side or browser approach to data (key, mouse and dwell time) collection provides an efficient alternative for accurately predicting relevant web documents [8]. The aim of interpreting this usage data is to predict user's perceived relevance of a document [4]. Client-side data have become common due to the fact that most browsers use JavaScript as their common technology. Claypool et al. [5] used "The Curious Browser" to examine promising implicit indicators that can predict relevant web documents. They found out that time spent on a page is a good indicator that signifies user's interest on a page. Kim et al [13] also supported this assertion with a focus on user active time on a page and they added that duration on a page is closely related to user's interest. The concept of contextualization was not examined by [5, 13].

In this work, we attempt to examine and correlate some implicit indicators with
user explicit ratings based on 15 web documents. We juxtaposed user's interaction on
15 web documents and their explicit ratings of how relevant the documents are to the
specific task. We used mouse activity and dwell time to represent the user's interest.

The remaining part of the paper is organized as follows: Section 2 presents a review of related work. Section 3 describes in detail the implicit indicators studied. Section 4 is on the approach used for the study. Section 5 presents and analyse the results. Section 6 is the discussion. Section 7 is the conclusion and future work.

## 2 Related work

Morita and Shinoda [15] used implicit approach to transparently capture data. They investigated if the time a reader spends reading Usenet news article can be an important source for measuring relevance. They infer that the longer a user spends reading an article, the more interesting it is to the user. Nichols [16] discussed the benefits of implicit feedback and he listed some behavioural characteristics that can be used as a source for implicit feedback. He however did not conduct any experiment to examine and validate the efficacy of the implicit indicators listed. Oard et al [18] studied how implicit indicators can be used in place of explicit ratings for a recommender system. They centred their observation on three broad categories of retention, examination and reference as useful criteria for making prediction. They also found that reading time is a good indicator for measuring relevance. This assertion was affirmed by [9, 12, 21].

Claypool et al. [5] studied the relationship that exists between different implicit
ratings and explicit ratings on a particular webpage. They developed a web browser
("The Curious Browser") to capture the following implicit indicators: mouse move-
ment, mouse click, scrolling and the time spent on a particular page. The main goal of their study was to collect some implicit indicators that are promising. They found out that although time spent on a particular webpage is a good indicator to predict user interest, a combination of time and scrolling activity gave a better prediction. Joachims [10] study focused on the use of click through as an implicit indicator for measuring relevance. They found out that click through closely follows relevant judgment and can use Ranked SVM algorithm to learn a ranking function. These studies [5], [10] argues that click through by itself is not a significant indicator to measure user interest.

Kim and Chan [13] used experimental measures to examine [16] list of promising indicators. Their focus was to know whether the time spent on a page signifies relevance. 11 students' behavioural characteristics were captured as they search the internet for 2 hours. Their findings suggest that the dwell time and distance of mouse movement are more reliable in predicting user interest than other indicators examined.

Buscher et al [3] examined the correlation between user reading behaviour and user explicit judgment. In the experimentation, participants were given 16 documents to read and rank them in relation to a given task area. An eye tracker was used to measure users' gaze traces on each of the documents. They found out that readers are induced in some way to documents that are of topical relevance. This research is somewhat similar to that of [3]. It however captures user's interest through mouse activity and dwell time.

# 3 Implicit feedback indicators

In this study, a number of implicit indicators were used to capture participant's interest on the given web documents. The implicit behaviour captured by the automated software includes:

## • Active Time Spent on the Document (TS)

Since users can open multiple browsers and run several applications, it is imperative to note that the active time on a particular window is the period at which the window or web document is in focus. The Active Time Spent (TS) also called Dwell time is the accumulated time spent by a user on an active page during browsing. It starts counting immediately the page is open and stops when focus is moved away from the page or when the page is closed.

## *Hypothesis*

The more the time the page is active, the more it is interesting, informative and of topical relevance [2], [5], [7], [12, 13, 14, 15], [22]. Display time can be used as a substitute for eye tracker for a sophisticated feedback mechanism [4].

## Mouse Activities

Most people move the mouse when they read a web document. A user may move the mouse more frequently when viewing a document of interest. The mouse activities are
captured only when the current document is in focus. When a user views another application or documents that are not related to the task, the mouse activities are not captured.

### Distance of Mouse Movement (DMM)

The distance of mouse movement is calculated by its x and y coordinates on the monitor. The formula is given as:

$$DMM = \sum_{i=1}^{n} \sqrt{(x - x_i)^2 + (y - y_i)^2} \quad (1)$$

Where x and y are the mouse location along the x and y coordinates of the monitor screen.

#### Hypothesis

The more the distance the mouse move, the more interesting is the page to the user [5], [11], [13], [20].

### Total Mouse Movement (TMM)

This is the total mouse movement calculated by its x and y coordinates on the monitor. The count increment as the mouse hovers on the document.

#### Hypothesis

The more the mouse hovers on a page, the more interesting the page. Cursor movement complements dwell time information [7].

### Total Mouse Velocity (TMV)

This is the total speed covered by the mouse on the monitor. It is given as:

$$TMV = \sum_{i=1}^{n} \sqrt{(x - x_i)^2 + (y - y_i)^2} / (t - t_i) \quad (2)$$

Where x and y are the mouse location along the x and y coordinates. And t is the time of the mouse distance in an interval of 100ms.

#### Hypothesis

When the mouse speed is low, it shows that the reader is actually reading the document and not skimming it. Therefore, the more the mouse speed the less the user's interest on the page. Cursor movement and cursor speed especially along the vertical axis of a Search Engine Result Page are good predictive indicators of document relevance [7].

### Explicit Ratings (ER)
This is the actual user judgement of the visited documents. The buttons for explicit
rating was attached on top of each document on a 0 to 5 rating scale. After reading a
web document, the user rated the document according to his/her interest and topical
relevance.

4 Study design

The main aim of the study was to capture participant's interest on web documents via some implicit indicators and to correlate the user's interest against their explicit ratings of the given documents. The participants were 4 PhD students, 1 Research assistant, 6 MSc students and 2 undergraduate students. Data for this research was collected by our own automated software developed with JavaScript. The software was injected in 15 web documents to record users' mouse activity, dwell time and explicit rating. Participants were given a task brief to read and a consent form to complete, after which they were allowed to perform the experiment at their convenient time. They were to login into a website containing links to the 15 web documents and read the documents. The implicit data was captured unobtrusively as the participants read through the documents and then sent to MySQL database when they rated the document by clicking on any of the buttons on the rating scale as shown in fig 2. The task ended after the participants read rated the 15 documents. Fig 1 shows a step to step schema of the task process.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Step to step schema of the task process</figcaption></figure>

4.1 User task

The same task was given to all the participants. They were asked to prepare a presen-
tation on the topic - Ethical issues in Big Data. We provided them with 15 documents
of equal length containing 350 words with a font size of 20px and a font type of Arial,
making the documents one screen view. The documents were created from web arti-
cles on ethical issues in Big Data. Two of the documents were however not related to
the topic. The participants were asked to read each of the 15 documents and rate them
according to how relevant the documents are to the task related topic. The rating was
on a scale of 0 - 5. Six buttons were attached on top of the documents and labelled 0
to 5 for explicit rating of the documents: 5 - means very relevant; 4 - means more
relevant; 3 - means relevant; 2 - means slightly relevant; 1 - means very low rele-
vance; 0 - means not relevant
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Login page, Index page with links to the 15 documents and document page with explicit rating buttons</figcaption></figure>

The experiment was given a realistic feel by creating a second phase of the task which we called the presentation writing phase. The participants were told that documents will be presented to them according to how they rated them for later use in the presentation writing phase. To avoid Hawthorne effect (the alteration of behaviour by the subjects of a study due to their awareness of being observed), participants were told to do the experiment when and where they are most comfortable. The participants did not actually perform the second phase of the experiment (the presentation writing phase). Fig. 2 shows the user interface of the automated software.

# 5 Results

This section analyses the data collected from the participants. The analysis is as follows:

## 5.1 Implicit/Explicit Relationship

Each of the participant's data was analysed separately and as a group. Pearson Correlation Coefficient was used to correlate the parameters of active time, total mouse distance, average mouse velocity, total mouse movement along the x and y axis with user explicit rating. We obtain a correlation coefficient of 0.21 between the user rating and the dwell time. We then ranked the implicit indicators by their predictive strength, using stepwise linear regression to obtain the most promising predictive indicators. The indicators that showed much prominence in relation to the explicit ratings were the dwell time and mouse movement along the x-axis.

The result from the experiment shows a positive correlation between the explicit rating and the dwell time. The dwell time also has a positive correlation with the mouse movement/mouse distance. Fig. 3 shows a Box plot of varying median of the user explicit rating and the dwell time of the participants, and it shows that the values of the ratings for 3 and 4 is the most consistent. The inconsistencies in the other values might be due to noise in the data. The Kruskal-Wallis test on the median for each
of the explicit ratings rejected the null hypothesis, meaning that the median values are
not the same.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Graph showing the Boxplot for the combination all the participant's time/explicit rating relationship</figcaption></figure>

We also found out that although users vary in their reading behaviour, some of them have similar behavioural pattern. We analysed the first two participants' data separately to find out the extent of individual differences. We discovered that they have a relatively similar pattern in dwell time and mouse activity on the documents visited as shown in Fig. 4.

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Graph showing participant 1 and 2 dwell time, mouse distance and average speed on the documents.</figcaption></figure>

In order to regroup the user ratings, the five levels of ratings were then reduced into two levels - Relevant and Non-relevant. The user ratings from 0 to 2 were merged together and represented as 0 (Non-relevant) while the ratings from 3 to 5 were represented as 1 (Relevant). A Multilayer Perceptron was used to conduct further analysis on the primary data set (mouse movement along the x and y axis, dwell time and mouse velocity time count) to predict relevant and non-relevant user rating. We obtained a 65% successful mapping with the user ratings after testing the trained data set. Table 1 shows the training and testing of the observed samples.
<table><caption>**Table 1.** Multilayer Perceptron cluster analysis</caption><thead><tr><th rowspan="2">Sample</th><th rowspan="2">Observed</th><th colspan="3">Predicted</th></tr><tr><th>0</th><th>1</th><th>Percent Correct</th></tr></thead><tbody><tr><td rowspan="3">Training</td><td>0</td><td>40</td><td>27</td><td>59.7%</td></tr><tr><td>1</td><td>23</td><td>45</td><td>66.2%</td></tr><tr><td>Overall Percent</td><td>46.7%</td><td>53.3%</td><td>63.0%</td></tr><tr><td rowspan="3">Testing</td><td>0</td><td>13</td><td>9</td><td>59.1%</td></tr><tr><td>1</td><td>10</td><td>23</td><td>69.7%</td></tr><tr><td>Overall Percent</td><td>41.8%</td><td>58.2%</td><td>65.5%</td></tr></tbody></table>

Dependent Variable: user_rating

# 6 Discussion

The most promising indicator in the measure of perceived relevance is the active time. Participants spent more time on documents perceived to be of topical relevance. We also found some individual behavioural differences among the participants while reading. The correlation of mouse activity with the explicit rating is relative low probably because of the length of the documents which were of 350 words and could be mostly viewed on screen at once. There is however evidence of a positive relationship between dwell time and explicit user rating. This conclusion is in line with previous research conducted in relation to implicit feedback [1], [12, 13, 14, 15], [17], [22]. The mouse distance/ movement are closely related to the dwell time. We can substitute in some way the dwell time by mouse movement or mouse distance in an implicit system.

We can also infer that learners dwell more on documents that are of topical importance and interest to their current activity. The effect of the concept of prior knowledge and cognition on the reader's behaviour was not examined. We assumed that the selected participants barely had knowledge of the task domain. We also observed some individual behavioural differences among the participants. To examine the variety in reading behaviour, we took a closer look at the data for two of the participants (Participants 1&2) and we discovered some level of similarity in their behaviour in terms of dwell time and mouse activity.

Multilayer Perceptron was used to further analyse the primary data set, with user rating as the dependent variable. We obtained a fair result of 65% mapping after training the data set. This suggests a relationship between the user behaviour and their explicit ratings.

Since the experiment was not closely monitored in a controlled environment, we did not get a significant correlation between the user ratings and the mouse activities as hypothesized; we suspect that some of the participants did not follow the instructions carefully as stipulated for the task. This probably added some noise to the data collected.

# 7 Conclusion and future work

Implicit data can be used to predict user's interest on a web documents. When users give explicit feedback, it affects their normal reading or browsing pattern [5]. Implicit
methods are a cost effective and objective approach of rating user's interest. This
study shows that dwell time on a document is influenced by user perceived relevance
and topicality. We also found a relationship between homogenous clusters of user
reading behaviour and their explicit ratings.

The next phase of this research is to develop an add-on that will implicitly capture
user data on the web as they browse. Additional implicit indicators like copy and
paste [6], amount of scroll and keystroke will be examined. The data analysis will be
centred on aggregating the most promising indicators and developing a model for
effective personalization of relevant documents to learners based on their interest.
The effect of document familiarity will also be examined.

References

1. Akuma, S.: Investigating the Effect of Implicit Browsing Behaviour on Students' Performance in a Task Specific Context, International Journal of Information Technology and Computer Science(IJITCS), 6(5), 2014, 11-17.

2. Brusilovsky, P., Tasso, C.: Preface to special issue on user modeling for Web information retrieval, User Modelling and User-Adapted Interaction, 14(2-3), 2004, 147-157.

3. Buscher, G., Dengel, A., Biedert, R., Van Elst, L.: Attentive Documents: Eye Tracking as Implicit Feedback for Information Retrieval and Beyond, ACM Transactions on Interactive Intelligent Systems, 2(1), 2012, 1-30.

4. Buscher, G., Van Elst, L., Dengel, A.: Segment-level display time as implicit feedback: A comparison to eye tracking, Proceedings - 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, 2009, 67-74.

5. Claypool, M., Le, P., Wased, M., Brown, D.: Implicit interest indicators, International Conference on Intelligent User Interfaces, Proceedings IUI, 2001, 33-40.

6. Grzywaczewski, A., Iqbal, R.: Task-Specific Information Retrieval Systems for Software Engineers, Journal of Computer and System Sciences, Elsevier, 78(4), 2012, 1204-1218.

7. Guo, Q., Agichtein, E.: Beyond dwell time: Estimating document relevance from cursor movements and other post-click searcher behavior, WWW'12 - Proceedings of the 21st Annual Conference on World Wide Web, 2012, 569-578.

8. Hauger, D., Paramythis, A., Weibelzah, S.: Using browser interaction data to determine page reading behavior, 2011.

9. Huai, Y.: Study on ontology-based personalized user modeling techniques in intelligent information retrievals, 2011 IEEE 3rd International Conference on Communication Software and Networks, ICCSN 2011, 2011, 204-207.

10. Joachims, T.: Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July, 2002.

11. Jung, K.: Modeling web user interest with implicit indicators, Master Thesis, Florida Institute of Technology, 2001.

12. Kellar, M., Watters, C., Duffy, J., Shepherd, M.: Effect of task on time spent reading as an implicit measure of interest, Proceedings of the ASIST Annual Meeting, 41, 2004, 168-175.

13. Kim, H. -, Chan, P. K.: Implicit indicators for interesting Web pages, WEBIST 2005 - 1st International Conference on Web Information Systems and Technologies, Proceedings, 2005, 270-277.
14. Lee, T. Q., Park, Y., Park, Y. -: A time-based approach to effective recommender systems using implicit feedback, Expert Systems with Applications, 34(4), 2008, 3055-3062.

15. Morita, M., Shinoda, Y.: Information Filtering Based on User Behaviour Analysis and Best MatchText Retrieval, 1994, 272-281.

16. Nichols, D. M.: Implicit Rating and Filtering, 1997, 31-36.

17. Núñez-Valdéz, E. R., Cueva Lovelle, J. M., Sanjuán Martínez, O., García-Díaz, V., Ordoñez De Pablos, P., Montenegro Marín, C. E.: Implicit feedback techniques on recommender systems applied to electronic books, Computers in Human Behavior, 28(4), 2012, 1186-1193.

18. Oard, D., Kim, J.: Implicit feedback for recommender systems, 1998, 81-83.

19. Takano, K., Li, K. F.: An adaptive personalized recommender based on web-browsing behavior learning, Proceedings - International Conference on Advanced Information Networking and Applications, AINA, 2009, 654-660.

20. Zemirli, N.: WebCap: Inferring the user's interests based on a real-time implicit feedback, 7th International Conference on Digital Information Management, ICDIM 2012, 2012, 62-67.

21. Zhang, B., Guan, Y., Sun, H., Liu, Q., Kong, J.: Survey of user behaviors as implicit feedback, 2010 International Conference on Computer, Mechatronics, Control and Electronic Engineering, CMCE 2010, 6, 2010, 345-348.

22. Zhu, Z., Wang, J. -, Chen, M. -, Huang, R. -: User interest modeling based on access behavior and its application in personalized information retrieval, Proceedings - 3rd International Conference on Information Management, Innovation Management and Industrial Engineering, ICIII 2010, 1, 2010, 266-270.