# Design and Applications of GLANCE: GLanceable Alarm Notification for a User Centered Experience
## 3 Results of the 1 st Macro-Phase: The GLANCE Project
### Third Iteration: Final Release of the Proof-of-Concept and Definition of a General Formal
#### 3.3.3 Evaluation and Learning
The notification sequence included information related to 16 different events and a total of 13 animations of display content (Figure 9c is actually an example notification from the tested prototype), with a uniform distribution of notifications and display changes over the four notification levels (change-blind changes, make-aware, interrupt, and demand-action). A pilot test was conducted with internal users to tune the timing of the notifications relatively to the note-taking activities of observers. Based on the results of the pilot test, we decided to have two observers per user, alternating within a single user observation during the sequence of display changes, so that each of them could have enough time for recording details about the user's reaction to each display change. Observers were trained to judge user's reaction based on users' eye movements and face expression in response to a notification and were asked to collect structured quantitative data to be organized as an _observation matrix:_\(displayChange\times user\to rating\), where _rating_ was to be assigned by the observers according to a four values scale depending on the attention zone corresponding to the user reaction (0 = inattention, 1 = divided attention, 2 = focused attention, and 3 = focused attention plus action). The _observation matrix_ was to be compared with the _\"expected matrix\"_ after the experiment to measure the distance between actual users' behavior and expected behavior based on the notification level. _Experiment sessions._ A total of 15 participants (six female), aged 22-51 (AVG = 33,6; M = 30), took part in the study on a voluntary basis, recruited among students and personnel of the university campus (five participants were Master students, two PhD students, two laboratory technicians, two were from administrative personnel, and four were employees not using computers for their jobs). None of the participants had previous experiences with the tested notifier, but seven of them had been involved in the scalability test of the previous prototype. Participants were invited individually to the lab and were given a brief introduction to the experiment. A facilitator gave a description of GLANCE features and explained to users what they were requested to do, what they would be observed for, and how data collected from the study would be used. Then each user was asked to read a one-page text (foreground task) while simultaneously monitoring the asynchronous notifications delivered by the GLANCE display (background task). In the case of critical notifications (corresponding to the demand-action level) they were requested to click on the display, while in all other cases they were not asked to do anything in particular. Participants were not instructed on how to prioritize the two tasks nor were they preliminarily informed about the speed with which the notifier would update (actually, notifications arrived in an asynchronous manner). At the end of the session, participants were asked to recall details of both the read text and information notified by the GLANCE display by answering a questionnaire with 12 questions about event information delivered by GLANCE during the test. Furthermore, in a post-experiment debriefing, participants were asked to subjectively evaluate GLANCE as to intuitiveness, glanceability, intrusiveness, and usefulness of the display. _Results._ At the end of the test sessions we analyzed the data collected by the observation matrix and questionnaires. As to the former, gathered data were compared with the expected values and discrepancies were analyzed and summarized (Table 7). In particular, we observed that anomalies in the observed attention zone involved a distance of only one level up (e.g., a \"divided attention\" was observed when an \"inattention\" was expected) or one level down (e.g., a \"divided attention\" was observed when a \"focused attention\" was expected).