# Design and Applications of GLANCE: GLanceable Alarm Notification for a User Centered Experience
## 3 Results of the 1 st Macro-Phase: The GLANCE Project
### Second Iteration: Dealing with Multiple Alarms
#### 3.2.2 Evaluation and Learning  
The Expert-Based Evaluation  
The evaluation meeting was organized and conducted similarly to the evaluation session performed at the end of the 1 st iteration, with the goal of evaluating _glanceability_, _readability_, and _learnability_ of the notifier in the new context of the added overview capability. The meeting included a presentation of the enriched model, a cognitive walkthrough session with domain experts, an informal think aloud testing with experienced users,  
Figure 6: Redefinition of the display format in the case of multiple notifications: (**a**) Multiple faults in the same NE (Case 1) and (**b**) sequence of multiple faults in two NEs (Case 2). and a final discussion with both domain experts and experienced users. Overall it lasted two and a half hours. The evaluation group included one facilitator, one product expert, two Technolabs people with the twofold role of evaluators and domain experts (who also participated in the previous evaluation session), and two Technolabs people with the role of experienced users (who did not participate in the previous evaluation session). Again, the session was conducted in a real working environment and the examined notification sequence was built from a real alarm log generated by the optical fiber network under consideration in the problem-solving cycle. In particular, the sequence was set up by collecting a number of log portions selected so to include progressively different combinations of intra-NE, inter-NE, and hybrid case scenarios, with notifications presented in an asynchronous way. The same sequence was used for the cognitive walkthrough session with domain experts and the think aloud testing with experienced users. In both cases, the facilitator walked through the example notification sequence to evaluate whether real users would achieve the desired comprehension effect and would associate the correct reaction. In particular, in the think aloud testing users were asked to comment on how they interpreted the visualized information so as to evaluate the learnability of the GLANCE display and its readability against a benchmark situation, in particular in the tricky case of hybrid situations (Case 3), with the main attention on the comprehension of the network status based on the overview provided by the synthetic component. Experienced users involved in the think aloud testing did not participate in the walkthrough session, so to obtain some first rough indications on the learnability of the system: The test was indeed promising, with users grasping correctly the network status from the status of the synthetic component after few steps of the sequence, necessary to understand that the display was cycling over alarmed network elements. Beside the legibility of the synthetic component, the other crucial aspect to be evaluated and discussed was the necessary visual representation power trade-off (dictated by space constraints) between overview information and individual alarm information, and in particular, the choice of reserving the detailed component to one single alarm (the one with highest severity) for each alarmed network element. All domain experts and users judged the choice appropriate with reference to the main goal of a notification in this context, i.e., _pushing operators to take charge of incoming alarms starting from the most severe ones_, as soon as the most sever alarm in a NE is acknowledged by an operator, it will disappear from the GLANCE display while the next in the priority list of its NE will appear, making the notifier behave like an intervention task dispatcher. It was observed that in an operative release of the notifier, some kind of interactivity would be desirable to allow operators to manually access detailed information on less sever alarms on the same NE on-demand if judged necessary (e.g., the overview might reveal that less severe alarms of a given NE are more severe than displayed alarms of other NEs and operators might wish to inspect all alarms of that NE).