# Visualizations of User's Paths to Discover Usability Problems

Paolo Buono, Giuseppe Desolda, Rosa Lanzilotti, Maria Francesca Costabile, Antonio Piccinno

► To cite this version:

Paolo Buono, Giuseppe Desolda, Rosa Lanzilotti, Maria Francesca Costabile, Antonio Piccinno. Visualizations of User's Paths to Discover Usability Problems. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.689-692, 10.1007/978-3-030-29390-1_64. hal-02878632

HAL Id: hal-02878632

https://inria.hal.science/hal-02878632

Submitted on 23 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Visualizations of user's paths to discover usability problems

Paolo Buono, Giuseppe Desolda, Rosa Lanzilotti,
Maria Francesca Costabile, Antonio Piccinno

¹ University of Bari Aldo Moro, Computer Science Department
{name.surname}@uniba.it

**Abstract.** This paper reports on an on-going work that investigates the use of visualization techniques to help evaluators discover usability problems by visualizing data collected during usability tests of web sites. Two visualization techniques are described and some results of the evaluation study that compared the two techniques are provided.

**Keywords:** Usability test; Visualization techniques; Usability problems discovery; Web site evaluation.

# 1 Introduction

Usability testing is performed by HCI experts to identify usability problems of interactive software systems. During a test, data like audio/video, PC logs, questionnaire results are gathered and analyzed to detect usability problems. This technique stands out positively from others (e.g. inspection techniques) since users are involved, critical tasks are executed, and a lot of qualitative and quantitative data can be collected and analyzed. It is considered expensive, especially due to the time needed to analyze data.

Some researchers propose the use of visualization techniques to help, in different ways, the evaluator discovering usability problems. For example, WebQuilt is a usability logging and visualization tool that presents logged data in an interactive graph, whose nodes are web pages visited by the users [1]. Each node details usability issues that occurred at that page. In [2], the user's actions during a task are shown on a timeline, and the optimal actions to perform a task are also represented; this should help the evaluator to detect deviations of the performed actions from the correct ones, speeding up the time to investigate something that occurred at a certain time.

In our research, we are primarily working on usability evaluation of web sites and we aim at supporting the evaluator on easily identifying the web pages presenting usability issues. Thus, we propose two techniques that visualize a graph whose nodes are web pages. The presented graphs also show the optimal path to perform a task, since the visual comparison with the optimal path provides useful indications of something that the user could not correctly perform (this was inspired by [2]). In the following, we
describe the proposed interactive visualizations, and briefly report on an evaluation
study that compared the two techniques.

## 2 Two Visualization Techniques to Identify Usability Problems

In usability tests, the identification of sources of usability problems often needs the triangulation of different quantitative and qualitative methods. Quantitative data, like average task time and task success rate, provide indications on tasks that caused evident problems but not where the problem occurred. To get this information, evaluators should look at the videos, paying attention to which pages led participants to fail the tasks or too much longer routes to complete the task.

To support evaluators we propose two interactive visualization techniques that de-
pict, for each task, the routes followed by participants, as well as the ideal path that
participants would follow to successfully complete a task. The problematic web pages
are the ones that mostly confused the participants, leading them to follow wrong paths.

The first visualization is based on the Arc Diagram [3], which places vertices along a horizontal line and arcs represent connections between nodes. Fig. 1a, shows the adaptation of Arc Diagram to support the identification of web pages that caused problems during the execution of a task by all participants to the test. Nodes represent web pages visited by participants, their size is directly proportional to the number of visits. Blue node refers to the task starting page, red if participants wrongly terminated the task at that page, green node refers to the task ending page, black for the other pages. Arc thickness is proportional to the frequency of movement between connected pages. Green arc is the ideal path, otherwise it is light grey. Violet arcs represented below the node line indicate backward paths. The graph in Fig. 1a shows that 13 users correctly moved from the task starting page to the next one (the green arc with ‘13’ label); then only 9 users correctly moved to the next page, and eventually all 15 users moved toward the page that completed the task. Only 1 user wrongly terminated the task, as shown by the red node. One of the most problematic web pages is the one where the arc with 13 users ends, since only 9 users were able to correctly move from this page to the next correct page.

The second visualization is called Page Tree because it shows a tree-like structure that better recalls the ‘hierarchical’ organization of a Website (see Fig. 1b). Nodes and arcs (branches) have the same meaning as other visualization, both in term of colors and dimensions/thickness. In this visualization, the backward paths are not explicitly shown. Here the 47% of participants correctly moved from the task starting page toward the final task page (percentage or absolute values can be switched), while the other participants followed wrong paths (in grey), sometimes wrongly terminating the task (to the red nodes) and sometimes correctly finishing the task (the other green nodes). Lack of space prevents us from detailing the motivation of the choice of these visualizations and reporting all their details and functionalities.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Visualizations of participants paths inspired by the Arc Diagram (a) and Page Tree (b)</figcaption></figure>

## 3 Evaluation of the proposed visualizations

The design and development of the two visualizations followed the User-Centered Design methodology. Initial ideas were sketched and discussed among designers and evaluation experts. Prototypes of increasing complexity (starting from paper-based ones) were tested with few evaluators using the thinking aloud protocol. They allowed to detect various problems that were fixed in the next prototype. Fig. 1a and 1b refer to the current interactive prototypes, implemented using D3 JavaScript library. These prototypes were used in an experimental study to understand to what extent the visualizations support HCI experts in identifying problematic Webpages. We recruited 15 participants (4 females, 11 males) among the students of the Computer Science and Digital Communication curriculum at our university, who completed a course on Human-Computer Interaction and had some basic experience about performing a usability test. Their average age was 26 years (min= 23, max=29). The study was organized according to a within-subject design, with visualization as an independent variable and as within-subject factors the two visualizations. Each visualization showed data gathered in a study conducted with 15 employees of an Italian Administration, who used a Web platform for onsite and remote usability tests [4, 5]. With each system, the participants performed the following tasks: 1. Identify the page that caused most problems; 2. Identify the path(s) that led to the task's failure; 3. Identify the path that led to the task's success following the ideal path; 4. Identify the paths that led to the task's success following alternative paths; 5. Identify backward paths, if possible.

Visualizations and order of tasks were balanced according to a Latin-square design. At the end of all tasks with each visualization, participants filled the UES-short form questionnaire.
# 4 Results and Conclusion

Visualizations have been analyzed and compared according to their effectiveness, efficiency and satisfaction in discovery usability problems. The main findings are mentioned here.

Efficiency was evaluated considering the time that participants needed to complete the assigned tasks. Participants were slower when using Arc Diagram (x=34.93, SD = 5.77) than Page Tree (X=27.76, SD = 5.13). Paired sample t-test showed that this difference is statistically significant (t(14) = -5.287, p = .000).

Effectiveness was analyzed calculating the task success rate. Page Tree had a lower rate (x = 86,56%) than Arc Diagram (X= 93,00%). Even in this case, paired sample t-test revealed a significant difference (t(14) = -2.512, p = .025).

Satisfaction was analyzed calculating the UES score of the two visualizations (Arc Diagram = 4.61/5; Page Tree = 4,78/5). However, paired sample t-test did not reveal any significant difference (t(14) = -1.878, p = .081).

In conclusion, this on-going work proposes two interactive visualization techniques to support usability evaluators in identifying web pages that cause usability problems, speeding up data analysis and encouraging the adoption of usability test in software development lifecycle. Results have shown that both visualizations are adequate to identify web pages that cause usability errors. Page Tree resulted better in term of time to find errors and success rate, while no differences emerged in user engagement. Studies are in progress to investigate other visualization techniques to focus on other aspects of the user flow during task execution or to visualize user activities [6].

# References

1. Waterson S. J., Hong J. I., Sohn T., Landay J. A., Heer J., and Matthews T. 2002. What did they do? understanding clickstreams with the WebQuilt visualization system. In *Proc. of Conference on Advanced Visual Interfaces (AVI '02)*. ACM, 1556276, 94-102.

2. Paternò F., Schiavone A. G., and Pitardi P. 2016. Timelines for Mobile Web Usability Evaluation. In *Proc. of International Conference on Advanced Visual Interfaces (AVI '14)*. ACM, New York, NY, USA, 88-91.

3. Wattenberg M. 2002. Arc diagrams: visualizing structure in strings. In *Proc. of IEEE Symposium on Information Visualization (INFOVIS 2002)*, 110-6.

4. Federici S., Mele M. L., Lanzilotti R., Desolda G., Bracalenti M., Gaudino G., Cocco A., and Amendola M. 2018. UTASSISTANT: A New Semi-Automatic Usability Evaluation Tool for Italian Public Administrations. In *Proc. of International Conference on Advanced Visual Interfaces - ECONA Workshop (AVI '18)*, 1-3.

5. Federici S., Mele M. L., Lanzilotti R., Desolda G., Bracalenti M., Meloni F., Gaudino G., Cocco A., and Amendola M. 2018. UX Evaluation Design of UTAssistant: A New Usability Testing Support Tool for Italian Public Administrations. In: Kurosu M. (Eds), *Human-Computer Interaction. Theories, Methods, and Human Issues - HCI 2018*. Lecture Notes in Computer Science. Springer International Publishing, 55-67.

6. Buono P., Costabile M. F., and Lanzilotti R. 2014. A circular visualization of people's activities in distributed teams. *Journal of Visual Languages and Computing*, 25, 6 (2014), 903-11.