# User Experience Evaluation – Which Method to Choose?

Virpi Roto, Arnold Vermeeren, Kaisa Väänänen-Vainio-Mattila, Effie Law

## ► To cite this version:

Virpi Roto, Arnold Vermeeren, Kaisa Väänänen-Vainio-Mattila, Effie Law. User Experience Evaluation – Which Method to Choose?. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.714-715, 10.1007/978-3-642-23768-3_129. hal-01596898

HAL Id: hal-01596898

https://inria.hal.science/hal-01596898

Submitted on 28 Sep 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>
# User Experience Evaluation – Which Method to Choose?

Virpi Roto<fnref n="1" />, Arnold Vermeeren<fnref n="2" />, Kaisa Väänänen-Vainio-Mattila<fnref n="3" />, Effie Law<fnref n="4" />

**Abstract.** User experience has many dimensions and therefore, it is tricky to evaluate it. When the goal of user experience evaluation is to investigate how people feel about using an interactive system, the traditional usability methods are hardly applicable. In this tutorial, we introduce a set of 78 user experience evaluation methods that we have been collecting from the user experience community 2008-2010. We give both an overview of the different types of methods and examine a selected set of methods in detail.

**Keywords:** User experience, Evaluation, Assessment, Method.

## 1 Introduction

High quality user experience (UX) has become a central competitive factor of product development in mature consumer markets. Improving UX of systems requires evaluation, but traditional usability testing methods are not adequate for evaluating UX. The evaluation methods for investigating how users *feel* about the tested system are still largely unknown.

Since 2008, we have been collecting a comprehensive set of UX evaluation methods (UXEM) both from academia and industry. We have collected the methods from workshops [2,3,4], a Special Interest Group session [1], online survey, literature, and existing smaller collections of experiential evaluation methods.

Based on a structured description of each method, we have been able to categorize the methods based on various criteria. Figures 1 and 2 show examples of UXEMs categorized by the time span of UX that is investigated and by the product development phase they can be used in.

In this tutorial, we will share our knowledge on UX evaluation methods by both giving an overview of the different types of UXEMs and by examining a selected set of methods to gain practical understanding of the method.

<footnote n="1">¹ University of Helsinki, P.O.Box 68, 00014 University of Helsinki, Finland.
virpi.roto@helsinki.fi</footnote>

<footnote n="2">² Delft University of Technology, Landbergstraat 15, 2628 CE Delft, The Netherlands.
a.p.o.s.vermeeren@tudelft.nl</footnote>

<footnote n="3">³ Tampere University of Technology, Korkeakoulunkatu 6, 33720 Tampere, Finland.
kaisa.vaananen-vainiomattila@tut.fi</footnote>

<footnote n="4">⁴ University of Leicester, LE1 7RH Leicester, UK. elaw@mcs.le.ac.uk</footnote>
<table><caption>Figure 1. Examples of UX evaluation methods for studying different periods of experience</caption><thead><tr><th>Evaluating emotions</th><th>Evaluating an episode</th><th>Evaluating long-term UX</th></tr></thead><tbody><tr><td><strong>Observation</strong><br>Facial, body, vocal expressions (e.g. smile, lean back, sigh)</td><td><strong>Observation</strong><br>Experience think aloud</td><td><strong>Self-reporting</strong><br>Questionnaires, Laddering, iScale, Repertory Grid Technique</td></tr><tr><td><strong>Psychophysiological measurements</strong><br>Muscle, pupil, heart, skin reactions detected with sensors</td><td><strong>Self-Reporting</strong><br>Experience sampling, AttrakDiff, Interviews, Day Reconstruction</td><td></td></tr><tr><td><strong>Self-reporting</strong><br>Verbal: PANAS, AffectGrid<br>Non-verbal: EmotionSlider, EmoCards, PrEmo</td><td></td><td></td></tr></tbody></table>

<table><caption>Figure 2. Examples of UX evaluation methods for different phases of product development</caption><thead><tr><th>Concepting</th><th>Non-functional prototype</th><th>Functional prototype</th></tr></thead><tbody><tr><td><strong>Visual design</strong><br>Emotional expressions, reactions</td><td><strong>Visual design</strong><br>Emotional expressions, reactions</td><td><strong>Lab test</strong><br>Emotional expressions, reactions, AttrakDiff</td></tr><tr><td><strong>Idea description</strong><br>Expert evaluation, Role play: Perspective-Based Inspection</td><td><strong>Interaction</strong><br>Experience think aloud</td><td><strong>Field study</strong><br>Experience sampling, Diary, Day Reconstruction Method</td></tr><tr><td></td><td></td><td><strong>Market feedback</strong><br>Questionnaires, UX Curve / iScale</td></tr></tbody></table>

By the end of this tutorial, participants will know

- the general targets of UX evaluation

- the various kinds of UX evaluation methods available for different purposes

- how to choose the right method for the purpose

- the basics of the selected UX methods of different types

- where to find more information on those methods

## References

1. Obrist, M., Roto, V., Väänänen-Vainio-Mattila, K. (2009). User Experience Evaluation - Do you know which method to use? In Extended Abstracts of CHI'09, pp. 2763-2766.

2. Roto, V., Hoonhout, J. (2009) Experiential Evaluation Methods. Workshop in the 4th International Conference on Designing Pleasurable Products and Interactions, DPPI'09.

3. Roto, V., Väänänen-Vainio-Mattila, K., Law, E., Vermeeren, A. (2009) Workshop on User Experience Evaluation Methods in Product Development (UXEM'09). Proceedings of INTERACT 2009, Part II, LNCS 5727, pp. 981-982, 2009.

4. Väänänen-Vainio-Mattila, K., Roto, V., Hassenzahl, M. (2008). Now Let's Do It in Practice: User Experience Evaluation Methods in Product Development. CHI 2008 Proceedings - Workshops, pp. 3961-3964. April 6th, 2008, Florence, Italy. Workshop site: http://www.cs.tut.fi/ihte/CHI08_workshop