<figure><img src="image_1.png" /></figure>

HAL
open science

# In Two Minds about Usability? Rationality and Intuition in Usability Evaluations

Volker Thoma, Elliott P. White

<figure><img src="image_2.png" /></figure>

## To cite this version:

Volker Thoma, Elliott P. White. In Two Minds about Usability? Rationality and Intuition in Usability Evaluations. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.544-547, 10.1007/978-3-642-23768-3_78. hal-01596982

HAL Id: hal-01596982

https://inria.hal.science/hal-01596982

Submitted on 28 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# In two Minds about Usability? Rationality and Intuition in Usability Evaluations

Volker Thoma¹, Elliott P. White¹

¹University of East London, Romford Road, London E15 4LZ

**Abstract.** Usability ratings of a university website by 60 students were analysed together with participant's self-ratings of their cognitive style. The degree of users' "rational" as well as their "intuitive" style correlated with evaluation scores. In particular, self-reported rational ability was connected with evaluations of Controllability, intuitive ability was related to Helpfulness scores of the interface. Thinking style significantly affects usability ratings (explaining over 9% of the ratings' variation), which has implications for evaluations across user groups.

## 1 Introduction

User centred design relies on a variety of methods and techniques which aim to involve potential users at different stages of product development. A common component of such projects often a summative evaluation that aims to measure "ease of use" [1, 2]. The results of user evaluations affect decision making in IT projects, but the factors influencing the users' judgments are rarely analysed, even if they are subdivided into "user groups" according to socio-economic criteria or persona characteristics [3]. Furthermore, only recently researchers and practitioners have embraced the concept of "affect" or "attractiveness" to account for "hedonic" or "fun" aspects of use [4]. However, modern research on human judgment, decision making, and social psychology has identified a great number of factors that may systematically influence preferences or ratings. For example, Slovic [5] identified the *affect heuristic* - a positive or negative feeling experienced by the evaluator informs their preferences between items or images.

There is growing evidence that human reasoning and decision making is the product of two distinct cognitive systems. Researchers share the labels 'System 1' and 'System 2' for these two systems [6], although they are more commonly referred to as "intuition" and "rationality". System 1 is a fast, effort-free collection of autonomous processes that are susceptible to emotional stimulation. System 2 is characterised by slow, deliberative, rule-based reasoning [6]. Humans seem to vary in their propensity for intuitive versus reflective thinking in their daily judgments. Thus, cognitive style, also called thinking style, refers to a person's preferred way of processing information and the way he or she perceives, conceptualises, and judges information [6]. One of the earliest and most reliable methods to investigate cognitive style is the Rationality and Experientiality Inventory (REI) developed by Epstein [7]. The subscales of this questionnaire correspond to self-rated ability and preference of utilizing System 1 and
System 2 respectively. Rationality and Experientiality have been found to be
orthogonal constructs [7] and are dissimilar to cognitive abilities because they do not
describe peak performance, but they do influence attitudes and social interaction.

The aim of the current study is to test whether individual differences in
“Experientiality” and “Rationality” may influence user evaluations, and whether
aspects of usability ratings depend more on intuition rather than reflective thought.
We used the WAMMI (Website Analysis and Measurement Inventory) [2] (after
considering SUMI, QUIS, SUS [1]) to elicit usability ratings of a website, because of
its widespread use, and because it employs a separate subscale of Attractiveness (A).
Our predictions were that a) WAMMI scores are influenced by individual differences
in thinking as measured by the REI and b) that some WAMMI subscales would
correlate more strongly with rationality than others (A), as people would rely more on
system 1 processes (i.e., automatic, implicit visual evaluations) in this case.

## 2. A web site evaluation and the role of intuition and reflexivity

### Participants, Measures and Procedure

Sixty students (40 females, age range between 18 and 43, mean = 24.45) in a
modern UK university were approached in one of the campus computer centres and
invited to take part in a study to evaluate the university's website.

Once participants agreed they were asked to visit to the university's homepage. They were not required to perform any tasks but simply asked whether they had used the website before. If so, participants completed the two questionnaires after brief inspection of the website. The WAMMI was used to elicit ratings regarding the usability of the website. The WAMMI is a 20 item questionnaire in which participants rate their strength of approval for statements like "The pages on this web site are very attractive" and "This web site has some annoying features", on a 1–5 scale. Data from this scale were scored by the WAMMI's authors using a proprietary algorithm. Results are provided on five sub-scales: Learnability (L), Helpfulness (H), Efficiency (EC), Controllability (C), and Attractiveness (A), and a combined global usability score (G). After that participants filled in the Rational-Experiential Inventory-short form, a 24 item version of the original scale [7]. Participants rate the strength of their agreement with statements like, "I have a logical mind" (rational ability) and "I believe in trusting my hunches" (experiential favourability), on a 1-5 scale. These items combine into four subscales: Rational Ability (RA), Rational Favorability (RF), Experiential Ability (EA), and Experiential Favorability (EF), which can be summed up for overall 'Rationality' (R) and 'Experientiality' (E) scores.

### Results

Pearson correlation coefficients were calculated. Table 1 shows the pattern of
correlations and level of significance. Global WAMMI scores correlated significantly
with RA and Rationality, as well as with Experientiality. The WAMMI subscale for
Controllability and Efficiency correlated with RA and Rationality, whereas
Helpfulness correlated with RA, EA and Experientiality. Learnability correlated with
EF only. Interestingly, against our prediction the subscale of Attractiveness did not correlate with experiential scales, but it did correlate with RA.

<table><caption>**Table 1.** Correlations of WAMMI and REI scales</caption><thead><tr><th rowspan="2"></th><th colspan="5">WAMMI</th><th colspan="5">REI</th></tr><tr><th>A</th><th>C</th><th>E</th><th>H</th><th>L</th><th>G</th><th>RA</th><th>RF</th><th>R</th><th>EA</th><th>EF</th><th>E</th></tr></thead><tbody><tr><th rowspan="6">WAMMI</th><th>Attractiveness</th><td>.59**</td><td>.60**</td><td>.60**</td><td>.54**</td><td>.83**</td><td>.28*</td><td>.11</td><td>.23</td><td>.186</td><td>.05</td><td>.15</td></tr><tr><th>Controllability</th><td></td><td>.67**</td><td>.56**</td><td>.50**</td><td>.84**</td><td>.39**</td><td>.24</td><td>.37*</td><td>.17</td><td>.16</td><td>.20</td></tr><tr><th>Efficiency</th><td></td><td></td><td>.48**</td><td>.39**</td><td>.79**</td><td>.31*</td><td>.23</td><td>.33*</td><td>.06</td><td>.10</td><td>.09</td></tr><tr><th>Helpfulness</th><td></td><td></td><td></td><td>.52**</td><td>.8**</td><td>.31*</td><td>.10</td><td>.25</td><td>.41**</td><td>.21</td><td>.39**</td></tr><tr><th>Learnability</th><td></td><td></td><td></td><td></td><td>.72**</td><td>.08</td><td>.15</td><td>.14</td><td>.09</td><td>.34**</td><td>.25</td></tr><tr><th>Global</th><td></td><td></td><td></td><td></td><td></td><td>.35**</td><td>.21</td><td>.33**</td><td>.23</td><td>.21</td><td>.27*</td></tr></tbody></table>

In order to test whether individual differences in cognitive style can explain variations in how people rate the usability of this interface, we ran simple linear regression analyses with the REI subscales as predictors and the WAMMI scores as criterion. In the first analysis with the global WAMMI score as predictor, the ANOVA was marginally significant, F (4, 55) = 2.51, p = .052, meaning that the REI model predicted the global WAMMI scores to a significant degree (R=.39). However, only RA approached significance as a predictor (Beta = .26, t = 1.850, p = .07) all other predictors were t = < 1.07. Thus, the REI model explained 9% of the variance in WAMMI-G scores. For the analyses with a WAMMI subscore (A, EC, L) as criterion we obtained no significant model, all p > .1. However, for Controllability the REI model significantly predicted the WAMMI scores, F (4, 55) = 2.70, p < .05 (adjusted R²= 0.10), again with RA as the only significant predictor (t = 2.29, p < .05). The model reliably predicted Helpfulness, F (4, 55) = 3.69, p < .05, (adjusted R²= 0.15), this time only with EA explaining a significant part of the variance (t = 2.62, p < .05).

## 3 Conclusions

We asked whether usability measures could be differentially influenced by individual differences in cognitive style. There were two important findings. First, usability scores were higher the more people believed in their thinking ability: both “rational” and “experiential” scores correlated with WAMMI ratings. However, only rational ability self-ratings significantly predicted variations in WAMMI scores in a multiple regression model. It is outside the scope of the study to determine whether this means that people who feel they can rely on their (rational) thinking style in general feel more positive (“in control”) about an interface, or whether this reflects overconfidence in participants about their abilities which may result in a sense of mastery of the interface. Whatever the interpretation of this effect, if confirmed this result would mean that comparing summative usability evaluations across different populations may be affected by differences in thinking style. For example, some Asian populations are found to be less overconfident in their judgments than Westerners and prefer a more global style of information processing [8, 9]. Hence, cross-cultural user research may be susceptible to thinking style.

<footnote>**= significant at the 0.01 level, *=significant at the 0.05 level</footnote>
The second result is more complex: self-belief in rational ability positively affects the experience with an interface, in particular in terms of its controllability. One explanation is that people who score high on RA are able to cope with poor interactive design features and feel less affected by it. For example, they may work out how to navigate even a poorly organised website and attribute this relative ease as inherent to the website - thereby inflating the Controllability scores. On the other hand, Helpfulness (but not Attractiveness) scores correlated highly with a global "experiential" style. The reasons for this are probably complex, but it is possible that the more "intuitive" participants relied successfully on processes which are unconscious and work automatically. Compared to the task of deciding whether a website is easy to control by recalling specific episodes of usage, judgments on Helpfulness may be harder to break down into distinct instances. As the experiential system operates in an automatic and holistic manner, users that score high on experiential ability may have simply a better mental "toolset" for making such a more global judgment, resulting in higher scores. In contrast, users may rely on rational criteria when asked to judge Attractiveness [10].

In conclusion, the current study provides evidence for a role of individual thinking styles in usability evaluations. Further research would need to establish - for example by using "verbal protocols" and different memory tests - whether user ratings can indeed be traced to reflective or intuitive processes.

## References

[1] Holzinger, A.: Usability Engineering for Software Developers. Commun. ACM, vol. 48, 1, 71-74 (2005)

[2] Kirakowski, J., Claridge, N., Whitehand, R.: Human Centered Measures of Success in Web Site Design. In Proceedings of the Fourth Conference on Human Factors & the Web, Basking Ridge, NJ (1998). http://www.ucc.ie/hfrg/questionnaires/wammi/research.html

[3] Thoma, V., Williams, B.: Developing and Validating Personas in e-Commerce: A Heuristic Approach. INTERACT 2009, LNCS, 5727, pp. 524-527 (2009)

[4] Hassenzahl, M.: The interplay of beauty, goodness, and usability in interactive products. Int. J. Hum-Comput. Int., vol.19, 319-349 (2004)

[5] Slovic, P., Finucane, M.L., Peters, E., MacGregor, D.G.: The affect heuristic. In: Heuristics and biases: The psychology of intuitive judgment. Gilovich, T., Griffin, D.W., Kahneman, D. (eds) pp. 397-420. Cambridge University Press (2002)

[6] J. Evans, "In two minds: dual-process accounts of reasoning," Trends Cogn. Sci., vol. 7, Oct. 2003, 454-459.

[7] Epstein, S., Pacini, R., Denes-Raj, V., Heier, H.: Individual differences in intuitive-experiential and analytical-rational thinking styles. J. Pers. Soc. Psychol., vol. 71, 390-405 (1996)

[8] Lee, J.W., Yates, J.F., Shinotsuka, H., Singh, R., Onglatco, M.L.U., Yen, N.S., Gupta, M. Bhatnagar, D.: Cross-national differences in overconfidence. Asian J. Psy., vol. 1, 63-69 (1995)

[9] Zhu, Y., Ronis, D.L., Wang, D.F., Shinotsuka, H., Toda, W.: Probability judgment accuracy: China, Japan, and the United States. Organ. Behav. Hum. Dec., vol. 43, 145-171 (1989)

[10] Leder, H., Belke, B., Oeberst, A., Augustin, D.: A model of aesthetic appreciation and aesthetic judgments. Brit. J. Psychol., vol. 95, 489-508 (2004)