# Usability Evaluation in a Digitally Emerging Country: A Survey Study

Fulvio Lizano, Maria Marta Sandoval, Anders Bruun, Jan Stage

► To cite this version:

Fulvio Lizano, Maria Marta Sandoval, Anders Bruun, Jan Stage. Usability Evaluation in a Digitally Emerging Country: A Survey Study. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.298-305, 10.1007/978-3-642-40498-6\_22 hal-01510556

HAL Id: hal-01510556

https://inria.hal.science/hal-01510556

Submitted on 19 Apr 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Usability Evaluation in a Digitally Emerging Country:
a survey study

Fulvio Lizano¹, Maria Marta Sandoval², Anders Brunn¹ and Jan Stage¹

¹ Aalborg University, Department of Computer Science, Selma Lagerlöfs Vej 300,
Aalborg East, Denmark

² National University, Informatics School, PO-Box 86-3000, Heredia, Costa Rica

{fulvio, bruun, jans}@cs.aau.dk, msandova@una.cr

**ABSTRACT.** Several emerging countries experience increasing software de-
velopment activities. With the purpose of provide useful feedback on possible
courses of action for increasing application of usability evaluation in such coun-
tries, this paper explores the status of usability evaluation in a digitally emerg-
ing country. Our aim is to identifying common characteristics or behavioral
patterns that could be compared with digitally advanced countries. We used an
online survey answered by 26 software development organizations, which gave
a snapshot of the application of usability evaluation in these organizations. We
found many similarities with advanced countries, several completely new ob-
stacles more connected with software development matters and a relatively pos-
itive improvement in the lack of "usability culture". These findings suggest
good conditions to improve conduction of usability evaluations in digitally
emerging countries.

**Keywords:** Usability evaluation, advantages, obstacles, digitally emerging countries.

# 1 INTRODUCTION

Usability evaluation is a relevant and strategic activity in software projects [6]. For the user, a high level of usability in a software system is important [8]. For the user organization, usability is important because it can provide benefits such as increased sales, productivity, lower training costs and reduced technical support for users [2].

Previous studies of the perception of usability evaluation have been focused on ob-
stacles and advantages. Two specific cases have identified obstacles in software or-
ganizations. The first one was a survey in Northern Jutland, Denmark [2] (known in
this paper as the “D-Study”) and the second, which replicated the first, was made in
Southern Italy [1] (known in this paper as the “I-Study”). The D-Study identified
several obstacles to increased application of usability evaluation, e.g. developer mind-
set, resource demands, lack of understanding, customer participation, conducting tests
and test participants [2]. In the case of the I-Study, major obstacles identified were
resource demands, no suitable methods, developers' mindset and user availability. In addition, this study identified advantages of usability evaluation such as quality improvement, the users' satisfaction, resource saving and competitiveness [1].

These facts, together with other problems found in digitally advanced countries [3, 4, 5, 7, 9, 10, 11], can be visualized, as a whole, in a view formed by several dimensions of types of actors (e.g. users or clients, software developers, and organizations) plus types of facts (e.g. facts related to understanding, advantages and obstacles of usability evaluation) This is a way to put into context the real implications for usability evaluation in digitally advanced countries. In Table 1 we present the distribution of these findings into the cited dimensions (some references are related with different dimensions at once).

<table><caption>**Table 1.** Main findings related to usability evaluation.</caption><thead><tr><th rowspan="2">Types of actors</th><th colspan="3">Types of facts (related to...)</th></tr><tr><th>Understanding of usability evaluation</th><th>Advantages of usability evaluation</th><th>Obstacles of usability evaluation</th></tr></thead><tbody><tr><td>Users / Customers</td><td>- User involvement[1,2]<br>- Customer involvement[2]</td><td>- User satisfaction[1]</td><td>- Test participants[2]<br>- Customer participation[2]<br>- User availability[1]</td></tr><tr><td>Software developers</td><td></td><td></td><td>- Developer mindset[2]<br>- Developer mindset[1]<br>- Lack of trained eng.usab/HCI[10]<br>- Personal developers tools[11]</td></tr><tr><td>Organizations</td><td>- Functionality[2]<br>- Problem/task solving[2]<br>- Possibility test[2]<br>- Usability evaluation[1]<br>- Usability definition[1]<br>- Accessibility test[1]</td><td>- Quality.improvement[1]<br>- Competitiveness[1]<br>- Resource saving[1]</td><td>- Lack of understanding[2, 10, 11]<br>- Resource demands[2]<br>- Conducting tests[2]<br>- No suitable methods[1]<br>- Resource demands[1, 10]<br>- Resistance to UCD-Usability[10]<br>- Lack of comms. of impact[10]<br>- Lack coupling UCD &amp; S-Dev cycle[11]<br>- Gap SD &amp; usability[11]<br>- Edu. lack coupling (SD&usab.) [11]<br>- Lack of respect and support[5]<br>- Limited description HCI in SE[4]<br>- Strong differences HCI &amp; SE[7]<br>- Lack of "usability culture" [3, 9]</td></tr></tbody></table>

In this paper, we present the results of a study that explored the application of usability evaluation in software development organizations in a digitally emerging country. The aim of our study was to explore the understanding of the usability evaluation
concept, their advantages and obstacles. Our interest was identifying similarities,
differences and patterns that could enhance the application of usability evaluation in
other digitally emerging countries. This explains why we will present our results and
compare them to other studies, especially with the D-Study and the I-Study.

2 METHOD

2.1 Settings

We conducted a survey at software development companies in Costa Rica. According
The Global Information Technology Report 2012 (World Economic Forum -
www.weforum.org/gitr), Costa Rica is a digitally emerging country ranked in the 58
position of the Networked Readiness Index (NRI). Costa Rica has a NRI of 4 in a 1-
to-7 scale.

## 2.2 Participants and procedure

The study has involved companies that could potentially conduct usability evaluations
and in addition met the specific criteria, e.g. located in a specific geographical area
(Costa Rica), that develop software or hardware with graphical user interfaces, that
develop software for customers or for internal use and that employ more than a single
person. Initial set of participants was made using the list of organizations affiliated to
the Chamber of Information and Communication Technology - CAMTIC, by its
Spanish acronym (www.camtic.org) (148 organizations). This organization is open to
any IT organization of Costa Rica and was founded in 1998. Because CAMTIC is
open to a broad range of IT companies, we decided to filter the original list obtaining
a final list of 35 organizations. Our survey was completed by 26 organizations
(74%). The average number of years of operation of the organizations participating in
the study was 11. The average of age of the persons, who filled the questionnaire,
was 39. 15% of them were females. All these companies were located at the central
valley of Costa Rica (the most developed zone of Costa Rica). The organizations had
this distribution on number of employees: 58% (1-10), 19% (11-50) and 23% (51-
250). In order to find the person most appropriate to participate in the study, we con-
tacted every company personally by phone in order to enquire who could provide an
opinion that could reflect the position of the company in the survey. These persons
received an electronic token to access the online survey which was active for 4 weeks.

## 2.3 Data collection and analysis

The questionnaire used in the study contained several parts. The main parts were:
demographic and general information, products/services provided by the organization,
methodology used to develop software applications, understanding of the usability
evaluation concept, and obstacles and advantages of the application of usability eval-
uation. We used a combination of open-closed questions was used. The aim of open
questions was to permit participants to express themselves in their own words. Closed questions were used in order to allow them to reconfirm data previously provided. We used different analytical approaches to analyze the data generated in the closed and open questions. A quantitative analysis was used on the closed questions, and grounded theory by Strauss and Corbin [12] was used for the analysis of the open questions.

## 3 RESULTS

We were interested in exploring the understanding of the concept of usability evaluation in the software development organizations. Our results allowed us to identify several categories of understanding that the organization have about this concept. In order to verifying what the organizations had understood by usability evaluation, after provide their definition of usability evaluation, we showed them a definition of usability evaluation based on the ISO-9241 standard. Next, we asked them if they made usability evaluations in their companies in accordance with this ISO definition and the strategy followed by them to do it. The participants basically reported two categories of strategy: internal or external conduction of usability evaluation. However, a relevant number of participants reported do not conduct usability evaluations at all. Finally, some participants did not provide a response for this enquiry. In Table 2, we present these results.

<table><caption>**Table 2.** Distribution of the strategy used to conduct usability evaluation related to the understanding of the concept of usability evaluation</caption><thead><tr><th rowspan="2">Strategy used to conduct usability evaluation</th><th colspan="5">Category of understanding (In terms of...)</th></tr><tr><th>Usability concept</th><th>Usability evaluation concept</th><th>Another kind of testing</th><th>No response</th><th>Percent</th></tr></thead><tbody><tr><td>Internally</td><td>4</td><td>8</td><td>4</td><td></td><td>62%</td></tr><tr><td>Externally</td><td></td><td>1</td><td>2</td><td></td><td>12%</td></tr><tr><td>No Usab.evalution</td><td>1</td><td>2</td><td>2</td><td></td><td>19%</td></tr><tr><td>No response</td><td></td><td></td><td></td><td>2</td><td>8%</td></tr><tr><td>Percent</td><td>19%</td><td>42%</td><td>31%</td><td>8%</td><td></td></tr></tbody></table>

On the other hand, following the methodological approach established in our method, we identified the main advantages and obstacles for applying usability evaluation. First, using an open question we identified advantages and obstacles of the application of usability evaluation. Some participants offered more than one advantage/obstacle and others did not respond. The results were grouped in different categories of advantages/obstacles. In the case of the advantages, main results were product quality (35%), user acceptance (32%) and no advantages (32%). The main obstacles detected were users (22%), software design (19%), software development method (15%), costs (11%), software developers (4%) and no obstacles (30%). The organizations which reported do not conduct usability evaluation in their process, were the ones which had not found advantages or obstacles.
In order to complement the previous results, we presented to the participants with a list of common advantages and obstacles of usability evaluation. The participants could select more than one option. Results are showed in Table 3.

<table><caption>**Table 3.** Advantages and obstacles provided through closed questions.</caption><thead><tr><th>Advantages.</th><th>#</th><th>P</th><th>Obstacles</th><th>#</th><th>P</th></tr></thead><tbody><tr><td>User satisfaction</td><td>19</td><td>39%</td><td>Recruitment of test participants</td><td>9</td><td>20%</td></tr><tr><td>Quality improvement</td><td>18</td><td>37%</td><td>Conduct test / no method</td><td>6</td><td>13%</td></tr><tr><td>Competitiveness</td><td>5</td><td>10%</td><td>Developer mindset</td><td>17</td><td>38%</td></tr><tr><td>Resource saving</td><td>6</td><td>12%</td><td>Resources demands</td><td>8</td><td>18%</td></tr><tr><td>Other</td><td>1</td><td>2%</td><td>Other</td><td>5</td><td>11%</td></tr><tr><td>Total</td><td>49</td><td></td><td></td><td>45</td><td></td></tr></tbody></table>

These results represent some of the main facts related to usability evaluation in digitally emerging countries. To facilitate a comparison of these facts with the ones presented in digitally advanced countries, we present them in a similar way as we did at table 1. Thus, as it is possible to see in Table 4, users, customers and software developers are not presented into the understanding of usability evaluation, by the software development organizations at the digitally emerging countries. In addition, our results suggest a weak visualization of advantages and obstacles in the same context. Similar to the digitally advanced countries, at digitally emerging countries it is possible to find more facts related to the understanding, advantages and obstacles of usability evaluation, in the context of the organizations.

<table><caption>**Table 4.** Summary of results.</caption><thead><tr><th rowspan="2">Types of actors</th><th colspan="3">Types de facts (related to...)</th></tr><tr><th>Understanding of usability evaluation</th><th>Advantages of usability evaluation</th><th>Obstacles of usability evaluation</th></tr></thead><tbody><tr><td>Users / Customers</td><td></td><td>- User acceptance<br>- User satisfaction</td><td>- User<br>- Test participants</td></tr><tr><td>Soft.Dev.</td><td></td><td></td><td>- Software developers mindset</td></tr><tr><td>Organizations</td><td>- Usability concept<br>- Usability evaluation concept<br>- Other test</td><td>- Product quality<br>- Quality improvement<br>- Competitiveness<br>- Resource saving</td><td>- Software design<br>- Software development method<br>- Costs<br>- Usability evaluation conduct.<br>- No usability evaluation method</td></tr></tbody></table>

# 4 DISCUSSION

Our results suggest a relatively good understanding of the understanding of the usability evaluation concept, including some similarities to previous studies, e.g. in some
aspects of the notion of usability [1], and in usability matters as a whole, specifically
in some responses related to user involvement [2]. The good understanding about the
definition of usability evaluation was obtained from organizations which conduct
usability evaluation internally. This practical experience supported this better under-
standing. This is even more evident when we analyze the reasons given by those
organizations that do not conduct usability evaluations (19%). Although the distribu-
tion of their understanding is uniform in the different categories of understanding,
these participants provided opinions that are clear signals of a misunderstanding about
usability evaluation, e.g. "in open source software projects you do not need usability
evaluations" or "some projects do not require usability evaluations" and "a software
project only needs functional tests". Here, we can see and excellent example of what
the lack of "usability culture" is [3, 9].

In the case of advantages, our results are fully in agreement with the I-Study [1]. However, in the case of the obstacles, our study found very interesting results. The 'user' was identified as one of the most relevant obstacle. This was emphasized by participants who conduct usability evaluations internally, which makes this result conclusive. Both the D-Study and I-Study also identified this obstacle but with a lower level of importance [1, 2]. This finding allows us to notice that in a digitally emerging country, participation of users in usability evaluation seems to be particularly challenging. Consistently with the D-Study and the I-Study, our study confirmed obstacles related to resource demands and software developers' mindset [1, 2]. It is interesting to notice that the level of relevance given to this last obstacle has changed across the D-Study, the I-Study and our study. This obstacle was very important in the D-Study. In the I-Study, its relevance was lower. Finally, in our study this obstacle is the last one mentioned by the participants. This change could, initially, reinforce our perception of a positive change in the lack of a "usability culture". However, other results obtained in the closed questions seem to offers contradictory results. In this case, the most important obstacle selected by participants was related to the software developers' mindset problem. These different levels of relevance are not necessarily a contradiction. Actually, the fact that this obstacle was cited twice in our study allow us to conclude that this matter continues been one of the most recognized obstacles against increased use of usability evaluation. The second obstacle identified in this part of our study is related to resource demands, which is not surprising.

In addition, there are new obstacles that were identified in our study. First obstacle is related to problems in the design process of the software, which subsequently could hinder conduction of usability evaluations. Second, a new obstacle was identified in some problems related to the software development method. This obstacle was identified by the I-Study in 2011 but not by the D-Study in 2008. Here it is possible to observe a change of tendency in the lack of "usability culture", into those organizations that have practical experience; an alternative view about the new obstacles, which is more connected to the software development process, seems to emerge to reduce some problems such as the confusion, the lack of coupling and some gaps between SE and HCI [11].

In digitally advanced countries the main facts related to understanding, advantages and obstacles to conduct usability evaluation are more connected with methodology
and the organization (see Table 1). Users, customers and software developers have a lower visualization. More remarkable is the fact that software developers are not presented at all in such dimensions. Only in the case of the obstacles, it is possible to find more facts related to users, customers and developers.

In digitally emerging countries, this situation seems no to be better (see Table 4). Into the understanding of usability evaluation, the users, the clients and the developers are excluded at all. Only a limited number of advantages were noticed for users and clients, none related to developers.

We think that our study provide interesting results that can be extended to other similar contexts. The digitalization level and other human and economical indicators are pretty similar to others countries in the same region, e.g. Ecuador, Trinidad & Tobago, Panama, Peru, Brazil and The Bolivarian Republic of Venezuela. The average on Networked Readiness Index (NRI) in these countries (including Costa Rica) is 3,71 (SD=0,32), the mean value for the GNI per capita in PPP terms (constant 2005 international $) is 12,051 (SD= 4,740), the average of expected years of schooling is 9,05 (SD=2,21), the mean value for expectancy of life (years) is 74,81 (SD= 2,79). Main differences are related to population and territorial extension. Considering all these facts, the context studied in our research can be considered a good referent about how usability evaluation is conducted in other digitally emerging countries.

# 5 Conclusion and future works

In this research we have explored the application of usability evaluation in software development organizations in the digitally emerging countries. To accomplish this, we conducted a questionnaire survey with 26 participating software development organizations. As part of our research, our findings were contrasted with results from similar studies in digitally advanced countries. The aim of our study was to obtain valuable feedback that could orientate future enhancement actions of application of usability evaluation in digitally emerging countries.

Our study found a relatively acceptable conduction of usability evaluation in digitally emerging countries, embodied by a fairly clear understanding about the meaning of usability evaluation and similar advantages and obstacles to the found in other digitally advanced countries. In addition, our research has identified new obstacles such as the users' behavior and problems related to the design of the software. These new obstacles can offer to HCI theory a complementary perspective on usability evaluation. These new findings seem to imply a decreasing tendency in the lack of "usability culture". However, our results do not permit strong conclusions about this matter as it was not a focus of our study.

However, any improvement of conduction of usability evaluation at the context studied must necessarily go through an empowerment process of users, clients and software developers, as main actors in such processes. In the case of users and clients, reasons to do that are more than evident; in some sense, these actors are a main cornerstone of theory of HCI. For developers, this strategy should help continuing improvement of some well studied problems, e.g. confusion, the lack of coupling and the
gaps between software engineering and HCI. Future works could focus on exploring specific forms to enhance and increase the use of usability evaluations in software development organizations located in digitally emerging countries.

## Acknowledgments

The research behind this paper was partly financed by Universidad Nacional (Costa Rica), MICIT (Costa Rica), CONICIT (Costa Rica), and the Danish Research Councils (grant number 09-065143).

## References

1. Ardito, C., Buono, P., Caivano, D., Costabile, M.F., Lanzilotti, R., Bruun, A., Stage, J.: Usability Evaluation: a survey of software development organizations. In Proceedings of 33 International Conference on Software Engineering & Knowledge Engineering. Miami, FL, USA (2011)

2. Bak, J.O., Nguten, K., Risgaard, P., Stage, J.: Obstacles to Usability Evaluation in Practice: A Survey of Software Development Organizations. In Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, pp.23-32. ACM, New York, NY, USA (2008)

3. BugHuntress QA Lab: Mobile usability testing Problems and solutions. Research Report, Quality Assurance: Management & Technologies (QAMT), Ukraine. (2007)

4. Ferre, X., Juristo, N., Moreno, A.M.: Obstacles for the Integration of HCI Practices into Software Engineering Development Processes. C. Ghaoui & Idea Group Reference (Eds.). Encyclopedia of Human Computer Interaction, pp. 422-42 (2006).

5. Gulliksen, J., Boivie, I., Persson, J., Hektor, A., Herulf, L.: Making a difference: a survey of the usability profession in Sweden. In Proceedings of the third Nordic conference on Human-computer interaction (NordiCHI '04), pp 207-215, ACM, New York, USA (2004).

6. IEEE Computer Society: SWEBOK Guide to the Software Engineering Body of Knowledge, http://www.computer.org/portal/web/swebok (2004)

7. Juristo, N., Ferre, X.: How to integrate usability into the software development process. In Proceedings of the 28th international conference on Software engineering (ICSE '06), pp 1079-1080, ACM, New York, NY, USA (2006)

8. Lindgaard, G., Chattratichart, J.: Usability Testing: What Have We Overlooked?. In CHI 2007 Proceedings, Usability Evaluation. San Jose, CA, USA. (2007)

9. Muzaffar, A., Azam, F., Anwar, H., Khan, A.: Usability Aspects in Pervasive Computing: Needs and Challenges. International Journal of Computer Applications, 32(10):18-24, October 2011. Published by Foundation of Computer Science, New York, USA. (2011)

10. Rosenbaum, S., Rohn, J.A., Humburg, J.: A toolkit for strategic usability: results from workshops, panels, and surveys. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '00), pp 337-344, ACM, New York, USA, (2000)

11. Seffah, A., Metzker, E.: The obstacles and myths of usability and software engineering. Commun. ACM, 47, pp 71-76, December (2004).

12. Strauss, A., Corbin, J.: Basics of qualitative research. Techniques and Procedures for Developing Grounded Theory, 2.edition. SAGE Publications, (1998)