# Factors Affecting the Choice of Usability Evaluation Methods for Interactive Adaptive Systems

Amira Dhouib, Ahlem Assila, Abdelwaheb Trabelsi, Christophe Kolski, Mahmoud Neji

► To cite this version:

Amira Dhouib, Ahlem Assila, Abdelwaheb Trabelsi, Christophe Kolski, Mahmoud Neji. Factors Affecting the Choice of Usability Evaluation Methods for Interactive Adaptive Systems. 7th International Conference on Human-Centred Software Engineering (HCSE), Sep 2018, Sophia Antipolis, France. pp.270-282, 10.1007/978-3-030-05909-5_16. hal-02270701

HAL Id: hal-02270701

https://inria.hal.science/hal-02270701

Submitted on 26 Aug 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Factors Affecting the Choice of Usability Evaluation Methods for Interactive Adaptive Systems

Amira Dhouib<fnref n="1" />, Ahlem Assila², Abdelwaheb Trabelsi³, Christophe Kolski⁴, and Mahmoud Neji<fnref n="1" />

¹ Miracl Laboratory, Faculty of Economics and Management Sciences University of Sfax, B.P. 1088, Sfax 3000 Tunisia {amira.dhouib, mahmoud.neji}@fsegs.rnu.tn

² CESI, LINEACT, 7 Bis Avenue Robert Schuman, Reims, France aassila@cesi.fr

³ Saudi Electronic University. Saudi Arabia, Dammam atrabelsi@seu.edu.sa

⁴ LAMIH-UMR CNRS 8201, Université Polytechnique Hauts-de-France, Valenciennes, France Christophe.Kolski@uphf.fr

**Abstract.** Choosing the appropriate usability evaluation methods is a key part of the usability evaluation process of interactive adaptive systems. This step needs the consideration of different factors, leading to a multi-criteria decision analysis problem. In this paper, we present a review of the main factors reported in the literature which can affect the selection of usability evaluation methods for interactive adaptive systems. Three of the most commonly used usability evaluation methods are selected and classified according to these factors. The results of this research are used by applying a decision aid method in order to guide the choice of suitable usability evaluation methods for a given adaptive system in the field of tourism.

**Keywords:** Interactive adaptive system, Usability evaluation method, Decision process.

## 1 Introduction

As for all interactive systems, usability plays an important role in the success of Inter-active Adaptive Systems (IAS)<fnref n="1" /> [1]. The usability evaluation of IAS represents an essential part of their development process. It may be conducted through the use of suitable Usability Evaluation Methods (UEMs). In the IAS literature, several UEMs are available [3,4]. These methods aim mainly to detect the usability issues. Given this variety, non-specialists and even specialists can encounter difficulties in selecting

<footnote n="1">¹ According to Jameson and Gajos [2], an interactive adaptive system represents an "Interactive system that adapts its behavior to individual users on the basis of processes of user model acquisition and application that involve some form of learning, inference, or decision making".</footnote>
the most appropriate UEM(s) in particular settings [4,5]. Choosing appropriate usability evaluation method(s) is a crucial task of the IAS evaluation process [6]. This task depends usually on different factors, such as number of stakeholders, available time, etc. [4]. Applying a Multi-Criteria Decision Analysis (MCDA) method for the suitable UEMs choice is one strategy to deal with multiple and conflicting factors. One of the main steps of the decision analysis process is to define the factors that impact the choice of alternatives. In this research, we provide a general overview of the factors affecting the choice of alternative UEMs. We also classify three common usability evaluation methods for IAS according to the considered factors. These include heuristic evaluation, cognitive walkthrough, and usability test. Lastly, we use the finding of this analysis to guide the choice of suitable methods for the usability evaluation of a target adaptive system as a whole using ELECTRE I (Elimination and Choice Translating Reality) method.

It is common to identify in the IAS literature two kinds of evaluation. The first one is named *layered evaluation*, which aims to separate the adaptation process into its layers and to assess each one individually where necessary and feasible [7]. The second one is *evaluation as a whole* (or traditional evaluation), which considers the adaptation as one block. Various UEMs can be applied in conjunction with the layered evaluation as well as the evaluation as a whole. In the IAS field, a limited number of works have been focused on the guidance of the choice of appropriate UEMs. For instance, Paramythis et al. [7] proposed a layered evaluation framework that breaks the adaptation process into five separate layers. They provided a comprehensive overview of the appropriate evaluation methods and attributes to be applied in individual layers and evaluation as a whole. Regarding the use of MCDA, it has been observed that very few studies exist, apart from the ones in our previous works [8,9]. These studies focus on the choice of suitable evaluation methods for the layered evaluation given particular evaluation settings. While assessing adaptation layers individually allows one to answer questions which are not possible to approach in a "monolithic" entity, there are some assessment questions that require treating the adaptation process as a whole. One example of the questions that can be examined when conducting the evaluation as a whole is "does the adaptive system achieve its goals?" [7]. To the best of our knowledge, there are no attempts to date that provide a review of the common factors affecting the selection of UEMs and that explore the most potential benefits of MCDA to identify appropriate methods for the usability evaluation as a whole of IAS. For instance, the use of MCDA allows the consideration of a variety of criteria that are important for the decision analysis by considering both quantitative and qualitative aspects.

The present paper is organized as follows. First, we briefly outline the usability evaluation methods for IAS, focusing on three common ones (Section 2). Second, we present the proposed decision process for choosing the appropriate usability evaluation methods for IAS (Section 3). Then, the study investigates the main factors that can affect the choice of usability evaluation methods for IAS and analyzes the considered UEMs in relation to these factors (Section 4). An application of an MCDA method is provided in Section 5. The aim is to guide the selection of the most suitable
methods for the usability evaluation of a given adaptive system as a whole. Lastly, we
conclude the paper with a summary and some future directions (Section 6).

## 2 Focus on Three Usability Evaluation Methods for Interactive Adaptive Systems

### 2.1 Heuristic Evaluation

It describes a method in which expert evaluators examine a user interface in order to discover the usability problems [10]. In the IAS field, expert evaluators need to have expertise in heuristic evaluation and they are required to understand the meaning of the particular heuristics applied for adaptive systems [7].

### 2.2 Usability Test

The main purpose of this method is to give a group of real users well-defined tasks to perform and to ask them to record what happens [11]. Certain observational methods can be applied in conjunction with this method such as co-discovery, retrospective testing, etc. In the IAS field, certain modifications to observational methods are required. For instance, in contrast to interactive (non-adaptive) systems, an IAS necessitates interrupting the users in order to ask them about the adaptations that occur explicitly.

### 2.3 Cognitive Walkthrough

During this evaluation method, a group of expert evaluators construct typical user tasks in order to detect the difficulties encountered by novice users [12]. When evaluating adaptive systems, some modifications to this method are needed. For instance, multiple-action sequences per task have to be given to expert evaluators. For each action, the expert evaluators have to examine four main questions: "Will the user expect to be asked to do this?", "Will they notice the control (e.g., button)?", "Will they realize that the control is appropriate for this step?", and "Will progress be apparent once it has been used?" [7].

## 3 Usability Evaluation Methods' Choice Process

As shown in Fig. 1, the decision analysis process starts with a preparatory step, where the goal of the decision problem is defined. The aim consists in selecting the appropriate UEMs for interactive adaptive systems. This step also defines the actors involved in the decision process, including a Decision Maker (DM) and an analyst. In this research, a decision maker can be a novice evaluator; s/he can also be an expert evaluator who needs to be assured in the suitable UEMs. Finally, s/he can be a project manager who needs to be aware of the UEMs to be applied given particular
evaluation settings. The next step consists in determining the set of alternative UEMs that define the aspects relevant to the decision problem. Then, the factors (or criteria) that can affect the choice of these UEMs are identified. Different criteria should be considered when selecting appropriate UEMs for interactive adaptive systems. Once relevant criteria are retained, the performance table should be established. Each alternative UEM is classified with respect to the considered criteria after a detailed analysis of studies such as [3,4,7]. Next, an appropriate MCDA method needs to be selected in order to solve the considered decision problem. The decision maker has then to give information about the evaluation constraints of the target IAS. Usually, some parameters need to be set up in an MCDA method, such as the weight associated to every criterion. The weight refers to the relative importance of each criterion. It can be determined by the DM or estimated using a specific weighting method<fnref n="2" />. The next step consists to establish the outranking relations for the different alternative UEMs. Before proposing the final list of appropriate UEMs, it is essential first to test the robustness of the results by varying the MCDA method's parameters and observing the effect on the results. On the basis of such an analysis, it is possible to study the validity of the results. The results are said to be robust only in the case where they are not modified to any significant extent by varying the parameters [13]. After an analysis of the results, the DM has to express the satisfaction level s/he obtained from the proposed UEMs.

<figure><img src="image_2.png" /><figcaption>**Fig.1.** A flowchart illustrating the decision analysis process for the choice of Usability Evaluation Methods (UEMs).</figcaption></figure>

<footnote n="2">² Different weighting methods have been proposed in the MCDA literature to assign weights to decision criteria. One example of these methods is the 'equal weights' method. It consists in distributing weights equally among all criteria and has been used in different problems [13].</footnote>
A score is attributed; it may adopt either (1) useful, when the list of UEM(s) satisfies the evaluation constraints, or (2) not useful, when the list of UEM(s) is not suitable due either to the non-satisfaction of the evaluation constraints (e.g., exceed the available number of users, etc.) or to the lack of proposals. In this case, a relaxation of constraints has to be performed by proposing a list of actions to the DM.

# 4 Factors Affecting the Choice of Usability Evaluation Methods

One of the main parts of the UEMs' choice process is to define the factors that reflect the impact of each alternative on the decision problem. This step has a great influence on the decision process's success. As already stated, this research seeks to present the common factors that influence the choice of UEMs in the IAS field. As shown in Table 1, three groups of criteria are considered (e.g., situational factors, characteristics of stakeholders, and adaptivity aspects). It is essential to highlight here that this list is not exhaustive and may be completed with other factors.

## 4.1 Situational Factors

The choice of UEMs for interactive adaptive systems depends on different factors related to the evaluation situation. In Table 2, we classify the characteristics of the considered UEMs (cf. Section 2) according to these factors.

**Stage of Development Life-Cycle.** The evaluation of IAS can occur at three stages of the development life-cycle of adaptive systems [7]. These include (1) *specification*, which refers to the phase taking place before any system implementation, (2) *design*, which occurs during the IAS' development, and (3) *implementation*, which occurs after the implementation of a prototype of the system functionality.

<table><caption>**Table 1.** List of criteria affecting the choice of usability evaluation methods for IAS.</caption><thead><tr><td>Group of criteria</td><td>Criteria</td></tr></thead><tbody><tr><td>Situational factors</td><td>Stage of the development life cycle [7]<br/>Temporal resources [4]<br/>Financial resources [4]<br/>Style of evaluation [15]<br/>Type of data [16]</td></tr><tr><td>Characteristics of stakeholders</td><td>Number of users [17]<br/>Number of evaluators [17]<br/>Availability of direct access to users [18]<br/>Level of expertise of evaluators [19]</td></tr><tr><td>Adaptivity aspects</td><td>Intrusiveness of adaptivity [17]<br/>Reusability adaptation rules [17]</td></tr></tbody></table>
**Temporal Resources.** The application time of a usability evaluation method represents an important factor which affects its choice [4]. The duration can be represented by an ordinal scale (i.e., low, medium, or high).

**Financial Resources.** Another factor distinguishing UEMs is the required budget [4]. This criterion can be assessed using a three-level scale (i.e., low, medium, or high).

**Style of Evaluation.** Usability evaluation methods may be performed under laboratory conditions, as well as in the work environment [15]. In order to evaluate this criterion, yes is used to model the evaluation conducted in laboratory conditions, and no otherwise.

**Type of Data.** Usability evaluation methods can be distinguished according to the type of data they deal with, specifically, whether these data are qualitative or quantitative [16]. In order to evaluate this criterion, yes is used to model that an evaluation method provides qualitative data, and 0 otherwise.

<table><caption>**Table 2.** Classification of UEMs according to the situational factors.</caption><thead><tr><th></th><th>Heuristic evaluation</th><th>Cognitive walkthrough</th><th>Usability test</th></tr></thead><tbody><tr><td>Stage of the development lifecycle</td><td>Design</td><td>Design</td><td>Implementation<sup>3</sup></td></tr><tr><td>Temporal resources</td><td>Low</td><td>Medium</td><td>High</td></tr><tr><td>Financial resources</td><td>Low</td><td>Low</td><td>High</td></tr><tr><td>Style of evaluation</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Type of data</td><td>Yes</td><td>Yes</td><td>Yes</td></tr></tbody></table>

## 4.2 Characteristics of Stakeholders

A crucial consideration when selecting UEMs is the characteristics of the stakeholders involved in the usability evaluation process. In Table 3, we present a classification of the considered UEMs according to the characteristics of stakeholders.

**Number of Users.** This concerns the total number of users to be involved to use a specific UEM [17].

**Number of Evaluators.** This refers to the total number of evaluators to be involved to apply a given UEM [17].

**Availability of Direct Access to Users.** The evaluation of interactive adaptive systems can be carried out in the presence of real or representative users [18].

<footnote n="3">³ It is essential to highlight that it is possible to apply usability test using Wizard-of-Oz when a target system's functionality has not been implemented yet. In the case of this paper, the interest is on using usability test without Wizard-of-Oz technique.</footnote>
<table><caption>**Table 3.** Classification of UEMs with respect to the characteristics of stakeholders.</caption><thead><tr><th></th><th>Heuristic evaluation</th><th>Cognitive walkthrough</th><th>Usability test</th></tr></thead><tbody><tr><td>Number of users</td><td>0</td><td>0</td><td>15+</td></tr><tr><td>Number of evaluators</td><td>3+</td><td>3+</td><td>1+</td></tr><tr><td>Availability of direct access to users</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>Level of expertise of evaluators</td><td>High</td><td>High</td><td>Medium</td></tr></tbody></table>

**Level of Expertise of Evaluators.** This refers to the evaluator's expertise and knowledge of UEMs [19]. The level of expertise can be: *low*, *medium*, or *high*.

## 4.3 Adaptivity Aspects

One other important consideration when choosing UEMs is the adaptivity aspects (e.g., reusability adaptation rules, and intrusiveness of adaptivity). Table 4 presents a classification of the considered UEMs based on these aspects of adaptivity.

**Reusability Adaptation Rules.** The reusability adaptation rules can influence the choice of UEMs [17]. It may adopt two values either simple or complex. In order to evaluate this criterion, *yes* is used to model simple reusability adaptation rules, and *no* otherwise.

**Intrusiveness of Adaptivity.** This underlines the frequently-given suggestions of the interactive adaptive systems [17].

<table><caption>**Table 4.** Classification of UEMs according to the adaptivity aspects.</caption><thead><tr><th></th><th>Heuristic evaluation</th><th>Cognitive walkthrough</th><th>Usability test</th></tr></thead><tbody><tr><td>Reusability adaptation rules</td><td>No</td><td>No</td><td>No</td></tr><tr><td>Intrusiveness of adaptivity</td><td>Yes</td><td>Yes</td><td>Yes</td></tr></tbody></table>

# 5 Illustrative Example of Using an MCDA Method for Choosing Appropriate UEMs

Let us assume that a DM, who is in this case a novice evaluator, is interested in identifying the appropriate UEMs for the usability evaluation of a given adaptive tourist guide system. The adaptive system adapts the user interface and the content according to the needs and requirements of the tourists. It helps them to easily find the most appropriate itineraries, schedules, etc., according to their requirements and preferences. Furthermore, the system can be adapted based on the device on which it is displayed (i.e., desktop, Smartphone). Many adaptive systems of this type have been studied and proposed in the literature over the last decades [20,21].
As already presented, the interest of this study is related essentially to choosing the appropriate methods for the usability evaluation as a whole. To support the decision process of the choice of UEMs, the DM has to consider the evaluation constraints of the whole system to be evaluated without separation between its adaptation layers. This situation deals with choosing problem (P.α)<fnref n="4" />, where the goal is to select one or a combination of UEM(s). For this study, one MCDA method is adopted, namely the ELECTRE I method. In the sub-sections that follow, we give a brief overview of the ELECTRE I method and details to justify the choice of this MCDA method as well as an application of the ELECTRE I method.

## 5.1 Brief Description of the ELECTRE I Method

ELECTRE I method is an MCDA method based on an outranking relation (aSb)<fnref n="5" /> that aims to increase a set of alternatives in a reduced subset called kernel set [22]. This latter contains the best alternative(s). The ELECTRE I method is based essentially on the following steps:

*Calculating the Concordance Index (C(a,b))*. The aim is to test the strength of the criteria coalition in favor of the agreement to the outranking relation. In this step, the discordance index is computed as shown in (eq. (1)), where W<sub>k</sub> represents the weight for each criterion and *f*<sub>k</sub>(*a*) is the score for alternative *a* under criterion *j*.

$$ 
\begin{gathered}
C(a,b) = \frac{1}{W} \sum_{j: f_k(a) \le f_k(b)} W_k \\
\text{Where } W = \sum_{k=1}^{m} w_k, w_k > 0
\end{gathered}
\qquad (1)
$$

*Calculating the Discordance Index (D(a,b))*. The aim is to measure the rejection against the assertion aSb. In this step, the discordance index is computed as shown in (eq. (2)). More details about the ELECTRE I method can be found in [22].

$$ 
\begin{gathered}
D(a,b) = \begin{cases} 0 & \text{if } f_k(a) > f_k(b), \forall k \\ \frac{1}{\partial} \max [f_k(a) - f_k(b)], & \text{Otherwise} \end{cases} \\
\text{Where } \partial = \max_{a,b,k} [f_k(a) - f_k(b)]
\end{gathered}
\qquad (2)
$$

## 5.2 Why Adopt ELECTRE I Method?

Numerous MCDA methods exist in the literature. Each one has some advantages depending on where it is applied. According to [24], the choice of a suitable MCDA method depends mostly on the type of information available and the nature of the decision problem to be solved. As stated earlier, the decision problem to be treated in this study corresponds to (P.α). A number of MCDA methods are appropriate to decision problems involving choice, such as the ELECTRE I method and its variant

<footnote n="4">⁴ Three types of decision problems can be distinguished according to Roy [23], namely choosing problem (P.α), ranking problem (P.γ), and (3) sorting problem (P.β).</footnote>

<footnote n="5">⁵ Where a and b are two alternatives to compare.</footnote>
ELECTRE IS. Some differences between these methods exist. The main novelty of the ELECTRE IS method, for example, is the use of indifference and preference thresholds [22]. These discrimination thresholds aim essentially, in this case, to take into account the imperfect knowledge character of the DM with respect to the evaluation of alternatives. Such imperfect knowledge may arise when two alternatives are susceptible to be characterized with the same performance, which disables the DM to clearly express a preference relation for any pair of alternatives. For the problem analyzed in this case, it is presumed that neither indifference nor preference thresholds are necessary to model the preferences of our DM. Then, ELECTRE I method is retained since it seems to be appropriate for this decision problem. The main advantage of this MCDA method consists in using pair-wise comparisons between alternative UEMs, so that one can select the appropriate one or a combination of UEMs according to different criteria [25].

## 5.3 Application of the ELECTRE I Method

The use of ELECTRE I method requires the identification of a set of input data about the given decision problem. Firstly, the DM has to identify the UEMs applicable to the candidate usability attributes to be assessed. Three representative UEMs are considered in this study, namely cognitive walkthrough, heuristic evaluation, and usability test (cf. Section 2). After determining the set of criteria, a performance table should be established which consists of the evaluation of alternative UEMs through the retained criteria (Cf. Table 2, Table 3, and Table 4). Then, the decision maker is asked to answer a questionnaire in order to explore the constraints about the usability evaluation of the given adaptive system. An example of these questions is: “When will the usability evaluation of the given adaptive system be done?”. In this study, the evaluation as a whole occurs in laboratory conditions during the implementation stage. As already stated, ELECTRE I is retained. A set of input data has to be determined for this MCDA method such as the relative importance of criteria. In this study, the DM considers that the decision criteria at the same level have equal weights. The outranking relation of ELECTRE I method involves two calculations, including the concordance and the discordance indices (Cf. Section 5.1). The outranking relation is based on the concordance and discordance thresholds. Table 5 presents the concordance matrix, which aims to measure the strength of the criteria coalition in favor of the agreement to the outranking relation.

<table><caption>**Table 5.** Concordance matrix.</caption><thead><tr><th></th><th>Heuristic<br>Evaluation (HE)</th><th>Cognitive Walkthrough<br>(CW)</th><th>Usability Test<br>(UT)</th></tr></thead><tbody><tr><td>Heuristic evaluation (HE)</td><td>--</td><td>0.616</td><td>0.4</td></tr><tr><td>Cognitive walkthrough<br>(CW)</td><td>0.916</td><td>--</td><td>0.45</td></tr><tr><td>Usability test (UT)</td><td>0.833</td><td>0.85</td><td>--</td></tr></tbody></table>
Then, the discordance matrix, which aims to measure the rejection against the assertion (aSb) is calculated (Table 6).

<table><caption>**Table 6.** Discordance matrix.</caption><thead><tr><th></th><th>Heuristic<br>Evaluation (HE)</th><th>Cognitive Walkthrough<br>(CW)</th><th>Usability Test<br>(UT)</th></tr></thead><tbody><tr><td>Heuristic evaluation (HE)</td><td>--</td><td>0.266</td><td>1</td></tr><tr><td>Cognitive walkthrough<br>(CW)</td><td>0.333</td><td>--</td><td>1</td></tr><tr><td>Usability test (UT)</td><td>0.166</td><td>0.15</td><td>--</td></tr></tbody></table>

In order to interpret the information shown in the concordance and discordance matrices (Table 5, and Table 6), two thresholds (p and q) should be defined. These thresholds aim to establish the outranking relations between alternatives and to define the desired concordance and tolerated discordance. The concordance threshold p refers to the minimum concordance index needed for outranking whereas the discordance threshold q reflects the maximum discordance index required for outranking. The values for both threshold parameters are fixed as follows: p= 0.67 (represents the average of the concordance matrix); q= 0.48 (represents the average of the discordance matrix). Table 7 illustrates the outranking relations between alternative UEMs.

<table><caption>**Table 7.** Outranking relations between alternative UEMs</caption><thead><tr><th>CUEMI, UEMj</th><th>CUEMI, UEMj<br>≥ p</th><th>DUEMI, UEMj</th><th>DUEMI, UEMj<br>≤ q</th><th>UEMj=>UEMj</th></tr></thead><tbody><tr><td>CHE,CW = 0.616</td><td>No</td><td>DHE,CW = 0.266</td><td>Yes</td><td>--</td></tr><tr><td>CHE,UT = 0.4</td><td>No</td><td>DHE,UT = 1</td><td>No</td><td>--</td></tr><tr><td>CCW,HE = 0.916</td><td>Yes</td><td>DCW,HE = 0.333</td><td>Yes</td><td>CW=>HE</td></tr><tr><td>CCW,UT = 0.45</td><td>No</td><td>DCW,UT = 1</td><td>No</td><td>--</td></tr><tr><td>CUT,HE = 0.833</td><td>Yes</td><td>DUT,HE = 0.166</td><td>Yes</td><td>UT=>HE</td></tr><tr><td>CUT,CW = 0.85</td><td>Yes</td><td>DUT,CW = 0.15</td><td>Yes</td><td>UT=>CW</td></tr></tbody></table>

The results obtained by the ELECTRE I method may be expressed in the form of a graph, as shown in Fig. 2. The arrows emerging from the nodes represent the outranking relations between alternatives. Each node corresponds to an alternative UEM. Once the analysis of the robustness of results is carried out, the decision maker has to express his/her satisfaction level obtained from the proposed UEMs. In this study, the Usability Test (UT) method has no incoming arrows. It outranks Heuristic evaluation (HE) and Cognitive Walkthrough (CW) method. The latter outranks heuristic evaluation method. Hence, the usability test can be stated to be the most appropriate method for the evaluation as a whole of the given adaptive system. It is essential to highlight that these results depend essentially on the given evaluation context and can change from a situation to another.
<figure><img src="image_3.png" /><figcaption>Fig.2. Outranking graph</figcaption></figure>

6 Conclusion and Future Work

During the usability evaluation of IAS, a careful selection of UEMs should be per-
formed in order to fit better the situation of the evaluated system. As a contribution to
the IAS field, this paper summarizes firstly the most common factors that affect the
selection of UEMs for a particular situation. Three of the most commonly used meth-
ods for evaluation as a whole are classified then according to the identified factors.
Indeed, the knowledge obtained in this research is used in order to support the choice
of appropriate methods for the usability evaluation as a whole. Towards this end, the
ELECTRE I method is retained. An application of this MCDA method is presented to
support the UEMs choice process for a target adaptive system in the field of tourism.

Future work will investigate to apply our proposal in other areas of adaptive sys-
tems (e.g., adaptive learning system, adaptive e-commerce system, etc.). We also
intend to propose a Multi-Criteria Decision Support System (MCDSS) that integrates
ELECTRE I into its model base subsystem. This MCDSS will guide our DM by
providing powerful capabilities in the exploration and the comparison of alternative
UEMs.

References

1. Benyon, D.: Adaptive systems: a solution to usability problems. User Modeling and User-Adapted Interaction, 3, 65-87 (1993).

2. Jameson, A., Gajos, K Z.: Systems that adapt to their users. In: Julie A. Jacko J. (ed.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 431-456. CRC Press, Boca Raton (2012).

3. Gena, C., Weibelzahl, S.: Usability engineering for the adaptive web. In: Brusilovsky P. Kobsa A. Nejdl W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization, 720-762, Springer, Berlin (2007).

4. Dhouib, A., Trabelsi, A., Kolski, C., Neji, M.: A classification and comparison of usability evaluation methods for interactive adaptive systems. 9th International Conference on Human System Interactions, Portsmouth, UK, pp. 246-251(2016).

5. Xavier, F., Nigel, B., Tomás, A. E.: UCD method selection with usability planner. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, pp. 829-830 (2010).

6. Effie, L., Ebba, H., Gilbert C (Eds.).: Maturing Usability: Quality in Software, Interaction and Value, Springer, London (2007).

7. Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Modeling and User-Adapted Interaction. 20, 383-453 (2010).
8. Dhouib, A., Trablesi, A., Kolski, C., Neji, M.: EvalCHOICE: A Decision Support Approach for the Usability Evaluation of Interactive Adaptive Systems. Proceedings of the 21st International Conference KES, Elsevier Procedia Computer Science, Marseille, France, pp. 864-873, September (2017).

9. Dhouib A., Trabelsi, A., Kolski, C., Neji, M.: Towards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method. Proceedings of the 12th International Conference on Software Technologies (ICSOFT 2017), Madrid, pp. 163-170 (2017).

10. Nielsen, J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection Methods, pp. 25–64. John Wiley & Sons, New York (1994).

11. Dumas, J. S., Redish, J. C.: A Practical Guide To Usability Testing. Norwood, N.J. Ablex Publishing Corp (1999).

12. Mahatody, T., Sagar, M., Kolski C.: State of the art on the cognitive walkthrough method, its variants and evolutions, International Journal of Human-Computer Interaction, 26, 741-785, (2010).

13. Zardari, K., Ahmed, S. M., Shirazi, Z., Yusop, B.: Weighting Methods and their Effects on Multi-Criteria Decision Making Model Outcomes in Water Resources Management, Springer Briefs in Water Science and Technology, (2015).

14. Wang, J. J., Jing, Y. Y., Zhang, C. F., Zhao, J.H.: Review on multi-criteria decision aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, 13, 2263-2278, (2009).

15. Dix, A. J., Finlay, J.E., Abowd, G.D., Beale, R.: Human-Computer Interaction. 2nd Ed. Staffordshire Hemel Hempstead: Prentice-Hal (1998).

16. Assila, A., Oliveira, K., Ezzedine, H.: Integration of Subjective and Objective Usability Evaluation based on ISO/IEC 15939: a Case Study for Traffic Supervision Systems. International Journal of Human-Computer Interaction. 32, 12, 931–955 (2016).

17. Primož, K., Matjaž, D., Danijela, M.: Adequateness of usability evaluation methods regarding adaptivity. Simpozijum o računarskim naukama i informacionim tehnologijama, (2007).

18. Wixon, D., Wilson, C.: The Usability Engineering Framework for Product Design and Evaluation. Handbook of Human-Computer Interaction. M. Helander, T. Landauer and P. Prabhu. Amsterdam, Elsevier Science B.V: 653-688 (1997).

19. Karat, J.: User-centered software evaluation methodologies. In M. Helander, T.K. Landauer and P. Prabhu (eds.), Handbook of Human-Computer Interaction, 2nd edition, 689-704. Amsterdam: Elsevier Science B.V (1997).

20. Hariri, M., Tabary, D., Lepreux, S., Kolski, C.: Context aware business adaptation toward user interface adaptation. Communications of SIWN, 3, pp. 46-52 (2008).

21. Taktak, H., Moussa. F.: Service-oriented application creation process in ubiquitous environments: Travel assistant mobile application. International Journal of Pervasive Computing and Communications, 13(3), 300-330 (2017).

22. Roy, B.: The Outranking Approach and the Foundations of ELECTRE Methods. Theory and Decision (1991).

23. Roy, B.: Méthodologie Multicritère d'Aide à la Décision, Economica, Paris (1985).

24. Guitouni, A., Martel, J-M., Vincke, P.: A Framework to Choose a Discrete Multicriterion Aggregation Procedure, Defence Research Establishment Valcatier (DREV) (1998).

25. Bouyssou, D., Duckstein, L., Goicoechea, A., Zions, S.: On Some Properties of Outranking Relations Based on a Concordance-Discordance Principle, Multiple Criteria Decision Making. Springer-Verlag, Berlin, 93-106 (1992).