# Using fuzzy coding with qualitative data: example with subjective data in human-computer interaction

Pierre Loslever, Taisa Guidini Gonçalves, Káthia Marçal de Oliveira, Christophe Kolski

<figure><img src="image_1.png" /></figure>

## To cite this version:

Pierre Loslever, Taisa Guidini Gonçalves, Káthia Marçal de Oliveira, Christophe Kolski. Using fuzzy coding with qualitative data: example with subjective data in human-computer interaction. Theoretical Issues in Ergonomics Science, 2019, 20 (4), pp.459-488. 10.1080/1463922X.2019.1574932. hal-03274971

HAL Id: hal-03274971

https://uphf.hal.science/hal-03274971

Submitted on 21 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Using fuzzy coding with qualitative data. Example with subjective data in Human-Computer Interaction

Pierre Loslever*, Taisa Guidini Gonçalves, Káthia Marçal de Oliveira, Christophe Kolski

LAMIH CNRS UMR 8201, Université Polytechnique Hauts-de-France, Valenciennes - France

* Contact: Pierre Loslever - Pierre.Loslever@uphf.fr

LAMIH CNRS UMR 8201, Université Polytechnique Hauts-de-France,
Le Mont Houy, 59313 Valenciennes cedex 9, France

**Abstract.** This article shows the role that fuzzy sets may play in the prospect of analyzing qualitative data. To underline this role, a Human-Computer Interaction (HCI) study is presented. The data coming from 20 experts concerns their judgment regarding 33 questions related to the use of HCI approaches in order to support interactive system development phases. Each response scale features three main modalities, i.e. *Agree*, *Partially agree*, and *Disagree*. The dataset example is analyzed using Multiple Correspondence Analysis (MCA) with both crisp and fuzzy coding models where the intermediate modality, Partially agree, is removed and considered with ½ membership values to the two extreme modalities. A comparative analysis is performed and the discussion states the interest of fuzzy coding with several kinds of qualitative factors or measurement variables. With qualitative measurement variables (our example), the main drawback of fuzzy coding could be the information loss, which is counterbalanced by the possibility of having fewer modalities and therefore of simplifying the multivariate analysis.

**Keywords:** Fuzzy sets; multiple correspondence analysis; questionnaire; qualitative data; human-computer interaction

## Relevance to human factors/ergonomics theory

Studies into Human Component Systems (HCS) often involve recording qualitative and/or quantitative data (Jacko, 2012) (Wilson & Sharples, 2015). The analysis of such data gets complex if (1) both types of data exist and (2) both the multifactor and the multivariate aspects are kept (the multivariate aspect is inherent to the system approach and reduces the information loss to a lesser degree compared with the univariate or bivariate analysis). If the concept of Zadeh's fuzzy membership value is interesting and proven with quantitative data, e.g. with scales such as temperature or geometric size in the fields of automatic control or pattern recognition, this concept can be used with qualitative data (as suggested in this article). The concept can also be used when both qualitative and quantitative data is present: using membership values yields homogeneous data, the latter can be analysed by multivariate factorial or clustering methods.
# 1. Introduction

The title of this article includes some words that may imply different meanings, either within or between different research fields (ergonomics, psychology, human-computer interaction, ecology, statistics, artificial intelligence, big data ...).

## 1.1. Qualitative data

This expression is often linked to the concept of measurement scale models. If several classifications exist, thus with specific terms for each scale of measurement (Friendly & Meyer, 2016) (Gray, 2004) (Jobson, 1999) (Stevens, 1946) (Thole, Zimmermann, & Zysno, 1979) (Tukey, 1977), we suggest the quantitative (QT) vs. qualitative (QL) scale as a main taxonomic dimension. The qualitative family contains variables where possible values (hereby named modalities) are either without any order, such as gender, dominant hand (Jones, Reed, & Chaffin, 2015), type of driver impairment (Lenné & Jacobs, 2016) and study participant experience (Ingram, Duncan, Mansfield, Byrne, & McIlroy, 2016) or with an order, such as minimal/little/no risk as regards “very little risk for the particular level of a given risk factor” (Bell & Crumpton, 1997), 12 working/clock positions relative to the dental hygienist’s client lying supine in the patient’s chair (Howarth, Grondin, La Delfa, Cox, & Potvin, 2016) (La Delfa, Grondin, Cox, Potvin, & Howarth, 2017), directions and speeds in 2D tracking (Miyake, Loslever, & Hancock, 2001) and “neutral” (flexion/extension ± 30°), “flexion” (<-30°) and “extension” (>30°) to estimate hand/wrist postures in Carpal Tunnel Syndrome study (Kapellusch et al., 2013) or any Likert-type scales (Saporta, 2011). As suggested below, the same variable may involve different scale types depending on the context (e.g. how it is measured and the goals of the data analysis).

## 1.2. Subjective data

In ergonomics, numerous physical concepts (position, force, temperature, color, ...) can be measured using specific measurement systems (Fraden, 2004) (Laeng & Teodorescu, 2002), yielding objective variables. Such concepts can also be obtained regarding the Human being, the “measurement system” including both the individual (either an expert or not) and a tool allowing the Human being to give his/her assessment. In such a case and, of course, in all cases where intrinsically subjective concepts are to be reached (mental or physical workload, pain, task difficulty, ...) (Bainbridge & Sanderson, 2005) (Gray, 2004) (Lannoy & Procacia, 2001) (Meshkati, Hancock, & Rahimi, 1990) (Prytz & Scerbo, 2015) (Stanton, Salmon, Walker, Baber, & Jenkins, 2000) (Visser, Krosnick, & Lavrakas, 2000) (Young, Brookhuis, Wickens, & Hancock, 2015), a subjective variable is present. In many experimental or observation situations, the tool can be seen as being composed of both a question and a response fields where several scale models can be proposed.

## 1.3. Fuzzy coding

Here, the main notion is Zadeh’s membership value, which can be within [0, 1] instead of within {0, 1} (for an overview of fuzzy sets in the fields of ergonomics, see (Karwowski et al., 2006)). It is worth noting that, in ergonomics, quantitative scale
fuzzy windowing is sometimes used (Karwowski and Ayoub 1984; Loslever 1993 and 2014). Nevertheless, the frequency of use is much lower compared with cases where the measurement scale is kept as it is in the data analysis (with either descriptive or inferential approach), i.e. in more than 95% of cases. Still, with quantitative scales, fuzzy sets may be found in the analysis of data related to either subjective or objective scales. Concerning the first case, one can evoke perceived workload in manual lifting tasks (Karwowski & Ayoub, 1984), car speed (Brackstone, 2000), subjective trust in the context of fuzzy signal detection theory (Parasuraman, Masalonis, & Hancock, 2000) or assessment of comfort and pain in data entry tasks (Loslever & Lepoutre, 2004). Concerning the objective scales, let us mention risks in an occupational environment (Bell & Crumpton, 1997), flexion/extension angles and forces in tasks leading to the Carpal Tunnel Syndrome (Loslever & Ranaivosoa, 1993), direction and speed in 2D tracking (Loslever, 1993), eye movement in driving (Loslever, Popieul, & Simon, 2003). For a qualitative variable, it could seem odd to suggest that the membership values linked to the scale modalities are not only either 0 or 1. If one considers both the example above about the gender variable and that the scale depends on the context, we can consider the two following empirical situation examples: (1) an epidemiologic study when the gender variable must be taken into account and (2) a psychological or criminal study where several individual faces are shown, some with effeminate or androgynous aspects. If in the first situation, the numbers 0 and 1 can be imagined as the only possible membership values, other membership values can be imagined in the second case, e.g. 0.5.

Given these points of view for some words mentioned in the title of the article, the rest of the paper will focus on the role played by data coding in multivariate analysis where either all the variables or very few variables (one at least), feature a qualitative scale. The rest of this article is organized as follows: Section 2 briefly recalls some differences when using initial data, crisp and fuzzy coding with the quantitative scale and a multivariate analysis perspective. Section 3 focuses on the qualitative scale and presents the method (participants, data collection, and data analysis procedure) used in an example in the context of Human-Computer Interaction (HCI). Section 4 presents the results generated with the coding and analysis methods considered in this paper. Section 5 discusses these results. Section 6 presents the threats to validity considered in the analysis of this study. Finally, section 7 presents the conclusion of this study.

## 2. Fuzzy coding and multivariate analysis in the case of quantitative data

Fuzzy set theory and fuzzy logic (FSTL), which were first introduced by Zadeh in 1965, have been used essentially together for about fifty years in fields such as automatic control, pattern recognition or expert systems; see (Komem & Schneider, 2005) for an overview. It is worth noting that the use of FSTL is less frequent in the field of statistics than in the domains mentioned above, even if, from the very beginning of his 54 years-old theory, Zadeh stated that “Probability theory/statistics and fuzzy logic should be viewed as complementary rather than competitive” (D’Urso & Gil, 2017). The main reason is that FSTL based tools have been initially designed for the data processing, in the sense of real-time operation of signals, whereas for statistics, the data analysis is required, in the sense that the Human being must be present for each of the following
three steps: (1) observing the data (e.g. through graphical summaries), (2) choosing the statistical methods (to get effects, relationships or classes), and (3) interpreting the results in relation to the domain the data come from (the first 2 steps require data processing). Several statistical tools used for several decades (arithmetic mean, analysis of variance, regression, principal component analysis or clustering) do exist with FSTL, see (D'Urso & Gil, 2017) for an overview; nevertheless these tools are first much more complex mathematically speaking and then less present (or not present at all) in usual statistical packages. There is an abundant research in FSTL connected to statistics with possible applications to ergonomics studies. One of the most important topics is related to step 2 evoked above, i.e. fuzzy clustering (Bian, Ishibuchi, & Wang, 2019), which can be applied on study participants or time data. Another topic worthy to mention is the possibility to get fuzzy data directly from the study participants (Jónás, Tóth, & Árva, 2018); with the 3 steps evoked above, getting data (objective and subjective, qualitative and quantitative) could be in step 0.

Here we focus on the combination of data fuzzification and usual statistics in the field of ergonomics, which often yields both several kinds of data and large data sets (e.g. when biomechanical signals are recorded using 3D measurement devices).

Before considering the qualitative case, which is the focus of this article, it is worth remembering the concept of fuzzy coding with quantitative scales. Keeping in mind that ergonomic studies can be performed using either an experimental design or observational design (Sheskin, 2007), here are two generic examples with V quantitative scales:

(1) Generic example 1: physical comfort and pain indicators are obtained using a questionnaire with a continuous response segment (Stanton et al., 2000);

(2) Generic example 2: effort and movement signals are obtained using specific measurement devices (Allard, Stokes, & Blanchi, 1995).

In both generic examples, two main statistic analysis objectives are often present: (i) to show factor effects on the V variables (or any summary found from these variables, more particularly with signals), and (ii) to show connections between these V variables (or summaries). If the multivariate aspect is kept in the statistical analysis, these two objectives can be reached using, firstly, the factor analysis principle (eigendecomposition of a specific matrix); secondly, methods that are more specific can be used in an inference context. Let us consider a didactic example with V variables and 2 values for each variable, Figure 1 (given our 2 generic examples, these values may correspond to 2 individuals and to 2 time samples respectively). Three situations are worth being mentioned:

(1) the data is kept as it is. This is the most frequent case, the easiest approach but the information loss may be considerable as soon as an averaging has to be done (to show factor effects). For instance, two very different sets can yield identical summaries; see Figure 1 (a) and the famous example of (Anscombe, 1973) where different bivariate data sets give the same arithmetic mean, the same standard deviation and the same correlation coefficient. Then the initial data values can be included within a table where the columns correspond to the variables and the rows to the observations (individuals or time samples with our two generic examples). This table can be studied using Principal Component Analysis (PCA), which mainly shows linear relationships;
(2) a (usual) windowing is performed (often named 0/1 coding or crisp coding). With this approach, the membership value of each data piece to any window is within the set {0,1}. The information loss can be lower than previously (e.g. the two membership value average triplets are different - Figure 1 (b)). PCA can be replaced by Multiple Correspondence Analysis (MCA) (Benzécri, 1992) (Miyake et al., 2001). MCA allows complex relational phenomena to be found, but with few observations (several tens), the arbitrary choice of the border type for two adjacent intervals may have an impact on the (average) membership values (e.g. a possible interval series can be [0, 10], ]10, 20] and ]20, 30] but also [0, 10[, [10, 20[ and [20, 30]). Still with the arbitrary aspect of the position of the boundaries between 2 intervals (e.g. 10 and 20), two close data pieces may belong to two different windows (e.g. 9 and 11), and two far data pieces may belong to the same window (e.g. 11 and 19);

(3) a fuzzy windowing is performed. The membership value is now within [0, 1] and to remain in a statistical context, the sum of the membership values is 1 for any data piece, Figure 1 (c). The information loss is lower than with 0/1 coding and the windowing can be data adapted, which allows to introduce the ergonomist's point of view (Loslever, 1993) (Loslever, 2014).

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Different data coding approaches in the perspective of a multivariate analysis using the factor analysis principle (case of quantitative data). (a) the data is kept as it is, (b) crisp windowing, (c) fuzzy windowing.</figcaption></figure>
This section briefly recalled the principle and interest of fuzzy coding with
quantitative data and factor analysis; let us now turn to qualitative data, the focus of this
article.

**3. Method**

A study was performed to validate propositions from approaches in the Human-
Computer Interaction (HCI) domain (Gonçalves, Oliveira, & Kolski, 2018). A literature
review was carried out to identify and collect HCI categories (with examples of
approaches) regarding CMMI-DEV - Capability Maturity Model Integration for
Development (CMMI Product Team, 2010) in the first phase. The objective was to
integrate HCI approaches in the CMMI-DEV process coming from the Software
Engineering domain. The bibliographic analysis (Gonçalves, 2017) gave ten HCI
categories:

* Task Analysis Methods for HCI

* Prototype for HCI requirements

* Operational Concepts and Scenarios Specification for HCI

* Standards and Guidelines for design and documentation of HCI

* Techniques to validate HCI requirements

* Architecture Patterns for HCI

* Design patterns for HCI

* Functional Prototype to validate HCI

* Evaluation methods for HCI verification tests

* Evaluation methods for HCI review

The second phase consisted in collecting expert points of view about the support
of HCI categories related to CMMI-DEV practises.

**3.1. Participants**

Twenty high-level experts with a Ph.D. degree in the field of Computer Science and more specifically in the HCI domain were considered. These experts (see Table 1) had from 7 to 40 years of experience in this domain; they are known in the HCI community; and were recruited from five countries (Algeria, Belgium, Brazil, France and Tunisia). We interviewed the experts one-to-one using a questionnaire. Each interview lasted about 2 hours, which limited our sample to 20 experts.

**3.2. Data collection**

A questionnaire (see extract in Figure 2) was designed according to the different
components (*Process Area* – Requirements Development, Technical Solution, Product
Integration, Validation and Verification; *Specific Goal* (SG); and *Specific Practice* (SP))
of the CMMI-DEV model (CMMI Product Team, 2010). For each specific practice, we
proposed one or more HCI categories; each HCI category proposed for one practice
corresponds to an item in the questionnaire. The questionnaire contained 33 items, each
item in the questionnaire being represented using a label with 3 letters: first letter -
process area; second letter - specific practice; and third letter - HCI category, (for more
details, see **Appendix A. Questionnaire description**, where each letter is in bold type for each corresponding column).

<table><caption>**Table 1.** Experts' information - adapted from (Gonçalves et al., 2018)</caption><thead><tr><th rowspan="2">Expert</th><th colspan="3">Background</th><th rowspan="2">Origin</th><th rowspan="2">Interview type</th></tr><tr><th>Experience in HCI (years)</th><th>PhD domain</th><th>Current interest in interactive systems</th></tr></thead><tbody><tr><td>E1</td><td>13</td><td>HCI</td><td>Methods and models for HCI design and evaluation</td><td>France</td><td>In person</td></tr><tr><td>E2</td><td>25</td><td>HCI</td><td>Tools for the design, realization and evaluation</td><td>France</td><td>In person</td></tr><tr><td>E3</td><td>8</td><td>HCI</td><td>Agent-based architecture models and HCI evaluation</td><td>France</td><td>In person</td></tr><tr><td>E4</td><td>8</td><td>SE-HCI</td><td>Interaction and Automatic Reasoning</td><td>France</td><td>In person</td></tr><tr><td>E5</td><td>25</td><td>SE-HCI</td><td>Methods and tools of systems engineering</td><td>France</td><td>In person</td></tr><tr><td>E6</td><td>26</td><td>HCI</td><td>HCI</td><td>France</td><td>In person</td></tr><tr><td>E7</td><td>27</td><td>SE-HCI</td><td>SE and HCI</td><td>Belgium</td><td>In person</td></tr><tr><td>E8</td><td>20</td><td>HCI</td><td>HCI</td><td>Brazil</td><td>Video conference</td></tr><tr><td>E9</td><td>10</td><td>HCI</td><td>HCI</td><td>Brazil</td><td>Video conference</td></tr><tr><td>E10</td><td>25</td><td>HCI</td><td>HCI</td><td>France</td><td>In person</td></tr><tr><td>E11</td><td>20</td><td>SE-HCI</td><td>User Interfaces Plasticity, Creativity Support Tools, and Persuasive Technology</td><td>France</td><td>In person</td></tr><tr><td>E12</td><td>40</td><td>SE-HCI</td><td>Innovative interfaces, mobility</td><td>France</td><td>In person</td></tr><tr><td>E13</td><td>12</td><td>SE-HCI</td><td>Quality of Human-Computer Interfaces</td><td>France</td><td>In person</td></tr><tr><td>E14</td><td>7</td><td>SE-HCI</td><td>HCI</td><td>France</td><td>In person</td></tr><tr><td>E15</td><td>10</td><td>HCI</td><td>HCI</td><td>Brazil</td><td>Video conference</td></tr><tr><td>E16</td><td>30</td><td>CS-HCI</td><td>Interactive critical systems</td><td>France</td><td>Video conference</td></tr><tr><td>E17</td><td>27</td><td>CS-HCI</td><td>HCI design, Ubiquitous computing</td><td>Tunisia</td><td>Video conference</td></tr><tr><td>E18</td><td>21</td><td>CS-HCI</td><td>Semiotic engineering, evaluation and design of interfaces</td><td>Brazil</td><td>Video conference</td></tr><tr><td>E19</td><td>10</td><td>CS-HCI</td><td>Organizational Semiotics, Culture and Values in design</td><td>Brazil</td><td>Video conference</td></tr><tr><td>E20</td><td>27</td><td>CS-HCI</td><td>Service Design, Ubiquitous Computing, SOA</td><td>Algeria</td><td>In person</td></tr></tbody></table>

CS – Computer Science, SE – Software Engineering, and HCI – Human-Computer Interaction

The questionnaire was elaborated in three languages: English (see **Appendix B. Questionnaire for interview**), French, and Portuguese. Each expert was interviewed in person or by video conference. One or two researchers (among the authors of this article) performed the interviews. For each specific practice (see Figure 2, column 2), the experts had to answer if they agreed (A), partially agreed (a) or did not agree (D) – see Figure 2 (columns 4, 5, and 6) – that the associated HCI category supports the practice. The partially agree and disagree responses were justified by experts (see Figure 2, column “justification”), and when necessary they explained regarding other proposals.
<table><caption>**Figure 2.** Example of questions</caption><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>A</th><th>a</th><th>D</th></tr></thead><tbody><tr><td rowspan="2"><strong>Product Integration</strong><br>SG 1 Prepare for Product Integration<br>Preparation for product integration is conducted.</td><td><strong>SP 1.1 Establish an Integration Strategy</strong><br>Establish and maintain a product integration strategy.</td><td><strong>Prototype for HCI requirements</strong><br>Examples:<br>❖ Rapid Prototyping<br>• Offline techniques: Paper and pencil (paper sketches, storyboards), Mockups, Wizard of Oz, Video prototyping<br>• Online techniques using software tools: No interactive simulations, Interactive simulations, Scripting languages</td><td></td><td></td><td>x</td><td>“This type of prototype does not contribute to integration.”</td></tr><tr><td><strong>SP 1.1 Establish an Integration Strategy</strong><br>Establish and maintain a product integration strategy.</td><td><strong>Functional Prototype to validate HCI</strong><br>Examples:<br>❖ Iterative and Evolutionary Prototypes<br>• User interface toolkits<br>• User interface builders<br>• User interface development environments</td><td></td><td>x</td><td></td><td>“You should use a good IDE.”</td></tr></tbody></table>

### 3.3. Data analysis procedure

Multiple-choice questionnaires are increasingly studied using Multiple Correspondence Analysis (MCA) (Benzécri, 1992) (Jobson, 1999) (Greenacre & Blasius, 2006) (Beh & Lombardo, 2014) (Friendly & Meyer, 2016). Since this method is not very well-known in the field of Human Factor/Ergonomics, a summary can be found in (Loslever & Lepoutre, 2004). In this reference, MCA was used with quantitative data cut using specific membership functions, e.g. the membership values were either within {0, 1} with crisp windowing, also called the 0/1 “dummy” indicator (Friendly & Meyer, 2016), or within [0, 1] in the fuzzy case. Here, as stated above, the data are qualitative. The statistical analyses are performed using software R with the package FactoMineR (Dalgaard, 2008) (Cornillon, 2012).

#### 3.3.1. Analysis in the crisp coding case

Considering the 33 variables linked to the questionnaire, the 0/1 coding approach is worthwhile whenever the expert does not provide any answer because a new *No response* (N) modality can be added to the three initial ones, yielding a total of 4 modalities, i.e. A, a, D and N (the modality N was present for 4 questions and used by 3 experts only. The binary sub-table corresponding to an expert giving No response for
a given variable is coded using the quadruplet (0, 0, 0, 1). Still considering these 33 variables, the number of modalities ranged from 2 to 4, e.g. 2 corresponds to the case where only *a* and *A* were used (for 10 questions). Given all the answer possibilities used for the 33 variables, the total number of modalities was 91, thus yielding a binary table with a rather large number of columns. To summarize, using 0/1 coding yields table *Y1* with *R1*=20 rows and *C1*= 91 columns; the generic value is $y1_{ivs} \in \{0,1\}$, where *i*, *v* and *m* indicate the individual, variable and modality respectively, and the generic set of modalities is $\mathcal{M1}=\{A, a, D, N\}$. The MCA of this table will be named MCA1 (Figure 3 left side).

### 3.3.2. Analysis in the Fuzzy coding case

The main idea is to state that modality *a* is between modalities *A* and *D*, i.e. using *a* gives the membership value triplet (1/2, 1/2, 0) if the remaining modalities are *A*, *D*, and *N*. Keeping in mind that modality *N* is present for 4 variables, the membership value table is *Y2* with *R2*=20 rows and *C2=((33-4)*2)+(4*3)=70* columns; the generic value is $y2_{ivs} \in [0,1]$ and the generic set of modalities is $\mathcal{M2}=\{A, D, N\}$. MCA of this table will be named MCA2 (Figure 3 right side).

Given both our didactic context (here the aim is to focus on the impact of different coding schemes using the HCI context example) and the presence of rather large data tables (*Y1* and *Y2* with 91 and 70 columns respectively), we consider only the MCA output related to the Main plane obtained when crossing the first two principal axes (called Axes 1 and 2). Moreover, being undeniable that the graphical results are relatively complicated (for example, for the main plane of MCA1, there are 90 column points related to 33 variables), only the points with the strongest contributions to the control of Axes 1 and 2 will be shown (some points can be moved a little to show the corresponding labels).

The last stage of our statistical analysis procedure will consist of comparing MCA1 and MCA2 outputs. The modalities being different, only a qualitative comparison of modality trajectories can be performed. To compare MCA1 and MCA2 results in a more quantitative way, the 190 distances between the 33 experts' points can be computed with both MCA1 and MCA2, and visualized using a bivariate scatterplot, Figure 3 bottom.

## 4. Results

MCA1 gives a first main axis which is mainly controlled by modality *a* for 11 questions; see Figure 4 (a), right side (so as not to overcomplicate the graph, the points are not all displayed, see Table 2 for details). For the corresponding variables, the relative positions of modality points *A* and *a* along Axis 1 yield that:

(1) axis 1 opposes modality *A*, on the left side, to modality *a*, on the right side,

(2) modality *A* is used more often than modality *a* (abiding by the usual barycenter principle, modality points *A* are closer to the center of gravity, i.e., the intersection between the main axes).
<figure><img src="image_4.png" /><figcaption>**Figure 3.** Overview of the analysis procedure with crisp (in blue) and fuzzy (in green) coding.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>**Figure 4.** Data analysis with crisp (left side) and fuzzy (right side) coding (see Figure 3 for an overview). (a) and (b) the Main plane of MCA1 with 20 experts who answer 33 questions about HCI practice (only modality points with the highest contributions are shown). (c) and (d) idem with MCA2. (e) MCA1 vs. MCA2 comparison.</figcaption></figure>
Figure 4 (b) is consistent with this result, since it opposes a cluster of 9 experts on the left side (thus experts often using modality *A* for questions linked to Axis 1, see Figure 4 (a)) to a cluster of 5 experts on the right side (these experts often use modality *a*). Table 2, which describes the responses for the two clusters of experts, confirms the trends shown by Axis 1.

<table><caption>**Table 2.** Main results underlined by Axis 1 of MCA1, Figure 4 (a) and (b): the rows correspond to the most discriminant and connected modalities; the columns to 2 clusters of experts (the 14 experts present high contributions in Axis 1 control); a row/column intersection (in italic) gives the expert's response with *A*=Agree, *a*=partially agree and *N*=No answer.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">labels</th><th rowspan="2">Relative<br>contribution<br>in %</th><th colspan="10">Cluster of Experts<br>situated on Axis 1 left side<br>Leftmost position</th><th colspan="5">Cluster of Experts<br>situated on Axis 1<br>right side<br>Rightmost position</th></tr><tr><th>3</th><th>12</th><th>9</th><th>2</th><th>1</th><th>13</th><th>17</th><th>14</th><th>10</th><th>7</th><th>19</th><th>5</th><th>18</th><th>8</th></tr></thead><tbody><tr><td rowspan="10">Highest<br>contribution</td><td>ESV.a</td><td>5</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>EEV.a</td><td>5</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>N</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>EPV.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>A</td><td>a</td><td>a</td><td>a</td></tr><tr><td>ERR.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>ECR.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>AEV.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>N</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>AVS.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>APV.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>ASV.a</td><td>4</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>A</td><td>a</td><td>a</td><td>a</td></tr><tr><td>EVS.a</td><td>3</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td><td>a</td></tr><tr><td>EDR.a</td><td>3</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>A</td><td>a</td><td>a</td><td>A</td><td>a</td><td>a</td><td>a</td><td>a</td></tr></tbody></table>

Axis 2 is mainly positioned by modality *D* of variables ESV, EEV, EPV, EAV, and TPA, see Figure 4 (a) top and Table 3. In fact, the close positions for these 5 points are mainly due to *Expert 16*, Figure 4 (b) top, who uses the *Disagree* modality for these 5 questions. *Expert 6* and *Expert 3* also have the main contribution in Axis 2 control. *Expert 6* uses *D* modality for ESV, EEV, EPV, EAV but not TPA, yielding an average top position compared with the extreme top position of *Expert 16*. The opposite position of *Expert 3* is due to the absence of use of *D* modality for these 5 variables, see Table 3.

When comparing MCA2 with MCA1, Figure 4 (a) to (d), a first difference originates from the higher relative inertia (38% vs. 31%), which sounds logical since the number of modalities is lower (70 vs. 91). The second difference is related to the points with the strongest contributions, for instance the modality *N* of AEV, EEV and TID plays a pivotal part in Axis 1 control with MCA2, see Figure 4 (c) right side, while the contributions for these 3 points were high for Axis 3 of MCA1. The high contributions for these 3 points are due to *Expert 10*, see Figure 4 (d) right side, who used the modality *N* for AEV, EEV, and TID (and only for these 3 variables).
<table><caption>**Table 3.** Main results underlined by Axis 2 of MCA1: the rows correspond to the most discriminant and connected modalities; the columns to the 3 experts with the 3 highest contributions in Axis 2 control; a row/column intersection (in italic) gives the expert's response with A=Agree, a=partially agree and D=Disagree.</caption><thead><tr><th></th><th></th><th></th><th>Expert situated on<br>Axis 2 bottom side</th><th colspan="2">Experts situated on<br>Axis 2 top side<br><i>Topmost position</i></th></tr><tr><th></th><th>labels</th><th>Relative<br>contribution<br>in %</th><th>3</th><th>6</th><th>16</th></tr></thead><tbody><tr><td rowspan="6">Highest<br>contribution</td><td>ESV.D</td><td>8</td><td>A</td><td>D</td><td>D</td></tr><tr><td>EEV.D</td><td>8</td><td>A</td><td>D</td><td>D</td></tr><tr><td>EPV.D</td><td>8</td><td>A</td><td>D</td><td>D</td></tr><tr><td>EAV.D</td><td>8</td><td>A</td><td>D</td><td>D</td></tr><tr><td>TPA.D</td><td>7</td><td>A</td><td>A</td><td>D</td></tr><tr><td>PEF.a</td><td>4</td><td>D</td><td>A</td><td>a</td></tr></tbody></table>

Figure 4 (e), which displays the differences between MCA1 and MCA2 more directly, shows that distances d1 and d2 are not highly linked (cor(d1, d2)=0.46 only): for about two thirds of expert pairs, there are connections but for the remaining third, d1 and d2 are quite independent. Such a result is consistent with the points that played a main role in the control of Axes 1 and 2. Thus, the introduction of fuzzy windowing changes the MCA output (in our case, see the discussion for generalization).

# 5. Discussion

## 5.1. Regarding the result of the HCI example

It is worth remembering that the crisp vs. fuzzy comparative analysis must not be generalized because the number of experts is small (20), which may give results that are not robust (with MCA but also as with any other method, such as Pearson's Chi-squared test or Fisher's exact test). The main advantage is the reduction in the number of modalities (from 91 to 70), which yields a higher inertia spanned by the principal plane (22% more) and simpler modality trajectories. If Figure 4 (a) and (b) are considered as a reference (all the initial modalities are present), Figure 4 (c) and (d) are rather far, which may suggest that coding ½/½ is “bad”. Figure 4 (e) confirms the major difference because many points (almost one third) are not plotted along a line. Nevertheless, one must keep in mind that the 4 plots (Figure 4 (a) to (d)) only consider Axes 1 and 2 which display (1) different relative inertia and (2) different statistical phenomena (see Figure 4 (c) and (d)). For instance, *Expert 10* has a very high contribution in Axis 1 control because he/she uses the modality *N* for questions *AEV*, *EEV*, and *TID*, Figure 4 (c) (remember that this result was shown on Axis 3 of MCA1). This is quite an interesting result, in that this expert does not want to give any appreciation (in the same way that an individual refuses to give financial information, such as salary). The expert did not respond to the questions *AEV* and *EEV* because he/she was not sure that the propositions suggested could be applied to these practices. For the question *TID* he/she
argued that his/her knowledge about the category (Design patterns for HCI) was not sufficiently extensive to answer the question. This interesting result being noted, *Expert 10* may be removed from MCA (it is worth stating that MCA based data analysis must be seen as a process involving several loops; for instance, if there are outliers or points with a high contribution, as with *Expert 10*, the corresponding row or column may be removed, or considered as a supplementary point, before restarting the MCA) (Benzécri, 1992).

Figure 5 schematically shows both the main advantage of using fuzzy coding, i.e. only two modalities (*A* and *D*) remain, and the main disadvantage, i.e. on Axis 1 right side, some experts use the 3 modalities (*A*, *a* and *D*). The distinction between Axis 1 left and right sides, Figure 5, is less obvious than when the 3 modalities are present, see Figure 4 (a) and (b) and Table 2.

<figure><img src="image_6.png" /><figcaption>**Figure 5.** Overall principle of the MCA main plane with the HCI example: 19 experts (Expert 10 is removed, see the text) answer 33 questions about the HCI practice. Case of the coding 1/2/1/2 for A/D modality pair (thus each modality has the same membership function, modality *a* being removed).</figcaption></figure>

**5.2. Generalization of fuzzy windowing based data analysis**

Before assessing the pros and the cons of qualitative-variable fuzzy windowing, let us focus on (1) the possible tools for the analysis of membership values and (2) the qualitative vs. quantitative aspects.

(1) Given the sets of factors, measurement variables and objectives inherent in any Human Component System (HCS) study, there are not many data analysis tools capable of keeping both multifactor and multivariate aspects with membership values. Indeed in the previous example, only one factor was present, the individual factor, but in most cases, the number of factors is higher; for instance, in an experimental design with 3 experimental factors and yielding signals, the factor set contains these 3 factors plus the individual and time factors, thus there are 5 factors. These data analysis tools can be based on: (i) fuzzy set theory, the latter being introduced in method families such as factor analysis (Giordani & Kiers, 2004) (Nakamori & Ryoke, 2006) (Theodorou,
Drossos, & Alevizos, 2007), classification (Guerra & Loslever, 1993) (Peng, Zhang, Zhang, & Yang, 2017), or regression (Manton, Woodbury, & Tolley, 1994) (Peters, 2011); or (ii) usual Statistics with fuzzy field data, but more or less with the same families (Bordet, 1973) (Guitonneau & Roux, 1977) (Gallego, 1982) (Guerra & Loslever, 1993) (Murtagh, 2005) (Loslever, Simon, Rousseau, & Popieul, 2008) (Huang et al., 2017). For a very first approach to data, the second category with MCA was chosen because even if this multidimensional method is complex (more particularly compared to Principal Component Analysis or Hierarchical Clustering), it is increasingly well-known (Beh & Lombardo, 2014) (Taha & Hadi, 2016) and used. This possible use concerns complex data sets (multivariate *and* multifactor aspects are present, with qualitative *and* quantitative scales) or large data sets (presence of many multidimensional signals or time series) (Chevene, Doleadec, & Chessel, 1994) (Loslever, 2014) (Schiro, Loslever, Gabrielli, & Pudlo, 2015) (Gagolewski, Bartoszuk, & Cena, 2016) (Taha & Hadi, 2016) (Xu, Wang, & Deng, 2016) (Mears, Roberts, & Forrester, 2018).

(2) Let us focus again on qualitative and quantitative aspects (see the introduction for quantitative vs. qualitative measurement scale). Philosophically, the main characteristic of a *quantity* is that one can use the words *equal* and *unequal*, while words *similar* or *dissimilar* stand for a *quality* (Benzécri, 1988). Secondly, given some methods mentioned above (e.g. regression or modeling based on fuzzy set theory) and the data analysis method used in this paper (MCA), the former are often considered *quantitative* (with the possibility of inference) while the latter is considered *qualitative* (the word *descriptive* is also often used). Thirdly, there are the specialists in the domains from which the data come from (ergonomics, medicine, psychology, engineering, ...) who use both quantitative and qualitative measurement scales, and specialists in the domains of data analysis (statistics, computer science, automation, ...) who suggest mathematical descriptions through quantitative forms (a (fuzzy) set and corresponding membership values, a point in large multidimensional space, a distance, ...). As a consequence of these three points, faced with a complex and/or large data set, one must acknowledge that MCA, as all the methods mentioned previously, mainly yields qualitative results (although MCA outputs are purely quantitative). For instance, if one focuses on Figure 4, this qualitative aspect stands out from the analysis of both the tables that aid interpretation (e.g. row and column point contributions shown through lozenge sizes) and the graphical views (e.g. point relative positions). In the same way, when comparing several sets of results, either from a same method (e.g. using a scatterplot, see Figure 4) or different methods (e.g. using a box or violin charts when comparing algorithms (Gagolewski et al., 2016) (Peng et al., 2017), the qualitative aspect is preponderant.

Keeping in mind (1) and (2), one must acknowledge that removing one modality among the initial set of modalities ({A, *a*, D, N} in our example), entails an information loss. This is particularly the case when subjective data is present, especially when such data comes from experts: the observer or experimenter spent much time in (i) building a tool making it possible to get the expert's point of view, i.e., a questionnaire with several questions and response fields, and (ii) collecting the responses. Thus, it is frustrating to modify the answer set. What counterbalances this drawback is firstly the
possibility of facilitating the data analysis. For instance, in the HCI study, MCA output
is complicated even with only 20 experts, 33 variables and 91 modalities.

A second advantage likely to counterbalance the information loss disadvantage
is that the analysis can become more robust. For instance, in a Carpal Tunnel Syndrome
Study (CTS) study (Loslever & Ranaivosoa, 1993), considering very few cases where
the dominant hand is neither left nor right, but both, using coding ½/½ may be better
than other usual procedures such as removing the ambidextrous individuals or
performing a random assignment to either the left or right dominant hand class.

The final but main advantage of the fuzzy coding is the possibility to be used in
many cases whenever a continuum exists or may exist between two verbal nuances.
Obviously, such a coding can be used with measurement variables (dependent
variables) as in our example but also the factors (independent variables). For instance, if
a dichotomy is often considered for novice/expert (with the membership values 0/1 and
1/0), we can have ½/½, but also ¹/³²/³³, etc.

**6. Threats to Validity**

In the analysis of the results found in the study with the experts, we considered the four
threats of validity proposed by (Wohlin et al., 2012): construct validity, internal
validity, conclusion validity, and external validity.

The *construct validity* “is concerned with the relation between theory and observation” (Wohlin et al., 2012). This threat concerns the construction of our questionnaire (see **Appendix B**) and its use for our study. First, to minimize this threat, we built the questionnaire using the original text extracted (specific goals and specific practices) from the official documentation of CMMI-DEV. The HCI categories were collected from literature and pre-validated by one of the authors. Moreover, two authors who conducted the interviews have good knowledge of CMMI-DEV; in addition, we had the official CMMI-DEV documentation on hand during the interviews. CMMI-DEV was used to structure the study considering only the engineering process areas that correspond to classical activities of software development processes, well known to software practitioners. Therefore, we consider this risk to be under control.

Threats to the *internal validity* “are influences that can affect the independent variable with respect to causality, without the researchers’ knowledge” (Wohlin et al., 2012). This threat is associated with the subjects (experts). The first group of experts was selected by convenience from the professional network of one of the authors. After that, these experts suggested other names following the pre-defined profile. The literature shows that some studies have used small samples to obtain expert feedback. For instance, (Dyba, 2000) had 11 experts who conducted his review process; and (Beecham, Hall, Britton, Cottee, & Rainer, 2005) had 20 experts who evaluate their requirements process improvement model. Therefore, we accepted this risk. Another threat is the knowledge of the experts regarding HCI categories and examples. We assumed that the experts knew all the proposed categories. To mitigate this risk, we selected only professionals that have experience (academic and/or industrial) in the HCI domain and have a Ph.D. degree in the HCI field. In addition, we decided that it was not
necessary to be familiar with CMMI-DEV, since the practices of engineering process areas are typically in system development.

The *conclusion validity* “is concerned with the relationship between the treatment and the outcome” (Wohlin et al., 2012). In this study, the conclusion threat is the relation between the HCI categories and each specific practice evaluated by experts. To reduce this risk, we decided to perform interviews individually and not using a survey. In this way, we could clarify each doubt of the experts about the objective of the evaluation, the CMMI-DEV and the HCI categories. In addition, when the experts partially agreed or disagreed with one or more propositions, they were asked to justify their opinion and include any other proposals (when necessary).

Threats to the *external validity* are “concerned with generalization” (Wohlin et al., 2012). The result could be biased if experts come from only one domain of expertise. To mitigate this risk, we interviewed experts with different expertise in HCI and with experience recognized by the HCI community (e.g., program chair or member of the program committee of HCI conferences, editor of journals and members of HCI associations). We also invited experts who are well known for working on different technologies (e.g., web applications, information systems, critical systems, table top applications, and so on). Even with these precautions, we accepted the risk of it not being possible to generalize the results.

# 7. Conclusion

This paper has investigated data analysis whenever a human component system study yields qualitative data. Fundamental to our work is (1) the use of membership values within [0, 1] instead of within {0, 1} (if it makes sense), and (2) the suggestion of the Multiple Correspondence Analysis (MCA) method due to its possibility of showing both relationships between such coded variables and factor effects. Through a representative example, in the HCI context, we have shown that the information loss effect when removing a modality can be mitigated by the possibility of making the analysis easier and more robust. This is particularly worthwhile within the prospect of analyzing a large dataset where both the multivariate aspect (e.g. with several tens of variables) and the multifactor aspect are present (the individual being one factor and the other factors may be linked to individuals and/or any other factors present in the experimental or observational design). Using membership values, which makes it possible to obtain homogeneous data in which both qualitative and quantitative scales are initially present, is a clear future direction to consider (for a preliminary statistical analysis, at least).

# Key Points

* The most discriminating variables correspond to HCI approaches to support the verification and validation phases of system development, which are fields where HCI research is well consolidated. That means that some of the experts have wide experience in these fields.

* The experts have experience in academic and industrial environment, working on different technologies (e.g., critical systems, web applications, tabletop
applications, and so on) and HCI fields (requirements, design, verification, validation), which implies different points of view.

* The percentage of variance explained by the Multiple Correspondence Analysis Main Plane (when crossing Axes 1 and 2) was about 20% higher with fuzzy coding. Such a result is often found when comparing crisp and fuzzy windowing with quantitative data.

* Fuzzy coding can be used in many cases with qualitative factors or qualitative measurement variables.

## Acknowledgments

The authors would like to acknowledge the financial support granted by CAPES – Science without Borders Program. They also thank the experts who participated in this study, and the anonymous reviewers for their constructive remarks.

## Notes on contributors

**Pierre Loslever** has a Ph.D. degree in automatics for industrial and human systems from the Université Polytechnique Hauts-de-France (France) and is a Professor in Valenciennes at the engineering school ENSIAME. He is a member of the LAMIH (Laboratory of Industrial and Human-Automation, Mechanics, and Computer Science). He primarily teaches courses in data analysis related domains, such as Signals and Systems, Statistics, Maintenance or Quality management. His research focuses mainly on human component systems studies with aspects such as ergonomics or biomechanics.

**Taisa Guidini Gonçalves** received her Ph.D. degree in computer science from the LAMIH CNRS UMR 8201 - Université Polytechnique Hauts-de-France (France) in 2017, and her M.Sc. degree in computer science (software engineering) from the Federal University of Rio de Janeiro in 2014. She specializes in Human-Computer Interaction and integration of HCI models and tools in Software Engineering models, particularly in the CMMI-DEV model.

**Káthia Marçal de Oliveira** is an associate professor at the Université Polytechnique Hauts-de-France (France) and a member of the "INTERaction and Agents" research group in the LAMIH. She has a Ph.D. in software engineering focused on quality assurance. She works on the integration of Human-Computer Interaction and software engineering issues.

**Christophe Kolski** is a Professor in computer science at the Université Polytechnique Hauts-de-France (France) and a member of the "INTERaction and Agents" research group in the LAMIH. He specializes in human-computer interaction, software engineering for interactive system design and evaluation, and adaptive and tangible user interface.
## References

Allard, P., Stokes, I. A. F., & Blanchi, J.-P. (1995). *Three-Dimensional Analysis of Human Movement*. Champaign (IL): Human Kinetics. Retrieved from https://www.amazon.com/Three-Dimensional-Analysis-Human-Movement-Allard/dp/0873226232

Anscombe, F. J. (1973). *Graphs in Statistical Analysis*. *The American Statistician*, 27(1), 17–21.
https://doi.org/10.1080/00031305.1973.10478966

Bainbridge, P. E., & Sanderson, P. (2005). Verbal protocol analysis. In J. R. Wilson & E. N. Corlett (Eds.), *Evaluation of human work: a practical ergonomics methodology* (pp. 159–184). Cambridge: CRC Press.

Bass, L., Little, R., Pellegrino, R., Reed, S., Seacord, S., Sheppard, S., & Szesur, M. (1991). *The Arch model: Seeheim revisited*. In *Proceedings of User Interface Developers Workshop*. Seeheim.

Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an expert panel to validate a requirements process improvement model. *Journal of Systems and Software*, 76(3), 251–275. http://dx.doi.org/10.1016/j.jss.2004.06.004

Beh, E. J., & Lombardo, R. (2014). *Correspondence analysis: theory, practice and new strategies* (1st ed.). UK: John Wiley & Sons.

Bell, P. M., & Crumpton, L. (1997). A fuzzy linguistic model for the prediction of carpal tunnel syndrome risks in an occupational environment. *Ergonomics*, 40(8), 790–799.
https://doi.org/10.1080/001401397187784

Benzécri, J.-P. (1988). *Qualité et quantité dans la tradition des philosophes et en analyse des données*. *Les Cahiers de l'analyse Des Données*, 13, 131–152.

Benzécri, J.-P. (1992). *Correspondence analysis handbook*. CRC Press.

Bian, Z., Ishibuchi, H., & Wang, S. (2019). Joint Learning of Spectral Clustering Structure and Fuzzy Similarity Matrix of Data. *IEEE Transactions on Fuzzy Systems*, 27(1), 31–44.
https://doi.org/10.1109/TFUZZ.2018.2856081

Borchers, J. (2001). *A Pattern Approach to Interaction Design*. New York, NY, USA: John Wiley &amp; Sons, Inc.

Bordet, J.-P. (1973). *Etude de données géophysiques - Modélisations statistiques par régression factorielle* (PhD dissertation). University of Paris VI.

Brackstone, M. (2000). Examination of the use of fuzzy sets to describe relative speed perception. *Ergonomics*, 43(4), 528–542. https://doi.org/10.1080/001401300184396

Chevene, F., Doleadec, S., & Chessel, D. (1994). A fuzzy coding approach for the analysis of long-term ecological data. *Freshwater Biology*, 31(3), 295–309. https://doi.org/10.1111/j.1365-2427.1994.tb01742.x

CMMI Product Team. (2010). *CMMI® for Development* (Version 1.3 No. CMU/SEI-2010-TR-033). Pittsburgh, PA, USA: Software Engineering Institute, Carnegie Mellon University.
Retrieved from http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=9661

Cornillon, P. A. (2012). *Statistiques avec R*. Rennes, France: Presses Universitaires de Rennes.

Coutaz, J. (1987). PAC, an Object-Oriented Model for Dialog Design. In *Proceedings of 2nd IFIP International Conference on Human-Computer Interaction (INTERACT 87)* (pp. 431–436). Stuttgart, Germany.

Dalgaard, P. (2008). *Introductory Statistics with R*. New York, NY: Springer New York.
https://doi.org/10.1007/978-0-387-79054-1
D'Urso, P., & Gil, M. Á. (2017). Fuzzy data analysis and classification: Special issue in memoriam of Professor Lotfi A. Zadeh, father of fuzzy logic. *Advances in Data Analysis and Classification, 11*(4), 645-657. https://doi.org/10.1007/s11634-017-0304-z

Dyba, T. (2000). An Instrument for Measuring the Key Factors of Success in Software Process Improvement. *Empirical Software Engineering, 5*(4), 357-390.
https://doi.org/10.1023/A:1009800404137

Fraden, J. (2004). *Handbook of Modern Sensors: physics, designs and applications*. New York: Springer-Verlag. https://doi.org/10.1007/b97321

Friendly, M., & Meyer, D. (2016). *Discrete Data Analysis With R: Visualization and Modeling Techniques for Categorical and Count Data* (Har/Psc). Boca Raton London New York: Productivity Press.

Gagolewski, M., Bartoszuk, M., & Cena, A. (2016). Genie: A new, fast, and outlier-resistant hierarchical clustering algorithm. *Information Sciences, 363*, 8-23.
https://doi.org/10.1016/j.ins.2016.05.003

Gallego, F. J. (1982). Codage flou en analyse des correspondances. *Les Cahiers de l'analyse Des Données, 7*, 413-430.

Giordani, P., & Kiers, H. A. L. (2004). Principal Component Analysis of symmetric fuzzy data. *Computational Statistics & Data Analysis, 45*(3), 519-548. https://doi.org/10.1016/S0167-9473(02)00352-3

Gonçalves, T. G. (2017). *Integration of Human-Computer Interaction Engineering issues into Software Process Capability Maturity Models* (PhD thesis). University of Valenciennes and Hainaut-Cambrésis, Valenciennes, France.

Gonçalves, T. G., Oliveira, K. M., & Kolski, C. (2018). Identifying HCI Approaches to support CMMI-DEV for Interactive System Development. *Computer Standards & Interfaces, 58*, 53-86. https://doi.org/10.1016/j.csi.2017.12.003

Gray, D. E. (2004). *Doing research in the real world*. Los Angeles: Sage Publications.

Greenacre, M. J., & Blasius, J. (Eds.). (2006). *Multiple correspondence analysis and related methods*. Boca Raton: Chapman & Hall/CRC.

Guerra, T.-M., & Loslever, P. (1993). Probabilistic sets and fuzzy reasoning to build relation between "subjective" and "objective" variables sets. *Information Sciences, 73*(1-2), 117-137.
https://doi.org/10.1016/0020-0255(93)90017-G

Guitonneau, G. G., & Roux, M. (1977). Sur la taxinomie du genre Erodium. *Les Cahiers de l'analyse Des Données, 2*, 97-113.

Howarth, S. J., Grondin, D. E., La Delfa, N. J., Cox, J., & Potvin, J. R. (2016). Working position influences the biomechanical demands on the lower back during dental hygiene. *Ergonomics, 59*(4), 545-555. https://doi.org/10.1080/00140139.2015.1077274

Huang, T.-H., Huang, M. L., Nguyen, Q. V., Zhao, L., Huang, W., & Chen, J. (2017). A Space-Filling Multidimensional Visualization (SFMDVis) for Exploratory Data Analysis. *Information Sciences, 390*, 32-53. https://doi.org/10.1016/j.ins.2015.06.031

Ingram, T. G. J., Duncan, C. A., Mansfield, A., Byrne, J. M., & McIlroy, W. E. (2016). The influence of previous experiences on participant performance during maritime simulation testing. *Theoretical Issues in Ergonomics Science, 17*(3), 324-336.
https://doi.org/10.1080/1463922X.2016.1138153

Jacko, J. A. (Ed.). (2012). *The human-computer interaction handbook: fundamentals, evolving technologies, and emerging applications* (3rd ed). Boca Raton, FL: CRC Press.
Jobson, J. D. (1999). *Applied multivariate data analysis: Volume II - categorical and multivariate methods*. New York: Springer.

Jónás, T., Tóth, Z. E., & Árva, G. (2018). Applying a fuzzy questionnaire in a peer review process. *Total Quality Management & Business Excellence*, 29(9–10), 1228–1245.
https://doi.org/10.1080/14783363.2018.1487616

Jones, M. L. H., Reed, M. P., & Chaffin, D. B. (2015). Identifying and classifying force-generation strategies for one-hand isometric force exertion tasks with bracing availability. *Theoretical Issues in Ergonomics Science*, 16(3), 326–344.
https://doi.org/10.1080/1463922X.2014.995782

Kapellusch, J. M., Garg, A., Bao, S. S., Silverstein, B. A., Burt, S. E., Dale, A.-M., ... Rempel, D. M. (2013). Pooling job physical exposure data from multiple independent studies in a consortium study of carpal tunnel syndrome. *Ergonomics*, 56(6), 1021–1037.
https://doi.org/10.1080/00140139.2013.797112

Karwowski, W., & Ayoub, M. M. (1984). Fuzzy modelling of stresses in manual lifting tasks. *Ergonomics*, 27(6), 641–649. https://doi.org/10.1080/00140138408963537

Karwowski, W., Gaweda, A., Marras, W. S., Davis, K., Zurada, J. M., & Rodrick, D. (2006). A fuzzy relational rule network modeling of electromyographical activity of trunk muscles in manual lifting based on trunk angles, moments, pelvic tilt and rotation angles. *International Journal of Industrial Ergonomics*, 36(10), 847–859.

Komem, J., & Schneider, M. (2005). On the Use of Fuzzy Logic in Data Mining. In O. Maimon & L. Rokach (Eds.), *Data Mining and Knowledge Discovery Handbook* (pp. 517–533). New York: Springer-Verlag. https://doi.org/10.1007/0-387-25465-X_23

La Delfa, N. J., Grondin, D. E., Cox, J., Potvin, J. R., & Howarth, S. J. (2017). The biomechanical demands of manual scaling on the shoulders & neck of dental hygienists. *Ergonomics*, 60(1), 127–137. https://doi.org/10.1080/00140139.2016.1171402

Laeng, B., & Teodorescu, D.-S. (2002). Eye scanpaths during visual imagery reenact those of perception of the same visual scene. *Cognitive Science*, 26(2), 207–231.
https://doi.org/10.1207/s15516709cog2602_3

Lannoy, A., & Procacia, H. (2001). *L'utilisation de jugements d'experts en sûreté de fonctionnement*. Paris: Tec&Doc.

Lenné, M. G., & Jacobs, E. E. (2016). Predicting drowsiness-related driving events: a review of recent research methods and future opportunities. *Theoretical Issues in Ergonomics Science*, 17(5–6), 533–553. https://doi.org/10.1080/1463922X.2016.1155239

Loslever, P. (1993). Analysis in man-task system behavior studies. *Behavioral Science*, 38(2), 101–113. https://doi.org/10.1002/bs.3830380203

Loslever, P. (2014). Membership Function Design for Multifactorial Multivariate Data Characterizing and Coding in Human Component System Studies. *IEEE Transactions on Fuzzy Systems*, 22(4), 904–918. https://doi.org/10.1109/TFUZZ.2013.2278410

Loslever, P., & Lepoutre, F. X. (2004). Analysis of objective and subjective data using fuzzy coding and multiple correspondence analysis: principle and example in a sitting posture study. *Theoretical Issues in Ergonomics Science*, 5(5), 425–443.
https://doi.org/10.1080/14639220412331298910

Loslever, P., Popieul, J. C., & Simon, P. (2003). From classic statistical characterization to fuzzy windowing based characterization for the exploratory analysis of miscellaneous time variables: example in the field of car driving studies. *Fuzzy Sets and Systems*, 137(2), 271–296.
https://doi.org/10.1016/S0165-0114(02)00302-0
Loslever, P., & Ranaivosoa, A. (1993). Biomechanical and epidemiological investigation of carpal tunnel syndrome at workplaces with high risk factors. *Ergonomics*, **36**(5), 537–555.
https://doi.org/10.1080/00140139308967911

Loslever, P., Simon, P., Rousseau, F., & Popieul, J. C. (2008). Using space windowing for a preliminary analysis of complex time data in human component system studies. Examples with eye-tracking in advertising and car/head movements in driving. *Information Sciences*, **178**(19), 3645–3664. https://doi.org/10.1016/j.ins.2008.05.037

Manton, K. G., Woodbury, M. A., & Tolley, H. D. (1994). *Statistical application using fuzzy sets*. New York: Wiley.

Mears, A., Roberts, J., & Forrester, S. (2018). Matching Golfers' Movement Patterns during a Golf Swing. *Applied Sciences*, **8**(12), 2452. https://doi.org/10.3390/app8122452

Meshkati, N., Hancock, P., & Rahimi, M. (1990). Techniques in mental workload.”In Evaluation of human work. In J. R. Wilson & E. N. Corlett (Eds.) (pp. 749–782). London: Taylor & Francis.

Miyake, S., Loslever, P., & Hancock, P. A. (2001). Individual differences in tracking. *Ergonomics*, **44**(12), 1056–1068. https://doi.org/10.1080/00140130110084782

Murtagh, F. (2005). *Correspondence analysis and data coding with Java and R*. Boca Raton: Chapman & Hall.

Nakamori, Y., & Ryoke, M. (2006). Treating fuzziness in subjective evaluation data. *Information Sciences*, **176**(24), 3610–3644. https://doi.org/10.1016/j.ins.2006.02.015

Parasuraman, R., Masalonis, A. J., & Hancock, P. A. (2000). Fuzzy Signal Detection Theory: Basic Postulates and Formulas for Analyzing Human and Machine Performance. *Human Factors: The Journal of the Human Factors and Ergonomics Society*, **42**(4), 636–659.
https://doi.org/10.1518/001872000779697980

Peng, L., Zhang, H., Zhang, H., & Yang, B. (2017). A fast feature weighting algorithm of data gravitation classification. *Information Sciences*, **375**, 54–78.
https://doi.org/10.1016/j.ins.2016.09.044

Peters, G. (2011). Granular Box Regression. *IEEE Transactions on Fuzzy Systems*, **19**(6), 1141–1152. https://doi.org/10.1109/TFUZZ.2011.2162416

Prytz, E. G., & Scerbo, M. W. (2015). Changes in stress and subjective workload over time following a workload transition. *Theoretical Issues in Ergonomics Science*, **16**(6), 586–605.
https://doi.org/10.1080/1463922X.2015.1084397

Saporta, G. (2011). *Probabilités, analyse des données et statistique*. Paris: Technip.

Scapin, D. L., & Bastien, J. M. C. (1997). Ergonomic criteria for evaluating the ergonomic quality of interactive systems. *Behaviour & Information Technology*, **16**(4–5), 220–231.
https://doi.org/10.1080/014492997119806

Schiro, J., Loslever, P., Gabrielli, F., & Pudlo, P. (2015). Inter and intra-individual differences in steering wheel hand positions during a simulated driving task. *Ergonomics*, **58**(3), 394–410.
https://doi.org/10.1080/00140139.2014.978899

Sheskin, D. J. (2007). *Handbook of Parametric and Nonparametric Statistical Procedures* (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.

Stanton, N., Salmon, P., Walker, G., Baber, C., & Jenkins, D. (2000). *Human factors methods: A practical guide for engineering and design*. Burlington: Ashgate Publishing.

Stevens, S. S. (1946). On the Theory of Scales of Measurement. *Science*, **103**(2684), 677–680.
https://doi.org/10.1126/science.103.2684.677
Taha, A., & Hadi, A. S. (2016). Pair-wise association measures for categorical and mixed data. *Information Sciences*, 346-347, 73-89. https://doi.org/10.1016/j.ins.2016.01.022

Theodorou, Y., Drossos, C., & Alevizos, P. (2007). Correspondence analysis with fuzzy data: The fuzzy eigenvalue problem. *Fuzzy Sets and Systems*, 158(7), 704-721.
https://doi.org/10.1016/j.fss.2006.11.011

Thole, U., Zimmermann, H.-J., & Zysno, P. (1979). On the suitability of minimum and product operators for the intersection of fuzzy sets. *Fuzzy Sets and Systems*, 2(2), 167-180.
https://doi.org/10.1016/0165-0114(79)90023-X

Tukey, J. W. (1977). *Exploratory Data Analysis*. Boston, MA: Addison-Wesley.

Visser, P. E., Krosnick, J. A., & Lavrakas, P. J. (2000). Survey research. In H. T. Reis & C. M. Judd (Eds.), *Handbook of research methods in social and personality psychology* (pp. 223-252). Cambridge: Cambridge Press.

Wilson, J. R., & Sharples, S. (2015). *Evaluation of Human Work*. CRC Press.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). *Experimentation in Software Engineering*. Springer-Berlin Heidelberg.

Xu, J., Wang, G., & Deng, W. (2016). DenPEHC: Density peak based efficient hierarchical clustering. *Information Sciences*, 373, 200-218. https://doi.org/10.1016/j.ins.2016.08.086

Young, M. S., Brookhuis, K. A., Wickens, C. D., & Hancock, P. A. (2015). State of science: mental workload in ergonomics. *Ergonomics*, 58(1), 1-17.
https://doi.org/10.1080/00140139.2014.956151
# Appendix

## Appendix A. Questionnaire description

<table><thead><tr><th>Labels of the variables</th><th>Process Area and Specific Goal (SG)</th><th>Specific Practice (SP)</th><th>HCI categories</th></tr></thead><tbody><tr><td>REM</td><td>Requirements Development</td><td>SP 1.1 Elicit Needs</td><td>Task Analysis Methods for HCI</td></tr><tr><td>REP</td><td>SG 1 Develop Customer Requirements</td><td></td><td>Prototype for HCI requirements</td></tr><tr><td>RTM</td><td></td><td>SP 1.2 Transform Stakeholder Needs into Customer Requirements</td><td>Task Analysis Methods for HCI</td></tr><tr><td>RPM</td><td>Requirements Development</td><td>SP 2.1 Establish Product and Product Component Requirement</td><td>Task Analysis Methods for HCI</td></tr><tr><td>ROO</td><td>Requirements Development</td><td>SP 3.1 Establish Operational Concepts and Scenarios</td><td>Operational Concepts and Scenarios Specification for HCI</td></tr><tr><td>RDS</td><td>SG 3 Analyze and Validate Requirements</td><td>SP 3.2 Establish a Definition of Required Functionality and Quality Attributes</td><td>Standards and Guidelines for design and documentation of HCI</td></tr><tr><td>RAM</td><td></td><td>SP 3.3 Analyze Requirements</td><td>Task Analysis Methods for HCI</td></tr><tr><td>RRT</td><td></td><td>SP 3.4 Analyze Requirements to Achieve Balance</td><td>Techniques to validate HCI requirements</td></tr><tr><td>RVP</td><td></td><td>SP 3.5 Validate Requirements</td><td>Prototype for HCI requirements</td></tr><tr><td>TDA</td><td>Technical Solution</td><td>SP 1.1 Develop Alternative Solutions and Selection Criteria</td><td>Architecture Patterns for HCI</td></tr><tr><td>TCO</td><td>SG 1 Select Product Component Solutions</td><td>SP 1.2 Select Product Component Solutions</td><td>Operational Concepts and Scenarios Specification for HCI</td></tr><tr><td>TPP</td><td>Technical Solution</td><td>SP 2.1 Design the Product or Product Component</td><td>Prototype for HCI requirements</td></tr><tr><td>TPA</td><td></td><td></td><td>Architecture Patterns for HCI</td></tr><tr><td>TPS</td><td></td><td></td><td>Standards and Guidelines for design and documentation of HCI</td></tr><tr><td>TID</td><td>Technical Solution</td><td>SP 3.1 Implement the Design</td><td>Design patterns for HCI</td></tr><tr><td>TSS</td><td>SG 3 Implement the Product Design</td><td>SP 3.2 Develop Product Support Documentation</td><td>Standards and Guidelines for design</td></tr></tbody></table>
<table><thead><tr><th>Labels of the variables</th><th>Process Area and Specific Goal (SG)</th><th>Specific Practice (SP)</th><th>HCI categories</th></tr></thead><tbody><tr><td>PEP</td><td rowspan="2">Product Integration<br>SG 1 Prepare for Product Integration</td><td rowspan="2">SP 1.1 Establish an Integration Strategy</td><td>and documentation of HCI</td></tr><tr><td>PEF</td><td>Prototype for HCI requirements</td></tr><tr><td>ASV</td><td rowspan="4">Validation<br>SG 1 Prepare for Validation</td><td rowspan="2">SP 1.1 Select Products for Validation</td><td>Functional Prototype to validate HCI</td></tr><tr><td>ASF</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>AEV</td><td rowspan="2">SP 1.2 Establish the Validation Environment</td><td>Functional Prototype to validate HCI</td></tr><tr><td>AVS</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>APV</td><td rowspan="2">Validation<br>SG 2 Validate Product or Product Components</td><td rowspan="2">SP 1.3 Establish Validation Procedures and Criteria</td><td>Standards and Guidelines for design and documentation of HCI</td></tr><tr><td>AAV</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>ESV</td><td rowspan="4">Verification<br>SG 1 Prepare for Verification</td><td rowspan="2">SP 2.1 Perform Validation</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>ESF</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>EEV</td><td rowspan="2">SP 2.2 Analyze Validation Results</td><td>Functional Prototype to validate HCI</td></tr><tr><td>EVS</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>ERR</td><td rowspan="3">Verification<br>SG 2 Perform Peer Reviews</td><td rowspan="3">SP 1.3 Establish the Verification Environment</td><td>Standards and Guidelines for design and documentation of HCI</td></tr><tr><td>ECR</td><td>Evaluation methods for HCI review</td></tr><tr><td>EDR</td><td>Evaluation methods for HCI review</td></tr><tr><td>EPV</td><td rowspan="2">Verification<br>SG 3 Verify Selected Work Products</td><td rowspan="2">SP 3.1 Perform Verification</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>EAV</td><td>Evaluation methods for HCI verification tests</td></tr><tr><td>EDR</td><td>SP 3.2 Analyze Verification Results</td><td>Evaluation methods for HCI verification tests</td></tr></tbody></table>
# Appendix B. Questionnaire for interview

University of Valenciennes and Hainaut-Cambrésis (UVHC)
Laboratory of Industrial and Human Automation control, Mechanical engineering and Computer Science
(LAMIH UMR CNRS 8201)

## Questionnaire for interview

Domaine: Methods, techniques, standards and patterns of Human-Computer Interaction Engineering

Taísa Guidini Gonçalves
Kathia Oliveira
Christophe Kolski

June 2015

## Questionnaire of interview - Methods, techniques, standards, and patterns of Human-Computer Interaction Engineering

This interview aims to validate methods, techniques, standards, and patterns of HCI Engineering identified from an exploratory study. In this study was carried out an analysis of the Software Process Capability and Maturity Model (Capability Maturity Model Integration (CMMI-DEV) from the point view of the issues of Human-Computer Interaction Engineering. Therefore, we analyzed five process areas/processes. Engineering process areas cover the development and maintenance activities that are shared across engineering disciplines. The five Engineering process areas in CMMI-DEV are as follows:

* **Requirements Development (RD)**

* **Technical Solution (TS)**

* **Product Integration (PI)**
**Validation (VAL)**
**Verification (VER)**

From this analysis, we identified ten (10) groups of methods, techniques, standards, and patterns of HCI Engineering that were associated with the different processes areas analyzed. Each process area has different Specific Goals (SG) and these goals are associated with different Specific Practices (SP). *Do you agree, partially agree or not agree with each proposition? If you partially agree or do not agree justify our answer, please.*

**Respondent information**

**Name:**

**Date:**

**Formation and Profession:**

**The working period in the HCI area:**

**CMMI Model and Engineering Process Areas**
<figure><img src="image_7.png" /><figcaption>Figure 1. CMMI Model Components (CMMI Product Team, 2010)</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 2. Engineering Process Areas (CMMI Product Team, 2010)</figcaption></figure>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td rowspan="2">Requirements Development<br>SG 1 Develop Customer<br>Requirements<br>Stakeholder needs,<br>expectations, constraints, and<br>interfaces are collected and<br>translated into customer<br>requirements.</td><td><strong>SP 1.1 Elicit Needs</strong><br>Elicit stakeholder needs,<br>expectations, constraints,<br>and interfaces for all phases<br>of the product lifecycle.</td><td><strong>Task Analysis Methods for HCI</strong><br>Examples:<br>• CTT (Concur Task Tree)<br>• K-MAD (Kernel of Model for Activity Description)<br>• HTA (Hierarchical Task Analysis)<br>• SADT (Structured Analysis and Design Technique) or<br>SADT coupled with Petri Nets<br>• GTA (Groupware Task Analysis)</td><td></td><td></td><td></td><td></td></tr><tr><td><strong>SP 1.1 Elicit Needs</strong><br>Elicit stakeholder needs,<br>expectations, constraints,<br>and interfaces for all phases<br>of the product lifecycle.</td><td><strong>Prototype for HCI requirements</strong><br>Examples:<br>• Rapid Prototyping<br>□ Offline techniques: Paper and pencil (paper sketches,<br>storyboards), Mockups, Wizard of Oz, Video prototyping<br>□ Online techniques using software tools: No interactive<br>simulations, Interactive simulations, Scripting languages</td><td></td><td></td><td></td><td></td></tr><tr><td>Requirements Development<br>SG 1 Develop Customer<br>Requirements<br>Stakeholder needs,<br>expectations, constraints, and<br>interfaces are collected and<br>translated into customer<br>requirements.</td><td><strong>SP 1.2 Transform<br>Stakeholder Needs into<br>Customer Requirements</strong><br>Transform stakeholder<br>needs, expectations,<br>constraints, and interfaces<br>into prioritized customer<br>requirements.</td><td><strong>Task Analysis Methods for HCI</strong><br>Examples:<br>• CTT (Concur Task Tree)<br>• K-MAD (Kernel of Model for Activity Description)<br>• HTA (Hierarchical Task Analysis)<br>• SADT (Structured Analysis and Design Technique) or<br>SADT coupled with Petri Nets<br>• GTA (Groupware Task Analysis)</td><td></td><td></td><td></td><td></td></tr><tr><td>Requirements Development<br>SG 2 Develop Product<br>Requirements<br>Customer requirements are<br>refined and elaborated to<br>develop product and product<br>component requirements.</td><td><strong>SP 2.1 Establish Product<br>and Product Component<br>Requirement</strong><br>Establish and maintain<br>product and product<br>component requirements,<br>which are based on the<br>customer requirements.</td><td><strong>Task Analysis Methods for HCI</strong><br>Examples:<br>• CTT (Concur Task Tree)<br>• K-MAD (Kernel of Model for Activity Description)<br>• HTA (Hierarchical Task Analysis)<br>• SADT (Structured Analysis and Design Technique) or<br>SADT coupled with Petri Nets<br>• GTA (Groupware Task Analysis)</td><td></td><td></td><td></td><td></td></tr><tr><td>Requirements Development</td><td><strong>SP 3.1 Establish</strong></td><td><strong>Operational Concepts and Scenarios Specification for HCI</strong></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td></td><td>Examples:<ul><li>Context awareness</li><li>Adapting to context</li><li>User profile</li><li>Persona</li><li>Use cases</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Requirements Development<br>SG 3 Analyze and Validate<br>Requirements<br>The requirements are analyzed and validated.</td><td>SP 3.2 Establish a<br>Definition of Required<br>Functionality and Quality<br>Attributes<br>Establish and maintain a definition of required functionality and quality attributes.</td><td>Standards and Guidelines for design and documentation of HCI<br>Examples:<ul><li>Ergonomic Criterion (Scapin & Bastien, 1997)</li><li>ISO/IEC 9126-1 (2001)</li><li>ISO 9241-11 (1998)</li><li>ISO/IEC 25000 (2014)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Requirements Development<br>SG 3 Analyze and Validate<br>Requirements<br>The requirements are analyzed and validated.</td><td>SP 3.3 Analyze<br>Requirements<br>Analyze requirements to ensure that they are necessary and sufficient.</td><td>Task Analysis Methods for HCI<br>Examples:<ul><li>CTT (Concur Task Tree)</li><li>K-MAD (Kernel of Model for Activity Description)</li><li>HTA (Hierarchical Task Analysis)</li><li>SADT (Structured Analysis and Design Technique) or SADT coupled with Petri Nets</li><li>GTA (Groupware Task Analysis)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>SP 3.4 Analyze<br>Requirements to Achieve<br>Balance<br>Analyze requirements to balance stakeholder needs and constraints.</td><td>Techniques to validate HCI requirements<br>Examples:<ul><li>Proto Task (K-MAD)</li><li>Task Model Simulator (CTT)</li><li>Focus Group to validate requirements</li></ul></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td><strong>SP 3.5 Validate Requirements</strong><br>Validate requirements to ensure the resulting product will perform as intended in the end user's environment.</td><td><strong>Prototype for HCI requirements</strong><br>Examples:<ul><li>Rapid Prototyping</li><li>Offline techniques: Paper and pencil (paper sketches, storyboards), Mockups, Wizard of Oz, Video prototyping</li><li>Online techniques using software tools: No interactive simulations, Interactive simulations, Scripting languages</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Technical Solution<br><em>SG 1 Select Product Component Solutions</em><br>Product or product component solutions are selected from alternative solutions.</td><td><strong>SP 1.1 Develop Alternative Solutions and Selection Criteria</strong><br>Develop alternative solutions and selection criteria.</td><td><em>Architecture Patterns for HCI</em><br>Examples:<ul><li>MVC (Model-View-Controller) Model (Goldberg, 1983)</li><li>PAC (Presentation-Abstraction-Control) Model (Coutaz, 1987)</li><li>Arch Model (Bass et al., 1991)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>SP 1.2 Select Product Component Solutions</strong><br>Select the product component solutions based on selection criteria.</td><td><strong>Operational Concepts and Scenarios Specification for HCI</strong><br>Examples:<ul><li>Context awareness</li><li>Adapting to context</li><li>User profile</li><li>Persona</li><li>Use cases</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Technical Solution</td><td><strong>SP 2.1 Design the Product</strong></td><td><strong>Prototype for HCI requirements</strong></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td></td><td>Examples:<ul><li>Rapid Prototyping</li><li>Offline techniques: Paper and pencil (paper sketches, storyboards), Mockups, Wizard of Oz, Video prototyping</li><li>Online techniques using software tools: No interactive simulations, Interactive simulations, Scripting languages</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 2.1 Design the Product or Product Component</strong><br>Develop a design for the product or product component.</td><td><strong>Architecture Patterns for HCI</strong><br>Examples:<ul><li>MVC (Model-View-Controller) Model (Goldberg, 1983)</li><li>PAC (Presentation-Abstraction-Control) Model (Coutaz, 1987)</li><li>Arch Model (Bass et al., 1991)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 2.1 Design the Product or Product Component</strong><br>Develop a design for the product or product component.</td><td><strong>Standards and Guidelines for design and documentation of HCI</strong><br>Examples:<ul><li>Ergonomic Criterion (Scapin & Bastien, 1997)</li><li>ISO/IEC 9126-1 (2001)</li><li>ISO 9241-11 (1998)</li><li>ISO/IEC 25000 (2014)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Technical Solution<br><i>SG 3 Implement the Product</i></td><td><strong>SP 3.1 Implement the Design</strong></td><td><strong>Design patterns for HCI</strong></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td rowspan="3"></td><td rowspan="3">SP 3.2 Develop Product Support Documentation<br>Develop and maintain the end-use documentation.</td><td>Examples:<ul><li>A Pattern Approach to Interaction Design (Borchers, 2001)</li><li>Pattern Languages in Interaction Design: Structure and Organization (van Welie and van der Veer, 2003)</li><li>Designing interfaces (Tidwell, 2010)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Standards and Guidelines for design and documentation of HCI<br>Examples:<ul><li>Ergonomic Criterion (Scapin & Bastien, 1997)</li><li>ISO/IEC 9126-1 (2001)</li><li>ISO 9241-11 (1998)</li><li>ISO/IEC 25000 (2014)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>SP 1.1 Establish an Integration Strategy<br>Establish and maintain a product integration strategy.</td><td>Prototype for HCI requirements<br>Examples:<ul><li>Rapid Prototyping<ul><li>Offline techniques: Paper and pencil (paper sketches, storyboards), Mockups, Wizard of Oz, Video prototyping</li><li>Online techniques using software tools: No interactive simulations, Interactive simulations, Scripting languages</li></ul></li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Product Integration<br>SG 1 Prepare for Product Integration<br>Preparation for product integration is conducted.</td><td>SP 1.1 Establish an Integration Strategy<br>Establish and maintain a product integration strategy.</td><td>Functional Prototype to validate HCI<br>Examples:<ul><li>Iterative and Evolutionary Prototypes<ul><li>User interface toolkits</li><li>User interface builders</li><li>User interface development environments</li></ul></li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>SP 1.1 Select Products for</td><td>Evaluation methods for HCI verification tests</td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td></td><td>Examples:<ul><li>Usability tests<ul><li>☐ Exploratory tests</li><li>☐ Assessment tests</li><li>☐ Validation or verification tests</li><li>☐ Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.1 Select Products for Validation</strong><br>Select products and product components to be validated and validation methods to be used.</td><td><strong>Functional Prototype to validate HCI</strong><br>Examples:<ul><li>Iterative and Evolutionary Prototypes<ul><li>☐ User interface toolkits</li><li>☐ User interface builders</li><li>☐ User interface development environments</li></ul></li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.2 Establish the Validation Environment</strong><br>Establish and maintain the environment needed to support validation.</td><td><strong>Evaluation methods for HCI verification tests</strong><br>Examples:<ul><li>Usability tests<ul><li>☐ Exploratory tests</li><li>☐ Assessment tests</li><li>☐ Validation or verification tests</li><li>☐ Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.3 Establish Validation Procedures and Criteria</strong><br>Establish and maintain procedures and criteria for validation.</td><td><strong>Standards and Guidelines for design and documentation of HCI</strong><br>Examples:<ul><li>Ergonomic Criterion (Scapin & Bastien, 1997)</li><li>ISO/IEC 9126-1 (2001)</li><li>ISO 9241-11 (1998)</li><li>ISO/IEC 25000 (2014)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Validation<br>SG 2 Validate Product or</td><td><strong>SP 2.1 Perform Validation</strong><br>Perform validation on</td><td><strong>Evaluation methods for HCI verification tests</strong></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td></td><td>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>SP 2.2 Analyze Validation Results<br>Analyze results of validation activities.</td><td>Evaluation methods for HCI verification tests<br>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Verification</td><td>SP 1.1 Select Work</td><td>Evaluation methods for HCI verification tests</td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td></td><td></td><td>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.1 Select Work Products for Verification</strong><br>Select work products to be verified and verification methods to be used.</td><td><strong>Functional Prototype to validate HCI</strong><br>Examples:<ul><li>Iterative and Evolutionary Prototypes<ul><li>User interface toolkits</li><li>User interface builders</li><li>User interface development environments</li></ul></li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.2 Establish the Verification Environment</strong><br>Establish and maintain the environment needed to support verification.</td><td><strong>Evaluation methods for HCI verification tests</strong><br>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td><strong>SP 1.3 Establish Verification Procedures and Criteria</strong><br>Establish and maintain verification procedures and criteria for the selected work products.</td><td><strong>Standards and Guidelines for design and documentation of HCI</strong><br>Examples:<ul><li>Ergonomic Criterion (Scapin & Bastien, 1997)</li><li>ISO/IEC 9126-1 (2001)</li><li>ISO 9241-11 (1998)</li><li>ISO/IEC 25000 (2014)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Verification<br><em>SG 2 Perform Peer Reviews</em><br>Peer reviews are performed on</td><td><strong>SP 2.1 Prepare for Peer Reviews</strong><br>Prepare for peer reviews of</td><td><strong>Evaluation methods for HCI review</strong></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><thead><tr><th rowspan="2">Process Area and Specific Goal (SG)</th><th rowspan="2">Specific Practice (SP)</th><th rowspan="2">Methods, techniques, standards, and patterns of HCI</th><th colspan="3">Answer</th><th rowspan="2">Justification</th></tr><tr><th>I agree</th><th>I partially agree</th><th>I don't agree</th></tr></thead><tbody><tr><td rowspan="4"></td><td rowspan="2">SP 2.2 Conduct Peer Reviews<br>Conduct peer reviews of selected work products and identify issues resulting from these reviews.</td><td>Examples:<ul><li>Heuristic evaluation</li><li>Cognitive walkthrough</li><li>Groupware walkthrough</li></ul><strong>Evaluation methods for HCI review</strong><br>Examples:<ul><li>Heuristic evaluation</li><li>Cognitive walkthrough</li><li>Groupware walkthrough</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>SP 2.3 Analyze Peer Review Data<br>Analyze data about the preparation, conduct, and results of the peer reviews.</td><td>Examples:<ul><li>Heuristic evaluation</li><li>Cognitive walkthrough</li><li>Groupware walkthrough</li></ul><strong>Evaluation methods for HCI review</strong><br>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td>Verification<br>SG 3 Verify Selected Work Products<br>Selected work products are verified against their specified requirements.</td><td>SP 3.1 Perform Verification<br>Perform verification on selected work products.</td><td>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul><strong>Evaluation methods for HCI verification tests</strong><br>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>SP 3.2 Analyze Verification Results<br>Analyze results of all verification activities.</td><td>Examples:<ul><li>Usability tests<ul><li>Exploratory tests</li><li>Assessment tests</li><li>Validation or verification tests</li><li>Comparison tests</li></ul></li><li>Validation by HCI expert(s)</li></ul></td><td></td><td></td><td></td><td></td></tr></tbody></table>

Other suggestions: