A PROPOSAL OF USABILITY EVALUATIONMETHOD BY ROUGH SET THEORY

Katsuo Inoue¹, Masayuki Sakai2 and Yusuke Kinoshita3¹Department of Kansei Design, Hiroshima International University
2 School of Design, Sapporo City University

3 Graduate School, Kyushu Institute of Technology

ABSTRACT:We herein propose the usability method as a means of extracting the advantages anddisadvantages of an interface design. The proposed method is characterized by the combinationof the rough set theory, which is a type of quantitative method, and the conventional qualitative
method. In addition, we considered and verified the usefulness of the proposed method through acase study involving a digital music player. In the case study, we first calculated the evaluation
ranking of five digital music players by the AHP based on rough approximations, based on which
a non-transitive relation was calculated. Samples of five digital music players were divided into
two groups of high rank and low rank based on the results of evaluation. We then conducted atask analysis and a protocol analysis. As a result, we extracted the advantages and1disadvantages of the digital music players reported for each group. Furthermore, we calculatedrough sets using the table of cognitive parts of the Graphical User Interfaces of the digital music
players and the decision class of two groups in order to determine whether the GUI is ease to use.
Finally, combinations of cognitive advantages and disadvantages were extracted from theanalysis results.

1. BACKGROUND AND PURPOSE
The methods of usability evaluation can be classified into two main categories: quantitative
methods and qualitative methods. Quantitative methods are used primarily in selecting the best
prototype from a number of prototypes. In contrast, qualitative methods are generally used toinvestigate problems related to interface design. However, since qualitative data is not always
objective, methods of usability evaluation that combine qualitative and quantitative methods are
expected. Qualitative methods improve the user interface by identifying problems. However,methods that can determine both the advantages and disadvantages of a user interface design
are desired.
The present paper proposes a new usability evaluation method by which both the advantages anddisadvantages of a user interface design can be determined. The proposed method ischaracterized by combining the rough set theory (Mori, Tanaka and Inoue 2004), which is a typeof quantitative method, and the conventional qualitative method. We considered and verified theusefulness of the proposed method through a case study involving a digital music player.

2. EXPERIMENTIn an experiment using the KJ method, we first classified twelve digital music players into fivesamples (A, P, H, S, and V), each of which had a typical interface. The subjects were ten students(five male and five female). Each subject was asked to evaluate five samples with respect tovisual ease of use by means of a five-grade paired comparison. The subjects were not allowed totouch the samples.
We then conducted a usability evaluation of task analysis with the same subjects. Specifically, we
asked the subjects to evaluate each item of the three tasks shown in Table 1 for five real samples.
We recorded the experiment by video camera in order to later analyze protocol data that were2

specific to their comments on advantages and disadvantages of these samples. After these tasks
were completed, each subject evaluated 31 items in the three tasks (Task 1: fifteen items, Task 2:
two items, and Task 3: fourteen items) into five ranks in terms of ease of use.
Finally, each subject evaluated the synthetic ease of use into five ranks by paired comparison. In
order to confirm the statistical validity, we conducted an additional experiment in which twenty
students (ten males and ten females) evaluated the visual ease of use into five ranks by paired
comparison.
Table 1: Contents of Task Analysis.

■Task１（Beginner-class level）

Turn on the power supply. Confirm that the battery is in place and listen
to Song B by Singer A. Adjust to the volume to be suitable, and fast
forward to approximately 20 seconds of 3 minutes. Stop musicalplayback after confirm main part, and return to the main screen.
■Task２（Middle-class level）

Set the player to play one of your favorite songs repeatedly, and thenstop playback and return to the main screen.

■Task３（Upper-class level）

Set the player to play songs randomly. Fasting forward to find yourfavorite song and listen to this song. After listening briefly to the song,
turn off the power and return the player to its original position.

3. ANALYSIS METHOD AND CONSIDERATIONS

3. 1. CONSIDERATION OF PAIRED COMPARISON EVALUATION
We analyzed both the evaluation of paired comparison of visual ease of use before theexperiment and the synthetic usability evaluation conducted after the experiment by the AHP
method based on rough approximations (Tanaka and Sugihara 2001), and the obtained results

3are shown in Fig. 1. Specifically, we applied the AHP method using the geometric averages for
thirty subjects and ten subjects, respectively.
In addition, the non-transitive relation “if A>B and B>C then A<C” was not calculated by theconventional AHP method. However, the non-transitive relation could be calculated by the AHP
method based on rough approximations proposed by Prof. Tanaka using the linear programming
problem (Tanaka, Entani and Sugihara 2005). Therefore, we adopted this method in order to takeadvantage of the higher accuracy offered by this method, as compared to the conventional
method.
The transitive relation, excluding sample H, is shown in the evaluation of paired comparison ofvisual ease of use before the experiment, as illustrated in Fig. 1. Moreover, the width of theinterval of each sample is approximately the same. In particular, since the interval between thefirst and second samples does not overlap, each ranking has been decided. However, since thesamples after the third sample overlap, each ranking has not been decided clearly.
Visual evaluation (Beforethe experiment)
A

0.28

P

0.23

H

0.15

S
V

0.25

0.18
0.16

0.11

0.31

0.19

0.15

Usability evaluation (After the experiment)
A

0.2

P

0.25
0.2

0.2

H
0.13
S
V

0.1237
0.12

0.15
0.2

0.2

Fig. 1: Result of AHP based on Rough Approximations.
On the other hand, in the synthetic usability evaluation after the experiment, the lowest sample,
Sample V, shifts to a higher rank, and the interval is the largest, which shows that the evaluationof Sample V by ten subjects is dispersive. Moreover, Sample H, which was ranked in the middle
before the experiment, was ranked lower after the experiment, and the evaluation was barely4dispersive. Next, the rankings of Samples A and P were reversed after the experiment, as
compared with the rankings before the experiment. The interval between Samples A and P also
becomes wider. In the following chapters, we present a discussion and analysis of these factors.

3. 2. DISCUSSION OF TASK ANALYSIS
After each subject evaluated 31 items for the three tasks shown in Table 2, we calculated theaverage value of each item for the ten subjects and rearranged the items in ascending order withrespect to each sample, as shown in Fig. 2. The value “1 ~ 3” in the figure indicates the number oftasks. We also added a contour line at the average position (a) “2.5”, “3.0”, and “4.0”.
In addition, except for Sample A, the averages were omitted. The values (b) shown at the bottomof Fig. 2 are the averages of the individual average ranks (a) for the 31 items. Each taskcorresponds to Beginner-class level, Middle-class level, or Upper-class level at the top of Table 1.
In other words, each task corresponds with degree of ease to use.
A

V

P

S

H

Average (a)

22)

1.7

3

1)

1

29)

3

22)

3

32)

3

2.5

5)

2.5

1

16)

2

1)

1

16)

2

4)

1

3

32)
16)
15)

2.9
3.1
3.2

3
2
1

22)
19)
24)

3
3
3

22)
25)
23)

3
3
3

23)
29)
24)

3
3
3

3)
21)
15)

1
3
1

7)
19)
1)
10)
13)
23)
6)
29)
18)
24)
4)
14)

3.3
3.3
3.5
3.5
3.6
3.6
3.7
3.7
3.8
3.8
3.9
3.9

1
3
1
1
1
3
1
3
3
3
1
1

2)
11)
8)
5)
10)
13)
17)
14)
18)
23)
26)
27)

1
1
1
1
1
1
2
1
3
3
3
3

15)
24)
27)
30)
10)
26)
32)
16)
28)
21)
6)
4)

1
3
3
3
1
3
3
2
3
3
1
1

30)
1)
3)
15)
25)
32)
4)
26)
10)
27)
28)
18)

3
1
1
1
3
3
1
3
1
3
3
3

24)
25)
12)
17)
26)
7)
29)
27)
30)
16)
10)
20)

3
3
1
2
3
1
3
3
3
2
1
3

3)
17)
28)
30)
2)
9)
11)
26)
12)
20)
27)
21)
25)
8)
31)

4
4
4
4
4.1
4.1
4.1
4.2
4.3
4.3
4.3
4.4
4.4
4.5
4.6

1
2
3
3
1
1
1
3
1
3
3
3
3
1
3

32)
9)
15)
20)
12)
28)
25)
30)
31)
21)
3)
6)
7)
29)
4)

3
1
1
3
1
3
3
3
3
3
1
1
1
3
1

2)
3)
11)
17)
18)
19)
20)
31)
7)
12)
9)
13)
8)
14)
5)

1
1
1
2
3
3
3
3
1
1
1
1
1
1
1

2)
20)
31)
8)
19)
17)
9)
13)
14)
12)
21)
7)
11)
6)
5)

1
3
3
1
3
2
1
1
1
1
3
1
1
1
1

8)
14)
1)
5)
6)
19)
22)
2)
18)
28)
31)
13)
11)
23)
9)

1
1
1
1
1
3
3
1
3
3
3
1
1
3
1

4

Average (b)

3.76

3.73

3.59

3.2

2.73

Fig. 2: Ranking of Evaluation Criteria of Task Analysis.

5From the results of Fig. 2, the ranks for Task 3, which is the most difficult, are higher than those of
the other tasks for Samples A and V. In contrast, the ranks for Task 3 are low and those for Task
1, which is the easiest task, are higher for Sample P. For Samples H and S, the dispersion of theevaluation scores is too wide to recognize a clear tendency. In other words, the interface designconcepts of Samples H and S are less clear than the other samples.
Furthermore, the average ranks (b) for each of the 31 items for Samples H and S are much lower
than those for the other samples. As indicated by the third boundary line from the left-hand side ofFig. 2, for example, Sample H has no item with an average rank of more than “4.0”, and theaverage of “2.73” for all of the items for Sample H is the lowest among the samples. As a result,
we considered Sample P to be ranked highly after the experiment because it was designed basedon the basic concept of usability. In contrast, Samples A and V were ranked highly because forthese samples the difficult tasks were comparatively easy to understand.

3. 3. DISCUSSION OF PROTOCOL ANALYSIS
We extracted the advantages and disadvantages reported by subjects in the recorded video of theexperiment. The contents were arranged as shown in Table 2. We marked a sample
corresponding to an item with “*” and a sample only corresponding to the item with “***” especially.
In other words, the “***” symbols indicated as advantages are selling points, and the “***” symbols
indicated as disadvantages denote poorly evaluated items.
First of all, Samples A and V have more selling points because there are a number of advantages
and fewer disadvantages compared to the other samples, as suggested by the high evaluation
after the experiment, as shown in Fig. 1. In contrast, Sample H did not have any advantages, but
had several disadvantages, as indicated by the low evaluation in Fig. 1. Despite having several
advantages, Sample S had the greatest number of disadvantages. This result suggested the low
evaluation and wide dispersion after the experiment, as shown in Fig. 1.
Finally, since Sample P had no “***” symbols by evaluation items and its some marks in
disadvantages were common in the other samples, its some faults was not conspicuous.
Therefore, Sample P is considered to be evaluated highly.

6

Table 2: Advantages and Disadvantages.
Advantages
Screen

Display
ButtonA
Large screen size*

Large volume of information of the screen*

Color liquid crystal

*

P

H

S

*
*
*

*

*

***
*

GUI
Icon is comprehensible**

Buttons are easy to use because they are few

*

*
***

Large decision button
Power supply Power supply is comprehensible

***
***

Playback the tune at the same time as turning on the power supply
Function

Two or more kinds of song can be shuffled.

*

*
***

Use a shortcut during playback.

OthersEasy to use after become accustomed to the unit***

Easy to select songs

*

*

A

*
***
***

Can be used without reading the manual
Usage is similar to that of a cell phone

Disadvantages
Screen

DisplayP

H

S

Small screen

*

*

*

Small volume of information on the screen

*

*

*

Monochrome screen***

Characters are small
Character is English

Button

*

*

*

*

***
***

The button is of the slide type
No consistency in button use*

Buttons are difficult to use***

No feedback for button use

Menu
Function*****

Method of going menu panel is not clear***

Method of searching selection is not clear

***
***
*

Precision fast forwarding is difficult

Others*Location of menu panel is not clear

Only one song can be selected at a time****

Location of the selection button is not clear
The rotation button is difficult to use

V

*
***

*

Method of turning the power on is not clear
Method of turning the power off is not clear

V*

Location of operating now is not clear**

The screen structure is not clear

*

*
***

The power supply shuts down quickly

4. ANALYSIS AND CONSIDERATION OF DESIGN FACTORS

4. 1. CONSIDERATION OF VISUAL EASE TO USE
We analyze the design factor of visual ease of use for each sample shown in Fig. 1 by rough sets.
First, in preparation for calculating rough sets, we extracted the cognitive parts listed in the left-

7hand side of Table 3 by referring to the protocol (utterance) data in the usability evaluation
experiment.
In addition, we created a decision table of the decision class, in which five samples are arranged
in order of evaluation (sample A = 1, sample P = 2, sample S = 3, sample H = 4, sample V = 5).

Table 3: Two Types of Cognitive Parts.

Screen color

Main button

Main button position
Main button shape

Operation of main button

ButtonNumber of buttons
Size of decision button
Font size of button(Table above: Visual ease of use)

D2
E1
E2
F1
F2

Center

G1

Right side

G2

Round
SquareSpecialPush-button

H1
H2
H3
I1

Push-button + rotation

Slide
Stick

I2
I3
I4

Many
Few
Standard

J1
J2
K1

Small
Standard
Small

K2
L1
L2

Character

D1

Small
LengthSide
ColorBlack and whiteCharacter representation on menu GUI + character
Only character
Size of GUI
Large
Small
None
Character markJapanese
EnglishFont size on GUI
Large
Volume of information on display

Volume of

Direction of screenStandard

information

ScreenScreen size

sound

Thickness

A1
A2
A3
B1
B2
C1
C2
C3

Operation

Size

Portrait square
Oblong square
Barrel type
Standard
Small
Thick
Standard
Thin

Function

Whole shape

Main body shape

Usability evaluation (After the experiment)

Volume of information
when selecting

A1
A2
B1
B2
B3
C1
C2
D1

Standard

D2

Large
NormalSmall
One line
Two lines

E1
E2
E3
F1
F2

Four lines

F3

Six lines

F4

Operation electronic soundat each operation
Operation sound of button

Yes
No
Yes
None

G1
G2
H1
H2

Automatic playback

Yes
No
Yes

I1
I2
J1

No
Many
Few

J2
K1
K2

Yes
No
Time to turn off power automaticallyLong

L1
L2
M1

Short cut
Number of search functions
Scroll bar

OthersVisual evaluation (Before the experiment)"Hold" display on screen

Short
Yes (all buttons)
Yes(some buttons)
No

M2
N1
N2
N3

After the preparation, the decision rules of each sample, which are shown in the left-hand side ofFig. 3, were obtained by calculation of rough sets. In addition, since the CI value of each decision
rule of each sample is “1”, the CI value has been omitted from the left-hand side of Fig. 3. The CI
value is an index that shows the degree of contribution to the decision class. The decision rules
shown in the left-hand side of Fig. 3 can be used to determine the cognitive parts of the core. We8therefore surrounded these parts by square frames. For example, the core of Sample A consistsof “C3” and “J2”.
We used the decision rule analysis method to extract this important core. The result calculatedusing this analysis method for each sample is on the right-hand side of Fig. 4. The right-hand sideof Fig. 3 shows that the CI value of “C3” as a core is high in the standardization column score of
Sample A.

A
Y=1

P
Y=2

H
Y=3

S
Y=4

V
Y=5

I2

H2

A2

A3

C1 A1

C3 B1

B2 A1

I3

C2

C1 E1

C3 D1

B2 C3

C3 F1

B2 E1

C1 D2

H3

C3 H1

B2 G1

C1 E2

C3 K1

B2 J2

C1 F2

B2 L1

C1 G2

C1 D1
C1 F1

J2 B1

C1 G1
C1 I1

I4
I1 B1
L2

I1 D1

J2 D1

D2 A1

D2 B1

J2 F1

D2 C3

D2 H1

E2 B2

I1 H1

I1 F1

J2 H1

D2 E1

D2 K1

E2 F1

I1 K1

J2 K1

D2 G1

E2 K2

D2 J2

E2 B1
E2 F2

F1 B2

J1 B1 F1

F2 A1

E2 H1

F1 D2

J1 D1

F2 B2

E2 K1

F1 K2

F2 C3

E2 L1

F2 E1

J1 A1

J1 E1
J1 G1

G2 B2

J1 H1 F1

F2 G1

F2 B1

G2 F1

J1

F2 J2

F2 H1

G2 K2

J1 K1 F1

F2 K2

F2 K1

K2 A1

I1 B2

G2 B1

K2 C3

I1 C3

G2 F2

I1

J1 L1 F1
J1 B2

K2 E1

I1 D2

G2 H1

K2 G1

I1 F2

G2 K1

K2 J2

I1

G2 L1

K2 L1

I1 K2

J2

J1 K2

A P H S V
Y=1 Y=2 Y=3 Y=4 Y=5
A1
0.8
0.6
A2
0.5
A3
0.8
B1 1.2
1
0.5
B2
2
1.5
* C1
1
1.9
C2
0.8
C3 3 1D1 1.20.9
D2
1 1.2 0.4
E1
0.8
0.6
* E2
1.5 1.1
F1 1.2
1.9 1.5
F2
2 1.7
G1
0.8
0.6
* G2
1.5 1.1
H1 1.2
1
0.5
H2
0.4
H3
0.8
I1
1
2
I2
1
I3
0.5
I4
0.8
* J1
0.4 0.8 2
J2
3 1
K1 1.2
1
0.5
K2
2
1.5
L1
0.4 0.7
0.2
L2
0.8

J1 F2

CI value

Y= 2

CI value

Y= 2Standardization
column score

A1 C1
A1 J2
A1 K2
A1 M1
A1 I2
G1 C1
G1 M1
G1 J2
N2
E1
H1
L1

1
1
1
1
1
1
1
1
1
1
1
1

B1
D1
F3
C1 K1

0
0
0
0

C2
E2
J1 B2
J1 D2
J1 F1
J1 G1
J1 H2
J1 I1
J1 L2
K1 B2
K1 F1
M2 B2
M2 D2
M2 F1
M2 G1
M2 H2
M2 I1

1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

A1
A2
B1
B2
B3
C1
C2
D1
D2
E1
E2
E3
F1
F2
F3
F4
G1
G2
H1
H2
I1
I2
J1
J2
K1
K2
L1
L2
M1
M2
N1
N2
N3

0
0.32
0
1
0
0
0.32
0
0.59
0
0.32
0.48
0.48
0.32
0
0
0.32
1.3
0
1
0.48
1
1.3
0.27
1.1
0.48
0
1
0.27
1.3
0.32
0
1

Y= 1

（Omission）（Omission）

N3 A1
N3 G1
N3 K1

0
0
0

N3 B2
N3 E3
N3 G2
N3 H2
N3 K2
N3 L2

1
1
1
1
1
1

J1 L1

Decision rules

Fig. 3: Result of Visual Ease of Use.

Standardization column score（Total：47）（Total：46）

Fig. 4: Result of Display Screen Design.

First, we extracted the cognitive core parts of Samples A and P having high evaluation rankings in
Fig. 1, so that the cognitive core part consisted of “C3 (thin whole shape)”, “I2 (push button +rotation)”, and “J2 (few buttons)”. These three cognitive core parts were the only feature for these
two samples. Therefore, these parts were considered to be features related to high ranking.
On the other hand, we considered the cognitive core part of samples with low ranking from thesame view. “J1 (many buttons)” was a characteristic cognitive core part in Samples H, S, and V
because it was not found in samples with high ranking. In particular, since Sample V has the9highest value of “J1”, this feature worsens the first impression, and we considered that this worse
impression made ranking the lowest.
Next, although, unlike “J1”, “C1 (thick whole shape)”, “E2 (width type screen)”, and “G2 (mainbutton located on the right-hand side)” were not cognitive core parts common to all three samples,
these parts were considered to facilitate the evaluation of three samples. Since “G2” of Samples
H and S have higher values, we considered that “G” is the factor which lowered the evaluation.
Moreover, since the value of “C1” of Sample V is higher, “J1” and “C1” are considered to havemade the evaluation the lowest.
Furthermore, “A3 (body form with a width type and slack type)”, “C2 (standard whole-shaped
thickness)”, “H3 (special main button form)”, “I4 (main button with stick type)”, and “L2 (small
character size)” were features unique to Sample S. From the lower ranking of Sample S, these
cognitive core parts did not contribute to the visual ease of use.
Figure 3 shows that the number of decision rules in Sample A was extremely low. The small
number of features indicates that the features of Sample A are clear. Sample A was found to bedifferent form other samples, which may explain the why the subjects felt that the visual ease ofuse of Sample A was best from this point.

4. 2. CONSIDERATION OF EASE OF USE OF THE DISPLAY SCREEN
Next, we analyzed and considered the display screen after the experiment. We extracted
cognitive parts that contributed to ease of use of the display screen from the video record. The
usability of the cognitive part is shown on the right-hand side of Table 3. We separated thesamples into highly ranked samples (Samples P, V, and A) and poorly ranked samples (Samples
S and H) at the bottom of Fig. 1 and created a decision table with the decision class (Y) of these
two groups. The results calculated from the rough sets in the table are shown in Fig. 4.
The decision rule for which the CI value of the high evaluation group (Y = 1) was a maximum (CI= 1) is indicated by the combination of “A1 (GUI (Graphical User Interface) + character)” and “C1(Japanese character)” in Fig. 4. In decision rules in which the CI value was “0.67”, “A1” was acore of the combination, as illustrated by the square frames around the values.
Similarly, the cognitive part of “G1 (with operation electronic sound)” also serves as a core. In the
independent cognitive part, “N2 (“hold” display shown on the screen)”, “E1 (large amount of10

information on the display)”, “H1 (operation sound of a button)”, and “L1 (with a scroll bar)” were
calculated. The results indicate that the display screen design with a large amount of information
and a GUI with both Japanese characters and various kinds of electronic sounds and scroll bars
were easy to use.
On the other hand, for the group (Y = 2) of poorly evaluated samples in Fig. 4, we could not focus
on high CI values because all of the CI values were the same. Therefore, we examined the groupusing a decision rule analysis method. The analysis results are shown on the right-hand side ofFig. 4. The cognitive core parts with values greater than “0.96” for the standardization column
score, which is bordered by the square frame on the left-hand side of Fig. 4, were “B2, G2, H2, I2,
J1, K1, L2, M2, and N3”.
These results indicate that the display screen design both with a small GUI and without sounds
and other functions was difficult to use. In addition, the independent cognitive part of “A2, C2, E2,
F2, and N1” indicates that the display screen design with a menu using English characters and inwhich information presented at song selection required two lines was difficult to use.

5. CONCLUSIONSThe proposed method differs from the conventional usability evaluation method in that we canextract both the advantages and disadvantages of a user interface design based on the valuationbasis of ranking. Although the causal relationship (inverse problem) among a few samples, as in
the case study of the present study, was not able to be determined by the conventional
mathematical technique, the types of contents of the operation that contributed to ease of use
were clarified through factor analysis by rough sets.

REFERENCES:
Mori, Tanaka and Inoue (2004) Rough Sets and Kansei Engineering, Kaibundou Press, Japan.
Tanaka and Sugihara (2001) Dual Mathematical Models based on Rough Approximations, Japan Society
for Fuzzy Theory Vol.13, No.6, Japan, 592-599.
Tanaka, Entani and Sugihara (2005) Approach to Interval Evaluation in Decision Making, Japan Society for
Fuzzy Theory and Intelligent Informatics Vol.17, No.4, Japan, 406-412.

11