GUI Usability Development Tool with Dynamic Kansei Labeling--Kansei Evaluation Method for Understanding the UI Process-Katsuji MIYAZAWA*, Yuko KOMAMIYA**,Pieter Jan STAPPERS***,
Toshimasa YAMANAKA***** Graduate School of Art and Design, University of Tsukuba 1-1-1 Tennodai Tsukuba-shi Ibaraki
305-8574 JAPAN, katsujics@mac.com** TOSHIBA TEC CORPORATION 1-1 Kanda Nishiki-cho Tokyo 101-8442 JAPAN,
yuko_komamiya@toshibatec.co.jp*** Faculty of Industrial Design Engineering, Delft University of Technology,
Landberghstraat 15, NL-2628 CE Delft, the Netherlands
p.j.stappers@io.tudelft.nl**** Institute of Art & Design, University of Tsukuba 1-1-1 Tennodai Tsukuba Ibaraki 305-8574
JAPAN, tyam@geijutsu.tsukuba.ac.jp

Abstract: As for GUI usability evaluation, questionnaire form and protocol analysis that analyze users’
consciousness and unconsciousness are mainstream. As one of techniques of protocol analysis, development oflogging tool is very much done and research of method for handling statistical data that acquired with logging tool isalso popular. However, though logging tool could catch user's behavior as quantitative data, it couldn’t catch kansei
as qualitative data. In other words, even if a result (quantitative) data could be caught, a user's experience (qualitative)
data couldn't be caught. It is clear that an experience is important in the understanding process.
In this research, we develop an application for compressing protocol. Firstly, we classified user’s behavior from user
test movies. Consequently, a user's behavior has been classified into 13 kinds. 13 kinds of items were Stuck up,
Confidence, Confirm, Watch preview, notice own mistake, label following search in pop-up menu, trial and error
understanding, learned effect, select tab, confusion, comparison these behaviors were used as the kansei label.
Observer stuck it on user behavior context in movies and context was captured with kansei label. Next, user's
behaviors with kansei label were classified into each label. Then label was structuralized using KJ method and user's
protocol was indicated as a map.
In the result, since an observer's awareness of the issues are expressed as label, adding label and structuralized are an
act that creates an observer's mental model rather than user’s mental model. In order to express a user's mental model
more clearly, it is necessary to research on the usage of the label.
Key woods: usability, kansei evaluation, understanding process

1.Introduction
We carried out MFP’s* GUI** usability test in Germany, France, United States of America, and Japan lastyear. After these user tests, although we evaluated these user test movie with protocol analysis method, we
found some problem in protocol analysis method and evaluation. These problems can be summarized as follow:
・ Result of evaluation was lacking in a unified view, because we carried out user test in four countries andobserver who evaluate user test was different in each country. Therefore, it is doubtful whether results ofevaluation are reliable or not.
・ Importance of problem which was distilled from user test movie was not clear.
・ We cannot classify user model. (User model means structurized user behavior.)
・ We need a lot of time to evaluate user test movie with protocol analysis method.

・ Observer cannot tell problem to designer well, because a designer cannot realize a problem, unless he or she
see prolonged user test movie.
In this research, we try to develop a new analysis method that could catch user behavior efficiently and assist
the observer in creating user model to solve these problems. [1], [2] For that reason, we direct our attention tokansei behavior, because we thought that kansei is concerned with UI understanding process.
Understanding process is one of function of brain. Function of brain can divide into logical and illogical.
Kansei is a part of illogical, and it is surely concerned when user is going to understand GUI. We distill kansei
data from user test movie with kansei labeling method and application that we made. Then we generate acompressed description of the user behavior.
UI understanding process was defined by compressing and structurizing user behavior. By sharing result datawith designers, it becomes subject of discussion, and they can have problems as common recognition. Handlingboth of quantitative data and kansei data at the same time also raises efficiency of the protocol analysis.
Figure.1 shows framework of this research. First, we carry out user test experiment and record experiment withDV camera. Then, observer analyzes user test movies with application that we made. With this application, we areable to capture users’ behavior from user test movie. Also, it’s able to analyze these data with DEMATEL method.
Result data can be shared among designers and developers, and becomes subject of discussion to prove GUI.
*Multi Function Peripherals, **Graphic User Interface

Figure.1 Framework of research

2.ProcessThis research has three basic phases; developing kansei-labeling method for compress user behavior,developing application that equips kansei-labeling function, and case study. These three phases can besummarized as follow:
・ Kansei-labeling method: In this phase, we try to understand kansei, because we thought that kansei operateon understanding the UI process. First, we develop a method that called “kansei labeling” for extracting kansei
from user test movies. Then we apply this method for application.
・ Developing application: In this phase, we develop an application that equipped a function of kansei-labelingwith macromedia director. Then we evaluated user test movies.
・ Case study: In this phase, we use application, and analyze result data.
3. Kansei-labeling method
3.1. About Kansei
Kansei is surely a function of brain. Also, brain has two functions, logical and illogical. As kansei is often usedas emotion, passion, and feeling, we interpret that kansei exists in those compound domains. (Figure.2)
Human being uses various information in various environments. However, they don’t always have information
for understanding these phenomenon. They don’t always prospect for further action. Also, they always cannot deal
with problem correctly. Ironically, they don’t grasp completely even about the structure of the processing in manycases. They also have vague judgment standard. We considered that kansei has been ability to get through thesesituations with vague and incomplete judgment standard. Therefore, we direct our attention to kansei in users’ UI
understanding process and UI learning behavior.

Figure.2 Function s of brain.

3.2.Kansei in UI understanding process
Although we have written that kansei exists in illogical part on former section, the question remains whether
we deal with the kansei.
In UI understanding process, we can see some behavior that are Stuck up, Confidence, Confirm, Watch
preview, notice own mistake, and so on. Also, these actions are fed buck as user experience. We consider thatkansei exists and concerns user experience in UI understanding process. In this research, we tried to show thestructure of UI understanding process by structurizing kansei in each experience.

3.3. Make kansei label
Kansei label is a unit that observer picked up user’s behavior in each context of user test movie and added
additional information.
First, we had written out key word and comment about user behavior while observing user test movie. Then,
we had classified these key word and comment tag with KJ method*. After classified tag, we determined higher
rank concept of the group, and we call these kansei-label.
*KJ method is a classifying method that was created by Jiro Kawakita.

Figure.3. Label with comment for KJ method.

Figure.4. Classified labels with KJ method

4.Developping application for kansei labeling
4.1.Framework of application
We developed an application that equipped a function of kansei-labeling with macromedia director. The
specification of application is as follows.
・ Setup labels: Operator creates the labels that were made by KJ method in advance. Then operator enters theselabels into the application. Label and functions are set up at setup window. Capture mode, setup label mode,thumbnail information mode, structure model mode, and analysis mode are chosen with function button at thebottom part of the window.
・ Capture user test movie with label: Operator captured user’s behavior based on these label. Also, time data
are added to captured movie. Operator clicked movie, then movie was captured. Captured movie are classified
and accumulated in each label with thumbnails. [Figure.5, 6]
・ Replay and add additional information on captured movies: Operator can replay classified movie by clicking
each pictures. Operator also can add an additional comment and key words about user’s behavior. [Figure.7]

Figure.5.Movie Screen.

Figure.7.Function screen.

Figure.6.Captured movies.

4.2.Function of application
In this application, operator can capture users’ behavior with kansei label that was made in advance.
Application captures before two seconds, and after two seconds from the moment the operator clicked. Then,
captured movies are accumulated as picture at once, and accumulated movies are displayed every label with time.
When operator clicks these picture, captured movie is replay, and operator can add additional comment andkeyword on context of movie. Compressed protocol that is made by these methods can watch user behavior every
categories. If operator wants to watch stuck behavior of users, operator can watch only stuck behavior of usersthat application captured. As Application captures before two seconds, and after two seconds from the moment theoperator clicked, it becomes a clue to find cause of users’ behaviors.
When a designer watches at the compressed protocol data, it becomes possible to find the key point, without
watching prolonged user movie data. By comparing the same behavior, an observer also can deepen understandingto UI structure.
For finding a cause of user’s behavior and user model, we use DEMATEL analysis method. DEMATEL
analysis method is a one of method for analyzing structure. By the questionnaire form, special knowledge iscollected and structure of an object problem is clarified in DEMATEL method. It can be clarified the degree of
influence and relation of each element by describing relation between each elements which constitutes an object inmatrix
5. Case study
5.1.Capture users’ behavior with application
In this chapter, we consider validity of application. We use user test movie that recorded at United States forcase study. User's tests in the United States had five persons. They were one woman and four men. In this user test,
we give them task of printer driver operation, and had it done.
First, we picked up user behavior to make the kansei label, and classified these with KJ method. Table.1 is a
result of KJ method.
Then, we captured users’ behaviors in user test movie and stuck on these kansei labels with our application.
Result of capture the movie is as follow. We stuck these kansei labels on user test movie and capture. By using
application, we can collect 342 user behaviors. Detail is shown in a table.2.
Table.1.Kansei labels with KJ method
Stuck

Confidence

Confirm

Watch preview

Label following

Search in pop-up menu

Trial and error

Understanding

Learned effect

Select tab

ConfusionComparison

Notice own mistake

Table.2.Result of capture

Label following
Select tab
Search in pop-up menu
Trial and error
Confusion
Watch preview
Notice own mistake
Confidence
Confirm
Comparison
Stuck
Understanding
Learned effect

Count
89
41
40
35
26
24
17
15
14
10
10
10
6

%
27.4
12.6
12.3
10.8
8
7.4
5.2
4.6
4.3
3
3
3
1.8

Achieve goal5

1.5

Total Case

324

100

5.2. DEMATEL analysis
For DEMATEL analysis, we use kansei labels as element of matrix and number of times on which kansei label
was stuck as a value of a relation. (Table.3) About the number of times that stuck the kansei label, we counted
based on the order of stacking the label. Therefore, when context of users’ behavior moved from “Labelfollowing” to “Search in pop-up menu”, we counted relation between “Label following” and “Search in pop-up
menu” as one time. Figure.10 is a result of DEMATEL analysis. [3],[4],[5]:Figure.10. Result of DEMATEL

From result of DEMATEL, vertical axis expresses order of users’ behaviors. High degree expresses user
behavior that is able to watch at forepart, and lower degree expresses user behavior that is able to watch at latterpart. Since previous behavior lead to following behavior, vertical axis shows causality. Horizontal axis expresses
relation between each label. Since it is proportional to the number of labels, horizontal axis shows frequency ofusers’ behavior.
Result of DEMATEL can be summarized as follows:・ “Label following” behavior is most important action in UI understanding.
・ We expected that watch preview is important function to understanding this UI, but it is not so important.
・ Tab structure is easily accepted in a user.
・ In understanding UI process, user tends to select more confident labels.
・ Problem of UI is notation of labels.
“Label following” behavior is the highest frequency. From this result, it turns out that users tend to have paid
attention to label of UI, when they are going to understand this UI. This behavior called label following strategy,
and we can usually watch it in display-based interaction. [6] Since the frequency of a “watch preview” behavior isnot so high, it turns out that users have paid attention to the label rather than preview picture. “Select tab” and“Search in pop-up menu” are elements that are high degree of causality, and “trial and error” is element that is low
degree of causality. Therefore, degree of“select tab” shows that user tends to understand UI form tab structure,and it shows that an understanding of tab structure has permeated users. From this result, it turn out that user
behavior changes from “select tab” to “Label following”. Consequently, since “stack” occurs from these actions, itturns out that the problem of UI is notation of labels. As degree of causality of “confidence” is higher than “trial
and error”, it turn out that users tend to select more confidential labels.

6.Discussion
In this research, we tried to discover UI’s problems by dividing user behavior into each kinds with kansei
labels and replaying these behavior with application. By watching same kind users’ behavior in a lump, we could
find some problems from user test movie. Since the same operator uses this application, we have got unified resultof analysis. About the analysis of the user behavior with the DEMATEL method, it can change quantitive data into
qualitative data by expressing relationship between the frequency and the causality of user behavior as a map.
Consequently, we were able to express analysis result more clearly. Also, we were able to understand the cause of
UI’s problem by grasping the causality of user behavior. It is easy to communicate information with others byexpressing user behavior as a map.
7.Conclusion
Result of case study shows that our application improve problems of protocol analysis. Especially, sinceobserver can use same format for analyzing user test movies, result of evaluation have a unified view. Kansei label
was also efficient for making a user model.
Although we used our application and method for printer driver, we want to apply our application and method
for other cases. This research is the first step of the method to analyze UI understanding behavior, and we will also
develop our application and method in the following directions.
・ Developing application and method that can apply for other cases.
・ Developing method and application to acquire kansei data more efficiently.
Reference
1. Oshige Miyuki: Lingo super manual, 2002
2. Ito Noriyuki: Director8 technical note, 2001
3. Aoyama Koji, Ando Kimihide, and Kurosu Masaaki: Developmental process of the mental model in usability
testing situation, 1999
4. Kazuo Sugiyama, Katsuo Inoue: Introduction to research and analysis with Excel, 1996
5. Midori Fukuya: Evaluating web usability with state transition diagram and log analysis, 1999
6. Muneo Kitajima: A Comprehension Based Model of Deliberate Search, 1999