HAI シンポジウム 2016
Human-Agent Interaction Symposium 2016

G-13

Use Color, Sound and Vibration for An Appearance-Constrained Robot to Express
Emotions
Sichao Song1 and Seiji Yamada2
1 Department of Informatics

The Graduate University for Advanced Studies (SOKENDAI)
sichaos@nii.ac.jp
2 National Institute of Informatics
seiji@nii.ac.jp
Abstract: Many HAI and HRI researchers are now dedicating their efforts to study interactive modalities
such as facial expressions, gestures and gaze for affective communication. These modalities make such
communication between robots and individuals become natural and attractive. Although such humanoidoriented research is promising, approaches are time and cost consuming which come along with manytechnical difficulties in many research studies. In order to tackle this issue, we alternatively turn our focus
to other three simple but intuitive interaction modalities and their combinations, namely color, sound, andvibration. We conduct a structured study to evaluate the effects of such three modalities on a human`semotional perception towards our ball-shaped robot ‘Maru’. This results in a set of nine recommended
expressions that other appearance-constrained robots may readily employ.

INTRODUCTION

People respond to robots similar to how they respond topeople, especially if the robots communicate with peopleusing the same body language and other nonverbal cuesthat people use. Thus, many researchers are now
dedicating their efforts to studying interactive modalities
such as facial expressions, gestures and gaze for affective
communication. Unfortunately, these modalities are timeand cost consuming. In addition, they are also restrictedby their robot design. To tackle such limitations andmake interaction design simple, low cost, and intuitive,
we probe three modalities: color, sound, and vibration.
Previous studies have shown their impact on a person’sperception [3, 9, 8], but few works have comprehensively
evaluated the effect of these modalities in scenarios
involving affective communication with a social robot.
Thus, this leads our research questions as to how thethree modalities affect a human’s emotional perceptions
through expressions.

EXPRESSIONS DESIGN
Meta-Analysis

We first establish a set of basic expressions that represent
the mappings between each single modality of color,sound, and vibration and each emotional category of
relaxed, happy, sad, and angry. Further mixed-modality
expressions were built upon these basic expressions.
Color
According to [1], green elicits the feeling of excitement,
and red was stated as being tiring and depressive.
Similarly, [5] supports the idea that the color green

Figure 1. The circumplex model of affect.

attained the most positive emotion. [4] claims that astrong color (especially red) puts the brain into a highly
excited state and might induce a bad mood. Moreover,[6] reviews various studies on mapping emotional statesonto colors. The studies suggest that white means
peaceful, blue means depressed, and red means angry.
Sound[2] suggests that 1) when beep sounds with upwardslopes (increasing intonation) are presented from acomputer, people perceive the computer’s attitude asshowing “disagreement” regardless of the duration of thebeeps. 2) when slower downward slopes (decreasingintonation) with a longer duration are presented, thecomputer’s attitude is interpreted as “hesitation”.
Basically, “disagreement” can be interpreted as anemotion consisting of negative affection and a high levelof arousal, while “hesitation” consists of a negative

%QNQT

Table 1. Assumptions of mappings between single modality
and emotion, forming 12 basic expressions.
EmotionColor(c)Sound(s)relaxedwhite c1

happyflat beep
highly intense
green c2 sound (louder s2 vibration (lower v2
than s1 )than v4)

sadblue

c3

angry

redc4

flat beep
soundfalling beep
soundrising beep
sound

Vibration(v)
s1

s3
s4

mildly intense
vibration

low intense
vibrationhighly intense
vibration

v1

8KDTCVKQP

v3
v4

emotion with a low level of arousal. As a result, we claimthat suggestions 1) and 2) also hold if “disagreement”
and “hesitation” are replaced by the emotions “angry”
and “sad”.
Vibration[7] and [9] infer that levels of vibration intensity areassociated with different emotions. Hence, we decided toassociate relaxed with a mildly intense vibration, happy
with highly intense vibration (lower than that for angry),sad with low intense vibration, and angry with a highlyintense vibration.

5QWPF
Figure 2. Maru and its expressions made through color,sound, and vibration.

ranging from 20 to 39 years old (M = 29.09, SD = 5.90)
were recruited for the experiment. All of them were
native Japanese speakers with a certain amount ofknowledge on English.

Procedure

Our set of candidate expressions consists of basic
expressions (expressions through one single modality)
and mixed-modality expressions (expressions throughmultiple modalities). Table 1 shows 12 basic expressions.
Each of them was assigned with a unique code, forexample a white color expression was assigned with“c1”, and a falling beep sound was assigned with “s3”.
On the basis of them, we further designed 16 mixedmodality expressions. Specifically, each mixed-modality
expression was a combination of two or three basic
expressions from the same emotion category. Theirnames were decided by mixing codes of combinedmodalities followed by a number indicating which
emotion category of the basic expressions they belonged
to. For instance, cvs1 is a mixed-modality expression thatconsists of three basic expressions, c1, s1, and v1.

Maru was placed in front of the participants at a distanceof about 50 cm. Before the experiment started, theexperimenter briefly explained the purpose and setting of
the study. The participants were required to complete a
short pre-questionnaire consisting of demographic
information and questions regarding experience withrobots. After finishing the pre-questionnaire, theexperimenter started the experiment. In total, 28 trials
were conducted for each participant, where in each trial,
Maru repeatedly performed a single expression from thecandidate set. The order of expressions wascounterbalanced across participants. Each trial lasted for
10 seconds, and between each two trials, the participants
had a 20-seconds pause to select one emotion out of thefour (relaxed, happy, sad, and angry) that they believed
Maru had just expressed. After all the trials werecompleted, the experimenter ended the experiment andthanked the participant.

Maru the RobotRESULTS

Candidate Expressions

We built Maru to carry out the user experiments. FourLEDs (white, green, blue, and red) are assembled behindeach of its eyes. In addition, a speaker is used to generate
beep sound cues, and a vibration motor is attached to theinner body to produce vibration cues. An Arduino UNO
board is programmed to control the robot. Figure 2
shows Maru and how it expresses emotions through thethree modalities color, sound, and vibration and theircombinations.

EXPERIMENT

Twenty-four Japanese in total (12 males, 12 females)

Criteria for Selecting Expressions

We analyzed the candidate expressions with regard to thefour emotions separately. For the evaluation, we first
introduced two criteria to select good expressions: 1) an
expression must have a strong interpretation regarding anemotion (selection rate in the top quartile, or in otherwords above the third quartile), and 2) an expression
must be iconic, meaning that it has only one dominant
perception among the four emotions. For instance, anexpression is ambiguous and not desirable if theparticipants perceive it as more than one emotion.We
assessed the iconic-ness for each candidate expressionthat meets criteria (1). To evaluate, we used one-sample

c4

6cs1

6

cs4

6

cv1

5

cs2

5s4

5

s3

5

cvs1

4

c3

4v1

3

3

03
cs3
tests of proportions with multinomial
test. For each test
3
that was significant, we further v4vs2conducted post-hoc2
multinomial tests with Bonferroni correction
for multiple
2
cvs3
s1
1
comparisons. Because of the four emotion
categories, the1hypothesized probability that eachvs1emotion would bes2
1chosen at random regarding an expression
was
set
to
onev2
1
fourth (25%, which is the probabilityvs3of a random guess).
1
1
After a selection based on the twovs4criteria, we further
cvs4
1
discarded expressions that were notin line with the0cv3
participant’ comments (gathered fromopenquestionsinv30
the post-questionnaire).
cv40

cv1 cs2c2

cv3

v1v3

cs1 cvs1 c3

s1

cv2 vs1

s2

s4 cvs2 vs2

v2

cs3

c4

s3 cvs3 v4

cv4 cs4 vs3 vs4 cvs4

7

v3
vs1
c1

quartile 3
10

7

cvs1

6

cvs2

6

cs2

5v2

5

cv1

4

v1

4

vs2

4

cv1
cs2
c2
cv3
v1
v3
cs1
cvs1

c2
cv2
cvs2
c1
c4
cs1
cs4
cv1
cs2
s4
s3
cvs1
c3
v1
cs3
v4
vs2
cvs3
s1
vs1
s2
v2
vs3
vs4
cvs4
cv3
v3
cv4

none

0

3

cv2
vs1
s2

cv2

3

vs4

3

c1

2

c2

2cv42cs4

1v4

1c4

1

cvs4

0s4
cvs2
vs2v2
cs3
c4s3
cvs3v4
cv4
cs4
vs3
vs4
cvs4
04

81216Counting

Figure 3. Selection rates for relaxed emotion category.

Happy
No expressions met our selection criteria in the happy
category (see Figure 4). Although the four expressions
had scores above the third quartile, post-hoc tests showedthat none of the four expressions were iconic. This meets
participants’ comment (a) suggesting the difficulty ofrecognizing the happy emotion.

16912sum

Figure 4. Selection rates for happy emotion category.

c3s1

v1 cs3 v4 vs2 cvs3 s1 vs1 s2 v2 vs3 vs4 cvs4 cv3 v3 cv4

Sadinterpretation rates (IR)We recommend cvs3, vs3, cs3 and s3 for expressing a
sad emotion. They had the top five selection rates in thesad category except for s2 (see Figure 5). We conducted
multinomial tests for the top six expressions that met
criteria (1). The results of post-hoc tests with Bonferroni
correction indicats that all the six expressions also met
criteria (2).
We further discarded s2 and s1 because of theparticipants’ comments “Rising/falling sounds wereeasily recognised as angry/ sad, but flat sounds weredifficult to interpret”. All of the rest four expressions
0formedcvs3
vs3 cs3 s2 ours3 s1 recommend
cv3 c3 cs1 s4 v3 expressions
vs1 cvs1 cvs2 cs2 v2 forcv1 v1 sad,vs2 cv2 whichvs4 c1 c2 cv4 cs4 v4 c4 cvs4
consist of basic expressions that are mapped to the sademotion (c3: blue color; v3: low intense vibration; s3:
falling beep sound). This also met our assumptions.

quartile 3
10c1

c2 cv2 cvs2 c1 c4 cs1 cs4 cv1 cs2 s4 s3 cvs1 c3

quartile 3
6Relaxed
We recommend c1 for expressing a relaxed emotion. It
also has the highest selection rate in the relaxed category(see Figure 3). Although both cv1 and cs2 also met
criteria (1) and (2), they were discarded because of theparticipants’ comments “Vibration conveys negative
emotions, and especially that high intense vibration
conveys angry” and “It was difficult to recognise relaxed
Table 1emotion when sound was used”.
Modalitysum
A multinomial test indicated a significancedifference in 24
cvs3
19
c1 (p<0.01). Post-hoc tests with
Bonferroni correction
19vs3
suggest that the result for cs3
relaxed was significant
1918(significantly above 25%, p<0.001),
while results for16thes2
other three emotions are not (happy:n.s.; sad: n.s.; angry:
16s3
s1
significantly under 25%, p<0.01).
In addition, 13the 12
cv3
selection of c1 met our assumptions
as we assumed12the11c3
mapping between the color whiteand relaxed emotion
7cs1
6(see Table 1).
s4
7interpretation rates (IR)quartile 3
66

quartile 3
12

1

cvs3
vs3
cs3
s2
s3
s1
cv3
c3
cs1
s4v3
vs1
cvs1
cvs2
cs2
v2
cv1
v1
vs2
cv2
vs4
c1
c2
cv4
cs4v4
c4
cvs4
0

5

10

15

20

sum

Figure 5. Selection rates for sad emotion category.

Angry
Four expressions, cvs4, cv4, vs4, and cs4, arerecommended for expressing an angry emotion. Theseinterpretation rates (IR)able 1sum

24
23
22
20
18
1716
14
12
988
7
6
54
333222221100018

are the top five scores in the angry category except for v4
start withchoosing the expressions that have the highest
quartile 314(see Figure 6). Similar with sad category, all of the topselection
rates while meeting their hardware
12six expressions were iconic in the angry category.
configurations and that they further adjust their choices
Further, we discarded v4 and c4 because of theon the basis of the performance.
6
participants’comments “Using vibration alone wasconfusing” and “c2 11 Using multiple modalities is better
CONCLUSION
understandable cv2 8 than using single modality alone”.
In this paper, we discussed our research on how to0
v4 vs4four
cs4 c4 recommended
vs2 v2 vs1 s4 v3 v1 cv2expressions
cvs2 s1 s2 cv1 cvs1 vs3
cs2 cv3 cvs3 s3 ofc3 cs1 c1 c2 cs3
All cvs4
ofcv4theconsisted
express emotions with a social robot through threebasic expressions that were mapped to the angry emotion
modalities: color, sound, and vibration. We worked(c4: red color; v4: highly intense vibration; s4: risingthrough a structured design process to carefully select abeep sound), which again met our assumptions.
set of expressions that can well convey four emotions:relaxed, happy, sad, and angry. Our study began with aquartile 3
14
broad literature survey followed by a pre-design session.
cvs4
cv4
This resulted in a set of 28 candidate expressions thatv4
vs4
consist of one or multiple modalities. The results suggest,cs4
c4
in total, nine best expressions that can well convey
vs2
v2
relaxed, sad, and angry emotions, while no expression
vs1
can be recommended for the happy emotion. We claim
s4v3
that our findings possess good diversity, as each of thev1
cv2
four emotions was mapped onto a particular quadrant ofcvs2
s1
a valence-arousal space (the circumplex model).
s2
cv1
cvs1
vs3
cs2
cv3
cvs3
s3
c3
cs1
c1
c2
cs3

ACKNOWLEDGEMENT
This study was partially supported by JSPS KAKENHI"Cognitive Interaction Design" (No.26118005).
06

12

18

24

REFERENCE:
1.

DISCUSSION

sumWe conclude from our findings that expressions madethough the three modalities of color, sound, and vibration
are capable of conveying various emotions. In particular,
the color modality, among the three, was found to be themost important for communicating affection. Sound andvibration are considered to show a certain bias toward
some emotions. For example, falling and rising sounds
can be interpreted as sad and angry, and vibrations
convey negative emotions in general. Thus, we can offer
five suggestions as general design guidelines: 1) it isbetter to use expressions that contain color modality; 2)when expressing sadness, a falling sound is strongly
recommended; 3) when expressing anger, a rising soundand highly intense vibration are strongly recommended;
4) use multiple modalities rather than a single modality,if possible; 5) it is better not to use vibrations for positiveemotions.
For each emotion, we selected all of the expressions that
1 pick only one best expression.
met our criteria rather thanThis is because there might not be one best expressionthat holds for everyone. Instead, we offer a set of good
expressions so that variety and flexibility are promised.
For example, a practical issue could be of the various
designs of robotic platforms. A robot may not be able toperform expressions through all the three modalities,
especially vibration. Thus, for designers who wouldapply our findings to their projects, we suggest that they

2.

3.

4.
5.

6.
7.

8.
9.

http://news.mit.edu/2008/nexi-0409 — accessed
03.09.2016.
T. Komatsu. Toward making humans empathize withartificial agents by means of subtle expressions. InInternational Conference on Affective Computing andIntelligent Interaction, pages 458–465. Springer, 2005.
T. Komatsu, S. Yamada, K. Kobayashi, K. Funakoshi, andM. Nakano. Artificial subtle expressions: intuitive
notification methodology of artifacts. In Proceedings ofthe SIGCHI Conference on Human Factors in Computing
Systems, pages 1941–1944. ACM, 2010.
R. Kuller, B. Mikellides, and J. Janssens. Color, arousal,and performancea–a comparison of three experiments.
Color Research & Application, 34(2):141–152, 2009.
K. Naz and H. Helen. Color-emotion associations: Past
experience and personal preference. In AIC 2004 Colorand Paints, Interim Meeting of the International Color
Association, Proceedings, volume 5, page 31. Jose Luis
Caivano, 2004.
N. A. Nijdam. Mapping emotion to color. (Eds.): Book
Mapping emotion to colora (2009, edn.), pages 2–9, 2009.
S. U. Réhman and L. Liu. Vibrotactile emotions on amobile phone. In Signal Image Technology and Internet
Based Systems, 2008. SITIS’08. IEEE International
Conference on, pages 239–243. IEEE, 2008.
M. V. Sokolova and A. Fernández-Caballero. A review on
the role of color and light in affective computing. AppliedSciences, 5(3):275–293, 2015.
S. ur Réhman and L. Liu. ifeeling: Vibrotactile renderingof human emotions on mobile phones. In Mobile
multimedia processing, pages 1–20. Springer, 2010.