Investigation on Effects of Color, Sound, and Vibration on
Human’s Emotional Perception
Sichao SongDepartment of Informatics
The Graduate University for Advanced Studies(Sokendai)
2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japansichaos@nii.ac.jp

Seiji Yamada
National Institute of Informatics
The Graduate University for Advanced Studies(Sokendai)
2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japanseiji@nii.ac.jp

ABSTRACTAs robotics has advanced, research on conveying a robot’s
emotional state to a person has become a hot topic. Most
current studies are focused on interaction modalities such as
facial expressions and natural language. Although many of theresults seem to be promising, they suffer from high cost andtechnical difficulties. In this paper, we turn our attention to
three other interaction modalities: color, sound, and vibration.
Such modalities have the advantage of being simple, low cost,and intuitive. We conducted a pilot study to evaluate the effectsof the three modalities on a human’s emotional perception
towards our robot Maru. Our result indicates that humans tend
to interpret a robot’s emotion as negative (angry in particular)when vibration and sound are used, while they interpret theemotion as relaxed when only color modality is used. Inaddition, the participants showed preference towards the robot
when using all three modalities.

Figure 1. Maru and the experiment setting.

To make the interaction design simple, low cost, and intuitive,
we turn our attention to three other modalities: color, sound,and vibration. Previous studies have shown their impact ona person’s perception [2, 6, 7]. However, to the best of ourknowledge, there is no work that comprehensively evaluatedthe effect of these three modalities in a scenario involving theemotional expression of a robot.

ACM Classification Keywords

H.5.m. Information Interfaces and Presentation (e.g. HCI):
User Interfaces - Theory and methods

In this paper, we present our pilot study on this topic. We
evaluate a human’s perception towards a robot conveying its
emotions through the three modalities. A ball-shaped robot
named “Maru” was designed in order to conduct a user experiment. To be specific, we wanted to see how participantsinterpret Maru’s emotions in three cases: one modality (color),
two modalities (vibration and sound), and all three modalities.

Author Keywords

Color; Sound; Vibration; Emotional expression of a robot;
Multi-modalities; Human-Agent Interaction (HAI); HRI;HCI.
INTRODUCTION

As robotics has advanced, research on conveying a robot’s
emotional states to a person has become a hot topic. As over
80% of human communication is encoded in facial expressions and body movements [5], most of the current studies arefocused on interaction modalities, for instance, facial expressions, natural language, and body gestures. Although many ofthem have shown promising results, they suffer from high cost
and technical difficulties.

METHODS AND DESIGN

We used the circumplex model of affect to map emotions onto
a valence-arousal space. In particular, we focused on four
emotions, relaxed, happy, sad, and angry, where each emotion
can be mapped onto a different quadrant of the valence-arousalspace. We first conducted a pre-design phase to decide alimited candidate cue pool for our user experiment. This isbecause each modality can be associated with many factors,
resulting in a design space that is too large.

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact
the Owner/Author.
Copyright is held by the owner/author(s).
HAI ’16, October 04-07, 2016, Biopolis, Singapore
ACM 978-1-4503-4508-8/16/10.
http://dx.doi.org/10.1145/2974804.2980497

Pre-design SessionAt the beginning stage of the study, we wanted to prepare a candidate cue pool with a total of 12 candidate cues (3 modalities
× 4 emotions) in order to constrain our design space. Previousstudies suggest that colors can be considered as having theattribute to evoke different emotions [3, 6]. Accordingly, we225

RESULTS AND DISCUSSION

Figure 2 demonstrates our result. A two-way repeatedmeasure ANOVA on both factors for the dependent variable
was conducted. From the result, no significant difference in themain effect of combination of modalities [F(2,14) = 1.87, n.s.]
was found, indicating that hit across the four emotions does
not change much with different combinations of modalities.
However, emotion showed a significant effect [F(3,21) = 10.79,p<0.001]. A Tukey’s HSD test showed that among the fouremotions, angry (Mhit = 0.96) was recognized significantly
better than relaxed (Mhit = 0.38, p<0.001), happy (Mhit = 0.21,p<0.001), and sad (Mhit = 0.58, p<0.05). In addition, sad wasalso recognized significantly better than happy (p<0.05). No
other significant differences were found.
We also observed a significant difference in the combinationof modalities × emotion interaction effect [F(6,42) = 2.98,p<0.05]. This indicates that emotional perception changes
depending on the different combination of modalities. There isno single combination of modality that is effective for all kindsof emotions. Paired sample t-tests with Bonferroni correction
were conducted in light of the significant interaction. We seethat relaxed was recognized significantly better using only
color modality compared with using two modalities [t(7) =2.65, p<0.05] and three modalities [t(7) = 3.42, p<0.05].
However, we did not find significant differences regarding theother three emotions.

Figure 2. Modalities vs. Hit (C: color, V: vibration, S: sound)

decided to associate relaxed with white, happy with green,
sad with blue, and angry with red. [2, 1, 7] indicate thatboth sound and vibration can be informative in discriminating
between levels of valence (positive and negative) and levelsof arousal. For instance, [1] claims that sounds with upward
slopes were interpreted as "disagreement", and [2] shows thata decreasing ASE successfully conveyed "low confident" state
to the users. Thus, we decided to associate a flat beep soundwith a positive state, rising-pitch beep sound with a negativeand high arousal state, and falling-pitch beep sound with anegative and low arousal state. We also designed four levels ofvibration intensity, where a higher intensity indicates a higherlevel of arousal.

In addition, an analysis of the questionnaires indicated that theparticipants preferred when the robot used all three modalities
rather than only one or two modalities. From the result, wepropose our preliminary design guideline based on the conclusions from our result. We suggest using only color modality(white) to express relaxed. Negative emotions such as sadand angry, in order to be iconic, are strongly recommended
to be expressed through vibration and sound to. Specifically,
we suggest conveying angry using all three modalities whileconveying sadness using two modalities without color.

Designing the Robot Maru

We intentionally made Maru’s embodiment and appearance as
simple as possible while still having the attribute of anthropomorphism (see Figure. 1). The robot is made of two pieces ofhollow semi-spherical Styrofoam. Four LEDs (white, green,blue, and red) were assembled behind each eye. In addition, aspeaker is used to generate beep sound cues, and a vibration
motor is attached to the inner body to produce vibration cues.
An Arduino UNO board was programmed to control the robot.

FUTURE WORK

We will later conduct experiments on the three modalities
with larger number of participants. Particularly, we will focus
on the expression of happy and sad, and more combinationsof modalities will be examined. Further, we are considering
expanding the current system with biometric sensors such as
ECG and EDA. In addition, we are especially interested in theconcept of agent migration [4]. We will investigate how toshift the heavy amount of work on emotional communicationfrom the robot side to a third-party device such as a wearable
device through modalities such as color, vibration, and soundby applying the idea of agent migration.

EXPERIMENT

Eight Japanese (4 males and 4 females) ranging from 22 to
39 years old (M = 27.88, SD = 5.94) participated the study.
The experiment was a 3 × 5 within-subject design with two
independent variables: combination of modalities [one modality (color), two modalities (vibration, sound), three modalities(color, vibration, sound)] and emotion (relaxed, happy, sad,angry). We evaluated one dependent variable: hit [correctlyrecognized an emotion (1) and incorrectly recognized an emotion (0)]. In addition, we collected participants’ preferences
towards the three modalities via a post questionnaire using 7point Likert scales. An example question is “do you think you
recognize Maru’s emotions better when using three modalities(color, vibration, sound) rather than the other two cases (oneand two modalities)?”

ACKNOWLEDGEMENTS

This study was partially supported by JSPS KAKENHI "Cognitive Interaction Design" (No.26118005).
REFERENCES

1. Takanori Komatsu. 2005. Toward making humans
empathize with artificial agents by means of subtle
expressions. In Affective Computing and Intelligent
Interaction. Springer, 458–465.

226

2. Takanori Komatsu, Seiji Yamada, Kazuki Kobayashi,Kotaro Funakoshi, and Mikio Nakano. 2010. Artificial
subtle expressions: intuitive notification methodology ofartifacts. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems. ACM,1941–1944.
3. Niels A Nijdam. 2009. Mapping emotion to color.
)ˆ(Eds.):âĂŸBook Mapping emotion to colorâĂŹ(2009,edn.) (2009), 2–9.
4. Kohei Ogawa and Tetsuo Ono. 2008. ITACO: Effects tointeractions by relationships between humans andartifacts. In Intelligent Virtual Agents. Springer, 296–307.

5. Jocelyn Scheirer and R Picard. 2000. Affective objects.
MIT Media lab Technical Rep. 524 (2000).
6. Marina V Sokolova and Antonio Fernández-Caballero.
2015. A review on the role of color and light in affectivecomputing. Applied Sciences 5, 3 (2015), 275–293.
7. Shafiq ur Réhman and Li Liu. 2010. ifeeling: Vibrotactile
rendering of human emotions on mobile phones. In
Mobile Multimedia Processing. Springer, 1–20.

227