Do Audio-Visual Stimuli Change Hug Impressions?
Masahiro Shiomi1[0000-0003-4338-801X] and Norihiro Hagita1
1ATR-IRC, Kyoto, Japanm-shiomi@atr.jp

Abstract: This paper presents the effects of audio-visual stimuli on the impressions of hug interactions with a robot. In human-human interaction, perceived
gender is an essential factor to change the impressions of haptic interactions.
But in human-robot interaction, such perceived gender effects in haptic interactions have received less focus due to the difficulties of hardware limitations.
Therefore, to investigate perceived gender effects in human-robot haptic interaction, we developed a system called MetaHug that consists of a huggable robot
and a virtual reality application that uses a head-mounted-display to easilychange the audio-visual stimuli during hug interactions. We prepared both female and male virtual agents, which use identical hug animations in the virtual
application and synchronized them with the robot’s hug motions. We investigated whether our system can change the perceived genders of the robots during
hug interactions by different audio-visual stimuli, and whether their perceived
genders can change the impressions of the hugs. Experiment results with 16
participants showed that their perception of the robot genders was significantly
changed by the virtual agents. Their hug impressions were also significantly
changed by the gender of the virtual agents. These results suggest that we canchange the impressions of a robot’s hugs through audio-visual stimuli.
Keywords: Hug, Human-Robot Interaction, Virtual Reality

1

Introduction

Physical existence enables social robots to haptically interact with people throughsuch physical contact as touching and hugging. In human science literature, the positive effects of haptic interaction have already been broadly investigated, and bothmental and physical benefits have been identified [1-6]. Based on these results, researchers also investigated the following positive effects of haptic interaction between
robots and people: mental health benefits [7], improving motivation [8], decreasingstress levels [9], and encouraging prosocial behaviors [10]. These research works
showed the follow-up results of human-robot haptic interaction on haptic interaction
between people.
However, one essential factor, i.e., perceived gender, which changes the impressions of haptic interactions [11], has not been adequately investigated yet in the research field of human-robot haptic interaction. A few studies investigated gendereffects in touch interactions with an agent or a robot, but they are limited to cross-2

gender effects in tele-conference situations [12] or failed to address the robot’s genderbecause they used non-human-like robots [10]. Therefore, it remains unknown whether a robot’s perceived gender influences touching interactions with it. To design more
acceptable haptic interactions between robots and people, we must investigate theeffects of the perceived robot gender by the impressions of haptic interactions withpeople.
However, several difficulties impede the investigation of the effects of perceived
gender in human-robot touch interactions. To investigate such effects, changing botha robot’s appearance and its voice is needed. By simply altering their appearances andvoices, the perceived gender of computer-graphic-based agents can be easily changed;
on the other hand, for robots, hardware limitations complicate changing appearances.
Moreover, considering the balance of both appearance and functional designs is another difficult problem for building a robot. It would be too expensive to make multiple robots with adequate capability for natural motions with different appearances. A
few android robots might fulfill both requirements [13], but since the number of suchrobots in the world remains small, preparing so many robots with different appearances is unrealistic.
In this study, we addressed the difficulty related to changing a robot’s appearance
using a virtual reality application and investigated whether a robot’s perceived genderchanges the impressions of haptic interactions. We developed a system called MetaHug (Fig. 1) that consists of a huggable robot and a virtual reality application thatuses a head-mounted-display (HMD). We investigated whether our system canchange a robot’s perceived gender and the impressions of haptic interactions with it
due to such perceived gender changes. In this study, we addressed the following research questions:- Does a change in an agent’s appearance and voice in a virtual reality application
change the robot’s perceived gender?
- Does the perceived gender change the hug impressions?

Fig. 1 MetaHug system: a robot hugs a participant wearing a HMD that shows a computergraphics-based agent

3

2

Related works

In human science literatures, haptic interactions including touching and hugging havebeen broadly investigated. Haptic interaction provides physical and mental benefits.
For example, Grewen et al. reported that hug interactions reduce blood pressure andheart rates during stressor events [1]. Cohen et al. reported that hug interactions provide protection against the infectious virus that causes the common cold [2] andstress-buffering social support. Jakubiak et al. reported that imagined (virtual) touches
provide a better buffer against stress and pain than imagined verbal support [3].
Gender effects in haptic interaction have also been broadly investigated in the human-human interaction research field. Stier et al. investigated gender differences intouch interactions and concluded that females tend to respond more positively totouch than males [11]. Hubbard et al. investigated the effects of touch from bothsame- and opposite-gender effects and argued that being touched by the opposite
gender may be advantageous in certain situations [14]. Another study identified complex and negative effects in a nursing context where males were touched by othermales [15].
A few research works have started to investigate gender effects in haptic interactions with a robot or an agent. Suzuki et al. investigated the gender effects of mediated touch in a tele-conference situation and found that being touched by an oppositegender agent provides positive impressions. But they only investigated with maleparticipants [12]. Shiomi et al. investigated only participants’ gender differences in
hug interactions with a non-humanlike appearance robot [10]. Therefore, the gendereffects on haptic interactions with robots remain relatively uninvestigated. Also, noresearch works have untangled the difficulties related to the change of a robot’s appearance for haptic interactions using a virtual reality application. In this researchwork, we develop a novel system to solve such difficulties and investigate both samegender and opposite-gender effects in human-robot haptic interactions.

3

System overview

Figure 2 shows an overview of our system that consists of the following six components: position tracker, touch sensor, motion controller, HMD, virtual agents for audio-visual stimulus, and robot for hug stimulus.

PositontrackerTouchsensorHead position
Motioncontroller
Pressure

Audio-visual
stimulus
HMD(Virtual agents)
Robot

Fig. 2 System overview

Hug stimulus

4

3.1

Sensors: position tracker and touch sensor

We used Oculus Rift and two Oculus sensors as a position tracker because the participants also used Oculus Rift as a HMD (details are in the section 3.3). The Oculus
sensors increase the head-tracking accuracy of Oculus Rift by combining a gyro sensor, acceleration sensors, and a geomagnetic sensor.
We also placed a touch sensor, ShokacCube by Touchence, in the robot’s left armthat makes the first contact with the participants during the hug. This sensor, which
can measure the height change on the top surface of a soft material with 16 measurement points, is 36 x 20 x 30 mm and sends pressure information to the motion controller function with a maximum of 100 Hz.
3.2

Motion controller

This function autonomously controls the motions of both the virtual agents and therobot using position and pressure information from the sensors. After the participants
approach the robot, this function sends signals to both the virtual agent and the robot
to start the hug animations/behaviors. During hug interactions, if the pressure values
exceed a certain threshold, the motion controller interrupts the robot’s hug behaviors
and opens its arms for safety.
3.3

HMD

We used Oculus Rift as a HMD to show the virtual agents to the participants and to
track their head positions. Oculus Rift provides high-resolution (1080 x 1200 pixels
per eye) stereoscopic images with a 110 field of view with 90-HZ refresh rates. To
just investigate the effects of audio-visual stimuli from the virtual agents, we removed
all of the background images from the virtual reality application.
3.4

Virtual agent

Using Unity we prepared two virtual agents: a female and a male (Fig. 3). For thefemale agent’s appearance and voice, we combined a 3D model from the Futaba
Honoka Character Pack by Game Asset Studio and speech synthesis software from
VOICEROID+ Kyomachi Seika EX by AHS. For the male agent’s appearance andvoice, we combined a 3D model from the Taichi Character Pack by Game Asset Studio and speech synthesis software from VOICEROID+ Minase Kou EX by AHS.
The heights of both agents were identical. Also, their positions in the virtual reality
applications were adjusted based on the robot position to maintain the same distance
relationship between the application and the real world. We implemented eye-contact
and lip-sync behaviors for both virtual agents during idling states or speaking as wellas hug animation for both agents. The hug animation lasted one minute.
3.5

Robot

We used Moffuly, a robot that resembles a large teddy bear (Fig. 4) [10] as a huggablerobot platform. It is 200-cm tall with two elbows (1*2 DOF) and a speaker. Its 80-cm-

5long arms are adequate for reciprocating hugs. To ensure safety during hugs, we covered its frame with polypropylene cotton and used weak motors that can be easily
resisted if needed. We controlled the robot with a Raspberry Pi 2 Model B. The hug
behavior lasted one minute, during which the robot periodically patted the person on
the back.

Fig. 3 Virtual agents (left: female, right: male)

4

Experiment

4.1

Hypothesis and predictions

Fig. 4 Moffuly

In human science literature, perceived gender is essential for changing the impressions of haptic interactions [11]. Past studies reported that changes of perceived gender alter the impressions of touch stimuli from a mechanical device in a teleconference situation [12]. Therefore, in hug interactions with a robot, the changes ofaudio-visual stimuli will also influence the impressions of a robot’s hugs.
Past studies reported that opposite-gender touches are more often positively evaluated than same-gender touches [11]. Another study reported that same-gender touches
by males on males created complex and contradictory situations of acceptance, rejection, and suspicion in a nursing situation [15]. Based on these considerations, wemade the following predictions:
Prediction 1: People will report feminine hug impressions when a female agent isshown in the HMD than when a male agent is shown, even if the control method of
the robot's hug motion is the same.
Prediction 2: People will report more comfortable hugs and a greater willingness
for another one when an opposite-gender agent is shown in the HMD than when thesame-gender agent is shown.

6

4.2

Participants

Sixteen Japanese people (8 women and 8 men, whose average ages were 36.25, S.D
8.74) were paid for their participation. They had never experienced interaction with
our robot or used an HMD.
4.3

Environment

We conducted our experiment in a 5 x 2 m room in our laboratory. We attached therobot to a wall and installed two cameras and microphones on the ceiling and one
camera/microphone near it. We used the information from these sensors to analyze
the experiment.
4.4

Condition

The study had a within-participant design with the following two conditions whoseorder was counterbalanced:
Male-agent condition: Participants interacted with the robot using the HMD, whichshowed an agent with a male appearance and voice.
Female-agent condition: Participants interacted with the robot using the HMD,
which showed an agent with a female appearance and voice.
4.5

Procedure

Before the first session, the participants were given a brief description of our experiment’s purpose and procedure. In this explanation, we showed our robot and explained their interaction with it and literally demonstrated how to hug it. We addedthat the robot’s face part with which the participant faces make contact during hugging was replaceable to alleviate any sanitation concerns. We also explained how to
use the HMD and asked them to maintain the hug interaction with the robot until thevirtual agents stopped talking.
After the participants wearing the HMD approached the robot, it automaticallystarted the hug behavior and asked them to continue hugging for about one minute.
After the hug behavior ended, the participants filled out questionnaires about theirimpressions of the hug interaction. Then they wore the HMD again, hugged the robot,
and filled out questionnaires. In total, the participants did two hug interactions. The
order of showing the virtual agents was counter-balanced.
This research was approved by our institution’s ethics committee for studies involving human participants. Written, informed consent was obtained from them.
4.6

Measurement

We investigated whether the appearance and the voice of the virtual agents changed
their perceived gender and their effects on hug impressions by measuring the impressions of participants through two questionnaires. Questionnaire 1’s items are 1) per-

7

ceived agent gender and 2) degree of feminine hug impressions on a 1-to-7 point
scale, where 7 is the most feminine and 1 is the most masculine. Questionnaire 2’s
items are 3) willingness for another hug and 4) the pleasantness of the hug interaction
on a 1-to-7 point scale, where 7 is the most positive.

5

Results

5.1

Manipulation check

Figure 5 shows the questionnaire results of perceived gender. We conducted a twofactor mixed ANOVA for gender and agent factors, and the results showed significant
differences in the agent factor (F(1, 14)=51.946, p=.001, η2 = 0.788) . We found no
significant differences in the gender factor (F(1, 14)=0.615, p=.446, η2 = 0.042) orthe interaction effect (F(1, 14)=0.216, p=.649, η2 = 0.015). The perceived gender ofthe virtual agents was significantly different due to the changes of the appearancesand voices, by using our system.
***

***

*** : p = .001

7

7

6

6
5

5

4

4

3

3

2

2
1

1
Male agent

Male participants

Female agent

Female participants

Fig. 5 Perceived gender

5.2*** : p = .001

Male agent
Male participants

Female agent
Female participants

Fig. 6 Feminine hug impressions

Verification of predictions

Figure 6 show the questionnaire results of the degree of feminine hug impressions.
We conducted a two-factor mixed ANOVA for the gender and agent factors, and theresults showed significant differences in the agent factor (F(1, 14)=20.160, p=.001, η2= 0.590). But we found no significant differences in the gender factor (F(1,14)=1.577, p=.230, η2 = 0.101) or the interaction effect (F(1, 14)=0.560, p=.467, η2 =0.038). Prediction 1 was supported.
Figure 7 show the questionnaire results of the willingness for another hug. We
conducted a two-factor mixed ANOVA for the gender and agent factors, and the results showed significant differences in the agent factor (F(1, 14)=6.125, p=.020, η2 =0.331) and the interaction effect (F(1, 14)=5.091, p=.041, η2 = 0.267). However, wefound no significant differences in the gender factor (F(1, 14)=1.220, p=.288, η2 =

80.080). Multiple comparisons with the Bonferroni method revealed a significant difference for male agents: female participants > male participants, p=.047); but for female agents, no significant differences were revealed between female and male participants (p=.857). Multiple comparisons with a Bonferroni method revealed a significant difference for male participants: female agents > male agents, p=.047); but for
female participants, no significant differences were revealed between female and maleagents (p=.794)
Figure 8 show the questionnaire results of the pleasantness of the hug interactions.
We conducted a two-factor mixed ANOVA for the gender and agent factors, and theresults showed significant differences in the agent factor (F(1, 14)=10.889, p=.005, η2= 0.437) and the interaction effect (F(1, 14)=14.222, p=.002, η2 = 0.504). However,
we found no significant differences in the gender factor (F(1, 14)=1.032, p=.327, η2 =0.069). Multiple comparisons with the Bonferroni method revealed a significant difference for male agents: female participants > male participants, p=.029); but for female agents, no significant differences were revealed between female and male participants (p=.586). Multiple comparisons with the Bonferroni method revealed a significant difference for male participants: female agents > male agents, p=.001); but for
female participants, no significant differences were revealed between female and maleagents (p=.744).
Therefore, prediction 2 was partially supported; male participants reported significantly higher impressions of comfortable hugs and a willingness for another hug
when the opposite-gender agent was shown than when the same-gender agent was
shown. But questionnaire results from female participants did not significantly different among agents.
* : p <.05. ** : p <.01

7
6
5

**

6

*

* : p <.05. *** : p =.001

7

***
*

5

4

4

3

3

2

2

1

1
Male agent

Male participants

Female agent

Female participants

Fig. 7 Willingness for another hug

6Discussion

6.1

Design implications

Male agent

Male participants

Female agent

Female participants

Fig. 8 Pleasant hug feelings

Our experimental results revealed that audio-visual stimuli changed the genders perceived by the participants in hug interaction with a robot as well such impressions of

9the hug interactions as pleasantness. Moreover, the trends of same-gender and opposite-gender effects resemble human science literatures: a tendency for females to respond more positively to being touched than males [11] and negative attitudes to being touching by the same-gender in males [15]. We believe that these results provide
evidence for changing the hug impressions of a robot through audio-visual stimuli andwill contribute to scrutiny of the relationships between a robot’s hug impressions andits perceived gender because changing a virtual agent’s appearance/voice is easierthan performing the same manipulation in a robot.
One possible future work might investigate the effects of different virtual agents byfocusing on a preferred character of each participant. This current study used specificcombinations of appearances and voices in virtual agents, but if each participant could
tailor his/her own choice and type of character as an audio-visual stimulus, impressions of hug impressions would undoubtedly rise.
6.2

Limitations

Robot and agent generalities are limited since our experiment was conducted with ourdeveloped robot and specific virtual agents that have a huggable appearance and dolllike touch feelings or animation-appearances. We cannot ensure that our findings canbe applied to all huggable devices and virtual agent because appearances and feelings
about being touched are essential for hug interactions. To generalize hug impressions,
we must investigate them with different kinds of virtual agents and robots with different touch feelings. However, we believe that our setting offers essential knowledge
for researchers who are interested in hug interactions with VR applications.

7

Conclusion

We focused on the effects of audio-visual stimuli in hug interactions with a robot toinvestigate whether the perceived gender changes a hug’s impressions. We developeda system called MetaHug that consists of a huggable robot and a virtual reality application using an HMD and conducted an experiment where participants hugged a robot
by looking at both male and female virtual agents. The participants experienced different impressions to the hugs when looking at different virtual agents, even if the hug
behaviors were identical.

Acknowledgements
This research work was supported by JSPS KAKENHI Grant Numbers JP15H05322
and JP16K12505.

10References
1. Grewen, K. M., Anderson, B. J., Girdler, S. S., and Light, K. C., “Warm partner contact isrelated to lower cardiovascular reactivity,” Behavioral medicine, vol. 29, no. 3, pp. 123130 (2003).
2. Cohen, S., Janicki-Deverts, D., Turner, R. B., and Doyle, W. J., “Does hugging provide
stress-buffering social support? A study of susceptibility to upper respiratory infection andillness,” Psychological science, vol. 26, no. 2, pp. 135-147 (2015).
3. Jakubiak, B. K., and Feeney, B. C., “Keep in touch: The effects of imagined touch support
on stress and exploration,” Journal of Experimental Social Psychology, vol. 65, pp. 59-67(2016).
4. Gallace, A., and Spence, C., “The science of interpersonal touch: an overview,”
Neuroscience & Biobehavioral Reviews, vol. 34, no. 2, pp. 246-259 (2010).
5. Light, K. C., Grewen, K. M., and Amico, J. A., “More frequent partner hugs and higher
oxytocin levels are linked to lower blood pressure and heart rate in premenopausal
women,” Biological psychology, vol. 69, no. 1, pp. 5-21 (2005).
6. Field, T., “Touch for socioemotional and physical well-being: A review,” Developmental
Review, vol. 30, no. 4, pp. 367-383 (2010).
7. Yu, R., Hui, E., Lee, J., Poon, D., Ng, A., Sit, K., Ip, K., Yeung, F., Wong, M., andShibata, T., “Use of a Therapeutic, Socially Assistive Pet Robot (PARO) in Improving
Mood and Stimulating Social Interaction and Communication for People With Dementia:
Study Protocol for a Randomized Controlled Trial,” JMIR research protocols, vol. 4, no. 2(2015).
8. Shiomi, M., Nakagawa, K., Shinozawa, K., Matsumura, R., Ishiguro, H., and Hagita, N.,“Does A Robot’s Touch Encourage Human Effort?,” International Journal of Social
Robotics, pp. 1-11 (2016).
9. Sumioka, H., Nakae, A., Kanai, R., and Ishiguro, H., “Huggable communication medium
decreases cortisol levels,” Scientific Reports, vol. 3, pp. 3034 (2013).
10. Shiomi, M., Nakata, A., Kanbara, M., and Hagita, N., “A Hug from a Robot Encourages
Prosocial Behavior,” in Robot and Human Interactive Communication (RO-MAN), 2017
26th IEEE International Symposium on, pp. to appear, 2017.
11. Stier, D. S., and Hall, J. A., “Gender differences in touch: An empirical and theoreticalreview,” Journal of personality and social psychology, vol. 47, no. 2, pp. 440 (1984).
12. Suzuki, K., Yokoyama, M., Kionshita, Y., Mochizuki, T., Yamada, T., Sakurai, S.,
Narumi, T., Tanikawa, T., and Hirose, M., “Gender-Impression Modification Enhances theEffect of Mediated Social Touch Between Persons of the Same Gender,” Augmented
Human Research, vol. 1, no. 1, pp. 1-11 (2016).
13. Glas, D. F., Minato, T., Ishi, C. T., Kawahara, T., and Ishiguro, H., “Erica: The erato
intelligent conversational android,” in Robot and Human Interactive Communication (ROMAN), 2016 25th IEEE International Symposium on, pp. 22-29, 2016.
14. Ebesu Hubbard, A. S., Tsuji, A. A., Williams, C., and Seatriz, V., “Effects of Touch on
Gratuities Received in Same‐Gender and Cross‐Gender Dyads,” Journal of Applied Social
Psychology, vol. 33, no. 11, pp. 2427-2438 (2003).
15. Evans, J. A., “Cautious caregivers: gender stereotypes and the sexualization of men nurses'
touch,” Journal of advanced nursing, vol. 40, no. 4, pp. 441-448 (2002).