The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

Noname manuscript No.
(will be inserted by the editor)

Recommendation eﬀects of a social robot foradvertisement-use context in a shopping mall
Masahiro Shiomi · Kazuhiko Shinozawa · Yoshifumi Nakagawa ·Takahiro Miyashita · Toshio Sakamoto · Toshimitsu Terakubo · Hiroshi
Ishiguro · Norihiro Hagita

Received: date / Accepted: date

Abstract We developed a coupon-giving robot system for a shopping mall to explore possible applications using social robots in daily environments, particularly for advertising. The system provided information through conversations with people. The robot wassemi-autonomous, which means that it was partly controlled by a human operator, to cope with the diﬃculty of speech recognition in real environments. We
conducted two field trials to investigate two kinds of effectiveness related to recommendations: the presence ofa robot and diﬀerent conversation schemas. Although arobot can strongly attract people with its presence andinteraction, it remains unknown whether it can increase
the eﬀects of advertisements in real environments. Our
field trial results show that a small robot increased thenumber of people who printed coupons more than anormal-sized robot. The number of people who printed
coupons also increased when the robot asked visitors tofreely select from all coupon candidates or to listen to
its recommendation.
Keywords Social robot, Advertisement, Field trial

1 Introduction
One possible application for a social robot in city environments is advertising. In city environments, various information systems are used for advertisements,
Masahiro Shiomi, Kazuhiko Shinozawa, Yoshihumi Nakagawa,
Takahiro Miyashita, Hiroshi Ishiguro, and Norihiro Hagita
ATR-IRC, Kyoto, 619-0288, Japan
Tel.: +81-774-95-1432
Fax: +81-774-95-1408
E-mail: m-shiomi@atr.jp
Toshio Sakamoto, Toshimitsu Terakubo, and Hiroshi Ishiguro
Osaka Univ., Osaka, Japansuch as large displays. Unlike current systems, social
robots can interactively provide shopping informationby speech and gestures like humans [1–4]. Such intuitive interactions would be useful to advertise items toordinary people.
In fact, good salespersons already influence customer
behaviors to get them to buy through interactions in a
shop. For example, a salesperson might invite customers
to a shop and recommend items; this is a typical advertising task. In such situations, the customers must first
notice a shop or its salesperson. Moreover, the salesperson should consider the conversation schemas based onher situation. If she wants to sell a specific new item, sheshould recommend it. She also recommends additional
items to customers; thus, depending on the situation,
salespersons recommend one specific item or simultaneously recommend additional items.
Based on such human advertisements, for using asocial robot in advertisements, we must consider fourmain issues: grabbing customer attention, interrupting
their walking, attracting them to the advertising system, and recommending items. Loud sounds easily attract attention but they can be counterproductive anddiscouraging. Presenting attractive images on a large
display easily garners much attention and interrupts
walking, but people rarely approach such displays inmalls. From this context, the robot has an advantage
over the current system because its novel presence will
attract ordinary people. Moreover, its size is anotherimportant factor because a big robot attracts more people than a small one, but a small robot might convince
customers to come closer to it than a big robot. Furthermore, in interactions with such people, the conversation
schemas should be designed based on the advertising
purpose: advertising a specified item or everything in astore. The advertising eﬀects change based on how theThe final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

22 Related work
2.1 Recommendations from computers and robots

Fig. 1 Small robot advertising a shop in a mallrobot recommends items. These are considerable issues
for social robots in advertising cases.
However, most previous studies failed to focus on
the eﬀectiveness of social robots for advertising. For
example, previous studies revealed that social robots
can be used as museum guides [5, 6], receptionists [7],and peer-tutors [8]; they can also be used in the context
of mental-care for elderly people [9], in autism therapy[10, 11], in child-care [12], and in shopping malls [13,14].
Since these previous works focused on the eﬀectivenessof a developed robot system and/or social human-robot
interactions in specific actual situations, they greatly
advanced robotics technologies from the perspective of
increasing robot autonomy and the possible range of
their services. On the other hand, only a few researchworks have reported the possibilities of using a social
robot for advertising. It remains unknown how a social
robot can contribute to advertising contexts.
What kind of a robot is best suited for advertising? What kind of conversation schema is best for such
uses? This paper answers these questions by reporting
a number of field trials in a shopping mall using social
robots. In the trials, first, we investigate the eﬀectiveness of the presence of a social robot using diﬀerentsized robots. Second, we investigate what kinds of conversation schemas are useful in diﬀerent situations where
the advertising purpose is diﬀerent. In the trials, therobot had two tasks: route-guidance and coupon-giving,which is a very common advertising method in Japan(Fig. 1). The robot system was designed to control a display that can show multiple images and print coupons.
The robot is semi-autonomous due to the diﬃculty ofspeech recognition; a teleoperation technique is being
studied [15].

Advertising is a major persuasion tool in daily environments. In the field of human computer interaction, various researches have investigated the eﬀects of recommendations from computers known as persuasive technology [16,17]. Some focused on more eﬀective persuasion through interaction. For example, Powers et al.
compared a robot and a computer agent displayed on amonitor to investigate social interactions [2]. Shinozawa
et al. compared the eﬀect of persuasions in a laboratoryenvironment between a robot and a computer agent
displayed on a monitor [18]. Bainbridge et al. also compared the persuasion eﬀects between a computer agent
and a robot for various tasks [19]. These researches reported that real robots aﬀect subject decision-making
more eﬀectively than computer agents in real world environments.
However, these research works did not focus on such
diﬀerences of robots as size or appearance. Hiroi et al.
investigated the influences of robot size on impressions[20] without addressing the advertising context. Even if
they revealed the eﬀects of the robot s presence on recommendation uses, it remains unknown whether such
diﬀerences aﬀect advertisements. Thus, in this paper,
we investigate the eﬀects of such diﬀerences on advertising using diﬀerent kinds of robots.

2.2 Advertising with a robot in a real environment
A few related works reported field trials in shopping
malls and similar influences on shopping activity. For
example, Kanda et al. developed a robot system that
influenced people’s daily shopping activities [13] and asystem that enabled a social robot to anticipate pedestrian behaviors in malls [21]. They demonstrated how
a robot eﬀectively invited people to a shop by identifying browsers. Shiomi et al. conducted field trials withfour social mobile robots in a mall [14] and investigated
how they attracted more people to a shop with robots.
These research works evaluated developed systems andinteractions between robots and people; they did not
focus on the advertising context.
Moreover, a conversation schema becomes an important factor for advertising purposes. Past researchworks have argued that the talking script for such autonomous systems as a computer and a robot should
be designed carefully because the conversation schemais essential to reduce people s cognitive loads. For example, Lee et al. designed a conversation schema for
their reception robot [22] by considering psychological

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

3

status [23]. Kerstin investigated the eﬀects of conversation schemas on the perceptions of the interacting people [24]. But a conversation schema with an advertisingcontext is quite diﬀerent from such a reception context.
For example, a reception robot is basically designed torespond to such user requests as route-guidance; on theother hand, in advertising contexts, a robot not only responds to user requests but also recommends a specificitem or asks them to choose an item from candidates.
In such situations, a conversation schema must consider
not only reducing people’s cognitive loads but also increasing interest in the items being advertised. These
research works also failed to focus on the advertisingcontext.
Therefore, the eﬀects of the robot s presence on
advertising remain unanswered, as are the questions ofwhat kinds of conversation schemas are useful for advertising. In this research, we investigate the eﬀects of
conversation schemas using diﬀerent settings.
3 System configuration
To investigate the eﬀects of a robot s presence and conversation schemas on advertisements, our system provides a coupon based on recommendations. Such advertisements are very common in Japanese malls, especially at food courts, where salespersons distributecoupons. Even if customers fail to use the coupon immediately, it retains its advertising value. Such advertising leaves the final decision to use the coupon to thecustomers, who aren t compelled to buy items.
For this purpose, we designed a system that includes
a robot and a display to advertise shops in such realenvironments as malls. We used the display to simultaneously present multiple candidates with visual information, because for advertisements, such a function isneeded so that visitors can freely select from multiple
candidates. In fact, the shopping mall that cooperated
with us requested that we promote more than 15 shops;
it might be impossible to promote every shop in themall only through such verbal information as explaining all their names and coupon information.
Figure 2 shows an overview of our system configuration. The robot architecture is designed to control
the display. In this system, we involved a human operator to simulate part of the system, the Wizard-of-OZ(WOZ) method [25]. In this research, speech recognition and position estimation are conducted by a human operator. This information is sent to a behavior
selector that chooses an interactive behavior based on
pre-implemented rules called episode rules. Interactive
behavior prints coupons with the display controller andchanges the images displayed on the display.

Fig. 2 System configuration

Fig. 3 Robovie-II advertising a shop in a mall

3.1 Robot
We prepared two types of robots: small and normalsized. The major diﬀerence is their appearances. Bothhave equivalent capabilities to interact with people byengaging in conversations and printing coupons.
The small robot is based on Robovie-miniR2 (Fig.
1), which interacts with people in a home environment.
Robovie-miniR2 has a human-like appearance with twoarms (4*2 DOF), a head (3 DOF), and stands 30-cm
tall.
The normal-sized humanoid robot is based on RobovieII (Fig. 3), which is also used to interact with people.
Robovie-II has a human-like appearance with two arms
(4*2 DOF), a head (3 DOF), but it is 120-cm tall.
Both robots share other features. Speech synthesis software, XIMERA [26], was used for conversations.
They have a camera and a speaker on their heads. A
microphone was also attached to their fronts.
We implemented interactive behaviors with situatedmodules (called behaviors in this paper) and episoderules [30]. Each situated module controls the robot’s utterances, gestures, and non-verbal behaviors in reaction
to a person’s action. The pseudocode of a greeting behavior, which is described in Table 1, is executed whena person is located in front of the robot. The robot says,”Hello, how are you?” with a greeting motion and waitsa few seconds for a reaction from the person. If the per-

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

4son reacts to the robot, it says ”Nice to meet you” andfinishes the behavior.
Each behavior usually lasts for five to fifteen seconds. The behavior transition is handled by a ”behavior selector.” Only one behavior can be executed ata time. After the execution of each behavior, the behavior selector chooses one behavior based on the preimplemented episode rules (Table 2). The episode rules
were designed to deal with such situations considering
the interaction history to select an appropriate subsequent behavior and to prevent the execution of a behavior in specific situations. The behavior selector manages
the next behavior of the robot using the episode rules
and the sensor inputs. Note that part of the sensor inputs are substituted by a human operator as explainedat the end of this section; the behavior selector is partially autonomous.
In total, we implemented 141 behaviors and 233
episode rules with four kinds of behavior classes: routeguidance (101 behaviors), providing shop information(32 behaviors), greeting (7 behaviors), and coupon printing (1 behavior). For example, for the implementationof a route-guidance behavior, the robot explains a destination s route with utterances and gestures. It points
in the direction and says, Please go that way, with
an appropriate reference term. It continues the explanation: After that, you will see the shop on yourright. Since the robot knows all of the mall’s shops andfacilities (toilets, exits, nearest train station, etc.), it
can explain 101 destinations. As well as route-guidance,
we implemented 32 behaviors to provide information
about each shop. In this behavior, the robot explains
the sales items of each shop. If a person wants to print a
coupon, the coupon-printing behavior will be executed:
the robot says OK. Please wait a minute. While thecoupon is printing, the robot says, The printing will
finish soon. After it is finished, the robot says, Hereyou are and points to the coupon.
In this study, one operator supported the speech
recognition and the timing of the start of the interaction. Speech recognition is particularly crucial to realizesmooth interaction between robots and people. For the
robot’s teleoperation, we used an interface that displays
sensor information from the robot and the environment[15]. The operator listens to visitor responses to therobot s questions and chooses the appropriate button
as well as the robot’s speech recognition function. The
operator starts the interaction by monitoring people’s
behavior by camera; 1) if visitors greet the robot or 2)
stands more than three seconds within three meters of
it, it starts to interact with them.

Table 1 Pseudocode of ”GREETING” behavior
Precondition: IsHumanExist() == true
Situated processing:
1: Initially, returnValue = 0;
2: Say ( Hello, how are you? );3: PlayMotion ( greeting );4: startTime=GetNowTime();5: while ( GetNowTime() - startTime < 3 seconds )6: if (IsReactGreeting() == true )7: returnValue = 1;
8: break;
9: end if
10: end while
11: if (returnValue ==1)
12: Say ( Nice to meet you );13: end if
14: return returnValue;

Table 2 Grammar of episode rules
1: <ModuleID=retVal>...<...>NextModule
2: (<ModuleID1=retVal1> | <ModuleID2=retVal2>)...
3: (...)n,m...
4: !<...>NextModule
5: <M̂oduleID=r̂etVal>NextModule(1: basic structure of describing executed sequence,
2: OR , 3: repetitions, 4: negation of episode rule,
5: negation of Module ID and result value)

3.2 Display controller
We used a 50-inch display and an A7 size printer (Brother,
MW140BT type-F) to show shop images and to print
coupons. We prepared two control mechanisms: a GUI
and a network interface. Both mechanisms were used ineach field trial.
For the former, we developed a simple GUI thatworks on a PC to print a coupon. For the latter, wedeveloped a function to print a coupon by a network,
enabling the robot to control the displayed images andthe timing of printing the coupons. To realize such autonomous behavior, we improved our scripting language
for multi-robots [27] because it has adequate capabilities for describing multi-system communication and issimple enough for developers to use easily to control
robot behaviors and images with the display.
In this system, a set of script files was interpreted
and executed. Fig. 4 shows scenes where the small robot
interacts with people and prints a coupon. At the interaction’s beginning, the robot greets the people (Fig. 4a)and explains the coupon (Fig. 4b). Since they want it
printed, the robot does so (Fig. 4c). Finally, the people
get the coupon after listening to the robot’s explanation(Fig. 4d).

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-45(a) Robot greets people(b) Robot
coupons

explains

theFig. 5 Experimental environment(c) People wait for coupon to
be printed(d) Child picks up the coupon

Fig. 4 Interaction scenes of advertisement by a robot

4 Field trial I: eﬀects of robot presence
The purpose of this field trial is to measure how the
robot s presence attracts people and to measure such
eﬀects on advertising. Therefore, we designed a fieldtrial where the display was placed in a shopping mallwith/without either robot.
As described above, we prepared two social robots
whose main diﬀerence is their size: one is about 30 cm
tall and other is about 120 cm. We focused on diﬀerent
sizes because size aﬀects a robot’s attractiveness andthe ease of initiating interaction. A normal-sized robot
might attract more people than a relatively small robot.
On the other hand, interaction with a small robot isprobably easier. Based on these considerations, it remains unknown which size is better for advertising. Of
course, loud sounds or conspicuous movements easily
draw attention to a normal-sized robot, but they mightalso discourage people. So we focused on robot size andits eﬀect (e.g., ease of initiating interaction).
In this research, due to the diﬃculties of developing two robots with the same design but diﬀerent sizes,
we prepared two kinds of robots with diﬀerent appearances. Both have a general humanoid robot design andare commonly used in human-robot interaction.

4.1 Environment and settings
We conducted the field trial in a shopping mall environment. The robot was placed in a main corridor of a mallwhose visitors are mainly families, couples, and friends,
all of whom could freely interact with our system.
Figure 5 shows the environment where we installed
two cameras and the display. We placed the robot on astand next to the display. We adjusted the height of thestand and the display to prevent changes of the heightof the eye lines of visitors during interaction with therobots.
We obtained permission to record video and sensor
data from the mall authorities. The experimental protocol was reviewed and approved by our institutional
review board.

4.2 Procedures
The types of visitors diﬀered between day and evening,
so we divided the field trial times into daily time slotsthat covered both daytime (when people mainly havelunch, go shopping, or visit mall-sponsored events) andevening (when people have dinner, go shopping, or go
to the station) to avoid skewed results due to diﬀerent
types of participants.
We assigned a weekday for each condition and conducted a field trial for four hours on each condition, twohours for each time slot.

4.3 Conditions
We prepared three conditions to investigate the eﬀects
of the robot’s presence. To equalize the interaction modalities in the comparisons we decided to use only a GUI
to print coupons; the robot did not control the display.
Condition 1: GUI only
In this setting, neither of the robots was used. Only
the display and a mouse were installed in the environment. With a mouse, users can choose between two
coupon categories: restaurants or shops. Ten restaurants and eight shops were available. Second, the GUI

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

6shows all the coupons of the selected category. Afterthe user chooses a specific shop, the GUI showed detailed information about the coupon. If the user clicks
the print button, the system prints the coupon.
Condition 2: GUI with a small robot
In this setting, Robovie-miniR2 was installed withthe display and a mouse. We adjusted its height to equal
Robovie-II. When the robot was not interacting withanyone, it simply looked around and waited for a person
to talk with. At the beginning of the interaction, therobot greets the person, introduces itself, and asks,
Can I help you? I can give you route information, or I
can show you how to use the GUI to print coupons. If
someone requests route-guidance, the robot provides itby pointing and conversation. If someone wants to print
a coupon, the robot simply explains: Please select a
coupon with the mouse.
The GUI is basically the same as GUI only. The
only diﬀerence is that the robot explains the timing forprinting the coupons; if users click the print button, therobot executes the coupon-printing behaviors that are
written in Section 3.1.
Condition 3: GUI with a normal-sized robot
In this setting, Robovie-II was placed next to thedisplay and a mouse. The other settings are the sameas GUI with a small robot. The only diﬀerence is therobot; the dialogs of the robots are identical.

Moreover, we hypothesized that the ease of initiating interaction would aﬀect the number of printed
coupons. It is reasonable to assume that people who
initiate interaction with the robot will listen to it morethan people who are approached by it. In fact, past research reported that the acceptable distance for being
approached becomes smaller due to the robot s height[28][29]; this work used a mobile robot, but its result resembles our assumption. Moreover, we believe such influences aﬀect advertising. For example, in a shopping
mall, many shop persons greet customers to initiate interaction and recommend sales items, but the ratio of
buying by such invited people would be smaller thanthe case of people who independently approach suchshop persons.
Hypothesis 3: People who initiate interaction withthe robot will print more coupons than people who areapproached by the robot.
Finally, we hypothesized that robot size would aﬀect
the number of people who printed coupons by considering the second and third hypotheses.
Hypothesis 4: The small robot will increase thenumber of people who print coupons more than thenormal-sized robot.
4.4 Results
4.4.1 Number of people who used the system

4.3.1 Hypotheses
We hypothesized that the presence of each robot wouldaﬀect the number of people who used the system because we assumed that visitors would be attracted to
the robot.
Hypothesis 1: The number of people who uses thesystem will be larger when a robot is placed next to thedisplay.
We also hypothesized that the robot size would affect the ease of initiating interaction, because we assume that a smaller robot is more approachable. People
may feel more comfortable starting an interaction thanjust engaging in general interaction. Moreover, past research reported that a tall person increases the personal
distance of the interacting people [28] and a large robot
creates more anxiety than a small one [29]. In this experiment, the small-sized robot is 30-cm tall, which isshorter than the normal-sized robot, which is 120-cm
tall. This diﬀerence might aﬀect the ease of initiatinginteraction.
Hypothesis 2: The ease of initiating interaction
with the small robot will be higher than with the normalsized robot.

For analysis, we classified people into two categories
based on their behavior with the system: users andpassers-by (Table 1).
Users: People who tried the system.
Passers-by: People who just passed through theenvironment.
We verified the diﬀerences of the number of people
in both categories among the conditions with a Chisquare test (Table 3). The results revealed significant
diﬀerences among conditions (χ2 (2) =183.259, p<.001).
Multiple comparisons with the Bonferroni method revealed significant diﬀerences: GUI with a normal-sized
robot > GUI with a small robot (p<.001), GUI witha normal-sized robot > GUI only (p<.001), and GUI
with a small robot > GUI only (p<.001). Our results
confirm that the number of people who used the system
was significantly large when a robot was present.
Table 3 Number of people who used the system
GUI only
User
Passers-by

75698

GUI with
small robot
95
6325

GUI with
normal-sized robot
154
4300

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

7

4.4.2 Number of people who initiated interaction
To investigate the beginning of interaction, we also measured the number of people who initiated interaction
with the robot and those who were approached by it in
the user category. For classifications, we referred to theoperation logs for starting the interactions; if a visitor
greeted the robot, the visitor was classified as initiatedby the user. This analysis was conducted with conditions 2 and 3 because we were investigating the robot
eﬀects; GUI only did not include any robots.
We also verified the diﬀerences of the number of people in these categories between GUI with a small robot
and GUI with a normal-sized robot with a Chi-square
test (Table 4). The results revealed significant diﬀerences among conditions (χ2 (2) = 5.476, p<.05). Residual analysis revealed that initiated by user in GUI witha small robot was significantly high (residual = 2.340,
p<.05) and printed in GUI with a normal-sized robot
was significantly low (residual = -2.340, p<.05). Our results confirm that the number of people who initiatedinteraction was significantly large when the small robot
was present.

4.4.3 Number of people who printed coupons
We also classified people into three categories based on
their interaction histories with the system: printed, notprinted, and interaction only. We eliminated people who
used the system more than once. The following are thecategory details:
Printing: People who printed coupons after interaction
No printed: People who chose coupons without
printing coupons

Table 4 Number of people who initiated interaction

Initiated by user
Initiated by robot
Total

GUI witha small robot
70
25
95

GUI with
normal-sized robot
91
63
154

Table 5 Number of people who printed coupons in GUI in small
robot condition
PrintedNot printed
Interaction only
Total

Initiated by user
14
6
50
70

Initiated by robot
0
1
24
25

Interaction only: People who did not chose coupons.
This category includes people who only talked to therobots and requested route-guidance.
We verified the diﬀerences of the number of people
in all categories depending on the initiation patternsof interaction with a Chi-square test. In GUI with asmall robot (Table 5), the results revealed significant
diﬀerences among conditions (χ2 (2) = 6.950, p<.05).
Residual analysis revealed that printing initiated byusers was significantly high (residual = 2.422, p<.05),
and printing initiated by the robot was significantly low(residual = -2.422, p<.05). On the other hand, in GUI
with a normal-sized robot (Table 6), the results did notreveal significant diﬀerences among conditions (χ2 (2) =2.844, n.s.). Our results confirm that the number of people who printed coupons was significantly large amongpeople who initiated interaction with the robot when
the small robot was present.
We also verified the diﬀerences of the total numberof people in all categories between GUI with a small
robot and GUI with a normal-sized robot with a Chisquare test (Table 7). The results revealed significant
diﬀerences among conditions (χ2 (2) = 6.877, p<.05).
Residual analysis revealed that printing in GUI witha small robot was significantly high (residual = 2.577,
p<.01) and significantly low in GUI with a normal-sized
robot (residual = -2.577, p<.01). Our results confirm
that the number of people who printed coupons was
significantly large when the small robot was present.

4.5 Observations of visitor interactions
Here, we discuss how the robot interacted with people
by introducing scenes of visitor interaction as a casestudy. In this field trial, most people who interactedwith the robot asked that a shop’s coupon be printed,
Table 6 Number of people who printed coupons in GUI in
normal-sized robot condition
PrintedNot printed
Interaction only
Total

Initiated by user
7
6
78
91

Initiated by robot
1
4
58
63

Table 7 Total number of people in each category

PrintedNot printed
Interaction only
Total

GUI with
small robot
14
7
74
95

GUI with
normal-sized robot
8
10
136
154

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

8expressed thanks, and entered the shop. Before taking the coupon, almost all waited until the robot said,
Here you are, although the printing was obviously
finished. They seemed to be waiting for permission. On
the other hand, when the robot was not placed next
to the display, nobody thanked the system; such a difference indicates that the social presence of the robots
was recognized.
Some visitors repeatedly interacted with the small
robot. One family visited the robot three times overa six-hour period, printed coupons of diﬀerent shops
each time, and entered each shop. In this case, the family member who interacted with the robot changed: themother, the son, and finally the daughter. Another family visited the robot twice on diﬀerent days. We asked
why. The mother said, because my child wanted tointeract with the robot again. Such interactions wereobserved in past field studies at a shopping mall [13]and indicate that robots are a good device to attract
people and encourage interaction related to advertising.

5 Field trial II: eﬀects of advertising behaviors
by a robot
The first field trial suggested the eﬀects of the robot spresence in the advertising context, but it did not show
the eﬀects of the conversation schemas. Thus, it remains
unknown whether a recommendation from a robot increases customer interest in advertising contexts andhow to maximize the eﬀects of advertisements.
In our second field trial, we focused on the eﬀects of
conversation schemas during advertisements. Conversation schemas in advertisement contexts often change
depending a shop s situation. For example, if mall administrators want to promote a specific shop or item,
they can devise an advertising strategy that strongly
recommends them. Salespersons in the mall change conversation schemas to recommend the shop or the item.
On the other hand, from the visitors’ view, it is important to choose from multiple candidates to enjoy shopping. If they like window-shopping, recommendations
might be unnecessary, but they could be helpful whenbuying something. We prepared diﬀerent conversation
schemas to investigate their eﬀects on advertisementuse.

5.1 Method
In this trial, we used the same environment from thefirst trial, but the settings were slightly diﬀerent. First,
we only used Robovie-miniR2 because in the first fieldtrial, it outperformed the other robot in this setting.

Second, we eliminated the GUI to print a coupon; therobot printed coupons by a network interface link to
the display during interaction to investigate the eﬀectiveness of conversation schemas.
We also assigned daily time slots on weekdays foreach condition and conducted a field trial for four hours
on each condition, two hours for each time slot.

5.1.1 Conditions
In the context of providing a coupon, there are basicallytwo kinds of conversation schemas: recommend a specific coupon or give a choice to print a coupon from all
candidates. From this point of view, we prepared three
conditions by combining the two ways.
Condition 1: Choice only
In this condition, since the robot did not recommend any shop to the visitors, they themselves selected
coupons from available candidates. At the beginning of
the interaction, the robot greets the person, introduces
itself, and asks, Can I help you? I can give you route information or provide coupons you can use in the mall.
If the people request route-guidance, the robot guides
them with gestures and conversation.
If they want to print a coupon, the robot asks theinteracting person to choose a category: restaurants or
shops. Then the robot shows an image that includes all
the coupons of the selected category based on the results of the speech recognition function. It also displays
an image that includes detailed information about the
selected shop. The robot prints a coupon with the display controller if the visitor requests one. For example,
if the person says, Please print a coupon from the
Korean restaurant, the robot executes the couponprinting behavior that is written in Section 3.1.
Condition 2: Recommendation only
In this condition, the robot recommends a specificshop depending on the time slot; the visitor can choose
whether to select a coupon. In the daytime slot, therobot recommends the Korean restaurant. In the evening
time slot, the robot recommends the food court because
it provides desserts, unlike the Korean restaurant.
At the beginning of the interaction, the robot greets
the people, introduces itself, and asks, Can I help you?
I can give you route information or recommend a shop.
If they request route-guidance, the robot guides themwith gestures and conversation.
If they are interested in a recommendation, the robot
suggests a specific shop by providing information anddisplaying a coupon image on the display. For example,
the robot says, I suggest the Korean restaurant. You
will enjoy its spicy food and its modern interior! If

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

9the person wants to print the coupon, the robot does
so with the display controller, as described above.
Condition 3: Both choices
In this condition, the robot recommends a specificshop after it shows all the coupons from the selectedcategory. For example, when people want to print a
restaurant coupon, the robot oﬀers all the restaurant
coupons and asks, Listen, I can recommend a shop.
If the people are interested in the robot’s recommendation, the robot suggests a specific shop. This condition
combines Choice only and Recommendation only.

Table 8 Number of people who printed coupons in field trial II

PrintedNot printed
Interaction only

Choice
only
16
16
106

Recommendation
only
10
28
80

Both
choices19
976

Table 9 Rate of printed specific coupons in field trial II

Printed ratio

Choiceonly
1/138

Recommendation
only
10/118Bothchoices
1/104

5.1.2 Hypotheses
We hypothesized that the conversation schemas wouldaﬀect the number of people who printed coupons. To
maximize the number of specific coupons, only recommendation is better than the other way because therobot did not provide other candidates. On the other
hand, to maximize the total number of printed coupons,
the robot should provide both choices. Based on these
considerations, we made the following hypotheses:
Hypothesis 5: The total number of printed coupons
will increase when the robot provides both choices.
Hypothesis 6: The number of specific coupons will
increase when the robot only recommends a specificcoupon.

5.2 Results
5.2.1 Total number of printed coupons
We classified people into three categories based on theirinteraction histories with the system: printed, not printed,
and interaction only (Table 8). The definition of eachcategory is completely identical as in the first field trial.
We verified the diﬀerences of the number of people in each category among the conditions with a Chisquare test that revealed significant diﬀerences (χ2 (4)=15.122, p<.01). Residual analysis revealed that printed
in Both choices is significantly high (residual =
2.11, p<.05). Moreover, the analysis revealed that notprinted in Recommendation only is significantly high(residual = 3.368, p<.01) and significantly low in Bothchoices (residual = -2.071, p<.05). Our results confirm that the total number of printed coupons was significantly large when the robot provided both choices.
5.2.2 Number of specific coupons
To investigate the eﬀectiveness of the robot recommendations more deeply, we measured the number of people

who printed recommended coupons in Recommendation only (i.e., Korean restaurant and food court) (Table 9).
We verified the diﬀerences of the number of people who printed coupons for the Korean restaurant andthe food court among conditions with a Chi-square test
that revealed statistically significant diﬀerences (χ2 (2)= 13.031, p<.01). The number of people who printed
specific coupons in Recommendation only was significantly high (residual = 3.608, p<.01), and the numberof people who did not in Recommendation only wassignificantly low (residual = -3.608, p<.01). The number of people who printed specific coupons in Choice
only was significantly low (residual = -2.109, p<.05),
and the number of people who did not in Choice only
was significantly high (residual = 2.109, p<.05). Our results confirm that the number of specific coupons was
significantly large when the robot only recommends a
specific coupon.

5.3 Observations of visitor interactions
In this field trial, most people who interacted with therobot also asked that a shop’s coupon be printed, expressed thanks, and entered the shop like the observations in the first field trial. The main diﬀerence was thereactions to the recommendations from the robot. We
sometimes observed scenes where a recommendation
might have influenced the shopping activity of visitors,
who often hesitated over which candidate to choose
when selecting a coupon. In one case, the customers discussed among themselves for over two minutes in frontof the robot. Furthermore, some customers changed theirminds, decided not to print coupons, and left the robot
to directly check out each shop. If the robot could deal
with such situations well, it might increase the eﬀect ofadvertisements.

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

10

6 Discussion
6.1 Summary of results
Our results from the first field trial supported our firstand second hypotheses; the number of people who used
the system increased when a robot was present. The
number of people who initiated interaction with thesmall robot was larger than the case of the normal-sized
robot.
Our results also showed partial support for our third
hypothesis and support for our fourth hypothesis. When
we only used the small robot, people who initiated interaction with it printed more coupons than people
who were approached by it. The number of people who
printed coupons with the small robot was larger thanthe case of normal-sized robots. These results suggestthat the small robot has an advantage for advertisementuse settings.
The results from the second field trial supported our
fifth and sixth hypotheses; by providing both choicesfrom the robot, the number of printed coupons significantly increased. By only providing recommendations,the number of specific coupons also significantly increased. These results suggest that conversation schemas
are important for eﬃcient advertisements; a robot mustchoose an appropriate schema depending on the advertising situations.

6.2 Is the performance adequate for advertisements?
In the field trials, there were about 100 interactions
for each condition. In addition, there were about 5,000
pedestrians in each condition. As shown in each result,the number of interacting groups did not diﬀer over theexperimental period. We successfully elicited interest
from customers who interacted with the robot and theprinted coupons. Unfortunately, judging whether thenumber of printed coupons is suﬃcient as advertisements by comparing existing coupon-printing services
is diﬃcult, because such baseline information was notprovided by the shopping mall.
According to past related work, four mobile social
robots invited 15 people to a shop in a shopping mallout of 414 people with whom it interacted. There were2411 pedestrians in the trials [14]. In this research, however, in the second field trial, just one social robot gave
45 coupons to 350 people with whom it interacted; there
were more than 15,000 pedestrians in the trials. Therefore, we believe that our system’s performance at least
equals the previous one. Note that this is just a simple comparison, not laboratory experiments with precise control. Thus, the comparison suﬀers from unbal-

anced factors including environment, number of robots,
appearances, and so on. Yet we believe such a comparison is useful to understand the eﬀects caused by the
robot.

6.3 Eﬀects of size and appearance
In this research, we used diﬀerent sizes and appearancesof robots to investigate the eﬀectiveness of their presence. But the question remains unanswered: which ismore important, size or appearance? Such discussion isimportant even if it is diﬃcult to develop robots withidentical designs of diﬀerent sizes. We also discuss otherfactors such as the behavior of the robot to attract people.
We believe that investigating robot size is more important for advertising uses than other factors. We assume that the design of each social robot that works
in daily environments should be basically familiar. In
such situations, the robot design is not an essential factor to attract people. Of course, if the robot resembled
a popular children’s TV cartoon character, it would attract more people. But such character eﬀects are beyond the scope of this research. Moreover, even if thedesigner can make identical designs for diﬀerent sizesof robots, the size eﬀect remains crucial for attracting
people. Finally, the size of the robot directly influences
the cost; our experimental results provide solid knowledge for people who are interested in using a robot foradvertisements.
Loud sounds or conspicuous movements easily attract attention, but they are often discourage interaction. In particular, if the robot is normal-sized, people
might be more afraid of it, especially young children.
Polite invitation behavior might attract people, but theeﬀects of such behaviors can be easily anticipated without experiments.

6.4 Interaction times between robot and visitors
To understand how long the interaction between robot
and human lasted, we investigated the interaction time
between visitors and robots in the Both choices condition in field trial II.
The average interaction time was 77.0 seconds inthis condition. The average interaction time with people
who are printing a coupon basically exceeds two minutes because they need to select and wait until a coupon
is printed. However, the average interaction time withpeople who are only interacting with the robot, such asasking for route directions, is less than one minute. In

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

11fact, many people in the interaction only category left
after a simple request for directions.
We believe that visitors did not linger around therobot after they understood its role even if they wereinterested in it before interacting with it. Such a phenomenon shows that novelty eﬀectively attracts people’s attention, but keeping their attention only withnovelty is diﬃcult. Thus, the robot needs to quickly attract attention with its behavior to continue interaction
with visitors. Such a trend might become stronger due
to the robot s novelty; in fact, the number of people
in the interaction only in the GUI with a normal-sized
robot condition is larger than the GUI with a small
robot condition.

6.5 Limitations
Since our comparisons are based on a case study withexisting robots, Robovie-miniR2 and Robovie-II, theirgenerality is limited. As noted above, the appearancesof these robots are diﬀerent because of the diﬃculties
of developing robots with identical designs of diﬀerent
sizes. For example, more human-like robots like Geminoid [31] might aﬀect advertising diﬀerently. We cannot ensure that our findings can be applied to all social
robots.
Moreover, our comparisons were conducted in a shopping mall. Due to the diﬃculties of controlling visitor
flow in them, the number of people who used the system diﬀered among conditions. Therefore, we cannot
ensure that the findings can be applied to other kindsof facilities.
However, our trials involved about 5,000 ordinary
people, including families, couples, and friends and wereconducted under a situation where they freely interacted with the robots. Therefore, we believe that thesetting is realistic enough to oﬀer important knowledge
for researchers interested in field trials in real environments with social robots.

7 Conclusion
In this paper, we investigated the eﬀectiveness of thepresence of a robot and its conversation schemas ina real advertising context in a shopping mall. We developed an advertising robot system that consists of asocial robot and a display. The social robot provided
coupon-giving services by showing visual informationon the display, which is a reasonable service to expectfrom a social robot in a shopping mall, since their novelty eﬀectively attracts attention. The robot is semiautonomous due to the diﬃculty of speech recognition.

We conducted two kinds of field trials at a shopping
mall. In the first trial, we investigated how a robot attracts people and advertises shops to them in a public environment using two kinds of robots: small andnormal-sized. The robots increased the number of people who used the system, and the small robot outperformed the normal-sized robot for advertising uses.
In the second trial, we compared three diﬀerent conversation schemas. Our results show that presenting
coupon candidates with recommendations of specificshops more eﬀectively maximizes the number of printed
coupons; on the other hand, only recommending a specific shop increased the number of specific coupons morethan the other schemas. Room for improvement remains in how to increase the number of people who
print a coupon; however, we believe that this research
provides important knowledge for robotics researchers
who are interested in advertising applications.
Acknowledgements We thank the staﬀ of the Asia and PacificTrade Center Co., Ltd. for their kind cooperation. This research
was supported by the Ministry of Internal Aﬀairs and Communications of Japan.

References
1. Kidd, C. and Breazeal, C., Eﬀect of a Robot on User Perceptions, IROS’ 04, pp. 3559-3564, 2004.
2. A. Powers, S. Kiesler, S. R. Fussell, and C. Torrey, Comparing a computer agent with a humanoid robot, Human-Robot
Interaction 2007. (pp. 145-152).
3. Mutlu, B. et al., A Storytelling Robot: Modeling and Evaluation of Human-like Gaze Behavior,IEEE Int. Conf. on
Humanoid Robots (Humanoids2006), pp. 518-523, 2006.
4. Kahn, P. H. Jr. et al., Design Patterns for Sociality in Human
Robot Interaction, HRI2008, pp. 97-104, 2008.
5. W. Burgard, A. B. Cremers, D. Fox, D. Hanel, G. Lakemeyer,
D. Schulz, W. Steiner, and S. Thrun, The interactive museum
tour-guide robot, Proc. of National Conference on Artificial Intelligence, pp. 11-18, 1998.
6. R. Siegwart et al., Robox at Expo. 02: A Large Scale Installation of Personal Robots, Robotics and Autonomous Systems,
42(3), pp. 203-222, 2003.
7. R. Gockley, J. Forlizzi, and R. Simmons, Interactions witha Moody Robot, ACM/IEEE International Conference on
Human-Robot Interaction (HRI2006), pp. 186-193, 2006.
8. T. Kanda, T. Hirano, D. Eaton, and H. Ishiguro, Interactive
Robots as Social Partners and Peer Tutors for Children: A Field
Trial, Human Computer Interaction, 19(1-2), pp. 61-84, 2004.
9. K. Wada and T. Shibata, Living With Seal Robots - Its Sociopsychological and Physiological Influences on the Elderly ata Care House. IEEE Transactions on Robotics, 23(5), pp. 972980, 2007.
10. K. Dautenhahn and I. Werry, A quantitative technique foranalyzing robot-human interactions, IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS’02), pp.
1132-1138, 2002.
11. H. Kozima, C. Nakagawa, and Y. Yasuda, Interactive robots
for communication-care: A case-study in autism therapy, IEEE
International Symposium on Robots and Human Interactive
Communications (Ro-Man 2005), pp. 341-346, 2005.

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

1212. F. Tanaka, A. Cicourel, and J. R. Movellan, Socialization
between toddlers and robots at an early childhood education
center, Proc. of the National Academy of Sciences of the USA,104(46), pp. 17954-17958, 2007.
13. T. Kanda, M. Shiomi, Z. Miyashita, H. Ishiguro, and
N. Hagita, An aﬀective guide robot in a shopping mall,
ACM/IEEE International Conference on Human-Robot Interaction (HRI2009), 2009.
14. M. Shiomi, T. Kanda, D. F. Glas, S. Satake, H. Ishiguro,
and N. Hagita, Field Trial of Networked Social Robots in a
Shopping Mall, The 2009 IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS2009), 2009.
15. D. F. Glas, T. Kanda, H. Ishiguro, and N. Hagita, Simultaneous Teleoperation of Multiple Social Robots, ACM/IEEE International Conference on Human-Robot Interaction (HRI2008),
pp. 311-318, 2008.
16. B. J. Fogg,
Persuasive Technology: Using Computers toChange What We Think and Do, Morgan Kaufmann, 2003.
17. N. Tintarev and J. Masthoﬀ. A Survey of Explanations in
Recommender Systems, Workshop on Recommender Systems
and Intelligent User Interfaces associated with ICDE’07.
18. K. Shinozawa, F. Naya, and J. Yamato, Diﬀerences in Effect of Robot and Screen Agent Recommendations of Human
Decision-Making, International Journal on Human-Computer
Studies, Vol. 62, 2, Feb. 2005.
19. W. A. Bainbridge, J. Hart, E. S. Kim, and B. Scassellati,
The Eﬀect of Presence on Human-Robot Interaction, IEEE Int.
Symposium on Robot and Human Interactive Communication(RO-MAN2008), pp. 701-706, 2008.
20. Y. Hiroi and A. Ito, Influence of the size factor of a mobile
robot moving toward a human on subjective acceptable distace,
Z. Gacovski (Ed.), Mobile Robots - Current Trends, pp. 177190, INTECH., 2011.
21. T. Kanda et al., Who will be the customer? A social robotthat anticipates people’s behavior from their trajectories, UbiComp2008.
22. M. K. Lee, S. Kiesler, and J. Forlizzi, Receptionist or information kiosk? How do people talk with a robot? In Proceedings of CSCW 2010, pp. 31-40, 2010.
23. R. P. Abelson, Psychological status of the script concept,
American Psychologist, 36, 7, pp. 715-729, 1981.
24. K. Fischer,
The role of users’ preconceptions in talking
to computers and robots, Proceedings of the Workshop on’How People Talk to Computers, Robots, and Other Artificial
Communication Partners’, pp. 112-130, 2006.
25. D. Dahlback, A. Jonsson, and L. Ahrenberg. Wizard of oz
studies-why and how, Knowledge based systems, Vol. 6, No.
4, pp. 258-266, 1993.
26. H. Kawai, T. Toda, J. Ni, M. Tsuzaki, and K. Tokuda,
XIMERA: A New TTS from ATR Based on Corpus-Based
Technologies, Proc. of Fifth ISCA Workshop on Speech Synthesis (SSW5), pp. 179-184, 2004.
27. D. Sakamoto et al., Humanoid Robots as a Broadcasting
Communication Medium in Open Public Spaces, International
Journal of Social Robotics, 1(2), pp. 157-169, 2009.
28. John J. Hartnett, Kent G. Bailey & Craig S. Hartley, Body
Height, Position, and Sex as Determinants of Personal Space,
The Journal of Psychology: Interdisciplinary and Applied,
Volume 87, Issue 1, pp. 129-136, 1974.
29. Yutaka Hiroi and Akinori Ito, Influence of the Size Factor
of a Mobile Robot Moving Toward a Human on Subjective Acceptable Distance, Z. Gacovski (Ed.), Mobile Robots - Current
Trends, pp.177-190, INTECH., 2011.
30. T. Kanda, H. Ishiguro, M. Imai, and T. Ono, Development
and Evaluation of Interactive Humanoid Robots, Proceedingsof the IEEE, Vol. 92, No. 11, pp. 1839-1850, 2004.

31. D. Sakamoto, T, Kanda, T. Ono, H. Ishiguro, and N. Hagita
Android as a telecommunication medium with a human-likepresence, In ACM/IEEE International Conference on Humanrobot interaction, pp. 193-200, 2007.

Masahiro Shiomi received M. Eng. and Ph.D. degrees in engineering from Osaka University in 2004 and2007. From 2004 to 2007, he was an intern researcher
at the Intelligent Robotics and Communication Laboratories (IRC) and is currently a researcher at IRC at
the Advanced Telecommunications Research InstituteInternational (ATR) in Kyoto. His research interests
include human-robot interaction, interactive humanoid
robots, networked robots, and field trials.
Kazuhiko Shinozawa is a Group Leader, Interaction
System Laboratory, Intelligent Robotics and Communication Laboratories, Advanced Telecommunications
Research Institute International (ATR). He received
the B.E. and M.E. in Electrical Engineering from Keio
University in 1988 and 1990, respectively. In 1990, hejoined the NTT Human Interface Laboratories, where
he was engaged in solving optimization problems andpredicting weather radar images by using artificial neural networks. In 1998, he moved to the NTT Communication Science Laboratories. Since then, he has
been conducting research for building a robot that cancommunicate with humans naturally and smoothly. He
moved to ATR in May 2003.
Yoshifumi Nakagawa is a researcher at IRC at theAdvanced Telecommunications Research Institute International (ATR) in Kyoto and Osaka Urban Industry
Promotion Center. His research interest include humanrobot interaction, and advertising strategies.
Takahiro Miyashita received B.S., M.S., and Ph.D.
degrees in engineering for computer-controlled machinery from Osaka University, Japan in 1993, 1995, and2002. From 1998 to 2000, he was a fellow research of
the Japan Society for the Promotion of Science. From
April to September 2000, he was a researcher at theSymbiotic Intelligent Group at ERATO Kitano Symbiotic Systems Project of Japan Science and Technology
Agency. From October 2000 to July 2002, he was an
assistant professor at Wakayama University. Currently,
he is a group leader and a senior research scientist at
the Intelligent Robotics and Communication Laboratories (IRC), Advanced Telecommunications ResearchInstitute International (ATR). He is also a member ofthe Robotics Society of Japan, the Japan Society of Me-

The final publication is available at
http://link.springer.com/article/10.1007/s12369-013-0180-4

13

chanical Engineers, and the Japanese Society for Artificial Intelligence.
Toshio Sakamoto received the degree of B.Eng. inElectronic Engineering from Osaka Electro Communication University, Osaka, Japan in 1988. He establishedthe company of system development, Personal Technology, Inc. in 1994. He is working as a system engineer ofmeasurement, control and monitoring systems in industrial field. His research interests include human-robot
interaction and using robot technologies in nursing care.
Toshimitsu Terakubo is currently a managing director at the Hisamitsu Co. Ltd. His research interest include human-robot interaction, and advertising strategies.
Hiroshi Ishiguro received a D.Eng. in systems engineering from the Osaka University, Japan in 1991. He
is currently professor of Department of Systems Innovation in the Graduate School of Engineering Scienceat Osaka University (2009-). He is also group leader(2002-) of Hiroshi Ishiguro Laboratory at the Advanced
Telecommunications Research Institute and the ATR
fellow. He was previously Research Associate (19921994) in the Graduate School of Engineering Science at
Osaka University and Associate Professor (1998-2000)
in the Department of Social Informatics at Kyoto University. He was also Visiting Scholar (1998-1999) atthe University of California, San Diego, USA. He was
Associate Professor (2000-2001) and Professor (20012002) in the Department of Computer and Communication Sciences at Wakayama University. He then moved
to Department of Adaptive Machine Systems in the
Graduate School of Engineering at Osaka University
as a Professor (2002-2009). His research interests include distributed sensor systems, interactive robotics,
and android science. Especially, his android studies are
well-known very much in the world. He has received the
Osaka Cultural Award in 2011.
Norihiro Hagita received B.S., M.S., and Ph.D. degrees in electrical engineering from Keio University, Tsuruoka City, Japan in 1976, 1978, and 1986. From 19782001, he was with the Nippon Telegraph and Telephone
Corporation (NTT). He joined the Advanced Telecommunications Research Institute International (ATR) to
establish the ATR Media Information Science Laboratories and the ATR Intelligent Robotics and Communication Laboratories in 2001 and 2002. His current research interests include communication robots,
networked robot systems, interaction media, and pattern recognition. Dr. Hagita is a fellow of the Instituteof Electronics, Information, and Communication Engineers, Japan. He is a member of the Robotics Societyof Japan, the Information Processing Society of Japan,
and The Japanese Society for Artificial Intelligence. He
is a co-chair for the IEEE Technical Committee on Networked Robots.