The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

Design of a Gaze Behavior at a Small Mistake Moment for a
Robot
Masahiro Shiomi

Kayako Nakagawa

Norihiro Hagita

ATR Intelligent Robotics and Communication Laboratory
2-2-2 Hikaridai, Keihanna Science City,
Kyoto, Japan{m-shiomi, kayako_nkgw, hagita}@atr.jp

ABSTRACT
A change of gaze behavior at a small mistake moment is a natural response that reveals ourown mistakes and suggests an apology to others with whom we are working or interacting.
In this paper we investigate how robot gaze behaviors at small mistake moments change theimpressions of others. To prepare gaze behaviors for a robot, first, we identified byquestionnaires how human gaze behaviors change in such situations and extracted threekinds: looking at the other, looking down, and looking away. We prepared each gazebehavior, added a no-gaze behavior, and investigated how a robot’s gaze behavior at asmall mistake moment changes the impressions of the interacting people in a simple
cooperative task. Experiment results show that the looking at the other gaze behavior
outperforms the other gaze behaviors and indicates the degrees of perceived apologeticness
and friendliness.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

Key words: Communication robots, Gaze, Mistake, Mitigation

1. INTRODUCTION
Since human beings are fallible, completely preventing failures and mistakes is impossible.
When we make mistakes, we have various strategies to mitigate them, such as apologizing
or offering compensation. Even if the failure is minor, people do something to mitigate theunhappiness of others. Based on these considerations, we assume that since a robot will also
experience failure and make mistakes, it will also need mitigation strategies. Because of theincreasing development of robots that work with people [1~2], such a strategy for failures
will become especially important.
How should a robot behave at the moment of a mistake? Even if current robots do not cause
serious mistakes because of their limited uses, several researchers have started toinvestigate error recovery strategies from small mistake situations for robots [3~5]. For
example, Lee et al. investigated appropriate robotic service recovery strategies using
delivery robots: apologies, compensation, and user options [3]. Other research implemented
an apology behavior for robots that interact with people at small mistakes during interaction[4, 5]
However, these research works only focused an error recovery during conversations, i.e.,
verbal behavior. In interactions between people, they often change their nonverbal behavior,e.g., looking at the other, looking down, and so on. Since nonverbal behavior is essential inThe final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

interactions [6, 7], we believe that its design is also important for robot error recovery
strategies that resemble those of humans.
Gaze is one essential nonverbal behavior to indicate a robot’s intention [8]. In particular,
eye-contact is an important social cue that positively improves impressions toward robots invarious situations [9, 10]. These previous works show the significance of the gaze behaviorof robots in human robot interaction, but they did not focus on the effects of gaze behaviors
from the view of mitigation during mistake situations. It remains unknown how the robot’s
gaze behavior at mistake moments changes the impressions of others.

Fig. 1 Mistake: how should the robot change its gaze?
2. Data collection
Even though many works exist on gaze behavior in human-robot interaction [6-12], wefound none that focused on the design of gaze behavior at a robot’s mistake moment. So weinvestigated how people change their gaze behaviors at mistake moments by conducting aquestionnaire-based data collection.
We can empirically imagine that people change their gaze behavior depending on two
factors: social relationships and where the culpability lies. We asked participants to imagine
a situation where they made a mistake in a cooperative task. They assumed that their social
position is lower than the others (e.g., superiors, because people perceive robots as having

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

relatively lower social positions [13, 14]) and that they are responsible for the mistake.
They considered the following situation: “Describe how you change your gaze when you
make a mistake in front of a superior.”
Ten Japanese people (five women and five men whose average age was 20.5 years, standard
deviation (S.D) was 2.0) participated in our data collection. They freely described thechanges of their gaze behaviors at mistake moments when they themselves are responsible
for the mistake.
We gathered 20 sentences from the data collection (ten sentences for both assumptions
about responsibility). Two coders analyzed and classified the transcribed results. Cohen’s
kappa coefficient from their classifications was 0.964 and yielded the followingclassifications:(a) Looking at the other: they looked at the face of the person with whom they were
working(b) Looking down: they looked down because they could not face the other(c) Looking away: they looked away to avoid eye contact with the other

Table 1 shows each category for each assumption. The results indicated that people lookdown when they themselves feel responsible for the mistake; on the other hand, people lookat the other when they blame that person for the mistake. Based on these results, we used

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

four kinds of gaze behavior (three gaze behaviors and a no-gaze behavior) in our
experiments.
Table 1 Number of each category in different assumptions
Responsibility
on myself

on the other

Looking at other1

8

Looking down

91

Looking away

0

1

3. Experiments
3.1 Hypotheses and predictions about apologies
An apology signals the admission of one’s own mistake and suggests remorse and regret for
the action. For example, people who fail in cooperative tasks provide such signals. In fact,
our data collection suggests that people change their gaze behaviors at such moments. This
strategy is used not only in daily situations but also in medical malpractice litigation,
political contexts, and corporate culture [16].
A robot that works with people, e.g., doing a cooperative task, must be able to admit its
own mistakes and offer apologies at appropriate moments. Past research also reported theimportance of apologies from robots [3, 15]. To admit mistakes and apologize to others,gaze behavior at mistake moments would be useful. Based on these considerations, wemade the following hypothesis:
Prediction 1 Gaze behavior at a small mistake moment will increase the degree ofperceived apologeticness.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

3.2 Hypotheses and prediction for friendliness and dissatisfaction
Indicating an apology increases the degree of perceived friendliness and decreases thedissatisfaction toward the robot. However, as observed in the data collection, people change
their gaze behavior depending on their feelings about the cause of the mistake. Since
phenomena related to gaze behaviors at small mistake moments remain basically
unexplored, we established two contradictory hypotheses toward friendliness anddissatisfaction.

3.2.1 Hypothesis that assumes advantages of looking down
Based on the trend from the data collection, we assume that when a person accepts
responsibility for a mistake, he uses looking down behaviors to signal such intentions as“I’m sorry.” This attitude increases the degree of perceived friendliness and decreases
dissatisfaction. From these considerations, we made the following hypothesis:
Prediction 2-a: Looking down during small mistake moments will increase the degree ofperceived friendliness toward the robot more than other gaze behaviors.
Prediction 3-a: Looking down during small mistake moments will decrease dissatisfaction
toward the robot more than other gaze behaviors.

3.2.2 Hypothesis that assumes advantages of looking at the otherWe also have an opposite assumption: looking at others is friendlier than other gazebehaviors, because past research on gaze behaviors suggested that eye-contact improves

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

impressions toward robots [8-10]. Since looking down and looking away might be more
ambiguous for displaying intentions than looking at others, we made the following
hypotheses:
Prediction 2-b: Looking at the other at small mistake moments will increase the degree ofperceived friendliness toward the robot more than other gaze behaviors.
Prediction 3-b: Looking at the other at small mistake moments will decrease
dissatisfaction toward the robot more than other gaze behaviors.

3.3 Participants
Sixteen university students (eight women and eight men, whose age averaged 20.5, S.D. is2.13) participated.

3.4 Tasks
To investigate the effects of the gaze behaviors, we adopted a simple cooperation task inwhich the robot first asked the participant to put an object into its hand. After taking it fromthe participant, the robot put it into a box, rotated its body and released its arms. This task
was conducted twice in each condition. The robot successfully does the first trial, but in thesecond trial, it fails to put the object into the box.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

Fig. 2 Success(a) Looking down(b) Looking away(c) Looking at the otherFig. 3 Gaze behaviors during a mistake
3.5 Robot
In the experiment, we used Robovie-mR2, an interactive humanoid robot. To conduct a
cooperation task with participants, we prepared four kinds of motions: 1) taking an object
from a participant, 2) putting it into a box, 3) dropping it, and 4) changing gaze behaviors.
Fig. 2 shows both motions used in the first trial in each task. In the third motion, the robot
also turns to its right as the second motion, but it slightly spreads its arms in the middle ofthe turn. The object falls without going into the box (left side of Fig. 3). The robot changes
its gaze behavior depending on the conditions from the mistake moment; it waited one
second after the mistake moment based on research of robot response times [17].

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

3.6 Conditions
We used a within-participant experiment design to evaluate and compare the effects of thethree gaze behaviors from our data collection and the no-gaze behavior. Note that theparticipants reported that in the down/avoid/look conditions, the robot looked appropriately
at the mistake moments.
- No-gaze: when the robot fails, it does not change its gaze.
- Down: when the robot fails, it looks down (Fig. 3-a).
- Avoid: when the robot fails, it looks away (Fig. 3-b).
- Look: when the robot fails, it looks at the participant (Fig. 3-c).

3.7 Procedure
Robovie-mR2 was placed on a low table, and a cube was placed diagonally in front of it asa stand (Fig. 4). The box in which to drop the object was placed on the robot’s right.
Participants sat in front of the robot. In the interaction, the robot asks for the object twice.
The signal to start the trial was sent by the operator.
Before the first session, the participants were given a brief description of the experiment’s
purpose and procedure. Each participant participated in four sessions. They filled out
questionnaires after each session. The order of these conditions was counterbalanced.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

Fig. 4 Experiment environment

3.8 Measurement
To measure the subjective impressions, we prepared a questionnaire that addressed theapologeticness (the robot seemed to apologize), friendliness (I felt that the robot wasfriendly), and dissatisfaction (I felt dissatisfaction when the robot failed). After each session,
participants answered on a 1-to-7 point scale, where “7” is the most positive and “1” is themost negative. They could freely describe their impressions of each gaze behavior.

4. Results
4.1 Verification of prediction 1
We analyzed the apologeticness (Fig. 5) by conducting a one-factor within subject ANOVA.
Since Mauchly’s test indicated that the assumption of sphericity was violated (chisquare=16.174, p=.007), we corrected the degrees of freedom using the Huynh-Feldt
estimates of sphericity (epsilon=0.863). We found a significant difference among theconditions (F (2.589, 38.838) = 7.265, p =.001). Multiple comparisons with the Scheffe

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

method revealed significant differences: look > no-gaze (p = .001), down > no-gaze (p= .003), and avoid > no-gaze (p < .001). Prediction 1 was supported.

4.2 Verification of prediction 2
We analyzed the perceived friendliness toward the robot (Fig. 6) by conducting a one-factor
within subject ANOVA. We found a significant difference among the conditions (F (3, 45)= 15.496, p < .001). Multiple comparisons with the Scheffe method revealed significant
differences: look > no-gaze (p < .001) look > down (p = .004), look > avoid (p = .005),
down > no-gaze (p = .011), and avoid > no-gaze (p = .001). Prediction 2-b was supported,
but not prediction 2-a.

4.3 Verification of prediction 3
We analyzed dissatisfaction with the robot (Fig. 7) by conducting a one-factor within
subject ANOVA. Since Mauchly’s test indicated that the assumption of sphericity was
violated (chi-square=15.083, p=.010), we corrected the degrees of freedom using theGreenhouse-Geisser estimates of sphericity (epsilon=0.604). We found no significant
differences among the conditions (F (1.811, 27.164) = 2.853, p =.080). Neither predictions
3-a nor 3-b were supported.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

7
6
5
4
3
2
1
No-gaze Down Avoid

Look

Fig. 5 Perceived apologeticness

7
6
5
4
3
2
1
No-gaze Down Avoid

Look

Fig. 6 Perceived friendliness

7
6
5
4
3
2
1
No-gaze Down Avoid

Look

Fig. 7 Perceived dissatisfaction

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

5. Discussion
5.1 Analysis of free descriptions
Our results showed that the looking at the other condition outperformed looking down.
Since this phenomenon is contrary to the results suggested by the data collection, we
analyzed the free comments from the participants to determine why.
In our experiment, we gathered 64 sentences (16 for each gaze behavior). Two coders
analyzed and classified the transcribed results from them. Cohen’s kappa coefficient fromthe coder classifications was 0.625, which showed moderate agreement. They yielded thefollowing classifications:(a) The robot seemed to notice and to reflect on its own mistake.
(b) The robot seemed to notice without reflecting on its own mistake.
(c) The robot didn’t seem to notice its own mistake.

Table 2 shows each category for each gaze behavior. The results indicate that theparticipants felt that the looking at the other condition provides more reflection.

Table 2 Number of each category of gaze impressionsabc

No-gaze

2

7

7

Down

6

6

4

Avoid

6

8

2

Look

11

5

0

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

5.2 Responsiveness to mistakes
In this study, we situated the responsibility for the robot’s mistake in the cooperative task.
However, in cooperative tasks in real situations, a person might fail even if the robots
worked correctly, i.e., placing responsibility for the mistake on the person. In suchsituations, the gaze behaviors of the robot at mistake moments might have different
impressions, e.g., blaming a person. Groom et al. investigated the effects of blame from arobot in cooperation situations and argued that blame negatively affected impressions of it[15]. Therefore, designing a gaze behavior at the mistake moments of people who engage incooperative tasks with robots is fruitful future work.

6. CONCLUSION
We conducted an experiment during small mistake moments on four gaze behaviors:looking at the other, looking down, looking away, and no-gaze. Experimental results
revealed that participants perceived apologies from the gaze behaviors at mistake moments
and reported that the looking at the other condition increased the perceived friendlinessmore than the other conditions. This study provides useful knowledge for designing social
robots that work with people.

7. ACKNOWLEDGEMENTS
This work was supported by the Japan Society for the Promotion of Science, Grants-in-Aid
for Scientific Research No. 21118003.

The final publication is available athttps://benjamins.com/#catalog/journals/is.14.3.01shi

8. REFERENCES[1] Sara L, Jirina K, Mattias J, Henriette C, and Karol N, (2012), Hospital robot at work: Something alien or an
intelligent colleague?, Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 13-15.
[2] Mutlu B., and Forlizzi, J., (2008), Robots in Organizations: Workflow, Social, and Environmental Factors in
Human-Robot Interaction, Proceedings of the 3rd ACM/IEEE Conference on Human-Robot Interaction, 287-294.
[3] Lee, M. K., Kiesler, S., Forlizzi, J. Srinivasa, S., and Rybski, P., (2010), Gracefully Mitigating Breakdowns in
Robotic Services,” Proceedings of the 5th ACM/IEEE Conference on Human-Robot Interaction, 203-210[4] Peltason L. and Wrede, B., (2011), The Curious Robot as a Case-Study for Comparing Dialog Systems, AI magazine,
32, 4, 85-99.
[5] Lee, M. K., Kiesler, S., and Forlizzi, J, (2010), Receptionist or information kiosk: How do people talk with a robot?
Proceedings of the Conference on Computer-Supported Cooperative Work, 31-40.
[6] Birdwhistell R. L., (1970), Kinesics and Context: Essays on Body Motion Communication, Univ. of Pennsylvania
Press Philadelphia.
[7] Vargas M. F., (1986), Louder than Words: An Introduction to Nonverbal Communication, Iowa State Univ. Press.
[8] Srinivasan V, and Murphy, R. R., (2011), A survey of social gaze, In Proceedings of Human Robot Interaction, 253254.
[9] Yamazaki, A., Yamazaki, K., Kuno, Y., Burdelski, M., Kawashima, M., and Kuzuoka, H., (2008), Precision timing
in human-robot interaction: coordination of head movement and utterance. In Proceedings of CHI '08, 131-140.
[10] Yonezawa T., Yamazoe H., Utsumi A., and Abe S., (2007), Gaze-communicative Behavior of Stuffed-toy Robot
with Joint Attention and Eye Contact based on Ambient Gaze-tracking, Proceedings of the 9th International
Conference on Multimodal Interfaces, 140-145.
[11] Mutlu B., Kanda, T., Forlizzi, J., Hodgins, J., and Ishiguro, H, (2012), Conversational Gaze Mechanisms for
Humanlike Robots. ACM Transactions on Interactive Intelligent Systems, 1, 2.12
[12] Kirchner, N., Alempijevic, A., and Dissanayake, G. (2011). Nonverbal robot-group interaction using an imitatedgaze cue. In Proceedings of the 6th International Conference on Human-Robot Interaction. 497-504.
[13] Hayashi K., Shiomi, M. Kanda T., and Hagita N., (2011), Are Robots Appropriate for Troublesome andCommunicative Tasks in a City Environment? IEEE Transactions on Autonomous Mental Development, 150-160.
[14] P. H. Kahn, H. Ishiguro, B. Friedman, and T. Kanda, (2007) What is a human? – Toward psychological benchmarksin the field of human-robot interaction, Interaction Studies, 8, 3, 363-390.
[15] Groom, V., Chen, J., Johnson, T., Kara, F. A., and Nass, C, (2010), Critic, compatriot, or chump?: Responses torobot blame attribution, In Proceeding of the 5th ACM/IEEE International Conference on Human-Robot Interaction,
211-218.
[16] Ho, B, (2009), Apologies as signals: With evidence from a trust game, Management Science. 58, 1, 141-158.
[17] Shiwa T., Kanda T., Imai M., Ishiguro H., and Hagita N., (2009), How Quickly Should a Communication Robot
Respond? Delaying Strategies and Habituation Effects, International Journal of Social Robotics, 1, 2, 141-155.