IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x1LETTEREmbedding the awareness state and response state in animage-based avatar to start natural user interaction
Tsubasa MIYAUCHI† , Ayato ONO† , Hiroki YOSHIMURA† , Masashi NISHIYAMA†a) , and Yoshio IWAI†b) ,SUMMARY We propose a method for embedding the awareness stateand response state in an image-based avatar to smoothly and automaticallystart an interaction with a user. When both states are not embedded, theimage-based avatar can become non-responsive or slow to respond. Toconsider the beginning of an interaction, we observed the behaviors between
a user and receptionist in an information center. Our method replayed thebehaviors of the receptionist at appropriate times in each state of the imagebased avatar. Experimental results demonstrate that, at the beginning ofthe interaction, our method for embedding the awareness state and response
state increased subjective scores more than not embedding the states.
key words: Image-based avatar, Beginning of interaction, Awareness state,
Response state

Wait	State	Awareness	State	Response	State	Action	
State	Fig. 1 The image-based avatar requires the above four states to smoothly
start the interaction with the user.

1. Introduction
There is a demand for a system that enables a human andmachine to naturally interact through an avatar [1]–[5]. Sucha system would have the ability to interact with users invarious scenarios, for example, navigating in public spaces(e.g., airport, station, or city oﬃce) throughout 24 hours.
We discuss an image-based avatar system [3]–[5] in which
a video sequence acquired from a real person is used on alife-sized display. We consider that the image-based avatar
waits for a user and starts to navigate toward the user in an
information center. In particular, we focus on the beginningof the interaction until the image-based avatar is asked aquestion regarding the location to which the user wishes togo.
The image-based avatar requires wait, awareness, response, and action states to smoothly start the interaction
with the user. In the wait state, the image-based avatar ison standby because there are no users nearby. In the awareness state, the image-based avatar notices that the user isapproaching, but because the distance from the user is still
substantial, the avatar prepares to respond to the user at anytime. In the response state, the image-based avatar returns abehavior to indicate that it is ready to accept the interaction
with the user. In the action state, the image-based avatar talks
with the user and performs a close interaction. In particular,
the awareness state and response state play important roles
to smoothly start the interaction. If the response state is notembedded in the image-based avatar, it is diﬃcult for the user
Manuscript received January 1, 2015.
Manuscript revised January 1, 2015.
† Graduate School of Engineering Department of Informationand Electronics，Tottori University, Tottori 680–8550, Japana) E-mail: nishiyama@eecs.tottori-u.ac.jpb) E-mail: iwai@eecs.tottori-u.ac.jp
DOI: 10.1587/transinf.E0.D.1

to decide when to start the interaction because the avatar is
non-responsive despite the user approaching. Furthermore,
if the awareness state is not embedded in the image-basedavatar, the approaching user will be confused because thetiming of the response of the avatar is delayed. The existing
methods for image-based avatars mainly focus on the action
state [3], [4] and partly on the wait state [5]. However, these
existing methods do not suﬃciently consider how to address
the awareness state and response state for the image-based
avatar.
In this paper, we propose a novel method for embedding the awareness state and response state in the imagebased avatar to smoothly start an interaction with a user. We
observe the beginning of an interaction between a user andreceptionist in an information center. Based on the observation, we design a model of the receptionist’s behaviors.
Our method replays the behaviors of the receptionist whenstarting an interaction. We synthesize a video sequence of
the image-based avatar to adjust the movements of the user.
The results of a subjective assessment demonstrate that anavatar with the awareness state and response state obtains
higher scores than an avatar without the states.
2.

Design of the model for starting the interaction

2.1

Observation of receptionist

We observed an interaction between a user and receptionist in an information center. In many situations, the user
asked the receptionist for the location of a place, such as aticket counter or boarding gate. The receptionist performed
passive behaviors until she was spoken to by the user. The
receptionist was spoken to by the user infrequently. We interviewed the receptionist after the observation to design themodel of the receptionist’s behaviors. The details of theCopyright © 200x The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x2

S1

Wait for the occurrence of the interaction

Notice the approaching user?

NoYes

S2

Prepare to be able to return the response

Approached by the user?

NoYes

S3

Respond to the user using the behavior

Spoken to by the user?

NoYes

S4

Engage in an interaction with the user

Fig. 2 Model of the receptionist’s behaviors when starting an interaction
with a user.

model are described below.
2.2 Model of the receptionist’s behaviors
Figure 2 shows the model of the receptionist’s behaviors
when starting an interaction with the user. In S1, the receptionist waits for the occurrence of the interaction because
there was no user near to the receptionist. The receptionist’s
behavior involves looking forward or looking at the surroundings. In S2, the receptionist notices the approaching user andprepares to return a response to the user. The receptionist’s
behavior involves turning his/her face toward the user. InS3, the receptionist responds to the user to create an atmosphere in which it is easy to be spoken to. The receptionist’s
behavior involves looking at the user or bowing to the user.
In S4, the receptionist is engaged in a close interaction withthe user, such as a conversation about a location. We asked
why the receptionist performed the aforementioned behaviors. The following situation was reported: The receptionist
waited until the user approached before starting the interaction because a conversation was diﬃcult from a distance in apublic space. Thus, our model uses the distance between theuser and receptionist to determine the transitions from S1 toS2, and S2 to S3. When the receptionist was approached by
the user, the receptionist was asked for a location. The user
first said ‘Excuse me’ and then asked questions. Our model
uses the speech utterance by the user to the receptionist todetermine the transition from S3 to S4.
2.3 Our method for replaying the receptionist’s behaviors
In the following, we design our method for replaying thereceptionist’s behaviors using the image-based avatar. We
describe each state of the receptionist shown in Figure 2 asfollows: action state (S1), awareness state (S2), responsestate (S3), and action state (S4). The avatar performs thebehaviors as follows: looking forward or looking at the surroundings in S1, looking at the user or turning its face toward

the user in S2, looking at the user or bowing to the user in S3,and speaking and hearing in S4. The details of our method
for replaying the behaviors are as follows: When the distancedt from the user to the avatar is threshold D1 or less, the statetransition from S1 to S2 occurs. When dt is D2 or less, thestate transition from S2 to S3 occurs. When the average atof the audio powers, which are acquired from the directionof the user’s standing position, in the short term is threshold
A or greater, the state transition from S3 to S4 occurs. Inadvance, we determine D1, D2, A by measuring the distanceor audio power when transitioning from one state to another.
Our method in real time measures dt , at using a depth sensor
and microphone.
3. Synthesizing the video sequence of the image-based
avatar to replay the behaviors
3.1Video clips to represent the behaviors

The image-based avatar is represented by a video sequenceacquired from a real person. Thus, it is time-consuming toshoot all combinations of the receptionist’s behaviors in advance. Instead of acquiring a video sequence, we consider
acquiring video clips to represent the behaviors in S1, S2,and S3. Note that we do not address how to represent thebehaviors in S4 because we focus on the beginning of theinteraction. The video clip consists of the following: maintaining the initial posture in which the avatar faces the front,
making a movement, and returning to the initial posture. Our
method uses the following video clips:• C1: Keep looking forward in the initial posture.
• C2: Turn left from the initial posture and return to theinitial posture.
• C3: Turn right from the initial posture and return to theinitial posture.
• C4: Bow from the initial posture and return to the initialposture.
A facial image of a frontal direction on a flat display causes
the Mona Lisa eﬀect [6] such that the user feels that he/she isseen from the image. Thus, C1, which derives the eﬀect, isused to represent the behavior of looking at the user in S2 orS3. C1 is also used for the behavior of looking forward in S1.
C2 and C3 are repeatedly used for the behavior of looking at
the surroundings in S1. C2 or C3 is also used for the behaviorof turning the face toward the user in S2. C4 is the behaviorof bowing to the user in S3. Our method joins the video
clips by smoothly switching them, and controls the speed ofreplaying the behaviors by adjusting to the movement of theuser. The details of our method are described below.
3.2

Joining and controlling the video clips

We describe our method for joining the video clips of theimage-based avatar. When switching from one video clip toanother by simply connecting them, a discontinuous motion
is caused because there is a gap between the initial postures inLETTER3Receptionist

User

KinectApproach

1.4 m
14.0 mFig. 3

Setup for measuring the parameters of our method.

the video clips. Our method interpolates between the initial
postures using an optical flow technique and noise reduction
technique.
Next, we describe our method for controlling the speedof replaying the behaviors. To represent the behavior oflooking at the surroundings in S1, our method repeats C2
and C3 within a cycle of T1 . To represent the behaviorof turning the face toward the user in S2, our method fast
forwards or rewinds the video clip that is currently playing.
We finish this operation within T2 = (D1 − D2 )/v, where v isthe user’s walking speed. T2 is automatically computed eachtime using v measured in real time using a depth sensor. To
represent the behavior of bowing to the user in S3, we simply
play C4 within T3 . To represent the remaining behaviors, wecontinuously play the video clip of C1. We determine T1, T3
by observing the length of time of the receptionist’s behavior.
4. Experimental results
4.1 Measuring the parameters of our method to determine
the thresholds
We measured the parameters of the behaviors at the beginning of the interaction to determine the thresholds of ourmethod. We assumed that the receptionist guided the user
in the information center. We collected two receptionists(average age 23.5 ± 0.5; one male and one female; Japanese)and five users (average age 23.5 ± 0.5; four males and onefemale; Japanese). We explained the situation to each user,that is, the user was to ask the receptionist for the location of
the check-in counter at the airport. We assumed that the user
was approaching from the front of the receptionist. Figure 3
shows the setup for the measurement of the parameters. Theuser started walking from a position 14 m away from the receptionist, approached the range for which he/she felt it waseasy to speak, and spoke to the receptionist. We acquired
the parameters of our method twice for each user. From theresults of the measurements, we obtained D1 = 5.6 ± 1.6 m,
D2 = 3.9 ± 0.4 m, A = 20 dB in 0.08 s, T1 = 2.0 ± 0.3 s,
and T3 = 2.7 ± 0.3 s. Note that the average walking speed ofthe user was v = 1.4 ± 0.1 m/s. We used the averages of theparameters for the thresholds of our method.
4.2 Experimental conditions for the subjective assessment
using the image-based avatar
To conduct the subjective assessment of the beginning of theinteraction, we collected 14 subjects (average age 23.0 ± 1.3;

Subject

Display
Approach
1.9 m

14.0 mFig. 4

Setup for the subjective assessment using the image-based avatar.

12 males, and two females; Japanese) who did not participate
in the experiment in Section 4.1. We explained the situation
to each user, that is, the image-based avatar would guide
him/her in the information center. Figure 4 shows the setup
for a subjective assessment using the image-based avatar.
We played the video sequences of the avatar on a verticallyplaced 80-inch display (Sharp PN-A601). To easily find theavatar from a distance, we attached the title ‘information
center’ at the top of the display. To measure the distancefrom the subject to the avatar and the audio power of thesubject, we used a depth sensor and microphone (MicrosoftKinect v2). Each subject started walking from a position 14m away from the avatar, approached the avatar, and spoke tothe avatar. We tested using male and female avatars for eachsubject.
We conducted the subjective assessment to compare thefollowing image-based avatars:• I1: Look forward continuously.
• I2: Look forward and bow to the user when the user
approaches the avatar.
• I3: Look at the surroundings, turn the face toward theuser, and look at the user when the user approaches theavatar.
• I4: Look at the surroundings, turn the face toward theuser, and bow to the user when the user approaches theavatar.
I1 was a baseline method that did not embed the awarenessstate and response state. I2, I3, and I4 were methods thatembedded both states, but the behavior in each state was
diﬀerent. Each subject repeated approaching and speaking
to each avatar. Figure 5 shows the video sequences of I1 toI4. We used the video clips as C1: S1 to S3 in I1 and S3 in
I3; C2 and C3: S1 and S2 in I3 and I4; C4: S3 in I2 and I4.
Note that we simply displayed a message in S4 for all video
sequences. After beginning of the interaction was complete,
we asked the subjects the following questions:• Q1: Was it easy to understand the timing when speaking
to the image-based avatar?
• Q2: Did the avatar resemble the behavior of a real
receptionist?
• Q3: Was it possible to smoothly start the interaction
with the image-based avatar?
• Q4: Did you feel that the image-based avatar behaved
politely when starting the interaction?
Each subject provided a rated score using five response levels(1: disagreeable, 6: agreeable) for each question. We alsoIEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

4Look forward / the surroundings

Turn the face

Look at / Bow to the user

S1S2

S3

I1

I2

I3

I4TimeFig. 5S4

Video sequences of the image based-avatar for the subjective assessment.

asked the inverse questions of Q1 to Q4. We randomly
selected the order in which the video sequence of the avatar
was played for each user.

Figure 6 shows the average and standard deviation of thesubjective scores for each question. The scores of the inverse questions were inverted and added to the corresponding scores for Q1 to Q4. We used the Friedman test, theWilcoxon signed-rank test and Bonferroni correction after
checking the normality of the scores. Comparing the behaviors of looking forward (I1) and looking at the surroundings(I3), marginal diﬀerences (p < 0.05) emerged for Q2 andQ4, but did not emerge for Q3. We believe that it is less
eﬀective for the image-based avatar to perform these behaviors in S1. Comparing the behaviors of bowing to the user
(I2, I4) and looking at the user (I1, I3), significant diﬀerences (p < 0.01) emerged for all questions. Thus, bowing
in S3 was more agreeable to the user. We believe that it isimportant to embed the awareness state (S2) and response
state (S3) in the image-based avatar at the beginning of theinteraction.
5. Conclusions
We proposed a method for embedding the awareness state
and response state in an image-based avatar to smoothly
start an interaction with a user. We designed a model ofthe receptionist’s behaviors by observing an interaction in
an information center. Our method joined and controlled
video clips to replay the receptionist’s behavior. Experimental results demonstrate that the use of the awareness state
and response state worked well for the image-based avatarsystem. In future work, we will expand our assessment of
the interactive system, and we intend to develop a method tocombine the behaviors in the action state.

**
*****
*

** ** **** ** **** ** **** ** **

6Subjective score

4.3 Results of the subjective assessment using the imagebased avatar

**
**

543
2
1

Q1：I1

Q2：I2

Q3：I3

Q4：I4Fig. 6 Results of the subjective assessment (∗∗ : p < 0.01, ∗ : p < 0.05)
using the image-based avatar.

References[1] S. Robinson, D. Traum, I. Ittycheriah, and J. Henderer, “What would
you ask a conversational agent? observations of human-agent dialogues in a museum setting,” Proceedings of the Sixth International
Conference on Language Resources and Evaluation, pp.28–30, 2008.
[2] A. Lee, K. Oura, and K. Tokuda, “Mmdagent - a fully open-source
toolkit for voice interaction systems,” Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing,pp.8382–8385, 2013.
[3] R. Artstein, D. Traum, O. Alexander, A. Leuski, A. Jones, K. Georgila,
P. Debevec, W. Swartout, H. Maio, and S. Smith, “Time-oﬀset interaction with a holocaust survivor,” Proceedings of the 19th International
Conference on Intelligent User Interfaces, pp.163–168, 2014.
[4] A. Jones, J. Unger, K. Nagano, J. Busch, X. Yu, H.I. Peng, O. Alexander, M. Bolas, and P. Debevec, “An automultiscopic projector arrayfor interactive digital humans,” ACM SIGGRAPH 2015 Emerging
Technologies, p.6:1, 2015.
[5] M. Nishiyama, T. Miyauchi, H. Yoshimura, and Y. Iwai, “Synthesizing
realistic image-based avatars by body sway analysis,” Proceedingsof the Fourth International Conference on Human Agent Interaction,
pp.155–162, 2016.
[6] V. Bruce and A. Young, In the eye of the beholder: The science of faceperception, Oxford University Press, 1998.