Evaluating Effects of Hand Pointing
by an Image-based Avatarof a Navigation System
Michiko Inoue1 , Aya Shiraiwa2 , Hiroki Yoshimura2 ,Masashi Nishiyama2,3 , and Yoshio Iwai2
1

Graduate School of Sustainability Science, Tottori University
2
Graduate School of Engineering, Tottori University
3
Cross-informatics Research Center, Tottori University
101 Minami 4-chome, Koyama-cho, Tottori, 680-8550 Japans132007@eecs.tottori-u.ac.jp, nishiyama@eecs.tottori-u.ac.jp

Abstract. We investigate whether the behavior of pointing at a map
by an image-based avatar helps a user understand a route in an imagebased avatar navigation system. We also evaluate whether this behavior
is preferred by the user. Existing avatar-based methods inform the user ofa route by this behavior while talking. However, the existing methods do
not consider how to incorporate a map. Thus, we consider how to informthe user of a route using an image-based avatar that indicates the routeby pointing at a map. In the experiments, after users interacted withthe system, we conducted a route depiction test to determine whether auser was able to correctly understand the route on a map and performed
a questionnaire-based subjective assessment to determine whether theuser liked the image-based avatar system. The results of the experiments
show that the pointing behavior significantly increased the likeability ofthe system but did not help the user understand the route.
Keywords: Navigation system, Hand pointing, Image-based avatar, Routedepiction test, Questionnaire

1Introduction

An interaction system with a life-size display has many potential applications. In
particular, there is a demand for a system that uses an image-based avatar [1, 2,3, 4, 5] to smoothly communicate with users. The avatar provides good usability,
such when talking about past experiences [1] or acting as a guide at a museum [2].
This paper discusses a route navigation system that uses an image-based avatar
for intuitive interaction, that is, as if a real guide were directing the user, asillustrated in Figure 1. We assume the scenario of an information center in a
public space, such as a tourist information office.
In the design of a route navigation system [6, 7, 8, 9], the aim is for the user
to understand the explanation of the route and like using the system. When auser cannot understand the route, he or she will repeatedly ask about the routeIIImage-based avatar

UserFig. 1. Route navigation system that uses an image-based avatar that points at a map.

S1: informUser

S4: understandS2: understandS3: inform

Real guideFig. 2. Cycle of route navigation as performed by people.

and then feel uncomfortable using the system. To avoid this problem, we needto consider the interface between the image-based avatar and user.
When designing a user-friendly interface for route navigation, we aim tomimic the behaviors of a real guide. The real guide generally directs the useraccording the following steps.
S1: The user informs the real guide of the destination.
S2: The real guide understands the destination provided by the user.
S3: The real guide informs the user of the route.
S4: The user understands the route provided by the real guide.
The user’s understanding and liking of the system are determined in the cycleof informing and understanding, as illustrated in Figure 2. In this cycle, S3 isimportant in terms of smoothly satisfying the demands of the user. We thus
focus on developing an interface for S3 using an image-based avatar.
In existing avatar-based methods [10, 11, 12] the avatar informs the userof the route by pointing while talking from a first-person viewpoint. The useof the hand for pointing has merit in that it resembles natural communication
among people. However, existing methods have not considered incorporating amap with a bird’s-eye view. As described in [13], a map is an important part ofhelping a user to understand a route. A real guide frequently presents a routeIIIby indicating it on a map. We thus tackle the challenging issue of how to control
an image-based avatar so that it presents the route on a map.
To this end, we investigated the hypothesis that an image-based avatar thatindicates a route by pointing at a map helps the user understand it. We also
investigated the hypothesis that an image-based avatar that points is preferredby the user. After exposing users to the system, we conducted a route depiction
test to determine whether they correctly understood a route on a map, and thenperformed a questionnaire-based subjective assessment to determine whether
they liked the system.
The rest of this paper is organized as follows. Section 2 describes our experimental design, Section 3 presents the results of the route depiction test, Section 4
describes the results of the questionnaire-based assessment, and our concluding
remarks are given in Section 5.

2

Experimental design

2.1

Overview

We assumed that a user accesses the route navigation system in an information
center. We explored a scenario in which the user would like to visit some destinations in a particular order in a downtown area. We evaluated the effect when theimage-based avatar describes the route on a map using its finger. Twenty-fourparticipants (20 males, four females, average age 21.9 years) participated in thestudy. The details of our experimental design are described below.
2.2

Route on the map

We generated a fictional map containing 3 × 5 square blocks. Figure 3 shows
examples of the routes on the map. A route consists of a start point, destination
points, and path segments. We randomly set the start point and the destination
points during the experiment for each participant. We used six destination points
so that the user would not easily remember them. In general, a human canremember 4 ± 1 items in short-term memory time [14]. We believe that using
six destination points is a valid way to keep participants from easily getting fullmarks in the route depiction test. Furthermore, we set the paths that connect
the destination points so that they did not cross each other. Note that we fixed
the number of the corners in the paths to 12 to keep the experimental conditions
the same.
2.3Interface with the image-based avatar

We tested two interfaces as follows:
I1: An image-based avatar with map pointing,
I2: An image-based avatar without map pointing.

IV
S ： Start

G1 - G6 ： Des(na(ons： PathsFig. 3. Examples of the routes, which consist of a start point, destination points, andpath segments on the map.

We generated the video sequences of the interfaces, as illustrated in Figure 4.
Each participant viewed the video sequence of interface I1 or I2 in random
order. The length of each video sequence was 90 s. The sentences and speed ofthe avatars speech were the same for both I1 and I2.
Figure 5 shows the setup of the interface using the image-based avatar. Eachparticipant stood 1.5 m from the display and viewed the video sequences. We
used an 80-inch display with a resolution off 1, 920 × 1, 080 pixels (Sharp PNA601) to show the life-sized avatar. We placed the voice speaker (Towa electronicTW-S7B) behind the display.

2.4Procedure

To evaluate the hypothesizes for the interfaces, we executed the following procedure:
P1: We displayed the video sequence of the interface for the participant.
P2: We gave the route depiction test to the participant.
P3: We gave the questionnaire to the participant.

VI1

I2

TimeFig. 4. Video sequence of the interface using the image-based avatar with and without
map pointing.

2.0 m

1.5 m

Fig. 5. Setup of the interface using the image-based avatar.

We also gave an easy numerical calculation task to the participant between P2
and P3. We prepared six routes on the map and randomly selected a route whenassessing the interface. The order of interface I1 or I2 was randomly chosen.

3

Route depiction test

3.1

Overview

We prepared a blank map for the participant, as illustrated in Figure 6 (a),before starting the route depiction test. We slightly shifted the viewpoint of thisblank map with respect to the map displayed in the video sequences of Figure 4.
We asked the participants to depict the start point, destination points, and pathsegments at the same scale as displayed in the video sequence. Figures 6 (b) and(c) show the results of the route depiction test with respect to the ground-truthof the route illustrated in Figure 6 (d).

VI

(a)

(b)

(c)(d)

Fig. 6. Examples of the route depiction test. (a) Blank map. (b) and (c) Results of theroute depiction test. (d) Ground-truth.

We evaluated the interfaces using the following three metrics. The first one
was the correctness of the start point and destination points. The second one wasthe correctness of the path segments. The third one was the time taken for theuser to complete the depiction. The details of the metrics are described below.

3.2Metrics in the route depiction test

We first explain the correctness of the start point and destination points. We
evaluated whether a point depicted by the participant in the test was at thesame location as the point displayed in the video sequence. When checking thiscorrectness, we divided the blocks of the map into 3 × 3 = 9 regions. Figure 7 (a)shows an example of the regions in a block. The depicted point was consideredcorrect when it was more than half-way within the same region as the displayedpoint. Figure 7 (b) shows an example of a correct case, and Figure 7 (c) shows anexample of an incorrect case, where the depicted point is shifted by one region.
The maximum number of incorrect points was seven (one starting point and six
destination points).

VII
G1 : Displayed point: Depicted point

G1(a)

(b)

(c)

Fig. 7. Example of correct and incorrect destinations in the route depiction test. (a)Regions in the block. (b) Correct case. (c) Incorrect case.
: Depicted path: Displayed path(a)(b)(c)

Fig. 8. Example of correct and incorrect path segments in the route depiction test. (a)Displayed path. (b) Example of a correct case. (c) Example of an incorrect case.

We next explain the correctness of the path segments. We assigned correctness to a path segment when the depicted segment and the displayed segment
were the same. Figure 8 (a) shows an example of a displayed path segment,
Figure 8 (b) shows an example of a correct case, and Figure 8 (c) shows anexample of an incorrect case, where the path between the destination points wasincorrect even though the locations of the points were correct. The maximum
value of path incorrectness was six.
We finally explain the time taken by the user to create the depiction. We
used a stopwatch to record the times when the participant started the test andwhen he or she finished.
3.3

Results of the route depiction test

Figure 9 (a) shows the number of incorrect start and destination points foreach interface. Figure 9 (b) shows the number of the participants obtaining
an incorrect answer for each point. We used the Wilcoxon signed-rank test todetermine that there is no significant difference (p<.05) between interfaces I1and I2.
Figure 10 (a) shows the number of incorrect paths for each interface. Figure 10 (b) shows the number of the participants obtaining an incorrect path forVIII

Average number ofincorrect points

Number of participants
for each incorrect point

60

I1

24: I1: I20I2

S(a)

G1

G2

G3

G4

G5G6(b)Fig. 9. Results for destination points in the route depiction test.

Average number ofincorrect paths

Number of participants
for each incorrect path

60I2I1(a)6: I1: I2
0

S-G1 G1-G2 G2-G3 G3-G4 G4-G5 G5-G6(b)Fig. 10. Results for path segments in the route depiction test.

each path segment. We used the Wilcoxon signed-rank test and determined that
there is no significant difference (p<.05) between interfaces I1 and I2.
Figure 11 shows the time taken to depict the route. We used the Wilcoxon
signed-rank test and found that there is no significant difference (p<.05) betweeninterfaces I1 and I2.
Hence, we cannot claim that the pointing behavior of the image-based avatar
helps the user understand the route.

4

Questionnaire-based subjective assessment

4.1

Items of the questionnaire

After viewing the video sequences of interface I1 or I2, we asked the participant
the following questions:
Q1: Did you like interacting with the avatar?
Q2: Did the avatar provide a navigation service resembling that of a real guide?
Q3: Was it easy to understand where to go on the map?
Each participant provided a rated score using six response levels (1: disagreeableto 6: agreeable) for each question. We also asked the inverse questions of Q1, Q2,

Average time requiredfor depiction (s)IX
250

0I1I2Fig. 11. Results of the average time taken to depict the map.

and Q3. The purpose of Q1 was to evaluate the hypothesis that the image-based
avatar that points to the map was more liked by the user. The purpose of Q2
was to check whether the behavior of the avatar in the interface was close tothat of a real guide. The purpose of Q3 was to check whether the user felt thatthe avatar that pointed at the map had correctly presented the locations on theroute.
4.2

Results of the subjective assessment

Figures 12 (a) to (c) show the subjective scores of the questionnaire obtainedusing the Wilcoxon signed-rank test. In Figure 12 (a), in terms of Q1, there wasa significant difference between I1 and I2. We can hence claim that the imagebased avatar that points at the map is more liked by the user. In Figure 12 (b) interms of Q2, there was a significant difference between I1 and I2. We can hence
claim that the behavior of the avatar in interface I1 is closer to that of a realguide. In Figure 12 (c), for Q3, there was also a significant difference between
I1 and I2. Therefore, we can also claim that the avatar with pointing behavior
makes the user feel that the avatar has correctly presented the locations on theroute.

5Conclusions

We investigated two hypotheses regarding an interface with an image-based
avatar that points at a map. We evaluated the interface using a route depiction test and a questionnaire-based subjective assessment. We can claim that anavatar that points at a map significantly increases the likeability of the system,
but we cannot claim that this avatar better helps the user to understand theroute.
In future work, we will expand our assessment of the interactive system andintend to develop a method to add an explanation of landmarks, as used in [15],for the our avatar system.

X

65
4
32
1

*
Average subjective score

Average subjective score

*Q1

654
3
2
1Q2(a)(b)

Average subjective score*65
4: I1: I2

32
1* ： p < .05Q3(c)Fig. 12. Subjective scores of the questionnaire-based subjective assessment.

References[1] Artstein, R., Traum, D., Alexander, O., Leuski, A., Jones, A., Georgila, K., Debevec, P., Swartout, W., Maio, H., Smith, S.: Time-offset interaction with a holocaust survivor. In: Proceedings of the 19th International Conference on Intelligent
User Interfaces. (2014) 163–168[2] Robinson, S., Traum, D., Ittycheriah, I., Henderer, J.: What would you ask aconversational agent? observations of human-agent dialogues in a museum setting.
In: Proceedings of the Sixth International Conference on Language Resources andEvaluation. (2008) 28–30[3] Nishiyama, M., Miyauchi, T., Yoshimura, H., Iwai, Y.: Synthesizing realistic
image-based avatars by body sway analysis. In: Proceedings of the Fourth International Conference on Human Agent Interaction. (2016) 155–162[4] Jones, A., Unger, J., Nagano, K., Busch, J., Yu, X., Peng, H.I., Alexander, O.,
Bolas, M., Debevec, P.: An automultiscopic projector array for interactive digital
humans. In: ACM SIGGRAPH 2015 Emerging Technologies[5] Miyauchi, T., Ono, A., Yoshimura, H., Nishiyama, M., Iwai, Y.: Embedding theawareness state and response state in an image-based avatar to start natural userinteraction. IEICE Transactions on Information and Systems E100.D(12) (2017)
3045–3049

XI[6] Darken, R.P., Peterson, B.: Spatial Orientation Wayfinding and Representation,
Handbook of Virtual Environments: Design, Implementation, and Applications.
CRC Press (2002)[7] Devlin, A.S., Bernstein, J.: Interactive wayfinding: Use of cues by men and women.
Journal of Environmental Psychology 15(1) (1995) 23 – 38
[8] Blades, M., Spencer, C.: How do people use maps to navigate through the world?
Cartographica: The International Journal for Geographic Information and Geovisualization 24(3) (1987) 64 – 75
[9] Makimura, Y., Yoshimura, H., Nishiyama, M., Iwai, Y.: Decreasing physical burden using the following effect and a superimposed navigation system. In: Virtual,
Augmented and Mixed Reality. (2017) 533–543[10] Kopp, S., Tepper, P.A., Ferriman, K., Striegnitz, K., Cassell, J.: Trading spaces:
How humans and humanoids use speech and gesture to give directions. In: Conversational Informatics: An Engineering Approach. (2007) 1–26
[11] Bergmann, K., Kopp, S.: Gnetic – using bayesian decision networks for iconic gesture generation. In: Proceedings of the 9th International Conference on Intelligent
Virtual Agents. (2009) 76–89[12] Bergmann, K., Kopp, S.: Knowledge representation for generating locating gestures in route directions. In: Proceedings of Workshop on Spatial Language andDialogue. (2005) 1–13
[13] Taylor, H.A., Tversky, B.: The description of routes: A cognitive approach to theproduction of spatial discourse. 35 (1996) 371–391[14] N, C.: The magical number 4 in short-term memory: a reconsideration of mental
storage capacity. Behavioral and Brain Sciences 24(1) (2001) 87–114[15] Denis, M.: The description of routes: A cognitive approach to the production of
spatial discourse. 16 (8 1997) 409–458