This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

VISTURE: A System for Video-Based Gesture and Speech
Generation by Robots
Kaon Shimoyama

Kohei Okuoka

shimoyama@ailab.ics.keio.ac.jp
Keio University
Kohoku, Yokohama, Kanagawa, Japanokuoka@ailab.ics.keio.ac.jp
Keio University
Kohoku, Yokohama, Kanagawa, Japan

Mitsuhiko Kimoto

Michita Imaikimoto@atr.jp
ATR/Keio University
Seika, Soraku, Kyoto, Japanmichita@keio.jp
Keio University
Kohoku, Yokohama, Kanagawa, Japan

ABSTRACTThis paper proposes VISTURE, a system for generating a robot’s
gesture and speech by using video as input. VISTURE assumes asituation in which a robot conveys what it saw with a camera toa person who was absent. The value of this paper is that we have
performed a case study to investigate the expressions that Japanesepeople use to describe video scenes, and used the results to build
VISTURE. In particular, we found classi�cation of expressions depicting the video scenes throughout the case study: Foregroundinformation that is the relevant event of the scene and Background
one that is not the main point of the description giving the entire
scene. Foreground and Background are referred in combination.
VISTURE employs the classi�cation to generate human-like expressions. Moreover, we designed the method to determine Foregroundand Background, and it can generate multiple combinations ofexpressions. We investigated the people’s impression of a robot
performing the gestures and speech generated by VISTURE to evaluate the quality of those gestures and speech. The results showedthat the robot was perceived as more likable and capable when it
performed gestures.

CCS CONCEPTS• Human-centered computing ! Human computer interaction (HCI).

KEYWORDS
Human-robot interaction, Gesture generation, Speech generation

Figure 1: The human on the right is imaging the situationthat several people are walking and running near a streetcar.
If he focuses on the crowd (green circle), he may say “It wasvery crowded.” If he focuses on the streetcar (yellow circle),he may say “I saw a streetcar.”1

INTRODUCTION

Humans commonly communicate something they have seen to
people in multimodal ways including speech, gaze, and gestures.
In this paper, we focus on a robot which uses video acquired fromits own camera and conveys what it saw in a multimodal way.
There are many previous works that use gestures to convey what
it sees. ([12, 19]). Nihei et al. proposed a method to generate gesturesthat represent objects’ shape by using images as input [19]. For text
generation using video, there is video captioning ([7, 10, 15, 26]).
Donahue et al. proposed generating captions that express the input
videos by using long short-term memory (LSTM) [7].
However, previous works do not take into account information
selection. When humans conveys what they saw to someone who
did not see, they are inherently capable of selecting information. For
example, in a scene where a streetcar is running through a crowd, as
illustrated in Figure 1, whether to focus on the crowd or the streetcar,
or whether to focus on other aspects of the scene, will depend on
the person and the situation. In particular, if the place is famousfor being always crowded, humans would not mention the crowds,
but if it is not, humans would mention the crowds. This needs tobe taken into account when generating robot representations.

This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

HAI ’22, December 5–8, 2022, Christchurch, New ZealandIn this study, we consider a system that selects information andgenerates gestures and speech with reference to human representations, assuming a situation in which a robot conveys what it
saw to someone who did not see it. We performed a case study
to investigate the expressions that people use to describe video tosomeone who don’t see it, and used the results to build VISTURE.
VISTURE is a system for generating a robot’s gesture and speech byusing video as input, assuming a situation in which a robot conveys
what it saw with a camera to a person who was absent. VISTURE
can make human-like expressions inspired by humans’ expressions.
In particular, we found classi�cation of expressions depicting thevideo scenes throughout the case study: Foreground informationthat is the relevant event of the scene and Background one thatis not the main point of the description giving the entire scene(See Table 2 for details). Foreground and Background are referredin combination. Moreover, we designed the method to determine
objects to be mentioned as Foreground and Background, and it cangenerate multiple combinations of expressions. VISTURE computes
features based on the motions of objects in the video and thendecides which objects to represent.
The rest of this paper is organized as follows. Section 2 introduces
related research. Section 3 describes the target system and the datacollection, and Section4 explains the details of VISTURE. Section 5
details the experiments, and Section 6 presents the results of theexperiments. Section 7 discusses the results and Section 8 concludes
the paper.

2 RELATED WORK
2.1 Gestures Explaining Scenes
Gestures are important in communication ([3, 5, 11, 13, 20, 23]).
Cabibihan et al. showed that humans can understand spatial locations by using ambiguous explanations combined with pointing
gestures, as well as by using clear explanations [5]. Dijk et al. foundthat the performance of gestures related to the speech content helps
humans remember the verbs corresponding to the gestures [23].
Gestures are the one of the e�ective methods of explaining
scenes, and many works conduct on this topic. There are several
approaches generating gestures to explaining scenes ([12, 19]). Nihei et al. prepared seven types of gestures that express shapes ofobjects and developed a system that decides the appropriate gesture
to represent the shape of an object in an input image [19]. Kadono
et al. proposed the system which receives texts as input, checks the
pre-prepared dictionary of gestures for each noun obtained frommorphological analysis, and performs one of the three pre-preparedgestures if the noun is de�ned in the dictionary [12].

2.2

Texts Explaining Scenes

Text is also the one of the e�ective methods of explaining scenes[1, 21, 27, 28]. Image captioning is the task of generating text that describes the content of an image. Selvaraju et al. proposed Gradientweighted Class Activation Mapping (Grad-CAM) that can grasp
each neuron’s importance for the decision of interest [21]. Video
captioning is the task of generating text that describes the content ofa video. Previous research on video captioning uses LSTM or transformer [7, 10, 15, 26]. Yan et al. proposed the STAT video caption
framework, which uses the spatial-temporal attention mechanism

Shimoyama, et al.

(STAT) to exploit the temporal and spatial structure of video [26].
Man et al. proposed a scenario-aware recurrent transformer (SART)
that uses a recurrent transformer and includes scenario understanding module [15]. Focusing on the shared memory that a robot anda user acquire during the same experience, Matsumoto et al. developed a computational model of memory recall of visited places, anda robot which responds to a user using the model [17].

2.3Problem Setting

Previous works do not consider how to select information to convey
what it sees to someone who did not see. Humans inherently selectinformation and the representations should be di�erent dependingon the person or the situation. In this study, we performed a casestudy to investigate humans’ expressions that conveys what they
saw to someone who didn’t see it. And, based on the �ndings givenby the case study, we designed the criteria (see Section 4.3) of selecting information from the input and created a system that generates
gestures and speech from the information selected according to thecriteria.

3 CASE STUDY
3.1 Overview
People commonly communicate something they have seen to people
who didn’t see. Hence, we aim to build a system that enables a robot
to communicate a scenario that it saw to people who did not seethat scenario. For a system to generate gestures and speech basedon an input video, it must have a function to decide what to talkabout in the video. We thus performed a case study to investigate
the expressions that Japanese people use to describe video, withthe goal of �nding classi�cation for judging what they talk about.

3.2

Procedure

We asked experimental participants to watch videos and then explain in one sentence how they would describe each video to someone who had never seen it. The case study was conducted througha Japanese crowdsourcing platform with 30 participants. (20 male, 7
female and 3 undisclosed; age 30–55 years old), who each watched
10 videos. Each video was 10 seconds long and was randomly selected from the video description dataset VATEX [25]. VATEX uses
video from Kinetics-600 [6] validation and holdout test sets, andeach video has 10 English and 10 Chinese captions. The videos focus
on human behavior and include scenes such as playing musicalinstruments and shaking hands. Expressions for the same thing
may change depending on the culture and language. Therefore,translations of these captions could not be used, and descriptions
were collected in Japanese.

3.3

Results and Analysis

We collected 300 descriptions in total; Table 1 gives examples fromtwo videos. We analyzed the descriptions to investigate the tendencies in how the participants described the videos. First, we divided
the collected descriptions into clauses and extracted the subjects
and predicates. Next, we classi�ed the clauses into two groups:
those with a relation to the subject and predicate (main idea), andthose with no relation. We judged these relations by whether orThis is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

VISTURE: A System for Video-Based Gesture and Speech Generation by Robots

HAI ’22, December 5–8, 2022, Christchurch, New Zealand

Table 1: Examples of collected descriptions
Video

https://youtu.be/i2F6HrZKo34[16]Descriptions (translated from Japanese)A man kicks another man who is holding
crutches with both arms while smoking
a cigarette in his mouth.
A man standing on crutches is suddenly
kicked by another man.
A couple are dancing salsa in a room.

https://youtu.be/qcP1mb1IjjI[9]Figure 2: System con�guration.

A man and a woman are dancing
passionately in a room somewhere.

4
Table 2: De�nitions for determining what to talk about
Object to
talk aboutDe�nition

Event

An object whose motion has changed fromits initial state.

ForegroundAn element that is the main idea ofthe description: either an Event, or somethingthat is not an Event but is moving.

BackgroundAn element that is not the main ideaof the description and gives the entire scene.

VISTURE: SYSTEM FOR SPEECH ANDGESTURE GENERATION BASED ON VIDEO
4.1 System Structure
In this paper, we build VISTURE (VIdeo-based Speech and gesTURE), a gesture and speech generation system for robots that isbased on video input. From an input video, VISTURE generates
multiple candidate objects to talk about, ranks the candidates, andgenerates a gesture and utterance that express the chosen object’s
motion. VISTURE uses the de�nitions in Table 2, from the results ofour case study, to determine the object to talk about and to generatespeech. The system con�guration is shown in Figure 2. VISTURE is
composed of four functions: a video feature extraction function, an
information selection function, and a behavior generation function.
When a video is inputted, VISTURE outputs the robot’s gesture andspeech.

4.2
not the subject and predicate were a�ected when the clause wasdeleted. Finally, we examined the tendencies of the subjects andpredicates in the descriptions of each video.
As a result, we found three tendencies. First, in about 89% of
the descriptions for videos with behavior changes from the initialstates, the main ideas in the descriptions involved changes, suchas “things fell” or “things stopped.” Second, when there was nochange of behavior, the main ideas of about 93% of the descriptions
tended to involve motions, such as “dancing” or “moving.” Third,
about 32% of the descriptions express elements that had little to dowith the main ideas in descriptions. These references to extraneous
elements were seemingly intended to make it easier for the person
receiving the explanation to picture the scene. From this analysis,
we developed the de�nitions listed in Table 2 and used them toguide our system in determining what to talk about in a video
and generating utterances accordingly. We de�ned Event as anelement whose state has changed from its initial state. We de�ned
Foreground as an element which is the main idea of the description.
It is an Event, or something that is not an Event but is moving. We
de�ned Background as an element which is not the main idea of thedescription, but conveys an image of the entire scene. Foregroundand Background are referred in combination.

Video Feature Extraction Function

This function gets the classes of all the objects in the input video
and their coordinates in each frame. For this paper, we used Yolov5+ Deep Sort with PyTorch [4] for object detection and tracking.
The classes of the detected objects were based on Microsoft COCO
dataset [14].

4.3

Information Selection Function

This function determines the object to talk about by using thede�nitions in Table 2. Speci�cally, VISTURE determines the ranking of Foreground and Background candidates to talk about. First,
the information selection function receives a set of the detected
objects, >8 , 8 = 0, . . . , # , from the video feature extraction function as input. Then, it uses the coordinates of the detected objects to calculate E8 9 how well they match each criterion 2 9 , 9 =5 4;;, . . . , <>E8=65 0BC, . . . , <>E8=6B;>F, . . . , 148=6;0A64 via a function ⇠ !⇠* ! ) ⇢_" )⇠ #⌧_2 9 :E8 9 = ⇠ !⇠* ! ) ⇢_" )⇠#⌧_2 9 (>8 )(1)

The criteria are listed in Table 3. Based on the results in Section
3, we devised these criteria that could be computed from the coordinates of the recognized objects. For Event, there may be no
applicable object, in which case Event is not taken into account. For
each criterion, the function normalizes the degree to which eachobject �ts the criterion, with a value of 1 for the object that �ts

This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

HAI ’22, December 5–8, 2022, Christchurch, New ZealandShimoyama, et al.

Table 3: Criteria for the object to talk about
Criterion

Classi�cation
Foreground

Event
Object State

Background- fell- stopped- started- moving fast- moving long distance- moving from outside the frame
to inside- moving slow
- having many objects ofthe same class- being large

the criterion the best and a value of 0 for the object that �ts it theworst. Then, for each criterion, the di�erence, denoted as B2>A4 9 ,between the maximum normalized value < 9 and the next highest
normalized value = 9 is calculated:

Figure 3: Example of an input video [8] and the robot’s generated gesture. The yellow circle represents the Foregroundand the light blue circle represents the Background. The robot uses its right arm to perform a Background gesture andits left arm to perform a Foreground gesture. The class of
the Background object is “sports ball,” and its criterion is“moving slow.” The class of the Foreground object is “people,”and its criterion is “stopped.” The objects’ coordinates andthe robot’s hand position are reversed because VISTURE assumes a situation in which the robot conveys what it saw
and generates the gestures based on the robot’s viewpoint.

(3)

from the hand position in an upright state, which is the robot’s
default posture, the robot advances to the start of the gesture.
The robot uses an arm to perform a Foreground gesture. When
VISTURE uses the “moving slow” and “being large” criteria in Table
3 as the Background classi�cation, the robot uses the opposite
arm to describe the Background object, depending on whether theobject’s coordinates are closer to the left or right side of the frame,
and the hand position corresponds to the object’s coordinates. Then,the robot uses the other arm to describe the Foreground. When“having many objects of the same class” is the Background criterion,
the robot uses a metaphoric gesture with arms spread (prepared inadvance) as the Background gesture to express a large number ofobjects, and it performs another gesture to express the Foreground.
Metaphoric gesture is the one of the classi�cations of gestures by
McNeill, and represent abstract concepts [18]. We determined thisgesture based on McNeill’s classi�cation. When “moving fast” is theForeground criterion, VISTURE makes the robot move faster thanwhen using the other criteria. In addition, for the “moving long
distance” and “moving from outside the frame to inside” criteria,
it makes the working range of the robot’s arm larger than for theother criteria. We used SciPy [24] to calculate the angle of thearm joint from the position of the robot’s hand. If we used an
object’s coordinates in all frames to generate a gesture, the gesturewould be staggered; thus, we average the coordinates every 0.4
seconds. Figure 3 gives an example of an input video and the robot’s
generated gesture.

4.4.1 Gesture Generation. From the coordinates of the chosen object to talk about, VISTURE generates a gesture for which therobot’s hand position when viewed from the front corresponds tothe object’s coordinates. VISTURE assumes that the robot has seen
the input video, and that the robot’s hand positions are horizontally�ipped from the object’s coordinates. In addition, it adjusts thelength of the gesture to match the length of the generated speech,
so that the gesture is not much longer or shorter. To match gesturetiming with that of the speech, if a gesture’s starting point is far

4.4.2 Speech Content Generation. VISTURE generates utterances
based on the criteria used to determine the object to talk about andthe class of the object. For the class names of the Background andForeground objects to be talked about, VISTURE combines each
one with a sentence that is randomly selected from candidates foreach criterion that was used in selecting that object. The generated
speech refers to the Background and Foreground. For example,
suppose that the class of the Background object is “car,” and thatthe criterion is “moving slow.” Suppose further that class of theForground object is “person,” and that the criterion is “stopped.”

B2>A4 9 = < 9

=9

(2)

The object that best �ts each criterion is chosen as a candidate
to talk about. For example, for the criterion “moving long distance,” the information selection function calculates each detected
object’s moving distance in relation to its appearance time by⇠ !⇠* ! ) ⇢_" )⇠ #⌧_2<>E8=638BC0=24 (>8 ). It then normalizes the moving distance and calculates B2>A4<>E8=6;>=638BC0=24 .
The object with the longest moving distance is mentioned whenthe expression is generated based on this criterion.
As described in Section 3, objects that are Foreground candidates
tend to be objects that �t the Event de�nition. Accordingly, if an
object �ts the Event de�nition, it is given priority as a Foregroundcandidate, regardless of the B2>A4 values obtained for the non-Event(Object State) criteria.
As noted in Table 2, the Background is not the main point of thedescription. Therefore, to create combinations of the Foregroundand Background candidates from each criterion, after selecting a
Foreground candidate, a Background candidate is selected as an
object that is recognized as belonging to a di�erent class than the
Foreground object. Then, Foreground and Background combinations are created. VISTURE determines that they are suitable for
output in order of the sum of their B2>A4 values:
B2>A4C>C0; = B2>A4 >A46A>D=3 + B2>A4⌫02:6A>D=3

4.4

Behavior Generation Function

This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

VISTURE: A System for Video-Based Gesture and Speech Generation by Robots

In this case, VISTURE generates a sentence such as “A person
stopped, while there was a car in the background.” Speci�cally, for
the Background object, “There was” would be randomly selectedfrom the candidates for the criterion “moving slow,” and “a car”
is the class name of the object talked about; thus the resulting
sentence is “There was a car.” As for the Foreground object, “isstopped” would be randomly selected from the candidates for thecriterion “stopped,” and “a person” is the class name, resulting inthe sentence “A person stopped.” Finally, by connecting these twosentences in the order of Background and Foreground, VISTURE
generates the sentence “A person stopped, while there was a car in
the background.”

4.5

HAI ’22, December 5–8, 2022, Christchurch, New Zealand

Table 4: Questionnaire 1 (translated from Japanese)
MeasureItem

GestureQ1
Q2
Q3

SpeechQ5Robot

For the robot, we used Pepper, which was developed by SoftBank
Robotics [22]. It has two degree of freedoms in its head, six in its
arm, two in its waist, and one in its knee. Pepper is 121 cm tall andweights 29 kg.

Q6Overall
representation

Proposed
The robot performed both the gesture and utterance withthe highest ranking among the gestures and utterances generated by VISTURE.
Speech Only
The robot performed the same speech as under the Proposed
condition but without a gesture.
Baseline
The robot performed a gesture and utterance without referring to the Background. It talked about the same object as
under the Proposed and Speech Only conditions, and it used
Object State as the selection criterion for the Foregroundobject. We used this condition to measure the e�ect of Background and Event mentions, as found in the case study.

5.2

Procedure

We explained to the participants that the robot would express ascene that it had actually seen by focusing on the motion in thatscene. Participants watched three videos generated from an input
video for the three experimental conditions, as well as the input
video. The participants �rst watched a video of the robot’s performance. They then watched the input video, which was assumed
to be the actual scene viewed by the robot, and they answered aquestionnaire. The sequence of steps from watching the video ofthe robot to answering the questionnaire was repeated for each
video under each of the three experimental conditions. We counterbalanced the order of the presented videos.
For the input videos, we randomly selected four from the video
description dataset VATEX. As described in Section 3, VISTURE
uses objects recognized in di�erent classes as the Foreground and

Q7
Q8
Q9

5 EXPERIMENTS
5.1 Conditions
We conducted an experiment with an online questionnaire survey
to investigate the e�ects of gesture expression with VISTURE andthe �ndings from our case study. We compared the following three
conditions in an experiment having a within-subjects design.

Q4

Questionnaire content
The robot’s gesture was appropriate
for the input video.
The robot’s gesture was appropriate
for the speech content.
The robot’s gesture was natural.
The robot’s speech was appropriate
for the input video.
The robot’s speech was appropriate
for the gesture.
The robot’s speech was natural.
The robot’s explanation wasappropriate for the input video.
I understood what the robot wastrying to tell me.
The robot’s explanation wassu�cient for the input video.

Background. Accordingly, we used videos in which more than two
objects appeared and at least two or more classes of objects were recognized. From these four input videos, VISTURE generated expressions that mentioned the Foreground object, which was determinedaccording to the Event criteria in all the videos.

5.3

Measurements

We prepared Questionnaire 1, listed in Table 4, to investigate theparticipants’ impressions of the robot’s gestures and utterances.
The items on Questionnaire 1 were evaluated on a 7-point Likert
scale, where 1 was the most negative response and 7 was the mostpositive. We also used the Godspeed questionnaire [2], which isa scale to evaluate the perceived impressions of robots from �veperspectives: anthropomorphism, animacy, likeability, perceived
intelligence, and perceived safety. We used all of these perspectives
except the perceived safety. The Godspeed questionnaire items
were evaluated on a 5-point Likert scale, where 1 was the mostnegative response and 5 was the most positive.

5.4

Expected Results

We expected that the responses for anthropomorphism and likeability would be higher under the Proposed and Baseline conditions than under the Speech Only condition, because Salem et al.
found that people perceive a robot speaking with gestures as morehuman-like and likeability [20]. Accordingly, we made the following
prediction:
Prediction 1: As compared to the Speech Only condition, therobot’s performance under the Proposed and Baseline conditions
will be rated higher on the Godspeed questionnaire.
We also expected that the robot’s likeability would be increasedby taking the Background and Event into account, i.e., under the
Proposed condition. In the case study, the majority of participants
mentioned an Event as Foreground when an Event occurred. Thissuggests that inclusion of the Background and Event can moreThis is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

HAI ’22, December 5–8, 2022, Christchurch, New ZealandShimoyama, et al.

Figure 4: Results of Questionnaire 1 for all videos. The error
bars indicate the standard errors.

Figure 5: Results of the Godspeed questionnaire for all videos.
The error bars indicate the standard errors.

Table 5: Results of Questionnaire 1 for all videos.
Item
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9

(1.819, 141.890) = 3.021(1.859, 145.009) = 4.167(2, 156) = 5.630(1.416, 110.417) = 2.214(2, 156) = 9.735(1.763, 137.513) = 1.338(1.526, 118.995) = 0.722(1.368, 106.728) = 1.134(1.235, 96.359) = 3.849

?

[2

0.057
0.020
0.004
0.130
0.001
0.265
0.453
0.308
0.044

0.04
0.05
0.07
0.03
0.11
0.02
0.01
0.01
0.05

strongly express the characteristics of a video, and we thus madethe following prediction:
Prediction 2: As compared to the Baseline condition, the robot’s
performance under the Proposed condition will be rated higher onboth Questionnaire 1 and the Godspeed questionnaire.

5.5

ParticipantsThrough a Japanese crowdsourcing service, we recruited 20 participants for each input video, giving a total of 80 participants for four
input videos (51 male, 20 female, 9 undisclosed; age 20–65 yearsold). We removed one participant from the analysis as an outlier
because the participant’s response time for the questionnaire was
308 s, whereas the median response time was 664 s.

6

RESULTS

Figure 4 shows the results of Questionnaire 1. We conducted a oneway repeated-measures ANOVA on the results. What kind of video
was used is not treated as an independent variable. It revealed asigni�cant e�ect for Q2, Q3, Q5, and Q9 between the conditions. In
contrast, there was no signi�cant e�ect for Q1, Q4, Q6, Q7, or Q8.
Table 5 shows details.
We also conducted multiple comparisons using the Bonferroni
method, which revealed signi�cant di�erences for Q2 between the
Proposed and Speech Only conditions (? = 0.035), and between
the Speech Only and Baseline conditions (? = 0.038). For Q3, we
found a signi�cant di�erence between the Speech Only and Baseline
conditions (? = 0.003). For Q5, there were signi�cant di�erences
between the Proposed and Speech Only conditions (? = 0.005),
and between the Speech Only and Baseline conditions (? < 0.001).
Lastly, there were no signi�cant di�erences for Q9.

Table 6: Results of the Godspeed questionnaire for all videos.
Itemanthropomorphism
animacy
likeability
perceived intelligence(2, 156) = 7.269(2, 156) = 40.233(2, 156) = 15.598(1.765, 137.640) = 10.409

?

[2

0.001
0.001
0.001
0.001

0.09
0.34
0.17
0.12

Next, Figure 5 shows the results of the Godspeed questionnaire.
We found a signi�cant e�ect between the conditions for all four
tested perspectives. Table 6 shows details.
For the Godspeed questionnaire, we again conducted multiple
comparisons using the Bonferroni method. We found signi�cant differences for anthropomorphism between the Proposed and Speech
Only conditions (? = 0.002), and between the Speech Only andBaseline conditions (? = 0.008). For animacy, likeability, perceived
intelligence, there were signi�cant di�erences between the Proposed and Speech Only conditions (? < 0.001), and between the
Speech Only and Baseline conditions (? < 0.001).

7 DISCUSSION
7.1 Implications
For all items on the Godspeed questionnaire, we found signi�cantdi�erences between the Proposed and Speech Only conditions, andbetween the Speech Only and Baseline conditions. Because therobot’s performance under the Proposed and Baseline conditions
was rated higher than under the Speech Only condition, Prediction
1 was validated. VISTURE generates gestures that express motion,
and we found that the participants’ impressions of the robot wereenhanced when a gesture expressing motion was performed alongwith speech. In other words, it is worthwhile for a robot to perform
gestures that express motion.
On the other hand, we observed no trend for robot expressions
that mimicked human expressions of the Background and Event.
In other words, we could not validate Prediction 2. However, by
examining the trends for the individual videos, we found cases inwhich expressions that included the Background and Event were
evaluated better or worse. We describe these �ndings in the next
section.

This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

VISTURE: A System for Video-Based Gesture and Speech Generation by Robots

Figure 6: Results of Questionnaire 1 for Video 1. The error
bars indicate the standard errors.

7.2

HAI ’22, December 5–8, 2022, Christchurch, New ZealandFigure 7: Results of the Godspeed questionnaire for Video 1.
The error bars indicate the standard errors.

E�ects of Events and Background
Descriptions

Here, we discuss when a representation taking the Event and Background into account works well, when it does not, and what kindof representation should be used. The input video a�ected whether
the robot’s performance was evaluated more highly under the Proposed condition or the Baseline condition; hence, we focus on thisdi�erence.
For all of the input videos used in the experiment, VISTURE generated representations in which the Foreground object was obtained
from an Event. Therefore, the di�erence between the Foreground
representations under the Proposed and Baseline conditions waswhether or not the target object to talk about was obtained froman Event, and whether or not the generated expression took the
Background into account.
For Video 1 of a man throwing a ball, the robot’s performancewas evaluated more positively under the Proposed condition thanunder the Baseline condition. By conducting a one-way repeatedmeasures ANOVA and multiple comparisons using the Bonferroni
method, we found signi�cant di�erences between the Proposed andBaseline conditions. Figure 6 shows the results of the multiple comparisons test for Questionnaire 1. The test identi�ed the following
results: for Q1, Proposed > Speech Only (? = 0.006) and Proposed> Baseline (? < 0.001); for Q2, Proposed > Baseline (? = 0.006); for
Q3, Proposed > Baseline (? = 0.044); for Q4, Speech Only > Baseline (? = 0.035); for Q5, Proposed > Speech Only (? = 0.017) and
Proposed > Baseline (? = 0.011); for Q7, Speech Only > Baseline(? = 0.028) and Proposed > Baseline (? = 0.027); and for Q9, Speech
Only > Baseline (? = 0.004) and Proposed > Baseline (? = 0.001).
Overall, these results show that the robot’s performance for Video
1 was better evaluated under the Proposed condition than underthe Baseline condition.
Next, Figure 7 shows the results of the multiple comparisons test
for the Godspeed questionnaire. The test identi�ed the following
results: for anthropomorphism, Proposed > Speech Only (? = 0.002)
and Proposed > Baseline (? = 0.010); for animacy, Proposed >Speech Only (? < 0.001), Speech Only > Baseline (? = 0.001), andProposed > Baseline (? < 0.001); for likeability, Proposed > Speech
Only (? = 0.039); and for perceived intelligence, Proposed > Speech
Only (? = 0.009). Overall, these results further indicate that theperformance for Video 1 was better evaluated under the Proposed
condition than under the Baseline condition.
To a human observer, Video 1 appears to show an Event occurring during the video. The intent of mentioning the Background is

Figure 8: Results of Questionnaire 1 for Videos 2 and 3. The
error bars indicate the standard errors.

to make the explanation more detailed, and it is thus better to doso than not to do so. As a result, because there was no discrepancy
between Video 1 and the representation generated by VISTURE,
the robot’s performance was better evaluated under the Proposed
condition, which provided a more detailed explanation than underthe Baseline condition.
In contrast, for Video 2 of boys practicing dribbling and Video
3 of boys riding a scooter, the robot’s performance was evaluatedmore positively under the Baseline condition than under the Proposed condition. We again conducted a one-way repeated-measures
ANOVA and multiple comparisons using the Bonferroni method,
and we found signi�cant deferences between the Proposed andBaseline conditions. Figure 8 shows the results of the multiple comparisons test for Questionnaire 1. The test identi�ed the following
results: for Q1, Baseline > Speech Only (? = 0.004) and Baseline >Proposed (? = 0.011); for Q2, Baseline > Speech Only (? = 0.001)
and Baseline > Proposed (? = 0.010); for Q3, Baseline > Speech
Only (? < 0.001) and Baseline > Proposed (? = 0.049); for Q4, Baseline > Speech Only (? = 0.001) and Baseline > Proposed (? = 0.003);
for Q5, Baseline > Speech Only (? < 0.001) and Baseline > Proposed (? = 0.002); for Q7, Baseline > Speech Only (? = 0.031) andBaseline > Proposed (? = 0.036); for Q8, Baseline > Speech Only(? = 0.023); and for Q9, Baseline > Speech Only (? = 0.007). Overall,
the robot’s performance for Videos 2 and 3 was better evaluatedunder the Baseline condition than under the Proposed condition.
Next, Figure 9 shows the results of the multiple comparisons test
for the Godspeed Questionnaire. The test identi�ed the following
results: for animacy, Proposed > Speech Only (? = 0.001) and Baseline > Speech Only (? < 0.001); for likeability, Proposed > Speech
Only (? = 0.006) and Baseline > Speech Only (? < 0.001); and forperceived intelligence, Baseline > Speech Only (? < 0.001) andThis is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

HAI ’22, December 5–8, 2022, Christchurch, New ZealandFigure 9: Results of the Godspeed questionnaire for Videos 2
and 3. The error bars indicate the standard errors.
Baseline > Proposed (? = 0.013). Overall, these results further con�rm that the performance for Videos 2 and 3 was better evaluatedunder the Baseline condition than under the Proposed condition.
To a human observer, in contrast to Video 1, Videos 2 and 3
do not appear to show an Event occurring during the video. It
seems that there was a representation mismatch under the Proposed
condition, whereas the mismatch did not occur under the Baseline
condition because the Event was not considered. The main cause of
the representation mismatch seems to be that VISTURE generates
representations by using only coordinate changes. Comparing the
VISTURE output results with the input Videos 2 and 3, we canobserve that there were objects whose coordinates changed very
little in the videos. As a result, the “stopped” criterion for an Eventwas used to determine the target object to talk about. However, theobjects were still moving in the actual input videos, though thecoordinate changes were small. For example, Video 2 shows boys
kicking a soccer ball for dribbling practice and trying to go arounda colored cone. Because the boys’ coordinates obtained by the video
feature extraction function changed very little when they tried to
go around the cone, the information selection function decided thatthe “stopped” criterion for an Event was satis�ed. Thus, we foundthat it is not possible to detect all human actions from coordinatechanges alone.
The robot’s representation taking an Event into account did notnecessarily improve its evaluation by the participants. Expressions
that take an Event into account may be e�ective for objects such as
cars, whose actions are directly related to coordinate changes. However, humans may perform actions without changing coordinates,in which case taking an Event into account may result in representation mismatches. For the case when the target Foreground object
is a human, we should consider either generating a representationthat does not take an Event into account or introducing an action
classi�cation. This knowledge of meta-level rule can also bring useful insights to architectural design when creating models in deeplearning.

7.3

Limitations

We note here that our study has some limitations. The robot’s
gesture generation controls only the hand’s position and the rangeof motion of the robot’s arms is limited to two dimensions. It isbecause its gestures are generated from the coordinates of objects
in a video, but gestures are normally three-dimensional. It would beinteresting to conduct experiments that focus more on the e�ects ofgestures, such as comparing speech-related gestures with random
gestures. Regarding the limitations of the experimental design, theShimoyama, et al.

results may not be generalizable across cultures, as expressions vary
among di�erent languages and cultures. In addition, the experiment
was conducted with only one type of robot. Because Pepper has ahuman-like appearance, di�erent trends may emerge when using
a robot with a simpler appearance. Although the viewpoints of
the input videos should be an important factor because a robot isintended to describe a view that it has seen, we did not accountfor it in this study. Our experimental design limited the possible
types of communication. In the future, we aim to investigate moreinteractive settings by using real-world information.

8

CONCLUSION

We have proposed VISTURE, a system that generates robot gesturesand speech by using video as input, assuming a situation in whicha robot conveys what it saw with a camera to a person who wasabsent. We have performed a case study to investigate the expressions that Japanese people use to describe video scenes and foundclassi�cation: Foreground and Background. VISTURE generates
representations based on changes in the coordinates of objects ina video, according to the �ndings of a case study. We experimentally evaluated the quality of the gestures and speech generated
by VISTURE and the people’s impression of a robot performing
those gestures and speech. We found that the robot was perceived
as more likeable and capable when it performed gestures. However,some representations that took an Event and the Background in avideo into account worked well, whereas others did not. This was
because coordinate changes in a video could not always be used
to detect human actions, in which case the generated representation mentioning an Event did not match the person performing anaction.

ACKNOWLEDGMENTS
This work was supported in part by JST, CREST Grant Number
JPMJCR19A1, Japan and JSPS KAKENHI Grant Number JP20K19897.

REFERENCES[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson,
Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for
Image Captioning and Visual Question Answering. In 2018 IEEE/CVF Conference
on Computer Vision and Pattern Recognition. 6077–6086. https://doi.org/10.1109/
CVPR.2018.00636[2] Christoph Bartneck, Elizabeth Croft, and Dana Kulic. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence,and perceived safety of robots. International Journal of Social Robotics 1, 1 (2009),
71–81. https://doi.org/10.1007/s12369-008-0001-3[3] Paul Bremner and Ute Leonards. 2016. Iconic Gestures for Robot Avatars,
Recognition and Integration with Speech. Frontiers in Psychology 7 (2016).
https://doi.org/10.3389/fpsyg.2016.00183[4] Mikel Broström. 2020. Real-time multi-object tracker using YOLOv5 and deep
sort. https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch.
[5] John-John Cabibihan, Wing-Chee So, Sujin Saj, and Zhengchen Zhang. 2012.
Telerobotic Pointing Gestures Shape Human Spatial Cognition. International
Journal of Social Robotics 4 (April 2012), 263–272. https://doi.org/10.1007/s12369012-0148-9[6] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew
Zisserman. 2018. A Short Note about Kinetics-600. https://doi.org/10.48550/
ARXIV.1808.01340[7] Je� Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan,
Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2017. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. IEEE
Transactions on Pattern Analysis and Machine Intelligence 39, 4 (2017), 677–691.
https://doi.org/10.1109/TPAMI.2016.2599174[8] Drills. 2016. Soccer Coaching Attacking Drill: Warm Up | circular saw. https://youtu.be/jDrZ5Rzpn7A Accessed: Jun 18, 2022.

This is a preprint of an article published in Proceedings of the 10th International Conference on Human-Agent Interaction.
The final authenticated version is available online at: https://doi.org/10.1145/3527188.3561931

VISTURE: A System for Video-Based Gesture and Speech Generation by Robots[9] Martin Fernandez. 2013. Dami y Barby Bailando Salsa.
https://youtu.be/qcP1mb1IjjI Accessed: Jun 18, 2022.
[10] Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. 2017.
Video Captioning With Attention-Based LSTM and Semantic Consistency. IEEE
Transactions on Multimedia 19, 9 (2017), 2045–2055. https://doi.org/10.1109/
TMM.2017.2729019[11] Michita Imai, Tetsuo Ono, and Hiroshi Ishiguro. 2001. Physical relation andexpression: joint attention for human-robot interaction. In Proceedings 10th IEEE
International Workshop on Robot and Human Interactive Communication. ROMAN
2001 (Cat. No.01TH8591). 512–517. https://doi.org/10.1109/ROMAN.2001.981955[12] Yuki Kadono, Yutaka Takase, and Yukiko I. Nakano. 2016. Generating Iconic Gestures Based on Graphic Data Analysis and Clustering. In The Eleventh ACM/IEEEInternational Conference on Human Robot Interaction (Christchurch, New Zealand)(HRI ’16). IEEE Press, 447–448.
[13] Mitsuhiko Kimoto, Takamasa Iio, Masahiro Shiomi, and Katsunori Shimohara.
2021. Coordinating Entrainment Phenomena: Robot Conversation Strategy
for Object Recognition. Applied Sciences 11, 5 (2021). https://doi.org/10.3390/
app11052358[14] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common
Objects in Context. In Computer Vision – ECCV 2014, David Fleet, Tomas Pajdla,
Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing,
Cham, 740–755.
[15] Xin Man, Deqiang Ouyang, Xiangpeng Li, Jingkuan Song, and Jie Shao. 2022.
Scenario-Aware Recurrent Transformer for Goal-Directed Video Captioning.
ACM Trans. Multimedia Comput. Commun. Appl. 18, 4, Article 104 (mar 2022),
17 pages. https://doi.org/10.1145/3503927[16] Nicholas Masters. 2011. Guy on crutches gets dropped kicked. https://youtu.be/i2F6HrZKo34 Accessed: Jun 18, 2022.
[17] Takahiro Matsumoto, Satoru Satake, Takayuki Kanda, Michita Imai, and Norihiro
Hagita. 2012. Do you remember that shop? — Computational model of spatial
memory for shopping companion robots. In 2012 7th ACM/IEEE International
Conference on Human-Robot Interaction (HRI). 447–454.
[18] David McNeill. 1992. Hand and mind : what gestures reveal about thought. University of Chicago Press.
[19] Fumio Nihei, Yukiko Nakano, Ryuichiro Higashinaka, and Ryo Ishii. 2019. Determining Iconic Gesture Forms Based on Entity Image Representation. In 2019
International Conference on Multimodal Interaction (Suzhou, China) (ICMI ’19).
Association for Computing Machinery, New York, NY, USA, 419–425. https://doi.org/10.1145/3340555.3353736

HAI ’22, December 5–8, 2022, Christchurch, New Zealand[20] Maha Salem, Friederike Eyssel, Katharina Rohl�ng, Stefan Kopp, and Frank
Joublin. 2013. To Err is Human(-like): E�ects of Robot Gesture on Perceived
Anthropomorphism and Likability. International Journal of Social Robotics 5(2013), 313–323.
[21] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations from
Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV). 618–626. https://doi.org/10.1109/ICCV.2017.74[22] SoftBank. 2019. Robot SoftBank. https://www.softbank.jp/en/robot/ Accessed:
Jun 18, 2022.
[23] Elisabeth T Van Dijk, Elena Torta, and Raymond H Cuijpers. 2013. E�ects of eye
contact and iconic gestures on message retention in human-robot interaction.
International Journal of Social Robotics 5, 4 (2013), 491–501. https://doi.org/10.
1007/s12369-013-0214-y[24] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler
Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser,
Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern,
Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas,
Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero,Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa,
Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scienti�c Computing in Python. Nature Methods 17 (2020), 261–272.
https://doi.org/10.1038/s41592-019-0686-2[25] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang
Wang. 2019. VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Videoand-Language Research. In The IEEE International Conference on Computer Vision(ICCV).
[26] Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao,
Yongdong Zhang, and Qionghai Dai. 2020. STAT: Spatial-Temporal Attention
Mechanism for Video Captioning. IEEE Transactions on Multimedia 22, 1 (2020),
229–241. https://doi.org/10.1109/TMM.2019.2924576[27] Xu Yan, Zhengcong Fei, Zekang Li, Shuhui Wang, Qingming Huang, and Qi
Tian. 2021. Semi-Autoregressive Image Captioning. Association for Computing
Machinery, New York, NY, USA, 2708–2716. https://doi.org/10.1145/3474085.
3475179[28] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image
Captioning with Semantic Attention. In 2016 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR). 4651–4659. https://doi.org/10.1109/CVPR.2016.
503