Caption Generation of Robot Behaviors basedon Unsupervised Learning of Action Segments
Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

Abstract Bridging robot action sequences and their natural language captions is
an important task to increase explainability of human assisting robots in their recently evolving field. In this paper, we propose a system for generating natural
language captions that describe behaviors of human assisting robots. The systemdescribes robot actions by using robot observations; histories from actuator systems and cameras, toward end-to-end bridging between robot actions and natural
language captions. Two reasons make it challenging to apply existing sequence-tosequence models to this mapping: 1) it is hard to prepare a large-scale dataset for
any kinds of robots and their environment, and 2) there is a gap between the numberof samples obtained from robot action observations and generated word sequencesof captions. We introduced unsupervised segmentation based on K-means clustering
to unify typical robot observation patterns into a class. This method makes it possible for the network to learn the relationship from a small amount of data. Moreover,
we utilized a chunking method based on byte-pair encoding (BPE) to fill in the gap
between the number of samples of robot action observations and words in a caption.
We also applied an attention mechanism to the segmentation task. Experimental results show that the proposed model based on unsupervised learning can generatebetter descriptions than other methods. We also show that the attention mechanism
did not work well in our low-resource setting.

1 Introduction
In the recent advance of human assisting robot technologies, the ability to explain
robot behaviors is becoming an important task. Robot actions are often generated
1 Nara Institute of Science and Technology
8916-5, Takayama, Ikoma, Nara, 6300192, Japan e-mail: koichiro at is.naist.jp
2 Center for Advanced Intelligence Project (AIP), RIKEN

1

2

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

from uninterpretable systems, such as deep neural networks [8]. However, robots
are expected to be explainable for their behavior because they will cooperate withhumans in daily life areas. Most of the existing works for bridging robot actions
and natural language captions are oriented to generated robot actions given natural
language commands [21, 12, 25, 6]. They tackled the problem in a pipeline process
by using manually designed base action units, i.e., small basic robot actions, which
are easy to recognize and generate. Both robot actions and captions were converted
to sequences of base action units and then bridged each other.
In contrast, end-to-end approaches have been widely used to date by recent advances in the neural network field [4, 23, 3]. These approaches directly train mappings between two sequences. Such end-to-end mapping is applied to robot behavior captioning, which generates a natural language caption given a raw robot action
sequence [24, 20, 29, 26].
It is not easy for human assisting robots to generate natural captions from theirobservations, sequences of actions, and observation from robot sensors, because
robots encounter diverse environments and tasks. Collecting large-scale dataset forany different robots or environment is not realistic. Furthermore, one robot observation sequence has more samples than words in a caption due to sampling rates ofobservation devices. However, this gap makes it challenging for neural networks tolearn relationships between robot observations and natural language captions. This
problem is especially true if we apply neural network-based approaches, in particular recurrent neural networks, which are generally used for sequence-to-sequencelearning.
Segmenting robot actions to base action units will contribute to making the training process more straightforward. In the work we performed, we focused on using
unsupervised segmentation methods [16, 15, 17] for speeding up and stabilizing
sequence-to-sequence training, rather than using manually designed base robot action units [6]. We used K-means clustering and chunking based on byte-pair encoding (BPE) to learn the unsupervised base action segments of the robot. K-meansclustering can unify similar robot observation into one class. Chunking based on
BPE can wrap a typical sequence of robot actions into one class.
We also used an attention mechanism in sequence-to-sequence learning [13, 27],
which explicitly learns the correlation between input and output sequences. Attention mechanism calculates correlation weights between each input sample and each
output sample. It is known that the attention mechanism can solve a problem of sample number gaps in the area of speech recognition and synthesis [5, 28]. We expectthat the training results obtained for the attention can be incorporated into a robot
action unit vocabulary that corresponds to actual words in sentences. We combined
both methods; unsupervised action segment learning and attention mechanism, togenerate more natural captions since we expected that each method would provide
different contributions.
In experiments, we recorded videos of robots with their observation: actuator
sequences and videos recorded by a first-person viewpoint camera. We used thesevideos to collect robot behavior captions through crowdsourcing, which corresponds
to robot action sequences. However, we limited the number of collected samples

Title Suppressed Due to Excessive Length

3

Bring something that is on theliving room floor.
Yes.

…
I brought a bottle and blocks thatwere on the living room floor.

Fig. 1 Example human robot communication obtained through natural language commands/descriptions.

toward a realistic situation to apply the method. We conducted two experiments:
an automatic evaluation based on references in the test-set and human subjective
evaluation. These results showed that the proposed three types of base action units
contribute to improving better robot behavior captions in natural language. In particular, robot action units acquired by the unsupervised learning based on clusteringand chunking got the best performance in human subjective evaluation.

2 Natural Language Captions for Robot Action Sequences
2.1 Problem Setting and Related WorkFor daily life scenarios, various robot tasks have been designed as human assisting
robots have evolved. In particular, these tasks include human assisting tasks, such as
moving objects in rooms and recording videos of actual places. In these situations,
it is required for robots to communicate with users about their behavior in natural
language. There are two important directions. The first direction is deciding robot
actions according to the natural language commands by the user. Another direction
is providing robot behavior captions in natural language. Both directions are important to realize cooperative human assisting robots. In these tasks, there are sequencesof robot actions and natural language captions/descriptions, and the robot needs tolearn the mapping between them. Figure 1 shows an example of the task in a particular situation. In this example, the user requests the robot to “Bring something thatis on the living room floor,” and the robot does so. After this, the robot uses natural
language to tell the user what was done, e.g., “I brought a bottle and blocks that wereon the living room floor”. In this paper, we focus on the latter problem, generatinga natural language caption given a robot action sequence. We define the problem as
generating a word sequence Y given a sampled robot observation sequence S.
As the robot action sequence S, most reported existing works have been doneusing the pipeline process, such as handcrafting base action units, recognizing all
robot observation sequence portions with these base action units and constructing

4

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

Fig. 2 HSR robot on SIGVerse environment, conducting a task in the setting of WRS.

complicated actions [12]. However, these approaches for handcrafting base action
units are costly.
Based on the advance of technologies based neural networks, some works tried tobuild sequence-to-sequence learning system between robot observations and natural
language captions. Takano et al. [24] used bi-grams for end-to-end learning between
natural language captions and robot action sequences, to separate the semantics andthe language expressions in captions. Yamada et al. [29] utilized recursive autoencoders for the learning so as to minimize the distance between the latent variables ofautoencoders for robot actions and natural language captions, toward the end-to-end
bridging. In another study, a simple encoder-decoder was applied to learn the relation [20]. However, these methods require large-scale training data. It is not realistic
to prepare such large-scale training data whenever the robot is applied to a new environment, or when the robot functions are updated. The physical environments ofrobots themselves will be different if a new robot is developed.
To prevent problems caused by a lack of training data, in this paper, we applied
two robot action segmentation types in an unsupervised manner to make it possible
for the system to learn the relationship from a small data amount. Some related
studies have tried to define base action units with unsupervised learning [16, 15, 17]or adaptively [10]. These studies focused on structuring robot actions from data.
Such clustering-based approaches have the potential to be used for reducing training
data with simplifying symbols contained in training data.

2.2 Robot Task, Environment and Data in Simulator
In this section, we describe collected dataset for the training of captioning given
robot observations. We used the service category environment defined in the World
Robot Summit (WRS) [11] as the robot task for our action description generation.
We used Human Support Robot (HSR) [30], a human assisting robot in a domes-

Title Suppressed Due to Excessive Length5

tic environment, and also used a robot simulator SIGVerse [9]. We followed WRS
Partner Robot Challenge (Virtual Space) instruction to build our environment.1 The
HSR robot in SIGVerse environment is shown in Figure 2. The robot conductedtasks in this environment and described its actions with natural language behavior
captions. We generated robot actions in the simulator and added their captions, byusing crowdsourcing, as our training data.
The input S contains nine-dimensional motor actuation (rotations) and a threedimensional robot moving (two directions and one rotation) that is sampled every
0.3 seconds. The robot also uses 160×120-pixel images from a first person viewpoint camera mounted on robot arm’s tip. The image feature is embedded into aten-dimensional vector by using convolutional autoencoder (CAE) [14]. The robot
action actuation and the image features are concatenated into one vector, which will
be used as an input of encoder at each time-step.
We used crowdsourcing to add a caption Y for a robot action sequence S. We
made videos of robot action sequences from third-person viewpoint camera as
shown in Figure 2. We showed the video to crowd-workers and then requested themto give their captions that describe behaviors of the robot in natural language.2 We
made 50 videos and added 20 captions for each. Robot actions in the dataset contained “bring”, “put”, “pick up”, “drop” and “go to see” actions and 10 different
videos for each action. Note that, we did not give crowd-workers any instructions
on how they should include these verbs in descriptions. This means that, for example, they might use “get” as an alternative to “bring” in accordance with theirlanguage sense. In total, we collected 1000 sentences in Japanese, which were segmented by KyTea [18]. Annotated captions are simple sentences, which consist of
single predicate and several arguments, i.e., “Drop the toy dog from the table” or“Pick up the teapot on the table”.

2.3 Bridging based on Encoder-Decoder
We show the architecture of end-to-end relationship learning between the robot action sequence and the word sequence based on an encoder-decoder, which had been
proposed previously [20]. The overall architecture in a time sequence is shown in
Figure 3. As mentioned in the problem setting, the system gets an input robot observation sequence S to a generate natural language caption Y . Here, si indicates robot
observations, the raw trajectory of robot action and a frame recorded by cameras on
the robot, at time i and y j indicates the jth word in the natural language description.
The encoder embeds the robot action sk in time k to the hidden layer hk as
hi = σ (Wsh si +Whh hi−1 + bh ),1(1)

https://worldrobotsummit.org/download/rulebook-en/rulebooksimulation league partner robot challnege.pdf
2 Dataset including captions will be distributed when we publish the paper.

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

6

Decoder
࢟૛
࢟૚

࢟૜

࢟࢔

EOS

EOS

࢟૛

࢟࢔ି૚

࢟࢔

Encoder

࢙૚

࢙૛

࢙૜

࢙࢓ି૚

࢙࢓

࢟૚

Fig. 3 Basic encoder-decoder architecture. In our problem, the encoder receives a sample of robot
observation si in time i according to its sampling rate. Once the network receives the end-ofsequence symbol (EOS), the decoder start to generate a description word sequence.

where, Wsh and Whh are weight matrices for conversion and bh is a bias. σ is theactivation function. The decoder starts generating word w j as soon as the encoder
receives the end-of-sequence (EOS) symbol and continues to generate as it generates wn+1 .The network is updated byhi = σ (Wyh wi−1 +Whh hi−1 + bh ),(2)

yi = so f tmax(Why hi + by ),(3)

where Wyh , Whh and Why are decoder weight matrices, and bh and by are bias.
so f tmax is the activation function to predict words. This architecture was proposedin the task of machine translation and is widely used on a variety of mapping tasks
between sequences, such as conversational response generation. The difference frommachine translation or conversational response generation tasks is that the robot action sequence S will be very long in accordance with the robot trajectory sampling
rates. This property increases the risk of that vanishing gradients [1]. In the area ofspeech recognition or synthesis, attention mechanism is used to solve the problem[5, 28]. The problem can be mitigated if there is sufficient training data; however,this means that large-scale new data will be required for each new domain or eachrobot task.

3 Robot Action Segmentation
The dataset we used had only 1000 samples, corresponding to 50 actions, thus making it difficult to learn a good mapping with a vanilla encoder-decoder. However, as
we mentioned before, collecting a large-scale dataset for all domains and robots is
not a realistic option. Thus, we trained robot action segments as base action units byusing unsupervised learning: clustering and chunking. We also incorporated the at-Title Suppressed Due to Excessive Length7

tention mechanism into the encoder-decoder, since we expect the attention weights
could learn the action classes.

3.1 Robot Action Segmentation Based on Clustering and Chunking
Our system samples raw robot actions every 0.3 seconds. Figure 4 showing a clustering and chunking example. This graph shows robot actions in only two dimensions
for an easy description. Robot action vectors are quantized by k-means clustering[7], which decides what the centroids should be for the defined number of classes.
In this example, the number of samples is 18 and the number of class is set as 5. We
used the Elbow method [2] to determine the class number should be 150.

Cluster A

Cluster B

Cluster E

Cluster D
Cluster C

Clustering

AAAABBCCDEEEEDDEDA
Chunking

AA|AA|B|B|C|C|DE|E|E|E|D|DE|D|A
Fig. 4 Robot action segmentation process: clustering and chunking.

After the clustering, we extracted some chunks by using byte-pair encoding(BPE). BPE extracts frequent symbol sequences and replace them to another symbol
as maximizing the compression of a given raw sequence. The robot action sequencequantized by clustering is still longer than the word sequence; however, we canshorten the sequence by applying BPE. The BPE units can be used as base action
units. In the Figure 4 example, AA and DE are extracted as vocabularies. In our experiments, we set the BPE vocabulary size as 200 and trained the vocabulary fromthe training data. We refer to this method as “explicit segmentation” in the following
sections.

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

8

X

X

X

X

X

࢟૚

࢟૛

࢟૜

࢟࢔

EOS

࢙࢓

EOS

࢟૚

࢟૛

࢟࢔ି૚

࢟࢔

attention_score & softmax

࢙૚

࢙૛

࢙૜

࢙࢓ି૚

Fig. 5 Encoder-decoder with attention mechanism.

3.2 Segmentation Based on Attention Mechanism
We conducted explicit segmentation as a pre-processing of relation learning by using encoder-decoder neural networks. In contrast, the attention mechanism can consider input chunks in attention weight learning implicitly. The attention mechanism
learns the mapping from an input sample to an output sample by using a gating
mechanism. In our situation, the number of input samples is larger than the numberof output samples; thus input samples associated with the same output sample canbe interpreted that they belong to the same class.
Attention mapping is calculated between the hidden layer of encoder he,i and thehidden layer of decoder hd, j as,ai, j = hTe,iWa hd, j .

(4)

An attention weight is trained as a part attention vector a j in each decoding step.
Input samples that have similar attention weights for an actual output sample arestrongly used in the decoding of the associated output. We expect that this propertyof attention mechanism works as segmentation. We show an encoder-decoder thathas an attention mechanism in Figure 5. In this example, strong attention weights
are indicated in a deep color. States s2 and s3 have higher attention weights to y3 inthe output, and states sm−1 and sm have higher attention weights to y1 . We expectthat such weights in the attention mechanism work as base action units. We hereafter
call this method “implicit segmentation”.
Note that, it is known that the attention mechanism contributes to improve theentropy of the generated sentence; however, it also often causes an overfitting anddull generations. In other words, sentence naturalness will be improved by the attention mechanism; however, the system often generate similar sentences to different
input sequence.

Title Suppressed Due to Excessive Length

9

Table 1 Evaluation with BLEU scores. Note that the range is from 0 to 100.
Model BLEU-2 BLEU-3 BLEU-4
Vanilla0.1
0.0
0.0
Explicit
18.0
11.7
8.4
Implicit
21.4
14.7
10.4
Hybrid
23.4
16.2
11.7

3.3 Hybrid Method of Implicit and Explicit
Explicit segmentation segments sequences by focusing on input sequence information, in contrast to that of implicit segmentation, which focuses on the relationship between the input sequence and the output sequence for segmenting the input
sequence. In other words, these segmentations use different types of information.
Thus, we integrated these segmentations into a single method as “hybrid segmentation” to make the best of them both. In the hybrid segmentation, we used explicit
segmentation to segment the data as pre-processing, and then fed the data into theencoder-decoder with the attention mechanism to combine with the implicit segmentation method.

4 Experiments
In our experiments, we used the data collected in Section 2.2 to train an encoderdecoder that generates a word sequence of natural language caption T given a raw
robot action sequence S. As the baseline, we prepared a vanilla encoder-decoder,because the focus of this paper is investigating a good robot action segmentation,
which makes it possible to train the encoder-decoder from a small amount of data.
We compared three methods described in Section 3; explicit segmentation, implicit
segmentation, and hybrid segmentation, with the vanilla encoder-decoder. Experimental details follow.

4.1 Experimental Settings
We separated our 50 robot actions into 40/5/5 as training/validation/testing datasetsand conducted ten cross-fold validation to evaluate all the data. The number of decriptions associated with one robot action was 20; thus training/validation/testingdatasets cotains 800/100/100 pairs of sequences. We used a one-layered LSTM thathas 160 units as our encoder-decoder model. The batch size was 64, the dropout rate
was 0.5, the learning rate was 0.001, and the weight decay was 1e-0.6. We decided
the number of training epochs by using the validation dataset.

10

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

Table 2 Subjective evaluation result distribution for each method.
Model
a
b
c
d
e
a–c
Vanilla 0.0% 0.0% 0.0% 0.0% 100.0% 0.0%
Explicit 10.6% 17.2% 37.2% 28.3% 6.7% 65.0%
Implicit 3.9% 5.0% 35.0% 56.1% 0.0% 43.9%
Hybrid 7.2% 8.9% 33.3% 43.9% 6.7% 49.4%

4.2 Automatic Evaluation with BLEU
We used automatic evaluation metrics BLEU-2, 3 and 4 [19] to evaluate the generated captions, by using references annotated to videos in the testing set. BLEU-ncalculates the ratio of matched n-grams with smaller numbers of n, between thegenereation and the reference. Each video has 20 reference sentences; thus we used
multi-reference setting in the automatic evaluation. We calculated BLEU-n scoresfor each reference and selected a reference that has the best BLEU-n score for theevaluation.
Evaluation scores, BLEU-2, 3 and 4, for each model are shown in Table 1. Vanilla
encoder-decoder scores were deficient and scores are improved by introducing segmentation methods. Both the explicit segmentation and the implicit segmentation
were effective in improving scores. The hybrid segmentation achieved the best scorein each metric; it indicates the explicit segmentation and the implicit segmentation
made different contributions to each other. However, scores are slightly low even
if the task is simple. We will investigate the reason on the next human subjective
evaluation and generation analysis.

4.3 Human Subjective Evaluation
The automatic evaluation with BLEU calculates distance from references to generated sentences. The results have moderate correlations to the quality of generation;
however, there is still a gap between the result of evaluation with BLEU and human impression to the quality. For example, it is challenging to evaluate variationsof word selection by only using BLEU. Thus, we conducted a human subjective
evaluation to investigate the qualities of generated sentences from each method.
We showed both robot action videos and their associated captions generated bymodels to human subjects for scores these generated captions. In the experiment,each subject evaluated 20 sentences for each method, 80 sentences in total (for 4
methods). Following evaluation criteria are shown to human subjects.
a). The sentence is exactly describing the robot action in the video.
b). The sentence is mostly describing the robot action in the video; even it contains
some minor errors.

Title Suppressed Due to Excessive Length

11

Table 3 Generation examples. The “score distribution” indicates numbers of evaluators who gavea, b, c, d, and e.
MethodGeneration results (Translation)
Score distribution
Reference inu no omocha wo tehburu kara otoshite(Drop the toy dog from the table.)
Vanillate te te te te te te te te te te te te (Repeating function words.)
e:9
Explicit
ringo wo otoshite (Drop the apple.)b:2, c:7
Implicit
tehburu no ue no sohsu wo totte kite (Bring the sauce on the table.)d:9
Hybridtehburu no ue no koppu wo totte kite (Bring the cup on the table.)
d:9
Reference tehburu no ue wo nozoki konde (Look at the top of the table.)
Vanillate te te te te te te no no no no no no (Repeating function words.)
e:9
shinshitsu no yousu wo mite (Look at the bed room.)b:3, c:5, d:1
Explicit
Implicit
tehburu no ue no sohsu wo totte kite (Bring the sause on the table.)
a:1, d:7, e:1
Hybridheya no ue no yousu wo totte (Bring the state of the upper of the room.)d:2, e:7
Reference yuka no thipotto wo hirotte (Pick up the teapot on the floor.)
Vanillate te te te te te te te te te te te te (Repeating function words.)
e:9
Explicityuka no sohsu wo totte (Pick up the sauce on the floor.)
a:7, b:1, c:1
Implicit
tehburu no ue no sohsu wo totte (Pick up the sause on the table.)b:1, c:7, d:1
Hybridyuka ni aru sohsu wo totte (Pick up the sauce placed on the floor.)
a:7, c:2
Reference tehburu no koppu wo katadukete (Clear away the cup on the table.)
Vanillate te te te te te te te te te te te te (Repeating function words.)
e:9
Explicit
tehburu no ue no okimono wo totte kite (Bring the ornament on the table.)
a:1, b:5, c:3
Implicit
tehburu no ue no sohsu wo totte kite (Bring the sause on the table.)b:1, c:8
Hybridtehburu no ue no okimono wo totte kite (Bring the ornament on the table.) a:1, b:4, c:3, d:1
Reference tehburu no thipotto wo kauntah ni oite
(Bring the teapot from the counter to the table)
Vanillate te te te te te te no no no no no no (Repeating function words.)
e:9
kicchin no ue ni aru kan wo totte kite (Bring the can on the kitchen.)b:1, c:3, d:5Explicit
Implicit
tehburu no ue no sohsu wo totte kite (Bring the sause on the table.)a:6, b:3
kicchin no ue ni aru ka mite kite (Look at the upper of kitchen to check.)d:9
Hybridc). The sentence is moderately describing the robot action, but the description still
has some major errors on target objects or environments.
d). The sentence is grammatically correct; however, the action name and the object
name are different.
e). The sentence has wrong grammar and is meaningless.
a)–c) are prepared to check semantics of generated captions in different levels, compared with d) and e) focus more on sentence syntax. Human subjects selected one ofthem for the presented pair of a robot action video and its generated captions. The
number of subjects was 9; the total number of evaluated example for each method
was 180. Table 2 shows results.
From the results, we can know that the vanilla encoder-decoder did not generate any meaningful sentences; this result indicates that it is challenging to learn
the mapping between robot action sequences and caption word sequences from asmall portion of data. In contrast, tried segmentation methods make it possible themodel to generate many meaningful sentences; in particular, the explicit segmentation method generated meaningful sentence a lot. Human subjects judged that 65%
of generated sentences from the explicit segmentation method is meaningful (a, b
and c); this result indicates the usefulness of our proposed segmentation method.
Overall score is still slightly poor because of the number of training samples, as aresult, the majority in the evaluation result was c. This result indicates that the explicit segmentation method successfully learned predicates from robot action trajec-

12Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

tory; however, failed to learn about objects names from camera observations in mostcases. The implicit segmentation method also improved the score from the vanillasystem score; however, the ratio of meaningful generated sentences was 43.9%; it
was less than the result of the explicit segmentation method. This result indicatesthat it is challenging to learn the relation by using only the attention mechanism,
even if the effect is high. The implicit segmentation method generated no ungrammatical sentence; this result indicates the potential of using attention mechanism,
even if its score is less than the explicit segmentation method. This is probably because the attention mechanism is more focusing on fluency than content words. The
score of the hybrid segmentation method was slightly less than the explicit segmentation method, even if the method achieved the best BLEU score in the automatic
evaluation. This problem is probably caused by overfitting of word prediction in thesoftmax cross entropy loss calculation of encoder-decoder. However, the explicit
segmentation method lightened the problem of overfitting than other methods. In
summary, the explicit segmentation method, which uses clustering and chunking,achieved the best score for captioning robot actions in an end-to-end manner, inparticular, generating a caption of actions (predicates) from the robot action trajectory. However, we still have a problem with generation details, i.e., object names indescriptions.

4.4 Discussing Generated Sentences
Table 3 shows examples generated from each method with their scores on humansubjective evaluation.First, it can be seen that the vanilla method generated ungrammatical sentences in most cases. This supports our assumption that it is tough for avanilla encoder-decoder to train the mapping from a small data-set. In contrast, theexplicit segmentation method generated grammatical sentences. In particular, verbs
in sentences were correct in most cases. This indicates that our method successfully
learned the mapping between robot trajectories and verbs.
Generations from the implicit segmentation method was fluent; however, it sometimes contain similar sentences even if robot actions are different. For example, if
we look at the first and second examples, only the explicit segmentation method
can generate verbs “drop” and “look”, compared with implicit and hybrid generate“bring”, which is the most frequent verb in our training data. As observed elements,
the implicit segmentation method often generated the word “sauce”, because theword was frequent in the training data. This is probably caused by the overfittingof neural networks and generating dull generations in many cases. The explicit segmentation method reduced the problem in some cases.
The other problem that indicates the overfitting of the system is generating“sauce” and “table” as objects. The implicit segmentation method generated thesewords frequently than other methods. These words are more frequent than other objects; however, the implicit method generated these words in most cases, even if theother two methods generated different object and place names. This indicates theTitle Suppressed Due to Excessive Length13difficulty of our task, learning a mapping between robot observations and natural
language captions from a small amount of data, and the contribution of clusteringand chunking with explicit methods.
In many cases, robots do not successfully identify objects. This is probably since
the system misunderstood visual information because our method did not have an
object recognition module. Pre-training based on object labels is a possible solution
solve this problem.

5 SummaryThis paper presented a method that generates robot behavior captions in natural
language setting from robot action sequences, trajectories, and images, toward a cooperative human assisting robot that has explainability. The method uses two different segmentation types for robot action actuation for training from small-size data:
explicit segmentation based on clustering and chunking and implicit segmentation
based on the attention mechanism of neural networks. We also described the hybrid
segmentation that integrates both types. Experimental results indicated that the proposed methods improved generated caption quality, especially for generating verbs
related to the robot trajectories.
We are still trying to improve the method’s ability to describe objects, and so
we recognize that a subject for future works will be to use an object recognition
system as a module. We applied simple clustering and chunking methods; thus,
other segmentation methods or different types of robot action input to the network(e.g., VideoBERT [22]) have the potential to improve scores. Unsupervised learning
on generated data on the simulator is also a prospective approach to improve scores.
Another subject for future work will be to apply our method to generate robot action
sequences when it receives natural language commands from users.

Acknowledgement
Part of this work was supported by JSPS KAKENHI Grant Number JP17H06101.

References
1. Bengio, Y., Simard, P., Frasconi, P., et al.: Learning long-term dependencies with gradientdescent is difficult. IEEE transactions on neural networks 5(2), 157–166 (1994)
2. Bholowalia, P., Kumar, A.: Ebk-means: A clustering technique based on elbow method andk-means in wsn. International Journal of Computer Applications 105(9) (2014)
3. Chiu, C.C., Sainath, T.N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss,
R.J., Rao, K., Gonina, E., et al.: State-of-the-art speech recognition with sequence-to-sequence

14

Koichiro Yoshino1,2 , Kohei Wakimoto1 , Yuta Nishimura1 , Satoshi Nakamura1,2

models. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pp. 4774–4778. IEEE (2018)
4. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio,
Y.: Learning phrase representations using rnn encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pp. 1724–1734 (2014)
5. Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models
for speech recognition. In: Advances in neural information processing systems, pp. 577–585(2015)
6. Fasola, J., Mataric, M.J.: Using semantic fields to model dynamic spatial relations in a robot
architecture for natural language instruction of service robots. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 143–150. IEEE (2013)
7. Hartigan, J.A., Wong, M.A.: Algorithm as 136: A k-means clustering algorithm. Journal of
the Royal Statistical Society. Series C (Applied Statistics) 28(1), 100–108 (1979)
8. Hatori, J., Kikuchi, Y., Kobayashi, S., Takahashi, K., Tsuboi, Y., Unno, Y., Ko, W., Tan, J.:Interactively picking real-world objects with unconstrained spoken language instructions. In:
2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3774–3781.
IEEE (2018)
9. Inamura, T., Shibata, T., Sena, H., Hashimoto, T., Kawai, N., Miyashita, T., Sakurai, Y.,
Shimizu, M., Otake, M., Hosoda, K., et al.: Simulator platform that enables social interaction simulation―sigverse: Sociointelligenesis simulator. In: 2010 IEEE/SICE International
Symposium on System Integration, pp. 212–217. IEEE (2010)
10. Iqbal, T., Shah, A., Riek, L.: Toward a real-time activity segmentation method for humanrobot teaming. In: Proc. of the Robotics: Science and Systems (RSS), Towards a framework
for Joint Action: What about Theory of Mind Workshop (2018)
11. Kimura, T., Okugawa, M., Oogane, K., Ohtsubo, Y., Shimizu, M., Takahashi, T., Tadokoro,
S.: Competition task development for response robot innovation in world robot summit. In:2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR), pp.
129–130. IEEE (2017)
12. Kollar, T., Tellex, S., Roy, D., Roy, N.: Toward understanding natural language directions. In:
Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction, pp.
259–266. IEEE Press (2010)
13. Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machinetranslation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421 (2015)
14. Masci, J., Meier, U., Cireşan, D., Schmidhuber, J.: Stacked convolutional auto-encoders forhierarchical feature extraction. In: International Conference on Artificial Neural Networks,
pp. 52–59. Springer (2011)
15. Nakamura, T., Iwata, K., Nagai, T., Mochihashi, D., Kobayashi, I., Asoh, H., Kaneko, M.:
Continuous motion segmentation based on reference point dependent gp-hsmm. In: Proceedings of the IROS Workshop on Machine Learning Methods for High-Level Cognitive Capabilities in Robotics (2016)
16. Nakamura, T., Nagai, T., Iwahashi, N.: Grounding of word meanings in multimodal concepts
using lda. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems,
pp. 3943–3948. IEEE (2009)
17. Nakamura, T., Nagai, T., Mochihashi, D., Kobayashi, I., Asoh, H., Kaneko, M.: Segmenting
continuous motions with hidden semi-markov models and gaussian processes. Frontiers inneurorobotics 11, 67 (2017)18. Neubig, G., Nakata, Y., Mori, S.: Pointwise prediction for robust, adaptable japanese morphological analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pp. 529–533.
Association for Computational Linguistics (2011)19. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational
linguistics, pp. 311–318. Association for Computational Linguistics (2002)

Title Suppressed Due to Excessive Length15

20. Plappert, M., Mandery, C., Asfour, T.: Learning a bidirectional mapping between humanwhole-body motion and natural language using deep recurrent neural networks. Robotics
and Autonomous Systems 109, 13–26 (2018)
21. Sugiura, K., Iwahashi, N., Kashioka, H., Nakamura, S.: Learning, generation and recognitionof motions by reference-point-dependent probabilistic models. Advanced Robotics 25(6-7),
825–848 (2011)
22. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video
and language representation learning. In: Proc. ICCV (2019)23. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In:
Advances in neural information processing systems, pp. 3104–3112 (2014)
24. Takano, W., Nakamura, Y.: Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions. The International Journal of Robotics Research 34(10), 1314–1328 (2015)
25. Tellex, S., Kollar, T., Dickerson, S., Walter, M.R., Banerjee, A.G., Teller, S., Roy, N.: Understanding natural language commands for robotic navigation and mobile manipulation. In:
Twenty-Fifth AAAI Conference on Artificial Intelligence (2011)
26. Thomason, J., Padmakumar, A., Sinapov, J., Walker, N., Jiang, Y., Yedidsion, H., Hart, J.,
Stone, P., Mooney, R.J.: Improving grounded natural language understanding through humanrobot dialog. In: Proc. ICRA (2019)27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems,
pp. 5998–6008 (2017)
28. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y.,
Chen, Z., Bengio, S., et al.: Tacotron: Towards end-to-end speech synthesis. In: Proc. Interspeech 2017, pp. 4006–4010 (2017)29. Yamada, T., Matsunaga, H., Ogata, T.: Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions. IEEE Robotics and Automation Letters3(4), 3441–3448 (2018)30. Yamaguchi, U., Saito, F., Ikeda, K., Yamamoto, T.: HSR, human support robot as researchand development platform. In: The Abstracts of the international conference on advanced
mechatronics: toward evolutionary fusion of IT and mechatronics: ICAM 2015.6, pp. 39–40.
The Japan Society of Mechanical Engineers (2015)