Caption Generation of Robot Behaviors
Based on Unsupervised Learning of Action Segments
Koichiro Yoshino, Kohei Wakimoto,Yuta Nishimura, Satoshi Nakamura
Nara Institute of Science and Technology
RIKEN Center for Advanced Intelligence Project
RIKEN Robotics Project

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learningof Action Segments1Cooperative robots in living places• Robots will be a part of our daily life
– Helper robots will work to support
our daily life
• House keeping• automatic driving• Robots have to communicate with users for cooperation– House keeping assistants need to follow user’s instructions– Automatic driving systems have to follow both user’s intention andsurrounding situations including traffic rules

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

2

Miscommunication between robots and users• Problem: miscommunication between robots and users
Bring something that is on theliving room floor.
Yes.

…
Oh, I meant bring the bottle, but
you brought a block!

• Robots should have ability to explain its behaviors– User’s instructions are often ambiguous (e.g., “something”)– There are some gaps between what the robot intent to do, and what
the robot did (e.g., fail to pick-up something)2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments3

Ability to explain robot behaviors• Robots should have ability to explain their behaviors
Bring something that is on theliving room floor.
Yes.

…
I brought a bottle and blocks thatwere on the living room floor.

• Robots need to recognize:– not only what is the given command by users,– but also what the robot did, from its observation
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

4

Task: robot behavior captioning
Input

OutputI brought a rabbit ornament
from the kitchen table.
• Input: robot observations– Trajectory of each actuator– Video from a first person viewpoint camera• Output: caption to describe the behavior– A natural language sentence that describes what the robot did
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

5Related works and problems• Robot action segmentation [Nakmaura 2017]– Learned basic action units of robots using multi-modal LDA
to understand the robot’s behavior– They did not generate descriptionof robot actions• End-to-end mapping between
robot actions and their captions [Yamada 2018]– Try to learn alignments between encoded robot actions andmeaning vectors of sentences– Requires large-scale training data
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

6Difficulties in robot behavior captioning• End-to-end learning will be a promising approach; however,• It is impossible to collect large-scale dataset for any robots
and tasks– Possible actions depend on robot abilities– Action targets depend on the environment where the robot locates– Instructions depend on robot tasks• Mapping from a long sequence to a short sequence– The robot action sequence will be longer (100~10000) thanthe caption word sequence (~30)
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

7

Solution in this paper
1. Action classes learning by unsupervised learning
2. Attention mechanism to classify related action samples
raw
actions

captionraw
actionClustering(K-means)

Chunking(BPE)

actionclasses
actionclasses

caption
XX

X

X

X
𝒚𝟏

𝒚𝟐

𝒚𝟑𝒚𝒏EOS

EOS

𝒚𝟏

𝒚𝟐

𝒚𝒏−𝟏𝒚𝒏attention_score & softmax

𝒔𝟏

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAIST

𝒔𝟐

𝒔𝟑

𝒔𝒎−𝟏

𝒔𝒎

Encoder-Decoder w/attention mechanism(implicit chunking)

IWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

8

Clustering and chunking• We show an example when the robot
only has two dimensions of actuation

Cluster A

1. We applied K-means clustering
to discretize the action vector

Cluster B– We decided K=150 by Elbow method

2. We applied byte-pair encoding (BPE) Cluster E
to chunk frequent action patterns– To reduce the number of samples inrobot actions

We call this method “Explicit”
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTCluster DCluster CClustering
AAAABBCCDEEEEDDEDAChunking
AA|AA|B|B|C|C|DE|E|E|E|D|DE|D|A

IWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

9Attention mechanism for alignmentX

X

X

X

X

𝒚𝟏

𝒚𝟐

𝒚𝟑

𝒚𝒏EOS

𝒔𝒎EOS

𝒚𝟏

𝒚𝟐

𝒚𝒏−𝟏

𝒚𝒏

attention_score & softmax

𝒔𝟏𝒔𝟐𝒔𝟑𝒔𝒎−𝟏• Attention mechanism can learn a mapping between input andoutput sequences as an attention map– 𝒂𝒊,𝒋 = 𝒉𝑻𝒆,𝒊 𝑾𝒉𝒅,𝒋 : relation between 𝒊 in encoder and 𝒋 in decoder

We call this method “Implicit”
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

10Robot environment• World Robot Summit (WRS)
service category– Housekeeping task

• Human support robot (HSR)
made by TOYOTA• Used robot simulator– SIGVerse + Unity• Sampling rate = 0.3sec

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

11Dataset• Input (robot action sequence)– 9 dimensional motor actuation (rotation)– 3 dimensional robot moving (2 direction plus 1 rotation)– 160 x 120-pixel images from a first person viewpoint camera• 10 dimensional embedding by convolutional autoencoder (CAE)– 50 different robot actions• Output (caption)– Crowd-workers annotated a caption to each action– 20 different worker annotated different sentences for each– 1000 sentences in total

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

12Experimental setting• We used LSTM-based encoder-decoder to train the mapping– We split the data to 800/100/100 as train/development/test– 10-fold cross-validation• Compared models– Vanilla: raw action vectors are used in normal encoder-decoder– Explicit: action classes made by clustering and chunking are used,instead of raw action vectors– Implicit: raw action vectors are used in encoder-decoder that hasattention mechanism– Hybrid: action classes made by clustering and chunking are used as
input of the encoder-decoder that has attention mechanism
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

13

Automatic evaluation with BLEU-N
MethodBLEU-2

BLEU-3

BLEU-4

Vanilla

0.1

0.0

0.0

Explicit

18.0

11.7

8.4

Implicit

21.4

14.7

10.4

Hybrid

23.4

16.2

11.7

• We used multi-reference setting– Each video has 20 reference sentences– Reference that had the best score was used for evaluation• “Hybrid” achieved the best performance– “Explicit” and “Implicit” had different contribution in generation

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

14

Human subjective evaluation• We showed robot action
videos to subjects• Then subjects saw thecaption by each method
in random ordera.
• Subjects selected one
from a-e evaluation– 9 subjects– 20 pairs (x 4 methods)
were evaluated2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTThe sentence is exactly describing the robot actionin the video
b. The sentence is mostly describing the robot actionin the video; even it contains some minor errorsc. The sentence is moderately describing the robot
action, but the description still has some major errors
on target objects or environmentd. The sentence is grammatically correct; however,
the action name and the object name are different
e. The sentence has wrong grammar and is meaningless

IWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

15Subjective evaluation results
Methodabcde

a-c

Vanilla

0.0%

0.0%

0.0%

0.0%

100.0%

0.0%

Explicit

10.6%

17.2%

37.2%

28.3%

6.7%

65.0%

Implicit

3.9%

5.0%

35.0%

56.1%

0.0%

43.9%

Hybrid

7.2%

8.9%

33.3%

43.9%

6.7%

49.4%

• “Explicit” had the best score– 65.0% of generated sentences correctly described actions– 6.7% of ungrammatical sentences• “Implicit” often generated incorrect sentences– Most of them were grammatically correct, but contained incorrectcontents– Language model overfitting?
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

16

Subjective evaluation results
Methodabcde

a-cVanilla

0.0%

0.0%

0.0%

0.0%

100.0%

0.0%

Explicit

10.6%

17.2%

37.2%

28.3%

6.7%

65.0%

Implicit

3.9%

5.0%

35.0%

56.1%

0.0%

43.9%

Hybrid

7.2%

8.9%

33.3%

43.9%

6.7%

49.4%

• Every method contain object errors (c)– The system correctly trained the action names, but still had aproblem on object recognition• “Hybrid” did not contribute to improve human scores– Got bad points of both “Explicit” and “Implicit”

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

17

Generation examples
Ref.

Drop the toy dog from the table(犬の置物をテーブルから落として)

VanillaRepeating function words (ててててててててててて)

e:9Explicit

Drop the apple (リンゴを落として)

b:2, c:7

Implicit

Bring the sauce on the table(テーブルの上のソースを取ってきて)

d:9Hybrid

Bring the cup on the table(テーブルの上のコップを取ってきて)

d:9• Only “Explicit” successfully learned“drop” action

2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

18

Generation examples
Ref.

Pick up the teapot on the floor(床のティーポットを拾って)

VanillaRepeating function words (ててててててててててて)

e:9Explicit

Pick up the sauce on the floor (床のソースを取って)

a:7, b:1, c:1

Implicit

Pick up the sauce on the table(テーブルの上のソースを取って)

b:1, c7, d:1

HybridPick up the sauce placed on the floor(床にあるソースを取って)

a:7, c:2• It was difficult for the system todistinguish the same color objects• Each method learned “pick up”
action2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

19

Conclusion and future works• We trained a captioning model of robot behavior given robot
an action sequence• We used two types of classification to learn from small data
– Explicit: clustering and chunking to make action classes– Implicit: attention mechanism may work as clustering• Any clustering methods contributed to improve captioning– “Explicit” performed the best

– Most actions were learned correctly– Our system had problems on object recognition
 used pre-trained model of object recognition• Analysis, other clustering methods
2020/09/21 ©Koichiro Yoshino
AHC-Lab. NAISTIWSDS2020 Caption Generation of Robot Behaviors based on Unsupervised Learning ofAction Segments

20