RELATIONAL FUTURE CAPTIONING MODELFOR EXPLAINING LIKELY COLLISIONS IN DAILY TASKS
Motonari Kambara and Komei Sugiura
Keio University, Japan
ABSTRACT
Domestic service robots that support daily tasks are a promising solution for elderly or disabled people. It is crucial for domestic service robots to explain the collision risk before they
perform actions. In this paper, our aim is to generate a caption about a future event. We propose the Relational Future
Captioning Model (RFCM), a crossmodal language generation model for the future captioning task. The RFCM has theRelational Self-Attention Encoder to extract the relationships
between events more effectively than the conventional selfattention in transformers. We conducted comparison experiments, and the results show the RFCM outperforms a baseline
method on two datasets.
Index Terms— Future Captioning, Domestic Service
Robots, Relational Self-Attention
1. INTRODUCTION
Domestic service robots (DSRs) that naturally communicate
with users to support household tasks are a promising solution
for elderly or disabled people. DSRs are expected to perform
most tasks autonomously, and so they could damage objects
and themselves. It therefore would be useful if they could explain the potential risks associated with their actions throughnatural language. However, a DSR’s ability to generate natural language explanations is still insufficient.
Given this background, we focus on future captioning fordaily tasks [1]. Fig. 1 shows a typical use case where a DSR
puts a plastic bottle on a table. In this situation, it would bedesirable to tell the user that “the hand may contact the whitebottle, which may cause the bottle to further contact the apple
next to it, causing the apple to fall.” This task is difficult inthat models need to predict future events and generate captions. In fact, there is a big gap in the quality of the reference
and the generated sentences by typical video captioning models, as shown in Sec. 5.
Although there are many existing video captioning models, those models cannot generate appropriate future captions.
This is because many existing methods insufficiently model
the temporal relationship between the visual features and sentences. The transformer self-attention [2] used in those methThis material is based upon work supported by JSPS KAKENHI Grant
Number 20H04269, JST Moonshot, and NEDO.

Fig. 1. Overview of RFCM: RFCM generates captions of future
events from past events.

ods typically models the relationship between the visual features of the event and the sentence at time t. In addition,
many video captioning methods are inappropriate for future
captioning because those models use the captions after the
next timestep to generate a caption for the next timestep.
In this paper, we propose the Relational Future Captioning
Model (RFCM). The RFCM can generate captions that take
into account the relationship between past events. This is because it has a source–target attention structure that generates
appropriate captions for future events from the relationships
between events. In this structure, the features derived frompast clips are used as a source, and the features derived fromboth past clips and captions are used as the target.
Fig. 1 shows an overview of the RFCM, which consistsof three modules: a Relational Self-Attention (RSA [3]) Encoder, transformer encoder, and transformer decoder. The difference with respect to existing methods is that our method
includes the RSA Encoder to more effectively extract the relationships between events than the conventional self-attention
in transformers. Our code is available at this URL1 .
The main contributions of this paper are as follows:• We propose the RFCM, a crossmodal language generation model for the future captioning task.
• The RSA Encoder is introduced to extract the relationships between events more effectively than the conventional self-attention in transformers. The output of the
RSA Encoder is used to compute a source–target attention to obtain fine-grained caption representations.
1 https://github.com/keio-smilab22/RelationalFutureCaptioningModelwhere xt ∈ Rdin and k ∈ N denote the clip at time t and thenumber of input events that occurred before t, respectively.
From x, Zt−k:t+1 ∈ R(k+2)×din is obtained as follows:fz (xτ ) (τ > t)zτ =fz (xτ , xt ) (τ ≤ t),
Zt−k:t+1 = {zt−k ; ...; zt+1 },where τ = t − k, ..., t + 1, fz (·) denotes a linear transformation. Zt−k:t+1 contains information on an event sequence.
Fig. 2. The framework of RFCM. In the figure, Norm denotes layer
normalization.
2. RELATED WORK
There have been many studies in the field of caption generation [4–10]. This field includes image captioning and video
captioning. [8] is a survey paper in the image captioning field.
It provides a comprehensive review of existing deep-learningbased image captioning models. [11] is a survey paper in thevideo captioning field. It provides a comprehensive survey on
the video captioning field and classifies the methods, standard
datasets, and standard metrics in this field.
The field of video captioning can be divided into several subtasks such as a video captioning task, dense video
captioning task, and future captioning task. The video captioning task involves generating a description about an input event. Many video captioning models have been proposed [4, 7, 12–14]. [13, 14] are typical examples of the numerous pre-training models. The effectiveness of these models has become well known thanks to recent studies. The
dense video captioning task involves the generation of detailed captions. [5, 15, 16] are representative dense video captioning models. The future captioning task is the task dealt
with in this paper. [8, 17, 18] are representative future captioning models. [17] generates explanatory sentences of near
future events using in-vehicle camera images and vehicle motion information observed from past to present. Tasks similar
to the future captioning task include the video-and-language
event prediction task [19] and the video prediction task [20].
These tasks all involve predicting future events.
3. PROBLEM STATEMENT
Our target is to generate a sentence describing the next eventin the video, which we call the future captioning task [1].
Fig. 1 shows a typical scene. In this task, the desired behavior
is to generate a description by inferring the situation at timet + 1 given the clips up to time t.
The task is characterized by the following:• Input: video clips up to time t. We define the term
“clip” as a sequence of frames representing an event.
• Output: a sentence describing an event that is expected
to occur at time t + 1.
We assume that pretraining on other tasks is not allowed because knowledge transfer is outside the scope of this study.
4. PROPOSED METHOD
Fig. 2 shows the framework of the RFCM. The input to the
RFCM x ∈ R(k+1)×din is defined as x = {xt−k , ..., xt },4.1. RSA Encoder
The RSA Encoder uses Relational Self-Attention [3] and extracts the relationships between the events. The RSA Encoder
consists of Nr RSA layers. In the first layer, positional encoding using trigonometric functions is performed for Zt−k:t+1
using the procedure described in [2].
Then, query Qr ∈ Rdrsa , key Kr ∈ R(k+2)×drsa , andvalue Vr ∈ R(k+2)×drsa are obtained as follows:
Qr = zt ,Kr = Vr = Zt−k:t+1 ,where drsa denotes the size of each layer. Next, the basic kernel φp ∈ Rk+2 and relational kernel φh ∈ Rk+2 are obtained
as follows:
φp = Wp Qr ,Qr′ = {Qr ; ...; Qr } ∈ R(k+2)×drsa ,φh = Wh fflatten (Qr′ ⊙ Kr ),where fflatten (·) denotes flattening. Next, relational context
Φg ∈ R(k+2)×drsa obtained as follows:Φg = Vr + Wg Vr⊤ Vr .
Finally, RSA φ ∈ Rdrsa is obtained as follows:φ = (φp + φh )⊤ Φg .
RSA φ contains information about the relationships between input events. To use φ as a latent feature at timet, we replace zt with φ. From this, we obtain hr ={zt−k ; ...; zt−1 ; φ; zt+1 } ∈ R(k+2)×drsa . The output of layerhnr (nr = 1, ..., Nr ) is obtained by applying a feedforward
network (FFN) and layer normalization (LN) layers to hr .
The output of the encoder hrsa ∈ R(k+2)×drsa is given byhrsa = fLN (fFFN (hNr )), where fFFN (·) and fLN (·) denote
the FFN and LN layers, respectively.
4.2. Transformer encoder/decoder
The transformer encoder consists of Ne encoder layers. Eachlayer consists of a Masked Multi-Head Attention (MMHA),
Multi-Head Attention (MHA), and FFN layers.
The input to the encoder is hc = {Zt−k:t+1 ; T } ∈R(k+I+2)×din , where I denotes the maximum length of sentences. During training, T ∈ RI×din denotes the text features
obtained by embedding a reference sentence at time t + 1,
yt+1 with the BERT embedder [21]. During inference, T
denotes the text features for the generated words ŷt+1,1:j−1
when the j-th word is generated.
The input to the ne +1-th layer is hne (ne = 0, ..., Ne −1).
We set h0 = hc . In the MMHA layer, query Qe ∈

R(k+I+2)×de , key Ke ∈ R(k+I+2)×de , and value Ve ∈Verse [23], which was used in the World Robot Summit 2018(e)
R(k+I+2)×de (de = denc /Nh ) are obtained as Qe = Wq hne , Partner Robot Challenge/Virtual Space competition [24]. In(e)(e)
the simulator, a DSR placed a randomly selected everyday
Ke = Wk hne , and Ve = Wv hne (e = 1, ..., Nh ), where
object (e.g., a bottle or can) in the center of one of five types
Nh and denc denote the number of attention heads and the sizeof furniture (e.g., a table or shelf). Each sample was annoof each layer, respectively. During training, in the MMHA
tated with a statement explaining the situation that occurred
layer, we mask the m-th and subsequent word tokens to preas a result of the robot placement action. The dataset consists
vent the encoder from using the information of the words inof 1K videos and 1K english captions. The total and average
the m-th word prediction (teacher forcing). To obtain atten(k+I+2)×denc
lengths of the videos are 2.2 h and 8 s, respectively. In thetion AMMHA ∈ R
from Qe , Ke , and Ve , wedataset, each clip was given a sentence explaining dangerous
used the computation shown in [2]. Then, in the MHA layer,
events (e.g., collision events) and their causes. The vocabuattention AMHA is obtained using the same computation usedlary size is 245 words. We also set the number of samples
in the MMHA layer, where henc , Qe , Ke , and Ve are reincluded in the training, validation, and test sets to 800, 100,placed by AMMHA , Qe′ , Ke′ , and Ve′ , respectively. The out(k+I+2)×denc
and 100, respectively.
put of the ne + 1-th encoder layer, hne +1 ∈ R,
5.2. Experimental setup
is given by hne +1 = fLN (fFFN (AMHA )). The output of theencoder is hNe .
We preprocessed clips in the datasets as follows. For the
YouCook2-FC dataset, we trimmed the videos using the
The transformer decoder takes as inputs hrsa and hNe .
given start and end times. For the BILA-caption dataset, weThis module consists of Nd decoder layers. The structure
trimmed the videos so that the clip starts when the arm of theof each layer is similar to that of the MHA and FFN layers
DSR starts to move and ends when one of the following two
in the transformer encoder. However, there is a difference
conditions have been met: a collision event has occurred andin that each layer has the source–target attention structure.
more than 4 s have passed since the start time.
Therefore, the query is created based on hrsa , whereas the
key and value are created based on hNe . The output is hNd ∈Each clip was first converted to 0.6 and 8 fps for theR(k+I+2)×ddec , where ddec denotes the size of each layer.
YouCook2-FC and BILA-caption datasets, respectively. For
Finally, the prediction probability of the generated word
the YouCook2-FC dataset, we used the procedure shownp(ŷt+1,j ) ∈ RNv is given by p(ŷt+1,j ) = softmax(fgen (hNd )), in [25]. For the BILA-caption dataset, we used S3D [26]where Nv denotes a vocabulary size and fgen (·) denotes the
pretrained on the Howto100m dataset [27] to obtain 512computation by the first fully connected, GELU, LN, anddimensional features. Then, we used a fully connected layer
final fully connected layers.
to obtain 384-dimensional features.
The global loss function L is defined as:
The experimental setup was as follows: the optimizer,
learning rate, batch size, and number of epochs were Adam
L = λce LCE (yt+1 , p(ŷt+1 )) + λiwp Liwp (yt+1,1 , p(ŷt+1,1 ))(β1 : 0.9, β2 : 0.999), 1.0e-4, 16, and 25, respectively. For each
+ λcorr Lcorr + λmse LMSE (xt+1 , zt+1 ),module, Ne , denc , Nh , Nd , ddec , Nr , and drsa were 3, 384, 12,
where LMSE (·, ·) and LCE (·, ·) denote the mean square error
3, 384, 2, 384, respectively. For the loss function, λce , λiwp ,and the cross entropy loss functions, respectively. Here, λce ,λcorr , λmse were 30, 1.0, 0.1, 0.005, and 10, respectively. For
λcorr , λmse and λiwp are hyperparameters. Lcorr penalizes the
Liwp , we handled the words that appear more than nth times.
case where ŷt+1 describes an event that occurs before or after
We set nth to 30. We also sets W for the YouCook2-FC and∗t. Liwp penalizes the case where the predicted word yt+1,1
BILA-caption datasets to 3000 and 1000, respectively. The
is incorrect. This is defined as Liwp (yt+1,1 , p(ŷt+1,1 )) =number of trainable parameters of RFCM and the number ofγiwp LCE (yt+1,1 , p(ŷt+1,1 )), where γiwp = 1/W .
multiply-add operations are 3.1M and 540M, respectively.
5. EXPERIMENTS
The training was conducted on a machine equipped with
5.1. Dataset
an NVIDIA Tesla V100 SXM2 with 16 GB of GPU memory,
240 GB RAM, and an Intel Xeon Gold 6148 processor. It took
In the experiment, we evaluated our model on the YouCook26.2 and 1.6 × 10−1 h to train our model on the YouCook2-FC
FC and BILA-caption datasets. The YouCook2-FC dataset isand BILA-caption datasets, respectively. Similarly, inference
a dataset for the future captioning task. Generally, in cooktook 1.7×10−2 and 4.9×10−2 s/sample on the YouCook2-FC
ing, the next procedure is determined based on the previous
and BILA-caption datasets, respectively. As a condition forprocedure. We built the YouCook2-FC dataset based on theearly stopping, we used the generalization described in [28].
YouCook2 dataset [22]. We set the number of samples in5.3. Quantitative results
cluded in the training, validation, and test sets to 7435, 1569,
and 3035, respectively.
We compared RFCM with Memory-Augmented Recurrent
The BILA-caption dataset was newly built to evaluate fuTransformer (MART [7]). We selected MART as the baseline
ture captioning models that describe likely collisions in object
because it is a representative method for video captioning
placement tasks. Fig. 1 shows a typical sample from of thetasks and can be applied to future captioning tasks. TaBILA-caption dataset. To build the dataset, we extended SIGble 1 shows the quantitative results on the YouCook2-FC andTable 1. Quantitative comparison and ablation studies. The best scores are in bold.
YouCook2-FC

BILA-caption

Methods

BLEU4↑METEOR↑

ROUGE-L↑

CIDEr-D↑

BLEU4↑METEOR↑

ROUGE-L↑

CIDEr-D↑

MART [7]6.85±0.18

14.24±0.07

30.80±0.2120.86±1.07

19.01±0.74

21.02±0.4530.30±0.64

37.33±4.37

Ours (w/o RSA)
Ours (w/o Transformer Decoder)
Ours (RFCM)

6.70±0.36
6.68±0.13
7.03±0.15

14.14±0.50
14.09±0.15
14.53±0.09

30.18±0.18
30.16±0.31
30.49±0.2121.26±2.83
19.75±1.41
21.32±1.09

20.37±0.36
21.08±1.62
21.74±1.02

22.04±0.19
22.39±0.84
22.74±0.57

40.67±0.48
40.92±1.32
41.44±0.86

44.65±4.89
45.05±6.72
49.61±8.02

BILA-caption datasets. The mean and standard deviation
were computed on five experimental runs.
The evaluation of the generated sentences was basedon several standard metrics for video captioning tasks:
BLEU4 [29], ROUGE-L [30], METEOR [31], and CIDErD [32]. The primary metric was CIDEr-D.
First, we compared the models on the YouCook2-FC
dataset. The table shows that the CIDEr-D score was improved by 0.46 points. The BLEU4 and METEOR scores
were also improved by 0.18 and 0.29 points, respectively.
Next, we compared the models on the BILA-caption
dataset. The table shows that the CIDEr-D score was drastically improved by 12.28 points. The other metrics scores
were also improved. These results indicate that the RFCM
generated sentences more appropriately than the baseline.
5.4. Qualitative results
Figs. 3-4 show the qualitative results on the YouCook2-FC
and BILA-caption datasets. In the figures, the events areshown in chronological order. Some events are omitted due
to space limitations.
The sample in Fig. 3 illustrates a successful case on the
YouCook2-FC dataset. In this example, the object added tothe pan was “chopped tomatoes.” The baseline method incorrectly described it as “the chopped onions and ginger.” In contrast, our method appropriately described it as “tomato puree.”This result indicates that our method was able to appropriately
predict the next step and generate a caption.
Similarly, the sample in Fig. 4 illustrates a successful caseon the BILA-caption dataset. In this example, the grasped object was “the white bottle” and the collided object was “thecamera.” The baseline method incorrectly described the collided object as “a black teapot.” In contrast, our method appropriately described them as “a white jar” and “the camera,”
respectively. This result indicates that our method could appropriately describe the characteristics of the objects.
5.5. Ablation study
We conducted ablation studies for each module. Table 1 quantitatively shows the mean and standard deviation of five experimental runs. We investigated which module contributed
the most to the performance improvement using two ablation
conditions: (a) w/o RSA, where we used the standard MHA
layer [2] instead of the RSA layer, and (b) w/o transformer
decoder, where we removed the transformer decoder.
Comparing the results obtained under conditions (a) and(b) with the results of the proposed method on the YouCook2FC dataset, the CIDEr-D score decreased by 0.06 and 1.57

Fig. 3. The successful examples on the YouCook2-FC dataset. Top
figures show events. The bottom table shows a reference sentence
and sentences generated by the baseline and our methods.

Fig. 4. The successful examples on the BILA-caption dataset.
points, respectively. This indicates that the transformer decoder contributed the most to the performance improvement.
Similarly, comparing results obtained under the conditions (a) and (b) with the results of the proposed method on
the BILA-caption dataset, the CIDEr-D score decreased by
4.96 and 4.56 points, respectively. This indicates that the RSA
layer contributed the most to the performance improvement.
6. CONCLUSIONSIn this paper, we focused on the future captioning task, which
is a task to generate a description about a future event. Specifically, we proposed a future captioning model for daily tasks.
The main contributions of this paper are as follows:• We proposed the RFCM, a crossmodal language generation model that can generate a description about afuture event.
• The RSA Encoder is introduced to extract the relationships between events more effectively than the conventional self-attention in transformers.
• The RFCM outperformed the baseline method on two
datasets, BILA-caption and YouCook2-FC.

7. REFERENCES[1] Mehrdad Hosseinzadeh et al., “Video Captioning of Future
Frames,” in WACV, 2021, pp. 980–989.

[19] Jie Lei, Licheng Yu, Tamara Berg, et al., “What is More Likely
to Happen Next? Video-and-Language Future Event Prediction,” in EMNLP, 2020, pp. 8769–8784.

[2] Ashish Vaswani, Noam Shazeer, et al., “Attention Is All You
Need,” in NeurIPS, 2017, pp. 5998–6008.

[20] Bohan Wu, Suraj Nair, et al., “Greedy Hierarchical Variational
Autoencoders for Large-Scale Video Prediction,” in CVPR,
2021, pp. 2318–2328.

[3] Manjin Kim, Heeseung Kwon, Chunyu Wang, et al., “Relational Self-Attention: What’s Missing in Attention for Video
Understanding,” in NeurIPS, 2021.

[21] Jacob Devlin et al., “BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding,” in NAACL-HLT,
2019, pp. 4171–4186.

[4] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, et al.,“Video Captioning via Hierarchical Reinforcement Learning,”
in CVPR, 2018, pp. 4213–4222.

[22] Luowei Zhou, Chenliang Xu, and Jason Corso, “Towards
Automatic Learning of Procedures From Web Instructional
Videos,” in AAAI, 2018, pp. 7590–7598.

[5] Ranjay Krishna, Kenji Hata, et al., “Dense-Captioning Events
in Videos,” in CVPR, 2017, pp. 706–715.

[23] Tetsunari Inamura et al., “Development of Robocup @ Home
Simulation towards Long-term Large Scale HRI,” in Robot
Soccer World Cup, 2013, pp. 672–680.

[6] Kelvin Xu, Jimmy Ba, Ryan Kiros, et al., “Show, Attend andTell: Neural Image Caption Generation with Visual Attention,”in ICML, 2015, pp. 2048–2057.
[7] Jie Lei, Liwei Wang, et al., “MART: Memory-Augmented
Recurrent Transformer for Coherent Video Paragraph Captioning,” in ACL, 2020, pp. 2603–2614.
[8] Zakir Hossain, Ferdous Sohel, Mohd Shiratuddin, et al., “A
Comprehensive Survey of Deep Learning for Image Captioning,” ACM CSUR, vol. 51, no. 6, pp. 1–36, 2019.
[9] Motonari Kambara and Komei Sugiura, “Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions,” IEEE RA-L, vol. 6, no. 4, pp. 8371–8378,
2021.
[10] Aly Magassouba, Komei Sugiura, et al., “Predicting and attending to damaging collisions for placing everyday objects inphoto-realistic simulations,” Advanced Robotics, vol. 35, no.
12, pp. 1–13, 2021.
[11] Nayyer Aafaq, Ajmal Mian, et al., “Video Description: A
Survey of Methods, Datasets, and Evaluation Metrics,” ACM
CSUR, vol. 52, no. 6, pp. 1–37, 2019.
[12] Wentian Zhao, Xinxiao Wu, et al., “Multi-modal Dependency
Tree for Video Captioning,” in NeurIPS, 2021.
[13] Chen Sun, Austin Myers, Carl Vondrick, et al., “VideoBERT:
A joint model for video and language representation learning,”
in ICCV, 2019, pp. 7464–7473.
[14] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, et al.,“UniVL: A Unified Video and Language Pre-training Modelfor Multimodal Understanding and Generation,” arXiv preprint
arXiv:2002.06353, 2020.
[15] Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu,“Sketch, Ground, and Refine: Top-Down Dense Video Captioning,” in CVPR, 2021, pp. 234–243.
[16] Jonghwan Mun, Linjie Yang, et al., “Streamlined Dense Video
Captioning,” in CVPR, 2019, pp. 6588–6597.
[17] Yuki Mori, Tsubasa Hirakawa, et al., “Image Captioning in
Near Future from Vehicle Camera Images and Motion Information,” in IEEE IV, 2021, pp. 1378–1384.
[18] Tahmida Mahmud, Mohammad Billah, et al., “Prediction and
Description of Near-Future Activities in Video,” CVIU, vol.
210, pp. 103230, 2021.

[24] “World Robot Summit 2018 Partner robot challenge VirtualSpace Rules & Regulations,” 2018.
[25] Simon Ging et al., “COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning,” in NeurIPS,
2020, pp. 22605–22618.
[26] Antoine Miech, Jean-Baptiste Alayrac, et al., “End-to-End
Learning of Visual Representations from Uncurated Instructional Videos,” in CVPR, 2020, pp. 9879–9889.
[27] Antoine Miech et al., “HowTo100M: Learning a Text-Video
Embedding by Watching Hundred Million Narrated Video
Clips,” in ICCV, 2019, pp. 2630–2640.
[28] Lutz Prechelt, “Automatic early stopping using cross validation: quantifying the criteria,” Neural Networks, vol. 11, no. 4,
pp. 761–767, 1998.
[29] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing
Zhu, “BLEU: a method for automatic evaluation of machinetranslation,” in ACL, 2002, pp. 311–318.
[30] Chin-Yew Lin, “ROUGE: A package for automatic evaluationof summaries,” in Text summarization branches out, 2004, pp.
74–81.
[31] Satanjeev Banerjee and Alon Lavie, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation
with Human Judgments,” in the ACL Workshop on IEEM for
MTS, 2005, pp. 65–72.
[32] Ramakrishna Vedantam et al., “CIDEr: Consensus-based Image Description Evaluation,” in CVPR, 2015, pp. 4566–4575.