Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4275–4284
Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Visual Grounding Annotation of Recipe Flow Graph
Taichi Nishimura1 , Suzushi Tomori1 , Hayato Hashimoto1 , Atsushi Hashimoto2 ,Yoko Yamakata3 , Jun Harashima4 , Yoshitaka Ushiku2 , Shinsuke Mori5
1

Graduate School of Informatics, Kyoto University, 2 OMRON SINIC X Corporation
Graduate School of Information Science and Technology, The University of Tokyo
4
Cookpad Inc, 5 Academic Center for Computing and Media Studies, Kyoto University
1,5
Yoshidahonmachi, Sakyoku, Kyoto, Japan, 606-8501
25-24-5, Hongo, Bunkyo-ku, Tokyo, Japan, 113–0033
3
7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan, 113–8656
4
4-20-3 Ebisu, Shibuya-ku, Tokyo, Japan, 150–6012
1{nishimura.taichi.43x,tomori.suzushi.72e,hashimoto.hayato.73e}@st.kyoto-u.ac.jp,
2{atsushi.hashimoto, yoshitaka.ushiku}@sinicx.com,
3
yamakata@mi.u-tokyo.ac.jp, 4 jun-harashima@cookpad.com, 5 forest@i.kyoto-u.ac.jp
3

Abstract
In this paper, we provide a dataset that gives visual grounding annotations to recipe flow graphs. A recipe flow graph is a representationof the cooking workflow, which is designed to understand the workflow from natural language processing. Such a workflow will increase
its value when grounded to real-world activities, and visual grounding is a way to do so. Visual grounding is provided as boundingboxes to image sequences of recipes, and each bounding box is linked to an element of the workflow. Because the workflows are also
linked to the text, this annotation gives visual grounding with workflow’s contextual information between procedural text and visualobservation in an indirect manner. We subsidiarily annotated two types of event attributes with each bounding box: “doing-the-action,”or “done-the-action”. As a result of the annotation, we got 2,300 bounding boxes in 272 flow graph recipes. Various experiments showedthat the proposed dataset enables us to estimate contextual information described in recipe flow graphs from an image sequence.
Keywords: Procedural Text, Bounding Box, Flow Graph, Visual Grounding

1 Introduction
Procedural texts are suitable for the target of natural language understanding (NLU) because they are goal-oriented
descriptions and we can almost define the understandingof their goals. Some studies have proposed the frameworkof understanding procedural texts with graphs, which
can represent the entire workflow (Mori et al., 2014;
Jermsurawong and Habash, 2014;
Kiddon et al., 2015).
These frameworks would be helpful for real-world systems,
such as smart kitchen (Hashimoto et al., 2008) and cooking
robot (Bollini et al., 2013), to understand the context ofthese workflows and execute next actions. Therefore, it isessential to ground visual observations (images or videos)with procedural texts.
Visual grounding is one of the solutions to help computers
to understand which objects are aligned with textual
descriptions. In the previous studies, they targetted thepair of general texts and visual observations. MSCOCO(Lin et al., 2014), Flickr30k (Plummer et al., 2015), andYouTube-BoundingBox (Real et al., 2017) are typicaldatasets for such tasks. They reported that annotatingbounding boxes with a textual description helps computers
to know which objects they have to pay attention to forimage captioning (Xu et al., 2015; Cornia et al., 2019),
visual question answering (VQA) (Fukui et al., 2016), andsome other visual grounding tasks (Huang et al., 2017;
Bojanowski et al., 2015; Zhang and Lu, 2018). Comparedwith such general visual grounding tasks, a dataset withprocedural texts is differentiated by an additional goal ofunderstanding the context information, i.e., a model should
consider the entire workflow of a procedural text. We callthis task contextual visual grounding.
In this background, this paper focuses on the domain of
cooking recipes and provides a new dataset that has visual grounding annotation of contextual information represented in recipe flow graph (r-FG) (Mori et al., 2014),
which can represent a structured workflow described in aprocedural text. Figure 1 shows an overview of our annotation scheme. Given a procedural text with its image sequence and flow graph, annotators put a bounding box ofan object in an image, which is linked to a node of the flow
graph. This annotation enables us to link visual objects tocontextual information directly. Through the process, as a
subsidiary contribution, we found that visual objects represent roughly in either of the following two events: “doing-”and “done-the-action.” In the dataset, we also provide attribute labels for these events. For example, in the “step 1”
in Figure 1, the chef is just dicing the broccoli, whereas thechef has done the dicing action in the “step 2” of Figure 1.
To make it obvious whether an action to a visual object is inperformance or has already been performed, we annotatedeach bounding box with an event attribute.
As a result of the annotation, we built an r-FG bounding box(r-FG-BB) dataset, which consists of 2,300 bounding boxes
in 272 r-FGs. In the experiments, we confirmed that, with
our dataset, we could obtain a model that grounds boundingboxes with nodes in a flow graph, which gives a connection
between the bounding boxes of images, while classifying
event attributes correctly.
Our dataset allows contextual visual grounding research.
Grounded caption generation is a typical usage of ourdataset, where the pairs of a node in the flow graph4275

Figure 1: An overview of our annotation.
Concept tag
F
T
D
Q
Ac
Af
Sf
St

Total

Meaning
Food
Tool
Duration
Quantity
Action by the chef
Action by foods
State of foods
State of tools–

Frequency
12.37
3.59
0.73
0.74
13.21
2.67
3.03
0.31
36.65

Table 1: Recipe named entity (r-NE) tags and frequencies
per recipe in our r-FG-BB dataset.

and bounding boxes help a model understand which region they have to pay attention to at decoding a sentence.
One may leverage the pairs for multimodal state tracking (Yagcioglu et al., 2018; Amac et al., 2019) by tracing
nodes via arcs in the flow graphs.

2 Recipe Flow Graph Corpus
The language part of our r-FG-BB dataset is compatible
with the r-FG corpus (Mori et al., 2014). The r-FG corpus consists of cooking recipe texts (or simply “recipes”
hereafter) annotated with flow graphs connecting important
terms in the texts. Our r-FG-BB dataset is an extension connecting some of the important terms to bounding boxes in
the image attached to each step in the text. In this section,
we explain cooking recipe texts and flow graphs of the r-FG
corpus.

2.1

Cooking Recipe Text

A recipe describes instructions for a dish. The left part of
Figure 1 shows an example in Japanese with its English
translation. A recipe has a sequence of steps, and a step
consists of instruction sentences. The recipes in the r-FG
corpus are randomly selected from a dataset consisting ofrecipes downloaded from the recipe hosting service Cookpad1 . In addition to texts, Cookpad allows users to attach
an image to each step to explain it better (see the image sequence column in Figure 1 for example). In our r-FG-BB
dataset, we capitalize on these characteristics.

2.2

Directed Acyclic Graph Representation

In a recipe, the order of the instructions is not fully but only
partially specified. For example, a chef needs to dice carrots before boiling them but does not need to dice potatoes to boil carrots. In this case, the action “dicing carrots”
has an order relationship with the action “boiling carrots,”
but not with the action “dicing potatoes.” Moreover, thediced carrots and boiled carrots are identical. A directed
acyclic graph (DAG) is suitable for representing such relationships. The r-FG is a DAG whose nodes correspond toimportant terms (word sequences) with a type in the recipe.
The arcs represent the relationships between two nodes. All
the nodes are connected indirectly to a single special node,root, corresponding to the final dish. Thus a meaning of arecipe in the r-FG corpus is represented as a rooted DAG.
2.2.1 Nodes
The nodes of an r-FG are the important terms appearing inthe sentences with a type. Table 1 lists the types and their14276

https://cookpad.com/ (Accessed on 2019/Nov/25)

Arc label
Agent
Targ
Dest
F-compT-compF-eq
F-part-of
F-set
T-eq
T-part-of
A-eq
V-tmother-mod

Total

Meaning
Action agent
Action target
Action destination
Food complement
Tool complement
Food equality
Food part-of
Food set
Tool equality
Tool part-of
Action equality
Head of a clause for timing
Other relationships–

Frequency
2.58
14.50
5.65
0.44
1.31
2.59
3.25
0.26
0.23
0.45
0.58
1.15
5.23
38.21
Figure 2: Annotation examples.

Table 2: Arc labels and their frequencies per recipe in ourr-FG-BB dataset.
frequencies per a recipe in the r-FG-BB dataset. An important term is a sequence of words annotated with a type
among a pre-defined set. This definition is similar to thatof named entities (NE) except for the type set. Thus in ther-FG corpus, it is called recipe NE (r-NE). Each node of aflow graph corresponds to an r-NE in the text. As we described above, there is a special node, root, corresponding
to the completed dish. In Figure 1, the root is the node
whose r-NE is “/ready to eat/Af.”
Our r-FG-BB dataset extends the r-FG corpus by annotatingbounding boxes in the images. We assume that each step inthe recipes of our dataset has an image. We annotate some
objects in the image with a bounding box and a link to the rNE in the corresponding step. As a first trial we limited ther-NE types into F (Food), T (Tool), and Ac (Action by thechef) among eight types listed in Table 1. The reason why
we excluded the others is that they are difficult or almost
impossible to be recognized from an image. In addition,
the statistics in Table 1 indicate that the three types occupy
approximately 79.6%. Thus we decided to take these threevisually obvious types, F, T, and Ac, leaving others for a
future challenge.
2.2.2 Arcs
An arc between two nodes indicates that they have a certain
relationship. The type of relationship is denoted by a labelof the arc. Table 2 lists the arc labels and their frequencies
per a recipe in our r-FG-BB dataset. Unlike an arc in a dependency parsing, an arc in an r-FG may connect two nodes
in different sentences. For example, in Figure 1, the r-NE
node “boil” in step 1 and “eggs” in step 3 are connected by
the F-eq arc indicating that the eggs in step 3 are identicalto the result of the action “boil the eggs hard”.
In our r-FG-BB dataset, all the bounding boxes are connected directly or indirectly by arcs in the r-FG. This characteristic distinguishes our r-FG-BB dataset from other existing visual datasets.

3 Dataset Annotation Standard
In this section, we explain our annotation framework in detail. As mentioned in Section 1, we performed two types ofannotations: bounding boxes with nodes in the flow graphand event attributes with bounding boxes. We describe
these annotations in the subsequent sections.

3.1

Bounding Box Annotation

Similar to other visually annotated datasets, we annotate
some objects in images with a bounding box. The edges
of a bounding box are parallel to the vertical or horizontal
edge of the image (assumed to be a rectangle). A bounding box, specified by two corner points, is the smallest area
covering the target object completely. In our r-FG-BB, welimited the target objects into those having a corresponding
r-NE in the flow graph, that had been manually annotatedto a recipe. We also annotate each bounding box with a linkto the corresponding r-NE. And some bounding boxes may
be linked to the same r-NE because the object can be multiple or separated in the image. Thus the relationship frombounding boxes to r-NEs is so-called one-to-many. These
links are the source of all the interesting points of this work.
For example, there are two bounding boxes contouring theeggs in the image at the top left of Figure 2 and they areconnected to a single r-NE “eggs.” By the linguistic annotation in the r-FG part, we can say many things about therelationships of this object with others.
A node of type F has a tendency to be linked to bounding boxes for unprocessed ingredients (see Figure 2). Here,“unprocessed ingredient” indicates a food item to which thechef has not performed any action yet (thus diced potatoes
and grilled chicken is not “unprocessed”). In some cases,
processed food is specified by a noun phrase corresponding to some bounding boxes. A node of type T should belinked to the bounding box of a tool. We also annotate tools
appearing only partially in an image with a bounding box
spreading to the edge of the image. Different from F and T,
the bounding box of an Ac is not defined in a straightforward way. It can be a bounding box of the tool to be usedor that of the processed food, which shows the result of theaction Ac.

3.2

Event Attribute Annotation

In addition to the link between an Ac and a bounding box,
we annotate the link with an attribute indicating that the4277

#Steps#Sent.

#r-NEs#Non-Leaves
36.65
13.48
23.16

#Words#Char.

113.99173.91#Leaves
4.97

7.34

Table 3: Per-recipe statistics of our r-FG-BB corpus of 272
recipes.

Agreement rate

F
0.81

T
0.88

Ac
0.76

Total
0.79

Table 4: Agreement rate between two annotators

Figure 3: Annotation screen of developed web tool.
image shows an on-going action (Acing ) or a completed one(Aced ). Note that this attribute is only for Ac. We call thisan event attribute. For example, at the top right in Figure 2,
the link between the action “Dice” in the text below andthe blue bounding box should have an event attribute Acing
indicating that the action is on-going. And at the bottom
right, the link between the action “Cut” in the text below
and the blue bounding box would have the attribute Aced
that indicates that the action is completed. In some cases, it
is unclear whether the action is on-going or completed. We
allow an annotator to label such a case with Acunc .

3.3

Annotation Web Tool

To facilitate annotation, we developed a web tool. Figure 3
shows a screenshot. On the left-hand side, the tool shows
an image corresponding to a step. On the right-hand side,
the tool shows all the steps of a recipe with all F, T, and Ac
nodes highlighted with a color depending on the type.
The annotator checks all r-NEs from the first to the lastin the step whose image is shown on the left-hand side
whether the image shows the corresponding object of the
focused r-NE or not. If it shows the object, the annotatordraws its bounding box in the image. The tool allows theannotator to adjust the box by moving or rescaling it. The
tool also has a function to delete a bounding box.
After annotating a bounding box, the annotator connects itwith a link to the node in the flow graph. If the node isan Ac, the annotator is forced to select one event attributefrom three categories (Aced , Acing , and Acunc ) listed in thepull-down menu.
A typical case for Acunc is an instruction “/Add/Ac salt toboiling water” with an image of boiling water only. Fromthe image, it is almost impossible to judge if the salt has
been added or not yet. The label Acunc allows the annotator
to avoid making a difficult decision.

4 Annotation Statistics
As the recipes for our r-FG-BB dataset, we selected, fromthe r-FG corpus, those having an image to each step. As aresult, we had 70 recipes. To increase the number of recipes
in our corpus, we also selected 202 recipes randomly fromthose consisting of 3 to 10 steps in the Cookpad Image

Dataset (Harashima et al., 2017) under the same condition.
The r-FG for each recipe was annotated carefully by thesame annotators as the r-FG corpus. Combining the above
two sources, we obtained 272 recipes and their r-FG annotations. Table 3 shows the statistics of the r-FG part ofour r-FG-BB dataset. Then two annotators drew boundingboxes in the images of these recipes and connected them tor-NEs.
In this section, we first report the agreement rates between
the two annotators decomposing the annotation into bounding boxes, selections of r-NEs connected to them, and event
attributes. Then we describe the procedure we have executed to build our r-FG-BB dataset considering the agreement rate. Finally, we show its statistics.

4.1

Agreement Rate

We calculated the agreement rates of r-NEs and bounding boxes (selection problems) like other image datasets.
The agreement rate is defined as the intersection over union(IoU) calculated by the following formula:
IoU(SA , SB ) =|SA ∩ SB |,|SA ∪ SB |(1)

where SA and SB are the sets of annotations by the annotator A and B, respectively.
For an evaluation of event attribute annotation (a classification problem), we calculated the confusion matrix.
4.1.1 r-NE
In the r-NE case, an element of the set S is an r-NE selected
to connect to bounding boxes. We calculated the agreement
rate of selected r-NEs between two annotators for each typeand in total.
Table 4 shows the result. We see that the agreement rate ishighest for T and lowest for Ac. The reason is that tools(T) are solid and do not change the shape, but the object
corresponding to an Ac tends to be processed food, which
is sometimes difficult to be identified completely. We seethat the total IoU of 79% can be seen relatively high to build
a dataset.
4.1.2 Bounding Box
In the case of bounding boxes annotation, an element of theset S is an area specified by the bounding box. We first
calculated the pixel-wise IoU for each pair of the bounding boxes annotated by the two annotators. We limited thebounding boxes to those connected to an r-NE selected byboth the annotators. In some cases, an annotator gives more

4278

Table 5 shows the numbers of r-NEs connected to boundingboxes per recipe and its average numbers of the boundingboxes. The bounding boxes from the two annotators arecounted independently. Hence, some regions are doublecounted. In this table, the numbers of Acing and Aced arecounted only when the two annotators are in agreement. We
see that the number of F is very small because raw ingredients are rarely photographed.

5
Figure 4: Event attribute confusion matrix between two annotators.
#r-NEs#BBs/#r-NEs

F
0.16
2.00

T
1.28
2.00

Ac (Acing , Aced )2.44 ( 0.62, 1.82)
2.28 ( 2.59, 2.18)

Total
3.88
2.18

Table 5: Numbers of r-NEs connected to bounding boxes(BBs) per recipe and numbers of bounding boxes per r-NE.
than one bounding box corresponding to the same r-NE.
Thus we calculated IoU scores of all the combinations of
the bounding boxes connected to the same r-NE. Then we
calculate the average IoU.
The average IoU was 0.772. In the computer visioncommunity, researchers build their datasets with bounding boxes that have IoUs larger than 0.5 (Su et al., 2012;
Papadopoulos et al., 2017). Therefore, we can say thatmany annotated bounding boxes are suitable for the dataset.
4.1.3 Event Attribute
As we described, if the connected r-NE type is Ac, the annotator chooses one event attribute among Acing , Aced , andAcunc . We limited the Acs to those selected by both the annotators. Since the numbers of total annotations by the twoannotators are the same, we calculated the confusion matrix. Figure 4 shows the result. As we see from the table,
87% (= 859/990) of the event attributes agreed. In terms of
Acunc , there are very few tags selected by each annotator.
The most frequent attribute that two annotators agreed isAced . This may be because the recipe authors need to clean
up their hands to use the camera and from processed foods,
it is obvious that the action has been completed.

4.2

Dataset Construction

After the annotation, we applied the following two filtering
processes to construct our r-FG-BB dataset.
1. We selected pairs of an r-NE and bounding boxes fromthe annotation result only when the two annotators arein agreement. As a result, 79.3% of the annotated rNEs are extracted as shown in Table 4. Then, we filtered out the pairs of an Ac and bounding boxes when
its event attribute is Acunc .
2. We further filtered out pairs of an r-NE and boundingboxes if the IoU of the bounding boxes is over 0.7.
Finally, we obtained 2,300 pairs of an r-NE and boundingboxes for our r-NE-BB dataset consisting of 272 recipes.

Experimental Evaluation

To investigate the usefulness of our dataset, we propose
three tasks based on its most interesting characteristics:
symbol grounding, bounding box linking prediction, andevent attribute classification. In this section, we define thesetasks, give a preliminary solution to each one, and report theexperimental results.

5.1

Symbol Grounding

5.1.1 Task Definition
Contextual visual grounding is a visual grounding task that
requires a model to understand contextual information described in a procedural text. Given a bounding box in animage and the corresponding step to the image, the task is
to select an r-NE (F, T, or Ac) in the step. Formally, let a
step be W = (w1 , w2 , . . . , wn , . . . , wN ), where wn is then-th word and N is the number of words in the step, and ther-NEs in the step be R = (r1 , r2 , . . . , rk , . . . , rK ), where
K is the number of r-NEs in the step. The goal of this task
is to select one r-NE r̂k among R given a bounding box andthe word sequence of the step corresponding to the image
W . To investigate whether the flow graph is effective ornot, we allow some models to refer to the flow graph of theentire recipe.
5.1.2 Proposed Model
To build a model for the symbol grounding task, we haveto consider that the step W contains each r-NEs R. Thus
we adopted an embedding-based approach, which enables
us to calculate the similarity between all combinations of
the bounding box and candidate r-NEs. The r-NE that has
the highest score is selected to be the answer.
Figure 5 shows an overview of our approach. (i) We convert the bounding box and candidate r-NEs into feature vectors. For the bounding box, the image encoder calculates
the image vector v. Similarly for the corresponding step
W the textual encoder outputs the sequence of word embedding vectors W = (w1 , w2 , . . . , wn , . . . , wN ). Then,
we calculate the embedding vector of the k-th r-NE as theaverage of the vectors of the words consisting that r-NE:
R = (r 1 , r 2 , . . . , r k , . . . , r K ). (ii) For models referringto the flow graph, we concatenate two vectors to r k . One
is the preceding context vector r pk calculated as the average of the vectors of all the r-NEs in-coming to the k-th
r-NE in the flow graph. The other is the following context vector r fk calculated as the average of the vectors of
all the r-NEs to which the k-th r-NE is out-going in theflow graph. If the k-th r-NE is a leaf or the root, we set
random values to all the elements of r pk or r fk , respectively. These vectors are expected to represent the context
information. Thus, the enhanced r-NE vectors are given4279

Figure 5: An overview of our symbol grounding model.
We used the pre-trained neural networks for the image andtextual encoder.
random baseline
F
T
Ac
Total
LSTM w/o flow graph
F
T
Ac
Total
LSTM w/ flow graph
F
T
Ac
Total
BERT w/o flow graph
F
T
Ac
Total
BERT w/ flow graph
F
T
Ac
Total

Recall
0.023
0.091
0.429
0.189

Precision
0.250
0.039
0.221
0.174

F1
0.042
0.054
0.291
0.181

0.250
0.480
0.634
0.580

0.250
0.461
0.662
0.592

0.250
0.471
0.648
0.586

0.125
0.772
0.612
0.608

0.250
0.654
0.603
0.602

0.167
0.708
0.607
0.605

0.000
0.720
0.762
0.733

0.000
0.750
0.716
0.717

0.000
0.734
0.739
0.725

0.000
0.782
0.785
0.734

0.000
0.750
0.761
0.750

0.000
0.766
0.772
0.742

Table 6: Result of the symbol grounding experiment.
as uk = concat(r pk , r f , r hk ), where connat(·) is the vector concatenation function. When we do not use the flow
graph representation, we skipped this phase. Thus r-NE
vectors are given as uk = r k . (iii) We trained the twoneural networks, f from the textual branch and g fromthe image branch, jointly to calculate the common mapping space from the r-NE vectors and the image vectors.
The goal is to train these networks to embed a given rNE/bounding box pair at closer positions while others in
the distance. To achieve this, we used the triplet margin
loss (Balntas et al., 2016) with the distance function as the
cosine distance.
5.1.3 Results
To perform this experiment, we used all the pairs of an
r-NE and a bounding box in our r-FG-BB dataset. We

Figure 6: An outline of our bounding box linking classifier. The image encoder and textual encoder are the samemodels to Section 5.1.
split the dataset into 80% for training, 10% for validation, and 10% for test. As the image encoder, we used
ResNet-50 (He and Sun, 2016) pre-trained with ImageNet(Deng et al., 2009), which is one of the state-of-the-art
image classifiers. We used the classifier after removing
the last layer with softmax; thus, the output layer dimension is 2048. For the textual encoder, we prepared
two choices: LSTM and BERT (Devlin et al., 2019). We
trained them using 0.5 million recipes from the Cookpad
Dataset (Harashima et al., 2016). For training LSTM, weused the tokenizer KyTea2 (Neubig et al., 2011). We replaced words appearing less than three times with an unknown word symbol to have a 17,982-word vocabulary
for the LSTM. The perplexity was 29.16. For the BERTbased model, we employed a pre-trained BERT model on
the Wikipedia corpus. Then, we fine-tuned the model withrecipes in the same manner as the LSTM training. The vocabulary size of BERT is 32,000 after converting words into
sub-word units (Sennrich et al., 2016).
Table 6 shows the results. Independently from the textual
encoder, methods referring to the flow graph performed better than those without flow graph reference. This suggeststhat the context information from the flow graph helps themmodel link a bounding box in an image and a named entity
in the corresponding text. We also see that BERT-basedmodels performed better than those based on LSTM. Thisis consistent with other tasks in natural language processing.

5.2

Bounding Box Linking Prediction

5.2.1 Task Definition
In our r-FG-BB dataset, some bounding boxes are connected through the flow graph. Thus we can try an interesting novel task: bounding box linking. We defined thetask as a classification task to predict whether there is alink or not between a given pair of bounding boxes in two
subsequent bounding boxes.
5.2.2 Proposed Model
Figure 6 shows an overview of our proposed approach. The
inputs are two bounding boxes in subsequent images corresponding to the k-th and (k + 1)-th steps. (i) Similar to24280

http://www.phontron.com/kytea/

Baseline
RandomCosine similarity (α = 0.8)
Proposed method
Image
Image + LSTM
Image + BERT

Accuracy
0.483
0.533
0.667
0.583
0.817

Table 7: Accuracies of bounding box linking classification.
the previous task, we converted the bounding boxes into theimage vectors, v 1 andv 2 . Then we calculated the step vectors, s1 ands2 , as the average vectors of the words in thestep. (ii) We concatenated them and obtain the feature vector u = concat(v 1 , v 2 , s1 , s2 ). Finally, the feature vector u is fed to a classifier h, which is a fully-connected
two-layer network followed by a sigmoid activator. In theclassification phase we take argmax to output whether there
is a link (y = 1) or not (y = 0). In the training phase, we
minimized the binary cross-entropy loss with the Adam optimizer (Kingma and Ba, 2015).
5.2.3 Results
To perform this experiment, first, we split the r-FG-BB
dataset into 80% for training, 10% for validation, and 10%
for test. Then we extracted pairs of bounding boxes fromeach of two subsequent steps. Finally we checked whether
the two bounding boxes have a link or not in the flow graphto prepare the label (y = 1 or 0).
In this experiment, we employed two baselines: random
and cosine similarity. The random baseline investigates
whether the link label is biased or not. In “cosine similarity,” the model calculates the cosine similarity between
the image vectors v 1 and v 2 and judges that there is a link
if the score is higher than a threshold α, which were set to
0.8. Note that two image vectors are L2 normalized.
Table 7 shows the results of these two baselines and the
proposed methods. The result of the random baseline showsthat the task is a difficult binary classification problem without the model. The accuracy of the “cosine similarity” isslightly higher than that of the random This indicates that
this task is not so simple to be able to be answered correctly
without textual information. Compared with the baselines,
the proposed models could predict bounding box linking
more correctly. In the proposed method, the model learned
with only images achieved the highest precision, but thatlearned with texts using BERT in addition to the images
achieved the highest recall and F1. In contrast, the resultof the model using LSTM was even worse than that trainedwith only images. This indicates that a sophisticated textual
representation is necessary to solve this task.

5.3

Event Attribute Classification

5.3.1 Task Definition
Each bounding box has two event attributes, “doing the action” and “done the action”. It is important for a computer
to be able to distinguish between these two from visual andtextual information because real-world systems require amodel to know the status of actions. Our dataset has an
event attribute for each bounding box connected to an ac-

Figure 7: An overview of our event attribute classification
model. The encoders are also the same model to the symbol
grounding models described in Section 5.1.

Image
+ LSTM w/o flow graph+ LSTM w/ flow graph+ BERT w/o flow graph+ BERT w/ flow graphAccuracy
0.750
0.839
0.893
0.839
0.875

Table 8: Accuracies of event attribute classification.
tion node Ac. The labels are Acing for on-going action andAced for a completed one. Thus our dataset allows us to test
methods for this classification as well. We formulated thisproblem as a binary classification task. Given a boundingbox connected to Ac along with the step, we train models to classify whether the action is on-going or completed(y = Acing or Aced ).
5.3.2 Proposed Model
Figure 7 shows an outline of our event attribute classification model. (i) Through the encoders, we get the image
embedding vector v and the r-NE vector r. (ii) If the modelis allowed to refer to a flow graph, we concatenated the preceding and following context vectors r p , r f to the r in thesame manner as Section 5.1. We get the feature vector by
conncatenating them: u = concat(v, r p , r, r f ). Then, we
provide it to a fully-connected two-layer network followed
by a sigmoid activator. The network outputs whether the
event attribute is Acing or Aced . In the training phase, we
searched for the parameters that minimize the binary crossentropy loss by Adam optimizer. (Kingma and Ba, 2015).
5.3.3 Results
To perform this experiment, we extracted pairs of an Ac
and bounding box whose event attributes is Acing or Aced .
Then, we split the data into 80% for training, 10% for validation, and 10% for test. Table 8 shows the results ofthe models. Before this experiment, we anticipated thatthe model could tell them apart only from image information because object appearance is significantly different between visual information annotated with Acing and Aced .
The model could distinguish them to a certain extent (accuracy: 75%), but the result revealed that incorporating textual encoders such as BERT and LSTM contributed to ahigher score. From this result, we can safely say that lin-4281

guistic information is effective for event attribute classification. Compared with the models without reference to theflow graph, the models with it perform better with both textual encoders, LSTM and BERT. This result indicates thatthe flow graph representation also helps the model to solve
this task.

6

Application

Our r-FG-BB dataset has many potential applications. In
this section, we describe two typical usages of our dataset.

6.1

Grounded Image Captioning

Grounded image captioning is an essential problem in thecomputer vision and natural language communities. In
this field, to output grounded caption, many researchers
have incorporated the attention mechanism into a model(Xu et al., 2015; Cornia et al., 2019). This attention mechanism encourages models to know which objects and regions they have to pay attention to when decoding words,
and it helps them to generate visually grounded captions.
Our dataset provides researchers with pairs of an r-NE
and bounding boxes, which leads a model to decode visually grounded words correctly. Moreover, we would
like to emphasize that the text part is not a general text
but a procedural text. Procedural text generation froman image sequence is a prominent area because it requires a model to consider the context and the output
coherency (Nishimura et al., 2019; Chandu et al., 2019).
Thus these studies focused on generating grounded captions: incorporating a structure into the model implicitly(Chandu et al., 2019) and preferentially decoding important terms (Nishimura et al., 2019). Our dataset accelerates this research because pairs of an r-NE and boundingboxes help a model output these terms, considering the entire workflow using the flow graph.

6.2

QA dataset, which requires a model to understand not only
flow graphs but also bounding boxes. For example, given
an image showing mixing ingredients in a bowl, the question is, “What ingredients are used in it?”. To answer thisquestion correctly, the model must understand the flow of
all the ingredients and their visual locations.

Multimodal State Tracking

State tracking, which detects changes of an object andits identicality in a text, is an essential problem for
NLU. Recently, procedural texts are becoming the target
for NLU researchers because their understanding requiresa model to anticipate the implicit causal effects of actions on entities (Bosselut et al., 2018; Tandon et al., 2018;
Mishra et al., 2018; Yagcioglu et al., 2018).
Some researchers tried to build a dataset for procedural text understanding (Tandon et al., 2018) for the scientific domain,and other researchers expand them to multimodal version(Yagcioglu et al., 2018) for the cooking area. Some researcher has proposed a dataset for the understanding ofprocedural texts using a “grid,” which represents changesof an object state using a table format (Tandon et al., 2018;
Mishra et al., 2018). Our dataset provides a flow graphof a procedural text, which represents an entire workflow
as a rooted DAG, and bounding boxes in an image connected to nodes in the flow graph. Our experimental results
showed that the model could predict which textual description is related to a bounding box and whether there exists
a linking between bounding boxes or not. Thus, sequential
bounding boxes and a flow graph will be useful for multimodal state tracking. It would be interesting to build a

7 Conclusion
In this paper, we presented details of our r-FG-BB dataset.
Its language part is procedural texts, each of which is annotated with a flow graph whose nodes correspond to important terms in the text and are connected to form a DAG. The
visual part of the dataset is bounding boxes in the images
attached to each text. The bounding boxes are connected to
the nodes of the flow graph, allowing the dataset users tohave contextual information of the visual objects.
We performed various experiments. The results showedthat the proposed simple models could ground boundingboxes with nodes in a flow graph, which gives a linking
between the bounding boxes over images while classifying
event attributes correctly. Therefore, our dataset is usefulfor contextual visual grounding.
With our dataset, one can try contextual visual grounding
research using interesting triplets: bounding boxes, procedural text, and its flow graph. This combination would be
effective for understanding the entire workflow of a procedural text, grounding contextual visual information.

Acknowledgement
This work was supported by JST ACT-I Grant Number JPMJPR17U5.

References
Amac, M. S., Yagcioglu, S., Erdem, A., and Erdem, E.
(2019). Procedural reasoning networks for understanding multimodal procedures. In Proceedings of the Conference on Computational Natural Language Learning.
Balntas, V., Riba, E., Ponsa, D., and Mikolajczyk, K.
(2016). Learning local feature descriptors with triplets
and shallow convolutional neural networks. In Proceedings of the British Machine Vision Conference, pages 1–
11.
Bojanowski, P., Lajugie, R., Grave, E., Bach, F., Laptev,
I., Ponce, J., and Schmid, C. (2015). Weakly-supervisedalignment of video with text. In Proceedings of the International Conference on Computer Vision, pages 4462–
4470.
Bollini, M., Tellex, S., Thompson, T., Roy, N., and Rus, D.
(2013). Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481–495.
Bosselut, A., Levy, O., Holtzman, A., Ennis, C., Fox, D.,
and Choi, Y. (2018). Simulating action dynamics withneural process networks. In Proceedings of the International Conference on Learning Representations.
Chandu, K., Nyberg, E., and Black, A. W. (2019). Storyboarding of recipes: grounded contextual generation. In
Proceedings of the Annual Meeting of the Association for
Computational Linguistics, pages 6040–6046.

4282

Cornia, M., Baraldi, L., and Cucchiara, R. (2019). Show,
control and tell: a framework for generating controllable
and grounded captions. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages
8307–8316.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and FeiFei, L. (2009). Imagenet: A large-scale hierarchical image database. In Proceddings of the Conference on Computer Vision and Pattern Recognition, pages 248–255.
Devlin, J., Chang, M. W., Lee, K., and Toutanova, K.
(2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of
the Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell,T., and Rohrbach, M. (2016). Multimodal compact bilinear pooling for visual question answering and visualgrounding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages
457–468.
Harashima, J., Ariga, M., Murata, K., and Ioki, M. (2016).
A large-scale recipe and meal data collection as infrastructure for food research. In Proceedings of the International Conference on Language Resources and Evaluation, pages 2455–2459.
Harashima, J., Someya, Y., and Kikuta, Y. (2017). Cookpad image dataset: An image collection as infrastructure
for food research. In Proceedings of the International
ACM SIGIR Conference on Research and Development
in Information Retrieval, pages 1229–1232.
Hashimoto, A., Mori, N., Funatomi, T., Yamakata, Y.,
Kakusho, K., and Minoh, M. (2008). Smart kitchen: auser centric cooking support system. In Proceedings ofthe Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 848–854.
He, K., Z. X. R. S. and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages
770–778.
Huang, D. A., Lim, J. J., Fei-Fei, L., and Carlos Niebles, J.
(2017). Unsupervised visual-linguistic reference resolution in instructional videos. In Proceedings of the Conference on Computer Vision and Pattern Recognition,
pages 2183–2192.
Jermsurawong, J. and Habash, N. (2014). Predicting thestructure of cooking recipes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2370–2377.
Kiddon, C., Ponnuraj, G. T., Zettlemoyer, L., and Choi, Y.
(2015). Mise en place: unsupervised interpretation of
instructional recipes. In Proceedings of the Conferenceon Empirical Methods in Natural Language Processing,pages 982–992.
Kingma, D. P. and Ba, J. (2015). Adam: A method forstochastic optimization. In Proceedings of the International Conference for Learning Representations.
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P.,
Ramanan, D., Zitnick, C. L., and Dollar, P. (2014). Mi-

crosoft coco: common objects in context. In Proceedings of the European Conference on Computer Vision,
pages 740–755.
Mishra, B. D., Huang, L., Tandon, N., Yih, W. T., andClark, P. (2018). Tracking state changes in procedural
text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the Conferenceof the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pages 1595–1604.
Mori, S., Maeta, H., Yamakata, Y., and Sasada, T. (2014).
Flow graph corpus from recipe texts. In Proceedingsof the International Conference on Language Resources
and Evaluation, pages 2370–2377.
Neubig, G., Nakata, Y., and Mori, S. (2011). Pointwise
prediction for robust, adaptable japanese morphologicalanalysis. In Proceedings of the Conference of Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 529–533.
Nishimura, T., Hashimoto, A., and Mori, S. (2019). Procedural text generation from a photo sequence. In Proceedings of the International Conference on Natural Language Generation.
Papadopoulos, D. P., Uijlings, J. R., Keller, F., and Ferrari,
V. (2017). Extreme clicking for efficient object annotation. In Proceedings of the International Conference on
Computer Vision, pages 4930–4939.
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C.,
Hockenmaier, J., and Lazebnik, S. (2015). Flickr30k
entities: collecting region-to-phrase correspondences forricher image-to-sentence models. In Proceedings of theInternational Conference on Computer Vision, pages
2641–2649.
Real, E., Shlens, J., Mazzocchi, S., Pan, X., and Vanhoucke, V. (2017). Youtube-boundingboxes: A large
high-precision human-annotated data set for object detection in video. In Proceedings of the Conference on
Computer Vision and Pattern Recognition, pages 5296–
5305.
Sennrich, R., Haddow, B., and Birch, A. (2016). Neuralmachine translation of rare words with subword units. In
Proceedings of the Annual Meeting of the Association for
Computational Linguistics, pages 1715–1725.
Su, H., Deng, J., and Fei-Fei, L. (2012). Crowdsourcing
annotations for visual object detection. In Proceedingsof the Workshops at the AAAI Conference on Artificial
Intelligence.
Tandon, N., Mishra, B. D., Grus, J., Yih, W. T., Bosselut,A., and Clark, P. (2018). Reasoning about actions andstate changes by injecting commonsense knowledge. In
Proceedings of the Conference on Empirical Methods in
Natural Language Processing, pages 57–66.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R. S., and Bengio, Y. (2015). Show,
attend and tell: neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pages 2048–2057.
Yagcioglu, S., Erdem, A., Erdem, E., and Ikizler-Cinbis, N.
(2018). Recipeqa: A challenge dataset for multimodal

4283

comprehension of cooking recipes. In Proceedings of theConference on Empirical Methods in Natural Language
Processing, pages 1358–1368.
Zhang, Y. and Lu, H. (2018). Deep cross-modal projection
learning for image-text matching. In Proceedings of theEuropean Conference on Computer Vision, pages 686–701.

4284