Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.DOI

Structure-Aware Procedural Text
Generation from an Image Sequence
TAICHI NISHIMURA 1 , ATSUSHI HASHIMOTO 2 (Member, IEEE), YOSHITAKA USHIKU(Member, IEEE), HIROTAKA KAMEKO 3 , YOKO YAMAKATA 4 (Member, IEEE), and
SHINSUKE MORI3

2

1

Graduate School of Informatics, Kyoto University, Kyoto 606-8501, JapanOMRON SINIC X Corporation, Tokyo 113-0033, Japan
Academic Center for Computing and Media Studies, Kyoto University, Kyoto 606-8501, Japan
4
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8656, Japan
2
3

Corresponding author: Taichi Nishimura (e-mail: nishimura.taichi.43x@st.kyoto-u.ac.jp).

ABSTRACT It is an important activity for our society to create new value by combining materials. From
daily cooking to industrial manufacturing, procedural texts describe the way to do it allowing readers
to reproduce procedures for these activities. As pointed by some previous studies for natural language
understanding, one important property of the procedural text is its context dependency, which is the merging
operations of materials and can be represented by a graph or tree structure. This paper aims to investigate
the impact of explicitly introducing such a structure on the vision and language task of procedural text
generation from an image sequence. To this end, we propose (1) a new dataset, which extends a definitionof a tree structure merging tree to a vision and language version and (2) a novel structure-aware procedural
text generation model, which learns the context dependency efficiently. Experimental results show that the
proposed method can boost the performance of traditional versatile methods.
INDEX TERMS Natural language processing, text generation, procedural text, vision and language

I. INTRODUCTION

reason about the path to the goal.

An archive of procedural texts, which have been written in
the same abstraction level to human understanding, is an
accumulation of wisdom in our practice. It is an ambitious
challenge to obtain such an abstract representation of practical knowledge from low-level visual observation. As oneof the tasks to this goal, procedural text generation froman image sequence [1], [2] has been proposed to investigate machine’s capabilities in human-like understanding of
instructional visual stories. This task is rising in popularity
due to its goal-oriented property; the models must generate
coherent sentences.

Inspired by these ideas, we aim to represent the context
dependency in an image sequence as a structure and introduce it into the text generation models explicitly, allowing
the models to generate coherent sentences. To this end, we
propose (1) a new dataset, which extends the above proposed
structure definition to a vision and language version and (2)a novel structure-aware procedural text generation model,
which learns the context dependency efficiently.

To generate coherent sentences, we assume that the model
must preserve context dependency, which is the global dependency of actions and involved materials to a single final
product. Previous work to generate a procedural text from animage sequence [1], [2] thoroughly focused on representing
the global context in an image sequence, but it is not yet
reached in capturing the context dependency explicitly. In
terms of natural language understanding, some studies haverepresented the context dependency of a procedural text asa graph or tree structure [3]–[5], which leads the models to
VOLUME 4, 2016

Among the structure definitions proposed in previous studies, we refer to the Simplified Ingredient Merging Map in
Recipes (SIMMR) dataset [3], which consists of proceduraltexts (recipes) and its structure, a “merging tree.” A merging
tree is a tree structure describing the path through whichmaterials are merged into a single final product. We adopt
this structure because it is minimal but can fully express thecontext dependency for a procedural text generation task. As
mentioned above, SIMMR is annotated to a dataset withoutvisual data. Thus, to provide a vision and language version,
we extend SIMMR to a new dataset called visual SIMMR(vSIMMR), where each merging tree is annotated to an
image sequence in a subset of the traditional world-largest
1

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

dataset1 .
Using this dataset, we propose a structure-aware procedural text generation model that gains the global coherency by
explicitly predicting a merging tree. Additionally, based on
the intuition that the content of the instructions should be
identical regardless of the representation modality, we add atree re-prediction module to facilitate the generated procedural texts to be more coherent by preserving the merging tree
structure.
We tested the proposed method on the task of generatingcooking recipes using a variety of traditional procedural text
generation models. The experimental results show that the
proposed method can lead the base models to generate more
coherent recipes in a versatile manner. Human evaluation andqualitative analysis support the impact of an increase in themetrics from the perspective of the human senses.
II. RELATED WORK

We discuss the proposed vSIMMR dataset by comparing
with other datasets in II-A and II-B. We also show the noveltyof the proposed method, introducing the other procedural text
generation models in II-C.
A. HOW-TO DATASETS FOR VISION AND LANGUAGE

How-to datasets consisting of procedural texts with visualobservations are rising in popularity for learning processes ofcomplex tasks [12]–[14]. Among them, the cooking domain
has been targeted by several researchers, and our proposed
dataset, vSIMMR is also constructed in this domain. In Table
1, we compare the vSIMMR dataset with other cross-modal
recipe datasets from multiple perspectives. To analyze theprocesses of food preparation, there are roughly two possible
visual information resources: videos and image sequences.
Cooking videos are a natural input that involves observation of food states. Zhou et al. [8] developed the YouCook2
dataset, which consists of 2,000 YouTube videos for 89
recipes. The EPIC-Kitchens dataset [7] and Breakfast dataset[6] are large-scale cooking video datasets, but recipe texts
are not attached with videos. Although cooking videos havean advantage in its observation of a continuous food-statetransition, the number of cooking observations is limited.
Hence, it suffers from a shortage of recipes covered in the
training data.
Image sequences are an alternative way to observe foodstates, and the number of covered recipes tends to be muchlarger than that of video datasets. Recipe QA [9] is a recent dataset for cross-modal QA tasks. In [1], data fromthe same web site are used for a recipe generation task.
Among this kind of dataset, the Cookpad Image Dataset [10]is the world’s largest dataset, which contains 1.7M recipesand (optional) image sequences. In addition, it is the only
cross-modal dataset with an ingredient list, and our vSIMMR
dataset is based on the Cookpad Image Dataset.
1 We are going to release our dataset for research purpose publicly.

2

B. STRUCTURE ESTIMATION FOR CONTEXT
DEPENDENCY

In the 1980’s, Momouchi [15] proposed PT (ProceduralText)-chart, which represents an entire workflow of actions
and materials as the Backus-Naur form. This work is pioneer
research to convert procedural texts into graphical representations. Recently, recipes have also been targeted in this field,
and Kiddon et al. [4] proposed an unsupervised method toestimate a graph structure, called an action graph from arecipe text. An action graph is also used in the challenging
visual reference resolution task described in [16], [17]. Moriet al. [5] provided a recipe flow graph dataset, which is a finestructured machine-readable recipe representation. Jermsurawong and Nizar [3] have proposed the SIMMR dataset,
which provides a merging tree, a tree structure tracking theingredient merging operations.
Although the above work has focused on parsing text-only
recipes, little work has not addressed cross-modal analysis
due to limited available datasets. Pan et al. [11] recently
created a novel cross-modal dataset, MM-ReS dataset. Thisdataset consists of recipes, image sequences, and annotatedtree structures, allowing us to analyze the cause-and-effectrelations between step texts and images in the recipe andimage sequence.
Our proposed dataset vSIMMR can be seen as a simple
extension of the MM-ReS dataset by connecting images not
only with step texts but also with ingredients, while theirwork connects images only with a recipe. This difference
allows us to further investigate cause-and-effect relations
between the ingredients and images. Therefore, our datasetvSIMMR can be said as the only cross-modal dataset withrecipes, ingredient lists, tree structures, and image sequences
as shown in Table 1.
C. PROCEDURAL TEXT GENERATION FROM VARIOUS
INPUTSIn the field of natural language generation, a procedural text
is a popular target because it requires a model to generate
coherent sentences by understanding its long context. Amongthe many types of procedural text, recipes with rich descriptions of a large variety of actions and involved ingredients arefeatured. Kiddon et al. [18] and Bosselut et al. [19] proposeda model to generate a recipe from its title and ingredients.
To deal with the long context of a recipe, the former uses
an attention mechanism, and the latter uses reinforcement
learning.
Recently, many vision and language datasets have appeared in the cooking domain, and a challenging task, recipe
generation from an image, has also been proposed. Salvador
et al. [20] firstly tackled this problem and proposed a model
based on Transformer [21] to generate a recipe from a single
image of a prepared dish. Later, Wang et al. [22] proposeda structure-aware model to generate a recipe from an image
by imitating a structure of the ground truth recipe in an
unsupervised manner.
VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 1. Comparative overview of relevant cooking datasets.

Datasets
Breakfast [6]EPIC-Kitchens [7]
YouCook2 [8]RecipeQA [9]Story boarding [1]Cookpad Image Dataset [10]Recipe Flow Graph [5]Action Graph [4]SIMMR [3]MM-ReS [11]vSIMMR (ours)

Materials

Merging tree

Recipes?

Ingredients?

X
X
X
X
X
X
X
X
X

X
X
X
X
X

Structure?

X
X
X
X
X

Visual data
Video
Video
Video
Image sequence
Image sequence
Image sequence
N/A
N/A
N/A
Image sequence
Image sequence#Recipes
N/A
N/A
89
19,77916,405
1,715,595
266
133
260
9,850
2,103

Procedural text

Tomatoes

Step 1(a) SIMMR

Pumpkin(b) vSIMMR (ours)

Cut the tomatoes.

Ketchup
Step 2

Mayonnaise

Image sequence
Image 1

1

Step 1Stir-fry the pumpkin.

Step 2

2

Step 3

3

Step 3

Put them on the tomatoes.

Image 2
Image 3
Image 4

Step 4

Step 4

4

Cover them withketchup and mayonnaise.

FIGURE 1. An overview of the vSIMMR dataset. Images in the image
sequence are aligned with the intermediate nodes in the merging tree and thestep texts in the procedural text.

There have also been two recent attempts to generate arecipe from an image sequence depicting the intermediate
food states [1], [2]. To represent the global context in animage sequence, Chandu et al. [1] incorporated a finite statemachine (FSM), and Nishimura et al. [2] incorporated a joint
embedding model into a recipe generator. In line with theabove studies, this paper proposes a method for explicitly
modeling the context dependency as a structure in a networkarchitecture that is compatible with previous methods.
III. PROPOSED DATASET: VSIMMR

We explain the proposed cross-modal dataset, vSIMMR,
which consists of 2,106 procedural texts, image sequences,material lists, and annotated tree structures. In the following
subsections, we firstly provide an overview of the SIMMR
dataset [3] (III-A) and then describe important extension toit (III-B) and annotation process of the vSIMMR dataset(III-C).
A. MERGING TREE: CONTEXT DEPENDENCYDESCRIPTION IN SIMMR

SIMMR was originally proposed to represent the context
dependency of a procedural text as a tree structure “merging
tree,” which describes the merging path of materials to asingle final product. Figure 1(a) exemplifies the merging
tree of the procedural text. The leaves (terminal nodes) andintermediate nodes correspond to the materials and steps,
respectively. The root node is the final step and indicates the
final product of the procedural text.

VOLUME 4, 2016

Image sequence

Merging tree

Procedural text

FIGURE 2. Our browser-based annotation platform. A user can annotate amerging tree by clicking unused ingredients and intermediate nodes
representing a mixture of ingredients at each step.

B. EXTENSION TO SIMMR: LARGER DATASET WITH
VISUAL REFERENCE

As shown in Figure 1(b), we extend SIMMR to vSIMMR
by annotating merging trees with image sequences. Thisextension allows us to represent the common tree structure
between the visual and textual world because images in
the image sequence are aligned with the step texts in theprocedural text. Moreover, in terms of the number of labeleddata items, the vSIMMR dataset is about eight times larger
than the SIMMR dataset as shown in Table 1, and this pointis also an important extension to SIMMR.
C. ANNOTATION PROCESS OF THE TREE STRUCTURE

To build the vSIMMR dataset, we annotated merging trees
to a subset of the Cookpad Image Dataset, which consistsof procedural texts (= recipes), material lists (= ingredient lists), and image sequences. The annotation process isroughly divided into two parts: (i) material name normalization and (ii) merging tree annotation.
(i) Material name normalization. Firstly, to annotate merging trees correctly, we manually merged any notational variations of material names. This normalization is necessary
because all procedural texts and material lists in the Cookpad
Image Dataset are user-generated and thus material names
contain inconsistency [23]; for example, some write “2tbsp
3

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

of oil” or “2tablespoon of oil” and others simply write “oil”.
We manually normalized them to the simplest form (in theabove example, we converted “2tbsp of oil” and “2tablespoonof oil” into “oil”).
(ii) Merging tree annotation. Then, we annotated a merging
tree referring to an image sequence and procedural text.
Figure 2 shows our annotation platform, presenting the image
sequence on the left pane and the corresponding procedural
text on the right pane. We asked one annotator to construct
the merging tree by connecting unused ingredients and intermediate nodes representing a mixture of ingredients at eachstep. In this process, if steps are irrelevant to the cooking
process, we instructed the annotator to remove such step texts
and images from an image sequence and procedural text;
for example, in Figure 2, text and image of step 1 “Prepare
ingredients. Left is the yogurt and right is the avocado” wasremoved. When the annotator submits the annotated structure
to the server, the system validates whether all ingredients areused and whether the created structure is a tree structure.

IV. PROPOSED METHOD

After presenting the task definition and an overview, twomain modules of the proposed method are described in IV-A
and IV-B, then IV-C presents an implementation idea forsemi-supervised learning.
Task definition. Let Dtr be a training dataset, where{x, y, g} ∈ Dtr is an input x, procedural text y, and its
structure (e.g., merging tree) g. x = {xv , xmat } has twocomponents; an image sequence xv = {xnv |n = 1, . . . , N }and a material list xmat = {xmmat |m = 1, . . . , M }. N and
M are the number of images and materials, respectively.
y = {y n |n = 1, . . . , N } consists of N steps of textual
instructions y n s, where {xnv , y n } has a one-to-one correspondence. Note that g is unavailable for most of the data points
in Dtr (semi-supervised setting). Hence, the procedural text
generation task described in this paper is to train a model that
estimates a procedural text ŷ from an input x.
Overview of the proposed method. Figure 3 shows an
overview of the proposed method. To incorporate context
dependency into the decoder for gaining global coherency,
our model tries to estimate the structure ĝ explicitly and use it
in the decoder. This is roughly described by the combinationof an encoder E : x → z, ĝ and a decoder D : z, ĝ → ŷ,
where z is the latent representation of input x. Note thatboth E and D are trained in an end-to-end manner insteadof training E and D separately.
Additionally, based on our intuition that a merging tree is
identical regardless of the representation modality, we add
ˆ to re-predict the merging tree ĝˆa module Ey : ŷ → ĝfrom ŷ. Here, this tree re-prediction module encourages thegenerated procedural text to be more coherent by preserving
the merging tree.

A. STRUCTURE-AWARE PROCEDURAL TEXTGENERATION MODEL

Our model generates a procedural text coherently through thefollowing three processes: First, (i) encodes x into a latent
representation z. Then, (ii) determines ĝ through a Gumbel
softmax resampling [24], which is employed for end-to-end
training. Finally, (iii) decodes ŷ from ĝ with a structured
decoder of a Tree-LSTM.
(i) Link probability matrix calculation. A graph structure,such as a merging tree, can be predicted from a probability
matrix whose ij-element represents the probability that alink exists from the i-th node to the j-th node. Because
a processed material cannot be merged by returning to aprevious step, a step-to-step link has an order constraint,
whereas a material-to-step does not. Hence, we prepare two
sub-modules separately to calculate the two link probability
matrices (LPMs): step-to-step LPM Pv→v ∈ RN ×N andmaterial-to-step LPM Pmat→v ∈ RM ×N .
(i-1) Step-to-step LPM calculation. Each image xnv is fed
to a pre-trained image encoder Fv . This projects xnv onto zvn ,where we define z v = {zvn |n = 1, . . . , N }. To calculate
a link probability between two steps, z v is further converted
into row and column feature vectors z 1v and z 2v using different
1
2
biLSTMs Ev→v
and Ev→v, respectively. Then, each elementij
pv→v in the step-to-step LPM Pv→v is calculated as follows:

pij
v→v =


ε

(i ≤ j)1
2
exp (zv,i·zv,j) P exp (z1 ·z 2 )kv,i z,k(otherwise)

(1)

1
2
where z 1v = Ev→v
(Fv (xv )), z 2v = Ev→v
(Fv (xv )).
2
1
is the j-th
In Eq (1), zv,iis the i-th element of z 1v and zv,j2
ij
−7element of z v . pv→v is set to ε (= 1.0 × 10 ) for numericalstability when i ≤ j to preserve the order constraint.
(i-2) Material-to-step LPM calculation. Next, we calculate
the material-to-step LPM. For the step side, we calculate
from z 1v and z 2v with an element-wise max-poolingz maxvlayer. The material-side feature is calculated in two stages.
First, each xm
mat→v is converted into a distributed representation by Fmat 2 . Second, we further transform them intoz mat , which is aware of ordering information of material list
xmat . This encoding is necessary because we assume thata user-generated material list implies some order, althougha material list is an unordered set generally. Specifically,
as described in [26], we utilize another biLSTM Emat→v ,which further encodes each distributed representation intoma material feature zmat→v∈ z mat . Finally, each elementij
pmat→v in the material-to-step LPM Pmat→v is calculated
as follows:

pij
mat→v

=

where z mat

=

i
max
exp (zmat→v· zv,j
)
P
,
i
max )
exp
(z
·
z
mat→v
k
v,k
Emat→v (Fmat (xmat )).

(2)
(3)

2 For Fmat , we use word2vec [25] pre-trained on all recipes in Dtr .

4

VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Tree Re-prediction(Section Ⅳ.B)

Structure-Aware Procedural Text Generation Model (Section Ⅳ.A)
Material-to-step
LPM

Materials
Pumpkin
KetchupMayonnaise
Image sequence

Merging tree

Pmat→vTomatoes

Process(i)
Link
Probability
MatrixCalculation

Process(ii)

Step-to-step
LPM

e
es
p
kin
ais
ato ump tchu
nn
P
yo
Ke
a
M

Process(iii)
Structured
Decoder

Encoder
1
2
Ev→v
Ev→v

Emat→v

Decoder

Step 1

Cut the tomatoes.

Step 2

Step 2Stir-fry the pumpkin.

Pv→v
Order constraint

̂
Pmat→v

Step 1m
To

GumbelSoftmax
Resampling

Re-predicted
Material-to-step LPM

Procedural Text

Step 3

Put them on
the tomatoes.

Step 3

Merging
Tree
Reprediction

Step 4Re-predicted
Step-to-step LPM

̂Pv→v

Order constraint

Encoder

Step 2

Step 4Dg

Dy

Cover them withketchup andmayonnaise.

Step 1

Ey

Step 3
Step 4

FIGURE 3. An overview of the proposed method. The structure-aware procedural text generation model (Section IV-A) has three processes: (i) Link Probability
Matrix Calculation, (ii) Gumbel Softmax Resampling, and (iii) Structured Decoder. Additionally, the model re-predict a merging tree from a generated procedural text
(Section IV-B).

When the sample has an annotation of the structure g, we
calculate a tree prediction loss Ltp (g, Pv→v , Pmat→v ) as acolumn-wise cross-entropy loss.
(ii) Gumbel softmax resampling. To make the model fully
derivative, we use a Gumbel softmax resampling. To determine the tree structure ĝ directly from the LPMs, we apply
the straight-through (ST) version of Gumbel softmax resampling. The temperature parameter τ is set to 0.5 according to
the discussion in [27], [28]3 .
(iii) Structured decoder. To incorporate the context dependency based on ĝ into a text generator, we use a variantof Tree-LSTM, called Child-sum Tree LSTM [29]. Childsum Tree-LSTM Dg : z max, z mat , ĝ → h, where h =v{hn |n = 1, . . . , N }, converts intermediate nodes of ĝ into
hidden states h. In forwarding phase, the Child-sum TreeLSTM updates the hidden state hs and memory cell cs frommax
based on observation. See appendix A
each child and zv,nfor the calculation details.
The output of Tree-LSTM h is further decoded using aprocedural text generator Dy : zvn , hn → ŷ n . As instances of
Dy , we used five state-of-the-art procedural text generators
in our experiments. Finally, the model generates a procedural
text, and the cross-entropy loss Ltext is calculated.
B. MERGING TREE RE-PREDICTION

Because the tree structure is identical regardless of the visual and linguistic representation, it should be consistently
re-predictable from the textural information in nature. Explicitly preserving such a structure facilitates the generated
procedural texts to be more coherent. Hence, we prepared
Ey : ŷ → z ŷ for predicting P̂v→v and P̂mat→v . The repredicted matrices are then compared to g, where g is theground truth (for samples with an annotation) or a result
3 This setting is also supported by an experience appearing described inappendix C
VOLUME 4, 2016

determined by column-wise argmax on Pv→v and Pmat→v(for samples without an annotation). In addition, P̂mat→v
in Eq (3) into z ŷ .
is simply obtained by substituting z maxvHere, P̂v→v is obtained by substituting z 1v and z 2v in Eq (1)into z max
and z ŷ , respectively. Finally, a tree consistencyv
loss Ltc (g, P̂v→v , P̂mat→v ) is calculated as the column-wisecross-entropy loss on P̂v→v and P̂mat→v with supervision g.
C. VAE-BASED RECONSTRUCTION MODULES FOR
SEMI-SUPERVISED LEARNINGAs discussed in [30] for semi-supervised learning, we use
two variational autoencoder (VAE) architectures [31]: atree2image VAE and a step2image VAE. For the VAE models, we employ βVAE [32], which is widely used owing to its
ability to disentangle the latent representations. We calculate
the reconstruction losses Lt2i (h, z v ) and Ls2i (z ŷ , z v ) intree2image VAE and step2image VAE as follow:
Lt2i (h, z v )= E [log p(z v |h)] − βDKL (qφ1 (h|z v )||p(h)) (4)1qφ

Ls2i (z ŷ , z v )= E [log p(z v |z ŷ )] − βDKL (qφ2 (z ŷ |z v )||p(z ŷ )),
2
qφ

(5)
where qφ1 represents concatenated networks of the encoders
E and Tree-LSTM Dg and qφ2 represents the networks appended the re-prediction module Ey to qφ1 . Here, β is set to
be 4 according to the discussion in [32].
Finally, the model is trained with the loss function L,
which is defined as follows:
L = Ltext + Ltp + λtc Ltc + λt2i Lt2i + λs2i Ls2i ,(6)where the λ∗ s are hyper-parameters tune by grid search witha validation set (see appendix B).
5

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 2. Statistics of our Cookpad Image Dataset split (data used invSIMMER are excluded).

Cookpad Image Dataset#recipes#steps#words#ingredients

train
163,525
6.24
148.527.85

val
18,051
6.15
147.02
7.79

test
20,1936.26
148.507.86

TABLE 3. Statistics of vSIMMR.

vSIMMR#recipes#steps#words#ingredients

train
1,603
6.78
118.23
6.58

val
250
6.74
113.91
6.37

test
250
6.85
114.686.64

V. EXPERIMENTS

This section presents the evaluation of the generated procedural texts in the three criteria: automatic evaluation (V-C),
human evaluation (V-D), and qualitative analysis (V-E), witha thorough ablation. We also discuss the merging tree accuracy (V-D) and the satisfactory number of labeled data items
for procedural text generation (V-G).
A. DATASETS

We used the Cookpad Image Dataset, which has 1.7M recipes(= procedural texts) and ingredient lists (= material lists)
with (optional) image sequences. For our experiments, we
discarded recipes having steps without images. Each recipe
has a complete image sequence xv , a recipe y, and an
ingredient list xmat . We further discarded some recipes withless than two steps or two ingredients because such a recipe
has little ambiguity in its structure g. As a result, the size ofthe dataset is 200k.
Preprocessing. Each image was resized so that the longer
edge had 256 pixels while keeping the aspect ratio. A central
region with 224×224 pixels was then cropped. All recipes
in the Cookpad Image Dataset are written in Japanese.
Thus, to split the sentences into words, we used a Japanesemorphological parser, KyTea [34]. All words appearing lessthan three times were then replaced with an unknown word
symbol. Table 2 and 3 show the dataset’s statistics.
B. IMPLEMENTATION DETAILSAs the encoder for visual inputs Fv , we used the one proposed in [2]. This model, based on ResNet50 [35], waspre-trained by an image-step retrieval task on the Cookpad
Image Dataset. We preliminary confirmed that this metriclearning provides a better performance in recipe generation
than pre-training on ImageNet [36], owing to the domain
fitting. We set the output dimension of the image encoder Fv
to 2, 048. The parameters of the pre-trained model Fv were
fixed during training for recipe generation, as described inprevious studies [1], [2], [37], [38]. The output dimension of
word2vec Fmat for the ingredient encoder was set to 300,
and the dimension of the LSTM hidden layer of encoders,
1
2
Ev→v, Ev→vand Ey , was set to 512. The batch size was set
6

to 16, and Adam [39] was used as the optimizer.
Models. To prove the versatile impact of the proposed
method, we tested it on various state-of-the-art methods forimage-sequence-to-recipe generation, as described below.
• Images2seq [37] was originally proposed for the task ofvisual storytelling (ViST). This model uses biLSTM toconsider the global context in an image sequence.
• GLAC Net [38] achieved a state-of-the-art performanceon ViST. This model takes the global context by fusing
global and local image feature vectors encoded by biLSTM and Fv .
• SSiD [1] is a method proposed for procedural text
generation. It vectorizes steps using sent2vec [40], andthen they are clustered by k-means, where the obtained
clusters are treated as the states of an FSM. Based on thetransition probabilities of the FSM, the model explores
the change of the context to represent the global context
richly.
• SSiL [1] is a variant of SSiD, which has an additionalloss of Kullback-Leibler divergence to better imitate thestep transition of the ground truth.
• RetAttn [2] achieves a comparative method to SSiL,
but by a different approach; it pre-trains the model withimage-text retrieval task, then in the text generation task,
the retrieved text features are involved with an attention
mechanism to generate a better quality text.
Ablations. For all five methods above, we conducted ablation
studies on the following variations to reveal the impact of amerging tree estimation.
• Baseline (original) does not use any modules of the
proposed method4 .
• Half model computes the tree prediction loss Ltp for
samples with the ground truth with the text generation
loss Ltext . In addition, Lt2i is also calculated for samples with no annotation.
• Full model calculates Ltc in addition to the half model.
In the full model, two losses Lt2i and Ls2i 5 are additionally calculated for samples with no annotation.
Note that the baseline models originally have no ingredient lists in their inputs. Thus, for a fair comparison, we
calculate the ingredient vectors in the same manner as withthe proposed method and concatenate the average vectorof the ingredient vectors with each image feature vector.
When training and evaluating the baseline models, for afair comparison, we used recipes in both the vSIMMR and
Cookpad Image Dataset.
Enhanced baselines. The number of trainable parameters inthe baseline model is about 1.5 times smaller than that in
the half and full models. This difference occurs because thefull model has additional modules for encoding the structure
although the number of parameters in the text generator is
4 The parameter setting of each model is followed to the original paper.
5 In the half model, L

s2i cannot be calculated because the step vectors z ŷ
2 , which is the full model
appear only in the re-prediction module Ey in qφ(see Eq (5)).
VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 4. Word-overlap metrics for the five base models with half and full models. The scores in bold are the best for each base model. B=BLEU, RL=ROUGE-L,
D=Distinct, I=Ingredient, and Ac=Action. * indicates statistically significant difference (p < 0.001) from the base models (original) through bootstrap sampling [33].

Images2seq (original)
Images2seq (wide)
Images2seq (deep)+ Half+ FullGLAC Net (original)
GLAC Net (wide)
GLAC Net (deep)+ Half+ FullSSiD (original)
SSiD (wide)
SSiD (deep)+ Half+ FullSSiL (original)
SSiL (wide)
SSiL (deep)+ Half+ FullRetAttn (original)
RetAttn (wide)
RetAttn (deep)+ Half+ Full

B1
27.5
25.2
22.2
27.8∗
29.6∗
28.5
26.3
27.7
28.5
28.9
28.7
28.6
27.6
30.3∗
31.1∗
31.0
26.2
27.8
28.1
30.4
32.2
28.3
29.8
32.2
33.2

B4
5.1
4.6
3.4
5.8∗
6.3∗
5.9
5.1
5.3
5.9
6.1
6.0
5.7
5.2
6.2∗
6.4∗
6.3
5.5
5.6
5.4
6.4
6.5
5.9
6.3
6.5
7.1

RL
18.4
17.5
17.3
20.6∗
21.7∗
21.4
19.3
19.8
21.8∗
21.3
20.9
19.6
19.7
20.8
21.6∗
21.4
20.6
20.3
21.4
21.9∗
21.6
21.1
21.5
21.8
22.1

D1
38.3
38.1
38.3
51.1∗
47.6∗
46.6
40.6
41.5
46.7
47.2∗
45.5
41.0
41.0
43.9
48.3∗
45.5
46.3
42.9
46.7∗
47.3∗
40.2
44.5
44.1
52.4∗
52.7∗

D2
54.7
56.8
54.6
75.0∗
71.0∗
69.0
58.9
60.4
68.8
69.9∗
66.6
60.0
59.1
65.1
71.0∗
66.8
67.4
62.8
68.2∗
70.9∗
60.3
65.1
65.4
77.8∗
78.6∗

I-B1
7.0
4.0
4.2
7.9∗
8.8∗
9.5
7.7
8.2
10.5∗
11.8∗
8.6
7.9
6.9
11.4∗
12.9∗
8.3
9.2
8.3
11.3∗
12.4∗
11.2
12.0
12.1
11.9∗
12.1∗

I-B2
1.8
0.8
0.8
2.2∗
2.7∗
2.9
2.1
2.3
3.4∗
3.8∗
2.7
2.1
1.9
3.7∗
4.1∗
2.4
2.7
2.4
3.7∗
4.2∗
3.3
3.1
3.7
3.7∗
3.8∗

I-B3
0.5
0.1
0.1
0.7∗
0.9∗
0.9
0.6
0.7
1.1∗
1.2∗
0.8
0.6
0.5
1.3∗
1.2∗
0.7
0.8
0.7
1.2∗
1.4∗
1.0
1.2
1.2
1.2
1.2

I-B4
0.1
0.0
0.0
0.2
0.3∗
0.3
0.2
0.2
0.4
0.4
0.2
0.1
0.2
0.4∗
0.4∗
0.2
0.2
0.2
0.4∗
0.4∗
0.3
0.3
0.3
0.3
0.3

I-RL
9.7
0.8
6.6
12.7∗
13.8∗
13.2
10.8
11.2
15.6∗
16.3∗
13.1
11.3
10.6
16.4∗
16.7∗
12.5
13.0
12.1
16.9∗
17.0∗
14.5
14.1
14.7
14.8
14.8

Ac-B1
18.2
13.0
11.1
18.8∗
21.4∗
21.2
18.1
20.5
21.0
21.0
20.1
17.9
20.1
19.8
21.5∗
19.8
21.1
19.9
18.7
21.4∗
22.1
22.0
22.1
21.8
22.3

Ac-B2
8.1
5.3
5.0
8.4∗
9.4∗
9.3
8.0
9.2
9.4
9.1
8.7
7.6
9.0
8.7
9.5∗
8.7
9.4
8.6
8.1
9.5∗
9.0
8.9
9.1
9.4∗
9.5∗

Ac-B3
3.8
2.6
2.6
3.9
4.3∗
4.3
3.9
4.3
4.4
4.4
4.0
3.5
4.3
4.4∗
4.5∗
4.0
4.5
4.0
3.8
4.5∗
3.2
4.0
4.2
4.2∗
4.2∗

Ac-B4
2.0
1.4
1.5
2.0
2.1
2.1
2.1
2.4
2.3
2.3
2.1
1.7
2.2
2.2
2.2
2.0
2.3
2.1
2.0
2.3∗
1.5
1.9
1.9
2.0∗
2.0∗

Ac-RL
18.4
16.6
16.2
21.2∗
22.9∗
22.8
20.4
21.4
23.1
22.5
21.6
19.4
21.8
22.1∗
23.5∗
22.1
22.3
22.3
22.2
23.0∗
21.2
22.0
22.5
22.9∗
23.1∗

TABLE 5. Human evaluation results. RetAttn was used for the “Baseline,” “Half,” and “Full” models. Among the three types of baselines, we employed the original
baseline in our experiments. “F,” “IU,” and “IF” indicate the fluency, ingredient use, and image fitting, respectively. (a) Evaluation with recipes randomly selected fromthe test set. (b) Evaluation with the longest 25% recipes in (a).

F
IU
IF

Baseline
26.7
30.8
31.7

Full
62.5
66.7
45.0

(a) with all 120 recipes.
Tie
Half
10.8
F
35.8
2.5
IU
24.2
23.3
IF
21.7

Full
55.8
70.8
59.2

Tie
8.3
5.0
19.2

exactly the same. Thus, for a fair comparison, we added twoenhanced baselines by changing the number of parameters inthe biLSTM encoder: Baseline (wide) and Baseline (deep).
• Baseline (wide) widely changes the hidden size H by
fixing the number of layers L = 1 in the encoder
biLSTM.
• Baseline (deep) deeply increases the number of layers
L = 4 and changes the hidden size H in the encoder
biLSTM.
Based on these strategies, we changed the total number ofparameters near to that in the full models. Detailed settings
are described in Appendix B.
C. IMPACT ON PROCEDURAL TEXT GENERATION

Scores. We computed the generated recipes with commonlyused metrics: word-overlap metrics (BLEU1/4 [41] andROUGE-L [42]) and diversity metrics (Distinct1/2 [43]:percentage of distinct unigrams/bigrams). However, these
metrics do not evaluate the coherency of generated recipes,
the global ordering of instructions grounded in the realworld. Therefore, as with [19], we also report word-overlap
metrics on the extracted sequence of actions and ingredients
described in the recipe. Each word sequence extracted from arecipe depict a simulated world where actions are performed
and ingredients are used. Thus, the orderings of actions and
VOLUME 4, 2016

F
IU
IF

(b) with the longest 25%(=30/120) recipes.
Baseline
Full
Tie
Half
Full
16.7
70.0
13.3
F
23.3 66.7
26.7
73.3
0.0
IU
16.7 83.3
36.7
50.0
13.3
IF
10.0 86.7

Tie
10.0
0
3.3

ingredients in the generated recipes should agree with thosein the ground truth.
To extract the action and ingredient sequences fromrecipes, we trained the named entity recognizer, Flair6 [44]with the recipe flow graph corpus [5], in which the action andingredient words are defined7 . We applied the trained model
to both the generated recipes and ground truth, and calculated
action- and ingredient-level BLEU n (n = 1, 2, 3, 4) and
ROUGE-L, respectively.
Results. Table 4 shows that our full model achieves steady
improvements from the original and enhanced baselines inboth of word-overlap metrics and diversity metrics. From thisresult, we can say that the supervision of the merging tree
has contributed to increasing both accuracy and diversity.
Moreover, in the word-overlap metrics on the action andingredient sequences, the proposed models, particularly thefull models, substantially outperform (p < 0.001) boththe original and enhanced baselines in most metrics. This
indicates that incorporating a merging tree gains the global
coherency in a versatile manner, and re-predicting a merging
6 https://github.com/flairNLP/flair
7 We trained the recipe NER with 2,072, 230, and 256 sentences fortraining, validation, and test, respectively. The obtained model scored F1
values of 91.8 and 94.7 for the action and ingredient word recognition,
respectively.

7

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients

Burdock, Carrot, ● Sugar, ● Sake, ● Soy sauce, ● Wasabi, White sesame, Sesame oil

ImagesStep 1ModelsStep 2

Step 3

Step 4

Step 5

Step 6Chop the burdock
into thin strips,
Slice the carrot.
and put them inwater.
Chop the burdock
Cut the carrot
into thin strips,
into fine strips.
and put them inwater.

Pour the sesame
After the carrot
Add the carrots becomes wilted,
oil into the pan,and stir-fry theand stir-fry them. add the hijiki andcarrot.
stir-fry it.
After preheating
the pan, stir-fry
the carrot.

After the sprout,
Season with salt
add the carrots
and pepper.
and stir-fry them.

RetAttn(baseline)

Mix allingredients.

RetAttn(half model)

Prepareingredients.

RetAttn(full model)

Prepareingredients, andmix seasonings
marked ● in theingredient list.

Chop the burdock
into this strips,
Cut the carrot
and put them in
into fine strips.
water.

Pour the sesame
oil into the pan,and stir-fry the
carrot.

After the carrot
becomes wilted,
add the burdockand stir-fry it.

Add seasonings
and mix them.
Serve on a plate.

Mix seasonings
Ground Truth marked ● in theingredient list.

Peel the burdock,chop it into 5-cm Cut the carrot
strips, and put
into fine strips.
them in water.

Pour the sesame oil
into the pan, andstir-fry the burdock.
Add carrots and stirfry them.

Add step 1
seasonings into
the pan, and stirfry them.

Turn off the heat,
and add whitesesame.
Serve on a plate.

Merging tree generated by half model

Merging tree generated by full model

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6● Sugar● SakeBurdock● Sugar● Sake● Wasabi

Carrot● Soy sauce● Soy sauce

Burdock● Wasabi

CarrotWhite sesame

Sesame oil

Sesame oil

White sesame

FIGURE 4. Example of generated recipes and merging trees. Here, the baseline (original), half, and full models are compared with the ground truth. This sample
has no ground truth of the merging tree because it belongs to the Cookpad Image Dataset, not to vSIMMR. Note that the recipes are originally in Japanese andhave been translated into English (Japanese version is shown in appendix D).

TABLE 6. Merging tree accuracy.

Baseline
Merging tree only
Half models
Images2seq
GLAC Net
SSiD
SSiL
RetAttn
Full models
Images2seq
GLAC Net
SSiD
SSiL
RetAttn

D. HUMAN EVALUATION

Material-to-step

Step-to-step

Total

70.1

90.0

80.3

73.2
72.1
73.3
73.1
73.1

90.7
91.4
91.0
90.6
90.4

81.5
81.2
81.6
81.3
81.3

72.9
73.7
72.8
73.8
73.0

90.0
90.5
90.7
90.5
91.0

81.0
81.7
81.3
81.7
81.5

tree further facilitates generated recipes to be more coherent.

8

To confirm the impact of the increase in the word-overlap
metrics for humans, we conducted a human evaluation on 120
recipes randomly selected from the test set. We evaluated ourmodel on the three aspects of recipe quality as examined in[1], [19]: fluency (F), ingredient use (IU), and image fitting(IF).
Settings. For each example, an annotator was asked tocompare a pair of recipes, each generated from an identicalimage sequence but by a different model. The annotator
was then asked to select the better recipe according to thethree criteria above. For the ingredient use, the annotator
selected the recipe that used the ingredients more correctly.
For image fitting, we asked the annotators to select the recipe
that was more suitable for the image sequence. For theimage-sequence-to-recipe model, we used RetAttn, whosebase model performed best in Table 4.
Results. Table 5 lists the results, showing the clear advantageof the proposed method over both the baseline and half
VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSImages2seq
I-B1

I-B2

12.5

SSiD
I-B3

3.0

7.5

0 8 4
1

1

1
2

1

1.0

1

1

Ac-B1

1
2

1

Ac-B2

22.5

8.0

17.5

15.0

0 8 4
1

1

1
2

1

0 8 4
1

1

1

1
2

6.0

0 8 4
1

1

1
2

1

0.0

10.0

0 8 4
1

2.0

3.0

1.5

0 8 4
1

1

1
2

1

1
2

1

7.5

0 8 4
1

Ac-B4

4.0

2.5

15.0

12.5

Ac-B3

3.5

7.0

I-RL

0.1

4.5

9.0

20.0

I-B4

0.2

0.5

0 8 4

RetAttn

0.3

1.0

2.0

SSiL
0.4

4.0

10.0
5.0

GLAC Net

1

1.0

1

1
2

1

Ac-RL
22.5
20.0

0 8 4
1

1

1
2

1

17.5

0 8 4
1

1

1
2

1

FIGURE 5. (Best viewed in color). Change in action- and ingredient-level word-overlap metrics with a controlled ratio of labeled data. Dotted horizontal lines showthat the performance of the baseline (original) shown in Table 4. Note that the entire labeled data accounted for approximately 1% of the training set.

model in all three aspects. To demonstrate the effect of longcontext recipes, we further evaluate the proposed methods
on the longest 25% (=30/120) of the recipes8 . As a result,the superiority of the proposed method is highlighted morestrongly. This reveals that re-predicting merging trees helped
maintain the global coherency of long recipes.
E. QUALITATIVE ANALYSIS

Figure 4 shows an example of the generated recipes. Other
examples are discussed in appendix D.
Insights. Maintaining global coherency is essential particularly for understanding the procedural text. The baselinemodel loses its maintenance because the model tends to generate unsuitable duplicate actions (e.g., “stir-fry the carrot” insteps 4 and 5). These repetitive phrases are suppressed in therecipe generated by the proposed model. In addition, owing
to a merging tree estimation, it is remarkable that the fullmodel generates a referred expression (e.g., the “•” in therecipe generated by the full model in step 1 in the figure),
which refers to specific ingredients marked in the ingredient
list.
Limitations. Although the estimated merging tree is completely correct, we still found some slight differences with theground truth. For example, in step 4, the model only adds carrots, but burdock and carrots are added by the ground truth. In
step 5, the model does not talk about adding seasonings, butthey are added in step 6. This is likely occurred because of thenature of user-generated recipes; the image-step alignment is
not regularized and fluctuations are likely. The image in step
5 is not typical in the sense that the image is not a resultof the instruction step (the seasoning is added but not stir8 The length of a recipe is based on the two criteria: the number of steps
as the primal criterion and the number of words in a recipe following it.
VOLUME 4, 2016

fried as the instruction). Although it is beyond the scope of
this paper, a more strict image-step alignment in the dataset
would improve the quality of the recipe generation.
F. MERGING TREE EVALUATION

Table 6 shows an evaluation of the tree prediction accuracy.
Here, the baseline only estimated the tree and not the recipe,
and the model was trained with the pairs of image sequences
and merging trees in the training set of vSIMMR. Fromthese results, we can see that the procedural text generation
contributes to predicting the merging tree. When comparing
the half model with the full model, we can also see that thebest score in every metrics is achieved by the full models.
This indicates that the tree re-prediction module will guidethe model to estimate the merging tree correctly.
G. SATISFACTORY NUMBER OF LABELED DATA ITEMSTo estimate the satisfactory number of labeled data items,
we controlled the ratio of labeled data to the values of 0,
1 1 1
8 , 4 , 2 , and 1. As evaluation metrics, we use the actionand ingredient-level BLEU n (n = 1, 2, 3, 4) and ROUGE-L
to indicate the tendency of controlling training labeled data
items. Note that, when the ratio is 0, the model is trainedwith the proposed architecture in an unsupervised manner.
Figure 5 shows the results. In general, most models achievea higher performance as the ratio of labeled data increases.
This shows that training the labeled data in a semi-supervisedmanner leads to an increase in the quality of the recipe
generation. By contrast, interestingly, RetAttn achieves ahigh performance without any labeled data, indicating thatthe proposed architecture itself enables RetAttn to generatea coherent recipe. Although the GLAC Net, SSiD, and SSiL
require using from 25% to 50% of the labeled data to perform
9

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

better than the baselines, Images2seq must use 100% of thelabeled data. This result indicates that models with highcomplexity (GLAC Net, SSiD, SSiL) can learn from lesslabeled data than the simple model (Images2seq).
VI. CONCLUSION

In this paper, we investigated the impact of explicitly introducing the context dependency on the task of procedural text
generation from an image sequence. To achieve this goal, we
proposed the vSIMMR dataset by annotating merging trees toa subset of the Cookpad Image Dataset. Using the vSIMMR
dataset, we then proposed the structure-aware procedural
text generation model, which gains the global coherency by
explicitly predicting a merging tree. Additionally, we added
a tree re-prediction module to facilitate the generated procedural texts to be more coherent, motivated by the assumptionthat the structure should be common regardless of the visualand linguistic representations.
We tested the proposed method in the cooking domain,
and the results showed that the proposed method can boost
the performance of the traditional models in a versatile way
and for various metrics. Moreover, a qualitative analysis
demonstrated that the model generates a coherent recipe byunderstanding the context dependency efficiently.
.
APPENDIX A CHILD-SUM TREE-LSTM CALCULATIONm
When a child is the m-th material zmat
, Child-sum Treemm
LSTM calculates c and h using trainable linear functions
m
m
m
→ cm
fc : zmat
mat and fh : zmat → hmat , respectively.
On the other hand, when a child is n-th step node, it always
outputs cn and hn .
Let C(n) denote the set of children of node n. Let W (∗)
and U (∗) be the weighted matrices, and b(∗) be the bias. The
notations σ(·) and tanh(·) mean the sigmoid function and thetangent hyperbolic function, respectively. Then, the Childsum Tree-LSTM is described as follows:

h̃n

X

=

hnj

(7)

j∈C(n)inmax= σ(W (i) zv,n
+ U (i) h̃n + b(i) )

fjn
n

=

n

=

o

u

n

c

hn

=

max
σ(W (f ) zv,n
+ U (f ) h̃n + b(f ) )max
σ(W (o) zv,n
+ U (o) h̃n + b(o) )maxtanh(W (u) zv,n
+ U (u) h̃n + b(u) )

= i

n

= on

n

u +

X
j∈C(n)ntanh(c ).

fjn

cj

(8)
(9)
(10)
(11)
(12)
(13)

Note that the superscript n of the non-leaf nodes is the sameas that of the images xv , and thus, the indices are in a oneto-one correspondence.

10

TABLE 7. Hyper-parameters we used in our experiments. They are
determined by a grid search with the validation set.

Images2seq
GLAC Net
SSiD
SSiL
RetAttn

λtc
0.1
0.01
0.001
0.1
0.001

λt2i
0.01
0.01
0.1
0.1
0.001

λs2i
0.1
0.1
0.001
0.001
0.01

APPENDIX B RESULTS OF HYPER PARAMETERSTUNING

Hyper-parameter settings. We tuned the hyper-parameters
λtc , λt2i , and λs2i for each of the five base models by agrid search with a validation set. Three different parameters,λ∗ ∈ {0.001, 0.01, 0.1}, were used in the experiment. Hence,
we searched 27 (= 3 × 3 × 3) points for each method. Table
7 lists the results of the hyper-parameter tuning.
Trainable parameter settings. As described in Section V-B,
the total number of parameters θb in the baselines is about
1.5 times smaller than that θf in the full models. For a faircomparison, we added the two enhanced baselines, whichhave θb close to θf by changing the hidden size H andthe number of layers L in the following process. First, we
calculated θf in the full models. Next, we simply fixed L to
1 (wide) and 4 (deep), and then incremented H by 50 untilθb is over θf . Table 8 shows the hidden size H, the numberof layers L, and the total number of trainable parameters θtuned in our experiments, respectively.
APPENDIX C TEMPERATURE PARAMETER OFGUMBEL SOFTMAXOne of the extra hyper-parameters, the Gumbel softmax
temperature τ , is known to significantly affect the gradient computation [24], [27], [28]. Here, we explored how
the temperature rate affects the performance. Four different
temperature rates τ ∈ {0.005, 0.05, 0.5, 5} were used inthis experiment. Figure 6 shows the results for the overlap
metrics, indicating that τ = 0.5 tended to achieve the better
performance among the candidates. When τ was too low,
the model suffered from a gradient vanishing problem, which
prevented it from learning to generate a procedural text witha sampled merging tree. By contrast, when τ was too high,it was difficult to stabilize the model. As described in [24],to tackle this problem, we also conducted an experiment onannealing the temperature from high to low at each iteration.
This annealing technique did not work in our experiment,
however, so we set τ = 0.5 during the training.
APPENDIX D GENERATED RECIPE EXAMPLES

In addition to the example described in the main paper, weshow other examples of generated recipes in Figures 7-11,
in which different numbers of steps are sampled. Note thatthe first example (Figure 7) is the Japanese version of theexample (Figure 4) described in the main paper. All recipes
are originally in Japanese and have been translated into
English. As the sequence becomes longer, the baseline model
VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 8. The model’s parameter settings used in the experiments. H, L, and θ represent the hidden size, the number of layers, and the total number ofparameters, respectively.

Images2seq
GLAC Net
SSiD
SSiL
RetAttn

Baseline (original)
H
L
θ
512
1
46,460,593
1,024
2
110,312,113
1,024
2
110,710,293
1,024
2
110,710,293
512
1
109,738,161

Images2seq
12.5

I-B1

H
1,600
2,250
2,400
2,400
1,300

Baseline (wide)Lθ
1
83,800,753
1
162,487,741
1
162,644,629
1
162,644,629
1
147,801,713

GLAC Net

I-B2
4.0

10.0

3.0

7.5

2.0

5.0

0.005 0.05

0.5

1.0
5.0
0.005 0.05

Ac-B1

22.5
20.0
17.5
15.0
12.5
0.005 0.05

10.0

0.5

5.0

0.005

0.05

0.5

Baseline (deep)Lθ
486,872,753
4
161,438,241
4
165,994,429
4
165,994,429
4146,137,713

SSiD

SSiL

5.0

I-RL

0.5
0.05

0.5

5.0 0.005 0.05

3.0

1.5
0.5

0.5

5.0

17.5
15.0
12.5
10.0
7.5

0.005

Ac-B4
2.0

0.05

Full modelLθ
182,385,073
2
161,284,017
2
161,460,145
2
161,460,145
1
145,662,641

RetAttn

1.0

4.0
2.0
0.005

H
512
1,024
1,024
1,024
512

I-B4

I-B3

Ac-B3

6.0
5.0

0.005

Ac-B2

8.0
0.5

17.5
15.0
12.5
10.0
7.5

H
700
1,000
1,050
1,050
650

1.0
5.0 0.005

0.05

0.5

5.0

Ac-RL
22.0
20.0

0.05

0.5

5.0

18.0

0.005

0.05

0.5

5.0

FIGURE 6. Change in the overlap metrics under controlled temperature rates τ .

tends to generate actions in the improper order. For example,
in steps 6 and 7 in Figure 11, the baseline model generates“add,” “mix,” and “serve,” whereas the full model generates“crush,” and “add,” which is the same order as in the groundtruth. This confirms that the proposed method can consider
a long context dependency by incorporating a merging tree
estimate into the recipe generator.
Although the proposed method facilitates the generated
recipes to be coherent, the quality of the generated recipes
can be still improved. For example, in step 2 in Figure 8,
the full model says “Add butter to the bowl and heat themwith the microwave.” but the butter is not included in theingredient list. This is because our method does not haveany mechanism to explicitly access the vocabulary used in
the ingredient list on its decoding process. To address thisproblem, incorporating a copying mechanism [45] into thedecoder may be one of the scenarios of generating words
in the ingredient list. However, as stated in Section III-C,
recipes and ingredient lists are inconsistent due to the natureof user-generated contents. Thus, it is not clear whether suchan additional mechanism works well for solving the problem
directly.

VOLUME 4, 201611

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients

ごぼう, にんじん, ● 砂糖, ● 酒, ● しょう油, ● わさび, 白ごま, ごま油

Step 1Models
RetAttn(baseline)

材料を全て
混ぜます。

RetAttn(half model)

材料を用意します。

Step 2
ごぼうはささがきにし, 水に浸します。

Step 3

Step 4

Step 5

フライパンにごま
人参は千切りにし人参を入れ炒め
油を熱し、人参を
ます。
ます。
炒めます。

フライパンが温ま もやしを入れた
人参は千切りにしにし, 水に浸しまったら、人参を入 後、人参を入れます。
れ炒めます。
す。
て炒めます。
ごぼうはささがき

Step 6
人参を炒め終わったらひじきを入れ炒めます。

塩と胡椒で味をつけます。

人参がしんなりした 全体に火が通った

RetAttn(full model)

ごぼうはささがき
フライパンにごま
材料を用意しまにんじんは千切り
ら、ごぼうを入れす。●の調味料を合 にし、水にさらし油を熱し、人参をわせておきます。

ておきます。

にします。

炒めます。

ごぼうは皮を剥いて５Ground Truth●の調味料を混ぜ㎝ぐらいの長さで細切ておきます。

りにし、水にさらしておきます。

熱したフライパンにごま油を にんじんに火がとおっ

火を止めて、白ごまをにんじんを細切り ひいてゴボウを炒めます。 たら①の調味料を入れふりかけて混ぜればでごぼうに火がとおったらにん て汁気がなくなるまでにします。
きあがりです。
じんを入れて炒めます。

Merging tree generated by half model

炒めます。

Merging tree generated by full model

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

ごぼう

● 砂糖
●酒
● わさび

にんじん

● しょう油● しょう油

ごぼう

● わさび

にんじん

白ごま

ごま油

ごま油

白ごま

● 砂糖
●酒

ら、調味料を入れて、全体がしんなり て、よく混ぜまするまで炒めます。 す。できあがり！

FIGURE 7. (Best viewed in color). This is the Japanese version of the example (Figure 4) in the main paper.

12

VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients

Spinach, White sesame, Soy sauce, Sugar

Step 1ModelsStep 2

Step 3

RetAttn(baseline)

Add sugar, sugar, sugar, andBoil the spinach with hot water
salt to the bowl.
Add step 1’s seasonings to stepand drain it.
Mix it.
1’s bowl.
(ほうれん草は塩茹でにし、水気(ボウルに砂糖、砂糖、砂糖を入 (1の調味料を1に入れる。)
を切っておく。)
れてよく混ぜる。)

RetAttn(half model)

Add flour, baking powder, sugar,
Boil the spinach with hot salted and cocoa into the bowl. Mix it.
water and Cut it into a bite size. (ボウルに薄力粉とベーキングパ Add step 1's result to step 2's
result, and mix them.
(ほうれん草は、塩茹でして食べ(1を2に入れて混ぜる。)
ウダーとココアを入れて混ぜやすい大きさに切る。)
る。)

Add butter to the bowl and heat
Boil the spinach with hot salted them with the microwave.
water and cut it into a bite size. Add sugar and mix them well.
(ほうれん草は、塩茹でして食べ (ボウルにバターを入れ, レンジで加熱し, 砂糖を入れよく混ぜやすい長さに切っておく。)

RetAttn(full model)

Add step 1's result to step 2's
result, and mix them well.
(1を2に入れ、よく混ぜる。)

る。)

Ground Truth

Wash the spinach and boil it with
hot salted water to skim thefoam.
Add white sesame, soy sauce,
Add the spinach (step 1), andDrain and cut it into 3-cm
and sugar to the bowl. Mix themmix them while loosening them.
pieces.
well.
(ほうれん草は水洗いし塩少々を (ボウルにごま・しょうゆ・砂糖 (水気を切って1のほうれん草を加え、ほぐしながら炒める。)
加えたお湯で茹でる。水にさら を入れてよく混ぜる。)
してアクを抜き, 軽く絞って3cmくらいの大きさに切っておく。)

Merging tree generated by half model
Step 1

Step 2

Merging tree generated by full model

Step 3

Step 1

SpinachSpinachWhite sesame

White sesame

Soy sauce

Soy sauce

Sugar

Sugar

Step 2

Step 3

FIGURE 8. (Best viewed in color). Example of generated recipes and merging trees. The baseline (original), half, and full models are compared with the groundtruth. The half model predicts the merging tree and uses it as the structure of the Child-sum Tree-LSTM. The full model re-predicts the merging tree from thegenerated recipe in addition to the modules of the half model. Compared with the baseline model, both proposed models (half and full) generate coherent recipes
by using the referred expressions (e.g., in step 3, they output “add step 1’s result to step 2’s result”).

VOLUME 4, 201613

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients

Pork roast steak, Salt and pepper, Oil, ★Miso, ★Sake, ★Sweet sake, ★Soy sauce,★Garlic paste, ★Black pepper, ★Fried sesame

Step 1

Step 2Mix all ingredients.
(材料を全て混ぜま

Cut the pork into
bite-size pieces.
(豚バラ肉を食べやModels

RetAttn(baseline)

す。)

RetAttn(half model)

RetAttn(full model)

Pour the oil and grill
both sides.
Serve on a plate.
(フライパンに油を (お皿に盛ってできすい大きさに切りま ひき、両面を焼きま あがりです。)
す。)
す。)
Pour the oil into the
Prepare ingredients. Cut the pork into
Put lettuce on a plate.
pan and grill theMix ★ ingredients. bite-size pieces.
Pour the sauce onto it.
pork.
It’s ready to eat.
(豚肉は食べやすい(材料を用意しま(フライパンに油を (レタスを皿に盛り付け、上からタレをかけて完成
す。★の材料を合わ 大きさに切りま
熱し豚肉を焼きまです。)
す。)
せておきます。)
す。)
す。)(★は合わせておき

Cut the muscle of the Pour the oil, grill thepork with the top of the pork (step 2), and add
Serve on a plateknife. Add the salt and step 1's result to it.
with your favorite
pepper.
Mix them well.
(豚肉は筋切りをし包丁 (フライパンに油を熱し vegetables.

ます。)の先で全体を刺しま

Step 2②を焼き肉に火が通っ

Step 3

Merging tree generated by full model
Step 4

Step 1

★Miso
★Sake

★Miso
★Sake

★Sweet sake

★Sweet sake

★Soy sauce★Soy sauce★Garlic paste
Salt and pepper★Garlic paste

Oil
Pork roast steak★Black pepper★Fried sesame★Fried sesame(好みの野菜と共にす。塩コショウを軽く たら①を加え肉に味を 器に盛ります。)
振ります。)
絡めます。)

Merging tree generated by half model
Step 1

Step 4

Pour the oil and grill
both sides.
Serve on a plate.
(フライパンに油を (お皿に盛ってできすい大きさに切りま ひき両面焼きま
あがりです。)
す。)
す。)

Cut the pork into
Prepare ingredients. bite-size pieces.
(豚バラ肉を食べや(材料を用意しま

Mix ★ ingredients.
Ground Truth

Step 3

Step 2

Step 3

Step 4★Black pepper
Salt and pepper
Pork roast steak
Oil

FIGURE 9. (Best viewed in color). Example of generated recipes and merging trees. The baseline (original), half, and full models are compared with the groundtruth. In step 1, the full model correctly generates a referred expression, whereas the other models do not. All models miss the proper timing for adding theseasonings in step 3. We assume that the models would predict the merging tree incorrectly. Although they predict that the seasonings from step 1 would connect
with step 4, the seasonings are actually used in step 3.

14

VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients Egg, Sugar, Chocolate, Milk, Flour, Cocoa, Baking powder, Soy pulp, Drained cherry

Step 1Models

RetAttn(baseline)

Step 2

Step 3Crack the egg into
Add chocolate
the bowl. Add
and mix it.
sugar and mix
well.
(チョコレート(ボウルに卵を割り

Add milk andmix it well.
(牛乳を加えて入れ、砂糖を加え混ぜる。)てよく混ぜる。)

を加えて混ぜる。)て混ぜる。)

GroundTruthMelt theAdd milk to the
chocolate withstep 2’s result
hot water.
Add milk into the and mix it withthe rubberbowl.
(チョコレートを spatula.
割りほぐし、砂 湯煎で溶かし、 (2に牛乳を加糖を加えてよく 牛乳を入れて溶 え、ゴムベラで
混ぜる。)
混ぜる。)
かす。)

Crack the egg
into the bowl.
Add sugar andmix well.
(ボウルに卵をStep 7

Add flour andmix it well.
(薄力粉を加えて混ぜる。)

180℃のオーブ型から取り出し、180度で25分焼Add flour and
Add flour and mix it with the Put dough into amold.
Finish baking.
mix it with the rubber spatula.
Bake it at 180℃
rubber spatula. (ふるっておい for 25 minutes. Garnish thecocoa.
(薄力粉をふるた薄力粉を加 (型に生地を入 (焼き上がり。
い入れ、ゴムベ
え、ゴムベラで れ、180℃のオ ココアを添え
ラでしっかり混
さっくり混ぜ ーブンで25分焼 て、完成。)
ぜる。)
く。)
る。)
Put dough into a
muffin cup.
Add baking
After baking,on a drained
Add flour andpowder and mix Put
cool it in thecherry.
mix it with theit well.
Bake them at 180℃ fridge. Now it’srubber spatula. (ベーキングパ for 25 minutes.
ready for eat.
(薄力粉を加えゴ(マフィン型に生地(焼きあがった
ウダーをくわえをいれ、ドランチェ
ムベラでよく混
ら、冷蔵庫で冷入れよく混ぜ リーを乗せる。

ぜる。)

る。)

180℃のオーブンでやして完成。)

25分焼く。)

Add flour and Put dough into a
muffin cup. Put on
baking powder a drained cherry. If it is firm after
Add step 3's
to
the
step
4’s
baking, serve
Bake them atresult to step
result and mix it 180℃ for 25
on a plate.
2'sresult.
Mix it well.
minutes.
well.
(焼き終わった(1におからを加 Mix them well. (粉類は合わせ (生地はカップに(3に2を少しずら、触ってみてえ、泡立て器で流し、ドレンチェ
つ加え、よく混 てふるってお弾力があればでリーを飾り、180
良く混ぜ合わせぜ合わせる。) き、4に加えよ
度のオーブンで約 きあがり。)
る。)
く混ぜる。)
25分焼く。)て沸騰直前まで
温め、刻んだチールに入れて溶かす。)

Merging tree generated by half model
1
2
5
6
3
4
Step
Egg
Sugar
Chocolate
Milk
Flour
Cocoa
Baking powder
Soy pulp
Drained cherry

Step 6

Put dough into a After baking, take itout. Bake it at
mold.
for 25Bake it at 180℃ 180℃
minuites.
for 25 minutes. (焼きあがったら、(型を流し入れ、

Heat milk in the
Add the soy
pot. Melt the
chocolate with it. pulp into the(鍋に牛乳を入れ step 1’s result.

割りほぐし、砂ョコレートのボ糖を加えて良く混ぜ合わせる)

粉を加えて混ぜStep 5ンで25分焼く。) く。)

Add theAdd milk to
chocolate tostep 2’s result.
step 2's result.
(2に牛乳を加(2にチョコレー
え、よく混ぜりほぐし、砂糖、 トを入れて混ぜる。)
牛乳、砂糖を入れ る。)

Crack the egg
into the bowl.
Add sugar andmix well.
(ボールに卵わ

Add flour andmix it well.
(ふるった薄力る。)

Crack the egg
into the bowl.
Add sugar, milk
and sugar. Mix
RetAttn well.
(half model) (ボウルに卵を割RetAttn(full model)

Step 4

Merging tree generated by full model
7

Step

1

2

3

4

5

6

7

Egg
Sugar
Chocolate
Milk
Flour
Cocoa
Baking powder
Soy pulp
Drained cherry

FIGURE 10. (Best viewed in color). Example of generated recipes and merging trees. The baseline (original), half, and full models are compared with the groundtruth. The baseline improperly generates duplicate actions (e.g., “Add flour” in steps 4 and 5, and “Bake” in steps 6 and 7). This is suppressed by the proposed
methods, although they each output duplicate actions once: The half model outputs “Add flour” in steps 4 and 5, and the full model outputs “Add milk” in steps 2 and
3). We also emphasize that the full model can verbalize the correct instruction, “Put on a drained cherry,” by using a merging tree estimate, whereas the baselineand half models do not generate this instruction.

VOLUME 4, 201615

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Ingredients Potatoes, Mayonnaise, Salt, Pepper

Step 1ModelsStep 2

Step 3

Peel theAdd hot
potatoes andcut them into water to the
1-cm pieces. pot and boil
(じゃがいもは it.

When boiled,boiled, boiled,simmer it for 10
minutes.
(沸騰したら、

RetAttn(baseline) 皮をむいて、 (鍋に水を入
れ、沸騰さ
せる。)

1cm幅に切る。)

Step 4

Step 5
When theRap them
potatoes are
and heattender, simmer
them with the it for 10
microwave. minutes.
沸騰したら、沸 (ラップをし (じゃがいもがStep 6

Step 7Serve on a

When theplate. Add
potatoes aretender, add milk mayonnaise
and mix it.
if you like.
(じゃがいもが (お皿に盛っStep 9Add thebutter and Serve on amix it.
plate.
(バターを加 (お皿に盛っ柔らかくなった て、マヨネー
えて、混ぜ騰したら、10分 てレンジで加 柔らかくなった
ら、牛乳を加えら、10分煮ズをかけて る。)
ほど煮る
熱。)
混ぜる。)
る。)
。)
完成。)

て完成。)

When theAfter mixing
Crush theServe on aServe on a
Simmer it
potatoes arepotatoes and plate and add well, add
tender,
crush
it
Peel
the
plate.
for 3
drain them. mayonnaise. the butter.
potatoes with and add salt.
(お皿に盛りminutes.
(じゃがいもが (じゃがいも (お皿に盛り付 (よく混ぜて(蓋をして弱 the knife.
柔らかくなった
包丁で皮をむ沸騰したら、弱(包丁で皮をを潰して、水 け、マヨネー から、バタ 付けてできら、潰し、塩をき、一口大に切 火にして蓋をし 火で3分ほどあがりで気を絞りま ズを入れまむきます。)ーを入れま
ふっておきま
ります。)
ます。)
煮ます。)
す。)
す。)
す。)
す。)
す。)
Peel thepotatoes with aknife.
(じゃがいもは

RetAttn(halfmodel)

Add water into
the pot and put
a lid on it.
(鍋に水を入れCrush theAfter mixing Serve on a
Add water
plate. Add
potatoes
Skim theAdd theAddwell, add
into the pot,
mayonnaise if
withapotatoes
and
foam
from
potatoestomayonnaise
and
put
the
PreparePeel the hot
you like.
masher.
mix
them.
potatoestoit.
the surface. potatoes.
a bowl.
and mix it.
ingredients.
(お皿に盛っ(鍋に水を入れ(混ざった(沸騰したら (皮をむきま (じゃがいも (柔らかくな (マヨネーズ(材料を用意て、お好みでて沸騰したら アクを取りをボウルに ったら、マ を入れて、 ら、じゃが マヨネーズを
します。)
す。)
じゃがいもをます。)
入れます。) ッシャーで 混ぜます。) いもをいれて かけていただ
入れます。)
混ぜます。) きます。)
つぶす。)

RetAttn(fullmodel)

My friendgave me theAdd water
potatoes.
Simmer it
Thank you. into the pot for 40-50(友人からい and heat it minutes.
ただいたじ at low heat. (40~50分く(水からゆっ
ゃがいもで
らいかけて
くり弱火で
す。ありが
茹でます。)
茹でます。)たいことでGroundTruthPeel thepotatoes.
They are hot,thus Keeppeeling away.
(皮をむきます。熱い！けRemove theeyes of thepotatoes.
(芽もとりまど、ここがふん す。)
ばりどころでMix it well andit’s ready to eat.
Add water or
milk if you like.
(まぜまぜして Serve on a

Add
Crush them tomayonnaise,a certain
salt, andplate.
出来上がりでextent.
pepper to it.
す。少し水か牛 (盛り付けま(ある程度潰し(マヨネーズ、乳かを入れてし す。)て)塩、胡椒。) っとりさせても
よいでしょ

す。)

う。)

す。)

Merging tree generated by half model

Step

Step 8

1

2

3

4

5

6

7

8

Merging tree generated by full model
9

Step

Potatoes

PotatoesMayonnaise

Mayonnaise

Salt

Salt

PepperPepper

1

2

3

4

5

6

7

8

9

FIGURE 11. (Best viewed in color). Example of generated recipes and merging trees. The baseline (original), half, and full models are compared with the groundtruth. This example has a large number of steps. Therefore, generating the recipe correctly requires the models to understand a long context dependency. The
baseline model tends to generate improper actions, because it can not understand the long workflow of the recipe (e.g. “rap,” “heat,” “simmer,” “add,” “mix,” and“serve” from step 4 to step 6). By contrast, the half and full models generate the recipe with the actions in the correct order, to a certain extent. Comparing thesemodels with each other, the full model generates actions suitable for the images, whereas the half model generates duplicate actions (e.g., “crush” in steps 5 and 6).
Moreover, the full model generates the instruction, “Add mayonnaise if you like,” which is suited to the image but does not appear in the ground truth. This isachieved because the model fully uses the image feature.
16

VOLUME 4, 2016

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSREFERENCES[1] K. Chandu, E. Nyberg, and A. W. Black, “Storyboarding of recipes:
Grounded contextual generation,” in Proc. ACL, 2019, pp. 6040–6046.
[2] T. Nishimura, A. Hashimoto, and S. Mori, “Procedural text generationfrom a photo sequence,” in Proc. INLG, 2019, pp. 409–414.
[3] J. Jermsurawong and N. Habash, “Predicting the structure of cooking
recipes,” in Proc. EMNLP, 2015, pp. 781–786.
[4] C. Kiddon, G. T. Ponnuraj, L. Zettlemoyer, and Y. Choi, “Mise en Place:
Unsupervised interpretation of instructional recipes,” in Proc. EMNLP,
2015, pp. 982–992.
[5] S. Mori, H. Maeta, Y. Yamakata, and T. Sasada, “Flow graph corpus fromrecipe texts,” in Proc. LREC, 2014, pp. 2370–2377.
[6] H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Recovering
the syntax and semantics of goal-directed human activities,” in Proc.
CVPR, 2014, pp. 780–787.
[7] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos,
D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling
egocentric vision: The EPIC-KITCHENS dataset,” in Proc. ECCV, 2018,
pp. 720–736.
[8] L. Zhou, C. Xu, and J. J. Corso, “Towards automatic learning of proceduresfrom web instructional videos,” in Proc. AAAI, 2018, pp. 7590–7598.
[9] S. Yagcioglu, A. Erdem, E. Erdem, and N. Ikizler-Cinbis, “RecipeQA: A
challenge dataset for multimodal comprehension of cooking recipes,” inProc. EMNLP, 2018, pp. 1358–1368.
[10] J. Harashima, Y. Someya, and Y. Kikuta, “Cookpad image dataset: An
image collection as infrastructure for food research,” in Proc. ACM SIGIR,
2017, pp. 1229–1232.
[11] L. Pan, J. Chen, J. Wu, S. Liu, C.-W. Ngo, M.-Y. Kan, Y.-G. Jiang, and T.S. Chua, “Multi-modal cooking workflow construction for food recipes,”in Proc. ACMMM, 2020, pp. 1132–1141.
[12] D. Zhukov, J.-B. Alayrac, R. G. Cinbis, D. Fouhey, I. Laptev, and J. Sivic,“Cross-task weakly supervised learning from instructional videos,” inProc. CVPR, 2019, pp. 3537–3545.
[13] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic,“HowTo100M: Learning a text-video embedding by watching hundred
million narrated video clips,” in Proc. ICCV, 2019, pp. 2630–2640.
[14] J.-B. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and
S. Lacoste-Julien, “Unsupervised learning from narrated instruction
videos,” in Proc. CVPR, 2016, pp. 4575–4583.
[15] Y. Momouchi, “Control structures for actions in procedural texts and ptchart,” in Proc. COLING, 1980, pp. 108–114.
[16] D.-A. Huang, J. J. Lim, L. Fei-Fei, and J. C. Niebles, “Unsupervised
visual-linguistic reference resolution in instructional videos,” in Proc.
CVPR, 2017, pp. 2183–2192.
[17] D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, andJ. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from asingle video demonstration,” in Proc. CVPR, 2019, pp. 8565–8574.
[18] C. Kiddon, L. Zettlemoyer, and Y. Choi, “Globally coherent text generation with neural checklist models,” in Proc. EMNLP, 2016, pp. 329–339.
[19] A. Bosselut, A. Celikyilmaz, X. He, J. Gao, P.-S. Huang, and Y. Choi,“Discourse-aware neural rewards for coherent text generation,” in Proc.
NAACL-HLT, 2018, pp. 173–184.
[20] A. Salvador, M. Drozdzal, X. Giro-i-Nieto, and A. Romero, “Inverse
Cooking: recipe generation from food images,” in Proc. CVPR, 2019, pp.
10 453–10 462.
[21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. NeurIPS,
2017, pp. 5998–6008.
[22] H. Wang, G. Lin, S. C. H. Hoi, and C. Miao, “Structure-aware generation
network for recipe generation from images,” in Proc. ECCV, 2020.
[23] J. Harashima and Y. Yamada, “Two-step validation in character-based
ingredient normalization,” in Proc. CEA/MADiMa, 2018, pp. 29–32.
[24] E. Jang, S. Gu, and B. Poole, “Categorical reparametrization with gumblesoftmax,” in Proc. ICLR, 2017.
[25] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,”in NeurIPS, 2013, pp. 3111–3119.
[26] A. Salvador, N. Hynes, Y. Aytar, J. Marin, F. Ofli, I. Weber, and A. Torralba, “Learning cross-modal embeddings for cooking recipes and foodimages,” in Proc. CVPR, 2017, pp. 3020–3028.
[27] C. Baziotis, I. Androutsopoulos, I. Konstas, and A. Potamianos, “SEQ3 :Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression,” in Proc. NAACL-HLT, 2019,
pp. 673–681.
VOLUME 4, 2016[28] J. Gu, D. J. Im, and V. O. Li., “Neural machine translation with gumbelgreedy decoding,” in Proc. AAAI, 2018, pp. 5125–5132.
[29] K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic representations from tree-structured long short-term memory networks,” in Proc.
ACL-IJCNLP, 2015, pp. 1556–1566.
[30] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, “Semisupervised learning with deep generative models,” in NeurIPS, 2014, pp.
3581–3589.
[31] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv
preprint arXiv:1312.6114, 2013.
[32] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick,
S. Mohamed, and A. Lerchner, “β VAE: Learning basic visual cencepts
with a constrained variational framework,” in Proc. ICLR, 2017.
[33] P. Koehn, “Statistical significance tests for machine translation evaluation,”in Proc. EMNLP, 2004, pp. 388–395.
[34] G. Neubig, Y. Nakata, and S. Mori, “Pointwise prediction for robust,
adaptable japanese morphological analysis,” in Proc. ACL, 2011, pp. 529–
533.
[35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. CVPR, 2016, pp. 770–778.
[36] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: Alarge-scale hierarchical image database,” in Proc. CVPR, 2009, pp. 248–
255.
[37] T. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin,
R. Girshick, X. He, P. Kohli, D. Batra, C. L. Zitnick, D. Parikh, L. Vanderwende, M. Galley, and M. Mitchell, “Visual storytelling,” in Proc. NAACLHLT, 2016, pp. 1233–1239.
[38] T. Kim, M. Heo, S. Son, K. Park, and B. Zhang, “GLAC net: Glocal
attention cascading networks for multi-image cued story generation,”
arXiv preprint arXiv:1805.10973, 2018.
[39] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inProc. ICLR, 2015.
[40] P. Matteo, P. Gupta, and M. Jaggi., “Unsupervised learning of sentence
embeddings using compositional n-gram features,” in Proc. NAACL-HLT,
2018, pp. 528–540.
[41] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method forautomatic evaluation of machine translation,” in Proc. ACL, 2002, pp. 311–
318.
[42] C.-Y. Lin and F. J. Och, “Automatic evaluation of machine translation
quality using longest common subsequence and skip-bigram statistics,” inProc. ACL, 2004, pp. 605–612.
[43] J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan, “A diversity-promoting
objective function for neural conversation models,” in Proc. NAACL-HLT,
2016, pp. 110–119.
[44] A. Akbik, D. Blythe, and R. Vollgraf, “Contextual string embeddings forsequence labeling,” in Proc. COLING, 2018, pp. 1638–1649.
[45] J. Gu, Z. Lu, H. Li, and V. O. Li, “Incorporating copying mechanism insequence-to-sequence learning,” in Proc. ACL, 2016, pp. 1631–1640.

TAICHI NISHIMURA received the B.S. degree
in engineering from Kyushu University in 2019,
and the M.S. degree in informatics from Kyoto
University in 2020. His research area is visionand-language, especially text generation systems
from images or videos. He received the Best Paper
Award at the 11th workshop on multimedia forcooking and eating activities in 2019.

17

Nishimura et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

ATSUSHI HASHIMOTO received the B.S. degree in engineering, the M.S. and Ph.D. degrees
in informatics from Graduate School of Informatics, Kyoto University, Japan, in 2005, 2008,
and 2013, respectively. He is currently a seniorresearcher with Omron SINIC X corp. from 2018.
He has been working on human activity analysis
and recognition based on human-object interaction, and interface design and development in thefield of cooking. His research interests include
pattern recognition, computer vision, and human-computer interaction. He
is a member of the IEEE, Institute of Electronics, Information and Communication Engineers of Japan, and Information Processing Society of Japan.

SHINSUKE MORI received B.S., M.S., andPh.D. degrees in electorical engineering from Kyoto University, Kyoto, Japan in 1993, 1995, and1998, respectively. Then he joined Tokyo Research Laboratory of International Business Machines Co. Ltd. (IBM). Since 2007 he was an
associate professor of Academic Center for Computing and Media Studies, Kyoto University. He iscurrently a professor there. His research interests
include computational linguistics and natural language processing. He received IPSJ Yamashita SIG Research Award in 1997,
IPSJ Best Paper Award in 2010 and 2013, and 58th OHM Technology Award
from Promotion Foundation for Electrical Science and Engineering in 2010.
He is a member of Information Processing Society of Japan, the Association
for Natural Language Processing, the Database Society of Japan, and the
Association for Computational Linguistics.

YOSHITAKA USHIKU received his B.E., M.A.,
and Ph.D. degrees from the University of Tokyoin 2009, 2011, and 2014, respectively. In 2014,
he joined NTT CS Labs, Japan, where he wasinvolved in research on image recognition. From2016 to 2018, he was a lecturer with the University of Tokyo, Japan. Currently, he is a Principal
Investigator at OMRON SINIC X and Chief Research Officer at Ridge-i since 2018 and 2019,respectively. His research interests lie in crossmedia understanding through machine learning, mainly for computer visionand natural language processing. He received ACM Mutlimedia Grand
Challenge Special Prize in 2011, ACM Multimedia Open Source Software
Competition Honorable Mention in 2017, and NVIDIA Pioneering ResearchAwards in 2017 and 2018.

HIROTAKA KAMEKO received his B.E., M.E.,
and Ph.D. degrees from the University of Tokyo in
2013, 2015 and 2018, respectively. He is currently
an assistant professor of Academic Center for
Computing and Media Studies, Kyoto University.
His research interests include natural language
processing and game AI. He is a member of IPSJ
and ANLP.

YOKO YAMAKATA received her Ph.D. degree
in Informatics from Kyoto University, Japan in
2006. She was an expert researcher of NICT, Japan
from 2007, a lecturer and an associate professorof Kyoto University from 2010, a JSPS research
fellow at the University of Tokyo from 2015, aninternational visiting researcher of the Universityof Sussex, UK between 2016-2017 and currently
an associate professor of the University of Tokyo,
Japan. Her research interests include multimedia
processing especially regarding cooking and eating activities.

18

VOLUME 4, 2016