JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Recipe Generation from
Unsegmented Cooking Videos

arXiv:2209.10134v1 [cs.MM] 21 Sep 2022

Taichi Nishimura, Atsushi Hashimoto, Member, IEEE, Yoshitaka Ushiku, Member, IEEE,
Hirotaka Kameko, and, Shinsuke Mori

Abstract—This paper tackles recipe generation from unsegmented cooking videos, a task that requires agents to (1) extract
key events in completing the dish and (2) generate sentences forthe extracted events. Our task is similar to dense video captioning(DVC), which aims at detecting events thoroughly and generating
sentences for them. However, unlike DVC, in recipe generation,
recipe story awareness is crucial, and a model should output
an appropriate number of key events in the correct order. We
analyze the output of the DVC model and observe that although(1) several events are adoptable as a recipe story, (2) the generated
sentences for such events are not grounded in the visual content.
Based on this, we hypothesize that we can obtain correct recipes
by selecting oracle events from the output events of the DVC
model and re-generating sentences for them. To achieve this, we
propose a novel transformer-based joint approach of training
event selector and sentence generator for selecting oracle eventsfrom the outputs of the DVC model and generating grounded
sentences for the events, respectively. In addition, we extend
the model by including ingredients to generate more accurate
recipes. The experimental results show that the proposed method
outperforms state-of-the-art DVC models. We also confirm that,by modeling the recipe in a story-aware manner, the proposedmodel output the appropriate number of events in the correct
order.
Index Terms—Cooking Recipe Generation, Video Understanding, Natural Language Generation

I. I NTRODUCTION
With a rapid increase of cooking videos uploaded on theweb, multimedia food computing has become an important
topic [1], [2], [3], [4], [5], [6]. Among various developing
technologies, recipe generation from unsegmented cooking
videos [7] is challenging. It requires artificial agents to (1)extract key events that are essential to dish completion and (2)generate sentences for the extracted events. This task is important for both scene understanding and real-world applications.
In terms of scene understanding, (1) and (2) are recognized as
temporal event localization [8], [9] and video captioning [10],[11], [12], respectively, and learning both simultaneously isan ambitious challenge in computer vision (CV) and natural
language processing (NLP). Meanwhile, from a perspectiveof real-world applications, this technology can support people
learning new skills by providing key events and their explanation sentences as a multimedia summarization of cooking
videos. These academic and industrial motivations impel us totackle recipe generation from unsegmented cooking videos.
Our task is similar to dense video captioning (DVC) [13],[14], [15], which aims at detecting events densely from videos
and generating sentences for them. Although the input/output
pairs of our task (video/(events, sentences)) and that of the…

Existin
DV
Model

…
T

r

C

g

Our Task

Event Candidate(N=10)

Input Video

s

 

s

Ou
Model

1

Add the butter
into a pan

2

Crack the egg
and stir

3

Add the egg mixtur
and cheese and stireFig. 1. Concept overview of our approach. Based on the output events fromexisting DVC models, the proposed method selects events that are suitable asa story of recipes and generates sentences grounded with the extracted events.

DVC are of the same format, our task is different from theperspective of the recipe’s story awareness. DVC allows agentsto exclusively detect false-positive events in its evaluation
metrics; our task, on the other hand, requires them to extract
the accurate number of key events in the correct order. Fujita
et al. [16] reported that DVC models produced more than
200 redundant events per video on average on the ActivityNet
captions dataset [13] while the number of manually-annotated
events is only 3-4. Such redundant events and sentences makeit difficult for users to grasp an overview of the video content.
Although the events predicted by DVC models are redundant, we observed that (1) several events are adoptable as astory of a recipe, but (2) the generated sentences for such
events are not grounded well to the visual contents (i.e.,
ingredients and actions in the sentences are incorrect). We
confirm this by analyzing the outputs of the state-of-the-art
DVC model. We refer to the DVC output events as event
candidates. Through an approach that we refer to as oracle
selection, we select events that have the maximum temporal
IoU (tIoU) to ground-truth events (i.e., manually-annotated
events) and compute the DVC scores [16], [13]. The results
support our observations. Although oracle selection ensuresthat events that are adoptable as the story of a recipe are
selected, the obtained sentences are not grounded in the visualcontent.
Our task requires models to predict the next step for thestory of a recipe by memorizing previously predicted eventsand generated sentences. For example, in Fig. 1, to generate“egg mixture” in step 3, the models should memorize thevisual and textual content of step 2. Existing DVC approaches
do not consider them explicitly; thus they miss or hallucinate
ingredients (e.g., say ingredients that do not appear in thevideo), resulting in performance degradation. Based on this,our hypothesis is that we can obtain correct recipes by select-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

ing oracle events from the event candidates and re-generating
sentences for them (Fig. 1).
To achieve this, we propose a novel transformer[17]-based
joint learning approach of an event selector and sentence generator. The event selector selects oracle events from the event
candidates in the correct order, and the sentence generator
generates a recipe that is grounded with the events. Both ofthe modules have memory representations as with [10], whichenables the model to predict the accurate number of events inthe correct order by remembering previously selected eventsand generated sentences. Moreover, the model is designed to
be trainable in an end-to-end manner because the modulesof the model are connected without breaking a differentiablechain. We refer to this model as the base model.
We also propose an extended model that can generate moreaccurate recipes in the settings where the inputs are videos
with ingredients as in [12], [18]. The task requires agents todescribe detailed manipulations of ingredients from cooking
videos. To achieve this goal, our extended model has two
additional modules: (1) a dot-product visual simulator and (2)a textual attention module. The dot-product visual simulator
is introduced by the insights from our previous work [12],where we discovered that the manipulations change the stateof the ingredients, yielding state-aware visual representations.
In Fig. 1, “eggs” are transformed into “cracked” then “stirred”
form to cook scrambled eggs. To address this, we introduced
the visual simulator, which reasons about the state transition ofingredients, into the model. The motivation of the extension isthe difference in the inputs. Although in our previous work, theinputs are pre-segmented ground-truth events with ingredients,
the inputs in this work are event candidates. The dot-productvisual simulator accepts them and reasons about the statetransition of ingredients. Furthermore, we also introduce a
textual attention module that verbalizes grounded ingredientsand actions in the recipes. These modules are effective forgrounded recipe generation from cooking videos.
In our experiments, we use the YouCook2 [8] dataset,
which contains 2,000 videos with event/sentence annotation
pairs. The proposed method outperforms the state-of-the-art
DVC approaches based on commonly used DVC metrics. In
addition, the extended models boost the model’s performance.
We also show that the proposed models can select the correct
number of events that effectively reflects the ground-truth
events. In addition, our qualitative evaluation reveals that the
proposed approaches can select events in the correct order andgenerate recipes grounded in the video contents. We discuss
the detailed experiment settings for optimal recipe generation.
II. R ELATED W ORK
In this section, we describe the novelty of the proposed
method in line with other works on recipe generation from visual observations (Section II-A) and general video captioning(Section II-B).
A. Recipe generation from visual observations
Recipes are a popular target in the NLP community because
they require artificial agents to generate coherent sentences2[19], [20]. The inputs of these works are the title and ingredients of recipes. Recently, many multimodal cooking datasets
have appeared [1], [21] and recipe generation from images
has also been proposed [2], [5]. Salvador et al. [2] tackled
this problem using a transformer-based model [17] to generate
high-quality recipes from a single image of a completed dish.
Other researchers focus on generating a recipe from an image
sequence depicting the intermediate food states [22], [23],[24].
The essential limitation of generating recipes from images
is the lack of detailed, continuous scene information of humanmanipulations, which images do not contain. Hence, thistask is essentially an ill-posed problem; that is, it depends
on the obtained language model to generate correct actions.
Meanwhile, videos contain this information and have been
attracting the attention of many researchers in recent years[12], [18], [7], [11]. As described in Section I, this task
requires agents to (1) extract key events from videos and(2) generate grounded sentences for events. Most researchers
have focused on (2) generating grounded recipes from presegmented key events [12], [11], [18]; only few researchersattempt to learn both (1) and (2) to generate recipes from
unsegmented videos. Shi et al. [7] are pioneer researchers thathave tackled this problem by utilizing the narration of videos
effectively. Although this approach is effective for narrated
videos, transcription is not always available for all cooking
videos. Speaking during cooking or adding narration to videos
requires significant effort. In addition, narrations may lead themodel to attend to textual features, ignoring the visual featuresof videos. Our task extends their work to generate recipes only
from videos. This enables the model to treat even non-narrated
videos and be more useful than the previous setting.
B. Video captioning
Video captioning is an attractive field for both CV andNLP communities. The task settings of video captioning vary
according to the nature of the input video (e.g., one short event,pre-defined key events, or long unsegmented video). Our aim
is to extract key events and generate sentences simultaneously
from long unsegmented videos. This task is similar to DVC
because the input/output pairs are in the same format as in ourtask. We first describe traditional DVC approaches and thenenumerate the difference between DVC and our task.
DVC aims at densely detecting events from the video andgenerating sentences for them. Recently, transformer-based
approaches are well investigated for this task. Zhou et al. [25]proposed a Masked Transformer, which detects events andgenerates sentences via a differentiable mask in an end-toend manner. Their approach, however, outputs more than 200
redundant events per video on average [16]. To handle thisissue, Wang et al. [14] proposed PDVC, which detects eventsin parallel, re-ranks the top K (K is also the prediction
target), and generates sentences for the re-ranked events. Deng[15] proposed a top-down DVC approach, whereby wholesentences generated for the video are aligned into timestamps,
based on which the sentences are refined.
The key difference between our task and DVC lies in thestory awareness of recipes. Our task requires agents to extract

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

3

TABLE IW ORD - OVERLAP METRICS OF THE ORACLE SELECTION ON THE
YOU C OOK 2 DATASET. N REPRESENTS THE NUMBER OF CANDIDATEEVENTS , A HYPER - PARAMETER OF PDVC. T HE BOLD SCORES ARE THE
BEST AMONG THE COMPARATIVE SETTINGS .

PDVC (reproduced)
Oracle
N=25
N=50
N=100
N=200

BLEU
0.89

dvc eval
METEOR
4.52

CIDEr-D
21.50

METEOR
3.98

SODA
CIDEr-D
25.30

tIoU
27.80

0.58
0.84
0.97
1.10

6.09
6.92
7.68
8.15

27.12
31.63
36.26
38.60

7.62
8.83
9.64
10.43

26.32
29.93
35.08
36.89

56.55
64.58
71.16
76.71

N=25

N=50

N=100

N=200

4000
3000
2000
1000
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.6

0.7

0.8

0.9

(a) Training
1200
900
600
300
0

0

0.1

0.2

0.3

0.4

0.5

(b) Validation

Fig. 2. tIoU distribution of oracle events on the training and validation setsof YouCook2.

an appropriate number of events in the correct order. DVC
allows a model to detect events densely; however, redundant
event/sentence pairs impede readability, and users are unable
to grasp an overview of the video contents. To achieve story
awareness, we propose a transformer-based joint learning
approach of an event selector and sentence generator basedon our observation that oracle events can be selectable fromthe output events of a DVC model.
III. O RACLE - BASED A NALYSIS OF THE E XISTING DVC
M ODELAs Fujita et al. [16] reported, the outputs of DVC models
are redundant. However, we observed that although (1) several
events are adoptable as a story of a recipe, (2) the generated
sentences for such events are not grounded well in the visualcontents. To verify this, we analyze the outputs of the stateof-the-art DVC model. Specifically, we select events with the
maximum tIoU scores to ground-truth events from event candidates and compute the DVC scores, an approach referred to as
oracle selection hereinafter. The analyzed results demonstratethat the oracle selection boosts the performance, especially on
story-oriented DVC metrics.
We use the YouCook2 dataset [8], one of the largest cooking
video-and-language datasets. For the DVC model, we employ
PDVC [14], the state-of-the-art DVC model. It detects N
events densely and then re-ranks the top K of them for its
outputs. Note that N is a hyper-parameter1 and K is theprediction target. We adopt this model because it achieves the
1 N is set to be a sufficiently large number. For YouCook2, the authors of[14] set N to be 100 as a default parameter.

best performance on DVC tasks and it is easy to control N
before training the model. We use the N detected events forthe analysis, not the re-ranked K events.
Evaluation metrics. We use two commonly-used DVC
metrics: dvc eval [13] and SODA [16].
• dvc eval firstly computes tIoU of all the combinations
between the prediction and ground-truth events, andthen computes word-overlap metrics (e.g., METEOR or
CIDEr-D), if the tIoU scores are over the threshold θ. θ
ranges from 0.3 to 0.9 by 0.2. Their average is the output
score of these metrics.
• SODA stands for story-oriented dense video captioning
evaluation, whereby the story awareness of the output
events is evaluated. Specifically, it uses dynamic programming to explore an alignment of events between prediction and ground-truth for obtaining the maximum story
scores. The story scores are computed as a product oftIoU and word overlap metrics. Because SODA evaluates
the predicted events by penalizing redundant outputs, it
is suitable for computing the story awareness of recipes.
SODA evaluates whether the output event/sentence pairs areappropriate as a story; thus, it is rated above dvc eval in thisstudy. As word-overlap metrics, we use BLEU [26], METEOR[27], and CIDEr-D [28], which are commonly-used metrics fortext generation tasks. We also introduce SODA tIoU, which
computes story scores as tIoU scores, rather than a product oftIoU and word overlap metrics. These metrics can evaluate
whether the selected events are appropriate as componentsof the generated recipes. In this analysis, we range N from25, 50, 100, and 200.
A. Quantitative evaluation
Table I shows the results of the oracle selection on dvc evaland SODA metrics, indicating that the oracle selection outperforms the PDVC. Specifically, the SODA scores of the oracle
selection are quite better than that of the PDVC, demonstratingthat the generated recipes are more suitable as a story than theones generated by the PDVC.
Fig. 2 shows the distribution of tIoU scores of the oracle
events on training and validation sets. The number of candidate
events N was directly proportional to the average tIoU. Thisis because the more N was, the more suitable oracle events
appear in the candidate. Both the training and validation setsconfirm this tendency.
B. Qualitative evaluation
Fig. 3 shows a comparison of the recipes generated by
the oracle selection and ground truth. The selected event
timestamps are close to the ground-truth events, indicatingthat the appropriate selection can construct the correct recipes.
However, the sentences differ from the ground truth, theingredients, especially, are different from the ground truth(e.g., “baking soda”, “salt”, and “pepper” are missing in step
(1)).
The main reason for this error is that the model does
not consider previously predicted events and sentences. This

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

…

…

…

…

Ground-Truth
(a)

(b)

4

(c)

Time

(d)

(e)

(d)

(e)

Oracle Selection
(a)

(b)

Ground-Truth(a)Add our eggs baking soda salt andpepper to the bowl and sti(b)Add cold water to the bowl and sti(c)Cover the shrimp in the batter andbreadcrumb(d)Place the shrimp into a pan of hot oi(e)Remove the shrimp from the pan
fl

s

l

r

r

l

(c)

Oracle Selection(a) Add our and mix to a bowl and mi(b) Mix the ingredients in the bow(c) Place the chicken in the batte(d) Fry the chicken in the oi(e) Remove the soup from the oil

r
 x
 l

Fig. 3. Comparison of the recipes generated by the oracle selection andground truth. N = 100, which is a default hyper-parameter of PDVC, is usedin this example.

causes the model to miss or hallucinate ingredients and motivates us to propose a model that re-generates sentences for thepredicted events, rather than using the corresponding generated
sentences without modification.
IV. P ROPOSED METHOD
Based on the oracle-based analysis, we hypothesize that we
can obtain correct recipes by selecting oracle events from theoutput events of the DVC model and re-generating sentences
for them. Our task requires models to predict the next stepfor the story of a recipe by memorizing previously predicted
events and generated sentences. To achieve this, we propose ajoint learning approach of event selection module and sentence
generation module (Fig. 4), which are memory-augmented
recurrent transformers [10]. The event selection module andsentence generation module are referred to as event selector and sentence generator, respectively. The event selector
chooses oracle events from event candidates repeatedly and thesentence generator outputs sentences for the selected events.
The memories are updated to remember the history of theoutputs (selected events/generated sentences) and used in thenext steps.
We formulate this task using notations. Let X =(x1 , x2 , . . . , xn , . . . , xN ) ∈ R2×N be event candidates, where
xn consists of start and end timestamps. Note that X
is sorted on the start time of the events in chronological order. Given X, the model generate pairs (C, Y ) =((c1 , y 1 ), . . . , (ct , y t ), . . . , (cT , y T )), where ct , y t , and T represent an index of the oracle event candidates, corresponding
generated sentences, and the number of the selected events,
respectively. The memories V lt in the event selector and S lt
in the sentence generator are updated at each t step, where lrepresents the layer number of transformers.
A. Event selector
The event selector can be divided into two main components: event encoder and event transformer. The event encoder
converts given candidate events X into event-level representations E = (e1 , e2 , . . . , en , . . . , eT ), based on which theevent transformer outputs a holistic representations of events
H = (h1 , h2 , . . . , hn , . . . , hT ). They are used for computing
event probabilities, which represent the likelihood of oracle
events. We apply Gumbel softmax resampling [29] to select
events and forward them to the sentence generator without
breaking a differentiable chain during the training phase. In
the inference phase, events are deterministically selected byapplying argmax to event probabilities.
Event encoder. The event encoder converts events xn into
representations en . First, we extract event clips from the video,
according to the start and end time of xn , and input them into
pre-trained visual encoders. One of the straightforward ways to
encode events is to average frame-level image representations
extracted by ResNet [30]. However, we find that this approachis unable to capture fine-grained event semantics for overlapped events. For example, assuming that the ground-truth
event is a scene of cutting potatoes, and the event candidates
have two scenes, where one is a scene of cutting potatoes
and tomatoes, and another is of cutting only potatoes. We
expect the model to select the latter scene because the former
contains extra information of cutting tomatoes. Representing
events based on the average of frame-level features cannot
capture the semantic difference of these events effectively.
Thus, it is necessary to employ an event encoder that focuses
on extracting fine-grained event-level semantics.
To achieve this, we focus on multiple instance learningnoise contrastive estimation (MIL-NCE) model [31] pretrained on Howto100M [32]. The model is trained on morethan 100M pairs of an event with narration, and it cancapture event-level semantics of overlapped events. Using
this encoder, we obtain event-level representations as E =(e1 , e2 , . . . , en , . . . , eT ). Then, the positional encoding (PE)[17] and relative encoding of events [33] are added to en .
Event transformer. Given E, the event transformer outputs holistic representations of events
H = (h1 , h2 , . . . , hn , . . . , hT ). This module is basedon the memory-augmented recurrent transformer [10],where each l layer has the memories V lt to remember
the history of the selected events. The output vectors
H = (h1 , h2 , . . . , hn , . . . , hT ) and memories V lt are used tocompute n-th event probability p(ct = n|V lt , X) as follows:Vtp(ct = n|V lt , H)

max(V 1t , . . . , V lt , . . . , V Lt ), (1)t T
exp {(hn ) V t }
,
(2)
= P
t T
i exp {(hi ) V t }=where max(·) represents an element-wise max-pooling ofvectors. During training, the event selector selects events
through a Gumbel softmax resampling [29], which enables us
to train the model in an end-to-end manner without breaking a
differentiable chain. This indicates that the loss computed on
the sentence generator is backpropagated to the event selector.
During inference, the model selects event index based on theargmax of p(ct = n|V lt , H).
B. Sentence generator
Based on the selected event representations, the sentence
generator outputs sentences grounded for them. Let the selected event index be ĉt , and the selected event representation

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20155

Event Selector

Sentence Generator
Softmax

h1

[END]

h2

h4

h3

h5

h6

h7

h8

h9

LinearAdd & Norm

Add & Norm

Feed Forward

Feed Forward
Multi-hea
Attention

Multi-hea
Attention

Q

V

K

l
Vt−1

Vtl

ConcatAdd & Norm
Masked Multi-hea
Attention

e1

e2

Event
End tokene3TESentenc
Transformer

e4

SelecteEvent Vector

Positiona
Encoding

+

e5

e6

e7

e8

e9

Next
step

Feed Forward

Multi-hea
Attention

×N

Stl

ConcatAdd & Norm

Even
Transformer
Timestam
Encoding

V

K

l
St−1

Next
step

Feed Forward

[END]

Q

w1

Event encoder

×N

h5
w2

Positiona
Encoding

+

w3

w4

w5

w6

Word embedding

add cold wate
to the bowl…
…

…

(Shifted right)
t
 
  
 d

e
d

l
p

d
 r

Fig. 4. Joint learning approach of the event selector and sentence generator for recipe generation from unsegmented cooking videos. The event selector
chooses oracle events from event candidates repeatedly and the sentence generator outputs sentences for the selected events. The memories are updated toremember the history of the outputs (selected events/generated sentences) and are used in the subsequent steps.

can be written as hĉt . This vector is added to the word
vectors W = (w1 , w2 , . . . , wk , . . . , wK ) from the wordembedding layer, which is the concatenated neural networks ofthe pre-trained global vectors for word representation (GloVe)[34]2 and one-layer perceptron with ReLU activation. We
also add PE to word vectors W and input them into thesentence transformer, another memory-augmented recurrent
transformer. The model generates words repeatedly by applying softmax and argmax operations to the output vectors of
the sentence transformer.
C. Memory update
The memory modules in the event and sentence transformers
V lt , S lt can be separately updated, according to the memory
update equation given in the original paper [10]. However,intuitively, mixing memories is effective for coherent recipe
generation because the history of the selected events contributes to recipe generation and vice versa. To achieve this,
we update the memory modules by mixing them as follows:V̂ t= f1 (V t )σ(g2 (g1 (S t ))),

(3)

Ŝ t

= g1 (S t )σ(f2 (f1 (V t ))),

(4)

where f∗ (·), g∗ (·) represents a single linear layer and , σrepresents the Hadamard dot product and sigmoid function,
respectively. The obtained V̂ t and Ŝ t are forwarded into thenext t + 1 step.
2 We employ pre-trained 300D word embedding, which can be downloaded
from http://nlp.stanford.edu/data/glove.6B.zip

D. Loss functions
To train the model in an end-to-end manner, we sum up two
types of losses: event selection loss and sentence generation
loss. These losses are formulated as a negative log-likelihoodof the event selection and sentence generation tasks as follows:
LtotalLe
Ls= Le + Ls ,X X
= −log p(ct |X, V l<t ),(X ,C ) t
X X
= −log p(y t |hĉt , S l<t ),(X ,C ,Y ) t

(5)
(6)
(7)

where Ltotal , Le , and Ls represents the total, event selection,and sentence generation losses, respectively.
V. E XTENDED MODEL
In addition to the base model, we also propose an extended
model that can generate more grounded recipes in the settings
where the inputs are videos with ingredients, as in [12], [18].
This task requires agents to describe detailed manipulations ofingredients from cooking videos.
To achieve this goal, our extended model has two additional
modules: (1) dot-product visual simulator and (2) textual
attention module (Fig. 5). The dot-product visual simulator isintroduced based on the insights from our previous work [12].
In our previous work, we discovered that the manipulations
change the state of the ingredients, yielding state-aware visual
representations. For example, “eggs” are transformed into“cracked,” then “stirred” form to cook scrambled eggs. To
address this, we introduced the visual simulator into theJOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

6

Event Selector

Sentence Generator
Softmax

h1

[END]

h4

h3

h2

h5

h6

h7

h8

Weighted actions

a· 1h

h9

a· 2h

Linear…

a· Rh

Dot-produc
Visual simulator

Textual Attention Module
Updated ingredients

g1t
g2t

Add & Norm…

Feed Forward

gMt

Add & Norm

Next
step

Feed Forward
Multi-hea
Attention

Multi-hea
Attention

Q

V

K

l
Vt−1

ConcatAdd & Norm

Multi-hea
Attention

Masked Multi-hea
Attention

Sentenc
Transformer

×N

SelecteEvent Vector

ConcatTimestam
Encoding

e1

Event
End token

e2

e3

TE

e4

e5

e6

e7

e8

Event encoder

Flou
Baking sod
Shrim

Batter

…

r

t
 
p

 
 d

e
d

l
p

a

ConcatPositiona
Encoding

+

w2

w3

w4

w5

w6

Word embedding

add cold wate
to the bowl…

Batter…

Ingredients

Ingredients
…

+

w1

e9
Ingredient encoder

Flou
Baking sod
Shrim

Next
step

×N

h5

Positiona
Encoding+

Stl

Feed Forward

Add & Norm

Even
Transformer

Ingredient encoder

V

K

ConcatNext
step

Feed Forward

[END]

Q

l
St−1

Vtl

(Shifted right)

d
 r
 t

Fig. 5. Overview of the extended model for recipe generation from unsegmented cooking videos. To generate more accurate recipes, it has additional two
modules: (1) dot-product visual simulator and (2) textual attention. The dot-product visual simulator is introduced by the insights based on our previous work
[12] to learn the state transition of ingredients. The textual attention module encourages the sentence generator to verbalize actions and ingredients moreaccurately.

action vectors
add

A

crack
stir

(1) Action selector

·
Ha

a1

a2
a3

Event-actio
Attentioeq. (9)event vectors

h1

oracle
event

h2

H

h3

(3) Updater

Gt

·
Ah

h5

(2) Ingredient selector
vector

g1n−1
g2n−1

G t−1

g3n−1
n

n
n

 
n

t

t

t
y
 

Ingredient-even
Attentioeq. (10)
Event-ingredien
Attentioeq. (11)e1̂

Action-ingredien
Attentioeq. (12)h4(n − 1)-th material

 Action-even
Attentioeq. (8)Computing
event probabiliteq. (2)

Ĝt−1
h

·
Hg

e2̂

e3̂

are concatenated neural networks of pre-trained GloVe [34]word embedding and multi-layer perceptrons (MLPs) with
ReLU activation function and are added to the event selector and sentence generator without sharing their parameters.
Multi-word ingredients (e.g., parmesan cheese) are represented
by the average embedding vector of the words. They areconcatenated with event/word representations and inputted tothe event/sentence transformers as shown in Fig. 5.

Next stepComputing
event probabiliteq. (2)t

Fig. 6. Overview of the extended dot-product visual simulator that reasons
about the state transition of ingredients. It has three components: (1) action
selector, (2) ingredient selector, and (3) updater. The dot-product attention is
employed to treat the event candidates.

model, which reasons about the state transition of ingredients.
Although pre-segmented ground-truth events with ingredientsare regarded as input in our previous work, they are considered
event candidates with ingredients in this study. Thus we extend
it to the dot-product visual simulator described in Section V-A.
In addition, we also introduce a textual attention module thatverbalizes grounded ingredients and actions in the recipes.
These modules are effective for grounded recipe generationfrom cooking videos.
Another extension is to add an ingredient encoder to convert
ingredients into representations G0 . The ingredient encoders

A. Dot-product visual simulator
We first revisit the original visual simulator and then describe how we extend it to the dot-product visual simulator.
Visual simulator revisit. In the original study, the inputs
are the pairs of ingredients and ground-truth events. Let G0 =(g 01 , g 02 , . . . , g 0m , . . . , g 0M ) and Ĥ = (ĥ1 , ĥ2 , . . . , ĥt̂ , . . . , ĥT̂ )be the ingredient and ground-truth event vectors encodedby ingredient encoder (e.g., GloVe) and event encoder (e.g.,
MIL-NCE), respectively. Given (G0 , Ĥ), the visual simulator
reasons about the state transition of the ingredients by updatingthem at each t̂-th step. To this end, it consists of threecomponents: (1) action selector, (2) ingredient selector, and(3) updater. The action selector and ingredient selector predict
executed actions and used ingredients at t̂-th step. Basedon the selected actions/ingredients, the (t̂ − 1)-th ingredientvectors G(t̂−1) are updated into t̂-th new proposal ingredientvectors G(t̂) , which are forwarded into the next step. The
visual simulator recurrently repeats the above process until
processing the end element of the ground-truth events.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

7

Proposed extension. The proposed model is similar to thevisual simulator because it also has the same three components
to update the ingredient vectors. However, instead of theground-truth event vectors Ĥ, the event candidates H are
assumed to be the inputs in this study, that is, the model needs
to predict not only the actions/ingredients but also the events
that constitute the recipe story.
To achieve this, we extend the visual simulator into the
dot-product visual simulator shown in Fig. 6. It computes therelationships between action-to- and ingredient-to-events asattention matrices and outputs four vectors: action-weighted
and ingredient-weighted event vectors, event-weighted action,and ingredient vectors. The former two vectors are used to
calculate the event probabilities and the latter two vectors areforwarded to the textual attention module.
(1) Action selector outputs event-weighted action andaction-weighted event vectors by predicting the executed actions and related events. Let the action vectors be A =(a1 , a2 , . . . , ar , . . . , aR ), that is, the pre-defined action embedding, where ar represents r-th actions and R is the numberof the actions. In Fig. 6, the actions “crack” and “stir” areexecuted at the oracle event h3 ; thus a∗ that corresponding
crack and stir indices should be selected with the relation to
h3 . This computation can be formulated as the dot-productattention as follows:Ȧh = {KT
(W Q
a A)(W h H)√}(W Vh H),
d

(8)

V
K
where W Q
a , W h , W h represents a linear layer and d represents the dimension size of A and H 3 . We also acquire
action-weighted event vectors Ḣ a as follows:Ḣ a = {K(W Qh H)(W a A)√d}(W Va A),

(9)

K
V
where W Q
h ,Wa ,Warepresents a linear layer.
(2) Ingredient selector outputs the event-weighted ingredient and ingredient-weighted event vectors by predicting theingredients used and related events. For example, in Fig. 6,
the raw “eggs” should be selected at h3 . This computation isachieved by replacing A with Gt−1 in Eq (8) and Eq (9) as
follows:
t−1
T
(W Q
)(W K
t−1
g G
h H)
√
Ĝh = {}(W Vh H),

d
K t−1
(W Q
H)(W
G )
h
√ g
Ĥ g = {}(W Vg Gt−1 ),
d

(10)
(11)

K
V
where W Q
g , W g , W g represents a linear layer.
(3) Updater represents the state transition of the ingredientsby updating the ingredient vectors Gt−1 . Based on the selectedt−1
actions and ingredients (Ȧh , ĜH ), the updater computes the
dot-product attention as follows:t−1

Gt = Gt−1 + Ĝhrepeat(max(Ȧh )),(12)

where repeat(·) expands the max-pooled action vector byrepeating it M times (M is the number of ingredients).
3 The dimension size of A is set to be equal to H.

Visual simulator loss
Ingredient label
Ingredient-event
attention matri
Eq. (10)Ingredien
selectio
loss

eggs areuse
at event 3

Oracle events (index=3)
h1

h2

h3

h4

h5

Action label
Add
Action-event
Crack attention matri
Eq. (8)Stir

Crack andStir areexecuteat event 3

Actio
selectioloss

Ground-truth sentence
Crack the eggs and stir it in a bowl

Textual attention loss
Ingredient label

Crack
the
eggs
and

Sentenc
Transformer

Crack
the
eggs
and
stir

…
d

n

n
 n
n
 

t
d

t
  

e
  

 

 

Ingredien
attentio
loss

Action label

Add Crack Stir

Crack
the
eggs
and
stir

stir

Word-ingredientattention matri
Eq. (13)

Word-action
attention matri
Eq. (15)

Actio
attentio
loss

x
  
 

Fig. 7. Overview of loss computation of visual selector loss and textual
attention loss.

Output representations. The action- and ingredientweighted event representations (Ḣ a , Ḣ g ) are used to computethe event probability by replacing H with Ḣ = H +Ḣ a +Ḣ g
in Eq (2). The event-weighted action vectors Ȧh and updated
ingredients Gt are forwarded to the textual attention module.
Gt is also set to be the ingredient vectors at (t + 1)-thprediction.
B. Textual attention
The textual attention module encourages the sentence generator to output a recipe grounded with the events based on theactions and ingredients. Let ŵk be the k-th output word vector
from the sentence transformer. Given (Gt , Ȧh ) and ŵk , thetextual attention module computes two attention matrices: (1)word-action attention and (2) word-ingredient attention. Thenit outputs the context vectors ûk as follows:αkmugk

exp{(ŵk )T W gu g tm }
P
g t ,
T
j exp {(ŵ k ) W u g j }X=
αkm g tm ,

=

(13)
(14)

m

αkr
uak

exp{(ŵk )T W au ȧrh }
,
P
a j
T
j exp {(ŵ k ) W u ȧh }X=
αkr ȧrh ,
=

(15)
(16)

r

ûk

=

concat(ŵk , ugk , uak ),(17)

where W gu , W au represents a linear layer and concat(·) indicates a concatenation function of vectors. ûk is forwarded
to the linear layer and softmax activation to obtain the word
probability across the vocabulary.
C. Loss functions
In addition to the losses described in Section IV-D, weintroduce the following two types of loss functions: (1) visual

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20158

simulator loss Lvsim and (2) textual attention loss Ltattn . Fig.
7 shows an overview of the loss computation for these twolosses.
(1) Visual simulator loss aims to train the visual simulator,
consisting of two losses: (1) ingredient selection loss and(2) action selection loss. Given the action- and ingredientevent attention matrices in Eq (8) and Eq (10), they arecomputed as the summed negative log-likelihood based on theingredients/actions and events that constitute the recipe story.
To avoid costly human annotations, we compute the loss
through distant supervision [35] following our previous work.
For the ingredient selection loss, labels are obtained whether
the sentence corresponding ground-truth event contains ingredients and the events are oracle or not at each step. For theaction selection loss, labels are obtained whether the sentence
has actions in the 384 actions defined by [36] and the events
are oracle or not at each step. For example, in Fig. 7, “eggs”
at the event index of 3 is extracted as an ingredient label, and“crack” and “stir” are extracted at the same event index as anaction label.
(2) Textual attention loss aims to train the textual attention,
consisting of two losses: (1) ingredient attention loss and(2) action attention loss. Given the word-ingredient/action
attention matrices in Eq (13) and Eq (15), they are computed
as the sum of the negative log-likelihood based on whether
the k-th word is the same as the ingredients/actions.
Total loss. We simply add these losses to the loss definedin Section IV-D as follows:
Lextended = Ltotal + Lvsim + Ltattn .

(18)

VI. E XPERIMENTS
We use the YouCook2 dataset [8], which consists of 2,000
cooking videos from 89 recipe categories. All of the videos
have 3–16 clips with a human-annotated start/end timestamp,and each clip is also annotated with an English sentence.
The average length of videos is 5.27 minutes per video.
The original YouCook2 dataset does not contain ingredientannotations, thus we use the YouCook2-ingredient dataset[12], which contains additional ingredient annotations. The
dataset also contains 1,788 videos with events, sentences,and ingredients. We use the official split proposed by [8] forevaluation. Because the test set is not available online, we use
the validation set for evaluation.
Event encoder. We employ different two encoders described below:• TSN [37] converts the appearance and optical flow intoframe-level representations, and then outputs the eventlevel representations by averaging them. For appearance, we use 2,048D feature vectors extracted from the“Flatten-673” layer in ResNet-200 [30]. For the opticalflow, 1,024D feature vectors extracted from the “globalpool” layer in BN-Inception [38]. Note that these models
are pre-trained on only vision resources (e.g., ImageNet[39]).
• MIL-NCE [31] converts the events into representations.
The model is pre-trained on Howto100M [32], which
consists of automatically constructed 100M clip-narration

pairs. We expect the MIL-NCE to yield better event
representations than the TSN because this model is pretrained on instructional vision-and-language resources.
Data preprocessing. As in [10], we truncated sequences
longer than 100 for the clip and 20 for the sentence and set
the maximum length of the clip sequence to 12. Finally, we
built the vocabulary based on words that occurred at least threetimes. The resulting vocabulary contained 991 words.
Hyper-parameter settings. For both the encoder and decoder transformers, we set the hidden size to 768, number oflayers to two, and number of attention heads to 12. We train themodel using the optimization method described in [40], [10];we use the Adam optimizer [41] with an initial learning rateof 0.0001, β1 = 0.9, and β2 = 0.999. The L2 weight decay is
set to 0.01, and the learning rate warmup is over the first fiveepochs. We set the batch size to 16, and continue training at
most 50 epochs using early stopping with SODA:CIDEr-D.
Models. We test the proposed method by comparing itwith four state-of-the-art dense video captioning models, as
described below:• Masked Transformer (MT) [42] is a transformer-based
encoder-decoder DVC model that can be trained in an
end-to-end manner by using a differentiable mask.
• Event-centric Hierarchical Representation for DVC(ECHR) [33] is an event-oriented encoder-decoder architecture for DVC. The ECHR incorporates temporal andsemantic relations into the output events for generating
captions accurately.
• SGR [15] is a top-down DVC model consisting of four
processes. The model (1) generates an overall paragraphfrom the input video, (2) grounds the sentences with
events in the video, (3) refines captions based on the
grounded events, and (4) refines events, referring to therefined captions.
• PDVC [14] is the state-of-the-art DVC model. It detects
N events densely, re-ranks the top K of them, andgenerates sentences for the re-ranked top K events. Notethat K is the prediction target. This model is used in our
preliminary experiments described in Section III.
Ablations. We examine the impact of the components ofthe proposed method through ablation studies on the following
variations:• Base model (B) is the model introduced in Section IV.
• B + Ingredient (BI) incorporates the ingredient encoder
into the base model.
• BI + Visual simulator (BIV) incorporates the visual
simulator into the BI model.
• BIV + Textual attention (VIVT) additionally incorporates the textual attention module into the BIV model.
A. Word-overlap evaluation
Table II demonstrates the results of the word-overlap evaluation on both dvc eval and SODA with BLEU, METEOR,
and CIDEr-D. We observe that the base model consistently
outperforms state-of-the-art captioning models by a significant
margin in both evaluations. In the ablation, the BIV model
outperforms the BI model, and the BIVT model further

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

9

TABLE IIW ORD - OVERLAP METRICS FOR THE BASELINE AND PROPOSED METHOD . T HE BOLD SCORES ARE THE BEST AMONG THE COMPARATIVE METHODS .
Input modality
MT
ECHR
SGR
PDVC (reported)
PDVC (reproduced)
Base (B)Base + Ingredients (BI)BI + Visual simulator (BIV)
BIV + Textual attention (BIVT)
OracleVideo feature

Video (V)V
V
V
V
V
Video + Ingredients (VI)VI
VI
V

dvc eval
METEOR
3.18
3.82
4.72
5.80
6.03
7.18
7.27
8.04
7.68

BLEU4
0.30
0.80
0.56
1.04
1.39
1.40
1.92
0.97

TSN
TSN
TSN
TSN
TSN
MIL-NCE
MIL-NCE
MIL-NCE
MIL-NCETSNTABLE IIIP ERCENTAGE OF RECIPES THAT SATISFY |p − q| ≤ η, WHERE p, q, η
REPRESENTS THE NUMBER OF PREDICTED EVENTS , GROUND - TRUTH
EVENTS , AND A THRESHOLD , RESPECTIVELY. I N THIS EXPERIMENT, WE
CHANGE η FROM 0 TO 3.

…

CIDEr-D
6.10
22.71
21.47
24.98
31.07
32.67
37.24
36.30

METEOR
4.35
4.42
3.99
5.45
6.44
6.46
7.29
9.64

0
14.4

1
40.0

2
63.0

(a)

Frequency

…

…

Time

(b)

(c)

(e)

(d)

(f)

(g)

PDVC

3
76.4

(b)

(c)(e) (d)

(f)

(g)

Base (B)
(a)

(b)

(c)

(d)

(f) (e)

BIVT

18.6
18.8
20.5
19.7

52.1
51.6
51.8
51.8

71.7
73.5
74.2
73.5

83.4
87.3
86.2
87.5

(a)

B
BI
BIV

135

…
Ground Truth

PDVC

180

tIoU
27.80
33.23
35.10
34.13
35.06
71.16

…

(a)

η
PDVC
Model
B
BI
BIV
BIVT

SODA
CIDEr-D
15.10
25.09
31.69
32.95
38.93
35.09

BIVT
GT

90
fl

(b)

(c)

(d)

(e)

(f)

Ground Truth

PDVC

Base (B)BIVT(a) Cut the pork into slice(b) Cover the pork inplastic wrap and poun(c) Sprinkle salt andpepper on top of the mea(d) Melt butter in the pa(e) Mix eggs milk salt andpepper togethe(f) Dip the pork in the egg
mixture and the bread
crumb(g) Fry the pork in the pan(a) Add oil and salt to abow(b) Mix the chicken in the
our and mi(c) Fry the chicken in apa(d) Fry the chicken in apa(e) Heat the oil in a pa(f) Fry the chicken in apa(g) Fry the chicken in thepan(a) Cut the fat off of thefa(b) Season the meat withsalt and peppe(c) Pour buttermilk into abowl and add salt andpeppe(d) Coat the veal in ou(e) Coat the meat in the
our mixtur(f) Place the veal in thepan(a) Cut the pork in half
and remove the por(b) Season the porkwith salt and peppe(c) Season the porkwith salt and peppe(d) Heat some butter ina pa(e) Coat the pork in the
break crumb(f) Fry the pork in a
pan

t
n
 l
 n
 s

r

e

x
 s
 r

r

flr

k

d
n

n

r
 s

t
 

45

Fig. 9. Examples of the generated recipes. We compare three models: PDVC,
B, and BIVT with the ground truth.

0
3

4

5

6

7

8

9

10

11

>11

#Predicted events

Fig. 8. Histogram of the number of predicted events and ground truth.

improves the BIV model. This indicates that both the dotproduct visual simulator and the textual attention module are
effective for accurate recipe generation.
B. Discussion on the number of predicted events
In addition to the word-overlap metrics, we discuss thegenerated recipes from the perspective of the number ofpredicted events. Table III shows the percentage of recipesthat satisfy |p − q| ≤ η, where p, q, η represents the number ofpredicted events, ground truth, and a threshold, respectively.
In this experiment, we change η from 0 to 3. This result
demonstrates that the proposed models consistently predict a
more precise number of events than the PDVC. Fig. 8 shows
the histogram of the number of the predicted events. While the
PDVC outputs the specific number of events (i.e., 5, 7, 10),the histogram of the proposed method draws a similar curveto that of the ground-truth.
C. Qualitative analysis
Fig. 9 shows the predicted events and generated recipes fromthe PDVC, B, and BIVT models, in comparison to that of theground-truth. In terms of the predicted events, PDVC outputs
highly overlapped events. The proposed method, on the other
hand, predicts events in the correct order, with minimal
overlap. This story-oriented sequential event prediction is anadvantage of the proposed method. In terms of the generated
recipes, the PDVC repeatedly generates the same contents,
ignoring the events (see (c) to (g)). The proposed methods
suppress this problem. A comparison of B and BIVT reveals
that the BIVT can generate sentences that are grounded with
events (e.g., “pork” is accurately verbalized in (a) in BIVT,
whereas “fat” is generated in (a) in B).
D. Discussion on the detailed model settings
Here, we discuss the detailed model settings from four
perspectives: (1) loss ablation studies, (2) memory updatestrategies, (3) event encoders, and (4) parameter sensitivity
and the event candidates N . The results demonstrate that theseparameters are important to succeed in our task.
1) Loss ablation studies: Table IV shows ablation studies
on the loss function for the BIVT model. The experimentyields two insights. First, all the losses are essential for training
the model. This is confirmed by removing all of them fromthe full model (compare (a) and (g)). Second, ingredient-based
losses are more necessary than action-based ones. Removing

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

10

TABLE IV
L OSS ABLATION STUDIES . IS, AS, IA, AND AA REPRESENT INGREDIENTSELECTION , ACTION SELECTION , INGREDIENT ATTENTION , AND ACTIONATTENTION LOSSES , RESPECTIVELY.
IS
(a)
(b)
(c)
(d)
(e)
(f)
(g)

AS

IA

AA

X
X
X
X
X

X
X
X
X
X

X
X

X
X

BLEU4
1.11
1.28
1.44
1.57
1.76
1.57
1.92

dvc eval
METEOR
6.37
7.31
7.32
7.45
7.79
7.70
8.04

CIDEr-D
26.54
31.59
32.19
33.56
36.84
34.58
37.24

METEOR
5.69
6.46
6.35
6.76
7.12
6.93
7.29

SODA
CIDEr-D
26.54
33.10
30.86
35.45
36.96
37.42
38.93

tIoU
33.15
33.78
33.35
34.56
34.91
34.53
35.06

TABLE VC OMPARISON OF MEMORY UPDATE STRATEGIES : SEPARATE VS JOINT.

B
Separate
Joint
BIV
Separate
Joint
BIVT
Separate
Joint

BLEU4

dvc eval
METEOR

CIDEr-D

0.92
1.04

5.67
6.03

1.39
1.40

7.24
7.27

1.78
1.92

7.85
8.04

METEORSODA
CIDEr-D

tIoU

24.19
24.98

5.19
5.45

23.12
25.08

33.96
33.26

33.02
32.67

6.54
6.46

33.00
32.95

33.81
34.13

36.65
37.24

6.92
7.29

36.84
38.93

33.85
35.06

the ingredient selection loss has a more significant influenceon the performance, compared to the action selection loss
(compare (b) with (d) and (c) with (d)). This tendency is thesame as the relationship between the ingredient attention andaction attention losses.
2) Memory update strategies: separate or joint?: Table V
shows a comparison of the memory update strategies. While
the separate memory update does not mix the memories
in the event and sentence transformers, the joint approach
proposed in Section IV fuses these memories. The result
demonstrates that the joint approach outperforms the separate
approach, indicating the effectiveness of the joint memory
update strategy.
3) Event encoders: Table VI shows the performance difference when changing the event encoders, indicating that the
MIL-NCE proved significantly superior to the TSN in all of
the settings. This occurs because the MIL-NCE is pre-trainedon the vision-and-language resource, Howto100M, which
captures the fine-grained event-level semantics of cooking
procedures. We conclude that pre-training on an appropriate
resource is essential to effective performance on our task.
4) Parameter sensitivity and the event candidates N :The PDVC allows users to select the number of candidates
N when training the model. In the original study on the
PDVC, N is set to be 100 on the YouCook2 dataset. In thisexperiment, we change the parameter N to be 25, 50, 100 and
200 to investigate the parameter sensitivity of the model. Table
VII shows the performance change of the proposed method
in different model settings: B, BIV, and BIVT. The results
demonstrate that the proposed method consistently performs
well on N = 25. We observe that increasing N degrades themodel performance. Although a higher N makes the maximumtIoU larger (shown in Section III), the ratio of events thatare not oracle but highly overlapped with the ground-truth
increases. This prevents the model from selecting oracle events
precisely and causes it to overfit the training set.

TABLE VIC OMPARISON OF THE MODEL’ S PERFORMANCE WHEN VARYING THE
EVENT ENCODERS : TSN AND MIL-NCE. N OTE THAT UNLIKE TSN,
WHICH IS PRE - TRAINED ON ONLY VISION RESOURCES , THE MIL-NCE IS
PRE - TRAINED ON INSTRUCTIONAL VISION - AND - LANGUAGE RESOURCE ,H OWTO 100M.

B
TSN
MIL-NCEBIV
TSN
MIL-NCEBIVT
TSN
MIL-NCE

BLEU4

dvc eval
METEOR

CIDEr-D

METEORSODA
CIDEr-D

tIoU

0.36
1.04

4.24
6.03

15.55
24.98

3.79
5.45

14.98
25.08

31.71
33.26

0.52
1.40

4.93
7.27

18.98
32.67

4.51
6.46

18.32
32.95

31.77
34.13

0.99
1.92

5.87
8.04

23.60
37.24

5.26
7.29

22.84
38.93

32.36
35.06

TABLE VIIC OMPARISON OF THE MODEL’ S PERFORMANCE WHEN VARYING THE
NUMBER OF THE EVENT CANDIDATES N .

B
N=25
N=50
N=100
N=200
BIV
N=25
N=50
N=100
N=200
BIVT
N=25
N=50
N=100
N=200

BLEU4

dvc eval
METEOR

CIDEr-D

METEORSODA
CIDEr-D

tIoU

1.27
0.98
1.04
0.93

6.49
6.42
6.03
6.16

27.84
27.12
24.98
25.93

6.19
5.89
5.45
5.52

29.34
26.34
25.08
26.13

35.26
33.79
33.26
33.93

1.71
1.51
1.40
1.23

7.50
7.52
7.27
6.81

34.18
33.37
32.67
29.51

7.02
6.86
6.46
5.95

36.39
34.24
32.95
28.86

36.13
35.60
34.13
33.15

1.87
1.81
1.92
1.78

7.95
7.99
8.04
7.66

36.12
36.48
37.24
35.21

7.51
7.36
7.29
6.77

39.06
38.49
38.93
35.64

36.74
36.38
35.06
33.48

VII. C ONCLUSION
In this paper, we tackled recipe generation from unsegmented cooking videos, a task that requires agents to (1)extract key events that are essential to dish completion and(2) generate sentences for the extracted events. We analyzed
the output of the DVC model and observed that, although (1)several events are adoptable as a recipe story of a recipe, (2)the generated sentences for such events are not grounded in
the visual content. Therefore, we hypothesized that we canobtain correct recipes by selecting oracle events from theoutput events of the DVC model and re-generating sentences
for them. To achieve this, we proposed a transformer-based
joint learning approach for the event selector and sentence
generator. The event selector selects oracle events from theevent candidate in the correct order, and the sentence generator
generates a recipe grounded in the events. We also extend themodel by including ingredient inputs. The additional modules
encourage the model to generate more grounded recipes effectively. In experiments, we confirmed that the base model
outperforms the state-of-the-art DVC models and the extended
model boosts the model’s performance. In addition, we showedthat the proposed models can select the correct number of
events, as with the ground-truth events. The qualitative evaluation revealed that the proposed approaches can select eventsin the correct order and generate recipes grounded in thevideo content. Finally, we discussed the detailed experimental
settings for optimal recipe generation.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

R EFERENCES[1] A. Salvador, N. Hynes, Y. Aytar, J. Marin, F. Ofli, I. Weber, andA. Torralba, “Learning cross-modal embeddings for cooking recipes andfood images,” in Proc. CVPR, 2017, pp. 3020–3028.
[2] A. Salvador, M. Drozdzal, X. Giro-i-Nieto, and A. Romero, “Inverse
Cooking: recipe generation from food images,” in Proc. CVPR, 2019,
pp. 10 453–10 462.
[3] L. Pan, J. Chen, J. Wu, S. Liu, C.-W. Ngo, M.-Y. Kan, Y.-G. Jiang,
and T.-S. Chua, “Multi-modal cooking workflow construction for foodrecipes,” in Proc. ACMMM, 2020, pp. 1132–1141.
[4] D. P. Papadopoulos, E. Mora, N. Chepurko, K. W. Huang, F. Ofli, andA. Torralba, “Learning program representations for food images andcooking recipes,” in Proc. CVPR, 2022, pp. 16 559–16 569.
[5] H. Wang, G. Lin, S. C. H. Hoi, and C. Miao, “Structure-aware generation
network for recipe generation from images,” in Proc. ECCV, 2020, pp.
359–374.
[6] M. Zhang, G. Tian, Y. Zhang, and P. Duan, “Reinforcement learningfor logic recipe generation: Bridging gaps from images to plans,” IEEE
Transactions on Multimedia, 2021.
[7] B. Shi, L. Ji, Y. Liang, N. Duan, P. Chen, Z. Niu, and M. Zhou, “Dense
procedure captioning in narrated instructional videos,” in Proc. ACL,
2019, pp. 6382–6391.
[8] L. Zhou, C. Xu, and J. J. Corso, “Towards automatic learning ofprocedures from web instructional videos,” in Proc. AAAI, 2018, pp.
7590–7598.
[9] V. Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem, “DAPs: deep
action proposals for action understanding,” in Proc. ECCV, 2016, pp.
768–784.
[10] J. Lei, L. Wang, Y. Shen, D. Yu, T. Berg, and M. Bansal, “Mart:
memory-augmented recurrent transformer for coherent video paragraphcaptioning,” in Proc. ACL, 2020, pp. 2603–2614.
[11] B. Shi, L. Ji, Z. Niu, N. Duan, M. Zhou, and X. Chen, “Learning
semantic concepts and temporal alignment for narrated video procedural
captioning,” in Proc. ACMMM, 2020, pp. 4355–4363.
[12] T. Nishimura, A. Hashimoto, Y. Ushiku, H. Kameko, and S. Mori,“State-aware video procedural captioning,” in Proc. ACMMM, 2021, pp.
1766–1774.
[13] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Densecaptioning events in videos,” in Proc. ICCV, 2017, pp. 706–715.
[14] T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo, “End-to-end
dense video captioning with parallel decoding,” in Proc. ICCV, 2021,
pp. 6847–6857.
[15] C. Deng, S. Chen, D. Chen, Y. He, and Q. Wu, “Sketch, ground, andrefine: Top-down dense video captioning,” in Proc. CVPR, 2021, pp.
234–243.
[16] S. Fujita, T. Hirao, H. Kamigaito, M. Okumura, and M. Nagata, “Soda:
Story oriented dense video captioning evaluation framework,” in Proc.
ECCV, 2020, pp. 517–531.
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc.
NeurIPS, 2017, pp. 5998–6008.
[18] J. Wu, L. Pan, J. Chen, and Y.-G. Jiang, “Ingredient-enriched recipe
generation from cooking videos,” in Proc. ICMR, 2022, pp. 249–257.
[19] C. Kiddon, L. Zettlemoyer, and Y. Choi, “Globally coherent text
generation with neural checklist models,” in Proc. EMNLP, 2016, pp.
329–339.
[20] A. Bosselut, A. Celikyilmaz, X. He, J. Gao, P.-S. Huang, and Y. Choi,“Discourse-aware neural rewards for coherent text generation,” in Proc.
NAACL-HLT, 2018, pp. 173–184.
[21] J. Harashima, Y. Someya, and Y. Kikuta, “Cookpad image dataset: An
image collection as infrastructure for food research,” in Proc. ACM
SIGIR, 2017, pp. 1229–1232.
[22] K. Chandu, E. Nyberg, and A. W. Black, “Storyboarding of recipes:
Grounded contextual generation,” in Proc. ACL, 2019, pp. 6040–6046.
[23] T. Nishimura, A. Hashimoto, and S. Mori, “Procedural text generationfrom a photo sequence,” in Proc. INLG, 2019, pp. 409–414.
[24] T. Nishimura, A. Hashimoto, Y. Ushiku, H. Kameko, Y. Yamakata, and
S. Mori, “Structure-aware procedural text generation from an image
sequence,” IEEE Access, vol. 9, pp. 2125–2141, 2020.
[25] L. Zhou, Y. Kalantidis, X. Chen, J. J. Corso, and M. Rohrbach,“Grounded video description,” in Proc. CVPR, 2019, pp. 6578–6587.
[26] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method forautomatic evaluation of machine translation,” in Proc. ACL, 2002, pp.
311–318.

11[27] S. Banerjee and A. Lavie, “METEOR: an automatic metric for MT
evaluation with improved correlation with human judgments,” in Proc.
ACL Workshop IEEMMTS, 2005, pp. 65–72.
[28] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: consensus-based
image description evaluation,” in Proc. CVPR, 2015, pp. 4566–4575.
[29] E. Jang, S. Gu, and B. Poole, “Categorical reparametrization withgumble-softmax,” in Proc. ICLR, 2017.
[30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. CVPR, 2016, pp. 770–778.
[31] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman, “End-to-end learning of visual representations from uncurated
instructional videos,” in Proc. CVPR, 2020, pp. 9879–9889.
[32] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic,“HowTo100M: learning a text-video embedding by watching hundred
million narrated video clips,” in Proc. ICCV, 2019, pp. 2630–2640.
[33] T. Wang, H. Zheng, M. Yu, Q. Tian, and H. Hu, “Event-centrichierarchical representation for dense video captioning,” IEEE Trans. on
Circuits and Systems for Video Technology, pp. 1890–1900, 2021.
[34] J. Pennington, R. Socher, and C. Manning, “Glove: global vectors forword representation,” in Proc. EMNLP, 2014, pp. 1532–1543.
[35] M. Mintz, S. Bills, R. Snow, and D. Jurafsky, “Distant supervision for
relation extraction without labeled data,” in Proc. ACL-IJCNLP, 2009,
pp. 1003–1011.
[36] A. Bosselut, O. Levy, A. Holtzman, C. Ennis, D. Fox, and Y. Choi,“Simulating action dynamics with neural process networks,” in Proc.
ICLR, 2018.
[37] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool,“Temporal segment networks for action recognition in videos,” IEEE
Trans. on Pattern Analysis and Machine Intelligence, pp. 2740–2755,
2019.
[38] S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, 2015,
pp. 448–456.
[39] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet:
A large-scale hierarchical image database,” in Proc. CVPR, 2009, pp.
248–255.
[40] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: pre-trainingof deep bidirectional transformers for language understanding,” in Proc.
NAACL, 2019, pp. 4171–4186.
[41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”in Proc. ICLR, 2015.
[42] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong, “End-to-end
dense video captioning with masked transformer,” in Proc. CVPR, 2018,
pp. 8739–8748.