Recipe Generation from Unsegmented Cooking Videos
TAICHI NISHIMURA∗ , Graduate School of Informatics, Kyoto University, Japan
ATSUSHI HASHIMOTO† , OMRON SINIC X Corporation, JapanYOSHITAKA USHIKU† , OMRON SINIC X Corporation, JapanHIROTAKA KAMEKO‡ , Academic Center for Computing and Media Studies, Kyoto University, JapanSHINSUKE MORI‡ , Academic Center for Computing and Media Studies, Kyoto University, Japan
This paper tackles recipe generation from unsegmented cooking videos, a task that requires agents to (1) extract key events incompleting the dish and (2) generate sentences for the extracted events. Our task is similar to dense video captioning (DVC), whichaims at detecting events thoroughly and generating sentences for them. However, unlike DVC, in recipe generation, recipe story
awareness is crucial, and a model should extract an appropriate number of events in the correct order and generate accurate sentences
based on them. We analyze the output of the DVC model and confirm that although (1) several events are adoptable as a recipestory, (2) the generated sentences for such events are not grounded in the visual content. Based on this, we set our goal to obtaincorrect recipes by selecting oracle events from the output events and re-generating sentences for them. To achieve this, we propose atransformer-based multimodal recurrent approach of training an event selector and sentence generator for selecting oracle eventsfrom the DVC’s events and generating sentences for them. In addition, we extend the model by including ingredients to generate moreaccurate recipes. The experimental results show that the proposed method outperforms state-of-the-art DVC models. We also confirm
that, by modeling the recipe in a story-aware manner, the proposed model outputs the appropriate number of events in the correct
order.
CCS Concepts: • Computing methodologies → Natural language generation; Scene understanding.
Additional Key Words and Phrases: cooking recipe, video understanding
ACM Reference Format:
Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori. 2018. Recipe Generation from
Unsegmented Cooking Videos. In . ACM, New York, NY, USA, 25 pages. https://doi.org/XXXXXXX.XXXXXXX

1INTRODUCTION

With a rapid increase of cooking videos uploaded on the web, multimedia food computing has become an important
topic [25, 26, 29, 30, 35, 40]. Among various developing technologies, recipe generation from unsegmented cooking
videos [31] is challenging. It requires artificial agents to (1) extract key events that are essential to dish completion and(2) generate sentences for the extracted events. This task is important for both scene understanding and real-world
applications. In terms of scene understanding, (1) and (2) are recognized as temporal event localization [8, 42] and video
captioning [18, 23, 32], respectively, and learning both simultaneously is an ambitious challenge in computer vision(CV) and natural language processing (NLP). Meanwhile, from a perspective of real-world applications, this technology
can support people learning new skills by providing key events and their explanation sentences as a multimedia
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2018 Association for Computing Machinery.
Manuscript submitted to ACM
1

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

DVC: parallel prediction
Input Video

Sentences

Events

Add the butter to the pa
Add the butter to the pa…

Paralleeven
prediction

Parallesentenc
generation

…

…

…

Crack the egg
Add them into the bow
Cook ingredient
Cook ingredientsT

Ours: multimodal recurrent prediction
Event Candidates (N=10)

Input Video

Events and sentences
Proposed method…
Paralleeven
prediction

Even Multimoda Sentenc
selector memor generator
mixing1

Add the butter
into a pan

2

Crack the egg
and stir

3

Add the egg
mixture and cheeseand stir

…
T

…

t
 y
t

l
 e

l

l
 e
  

s

 

s
  
 s

l

n

Fig. 1. A conceptual comparison of our approach and existing DVC studies. While the existing DVC models adopted parallel prediction,
our approach employ multimodal recurrent prediction, which estimates events and sentences by memorizing and fusing the previously
prediction results.

summarization of cooking videos. These academic and industrial motivations impel us to tackle recipe generation from
unsegmented cooking videos.
Our task is similar to dense video captioning (DVC) [5, 17, 37], which aims at detecting events densely from videos
and generating sentences for them. Although the input/output pairs of our task (video/(events, sentences)) and that of
the DVC are in the same format, our task is different from the perspective of the recipe’s story awareness. DVC allows
agents to exclusively detect false-positive events in its evaluation metrics. Our task, on the other hand, requires them toextract the accurate number of events in the correct order and generate sentences for them. Fujita et al. [9] reported
that DVC models produced more than 200 redundant events per video on average on the ActivityNet captions dataset[17] while the number of manually-annotated events is only 3-4. Such redundant events and sentences make it difficult
for users to grasp an overview of the video content.
The reason for the redundant outputs is that existing DVC approaches employed parallel prediction, wherein eventsand sentences are estimated independently (Fig. 1 top). These approaches consist of two modules: event prediction
and sentence generation modules. Both modules do not take into account the surrounding events and sentences when
predicting the current ones, leading to duplicated outputs. The choice of parallel prediction is reasonable for the DVC
objective of thoroughly detecting events and generating sentences for multimedia retrieval. However, it is not suitablefor our recipe generation task because we place high importance on providing an overview of videos that can be easily
grasped by humans.
2

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Although the events predicted by DVC models are redundant, we observed that (1) several events are adoptable asa story of a recipe, but (2) the generated sentences for such events are not grounded well to the visual contents (i.e.,
ingredients and actions in the sentences are incorrect). We confirm this by analyzing the outputs of the state-of-the-art
DVC model. We refer to the DVC output events as event candidates. Through an approach that we refer to as oracle
selection, we select events that have the maximum temporal IoU (tIoU) to ground-truth events (i.e., manually-annotated
events) and compute the DVC scores [9, 17]. The results support our observations. Although oracle selection ensuresthat events that are adoptable as the story of a recipe are selected, the obtained sentences are not grounded in the visualcontent.
Based on this analysis, we set our goal to obtain correct recipes by selecting oracle events from the event candidates
and re-generating sentences for them. To realize this, the crucial idea is multimodal recurrent prediction, which estimates
the next step by understanding previously predicted events and generated sentences. The bottom of Fig. 1 showsa concept of our idea. To predict the events and sentences in step 3, the models should understand both the visualand textual content of step 2. To achieve this, we propose a transformer[33]-based recurrent learning approach ofan event selector and sentence generator. As with [18], both of the modules have memory representations, whichremember previously selected events and generated sentences to predict the next step effectively. In addition, theproposed multimodal memory mixing method enables the modules to share the history of the previous prediction
results, contributing the better performance. The overall model is designed to be trainable in an end-to-end manner
because the modules of the model are connected without breaking a differentiable chain. We refer to this model as thebase model.
We also propose an extended model in the settings where the inputs are videos with ingredients as in [23, 39]. The
motivation behind this extension is to enhance the model’s ability to generate more accurate recipes by incorporating
the actual ingredients depicted in the videos. Relying solely on video inputs is problematic since even humans struggle
to verbalize precise ingredients without additional contextual information. For example, the kinds of seasoning (e.g.,salt and sugar) or meat parts (e.g., chuck and rib) are the exact cases. To develop a reliable recipe generation system, themodels need to accept supplementary ingredient inputs.
This task requires agents to describe detailed manipulations of ingredients from cooking videos. To achieve this goal,
our extended model has two additional modules: (1) a dot-product visual simulator and (2) a textual attention module.
The dot-product visual simulator is introduced by the insights from our previous work [23], where we discovered thatthe manipulations change the state of the ingredients, yielding state-aware visual representations. In Fig. 1, “eggs” aretransformed into “cracked” and then “stirred” form to cook scrambled eggs. To address this, we introduced the visual
simulator, which reasons about the state transition of ingredients, into the model. The motivation for the extension isthe difference in the inputs. Although the inputs in our previous work are pre-segmented ground-truth events withingredients, the inputs in this work are event candidates. The dot-product visual simulator accepts them and reasons
about the state transition of ingredients. Furthermore, we also introduce a textual attention module that verbalizes
grounded ingredients and actions in the recipes. These modules are effective for grounded recipe generation fromcooking videos.
In our experiments, we use the YouCook2 [42] dataset, which contains 2,000 videos with event/sentence annotation
pairs. The proposed method outperforms the state-of-the-art DVC approaches based on commonly used DVC metrics.
In addition, the extended models boost the model’s performance. We also show that the proposed models can selectthe correct number of events that effectively reflects the ground-truth events. In addition, our qualitative evaluation
3

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

Table 1. A comparison of our methods with previous studies on recipe generation and video captioning. T, I, IS, S, P, and U represents
text, images, image sequences, short event, pre-segmented sequential events, and unsegmented videos, respectively.
Methods
Kiddon et al. [14]Salvador et al. [30]
Chandu et al. [4]Shi et al. [32]Wu et al. [39]
Nishimura et al. [23]Lei et al. [18]Shi et al. [31]Zhou et al. [43]Wang et al. [37]
Ours

Input
T
I
IS
S
P
P
P
U
U
U
U

Modalities
Title + ingredients
Image
Image
Video + transcription
Video + ingredientsVideo + ingredientsVideo
Video + transcription
Video
Video
Video (+ ingredients)

DomainCooking
Cooking
Cooking
Cooking
Cooking
Cooking
GeneralCooking
GeneralGeneralCookingreveals that the proposed approaches can select events in the correct order and generate recipes grounded in the video
contents. We discuss the detailed experiment settings for optimal recipe generation.
In summary, our contributions are summarized as follows:• This study focuses on enhancing the story awareness of recipes, where agents need to accurately extract the
number of events in the correct order and generate corresponding sentences. Based on the analysis of the DVC
models, we set our goal to obtain accurate recipes by selecting oracle events from the DVC output events andre-generating sentences for them.
• To achieve this goal, we propose transformer-based multimodal recurrent learning of an event selector andsentence generator. Building upon the predicted events from DVC models, our method effectively recurrently
estimates the next step by memorizing and mixing previously predicted events and sentences.
• Furthermore, we extend our model to handle inputs comprising videos and ingredients. The motivation behind thisextension is to enhance the model’s ability to generate more precise recipes by incorporating actual ingredientsobserved in cooking videos. We introduce two additional modules to the base model, allowing it to verbalize
detailed manipulations of ingredients in the cooking videos.
• Our experiments demonstrate that the proposed methods obtain the story awareness of a recipe. They outperformthe DVC approaches especially on the story-oriented metrics, select the accurate number of events in the correct
order, and generate accurate sentences for them. In addition, we investigate the detailed experiment settings for
optimal recipe generation.
2

RELATED WORKIn this section, we describe the novelty of the proposed method in line with other works on recipe generation from visualobservations (Section 2.1) and general video captioning (Section 2.2). Table 1 shows a comparison of our approacheswith the previous studies.
2.1

Recipe generation from visual observations

Recipes are a popular target in the NLP community because they require artificial agents to generate coherent sentences[2, 15]. The inputs of these works are the title and ingredients of recipes. Recently, many multimodal cooking datasets
have appeared [10, 30] and recipe generation from images has also been proposed [29, 35]. Salvador et al. [29] tackled
this problem using a transformer-based model [33] to generate high-quality recipes from a single image of a completed
4

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

dish. Other researchers focus on generating a recipe from an image sequence depicting the intermediate food states[4, 22, 24].
The essential limitation of generating recipes from images is the lack of detailed, continuous scene informationof human manipulations, which images do not contain. Hence, this task is essentially an ill-posed problem; that is,
it depends on the obtained language model to generate correct actions. Meanwhile, videos contain this informationand have been attracting the attention of many researchers in recent years [23, 31, 32, 39]. As described in Section 1,
this task requires agents to (1) extract key events from videos and (2) generate grounded sentences for events. Most
researchers have focused on (2) generating grounded recipes from pre-segmented key events [23, 32, 39]. Our work
differs from the studies categorized in (2) because our task requires the models to predict both (1) and (2) jointly.
However, only a few researchers attempt to learn both (1) and (2) to generate recipes from unsegmented videos. Shi etal. [31] are pioneer researchers that have tackled this problem by utilizing the narration of videos effectively. Although
this approach is effective for narrated videos, transcription is not always available for all cooking videos. Speaking
during cooking or adding narration to videos requires significant effort. In addition, narrations may lead the model toattend to textual features, ignoring the visual features of videos. Our task extends their work to generate recipes only
from videos. This enables the model to treat even non-narrated videos and be more useful than the previous setting.
2.2

Video captioning

Video captioning is an attractive field for both CV and NLP communities. The task settings of video captioning vary
according to the nature of the input video (e.g., one short event, pre-segmented sequential events, or long unsegmented
video). We aim to extract key events and generate sentences simultaneously from long unsegmented videos. This task
is similar to DVC because the input/output pairs are in the same format as in our task. We first describe traditional
DVC approaches and then enumerate the difference between DVC and our task.
DVC aims at densely detecting events from the video and generating sentences for them. Recently, transformer-based
approaches are well investigated for this task. Zhou et al. [41] proposed a Masked Transformer, which detects eventsand generates sentences via a differentiable mask in an end-to-end manner. Their approach, however, outputs morethan 200 redundant events per video on average [9]. To handle this issue, Wang et al. [37] proposed PDVC, which
detects events in parallel, re-ranks the top 𝐾 (𝐾 is also the prediction target), and generates sentences for the re-ranked
events. Deng [5] proposed a top-down DVC approach, whereby whole sentences generated for the video are aligned
into timestamps, based on which the sentences are refined.
The key difference between our task and DVC lies in the story awareness of recipes. Our task requires agents to
extract an appropriate number of events in the correct order. DVC allows a model to detect events densely; however,redundant event/sentence pairs impede readability, and users are unable to grasp an overview of the video contents. To
achieve story awareness, we propose a transformer-based joint learning approach of an event selector and sentence
generator based on our observation that oracle events can be selectable from the output events of a DVC model.
3

ORACLE-BASED ANALYSIS OF THE EXISTING DVC MODELAs Fujita et al. [9] reported, the outputs of DVC models are redundant. However, we observed that although (1) several
events are adoptable as a story of a recipe, (2) the generated sentences for such events are not grounded well in the visualcontents. To verify this, we analyze the outputs of the state-of-the-art DVC model. Specifically, we select events with the
maximum tIoU scores to ground-truth events from event candidates and compute the DVC scores, an approach referred5Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

Table 2. Word-overlap metrics of the oracle selection on the YouCook2 dataset. 𝑁 represents the number of candidate events, a
hyper-parameter of PDVC. The bold scores are the best among the comparative settings.

PDVC
Oracle
N=25
N=50
N=100
N=200

BLEU4
0.89
0.58
0.84
0.97
1.10

dvc_eval
METEOR CIDEr-D
4.52
21.50
6.09
6.92
7.68
8.15

N=25

27.12
31.63
36.26
38.60

METEOR
3.98

SODA
CIDEr-D
25.30

tIoU
27.80

7.62
8.83
9.64
10.43

26.32
29.93
35.08
36.89

56.55
64.58
71.16
76.71

N=50

N=100

N=200

4000
3000
2000
1000
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.6

0.7

0.8

0.9

(a) Training
1200
900
600
300
0

0

0.1

0.2

0.3

0.4

0.5

(b) Validation
Fig. 2. tIoU distribution of oracle events on the training and validation sets of YouCook2.

to as oracle selection hereinafter. The analyzed results demonstrate that the oracle selection boosts the performance,
especially on story-oriented DVC metrics.
We use the YouCook2 dataset [42], one of the largest cooking video-and-language datasets. For the DVC model, we
employ PDVC [37], the state-of-the-art DVC model. It detects 𝑁 events densely and then re-ranks the top 𝐾 of them for
its outputs. Note that 𝑁 is a hyper-parameter1 and 𝐾 is the prediction target. We adopt this model because it achieves
the best performance on DVC tasks and it is easy to control 𝑁 before training the model. We use the 𝑁 detected events
for the analysis, not the re-ranked 𝐾 events.
Evaluation metrics. We use two commonly-used DVC metrics: dvc_eval [17] and SODA [9].
1 𝑁 is set to be a sufficiently large number. For YouCook2, the authors of [37] set 𝑁 to be 100 as a default parameter.

6

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY• dvc_eval firstly computes tIoU of all the combinations between the prediction and ground-truth events and thencomputes word-overlap metrics (e.g., METEOR or CIDEr-D), if the tIoU scores are over the threshold 𝜃 . 𝜃 ranges
from 0.3 to 0.9 by 0.2. Their average is the output score of these metrics.
• SODA stands for story-oriented dense video captioning evaluation, whereby the story awareness of the output
events is evaluated. Specifically, it uses dynamic programming to explore an alignment of events between
prediction and ground truth for obtaining the maximum story scores. The story scores are computed as a productof tIoU and word overlap metrics. Because SODA evaluates the predicted events by penalizing redundant outputs,
it is suitable for computing the story awareness of recipes.
SODA evaluates whether the output event/sentence pairs are appropriate as a story; thus, it is rated above dvc_eval
in this study. As word-overlap metrics, we use BLEU [27], METEOR [1], and CIDEr-D [34], which are commonly-used
metrics for text generation tasks. We also introduce SODA tIoU, which computes story scores as tIoU scores, rather thana product of tIoU and word overlap metrics. These metrics can evaluate whether the selected events are appropriate ascomponents of the generated recipes. In this analysis, we range 𝑁 from 25, 50, 100, and 200.
3.1

Quantitative evaluation

Table 2 shows the results of the oracle selection on dvc_eval and SODA metrics, indicating that the oracle selection
outperforms the PDVC. Specifically, the SODA scores of the oracle selection are quite better than that of the PDVC,
demonstrating that the generated recipes are more suitable as a story than the ones generated by the PDVC.
Fig. 2 shows the distribution of tIoU scores of the oracle events on training and validation sets. The number ofcandidate events 𝑁 was directly proportional to the average tIoU. This is because the more 𝑁 was, the more suitableoracle events appear in the candidate. Both the training and validation sets confirm this tendency.
3.2

Qualitative evaluation

Fig. 3 shows a comparison of the recipes generated by the oracle selection and ground truth. The selected event
timestamps are close to the ground-truth events, indicating that the appropriate selection can construct the correct
recipes. However, the sentences are different from the ground truth (e.g., “baking soda”, “salt”, and “pepper” are missing
in step (1)).
The main reason for this error is parallel prediction, where the events and sentences are predicted independently. This
causes the model to miss or hallucinate ingredients and motivates us to propose a model that re-generates sentences forthe predicted events, rather than using the corresponding generated sentences without modification.
4

PROPOSED METHOD

Based on the oracle-based analysis, we set our goal to obtain correct recipes by selecting oracle events from the output
events of the DVC model and re-generating sentences for them. To achieve this goal, multimodal recurrent prediction
is essential, which memorizes and mixes the previously predicted events and generated sentences to estimate the next
step (Fig. 4).
We realize this idea by proposing transformer-based models, consisting of event selection and sentence generation
modules (Fig. 5), which we refer to them as an event selector and sentence generator. The event selector tries to choose
oracle events from event candidates repeatedly (Section 4.1) and the sentence generator outputs sentences for theselected events (Section 4.2). Both encoders are based on memory-augmented recurrent transformers (MART) [18],7Conference acronym ’XX, June 03–05, 2018, Woodstock, NY…

Taichi Nishimura, et al.

…

…

…

Ground-Truth
(a)

Time

(c)

(b)

(d)

(e)

(d)

(e)

Oracle Selection
(a)

(b)

(c)

Ground-Truth

Oracle Selection(a)Add our eggs baking soda salt andpepper to the bowl and sti(b)Add cold water to the bowl and sti(c)Cover the shrimp in the batter andbreadcrumb(d)Place the shrimp into a pan of hot oi(e)Remove the shrimp from the pan
fl

s

l

r

r

l

(a) Add our and mix to a bowl and mi(b) Mix the ingredients in the bow(c) Place the chicken in the batte(d) Fry the chicken in the oi(e) Remove the soup from the oil

r
 x
 l

Fig. 3. Comparison of the recipes generated by the oracle selection and ground truth. 𝑁 = 100, which is a default hyper-parameter ofPDVC, is used in this example.

DVC approaches:parallel prediction

Ours:
multimodal recurrent prediction

Parallel event/sentence prediction

Multimodal Recurrent predictionof the events and sentences
previous events

Add the ingredientsto the bowl

next step

next step

previous sentences

3

3 add shrimp into
the frying pan

1 add shrimp to the bow
2 add eggs and waterand mix it

1
2

Multimodal memory mixing
MemoryEvent extracto(Transformer)…

Sentence generato(Transformer, LSTM)…

MemoryEvent extracto(Memoryaugmente
Transformer)

updatemix

Sentence generato(Memoryaugmente
Transformer)

MemoryMemory…
…
…

Frames

 

-
 

d

 
 r

 

r

l
 r

 

r

 

…

Event

Event

 

Fig. 4. An introductory overview of our approach. Unlike the previous DVC approaches, we propose a multimodal recurrent learning
approach to train the event selector and sentence generator. Both modules represent the previously predicted events and sentences as
memory vectors and predict the next step. These memory vectors are updated and mixed to effectively share the previous prediction
belonging to different modalities.

which contain memory vectors to learn the recurrence by remembering the previous prediction to estimate the next
step. In addition, the proposed multimodal memory mixing approach enables the model to share the previous prediction
belonging to different modalities by fusing their memory vectors (Section 4.3).
It is worth noting the advancements over MART. This study presents two improvements:
8

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Event Selector

Sentence Generator
Softmax

h1

[END]

h2

h4

h3

h5

h6

h7

h8

h9

LinearAdd & Norm

Add & Norm

Feed Forward

Feed Forward
Multi-hea
Attention

Multi-hea
Attention

Q

V

K

l
Vt−1

Q

Vtl

ConcatFeed Forward

Stl

Add & Norm
Multi-hea
Attention

e1

[END]

Event
End token

e2Multimoda
Memor
mixing

e3

TE

e4

e5

e6

e7

e8

e9

w1

Event encoder (frozen)

Masked Multi-hea
Attention

Sentenc
Transformer
SelecteEvent Vector

Positiona
Encoding+

Vtl

Add & Norm

Next
step

×N

Timestam
Encoding

Stl

ConcatFeed Forward

Even
Transformer

V

K

l
St−1

Multimoda
Memor
mixing

Next
step

×N

h5
w2

Positiona
Encoding

+

w3

w4

w5

w6

Word embedding

add cold wate
to the bowl…
…

…

(Shifted right)
t
 
 y

 
 d

e
d

l
p

l

d
 r

Fig. 5. Multimodal recurrent learning approach of the event selector and sentence generator for recipe generation from unsegmented
cooking videos. The event selector tries to choose oracle events from event candidates repeatedly (Section 4.1) and the sentence
generator outputs sentences for the selected events (Section 4.2). The memories are updated and mixed to effectively remember thehistory of the events/sentences for predicting the next step (Section 4.3).

• Task extension. We extend the scope of the model to verify both event selection and sentence generation tasks,
whereas the original paper only validated its effectiveness in generating sentences from pre-segmented key
events.
• Multimodal memory mixing. MART contains one memory vector to tackle only the sentence generation,
but two memory vectors are necessary for our task, which predicts events and sentences jointly. How to treat
multiple memory vectors is not investigated in MART, thus we first propose a memory mixing method that
integrates them from both modules achieve accurate recipe generation (Section 4.3).
Mathematically, we formulate our task using notations. Let 𝑿 = (𝒙 1, 𝒙 2, . . . , 𝒙 𝑛 , . . . , 𝒙 𝑁 ) ∈ R2×𝑁 be event candidates,
where 𝒙 𝑛 consists of start and end timestamps. Note that 𝑿 is sorted on the start time of the events in chronologicalorder. Given 𝑿 , the model generates pairs (𝑪, 𝒀 ) = ((𝑐 1, 𝒚 1 ), . . . , (𝑐𝑡 , 𝒚𝑡 ), . . . , (𝑐𝑇 , 𝒚𝑇 )), where 𝑐𝑡 , 𝒚𝑡 , and 𝑇 represent
an index of the oracle event candidates, corresponding generated sentences, and the number of the selected events,
respectively. The memories 𝑽 𝑙𝑡 in the event selector and 𝑺 𝑙𝑡 in the sentence generator are jointly updated at each 𝑡 step,
where 𝑙 represents the layer number of transformers.
4.1

Event selector

The event selector can be divided into two main components: event encoder and event transformer. The event encoder
converts given candidate events 𝑿 into event-level representations 𝑬 = (𝒆 1, 𝒆 2, . . . , 𝒆𝑛 , . . . , 𝒆𝑇 ), based on which theevent transformer outputs holistic representations of events 𝑯 = (𝒉1, 𝒉2, . . . , 𝒉𝑛 , . . . , 𝒉𝑇 ). They are used for computing
event probabilities, which represent the likelihood of oracle events. We apply Gumbel softmax resampling [13] to select9Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

events and forward them to the sentence generator without breaking a differentiable chain during the training phase.
In the inference phase, events are deterministically selected by applying argmax to event probabilities.
Event encoder. The event encoder converts events 𝒙 𝑛 into representations 𝒆𝑛 . First, we extract event clips fromthe video, according to the start and end time of 𝒙 𝑛 , and input them into pre-trained visual encoders. One of thestraightforward ways to encode events is to average frame-level image representations extracted by ResNet [11].
However, we find that this approach is unable to capture fine-grained event semantics for overlapped events. For
example, assume that the ground-truth event is a scene of cutting potatoes, and the event candidates have two scenes,
where one is a scene of cutting potatoes and tomatoes, and another is of cutting only potatoes. We expect the model
to select the latter scene because the former contains extra information about cutting tomatoes. Representing events
based on the average of frame-level features cannot capture the semantic difference of these events effectively. Thus, itis necessary to employ an event encoder that focuses on extracting fine-grained event-level semantics.
To achieve this, we focus on the multiple instance learning-noise contrastive estimation (MIL-NCE) model [19]pre-trained on Howto100M [20]. The model is trained on more than 100M pairs of an event with narration, and it
can capture event-level semantics of overlapped events. Using this encoder, we obtain event-level representations as
𝑬 = (𝒆 1, 𝒆 2, . . . , 𝒆𝑛 , . . . , 𝒆𝑇 ). Then, the positional encoding (PE) [33] and relative encoding of events [38] are added to𝒆𝑛 . Note that the event encoders are pre-trained and their weights are frozen during the training phase because of thecomputational costs.
Event transformer. Given 𝑬, the event transformer outputs holistic representations of events 𝑯 = (𝒉1, 𝒉2, . . . , 𝒉𝑛 , . . . , 𝒉𝑇 ).
This module is based on the memory-augmented recurrent transformer [18], where each 𝑙 layer has the memories 𝑽 𝑙𝑡 toremember the history of the selected events. The output vectors 𝑯 = (𝒉1, 𝒉2, . . . , 𝒉𝑛 , . . . , 𝒉𝑇 ) and memories 𝑽 𝑙𝑡 are used
to compute 𝑛-th event probability 𝑝 (𝑐𝑡 = 𝑛|𝑽 𝑙𝑡 , 𝑿 ) as follows:𝑽𝑡
𝑝 (𝑐𝑡 = 𝑛|𝑽 𝑙𝑡 , 𝑯 )=max(𝑽 𝑡1, . . . , 𝑽 𝑙𝑡 , . . . , 𝑽 𝑡𝐿 ),(1)=

exp {(𝒉𝑛𝑡 ) T 𝑽 𝑡 },Í𝑡 T
𝑖 exp {(𝒉𝑖 ) 𝑽 𝑡 }(2)where max(·) represents an element-wise max-pooling of vectors. During training, the event selector selects events
through a Gumbel softmax resampling [13], which enables us to train the model in an end-to-end manner without
breaking a differentiable chain. This indicates that the loss computed on the sentence generator is backpropagated tothe event selector. During inference, the model selects the event index based on the argmax of 𝑝 (𝑐𝑡 = 𝑛|𝑽 𝑙𝑡 , 𝑯 ).
4.2

Sentence generator

Based on the selected event representations, the sentence generator outputs sentences grounded for them. Let the
selected event index be 𝑐ˆ𝑡 , and the selected event representation can be written as 𝒉𝑐ˆ𝑡 . This vector is added to the word
vectors 𝑾 = (𝒘 1, 𝒘 2, . . . , 𝒘 𝑘 , . . . , 𝒘 𝐾 ) from the word embedding layer, which is the concatenated neural networks ofthe pre-trained global vectors for word representation (GloVe) [28]2 and one-layer perceptron with ReLU activation. We
also add PE to word vectors 𝑾 and input them into the sentence transformer, another memory-augmented recurrent
transformer. The model generates words repeatedly by applying softmax and argmax operations to the output vectorsof the sentence transformer.

2We employ pre-trained 300D word embedding, which can be downloaded from http://nlp.stanford.edu/data/glove.6B.zip

10

Recipe Generation from Unsegmented Cooking Videos

4.3

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Multimodal memory mixing

One of the important contributions of this study is to mix the memories for sharing the prediction results between
the event selector and sentence generator. The motivation of this approach is that mixing memories is intuitively
effective for accurate recipe generation because the history of the selected events contributes to sentence generationand vice versa. Specifically, the memory vectors 𝑽 𝑙𝑡 , 𝑺 𝑙𝑡 are first separately updated by following the equations describedin Section 3.2 in MART paper, and then mixed as follows:𝑽ˆ 𝑡=

𝑓1 (𝑽 𝑡 ) ⊙ 𝜎 (𝑔2 (𝑔1 (𝑺 𝑡 ))),

(3)

𝑺ˆ 𝑡

=

𝑔1 (𝑺 𝑡 ) ⊙ 𝜎 (𝑓2 (𝑓1 (𝑽 𝑡 ))),

(4)

where 𝑓∗ (·), 𝑔∗ (·) represents a single linear layer and ⊙, 𝜎 represents the Hadamard dot product and sigmoid function,
respectively. The obtained 𝑽ˆ 𝑡 and 𝑺ˆ 𝑡 are forwarded into the next 𝑡 + 1 step. Note that we compare this with the original
MART algorithm that updates memory vectors separately, and confirm its effectiveness in Section 6.4.2.
4.4

Loss functions

To train the model in an end-to-end manner, we sum up two types of losses: event selection loss and sentence generation
loss. These losses are formulated as a negative log-likelihood of the event selection and sentence generation tasks asfollows:
𝐿𝑏𝑎𝑠𝑒

=

𝐿𝑒

=

𝐿𝑠

=

𝐿𝑒 + 𝐿𝑠 ,∑︁ ∑︁−log 𝑝 (𝑐𝑡 |𝑿, 𝑽 𝑙<𝑡 ),(𝑿 ,𝑪 ) 𝑡∑︁ ∑︁−
log 𝑝 (𝒚𝑡 |𝒉𝑐ˆ𝑡 , 𝑺 𝑙<𝑡 ),( 𝑿 , 𝑪 ,𝒀 ) 𝑡

(5)
(6)
(7)

where 𝐿𝑏𝑎𝑠𝑒 represents the total loss of the event selection and sentence generation losses 𝐿𝑒 , 𝐿𝑠 .
5

EXTENDED MODEL

In addition to the base model, we also propose an extended model in the settings where the inputs are videos withingredients, as in [23, 39]. The motivation behind this extension is to enhance the model’s ability to generate moreaccurate recipes by incorporating the actual ingredients depicted in the videos. Relying solely on video inputs is
problematic since even humans struggle to verbalize precise ingredient names without additional contextual information.
In Fig. 10, for instance, the correct ingredient is “pork,” yet the base model mistakenly generates “meat” and “veal.” To
develop a reliable recipe generation system, the models need to accept supplementary ingredient inputs.
The challenge is to describe detailed manipulations of ingredients from cooking videos. To achieve this goal, ourextended model has two additional modules: (1) dot-product visual simulator and (2) textual attention module (Fig. 6).
The dot-product visual simulator is introduced based on the insights from our previous work [23]. In our previous work,
we discovered that the manipulations change the state of the ingredients, yielding state-aware visual representations.
For example, “eggs” are transformed into “cracked,” then “stirred” forms to cook scrambled eggs. To address this, we
introduced the visual simulator into the model, which reasons about the state transition of ingredients. Althoughpre-segmented ground-truth events with ingredients are regarded as input in our previous work, they are considered
event candidates with ingredients in this study. Thus we extend it to the dot-product visual simulator described in
11Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

Event Selector

Sentence Generator
Softmax

h3

h2

h1

[END]

h4

h5

h6

h7

h8

Weighted actions
a· 1

h9

h

a· 2h
…
a· R

Linearh

Dot-produc
Visual simulator

Textual Attention Module
Updated ingredients

g1t
Add & Norm

g2t…

Feed Forward

gMt

Add & Norm

Next
step

Feed Forward
Multi-hea
Attention

Multi-hea
Attention

Q

V

K

l
Vt−1

Q

l
St−1

Vtl

ConcatStl

Add & Norm

Multimoda
Memor
mixingMulti-hea
Attention

Even
Transformer

Event
End token

Ingredient encoder

e2

e3

TE

e5

e6

e7

e8

e9

Event encoder (frozen)

Ingredient encoder

Flou
Baking sod
Shrim

Batter

…

r

t
 
p

y

 
 d

e
d

l
p

l

a

+

ConcatPositiona
Encoding

+

w2

w3

w4

w5

w6

Word embedding

add cold wate
to the bowl…

Batter…

Ingredients

Ingredients
…

h5

w1

Flou
Baking sod
Shrim

Next
step

×N

Positiona
Encoding+

e4

Masked Multi-hea
Attention

Sentenc
Transformer
SelecteEvent Vector

Timestam
Encoding

Multimoda
Memor
mixingVtl

Add & Norm

Next
step

×N

e1

Stl

ConcatConcat

[END]

V

K

Feed Forward

Feed Forward(Shifted right)

d
 r
 t

Fig. 6. Overview of the extended model for recipe generation from unsegmented cooking videos. To generate more accurate recipes, it
has additional two modules: (1) dot-product visual simulator and (2) textual attention. The dot-product visual simulator is introducedby the insights based on our previous work [23] to learn the state transition of ingredients. The textual attention module encourages
the sentence generator to verbalize actions and ingredients more accurately.

action vectors
add

A

crack
stir

(1) Action selector

·
Ha

a1

a2
a3

Event-actio
Attentioeq. (9)event vectors

h1

oracle
event

h2

H

Gt

·
Ah

h4

(2) Ingredient selector(n − 1)-th material
vector

g1n−1
g2n−1

G t−1

g3n−1
 

(3) Updater

Action-even
Attentioeq. (8)

h3
h5

n

n
n

 
n

t

t

t
y
 

Computing
event probabiliteq. (2)Ingredient-even
Attentioeq. (10)
Event-ingredien
Attentioeq. (11)

·
Gt−1
h

·
Hg

e1̂

Action-ingredien
Attentioeq. (12)e2̂

e3̂

Next stepComputing
event probabiliteq. (2)t

Fig. 7. Overview of the extended dot-product visual simulator that reasons about the state transition of ingredients. It has threecomponents: (1) action selector, (2) ingredient selector, and (3) updater. The dot-product attention is employed to treat the event
candidates.

Section 5.1. In addition, we also introduce a textual attention module that verbalizes grounded ingredients and actionsin the recipes. These modules are effective for grounded recipe generation from cooking videos.
12

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Another extension is to add an ingredient encoder to convert ingredients into representations 𝑮 0 . The ingredientencoders are concatenated neural networks of pre-trained GloVe [28] word embedding and multi-layer perceptrons(MLPs) with ReLU activation function and are added to the event selector and sentence generator without sharing theirparameters. Multi-word ingredients (e.g., parmesan cheese) are represented by the average embedding vector of thewords. They are concatenated with event/word representations and inputted to the event/sentence transformers asshown in Fig. 6.
5.1

Dot-product visual simulator

We first revisit the original visual simulator and then describe how we extend it to the dot-product visual simulator.
Visual simulator revisit. In the original study, the inputs are the pairs of ingredients and ground-truth events.
0 , . . . , 𝒈 0 ) and 𝑯
ˆ = (𝒉ˆ 1, 𝒉ˆ 2, . . . , 𝒉ˆ ˆ, . . . , 𝒉ˆ ˆ ) be the ingredient and ground-truth event vectors
Let 𝑮 0 = (𝒈 01, 𝒈 02, . . . , 𝒈𝑚𝑡𝑇𝑀
encoded by ingredient encoder (e.g., GloVe) and event encoder (e.g., MIL-NCE), respectively. Given (𝑮 0, 𝑯ˆ ), the visual
simulator reasons about the state transition of the ingredients by updating them at each 𝑡ˆ-th step. To this end, it consistsof three components: (1) action selector, (2) ingredient selector, and (3) updater. The action selector and ingredientselector predict executed actions and used ingredients at 𝑡ˆ-th step. Based on the selected actions/ingredients, the(𝑡ˆ − 1)-th ingredient vectors 𝑮 (𝑡ˆ−1) are updated into 𝑡ˆ-th new proposal ingredient vectors 𝑮 (𝑡ˆ ) , which are forwarded
into the next step. The visual simulator recurrently repeats the above process until processing the end element of theground-truth events.
Proposed extension. The proposed model is similar to the visual simulator because it also has the same threecomponents to update the ingredient vectors. However, instead of the ground-truth event vectors 𝑯ˆ , the event candidates
𝑯 are assumed to be the inputs in this study, that is, the model needs to predict not only the actions/ingredients but
also the events that constitute the recipe story.
To achieve this, we extend the visual simulator into the dot-product visual simulator shown in Fig. 7. It computes
the relationships between action-to- and ingredient-to-events as attention matrices and outputs four vectors: actionweighted and ingredient-weighted event vectors, event-weighted action, and ingredient vectors. The former two vectors
are used to calculate the event probabilities and the latter two vectors are forwarded to the textual attention module.
(1) Action selector outputs event-weighted action and action-weighted event vectors by predicting the executed
actions and related events. Let the action vectors be 𝑨 = (𝒂 1, 𝒂 2, . . . , 𝒂𝑟 , . . . , 𝒂𝑅 ), that is, the pre-defined action embedding,where 𝒂𝑟 represents 𝑟 -th actions and 𝑅 is the number of the actions. In Fig. 7, the actions “crack” and “stir” are executedat the oracle event 𝒉3 ; thus 𝒂 ∗ that corresponding crack and stir indices should be selected with the relation to 𝒉3 . This
computation can be formulated as the dot-product attention as follows:𝑄ˆ ℎ = softmax{
𝑨(𝑾 𝑎 𝑨)𝑇 (𝑾 ℎ𝐾 𝑯 )},√𝑑ˆ ℎ (𝑾 𝑉 𝑯 )𝑇 ,¤ℎ = 𝑨𝑨ℎ(8)

where 𝑾 𝑎 , 𝑾 ℎ𝐾 , 𝑾 𝑉
represents a linear layer and 𝑑 represents the dimension size of 𝑨 and 𝑯 3 . Note that Softmax(·)ℎ
represents the row-wise softmax operation on a matrix. We also acquire action-weighted event vectors 𝑯¤ 𝑎 as follows:𝑄

𝑄

𝑯¤ 𝑎 = softmax{(𝑾 ℎ 𝑯 )𝑇 (𝑾 𝑎𝐾 𝑨)}(𝑾 𝑉𝑎 𝑨)𝑇 ,√𝑑

𝑄

where 𝑾 ℎ , 𝑾 𝑎𝐾 , 𝑾 𝑉𝑎 represents a linear layer.
3 The dimension size of 𝑨 is set to be equal to 𝑯 .

13

(9)

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

(2) Ingredient selector outputs the event-weighted ingredient and ingredient-weighted event vectors by predicting
the ingredients used and related events. For example, in Fig. 7, the raw “eggs” should be selected at 𝒉3 . This computationis achieved by replacing 𝑨 with 𝑮 𝑡 −1 in Eq (8) and Eq (9) as follows:(𝑾 𝑔 𝑮 𝑡 −1 )𝑇 (𝑾 ℎ𝐾 𝑯 )
},
√
𝑑
𝑄

𝑡 −1
𝑮ˆ ℎ = softmax{

𝑡 −1
𝑡 −1
𝑇
𝑮¤ ℎ = 𝑮ˆ ℎ (𝑾 𝑉ℎ 𝑯) ,(𝑾 ℎ 𝑯 )𝑇 (𝑾 𝑔𝐾 𝑮 𝑡 −1 )𝑡 −1 𝑇
}(𝑾 𝑉
) ,
√
𝑔𝑮
𝑑

(10)

𝑄

𝑯¤ 𝑔 = softmax{

(11)

𝑄

where 𝑾 𝑔 , 𝑾 𝑔𝐾 , 𝑾 𝑉
𝑔 represents a linear layer.
(3) Updater represents the state transition of the ingredients by updating the ingredient vectors 𝑮 𝑡 −1 . Based on the¤ ℎ , 𝑮¤ 𝑡𝐻−1 ), the updater computes the updated ingredient representations as follows:
selected actions and ingredients ( 𝑨𝑡 −1

𝑮 𝑡 = 𝑮 𝑡 −1 + 𝑮¤ ℎ¤ ℎ )),⊙ repeat(max( 𝑨(12)where repeat(·) expands the max-pooled action vector by repeating it 𝑀 times (𝑀 is the number of ingredients).
Output representations. The action- and ingredient-weighted event representations ( 𝑯¤ 𝑎 , 𝑯¤ 𝑔 ) are used to compute¤ ℎ andthe event probability by replacing 𝑯 with 𝑯¤ = 𝑯 + 𝑯¤ 𝑎 + 𝑯¤ 𝑔 in Eq (2). The event-weighted action vectors 𝑨
updated ingredients 𝑮 𝑡 are forwarded to the textual attention module. 𝑮 𝑡 is also set to be the ingredient vectors at(𝑡 + 1)-th prediction.
5.2

Textual attention

The textual attention module encourages the sentence generator to output a recipe grounded with the events based on
ˆ = (𝒘 1, 𝒘 2, . . . , 𝒘 𝑘 , . . . , 𝒘 𝐾 ) be the output word vector sequence from the sentence
the actions and ingredients. Let 𝑾¤ ℎ ) and 𝑾
ˆ , the textual attention module computes two attention matrices: (1) word-ingredienttransformer. Given (𝑮 𝑡 , 𝑨
attention and (2) word-action attention. Then it outputs the context vectors 𝑼ˆ as follows:
𝒁𝑚

=

𝒁𝑎

=

𝑼ˆ

=

ˆ 𝑇 𝑮𝑡𝑾
softmax( √ ), 𝑼 𝑚 = 𝒁 𝑚 (𝑮 𝑡 )𝑇 ,
𝑑
¤
ˆ 𝑇𝑨
𝑾
¤ ℎ )𝑇 ,softmax( √ ℎ ), 𝑼 𝑎 = 𝒁 𝑎 ( 𝑨𝑑ˆ (𝑼 𝑚 )𝑇 , (𝑼 𝑎 )𝑇 ),concat(𝑾,

(13)
(14)
(15)

where concat(·) indicates a concatenation function of vectors. 𝑼ˆ is forwarded to the linear layer and softmax activation
to obtain the word probability across the vocabulary.
5.3

Loss functions

In addition to the losses described in Section 4.4, we introduce the following two types of loss functions: (1) visual
simulator loss 𝐿𝑣𝑠𝑖𝑚 and (2) textual attention loss 𝐿𝑡𝑎𝑡𝑡𝑛 . Fig. 8 shows an overview of the loss computation for these
two losses.
(1) Visual simulator loss 𝐿𝑣𝑠𝑖𝑚 aims to train the visual simulator, summing up two losses: (1) ingredient selection
loss 𝐿𝑖𝑠 and (2) action selection loss 𝐿𝑎𝑠 . Given the action- and ingredient-event attention matrices in Eq (8) and Eq (10),they are computed as the summed negative log-likelihood based on the ingredients/actions and events that constitutethe recipe story.
14

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Visual simulator loss: Lvsim
··
Gt−1
h

Material label
Material-event
attention matri
Eq. (4.11)

Ĝt−1
h

Materiaselectio
loss

eggs areuse
at event 3

Eq. (16)

Action label
Add
Crack

Action-event
attention matri
Eq. (4.8)

Aĥ

Stir

h1

h2

h3

Crack the eggs and stir it in a bowl

Eq. (17)··
Zm

Material label

Crack
the

Sentenc
Transformer

and

Crack
the
eggs
and
stir

…
d

n

n
 n

l
 l

n
 

d
 

e
  

 

 

Materia
attentioloss

Word-material
attention matri
Eq. (4.15)

Eq. (19)

Zm

eggs

Action label

Add Crack Stir

Crack
the
eggs
and
stir

stir

h5

Ground-truth sentence

Textual attention loss: Ltattn

eggs

h4

··
Ah
Crack andStir areexecuteat event 3

Actio
selectio
loss

Oracle events (index=3)

··
Za

crack

Word-action
attention matri
Eq. (4.17)

Actioattentioloss

Za

Eq. (20)

stir

 

 

x

Fig. 8. Overview of loss computation of visual selector loss and textual attention loss.

To avoid costly human annotations, we compute the loss through distant supervision [21] following our previous
work. For the ingredient selection loss, labels are obtained on whether the sentence corresponding ground-truth event
contains ingredients and whether the events are oracle or not at each step.
For the action selection loss, labels are obtained whether the sentence has actions in the 384 actions defined by [3]and whether the events are oracle or not at each step. For example, in Fig. 8, “eggs” at the event index of 3 is extracted
as an ingredient label, and “crack” and “stir” are extracted at the same event index as an action label. Mathematically,
let 𝐴¥ℎ , 𝐺¥ 𝑡 −1 be the action- and material-label binary matrices respectively, where each element represents 0 or 1. Basedℎon 𝐴ˆℎ , 𝐺ˆℎ𝑡 −1, 𝐴¥ℎ , 𝐺¥ℎ𝑡 −1 , the ingredient and action selection losses are computed as:
𝐿𝑖𝑠=−SumMat(log(𝐺ˆℎ𝑡 −1 ⊙ 𝐺¥ℎ𝑡 −1 )),

(16)

𝐿𝑎𝑠

=

−SumMat(log(𝐴ˆℎ ⊙ 𝐴¥ℎ )),

(17)

𝐿𝑣𝑠𝑖𝑚

=

𝐿𝑖𝑠 + 𝐿𝑎𝑠 ,(18)

where SumMat(·) represents the sum of all of the elements of the matrix.
(2) Textual attention loss 𝐿𝑡𝑎𝑡𝑡𝑛 aims to train the textual attention, consisting of two losses: (1) ingredient attention
loss 𝐿𝑖𝑎 and (2) action attention loss 𝐿𝑎𝑎 . Given the word-ingredient/action attention matrices in Eq (13) and Eq (14),they are computed as the sum of the negative log-likelihood based on whether the 𝑘-th word is the same as the15Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

ingredients/actions. Mathematically, let 𝒁¥ 𝑎 , 𝒁¥ 𝑚 be the action- and material-word binary matrices, which represent
whether the 𝑘-th word is the same as the actions/ingredients. Given the word-ingredient/action attention matrices
𝒁 𝑚 , 𝒁 𝑎 and 𝒁¥ 𝑎 , 𝒁¥ 𝑚 , they are computed as:=−SumMat(log(𝒁 𝑚 ⊙ 𝒁¥ 𝑚 )),

(19)

𝐿𝑎𝑎

=

−SumMat(log(𝒁 𝑎 ⊙ 𝒁¥ 𝑎 )),(20)

𝐿𝑡𝑎𝑡𝑡𝑛=𝐿𝑖𝑎 + 𝐿𝑎𝑎 .

(21)

𝐿𝑖𝑎

Total loss. We simply add these losses to the loss defined in Section 4.4 as follows:
𝐿𝑒𝑥𝑡𝑒𝑛𝑑𝑒𝑑 = 𝐿𝑏𝑎𝑠𝑒 + 𝐿𝑣𝑠𝑖𝑚 + 𝐿𝑡𝑎𝑡𝑡𝑛 .
6(22)

EXPERIMENTS

We use the YouCook2 dataset [42], which consists of 2,000 cooking videos from 89 recipe categories. All of the videos
have 3–16 clips with a human-annotated start/end timestamp, and each clip is also annotated with an English sentence.
The average length of videos is 5.27 minutes per video. The original YouCook2 dataset does not contain ingredientannotations, thus we use the YouCook2-ingredient dataset [23], which contains additional ingredient annotations. The
dataset also contains 1,788 videos with events, sentences, and ingredients. We use the official split proposed by [42]for evaluation. Because the test set is not available online, we use the validation set for evaluation. When training the
proposed models, we first set the pre-trained weights of event encoders (e.g., MIL-NCE), word embedding (GloVe), andingredient embedding (GloVe). Then we freeze only the event encoders and finally train other modules. Note that themodel is trained from scratch except for these pre-trained modules, and the frozen event encoder accounts for 18%
parameters (9.6M/52.7M) of total parameters. For our experiments, we use a single NVIDIA RTX A5000 GPU.
Event encoder. We employ different two encoders.
• TSN [36] converts the appearance and optical flow into frame-level representations, and then outputs theevent-level representations by averaging them. For appearance, we use 2,048D feature vectors extracted fromthe “Flatten-673” layer in ResNet-200 [11]. For the optical flow, 1,024D feature vectors were extracted from the“global pool” layer in BN-Inception [12]. Note that these models are pre-trained on only vision resources (e.g.,
ImageNet [6]).
• MIL-NCE [19] converts the events into representations. The model is pre-trained on Howto100M [20], which
consists of automatically constructed 100M clip-narration pairs. We expect the MIL-NCE to yield better event
representations than the TSN because this model is pre-trained on instructional vision-and-language resources.
Data preprocessing. As in [18], we truncated sequences longer than 100 for the clip and 20 for the sentence and set
the maximum length of the clip sequence to 12. Finally, we built the vocabulary based on words that occurred at least
three times. The resulting vocabulary contained 991 words.
Hyper-parameter settings. For both the encoder and decoder transformers, we set the hidden size to 768, thenumber of layers to two, and the number of attention heads to 12. We train the model using the optimization method
described in [7, 18]; we use the Adam optimizer [16] with an initial learning rate of 0.0001, 𝛽 1 = 0.9, and 𝛽 2 = 0.999.
The L2 weight decay is set to 0.01, and the learning rate warmup is over the first five epochs. We set the batch size to16, and continue training at most 50 epochs using early stopping with SODA:CIDEr-D. The early stopping is introduced
from MART, which stops the model training based on CIDEr-D.
16

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 3. Quantitative evaluation for the baselines and proposed methods. We evaluate them on the word-overlap metrics andcomputational complexity. The bold and underlined scores are the best and second-best among the comparative methods. V and VI
represents video only and video+ingredients, respectively.

MT
ECHR
SGR
PDVC (reported)
PDVC (reproduced)
PDVC (comparable)
Base (B)Base + Ingredients (BI)BI + Visual simulator (BIV)
BIV + Textual attention (BIVT)
OracleInput modality

FLOPs#parameters

Training time

Inference time

Video feature

V
V
V
V
V
V
V
VI
VI
VI
V

148.00G
362.72G
37.21G
78.65G
78.9G
78.9G
-

20.64M
48.10M
47.09M
47.32M
52.7M
52.7M
-

2.71h
8.98h
2.41h
3.67h
4.72h
4.79h
-

0.06s
0.09s
0.17s
0.17s
0.19s
0.18s
-

TSN
TSN
TSN
TSN
TSN
TSN
MIL-NCE
MIL-NCE
MIL-NCE
MIL-NCE
TSN

BLEU4
0.30
0.80
0.56
0.45
1.04
1.39
1.40
1.92
0.97

dvc_eval
METEOR CIDEr-D
3.18
6.10
3.82
4.72
22.71
5.80
21.47
4.42
15.63
6.03
24.98
7.18
31.07
7.27
32.67
8.04
37.24
7.68
36.30

METEOR
4.35
4.42
3.99
3.06
5.45
6.44
6.46
7.29
9.64

SODA
CIDEr-D
15.10
7.34
25.09
31.69
32.95
38.93
35.09

tIoU
27.80
24.24
33.23
35.10
34.13
35.06
71.16

Models. We test the proposed method by comparing it with four state-of-the-art dense video captioning models, as
described below:• Masked Transformer (MT) [43] is a transformer-based encoder-decoder DVC model that can be trained in an
end-to-end manner by using a differentiable mask.
• Event-centric Hierarchical Representation for DVC (ECHR) [38] is an event-oriented encoder-decoderarchitecture for DVC. The ECHR incorporates temporal and semantic relations into the output events forgenerating captions accurately.
• SGR [5] is a top-down DVC model consisting of four processes. The model (1) generates an overall paragraphfrom the input video, (2) grounds the sentences with events in the video, (3) refines captions based on the
grounded events, and (4) refines events, referring to the refined captions.
• PDVC [37] is the state-of-the-art DVC model. It detects 𝑁 events densely, re-ranks the top 𝐾 of them, andgenerates sentences for the re-ranked top 𝐾 events. Note that 𝐾 is the prediction target. This model is used in
our preliminary experiments described in Section 3.
Ablations. We examine the impact of the components of the proposed method through ablation studies on thefollowing variations:• Base model (B) is the model introduced in Section 4.
• B + Ingredient (BI) incorporates the ingredient encoder into the base model.
• BI + Visual simulator (BIV) incorporates the visual simulator into the BI model.
• BIV + Textual attention (BIVT) additionally incorporates the textual attention module into the BIV model.
6.1

Quantitative evaluation

6.1.1 Word-overlap evaluation. Table 3 demonstrates the results of the word-overlap evaluation on both dvc_eval and
SODA with BLEU, METEOR, and CIDEr-D. We observe that the base model consistently outperforms state-of-the-art
captioning models by a significant margin in both evaluations. In the ablation, the BIV model outperforms the BI model,
and the BIVT model further improves the BIV model. This indicates that both the dot-product visual simulator and the
textual attention module are effective for accurate recipe generation. We also note that the BI model achieves the best
performance on SODA:tIoU, but performs worse than BIV and BIVT on other word-overlap metrics. This indicatesthat the additional modules especially contribute to verbalizing the correct recipes, rather than selecting oracle eventsaccurately.
17

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

Table 4. Percentage of recipes that satisfy |𝑝 − 𝑞 | ≤ 𝜂, where 𝑝, 𝑞, 𝜂 represents the number of predicted events, ground-truth events,
and a threshold, respectively. In this experiment, we change 𝜂 from 0 to 3.

𝜂
PDVC
Model
B
BI
BIV
BIVT

0
14.4

1
40.0

2
63.0

3
76.4

18.6
18.8
20.5
19.7

52.1
51.6
51.8
51.8

71.7
73.5
74.2
73.5

83.4
87.3
86.2
87.5

6.1.2 Discussion on the computational complexity. In Table 3, we also demonstrate the computational complexity fromtwo perspectives: (1) FLOPs and the number of parameters (#parameters) and (2) training and inference speed.
(1) FLOPs and #parameters. To calculate #parameters and FLOPs, we use the thop 4 , which automatically computes
FLOPs by forwarding the input data. Note that both FLOPs and #parameters include the frozen part of the model(i.e., event encoder). Table 3 indicates that #parameters of PDVC are smaller than the proposed models, but its FLOPs
are bigger. We investigate the reason by analyzing layer-wise FLOPs and discover that the primary bottleneck is
the LSTM decoder, which cannot be computed in parallel and costs 𝑂 (𝑛 ∗ 𝑑 2 ) as discussed in [33]. To fill the gap of#parameters between PDVC and our methods, we increase the number of Transformer layers of PDVC (from 2 to10) and compare it with the proposed methods. We refer to it as “PDVC (comparable)” in Table 3. However, PDVC
(comparable) underperforms both the proposed methods and the original PDVC model across all metrics. This may be
primarily attributed to its excessive parameters, leading to overfitting of the training data.
(2) Training and inference speed. We compute the total training time and video-wise inference speed. We confirm
that both the total training time and inference speed of PDVC are much shorter than our methods. This occurs
because PDVC can predict events and sentences in parallel, but our methods estimate the current ones recurrently byremembering the previously predicted results, causing increased computational costs.
6.2

Discussion on the number of predicted events

In addition to the word-overlap metrics, we discuss the generated recipes from the perspective of the number ofpredicted events. Table 4 shows the percentage of recipes that satisfy |𝑝 − 𝑞| ≤ 𝜂, where 𝑝, 𝑞, 𝜂 represents the numberof predicted events, ground truth, and a threshold, respectively. In this experiment, we change 𝜂 from 0 to 3. This result
demonstrates that the proposed models consistently predict a more precise number of events than the PDVC. Fig. 9shows the histogram of the number of the predicted events. While the PDVC outputs the specific number of events(i.e., 5, 7, 10), the histogram of the proposed method draws a similar curve to that of the ground truth. In contrast, theablation study does not show a clear consistency in the number of predicted events. As discussed in Section 6.1, theadditional modules can improve the word-event-based metrics but do not affect the period of events.
6.3

Qualitative analysis

Fig. 10 shows the predicted events and generated recipes from the PDVC, B, and BIVT models, in comparison to that of
the ground truth. In terms of the predicted events, PDVC outputs highly overlapped events. The proposed method,on the other hand, predicts events in the correct order, with minimal overlap. This story-oriented sequential event
4 https://github.com/Lyken17/pytorch-OpCounter18

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

PDVC

180

B
BI
BIV
BIVT

Frequency

135

GT

90

45

0
3

4

5

6

7

8

9

10

11

>11

#Predicted events
Fig. 9. Histogram of the number of predicted events and ground truth.
Table 5. Loss ablation studies. IS, AS, IA, and AA represent ingredient selection, action selection, ingredient attention, and action
attention losses, respectively.
IS
(a)
(b)
(c)
(d)
(e)
(f)
(g)

AS

IA

AA

✓
✓
✓
✓
✓

✓
✓
✓
✓
✓

✓
✓

✓
✓

BLEU4
1.11
1.28
1.44
1.57
1.76
1.57
1.92

dvc_eval
METEOR CIDEr-D
6.37
26.54
7.31
31.59
7.32
32.19
7.45
33.56
7.79
36.84
7.70
34.58
8.04
37.24

METEOR
5.69
6.46
6.35
6.76
7.12
6.93
7.29

SODA
CIDEr-D
26.54
33.10
30.86
35.45
36.96
37.42
38.93

tIoU
33.15
33.78
33.35
34.56
34.91
34.53
35.06

prediction is an advantage of the proposed method. In terms of the generated recipes, the PDVC repeatedly generates
the same contents, ignoring the events (see (c) to (g)). The proposed methods suppress this problem. A comparisonof B and BIVT reveals that the BIVT can generate sentences that are grounded with events (e.g., “pork” is accurately
verbalized in (a) in BIVT, whereas “fat” is generated in (a) in B).
6.4

Discussion on the detailed model settings

Here, we discuss the detailed model settings from six perspectives: (1) loss ablation studies, (2) memory update strategies,(3) input modalities, (4) event encoders, (5) parameter sensitivity and the event candidates 𝑁 , and (6) trade-off between
the model’s performance, computational overhead, and model complexity. The results demonstrate that these parameters
are important to succeed in our task.
6.4.1 Loss ablation studies. Table 5 shows ablation studies on the loss function for the BIVT model. The experimentyields two insights. First, all the losses are essential for training the model. This is confirmed by removing all of them
19Conference acronym ’XX, June 03–05, 2018, Woodstock, NY…

Taichi Nishimura, et al.

…

…

…

…

Time

Ground Truth
(a)

(b)

(c)

(e)

(d)

(f)

(g)

PDVC
(a)

(b)

(c) (e)

(d)

(f)

(g)

Base (B)
(a)

(b)

(c)

(d)

(f) (e)

BIVT
(a)

(b)

Ground Truth

(e)

(d)

(c)

PDVC

(f)

Base (B)BIVT(a) Cut the pork into slice(a) Add oil and salt to a bow(a) Cut the fat off of the fa(b) Cover the pork in plastic
wrap and poun(b) Mix the chicken in the ourand mi(b) Season the meat with salt
and peppe(c) Sprinkle salt and pepper
on top of the mea(c) Fry the chicken in a pa(c) Pour buttermilk into a bowl
and add salt and peppe(d) Fry the chicken in a pa(d) Melt butter in the pa(f) Dip the pork in the egg
mixture and the bread crumb(b) Season the pork with salt
and peppe(c) Season the pork with salt
and pepper(d) Coat the veal in ou(e) Heat the oil in a pa(e) Mix eggs milk salt andpepper togethe(a) Cut the pork in half andremove the por(d) Heat some butter in a pa(f) Fry the chicken in a pa(e) Coat the meat in the our
mixtur(g) Fry the chicken in the pan(f) Place the veal in the pan(e) Coat the pork in the break
crumb(f) Fry the pork in a pan(g) Fry the pork in the pan
e

s

x

r

d

k

r

t

fl

n
n

r

r
fl
s

 

n

t

fl
n

n
  

l
n

s

Fig. 10. Examples of the generated recipes. We compare three models: PDVC, B, and BIVT with the ground truth.
Table 6. Comparison of memory update strategies: separate vs joint.

BLEU4
B
Separate
Joint
BIV
Separate
Joint
BIVT
Separate
Joint

dvc_eval
METEOR CIDEr-D

METEORSODA
CIDEr-D

tIoU

0.92
1.04

5.67
6.03

24.19
24.98

5.19
5.45

23.12
25.08

33.96
33.26

1.39
1.40

7.24
7.27

33.02
32.67

6.54
6.46

33.00
32.95

33.81
34.13

1.78
1.92

7.85
8.04

36.65
37.24

6.92
7.29

36.84
38.93

33.85
35.06

from the full model (compare (a) and (g)). Second, ingredient-based losses are more necessary than action-based ones.
Removing the ingredient selection loss has a more significant influence on the performance, compared to the action
selection loss (compare (b) with (d) and (c) with (d)). This tendency is the same as the relationship between ingredientattention and action attention losses.
6.4.2 Memory update strategies: separate or joint? Table 6 shows a comparison of the memory update strategies. While
the separate memory update does not mix the memories in the event and sentence transformers, the joint approach
proposed in Section 4 fuses these memories. The results demonstrate that the joint approach outperforms the separate
20

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 7. Comparison of input modalities: video only and multimodal versions.

BLEU4B
Video only
Multimodal
BIV
Video only
Multimodal
BIVT
Video only
Multimodal

dvc_eval
METEOR CIDEr-D

METEORSODA
CIDEr-D

tIoU

1.04
0.44

6.03
3.91

24.98
15.49

5.45
3.51

25.08
13.21

33.26
27.33

1.40
0.60

7.27
4.68

32.67
19.02

6.46
4.03

32.95
14.17

34.13
27.32

1.92
0.78

8.04
5.18

37.24
21.00

7.29
4.66

38.93
19.51

35.06
28.94

Table 8. Comparison of the model’s performance when varying the event encoders: TSN and MIL-NCE. Note that unlike TSN, which
is pre-trained on only vision resources, the MIL-NCE is pre-trained on instructional vision-and-language resource, Howto100M.

BLEU4
B
TSN
MIL-NCEBIV
TSN
MIL-NCEBIVT
TSN
MIL-NCE

dvc_eval
METEOR CIDEr-D

METEORSODA
CIDEr-D

tIoU

0.36
1.04

4.24
6.03

15.55
24.98

3.79
5.45

14.98
25.08

31.71
33.26

0.52
1.40

4.93
7.27

18.98
32.67

4.51
6.46

18.32
32.95

31.77
34.13

0.99
1.92

5.87
8.04

23.60
37.24

5.26
7.29

22.84
38.93

32.36
35.06

one on the base and BIVT settings, but on BIV, the separate approach performs the equivalent or better performance
than the joint one. Based on these observations, we confirm the effectiveness of the proposed memory mixing. However,there is room for further research to improve the memory mixing method that is effective for all ablation models.
6.4.3 Input modalities: video only or multimodal? MIL-NCE has two branches of encoders: video and text encoders.
In our experiments, we use only the video branch but can employ a text encoder by inputting the predicted eventswith generated captions of PDVC. We compare video only version and the multimodal version; in the multimodal
version, we convert event timestamps and sentences into vectors using MIL-NCE branches, simply concatenate them,and input them into the models. Table 7 shows a comparison of the input modalities, indicating that the video-only
inputs achieve much better than the multimodal inputs. This occurs because the sentences generated by PDVC aresemantically overlapped (see Fig. 10) and do not contribute to our recipe generation task.
6.4.4 Event encoders. Table 8 shows the performance difference when changing the event encoders, indicating that the
MIL-NCE proved significantly superior to the TSN in all of the settings. This occurs because the MIL-NCE is pre-trainedon the vision-and-language resource, Howto100M, which captures the fine-grained event-level semantics of cooking
procedures. We conclude that pre-training on an appropriate resource is essential to effective performance on our task.
6.4.5 Parameter sensitivity and the event candidates 𝑁 . The PDVC allows users to select the number of candidates
𝑁 when training the model. In the original study on the PDVC, 𝑁 is set to be 100 on the YouCook2 dataset. In thisexperiment, we change the parameter 𝑁 to be 25, 50, 100, and 200 to investigate the parameter sensitivity of the model.
Table 9 shows the performance change of the proposed method in different model settings: B, BIV, and BIVT. The results
demonstrate that 𝑁 = 25 consistently performs better than other settings on SODA, but is not always best on dvc_eval.
21Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

Table 9. Comparison of the model’s performance when varying the number of the event candidates 𝑁 .

BLEU4
B
N=25
N=50
N=100
N=200
BIV
N=25
N=50
N=100
N=200
BIVT
N=25
N=50
N=100
N=200

dvc_eval
METEOR CIDEr-D

METEORSODA
CIDEr-D

tIoU

1.27
0.98
1.04
0.93

6.49
6.42
6.03
6.16

27.84
27.12
24.98
25.93

6.19
5.89
5.45
5.52

29.34
26.34
25.08
26.13

35.26
33.79
33.26
33.93

1.71
1.51
1.40
1.23

7.50
7.52
7.27
6.81

34.18
33.37
32.67
29.51

7.02
6.86
6.46
5.95

36.39
34.24
32.95
28.86

36.13
35.60
34.13
33.15

1.87
1.81
1.92
1.78

7.95
7.99
8.04
7.66

36.12
36.48
37.24
35.21

7.51
7.36
7.29
6.77

39.06
38.49
38.93
35.64

36.74
36.38
35.06
33.48

Table 10. SODA score changes when changing the number of transformer layers in both event selector and sentence generator.

B
BIV
BIVT

SODA: METEOR
1
2
4
5.39 5.45 5.25
6.47 6.46 6.33
6.94 7.29 7.11

SODA:CIDEr-D
1
2
4
26.26 25.09 25.52
32.59 32.95 31.44
36.96 38.93 38.30

1
32.52
34.17
33.59

SODA:tIoU
2
4
33.23 32.06
34.13 34.87
35.06 34.90

For example, 𝑁 = 100 on the BIVT model achieves better than 𝑁 = 25 on dvc_eval. Therefore, we could not find the
optimal 𝑁 that achieves the consistently best performance on all of the ablation methods but discovered that the lower
𝑁 performs better, especially on the SODA scores. In addition, we observe that increasing 𝑁 extremely degrades themodel’s verbalization ability (e.g., 6.19 → 5.52 on SODA:METEOR by changing 𝑁 = 25 → 𝑁 = 200 on the base modelin Table 9). Although a higher 𝑁 makes the maximum tIoU larger (shown in Section 3), the ratio of events that are notoracle but highly overlapped with the ground truth increases. This prevents the model from selecting oracle events
precisely and causes it to overfit the training set.
6.4.6 Trade-off between the model’s performance, computational overhead, and model complexity. To discuss the trade-off
between the model’s performance and computational overhead, we demonstrate the SODA:CIDEr-D changes at eachepoch in Fig. 11. This indicates that 30 − 40 epochs are necessary for training the base model and 50 are necessary for
BIV and BIVT, concluding that 2.4 and 4.8 hours are necessary for achieving the best performance on B and BIV/BIVT,
respectively (see Table 3). Then, to discuss the trade-off between the model’s performance and model complexity,
we investigate the SODA score changes when changing the number of transformer layers 𝑙 = 1, 2, 4 in Table 10. The
reason to investigate 𝑙 is that transformer layers are dominant in terms of the number of parameters of the models andincreasing 𝑙 gains the model’s complexity and computational costs. The results demonstrate that (1) in general, 𝑙 = 2
generally performs the best on all of the settings and (2) 𝑙 = 1 works well on B and BIV while 𝑙 = 4 achieves better than𝑙 = 1 on BIVT. Therefore, we conclude that 𝑙 is an essential parameter to be tuned, but 𝑙 = 2 shows a good balance
between performance and model complexity.
22

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Fig. 11. SODA:CIDEr-D changes at each epoch. Note that training the base model is stopped at epoch 35 due to early stopping.

7

CONCLUSION

In this paper, we tackled recipe generation from unsegmented cooking videos, a task that requires agents to (1) extract
key events that are essential to dish completion and (2) generate sentences for the extracted events. We first analyzed thestate-of-the-art DVC models and set our goal to obtain correct recipes by selecting oracle events from the output eventsof the DVC model and re-generating sentences for them. To achieve this, we proposed a transformer-based multimodal
recurrent learning model, which consists of the event selector and sentence generator. The event selector selects oracle
events from the event candidate in the correct order, and the sentence generator outputs a recipe grounded in theevents. Both modules have memory vectors to remember the history of previous predictions to estimate the next step.
The proposed memory mixing approach efficiently combines them, effectively sharing the previous predictions between
the event selector and sentence generator. To generate more accurate recipes, we also proposed an extended model byintroducing two additional modules: dot-product visual simulator and textual attention module. In experiments, we
confirmed that the proposed methods achieve story awareness. The base model outperforms the state-of-the-art DVC
models and the extended model boosts the model’s performance. In addition, we showed that the proposed models
can select the correct number of events, as with the ground-truth events. The qualitative evaluation revealed that the
proposed approaches can select events in the correct order and generate recipes grounded in the video content. Finally,
we discussed the detailed experimental settings for optimal recipe generation.
ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Number JP21J20250, 21H04910, 20H04210, and JST-Mirai Program
Grant Number JPMJMI21G2.
23

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Taichi Nishimura, et al.

REFERENCES[1] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In
Proc. ACL Workshop IEEMMTS. 65–72.
[2] Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. 2018. Discourse-Aware Neural Rewards for Coherent
Text Generation. In Proc. NAACL-HLT. 173–184.
[3] Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018. Simulating action dynamics with neural process
networks. In Proc. ICLR.
[4] Khyathi Chandu, Eric Nyberg, and Alan W Black. 2019. Storyboarding of recipes: Grounded contextual generation. In Proc. ACL. 6040–6046.
[5] Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. 2021. Sketch, Ground, and Refine: Top-Down Dense Video Captioning. In Proc. CVPR.
234–243.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proc. CVPR.
248–255.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language
understanding. In Proc. NAACL. 4171–4186.
[8] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. 2016. DAPs: deep action proposals for action understanding. In
Proc. ECCV. 768–784.
[9] Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. SODA: Story Oriented Dense Video Captioning
Evaluation Framework. In Proc. ECCV. 517–531.
[10] Jun Harashima, Yuichiro Someya, and Yohei Kikuta. 2017. Cookpad image dataset: An image collection as infrastructure for food research. In Proc.
ACM SIGIR. 1229–1232.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proc. CVPR. 770–778.
[12] Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc.
ICML. 448–456.
[13] Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumble-softmax. In Proc. ICLR.
[14] Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. 2015. Mise en Place: unsupervised interpretation of instructional
recipes. In Proc. EMNLP. 982–992.
[15] Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proc. EMNLP. 329–339.
[16] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. ICLR.
[17] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-Captioning Events in Videos. In Proc. ICCV. 706–715.
[18] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara Berg, and Mohit Bansal. 2020. MART: memory-augmented recurrent transformer for coherent
video paragraph captioning. In Proc. ACL. 2603–2614.
[19] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual
representations from uncurated instructional videos. In Proc. CVPR. 9879–9889.
[20] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. HowTo100M: learning a text-video
embedding by watching hundred million narrated video clips. In Proc. ICCV. 2630–2640.
[21] Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proc. ACL-IJCNLP.
1003–1011.
[22] Taichi Nishimura, Atsushi Hashimoto, and Shinsuke Mori. 2019. Procedural text generation from a photo sequence. In Proc. INLG. 409–414.
[23] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori. 2021. State-aware Video Procedural Captioning. In
Proc. ACMMM. 1766–1774.
[24] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, Yoko Yamakata, and Shinsuke Mori. 2020. Structure-aware procedural
text generation from an image sequence. IEEE Access 9 (2020), 2125–2141.
[25] Liangming Pan, Jingjing Chen, Jianlong Wu, Shaoteng Liu, Chong-Wah Ngo, Min-Yen Kan, Yu-Gang Jiang, and Tat-Seng Chua. 2020. Multi-modal
cooking workflow construction for food recipes. In Proc. ACMMM. 1132–1141.
[26] Dim P. Papadopoulos, Enrique Mora, Nadiia Chepurko, Kuan Wei Huang, Ferda Ofli, and Antonio Torralba. 2022. Learning Program Representations
for Food Images and Cooking Recipes. In Proc. CVPR. 16559–16569.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL.
311–318.
[28] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: global vectors for word representation. In Proc. EMNLP. 1532–1543.
[29] Amaia Salvador, Michal Drozdzal, Xavier Giro-i-Nieto, and Adriana Romero. 2019. Inverse Cooking: recipe generation from food images. In Proc.
CVPR. 10453–10462.
[30] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning cross-modal embeddings
for cooking recipes and food images. In Proc. CVPR. 3020–3028.
[31] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou. 2019. Dense procedure captioning in narrated instructional
videos. In Proc. ACL. 6382–6391.
24

Recipe Generation from Unsegmented Cooking Videos

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY[32] Botian Shi, Lei Ji, Zhendong Niu, Nan Duan, Ming Zhou, and Xilin Chen. 2020. Learning semantic concepts and temporal alignment for narrated
video procedural captioning. In Proc. ACMMM. 4355–4363.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is
all you need. In Proc. NeurIPS. 5998–6008.
[34] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: consensus-based image description evaluation. In Proc. CVPR.
4566–4575.
[35] Hao Wang, Guosheng Lin, Steven C. H. Hoi, and Chunyan Miao. 2020. Structure-Aware Generation Network for Recipe Generation from Images. In
Proc. ECCV. 359–374.
[36] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2019. Temporal Segment Networks for Action
Recognition in Videos. IEEE Trans. on Pattern Analysis and Machine Intelligence (2019), 2740–2755.
[37] Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. 2021. End-to-End Dense Video Captioning with Parallel Decoding.
In Proc. ICCV. 6847–6857.
[38] Teng Wang, Huicheng Zheng, Mingjing Yu, Qian Tian, and Haifeng Hu. 2021. Event-Centric Hierarchical Representation for Dense Video Captioning.
IEEE Trans. on Circuits and Systems for Video Technology (2021), 1890–1900.
[39] Jianlong Wu, Liangming Pan, Jingjing Chen, and Yu-Gang Jiang. 2022. Ingredient-enriched Recipe Generation from Cooking Videos. In Proc. ICMR.
249–257.
[40] Mengyang Zhang, Guohui Tian, Ying Zhang, and Peng Duan. 2021. Reinforcement Learning for Logic Recipe Generation: Bridging Gaps from
Images to Plans. IEEE Trans. on Multimedia (2021).
[41] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. 2019. Grounded video description. In Proc. CVPR. 6578–6587.
[42] Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proc. AAAI.
7590–7598.
[43] Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong. 2018. End-to-end dense video captioning with masked transformer.
In Proc. CVPR. 8739–8748.

25