Springer Nature 2021 LATEX template

State-aware Video Procedural Captioning
Taichi Nishimura1*, Atsushi Hashimoto2 , Yoshitaka
Ushiku2 , Hirotaka Kameko3 and Shinsuke Mori3
1* Graduate School of Informatics, Kyoto University,

Yoshidahonmachi, Kyoto-shi, 606–8501, Kyoto, Japan.
2 OMRON SINIC X Corporation, 5–24–5, Bunkyo-ku, 113–8656,

Tokyo, Japan.
3 Academic Center for Computing and Media Studies, Kyoto

University, Yoshidahonmachi, Kyoto-shi, 606–8501, Kyoto, Japan.

*Corresponding author(s). E-mail(s): taichitary@gmail.com;
Contributing authors: atsushi.hashimoto@sinicx.com;
yoshitaka.ushiku@sinicx.com; kameko@i.kyoto-u.ac.jp;
forest@i.kyoto-u.ac.jp;
Abstract
Video procedural captioning (VPC), which generates procedural text
from instructional videos, is an essential task for scene understanding
and real-world applications. The main challenge of VPC is to describe
how to manipulate materials accurately. This paper focuses on this challenge by designing a new VPC task, generating a procedural text fromthe clip sequence of an instructional video and material set. In thistask, the state of materials is sequentially changed by manipulations,yielding their state-aware visual representations (e.g., eggs are transformed into cracked, stirred, then fried forms). The essential difficulty
is to convert such visual representations into textual representations;that is, a model should track the material states after manipulations tobetter associate the cross-modal relations. To achieve this, we propose
a novel VPC method, which modifies an existing textual simulator for
tracking material states as a visual simulator and incorporates it intoa video captioning model. Our experimental results show the effectiveness of the proposed method, which outperforms state-of-the-art video
captioning models. We further analyze the learned embedding of materials to demonstrate that the simulators capture their state transition.
Keywords: Instructional video, Procedural text, Simulator

1Springer Nature 2021 LATEX template2

State-aware Video Procedural Captioning

1 Introduction
In recent years, there has been significant progress in vision and language
research that targets procedural text and instructional video [1–5]. Amongthe various tasks in such research, it is important to describe the content of
the video using natural language for both scene understanding and real-world
applications. For example, machines can help people learn new skills by providing a quick overview of instructional videos. To this end, video procedural
captioning [6, 7] (VPC), which is the task of generating procedural text from
instructional videos, has been proposed. VPC requires a model (1) to segment
important clips from a video, (2) enumerate materials used in the clips, and(3) describe how to manipulate them as a sentence for each clip. (1) and (2)have been independently studied as the task of temporal clip segmentation[1, 8] and object recognition [9, 10], respectively. In this paper, we focus on (3)by designing a new VPC task and the method of generating a procedural text
from a clip sequence pre-segmented in an instructional video and material set(Fig. 1).
Different from standard video captioning [11–15], our task requires a model
to describe detailed material manipulations from visual observations. The
manipulations change the state of materials sequentially, yielding their stateaware visual representations (e.g., in step 2 in Fig. 1, “eggs” are transformed
into “cracked” then “stirred”). The essential difficulty is to convert such visual
representations into textual representations; that is, a model should trackmaterial states after manipulations to generate a procedural text. For example, when generating step 3 in Fig. 1, the models should be aware that theyellow liquid in the bowl and white liquid in the pan correspond to the eggs
and butter manipulated in steps 1 and 2, respectively. Existing video captioning models cannot track material states; thus, they miss materials, hallucinate
them (e.g., materials that are not in the clip), and lack details (e.g., say “cookingredients”), thereby degrading the captioning performance. The above challenge is not specific to the cooking domain but is universal across multiple
domains, such as wet-lab experiments and furniture assembly.
To address this challenge, this paper aims to generate procedural texts bymodeling the state transition of materials from visual observations. The same
motivation, tracking materials against their state transition, is shared by the
research community in natural language understanding (NLU). To address
this, some NLU studies have proposed a simulator to reason the state transitionof materials in a procedural text [16, 17]. These simulators read one sentence
in a procedural text, predict executed actions and involved materials, andrecurrently update the state of the materials to simulate their state transition.
Inspired by these ideas, we propose a novel VPC method, which modifies an existing NLU simulator as a visual simulator and incorporates it into
an encoder-decoder architecture. Fig. 1 illustrates the idea of the proposed
method. Given a clip sequence and material set, the proposed method reasons
the state transition of materials and generates procedural text accurately. Weemphasize that this work is the first attempt to integrate the NLU simulator

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

3

into a video captioning model. In addition, based on the intuition that thestate transition of materials is consistently traceable from the generated procedural texts, we attach a novel textual re-simulator, facilitating the model togenerate a procedural text even more accurately.
In our experiments, we test the proposed method in the cooking domain
because of its large variety of actions and materials (= ingredients). We compare it with two state-of-the-art video captioning models on four evaluations:word-overlap evaluation, ingredient prediction, retrieval evaluation, and qualitative analysis, with thorough ablation. Our experimental results show thatthe proposed method outperforms the state-of-the-art video captioning models. In addition, ablation studies show the effectiveness of integrating visualand textual simulators into the model. Finally, we analyze the learned embedding of materials to demonstrate that the simulators effectively capture thestate transition of materials.
Key contributions and extensions from our previous work. The
preliminary version of our work has been published at ACM Multimedia 2021
[18]. The key contribution of [18] is (1) modifying an existing NLU simulator as
a visual simulator and incorporating it into an encoder-decoder architecture,and (2) incorporating textual re-simulator to facilitate the model to generatea procedural text more accurately. Furthermore, this work provides a detailed
explanation of the proposed method and additional experiments as following:• Modification of the material encoder for learning material set of permutation
invariant representations of materials. Although the previous work regarded
input materials as an ordered set, it is more reasonable to generate identicalsentences against changes in the order of materials.

Materiaset
Clip sequence

step 1

action(s) : statebuttereggs

cheeseState transition

Procedural text

add thebutter into a pan

addedstep 2

crackestirredcrack theggs and stir

step 3adde
stirredadd the egg mixture
and cheese and stir

d
 d
 l

 

e

 

Fig. 1 Concept of the proposed method. Given a clip sequence and material list, the proposed method generates a procedural text by reasoning the state transition of materials at
each step.

Springer Nature 2021 LATEX template
4

State-aware Video Procedural Captioning

Table 1 A comparison of video captioning approaches. Our work is the first attempt togenerate a procedural text more accurately than other general video captioning methods
without using transcription.
MethodMFT [14]AdvInf [19]Masked Transformer [20]Transformer-XL [13]MART [13]DPC [6]
TVPC [7]
Ours

Input
Video
Video
Video
Video
Video
Video + Transcription
Video + Transcription
Video + Materials

DomainGeneralGeneralGeneralGeneralGeneralProcedural
Procedural
Procedural

Base model
LSTM
LSTM
Transformer
Transformer
Transformer
LSTM
Transformer
Transformer• Increase of the dataset size by annotating new examples. We added 126
newly annotated recipes to our previous study.
• Additional quantitative and qualitative evaluations, such as sentence-level
word-overlap evaluation and additional examples of generated recipes.
• Quantitative evaluation of the visual simulators to investigate how accurately they can acquire the state-awareness of materials.
• Additional experiments on the full prediction setting, where the material
set is not given but is predicted from the video clips in advance. Theseexperiments provide insight into whether the proposed methods work wellwith the predicted ingredients.

2 Related WorkWe describe the novelty of the proposed method in line with other works
on video captioning in Section 2.1. In Section 2.2, we discuss simulators forprocedural text understanding because they are the main components of the
proposed method.

2.1 Video captioning
Video captioning is an attractive field for both computer vision and natural
language processing communities. Table 1 shows a comparison of recent video
captioning approaches. The task settings of video captioning vary according
to the nature of the input video (e.g., one short clip, clip sequence, or long
untrimmed video). Our VPC task targets a clip sequence and thus belongs tothe video paragraph description [13, 14, 19], which aims to generate multiple
sentences for a given clip sequence pre-segmented in the video.
In the video paragraph description, recently, Transformer-based [21] models have been featured because of their ability to capture long-term information
in a clip sequence, compared with LSTM-based models [14, 19]. MART [13] isa transformer-based models that achieves state-of-the-art performance in thevideo paragraph description. The key contribution of MART is the introduction of a gated recurrent memory module, which makes it easy for the model
to summarize important information in the previous step.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

5

Table 2 A comparison of methods for procedural text understanding. This work is thefirst attempt to apply a reasoning-based simulator to the video captioning task.
MethodAction graph [22]Recipe flow graph [23]
SIMMR [24]MM-ReS [25]vSIMMR [26]
NPN [16]
RPN [17]
Ours

Type
Graph-based parser
Graph-based parser
Graph-based parser
Graph-based parser
Graph-based parser
Reasoning-based simulator
Reasoning-based simulator
Reasoning-based simulator

Modality
Text
Text
Text
Image+Text
Image+Text
Text
Image+Text
Video+Text

Task
Parsing
Parsing
Parsing
Parsing
Parsing
Text generation
QA
Video captioning

Different from standard video captioning, VPC requires a model to generatedetailed material manipulations for input clips. To this end, previous VPC
works [6, 7] target narrated videos and utilize transcription as a cue to generatea procedural text by fusing video and transcription features. Although theseapproaches work well for narrated videos, transcription is not always available
for all instructional videos. Speaking during manipulations or adding narration
to videos requires significant effort. Thus, the proposed VPC task extends theprevious VPC works to describe manipulations even from non-narrated videos;
our task allows a model to refer to a human-created material set1 . Our task
requires a model to track material states that are changed by manipulations
to generate a procedural text accurately. To this end, we propose a novel
VPC method, which modifies an NLU simulator as a visual simulator andincorporates it into a transformer-based encoder-decoder.

2.2 Procedural text understanding
In NLU, procedural texts are a popular target for understanding, and recipes
have been featured in this field. Many researchers have proposed methods tounderstand recipes, which are roughly divided into two categories: (1) a graphbased parser and (2) a reasoning-based simulator. Table 2 shows a comparisonof these approaches.
A graph-based parser aims to train a model to map recipes into graphor tree structures. Kiddon et al. [22] and Maeta et al. [23] proposed a model
to estimate a graph structure called the action graph and recipe flow graph,
respectively. Jermsurawong and Nizar [24] proposed a parser with the SIMMR
dataset, which provides a tree structure for understanding ingredient merging operations. Then, Pan et al. [25] and Nishimura et al. [26] provided
new multimodal tree structure datasets and parsers, which can be regarded
as an extension of the text-only version to a vision-and-language version.
These parsers focus on enabling machines to interpret recipes as structuralrepresentations, rather than capturing the state transition of ingredients inrecipes.
A reasoning-based simulator aims to model the state transition of ingredients in recipes by updating the ingredient states at each step. Gupta and1

We perform an experiments on the full prediction setting in Section 4.7.

Springer Nature 2021 LATEX template
6

State-aware Video Procedural Captioning
Visual simulator (sec 3.3) Rv

Encoder (sec 3.2) E

Textual re-simulator (sec 3.5) Rt

Decoder (sec 3.4) D

Materials
e10

Material
encoderbuttereggs

e20

cheesee30

DecoderaddedcrackedstirredMaterial vectors

Clip sequence
Clip
sequence
encoder

1
2

step 1

h1

step 2h2

s1

step 2 crack…

step 2s2

step 1

u2

step 3

add…

step 3s3

step 2

crackedstirredstep 3added
stirredMateriallabel

step 1step 1

add…

step 2

crack…
stir…

step 3

add…

Constructdistant supervision

Actionlabel

add…

Generated

State-awareu ⊂step vectors n

h3

step 3

3

step 1

u1

u3

added
stirredTextualencoder

Clip vectors

added

Sentence vectors

Training
and inferenceTraining
only
Losscalculation

Ground truth

Fig. 2 An overview of the proposed method. To track material states in a clip sequence, we
incorporate the visual simulator Rv into the transformer-based encoder-decoder architecture(E and D). In addition, based on our intuition that the state transition of materials is
traceable from the generated procedural texts, we attach the textual re-simulator Rt to themodel.

Simulator
Materia
label

1

Clip
attention

vector

Actio
label

e2n−1

0
1
0
1

Recurrent
attention

Actio
selection loss

e3n−1

wp

n-th clip vector
hn

e

e1n−1

×

e2n−1

×

e3n−1

∑

×
×

crack

fcrack

add

fadd

boil

fboil

stir

fstir

State-awar
step vector

un ∈ ℝ3×d

Clip vector

hn

ēn

Action vectorf¯nMaterial vectorēn(3) Updater(1) Action selector
Action embedding×

MLP

Selectemateria
vector×

Next step

×
 
 n

 

e
 l
l

d

d

l
  
 

an

1

(n − 1)-th material
e1n−1(2) Material selector

Materiaselection loss

1

Input vectors

Updated material vectors

Selecteactio
vector∑f¯nBilinearlnAction-awar
proposa
vector

Updat
gate

e1̂

Training
and inference

e2̂

Loss
calculatione3̂
Next step

Forwarto next stepe  Fig. 3 An overview of the (visual) simulator. The simulator recurrently reasons the statetransition of the materials at each step. Specifically, it predicts executed actions and involved
materials in (1) the action and (2) material selector and then updates the state of materialsin (3) the updater. The updated materials are forwarded to the next step. The textual resimulator has the same modules.

Durrett [27] proposed a structured simulator that imposes constraints on thestate transition of ingredients (e.g., stir-fried potatoes are not cut in later
steps). Amac et al. [17] proposed a method to answer multi-modal question
answering (QA) tasks by reasoning the state transition of ingredients using
a relation reasoning network [28]. This paper builds upon the neural process
network (NPN) [16], which learns to estimate the state transition of ingredients by predicting executed actions and involved ingredients. Owing to theability to capture the state transition of ingredients, these simulators reported
competitive results in QA tasks [17, 29]. Our work adopts a reasoning-based
simulator. We modify an NPN as the visual simulator and incorporate it into
the transformer-based encoder-decoder architecture to enable the model tocapture the state transition of ingredients.

3 Proposed Method
In this section, we present the proposed method. After describing an overview
in Section 3.1, we explain the components of the proposed method from Section

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning
Material encoder

butterGloV(freeze)mean

7

Clip sequence encoder

ML+ ReLUe10

1st Transformer

h1

[CLS]

Parametesharedeggs

GloV(freeze)mean

ML+ ReLUe20

1st Transformer

2nTransformer

h2

[CLS]

Parametesharedcheese

d

P
 e

GloV
(freeze)mean

ML+ ReLUe30

1st Transformer

h3

[CLS]

r

Fig. 4 An overview of the material and clip sequence encoders.

3.2 to Section 3.5. Finally, in Section 3.6 we explain the loss functions to train
the model.

3.1 Overview
Fig. 2 shows an overview of the proposed method. Given a clip sequence
V = (v 1 , . . . , v n , . . . , v N ) and a material set G = (g 1 , . . . , g m , . . . , g M ), ourgoal is to output a procedural text Y = (y 1 , . . . , y n , . . . , y N ) by recurrently
generating sentences from corresponding clips. To generate a procedural text
accurately, it is essential for models to track material states in the clip sequence(e.g., eggs are transformed into cracked, stirred, then fried forms).
Inspired by recent advances in NLU, we achieve this by modifying theexisting reasoning-based NLU simulator NPN as the visual simulator Rv , andincorporate it into the transformer-based encoder-decoder architecture (E andD). Specifically, given clip H and material encoded vectors E 0 , the visual
simulator reasons the state transition of materials and outputs state-aware
step vectors as U = Rv (H, E 0 ). Then, the decoder D outputs a procedural
text conditioned on U . We observe that this simple integration of the visual
simulator is effective for the model to generate a procedural text accurately.
In addition, based on our assumption that the state transition of materials should be consistently traceable from the generated procedural text, we
attach a novel textual re-simulator Rt , encouraging the model to generate aprocedural text even more accurately. Specifically, the textual re-simulator Rt
reasons the state transition of materials from the generated procedural text as
Rt (S, E 0 ), where S represents the encoded sentence vectors. Note that the textual re-simulator is only used during the training phase and detached during
the inference phase.

Springer Nature 2021 LATEX template
8

State-aware Video Procedural Captioning

3.2 Encoder
The input has two components: a material set G and a clip sequence V. Thus,
we develop a suitable encoder for each component. Fig. 4 shows an overview
of the encoders.
Material encoder. To encode a material set, we input them to concatenated neural networks of pre-trained GloVe [30]2 word embedding andmulti-layer perceptrons (MLPs) with the ReLU activation function. Multiword materials (e.g., parmesan cheese) are represented by the average embedding vector of words. Then, we obtain the initial material vectors E 0 =(e01 , . . . , e0m , . . . , e0M ). Compared with our previous work, the key difference isthe learning of permutation-invariant representations of materials instead of
the position-sensitive implementation in the previous work. Although the previous work regarded input materials as ordered sets, it is more reasonable togenerate identical sentences against changes in the order of materials.
Clip sequence encoder. A clip sequence V is hierarchical because theclip sequence contains multiple clips, and each clip is composed of sequential frames. Thus, to encode a clip sequence effectively, we design a two-stagetransformer suitable to encode a sequence of sequences. First, the former
transformer encodes each clip into a feature vector by extracting the vector, which corresponds to the [CLS] token as in [13, 31, 32]. Then the lattertransformer is trained over the sequence to obtain the step-aware clip vectors
H = (h1 , . . . , hn , . . . , hN ) in the clip sequence.

3.3 Visual simulator
Based on the encoded vectors of the clip sequence and material set (H, E 0 ),the visual simulator, shown in Fig. 3, reasons the state transition of materials
at each step. Specifically, at the n-th step, given the n-th clip hn and (n−1)-thmaterial set E n−1 , the visual simulator predicts executed actions and involved
materials in (1) the action and (2) material selector and then updates the stateof materials in (3) the updater. After n-th reasoning, it outputs a state-aware
step vector un ∈ R3×d , which concatenates the n-th clip hn , selected actionf̄ n and material vectors ēn (d represents the dimension of these vectors).
The visual simulator recurrently repeats the above process until processing
the end element of the clip sequence. This simulation process is the same as
NPN except for the input modality; we replace the textual sentence and entity
vectors for NLU with visual clip H and material vectors E 0 for our task. Thus,
we only provide a high-level overview in this subsection, and further details of
the simulator are provided in Appendix A.
(1) Action selector. Given a clip vector hn , the action selector outputs
the selected action vector f̄ n by choosing actions executed in the clip from thepredefined action embedding F . For example, in Fig. 3, the actions “crack” and“stir” are executed in the clip, thus both f crack and f stir should be selected.
2
We employ pre-trained 300D word embedding, which can be downloaded from http://nlp.
stanford.edu/data/glove.6B.zip

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

9

To consider multiple actions, the action selector computes a soft selection wp
as an action probability for each action in F . Then it outputs the selectedaction vector f¯n as a weighted sum of the action embedding F and action
probability wp .
(2) Material selector. Based on the action probability wp and clip vector
hn , the material selector outputs a selected material vector ēn by choosing
materials that are involved in the clip from the material set E n−1 . For example,
in Fig. 3, the raw “cheese” and manipulated “eggs” and “butter” should beselected. To consider such a combination of raw and manipulated material
selection, the material selector has two attention modules: (1) clip attentionand (2) recurrent attention. While the clip attention selects materials fromthe current clip vector hn , the recurrent attention selects materials based on
both the current and previous clips. Using these modules, the material selector
computes a soft selection an as the material probability for each material in
the material set E n−1 . Then it outputs the selected material vector ēn as aweighted sum of the material vectors E n−1 and material probability an .
(3) Updater. Based on the selected actions and materials, the updater
represents the state transition of materials by computing a new material vector êm . To this end, it first calculates an action-aware proposal vector ln ofmaterials with a bilinear transformation of selected action and material vectors (f¯n , ēn ). Then, based on the material probability an , it computes the
new material vector êm by interpolating the action-aware proposal vector ln
and the current material vector en−1
m . The m-th new material vector êm is
assigned to E nm , which is forwarded to the next (n + 1)-th step.

3.4 Decoder
Fig. 5 shows an overview of the decoder. The decoder D outputs a procedural text by recurrently generating sentences from the n-th output vector
un of the visual simulator. As our decoder D, we use the transformer, which
achieves state-of-the-art performance in video captioning tasks [6, 7, 20]. Ourtask allows the model to refer to a material set, thus to encourage the decoder
to generate materials, we incorporate the copy mechanism [33] into the decoder
D. When generating the k-th word in the n-th sentence, given the updated
materials enm ∈ E n , the copy mechanism first calculates the attention probability β mn,k using the bilinear dot product of the vectors of the decoder output
on,k and each material enm ∈ E n as:
exp {(on,k )T W c enm }βmn,k = Pn ,Ti exp {(on,k ) W c ei }(1)where W c represents a bilinear map.
Then it calculates the copying gate gn,k (0 ≤ gn,k ≤ 1), which makes asoft choice between selecting a material from the material set and generating

Springer Nature 2021 LATEX template
10State-aware Video Procedural Captioningeggs
Copmechanism

Word probabilit
Eq. (3)gn,kUpdatedmaterial vectors

Copying gat
Eq. (2)

m
βn,k

e1̂
e2̂

Attentio
Eq. (1)

e3̂

voc
Pn,k
(w)

Softmax

Linearon,kAdd & norm
State-awar
step vector

Feed forward
Multi-hea
attentionhn
f¯n

Feed forwardēn

Add & norm
Masked multi-hea
attentionxN
Positionaencoding

Word embedding+
Add the … (shifted right)

y

n
 

l
e

d
e
y

d

Fig. 5 An overview of the decoder.

a word from the vocabulary:gn,k = σ(W g [on,k ;Xnβmn,k em ] + bg ),(2)

mwhere [·], σ(·), W g , and bg represent the concatenation function, sigmoid function, linear map, and bias, respectively. Based on gn,k , the final predicted word
probability Pn,k (w) is computed as the weighted sum of the copy probability
and generation probability as follows:




X
1
voc
Pn,k (w) = (1 − gn,k ) Pn,k(w) + gn,k 
β in,k  ,∥g m ∥ i:w ∈g
i

(3)

m

voc
where Pn,k(w) and ∥g m ∥ represent the probability of the n-th sentence’s k-th
word w in the vocabulary and the number of words of in the m-th material,
respectively3 .
3To consider multiple words of materials, we divide the probability by the number of words.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

11

3.5 Textual re-simulator
Because the state transition of materials is identical in the visual and textual
worlds, it should be consistently traceable from the generated procedural texts.
Based on this assumption, we add a novel textual re-simulator Rt , encouraging
the model to generate a procedural text even more accurately.
The textual re-simulator consists of two sub-modules: (1) a textual encoderand (2) a textual simulator. The textual encoder converts a generated procedural text into step-aware sentence vectors S = (s1 , . . . , sn , . . . , sN ). First,
it applies the straight-through version of Gumbel softmax resampling [34] tosample a procedural text, preserving the differentiable chain. The sampled procedural text is further converted into feature vectors by computing the average
vector of the word embedding at each step. Note that the word embedding isshared between the decoder and textual encoder. They are then converted into
step-aware sentence vectors S using a biLSTM encoder. Finally, based on S,
the textual simulator, another NPN described in Section 3.3, reasons the statetransition of materials again as Rt (S, E 0 ).

3.6 Loss functions
To train the model, we compute three types of losses: (1) sentence generation
loss Lsent , (2) visual simulation loss Lv sim , and (3) textual re-simulation loss
Lt sim .
(1) Sentence generation loss Lsent . This loss aims to train the decoder,and is computed as the summed negative log-likelihood for all input/output
pairs {(V, G), Y} in the training set.
(2) Visual simulation loss Lv sim . This loss aims to train the visual
simulator, and consists of two losses: (1) material selection loss and (2) action
selection loss. These losses are computed as the summed binary cross-entropy
loss based on whether the materials/actions are involved/executed in the clip.
To avoid costly human annotations, we compute the loss from distant supervision [35] following the original NPN training method [16]. For the material
selection loss, labels are obtained whether or not each step contains materials
in the material set; for the action selection loss, labels are obtained whether or
not each step contains actions in the 384 actions defined by [16]. For example,
from the sentence “crack the eggs and stir,” “eggs” is extracted as a material
label, and “crack” and “stir” are extracted as an action label. As the action
selection loss, the simple binary cross-entropy does not work in our preliminary
experiment because the ratio of positive to negative actions is imbalanced; a
few actions are positive and most of the actions are negative. Thus, as theaction selection loss, we use asymmetric loss [36], which is a weighted binary
cross-entropy loss, to defuse the imbalanced problem.
(3) Textual re-simulation loss Lt sim . To train the textual re-simulator,
we also compute the above visual simulation loss from the sampled proceduraltexts.

Springer Nature 2021 LATEX template
12State-aware Video Procedural Captioning

Table 3 YouCook2-ingredient+ dataset statistics.
#recipes#steps / #recipes#ingredients / #recipes

train
1,331
7.8
10.4

val
228
7.6
10.3

test
229
7.7
10.4

Total loss. Consequently, to train the entire model in an end-to-end
manner, the total loss is computed as
Ltotal = Lsent + Lv sim + λLt sim ,(4)where λ is a hyper-parameter used for weighting the importance of the textual
re-simulation loss.

4 Experiments
We test the proposed method in the cooking domain because procedural texts(= recipes) have a large variety of actions and materials (= ingredients). We
compare it with two state-of-the-art video captioning models on four evaluations: word-overlap evaluation (Section 4.2), ingredient prediction (Section4.3), retrieval evaluation (Section 4.4), and qualitative analysis (Section 4.5)
with thorough ablation. We also visualize the learned embedding of ingredients, demonstrating that the visual simulator effectively reasons the statetransition of ingredients (Section 4.6).

4.1 Experimental settings
Dataset. We use the YouCook2 dataset [1], which consists of 2,000 cooking
videos from 89 recipe categories. All of the videos have 3–16 clips with astart/end timestamp annotated by humans, and each clip is also annotatedwith an English sentence. Because ingredients are not annotated in the original
dataset, in our previous work, we prepared the YouCook2-ingredient dataset
by annotating ingredients for recipes with 1,662 valid videos that are available
online. This work increases the dataset size by obtaining the missing videos
through YouCook2 author and annotating ingredients with these additional
videos. As a result, we obtained the annotated 1,788 videos as the YouCook2ingredient+ dataset and divided them as follows: 1,331 for training, 228 forvalidation, and 229 for testing4 . Table 3 shows the statistics of the YouCook2ingredient+ dataset.
Data preprocessing. As clip features, we use concatenated features ofappearance and optical flow provided by [1]. With respect to the appearance,
2,048D feature vectors extracted from the “Flatten-673” layer in ResNet-200[37] are used, and for the optical flow, 1,024D feature vectors extracted fromthe “global pool” layer in BN-Inception [38] are used. As in [13], we truncatedsequences longer than 100 for the clip and 20 for the sentence and set the4

We will release annotated ingredients and the dataset split.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

13

maximum length of the clip sequence to 12. Finally, we built the vocabulary
based on words that occurred three times or more, and the resulting vocabularycontained 991 words.
Hyper-parameter settings. For both the encoder and decoder transformers, we set the hidden size to 768, the number of layers to two, and thenumber of attention heads to 12. We train the model following the optimization
method described in [13, 31]; we use the Adam optimizer [39] with an initial
learning rate of 0.0001, β1 = 0.9, and β2 = 0.999. The L2 weight decay is set
to 0.01, and the learning rate warmup is over the first five epochs. We set thebatch size to 16, and continue training at most 50 epochs using early stoppingwith CIDEr-D. We tune λ with four different values λ ∈ {0.25, 0.5, 0.75, 1.0}
and set λ to 0.5 in our experiments (for details, see Section 4.2).
Models. We test the proposed method by comparing it with two state-ofthe-art video captioning models, as described below.
• Transformer-XL [40] is a powerful transformer-based language model thatwas originally proposed for capturing long-term dependency in natural language. As in [13], we adapt it for our task; the model directly uses all of theprevious hidden states to generate a current sentence.
• MART [13] is a transformer-based video captioning model that achieves
state-of-the-art performance on the video paragraph description. This model
generates a sentence with a gated recurrent memory module, which does notpass all of the previous hidden states, but effectively summarize important
information in the previous step.
Note that these models originally have no ingredient set in the inputs andcopy mechanism in the decoder. Thus for a fair comparison, we prepare for
additional baselines, baseline + ingredient (-I) models, which incorporate thematerial encoder (Section 3.2) and the copy mechanism into the models (forimplementation details of the models, see Appendix C).
Ablations. To reveal the impact of the components in the proposed
method, we conduct ablation studies on the following variations.
• Video only (V) encodes a clip sequence with the clip sequence encoder,then generates a procedural text.
• V + Ingredient (VI) incorporates the material encoder and copy
mechanism in the model.
• VI + Visual simulator (VIV) incorporates the visual simulator into the
VI model.
• VIV + Textual re-simulator (VIVT) additionally incorporates the
textual re-simulator into the VIV model.

4.2 Word-overlap evaluation
Scores. To evaluate the captioning performance of the proposed method, we
compute commonly used word-overlap metrics, such as BLEU [41], ROUGEL [42], METEOR [43], and CIDEr-D [44] in the test set. We conduct two

Springer Nature 2021 LATEX template
14

State-aware Video Procedural Captioning

Table 4 Paragraph- and sentence-level word-overlap evaluation for the baseline and the
proposed models with ablation studies. The scores in bold are the best among thecomparative models. “I” indicates whether the model uses ingredient information or not.
B=BLEU, M=METEOR, C=CIDEr-D, RL=ROUGE-L.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

✓

B1
39.0
37.7
37.9
42.3

B2
22.0
22.5
21.7
26.2

Paragraph-level
B3
B4
M
12.1
6.7
15.2
13.4
8.2
15.4
12.4
7.6
15.0
16.1
9.9
17.6

C
22.7
35.4
29.1
48.2

RL
30.9
34.2
32.3
36.2

B1
26.5
29.6
28.0
31.3

B2
13.5
17.0
14.5
18.5

✓
✓
✓

43.2
49.1
49.4
49.4

24.5
29.5
30.1
30.9

14.0
17.6
18.0
18.3

32.4
63.3
66.1
67.1

31.9
35.2
36.8
37.1

28.0
31.9
33.7
33.5

13.9
18.0
19.3
19.4

I

✓

8.1
10.5
11.0
11.3

16.6
20.3
21.0
21.1

Sentence-level
B3
B4
M
5.3
1.5
10.6
8.2
3.5
12.6
6.1
2.4
11.1
9.4
4.4
13.7
6.0
9.3
9.7
10.1

1.9
3.8
4.4
4.9

11.4
13.9
15.2
15.2

C
57.3
69.6
62.1
81.0

RL
27.8
31.3
29.4
32.9

60.7
81.7
93.0
96.7

28.3
31.3
33.3
33.7

Table 5 Change in paragraph-level word-overlap evaluation with controlled λ.
λ = 0 (VIV)
λ = 0.25
λ = 0.5
λ = 0.75
λ = 1.0

B1
49.4
49.6
49.4
50.3
49.1

B2
30.1
30.4
30.9
30.4
30.0

B3
18.0
18.3
18.3
18.0
18.0

B4
11.0
11.3
11.3
10.8
11.0

M
21.0
21.1
21.1
21.2
21.0

C
66.1
65.5
67.1
65.7
64.4

RL
36.8
36.6
37.1
36.3
36.7

types of word-overlap evaluation: recipe (paragraph-level) [13, 14, 19] and step
(sentence)-level evaluation [7].
Results. Table 4 shows the results of the word-overlap evaluation. We
observe that the proposed method consistently outperforms the state-of-the-art
captioning models by a significant margin in both paragraph- and sentencelevel evaluation. Our ablation studies show that the VIV model performs better
than the VI model, and the VIVT model further improves the VIV model.
This indicates that both the visual simulator and the textual re-simulator are
effective for generating a recipe accurately.
Performance change of controlling the hyper-parameter λ. Table 5shows the results by varying the λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0}. Note that λ = 0
is equivalent to the VIV model, which does not have a textual re-simulator.
The results indicate that (1) the VIVT model achieves a performance thatis better or comparable to the VIV model for any λ values, and (2) λ = 0.5
performs the best among the three metrics (BLEU2-4, ROUGE-L, and CIDErD) and obtains competitive results in BLEU1 and METEOR. Thus, we set
λ = 0.5 in our experiments.

4.3 Ingredient prediction
To evaluate whether the models use correct ingredients at each step without
missing and hallucinating them, we design the ingredient prediction, whichmeasures the step-level overlap of ingredients between generated and groundtruth recipes. To this end, we first construct an ingredient dictionary from all
unique ingredients in the YouCook2-ingredient+ dataset. Then, at each step,
we extract ingredients that are exact-matched between generated recipes andthe ingredient dictionary. The same process is performed to extract ingredientsin the ground-truth recipes. Finally, based on the extracted ingredient sets, we
compute the micro- recall, precision, and F1 scores, respectively.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

Ingredients flour, eggs, baking soda, salt, pepper, water, shrimp, batter, breadcrumbs, oilstep 1step 215

step 3

Clip sequence
MART +
Ingredients(MART-I)

add flour salt pepper to a bowl (✗ baking
soda, egg)

add flour salt pepper and milk to the bowl and add flour and shrimp in the bowl (✗ batter,mix (✗ water)
breadcrumbs)

V + Ingredients(VI)mix flour salt pepper and breadcrumbs (✗baking soda, eggs)

mix flour salt pepper and breadcrumbs (✗water)

coat the shrimp in the batter (✗ batter,breadcrumbs)

VI + Visual
simulator (VIV)

mix flour baking soda salt pepper andbreadcrumbs (✗ eggs)

mix the eggs with eggs (✗ water)

coat the shrimp in the batter (✗ breadcrumbs)+ VIV + Textual
re-simulator(VIVT)

mix flour baking soda salt and pepper
together (✗ eggs)

mix the eggs with the watercoat the shrimp in the batter (✗ breadcrumbs)

Ground truth

add flour eggs baking soda salt and pepper
to the bowl and stir

add cold water to the bowl and stir

cover the shrimp in the batter andbreadcrumbs(a)step 4step 5

Clip sequence
MART + Ingredients(MART-I)

fry the shrimp in oilremove the shrimp from the oil

fry the shrimp in a pan with oilremove the shrimp from the oilheat oil in a pan and fry the shrimp

remove the shrimp from the oil

VIV + Textual re-simulator
heat oil in a pan and fry the shrimp(VIVT)

remove the shrimp from the oil

V + Ingredients(VI)
VI + Visual simulator(VIV)

Ground truth

place the shrimp into a pan of hot oilremove the shrimp from the pan

Ingredients soy sauce, brown sugar, water, garlic, green onions, sesame oil, ribs
step 1step 2step 3

Clip sequence
MART +
Ingredients(MART-I)

add soy sauce sesame oil soy sauce sesame oil andsesame oil to a pan
cut the ribs into small pieces(✗ brown sugar, garlic, green onions, water)

add chopped garlic sesame oil brown sugar
and red pepper flakes to a pan and mix (✗green onions, water)
add garlic soy sauce sugar garlic green
VI + Visual
simulator (VIV) onions and garlic to a bowl and mix(✗ sesame oil, water)+ VIV + Textual add garlic sesame oil sugar soy sauce garlicre-simulator
and water to a blender(VIVT)(✗ green onions)
V + Ingredients(VI)

Ground truth(b)

put the marinade in the ribs and soy sauce

add soy sauce sesame oil soy sauce sesame oiland sesame oil to a bowl (✗ marinade)add sesame oil garlic and the bowl (✗marinade)cover the ribs with plastic wrap and place in apour the marinade over the ribs
paper towelplace the ribs in a bag

combine soy sauce brown sugar water garlicplace the ribs in the baggreen onions and sesame oil

put the ribs into a bag (✗ marinade)pour the marinade into the bag

step 4step 5

Clip sequence
MART + Ingredients(MART-I)

cook the ribs in the pan

cook the ribs on a grill

V + Ingredients(VI)flip the pancakes overplace the ribs on the grill

flip the ribs overcook the ribs on a grill

VI + Visual simulator(VIV)

VIV + Textual re-simulator
grill the ribs
(VIVT)
Ground truth

oil the grate of the grill

place the ribs on the grill and cook them until it is golden brow onboth sides
cook the ribs on the grill

Fig. 6 Examples of generated recipes. Here, we compare four models, MART-I (baseline),
VI, VIV, and VIVT with the ground truth. Green bold and red words represent semanticallycorrect and incorrect ingredients, respectively. Words in parentheses indicate missing ingredients, which should be included in the sentence. Note that parallel words in a sentence are
not separated from the commas in the YouCook2 dataset (see step 1 in (a) in the groundtruth).

Springer Nature 2021 LATEX template
16

State-aware Video Procedural Captioning

Table 6 Results of ingredient prediction.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

Recall
12.6
17.3
11.3
21.9

Precision
19.1
30.4
20.2
34.0

F1
15.2
22.0
14.5
26.7

13.5
24.3
28.9
29.7

19.8
33.1
43.2
43.2

16.0
28.1
34.7
35.2

Table 7 Results of retrieval evaluation. ↓ indicates that lower is better.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)
Ground truth

MedR (↓)
162.5
139
138.5
79

R@1
2.0
1.6
1.9
3.1

R@5
6.3
8.1
7.1
11.9

R@10
11.0
13.6
11.4
19.4

134
65
49
49
7

2.2
4.6
4.6
5.2
19.0

8.4
14.5
17.3
17.3
45.2

13.6
21.2
25.1
25.2
59.7

Results. Table 6 shows the results of the ingredient prediction. This resultshows that the proposed method outperforms the state-of-the-art video captioning models. In our ablation, we observe the same tendency of performancechange to the word-overlap evaluation. We note that the VIV model performs
much better than the VI model by 6.6% in F1, indicating that not only the copy
mechanism but also the visual simulator are important for generating ingredients correctly. We also notice that the VIVT model improves the VIV modelby 0.5% in F1, demonstrating the effectiveness of the textual re-simulator.

4.4 Retrieval evaluation
To evaluate whether the generated procedural texts are sufficiently concrete
to describe the input clips, we design a step-level zero-shot sentence-to-clip
retrieval evaluation. As a retrieval model, we employ the MIL-NCE model[5] pre-trained on the HowTo100M dataset [4], achieving the state-of-the-art
performance. In this task, given a generated step-level sentence as a query,the MIL-NCE model embeds it and computes the cosine similarly as a score
between the query vector and all the 1,768 clip vectors from the test set. Then,
we sort scores with clips in descending order and calculate the median rank
(MedR) and recall rate at the top K (R@K). The median rank represents themedian ranking of retrieved corresponding clips, hence a lower is better; incontrast, R@K represents the percentage of all the step-level sentence queries
where the corresponding clip is retrieved in the top K, hence a higher is better.
Results. Table 7 shows the results of the retrieval evaluation. We observethat the proposed method significantly outperforms the state-of-the-art video
captioning models. In MedR, the VIVT model achieves 49, which is marginallySpringer Nature 2021 LATEX template
State-aware Video Procedural Captioning

17

Table 8 Quantitative evaluation of the visual simulators.

VIV
VIVT
Distant supervision

Recall
22.6
21.4
30.5

Ingredient
Precision
77.5
75.9
97.8

F1
35.0
33.4
46.5

Recall
49.5
49.1
94.7

ActionPrecision
18.7
19.9
57.1

F1
27.1
28.3
71.2

lower than that of Transformer-XL-I 139 and MART-I 79. This indicates thatthe proposed method generates a more concrete recipe based on clips than
the state-of-the-art video captioning models. In our ablation, the VIV model
dramatically improves the VI model, indicating that the visual simulator isessential for generating concrete recipes. In addition, the VIVT model showsa steady improvement from the VIV model, indicating the effectiveness of the
textual re-simulator.

4.5 Qualitative analysis
Fig. 6 shows two examples of the generated recipes.
Insights. For the VPC task, it is important to generate the correct ingredients that are manipulated in a clip. MART-I fails to generate ingredients
correctly; the model tends to miss and hallucinate ingredients (e.g., “flour”
and “milk” in step 2 (a) and “garlic” in step 1 (b)). A similar tendency can beobserved in the VI model, indicating that these models superfluously generate
ingredients listed in the ingredient set.
The VIV model suppresses these problems (e.g., “baking soda” and “batter” in steps 1 and 3 in (a) and “marinade” in step 3 in (b)). In addition,owing to the textual re-simulator, the VIVT model can generate ingredientsthat are missed or misrecognized in the VIV model (e.g., “water” in steps 1
and 2 in (a) and in step 1 in (b)).
Limitations. Although the proposed method generates recipes more accurately than the baseline models, we still found some differences from the groundtruth. For example, in step 2 in (a), the VIV and VIVT models refer to “eggs,”
superfluously, but only “water” is added to the ground truth. In addition, in
step 3 in (b), the VIV and VIVT models generate “ribs” but it is not used.
To solve these problems, we believe that incorporating fine-grained ingredient recognition modules [45] would help the model to generate a recipe more
precisely.

4.6 Discussion of the learned embedding
4.6.1 Quantitative evaluation of visual simulators
We evaluate the visual simulators by measuring the performance of the action
and material selectors. These models are trained from distant supervision, as
described in Section 3.6, which does not always agree with human judgment.
Thus, we manually annotated the action and material labels for 50 recipesthat were randomly sampled from the test set.

Springer Nature 2021 LATEX template
18

State-aware Video Procedural Captioning

Action annotation
Add cold water
to the bowl and mix

Ingredient annotation
our

eggs

water

battershrimp

Step 1

Add cold water
to the bowl and mix

add, mix
actio
label
fl

n

fl

fl

oil

Step 1
Add our eggs to the bowl
Used: ( our, eggs)
Step 2
Add water to the bowl
Used: (step1, water)

Step 2

Step 3coat shrimp in the batter
Used: (step2, shrimp, batter)

Step 3

Step 4fry shrimp in oil
Used: (step3, oil)

Step 4

Step 5

Step 5
Remove shrimp from oil
Used: (step4)

  Fig. 7 An annotation example of the visual simulator.

Fig. 7 shows an annotation example. For actions, we manually extracted
action words from sentences at each step (e.g., “add” and “mix” were extracted
in Fig. 7). For ingredients, step sentences do not explicitly include all of theingredients manipulated in the real world because they mention only the difference from the previous steps. For example, in step 2 in Fig. 7, a mixture of“flour” and “eggs” is used, but it is not mentioned in the sentence. To address
this, we annotated the ingredient tree structure [24, 26], where each step node
indicates a mixture of ingredients. The tree structure allows us to determine
which ingredients are used at each step by back-tracking its leaf nodes.
Results. We compute micro-recall, precision, and F1 as evaluation metrics
for the visual simulator. Table 8 shows that VIV and VIVT achieve competitive results; while VIV performs better in terms of ingredient selection, VIVT
performs better with respect to action selection. We also note that even distant supervision performs only F1=46.5% in the ingredient selection. This
happens because recipes mention only the difference from the previous steps;“flour” and “eggs” are manipulated in step 2 in Fig. 7, but are not mentioned.
Therefore, distant supervision is not perfect for simulating human material
manipulation in the real world. However, it effectively guides the model topredict incremental ingredients and is adequate for improving the captioning
performance of the VPC task.

4.6.2 Visualization of the learned embedding
To investigate how the visual simulator represents the state transition of ingredients, we visualize the ingredient embedding by projecting it to 2D space
using t-SNE [46]. Fig. 8 shows the projected learned embedding of ingredients obtained by the VIVT model. Note that only raw (red) and updated(blue)5 ingredients are shown in the figure. This result shows that the raw and
5

The attention weight in the material selector was higher than 0.5.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

19

Table 9 Paragraph- and sentence-level word-overlap evaluation on the full prediction
setting.
Video only (V)w/ Predict ingredients
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)
w/ Ground-truth ingredients (Table 4)
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

I

B1
43.2

B2
24.5

B3
14.0

B4
8.1

M
16.6

C
32.4

RL
31.9

B1
28.0

B2
13.9

B3
6.0

B4
1.9

M
11.4

C
60.7

RL
28.3

✓
✓
✓

42.5
42.5
42.7

24.3
24.3
24.0

13.9
13.9
13.3

8.1
8.1
7.7

16.5
16.7
16.7

28.4
33.9
31.6

32.3
32.5
31.9

28.5
28.9
28.3

14.6
14.6
14.0

5.9
6.2
5.5

2.2
2.3
2.3

11.8
11.9
11.7

62.1
63.8
60.8

28.7
28.8
28.3

✓
✓
✓

49.1
49.4
49.4

29.5
30.1
30.9

17.6
18.0
18.3

10.5
11.0
11.3

20.3
21.0
21.1

63.3
66.1
67.1

35.2
36.8
37.1

31.9
33.7
33.5

18.0
19.3
19.4

9.3
9.7
10.1

3.8
4.4
4.9

13.9
15.2
15.2

81.7
93.0
96.7

31.3
33.3
33.7

Clip

Predicte
ingredient

Recip
category

milk

eggs
benedict
croquemonsieur

milk
milk

milk

Clip

Predicte
ingredient

Recip
categorynaan

tomatoestomatoes fattoushetomatoestomatosoup

tomatoesbeanburritodFig. 8 Learned embedding of ingredients obtained by the VIVT model. Note that only raw
and updated (the attention weight in the material selector is higher than 0.5) ingredients aretransformed by t-SNE [46]. Red and blue colors represent the raw and updated ingredients,
respectively.

updated points are located on the outer and inner sides of the distribution,
respectively6 .
We also investigate the ingredients’ trajectory with the retrieved top-2
nearest ingredient vectors from updated ingredient vectors (see the zoomed
parts of the figure). The retrieved ingredients indicate their state-awareness;that is, ingredients with similar states are embedded into the same cluster inthe vector space regardless of the difference in their recipe categories defined by[1]. For example, the near vectors of updated “milk” with the “add” state arealso updated “milk” with “add”-like states (e.g., mix and pour). “tomatoes”
also have the same tendency.

Springer Nature 2021 LATEX template
20

State-aware Video Procedural Captioning

Ingredient(a)canadian bacon(b)lettuce

(c)

beef

(d)

chickenUpdated ingredient
cut
tuna
ad
tomatoefrchickecutunRaingredienttuna
tomatoes

Updated ingredien(nearest vector)
cut canadia
bacon
added
lettucechickenfried beef

tuna

adde
chicken (fail)rice

mixed draine
spaghetti(e)drainedspaghetti

mix
rice

(f)

cut
dogs

fry
chickenchickenfried cut
dogs

(g)

cu
potatoesfrychickenchickenmixed
potatoes (fail)

y
t

d

w

 
t
 d
a

 

n

s

d
 n

t

Fig. 9 Arithmetics using the learned embedding of ingredients. Examples (a) to (d) and(e) to (g) represent the first-order (raw-to-updated) and second-order (updated-to-updated)transformations, respectively. (d) and (g) show the failure cases for each transformation.

4.6.3 Semantic vector arithmetic
To demonstrate the state-awareness of learned embedding, we attempt to apply
simple arithmetic operations as performed in the literature [47–49]. In the context of our task, the state transition of ingredients is expected to be computed
as v(cut potatoes) = v(potatoes) + v(cut tomatoes) − v(tomatoes), where v
represents the map in the embedding space.
Fig. 9 shows seven examples of the arithmetic operations. From (a) to (d),first-order transformations (raw-to-updated) are described, and from (e) to(g), second-order transformations (updated-to-updated) are described. We cansee that the learned embedding simulates the state transition of ingredients by
specific actions in both transformations. For example, in (a) and (e), “canadian bacon” and “drained spaghetti” are converted into “cut canadian bacon”
and “mixed drained spaghetti,” respectively. However, we observe some failurecases, where (d) and (g), “raw chicken” is transformed into “added chicken” via“cut” and “cut potatoes” into “mixed potatoes” via “fry.” In these examples,
ingredients are consistent before and after, but executed actions are different
because of the failure in action selection. We believe that noisy action labeling causes this failure, and a sophisticated action selection would ease thisproblem.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

21

Table 10 Results of ingredient prediction on the full prediction setting.
Video only (V)w/ Predict ingredients
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)
w/ Ground-truth ingredients (Table 6)
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

Recall13.5

Precision
19.8

F1
16.0

14.8
16.2
15.8

21.5
22.5
21.9

17.5
18.8
18.3

24.3
28.9
29.7

33.1
43.2
43.2

28.1
34.7
35.2

Table 11 Results of retrieval evaluation on the full prediction setting.
Video only (V)w/ Predict ingredients
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)
w/ Ground-truth ingredients
V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

MedR (↓)
134

R@1
2.2

R@5
8.4

R@10
13.6

138.5
133.5
141

2.1
1.6
2.0

7.8
7.2
7.0

13.1
12.1
12.6

65
49
49

4.6
4.6
5.2

14.5
17.3
17.3

21.2
25.1
25.2

Table 12 Performance of an ingredient decoder on the full prediction setting. Note that
we compute the micro-recall, precision, and F1 on the multi-label classification setting.
VI
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

Recall
45.2
46.0
45.3

Precision
27.2
27.8
27.3

F1
34.0
34.7
34.1

4.7 Experiments on the full prediction settingThis work discusses the proposed method under the condition that the input
has ground-truth ingredients. Then, a natural question arises: is the proposed
method effective for VPC using the predicted ingredients? To answer this question, we conduct experiments on the full prediction setting, where the material
set is not given, but is predicted from the video clips in advance. To achievethis, we add an ingredient decoder of the multi-label classifier and train theentire model as multi-task learning (for details, see Appendix D).
Results. We conduct three types of evaluations: word-overlap evaluation,
ingredient prediction, and retrieval evaluation. Table 9, 10, and 11 shows theirresults. Compared with the video-only (V) model, the performance of VI, VIV,
and VIVT models is competitive in word-overlap evaluation and ingredientprediction, and is worse with respect to retrieval evaluation. Compared withthe models with ground-truth ingredients, we observe a performance drop by asignificant margin. This performance degradation likely occurs because of theaccumulated errors of the ingredient decoder and visual simulators; Table 12
indicates that the performance of the ingredient decoder is around F1=34% to35% on average. Therefore, we conclude that the proposed method performs
6
The raw and updated ingredients correspond to an embedding E 0 by the material encoder andan embedding E n updated from E 0 by the visual simulator, respectively.

Springer Nature 2021 LATEX template
22

State-aware Video Procedural Captioning

well with the input of ground-truth ingredients, rather than the predicted
ingredients. This is another limitation of our work.

5 Conclusion
In this paper, we proposed a new VPC task for generating a procedural text
from a clip sequence and material list. To generate a procedural text accurately, it is essential for models to track material states in a clip sequence. To
achieve this, we proposed a novel VPC method, which modifies the existing
simulator as a visual simulator and incorporates it into the encoder-decoderarchitecture. Our experimental results with thorough ablation demonstrate theeffectiveness of the proposed method, which outperforms the state-of-the-art
video captioning models. The learned embedding of materials demonstrates
that the simulator effectively captures their state transition.
Future work. We have two future directions: applying the proposed
method to harder settings and extending the scope of domains. Currently, ourmethod performs well on edited and segmented videos on the web, but it isunclear whether it works with unsegmented web videos and unsegmented anduntrimmed videos, such as egocentric videos [50]. These settings are harderthan our experimental setting. Towards industrial use, we need to develop amethod that accepts these inputs. In addition, to confirm the generalizability
across domains, we also need to extend the method to other domains, suchas the biochemical domain [51]. Our approach utilizes a large-scale dataset,
which is not always available on the web in other domains. Thus, we need totransfer procedural knowledge learned from cooking tasks to others.

Declarations
Funding and/or Conflicts of interests/Competing interests. This
work was supported by JSPS KAKENHI Grant Number JP21J20250 and
JP20H04210, and partially supported by JP21H04910, JP17H06100, JSTMirai Program Grant Number JPMJMI21G2, and JST ACT-I Grant Number
JPMJPR17U5. All of them are research grants from the Japanese government.
Disclosure of potential conflicts of interest. All authors state that nofinancial/non-financial support has been received from any organization that
may have an interest in this work.

Data Availablity Statement
The datasets generated during and/or analysed during the current study areavailable in our repository7 : https://github.com/misogil0116/svpc pp

7
Currently, the repository is a private mode. After our manuscript has been accepted, I will
release the code and dataset.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

Appendix A

23

Details of simulator

In this section, we describe the details of the visual simulator for the reproducibility of the proposed method. Given the encoded vectors of the clip
sequence and material list (H, E 0 ), the visual simulator, shown in Figure 3 inthe main paper, recurrently reasons the state transition of materials at eachstep. Specifically, at the n-th step, given the n-th clip hn and (n−1)-th material
list E n−1 , the visual simulator predicts executed actions and involved materials in (1) the action and (2) material selector and it then updates the stateof materials in (3) the updater. After n-th reasoning, it outputs a state-aware
step vector un ∈ R3×d , which concatenates the n-th clip hn , selected actionf̄ n and material vectors ēn (d represents the dimension of these vectors). The
visual simulator recurrently repeats the above process until processing the end
element of the clip sequence. For clarity, we explain the simulation process ofthe visual simulator at the n-th step.
Action selector. Given a clip vector hn , the action selector outputs theselected action vector f̄ n by choosing actions executed in the clip from predefined action embedding F . For example, in Figure 3 in the main paper, theactions “crack” and “stir” are executed in the clip, thus both f crack and f stirshould be selected. To consider multiple actions, the action selector computesa soft selection wp as action probability for each action in F . Then it outputs
the selected action vector f¯n as a weighted sum of the action embedding F
and action probability wp :wp = MLP(hn )wpw̄p = P j
j wp

(A1)
(A2)

f̄ n = w̄Tp F ,(A3)

where MLP(·) represents two-layer MLPs with the sigmoid function and wp ∈R∥F ∥ is the attention distribution over ∥F ∥ possible actions8 .
Material selector. Based on the action probability wp and clip vector
hn , the material selector outputs the selected material vector ēn by choosing materials involved in the clip from the material list E n−1 . For example,
in Figure 3 in the main paper, the raw “cheese” and manipulated “eggs” and“butter” should be selected. To consider such a combination of raw and manipulated material selection, the material selector has two attention modules: (1)clip attention and (2) recurrent attention.
(1) The clip attention chooses relevant materials from the clip vector hn
and action probability wp :8ĥn = ReLU(W 1 hn + b1 )(A4)T
dm = σ((en−1m ) W 2 [ĥn ; w p ])(A5)

wjp represents j-th value of wp ; thus, Eq.(6) indicates normalization of wp

Springer Nature 2021 LATEX template
24

State-aware Video Procedural Captioning

where W 1 and W 2 are linear and bilinear mapping, b1 and b2 are biases, anden−1and dm represent the m-th material vector and its attention weight.
m(2) Recurrent attention selects materials based on information from boththe current and previous clips. Using the result of clip attention, it computes asoft selection an as a material probability for each material in the material list:c = softmax(W 3 ĥn + b3 )(A6)

anm = c1 dm + c2 an−1
+ c3 0m(A7)

iswhere W 3 is a linear mapping, c ∈ R3 is the choice distribution, an−1mthe attention weight of the previous clip for each material, anm is the final
distribution for each material, and 0 is a vector of zeros (providing the option
not to select any materials). Finally, using the calculated attention weights,
the selected material vector ēn is computed as the normalized weighted sumof the selected materials.
anαnm = P m n
j aj
X
ēn =
αnm en−1
m .

(A8)
(A9)

m

Updater. Based on the selected actions and materials, the updater represents the state transition of materials by computing a new material vectorêm . To this end, it first calculates an action-aware proposal vector ln of materials with a bilinear transformation of the selected action and material vectors(f¯n , ēn ):ln = ReLU(f̄ n W 4 ēn + b4 ),(A10)

where W 4 is a bilinear mapping.
Then, based on the material probability an , it computes the new material
vector êm by interpolating the action-aware proposal vector ln and current
material vector en−1
m :
m n−1
êm = amn ln + (1 − an )em .

(A11)

The new m-th material vector êm is assigned to E nm , which is forwarded to
the next (n + 1)-th step.

Appendix B

Detailed Annotation Process

We additionally annotated ingredients with the rest of the recipes (126 recipes)
to the YouCook2-ingredient dataset, and built the YouCook2-ingredient+dataset.
We increased the dataset size by obtaining the missing videos through
YouCook2 author and annotating ingredients with these additional videos bySpringer Nature 2021 LATEX template
State-aware Video Procedural Captioning

Recipe pane

25

Video pane

Pre-estimated ingredients

Ingredient annotation pane
Fig. B1 A screen of our browser-based web annotation tool. Annotators write ingredientsthat appear in the recipes. To ease annotation, we preliminarily estimated ingredients using
the NER method [52] pre-trained on the English recipe flow graph corpus [53], and we set
the default values of inputs.

hiring one annotator to use the web tool shown in Fig. B1. This annotation tool
presents a recipe, corresponding video, and text boxes for writing ingredients.
In this paper, ingredients are defined as raw materials that are necessary tocomplete the dish. For example, “tomato” and “cucumber” should be written
as ingredients, although “salad” should not be written because it represents a
mixture of ingredients.
To annotate ingredients easily, “jump” buttons, which allow annotators to
see a clip corresponding to a step, are implemented based on the start/end
timestamp from the original YouCook2 dataset. Moreover, to encourage annotators to write ingredients easily, this tool displays estimated ingredients using
the named entity recognition (NER) model, flair9 [52] pre-trained on the
English recipe flow graph corpus (E-rFG corpus) [53]10 . If the estimated words
are not appropriate for the ingredients, the ingredients can be deleted or
rewritten.

Springer Nature 2021 LATEX template
26

State-aware Video Procedural Captioning

vocabulary

eggs

copy?
ok

MART
Transformer-XL
[CLS]

[SEP]

[BOS]

crack

the

Material encoder

eggs

crack the ..
(shifted right)
milk

herbs butter

Fig. C2 An overview of our baseline +ingredients (-I) implementation. These models also
incorporate the material encoder described in Section 3.2 and copy mechanism into thebaselines.

Appendix C

Baseline implementation details

As our comparative models, we employed two state-of-the-art transformerbased video captioning models: Transformer-XL [40] and MART [13]. Thesemodels originally have no ingredient set in their inputs and copy mechanismin their decoder; thus for a fair comparison, we prepare for additional baseline+ingredient (-I) models, which incorporate the material encoder (Section 3.2)
and the copy mechanism into the baselines.
These models are based on the transformer that encodes sequential inputs
and decodes a sentence by attending all of the elements in the input sequence.
Thus, to fit this characteristic, we concatenate the encoded ingredient andvideo vectors, and input them to the model, as shown in Fig. C2. When decoding, based on the output of the decoder ok and ingredient vectors E 0 , the copy
mechanism calculates the copying gate to make a soft choice between selecting
an ingredient from the ingredient set or generating a word from the vocabulary.

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

27

Ingredien
label

_ingr

h1
Cli
sequenc
encoder

h2
h3

+

P

p

t

t
 e

Probability

SampledIngredien
decoder

max

h̄

ML
Sigmoid

top k

eggs
cheeseDownstrea
Process

butterm

Fig. D3 An overview of how to integrate ingredient decoder into the model.

Appendix D

Implementation and Training
Details on full prediction settings

Here, we discuss the implementation and training details of the full prediction
settings, where the material set is not given, but is predicted from the video
clips in advance. To address this, as described in Section 4.7, we added an
ingredient decoder of the multi-label classifier and trained the entire model as
multi-task learning.
Fig. D3 shows how to integrate the ingredient decoder into the model. The
ingredient decoder consists of a two-layered MLP with a sigmoid function,and converts ĥ of a max-pooled vector of clip vectors into a probability vector
7∈ Rq of materials, where q indicates the number of unique ingredients appearing more than three times in the training set (we obtained q = 668 in theexperiment). During training, we compute the ingredient decoder loss Lingr ,which is an asymmetric loss [36] on the multi-label classification settings, andadd it to the total loss defined in Eq (4). Note that we adopt teacher-forcing[54] to stabilize the training; while the models learn using the ground-truth
ingredients for the downstream process in the training phase, they generate arecipe based on predicted ingredients at the inference phase (we sample the top
k = 15 ingredients from the probability). Another modification of the model isto remove the copy mechanism because we find that it degrades the captioning
performance with this setting.

References[1] Zhou, L., Xu, C., Corso, J.J.: Towards automatic learning of procedures
9

https://github.com/flairNLP/flair
In the tag definitions of the E-rFG corpus, we display food entities as the estimated ingredients. These entities cannot be directly used for our dataset because the definition of food slightly
differs from the ingredient definition in this paper (for example, “it” or “salad” are recognized asfood in the E-rFG corpus). Therefore, we asked annotators to delete or rewrite ingredients if they
were not appropriate.
10

Springer Nature 2021 LATEX template
28

State-aware Video Procedural Captioning
from web instructional videos. In: Proc. AAAI, pp. 7590–7598 (2018)[2] Alayrac, J.-B., Bojanowski, P., Agrawal, N., Sivic, J., Laptev, I., LacosteJulien, S.: Unsupervised learning from narrated instruction videos. In:
Proc. CVPR, pp. 4575–4583 (2016)[3] Alayrac, J.-B., Sivic, J., Laptev, I., Lacoste-Julien, S.: Joint discovery ofobject states and manipulation actions. In: Proc. ICCV, pp. 2127–2136(2017)[4] Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., Sivic,
J.: HowTo100M: learning a text-video embedding by watching hundred
million narrated video clips. In: Proc. ICCV, pp. 2630–2640 (2019)[5] Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., Zisserman, A.: End-to-end learning of visual representations from uncurated
instructional videos. In: Proc. CVPR, pp. 9879–9889 (2020)[6] Shi, B., Ji, L., Liang, Y., Duan, N., Chen, P., Niu, Z., Zhou, M.: Dense
procedure captioning in narrated instructional videos. In: Proc. ACL, pp.
6382–6391 (2019)[7] Shi, B., Ji, L., Niu, Z., Duan, N., Zhou, M., Chen, X.: Learning semantic concepts and temporal alignment for narrated video procedural
captioning. In: Proc. ACMMM, pp. 4355–4363 (2020)[8] Escorcia, V., Heilbron, F.C., Niebles, J.C., Ghanem, B.: DAPs: deep
action proposals for action understanding. In: Proc. ECCV, pp. 768–784(2016)[9] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time
object detection with region proposal networks. In: Proc. NeurIPS, pp.
91–99 (2015)[10] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko,
S.: End-to-end object detection with transformers. In: Proc. ECCV, pp.
213–229 (2020)[11] Donahue, J., Hendricks, L.A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional
networks for visual recognition and description. In: Proc. CVPR, pp.
2625–2634 (2015)[12] Tan, G., Liu, D., Wang, M., Zha, Z.-J.: Learning to discretely compose
reasoning module networks for video captioning. In: Proc. IJCAI, pp.
745–752 (2020)

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

29[13] Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T., Bansal, M.: Mart:
memory-augmented recurrent transformer for coherent video paragraphcaptioning. In: Proc. ACL, pp. 2603–2614 (2020)[14] Xiong, Y., Dai, B., Lin, D.: Move forward and tell: a progressive generatorof video descriptions. In: Proc. ECCV, pp. 489–505 (2018)[15] Zhou, L., Kalantidis, Y., Chen, X., Corso, J.J., Rohrbach, M.: Groundedvideo description. In: Proc. CVPR, pp. 6578–6587 (2019)[16] Bosselut, A., Levy, O., Holtzman, A., Ennis, C., Fox, D., Choi, Y.: Simulating action dynamics with neural process networks. In: Proc. ICLR
(2018)[17] Amac, M.S., Yagcioglu, S., Erdem, A., Erdem, E.: Procedural reasoning
networks for understanding multimodal procedures. In: Proc. CoNLL, pp.
441–451 (2019)[18] Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., Mori, S.: Stateaware video procedural captioning. In: Proc. ACMMM (2021)[19] Park, J.S., Rohrbach, M., Darrell, T., Rohrbach, A.: Adversarial inference for multi-sentence video description. In: Proc. CVPR, pp. 6598–6608(2019)[20] Zhou, L., Zhou, Y., Corso, J.J., Socher, R., Xiong, C.: End-to-end dense
video captioning with masked transformer. In: Proc. CVPR, pp. 8739–
8748 (2018)[21] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,
A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proc.
NeurIPS, pp. 5998–6008 (2017)[22] Kiddon, C., Ponnuraj, G.T., Zettlemoyer, L., Choi, Y.: Mise en Place:
unsupervised interpretation of instructional recipes. In: Proc. EMNLP,
pp. 982–992 (2015)[23] Maeta, H., Sasada, T., Mori, S.: A framework for procedural text
understanding. In: Proc. IWPT, pp. 50–60 (2015)[24] Jermsurawong, J., Habash, N.: Predicting the structure of cooking recipes.
In: Proc. EMNLP, pp. 781–786 (2015)[25] Pan, L., Chen, J., Wu, J., Liu, S., Ngo, C.-W., Kan, M.-Y., Jiang, Y.-G.,
Chua, T.-S.: Multi-modal cooking workflow construction for food recipes.
In: Proc. ACMMM, pp. 1132–1141 (2020)[26] Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., Yamakata, Y.,

Springer Nature 2021 LATEX template
30

State-aware Video Procedural Captioning
Mori, S.: Structure-aware procedural text generation from an image
sequence. IEEE Access 9, 2125–2141 (2020)[27] Gupta, A., Durrett, G.: Tracking discrete and continuous entity state forprocess understanding. In: Proc. NAACL Workshop SPNLP, pp. 7–12
(2019)[28] Santoro, A., Faulkner, R., Raposo, D., Rae, J., Chrzanowski, M., Weber,T., Wierstra, D., Vinyals, O., Pascanu, R., Lillicrap, T.: Relational
recurrent neural networks. In: Proc. NeurIPS, pp. 7299–7310 (2019)[29] Dalvi, B., Huang, L., Tandon, N., Yih, W.-t., Clark, P.: Tracking statechanges in procedural text: a challenge dataset and models for process
paragraph comprehension. In: Proc. NAACL, pp. 1595–1604 (2018)[30] Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word
representation. In: Proc. EMNLP, pp. 1532–1543 (2014)[31] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-trainingof deep bidirectional transformers for language understanding. In: Proc.
NAACL, pp. 4171–4186 (2019)[32] Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert:a joint model for video and language representation learning. In: Proc.
ICCV, pp. 7464–7473 (2019)[33] See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization withpointer-generator networks. In: Proc. ACL, pp. 1073–1083 (2017)[34] Jang, E., Gu, S., Poole, B.: Categorical reparametrization with gumblesoftmax. In: Proc. ICLR (2017)[35] Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for
relation extraction without labeled data. In: Proc. ACL-IJCNLP, pp.
1003–1011 (2009)[36] Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L.: Asymmetric loss for multi-label classification. arXiv (2020)[37] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: Proc. CVPR, pp. 770–778 (2016)[38] Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network
training by reducing internal covariate shift. In: Proc. ICML, pp. 448–456(2015)[39] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In:
Proc. ICLR, USA

Springer Nature 2021 LATEX template
State-aware Video Procedural Captioning

31[40] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., Salakhutdinov, R.:
Transformer-xl: attentive language models beyond a fixed-length context.
In: Proc. ACL, pp. 2978–2988 (2019)[41] Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method forautomatic evaluation of machine translation. In: Proc. ACL, pp. 311–318(2002)[42] Lin, C.-Y., Och, F.J.: Automatic evaluation of machine translation quality
using longest common subsequence and skip-bigram statistics. In: Proc.
ACL, pp. 605–612 (2004)[43] Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proc. ACL
Workshop IEEMMTS, pp. 65–72 (2005)[44] Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: consensus-based image
description evaluation. In: Proc. CVPR, pp. 4566–4575 (2015)[45] Chen, J., Ngo, C.-w.: Deep-based ingredient recognition for cooking recipe
retrieval. In: Proc. ACMMM, pp. 32–41 (2016)[46] van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of
Machine Learning Research 9, 2579–2605 (2008)[47] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed
representations of words and phrases and their compositionality. In:
NeurIPS, pp. 3111–3119 (2013)[48] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learningwith deep convolutional generative adversarial networks. arXiv (2015)[49] Salvador, A., Hynes, N., Aytar, Y., Marin, J., Ofli, F., Weber, I., Torralba,
A.: Learning cross-modal embeddings for cooking recipes and food images.
In: Proc. CVPR, pp. 3020–3028 (2017)[50] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos,
E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling
egocentric vision: The EPIC-KITCHENS dataset. In: Proc. ECCV, pp.
720–736 (2018)[51] Nishimura, T., Sakoda, K., Hashimoto, A., Ushiku, Y., Tanaka, N., Ono,
F., Kameko, H., Mori, S.: Egocentric biochemical video-and-language
dataset. In: Proc. CLVL, pp. 3129–3133 (2021)[52] Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings forsequence labeling. In: Proc. COLING, pp. 1638–1649 (2018)

Springer Nature 2021 LATEX template
32

State-aware Video Procedural Captioning[53] Yamakata, Y., Mori, S., Carroll, J.: English recipe flow graph corpus. In:
Proc. LREC, pp. 5187–5194 (2020)[54] Williams, R.J., Zipser, D.: A learning algorithm for continually running
fully recurrent neural networks. Neural Computation 1, 270–280 (1989)