State-aware Video Procedural Captioning
Taichi Nishimura1 Atsushi Hashimoto2 Yoshitaka Ushiku2 Hirotaka Kameko3 Shinsuke Mori3
taichitary@gmail.com,{atsushi.hashimoto,yoshitaka.ushiku}@sinicx.com,{kameko,forest}@i.kyoto-u.ac.jp
1 Graduate School of Informatics, Kyoto University, 2 OMRON SINIC X Corporation
3 Academic Center for Computing and Media Studies, Kyoto University

ABSTRACTVideo procedural captioning (VPC), which generates procedural
text from instructional videos, is an essential task for scene understanding and real-world applications. The main challenge of VPC
is to describe how to manipulate materials accurately. This paper
focuses on this challenge by designing a new VPC task, generating a procedural text from the clip sequence of an instructional
video and material list. In this task, the state of materials is sequentially changed by manipulations, yielding their state-aware visual
representations (e.g., eggs are transformed into cracked, stirred,then fried forms). The essential difficulty is to convert such visual representations into textual representations; that is, a model
should track the material states after manipulations to better associate the cross-modal relations. To achieve this, we propose anovel VPC method, which modifies an existing textual simulator
for tracking material states as a visual simulator and incorporates
it into a video captioning model. Our experimental results show
the effectiveness of the proposed method, which outperforms stateof-the-art video captioning models. We further analyze the learned
embedding of materials to demonstrate that the simulators capture their state transition. The code and dataset are available fromhttps://github.com/misogil0116/svpc

CCS CONCEPTS• Computing methodologies → Natural language generation;
Scene understanding; Temporal reasoning.

KEYWORDS
procedural text, instructional video, simulator
ACM Reference Format:
Taichi Nishimura1 Atsushi Hashimoto2 Yoshitaka Ushiku2 Hirotaka Kameko3
Shinsuke Mori3 . 2021. State-aware Video Procedural Captioning. In Proceedings of the 29th ACM International Conference on Multimedia (MM ’21),
October 20–24, 2021, Virtual Event, China. ACM, New York, NY, USA, 9 pages.
https://doi.org/10.1145/3474085.3475322

1

INTRODUCTION

In recent years, there has been significant progress in vision andlanguage research that targets procedural text and instructional
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
MM ’21, October 20–24, 2021, Virtual Event, China© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8651-7/21/10. . . $15.00
https://doi.org/10.1145/3474085.3475322

Material
list
Clip sequence

step 1

action(s) : statebuttereggs

cheeseState transition

Procedural text

add thebutter into a pan

addedstep 2

crackedstirredcrack theeggs and stir

step 3added
stirredadd the egg mixture
and cheese and stir

Figure 1: Concept of the proposed method. Given a clip sequence and material list, the proposed method generates aprocedural text by reasoning the state transition of materials at each step.
video [1, 2, 23, 24, 47]. Among the various tasks in such research,
it is important to describe the content of the video using natural
language for both scene understanding and real-world applications.
For example, machines can help people learn new skills by providing a quick overview of instructional videos. To this end, video
procedural captioning [37, 38] (VPC), which is the task of generating a procedural text from an instructional video, has been proposed.
VPC requires a model (1) to segment important clips from a video,(2) to enumerate materials used in the clips, and (3) to describe how
to manipulate them as a sentence for each clip. (1) and (2) havebeen independently studied as the task of temporal clip segmentation [12, 47] and object recognition [6, 33], respectively. In thispaper, we focus on (3) by designing a new VPC task and method ofgenerating a procedural text from a clip sequence pre-segmented
in an instructional video and material list (Figure 1).
Different from standard video captioning [11, 20, 40, 44, 46], ourtask requires a model to describe detailed material manipulations
from visual observations. The manipulations change the state ofmaterials sequentially, yielding their state-aware visual representations (e.g., in step 2 in Figure 1, “eggs” are transformed into “cracked”
then “stirred”). The essential difficulty is to convert such visual representations into textual representations; that is, a model should
track material states after manipulations to generate a proceduraltext. For example, when generating step 3 in Figure 1, the models
should be aware that the yellow liquid in the bowl and white liquidin the pan correspond to the eggs and butter manipulated in steps 1
and 2, respectively. Existing video captioning models cannot trackmaterial states; thus, they miss materials, hallucinate them (e.g.,
say materials that are not in the clip), and lack details (e.g., say“cook ingredients”), thereby degrading the captioning performance.
The above challenge is not specific to the cooking domain but isuniversal across multiple domains, such as wet-lab experiments
and furniture assembly.
To address this challenge, this paper aims to generate proceduraltexts by modeling the state transition of materials from visual observations. The same motivation, tracking materials against theirstate transition, is shared by the research community in natural
language understanding (NLU). To address this, some NLU studies
have proposed a simulator to reason the state transition of materials
in a procedural text [3, 5]. These simulators read one sentence in aprocedural text, predict executed actions and involved materials,
and recurrently update the state of materials to simulate their statetransition.
Inspired by these ideas, we propose a novel VPC method, which
modifies an existing NLU simulator as a visual simulator and incorporates it into an encoder-decoder architecture. Figure 1 illustrates
the idea of the proposed method. Given a clip sequence and material
list, the proposed method reasons the state transition of materials
and generates a procedural text accurately. We emphasize that thiswork is the first attempt to integrate the NLU simulator into a video
captioning model. In addition, based on the intuition that the statetransition of materials is consistently traceable from the generated
procedural texts, we attach a novel textual re-simulator, facilitating
the model to generate a procedural text even more accurately.
In our experiments, we test the proposed method in the cooking
domain because of its large variety of actions and materials (= ingredients). We compare it with two state-of-the-art video captioning
models on four evaluations: word-overlap evaluation, ingredientprediction, retrieval evaluation, and qualitative analysis, with thorough ablation. Our experimental results show that the proposed
method outperforms the state-of-the-art video captioning models.
In addition, ablation studies show the effectiveness of integrating
visual and textual simulators into the model. Finally, we analyze thelearned embedding of materials to demonstrate that the simulators
effectively capture the state transition of materials.

2

RELATED WORKWe describe the novelty of the proposed method in line with otherworks on video captioning in Section 2.1. In Section 2.2, we discuss
simulators for procedural text understanding because they are themain components of the proposed method.

2.1

Video captioning

Video captioning is an attractive field for both computer visionand natural language processing communities. The task settings ofvideo captioning vary according to the nature of the input video(e.g., one short clip, clip sequence, or long untrimmed video). Our
VPC task targets a clip sequence and thus belongs to the video
paragraph description [20, 30, 44], which aims to generate multiple
sentences for a given clip sequence pre-segmented in the video.
In the video paragraph description, recently, Transformer-based[42] models have been featured because of their ability to capture
long-term information in a clip sequence, compared with LSTMbased models [30, 44]. MART [20] is a transformer-based models
that achieves state-of-the-art performance in the video paragraphdescription. The key contribution of MART is the introduction ofa gated recurrent memory module, which makes it easy for themodel to summarize important information in the previous step.
Different from standard video captioning, VPC requires a model
to generate detailed material manipulations for input clips. To thisend, previous VPC works [37, 38] target narrated videos and utilize
transcription as a cue to generate a procedural text by fusing video
and transcription features. Although these approaches work wellfor narrated videos, transcription is not always available for all
instructional videos. Speaking during manipulations or adding narration to videos requires significant effort. Thus, the proposed VPC
task extends the previous VPC works to describe manipulations
even from non-narrated videos; our task allows a model to referto a human-created material list, which will be replaced with amachine-recognized one in the future. Our task requires a model to
track material states that are changed by manipulations to generatea procedural text accurately. To this end, we propose a novel VPC
method, which modifies an NLU simulator as a visual simulator
and incorporates it into a transformer-based encoder-decoder.

2.2

Procedural text understanding

In NLU, procedural texts are a popular target for understanding,and recipes have been featured in this field. Many researchers haveproposed methods to understand recipes, which are roughly divided
into two categories: (1) a graph-based parser and (2) a reasoningbased simulator.
A graph-based parser aims to train a model to map recipes into
graph or tree structures. Kiddon et al. [18] and Maeta et al. [22] proposed a model to estimate a graph structure called the action graphand recipe flow graph, respectively. Jermsurawong and Nizar [17]proposed a parser with the SIMMR dataset, which provides a tree
structure for understanding ingredient merging operations. Then,
Pan et al. [28] and Nishimura et al. [27] provided new multimodal
tree structure datasets and parsers, which can be regarded as anextension of the text-only version to a vision-and-language version.
These parsers focus on enabling machines to interpret recipes as
structural representations, rather than capturing the state transitionof ingredients in recipes.
A reasoning-based simulator aims to model the state transitionof ingredients in recipes by updating the ingredient states at eachstep. Gupta and Durrett [13] proposed a structured simulator that
imposes constraints on the state transition of ingredients (e.g., stirfried potatoes are not cut in later steps). Amac et al. [3] proposeda method to answer multi-modal question answering (QA) tasks
by reasoning the state transition of ingredients using a relation
reasoning network [35]. This paper builds upon the neural process
network (NPN) [5], which learns to estimate the state transition ofingredients by predicting executed actions and involved ingredients.
Owing to the ability to capture the state transition of ingredients,
these simulators reported competitive results in QA tasks [3, 9]. Our
work adopts a reasoning-based simulator. We modify an NPN as
the visual simulator and incorporate it into the transformer-based
encoder-decoder architecture to enable the model to capture thestate transition of ingredients.

3

PROPOSED METHOD

In this section, we present the proposed method. After describing an
overview in Section 3.1, we explain the components of the proposed

Encoder (sec 3.2) E

Visual simulator (sec 3.3) Rv

Textual re-simulator (sec 3.5) Rt

Decoder (sec 3.4) D

Materials
buttere10

Material
encodereggs

e20

cheesee30

Clip sequence

Material vectors

1

Clip
sequence
encoder

2

u1

added
crackedstirredstep 1

h1

step 2

h2
h3

step 3u3

Action
label

add…

add…

Generated

State-awareu ⊂step vectors nMateriallabel

step 1step 2 crack…

u2

added
stirredstep 33

Decoderstep 1

Constructdistant supervision

Clip vectors

step 1

s1

step 2s2

step 1step 3s3

step 2

crackedstirredstep 3added
stirredSentence vectors

added

add…

crack…
step 2stir…
step 3

Textualencoder

add…

Ground truth

Training
and inferenceTraining
only
Losscalculation

Figure 2: An overview of the proposed method. To track material states in a clip sequence, we incorporate the visual simulator 𝑅𝑣 into the transformer-based encoder-decoder architecture (𝐸 and 𝐷). In addition, based on our intuition that the statetransition of materials is traceable from the generated procedural texts, we attach the textual re-simulator 𝑅𝑡 to the model.

Figure 3: An overview of the (visual) simulator. The simulator recurrently reasons the state transition of the materials at eachstep. Specifically, it predicts executed actions and involved materials in (1) the action and (2) material selector and then updates
the state of materials in (3) the updater. The updated materials are forwarded to the next step. The textual re-simulator has
the same modules.
method from Section 3.2 to Section 3.5. Finally, in Section 3.6 we
explain the loss functions to train the model.

3.1

Overview

Figure 2 shows an overview of the proposed method. Given a
clip sequence V = (𝒗 1, . . . , 𝒗𝑛 , . . . , 𝒗 𝑁 ) and a material list G =(𝒈 1, . . . , 𝒈𝑚 , . . . , 𝒈 𝑀 ), our goal is to output a procedural text Y =(𝒚 1, . . . , 𝒚𝑛 , . . . , 𝒚 𝑁 ) by recurrently generating sentences from corresponding clips. To generate a procedural text accurately, it isessential for models to track material states in the clip sequence(e.g., eggs are transformed into cracked, stirred, then fried forms).
Inspired by recent advances in NLU, we achieve this by modifying the existing reasoning-based NLU simulator NPN as thevisual simulator 𝑅𝑣 , and incorporate it into the transformer-based
encoder-decoder architecture (𝐸 and 𝐷). Specifically, given clip H
and material encoded vectors E 0 , the visual simulator reasons thestate transition of materials and outputs state-aware step vectors

as U = 𝑅𝑣 (H, E 0 ). Then, the decoder 𝐷 outputs a procedural text
conditioned on U. We observe that this simple integration of thevisual simulator is effective for the model to generate a procedural
text accurately.
In addition, based on our assumption that the state transitionof materials should be consistently traceable from the generated
procedural text, we attach a novel textual re-simulator 𝑅𝑡 , encouraging the model to generate a procedural text even more accurately.
Specifically, the textual re-simulator 𝑅𝑡 reasons the state transitionof materials from the generated procedural text as 𝑅𝑡 (S, E 0 ), where
S represents the encoded sentence vectors. Note that the textual
re-simulator is only used during the training phase and detachedduring the inference phase.

3.2

Encoder

The input has two components: a material list G and a clip sequence
V. Thus, we develop a suitable encoder for each component.

Material encoder. To encode a material list, we input themto concatenated neural networks of pre-trained GloVe [31]1 wordembedding and multi-layer perceptrons (MLPs) with the ReLU
activation function. Multi-word materials (e.g., parmesan cheese)are represented by the average embedding vector of words. We
then add positional encoding (PE) to material indices, and obtain
0 , . . . , 𝒆 0 ).
the initial material vectors E 0 = (𝒆 01, . . . , 𝒆𝑚
𝑀
Clip sequence encoder. A clip sequence V is hierarchical because the clip sequence contains multiple clips, and each clip is
composed of sequential frames. Thus, to encode a clip sequenceeffectively, we design a two-stage transformer suitable to encode
a sequence of sequences. First, the former transformer encodes
each clip into a feature vector by extracting the vector, which corresponds to the [CLS] token as in [10, 20, 39]. Then the lattertransformer is trained over the sequence to obtain the step-aware
clip vectors H = (𝒉1, . . . , 𝒉𝑛 , . . . , 𝒉𝑁 ) in the clip sequence.

3.3

Visual simulator

Based on the encoded vectors of the clip sequence and material
list (H, E 0 ), the visual simulator, shown in Figure 3, reasons thestate transition of materials at each step. Specifically, at the 𝑛-th
step, given the 𝑛-th clip 𝒉𝑛 and (𝑛 − 1)-th material list E𝑛−1 , thevisual simulator predicts executed actions and involved materialsin (1) the action and (2) material selector and then updates the stateof materials in (3) the updater. After 𝑛-th reasoning, it outputs astate-aware step vector 𝒖𝑛 ∈ R3×𝑑 , which concatenates the 𝑛-th
clip 𝒉𝑛 , selected action 𝒇¯ 𝑛 and material vectors 𝒆¯𝑛 (𝑑 represents
the dimension of these vectors). The visual simulator recurrently
repeats the above process until processing the end element of theclip sequence. This simulation process is the same as NPN except
for the input modality; we replace the textual sentence and entity
vectors for NLU with visual clip H and material vectors E 0 for ourtask. Thus, we only provide a high-level overview in this subsection,and further details of the simulator are provided in Appendix A.
(1) Action selector. Given a clip vector 𝒉𝑛 , the action selector
outputs the selected action vector 𝒇¯ 𝑛 by choosing actions executedin the clip from the predefined action embedding F. For example,
in Figure 3, the actions “crack” and “stir” are executed in the clip,
thus both 𝒇 𝑐𝑟𝑎𝑐𝑘 and 𝒇 𝑠𝑡𝑖𝑟 should be selected. To consider multiple
actions, the action selector computes a soft selection 𝒘 𝑝 as an action
probability for each action in F. Then it outputs the selected action
vector 𝒇¯𝑛 as a weighted sum of the action embedding F and action
probability 𝒘 𝑝 .
(2) Material selector. Based on the action probability 𝒘 𝑝 andclip vector 𝒉𝑛 , the material selector outputs a selected material
vector 𝒆¯𝑛 by choosing materials that are involved in the clip fromthe material list E𝑛−1 . For example, in Figure 3, the raw “cheese”
and manipulated “eggs” and “butter” should be selected. To consider
such a combination of raw and manipulated material selection, thematerial selector has two attention modules: (1) clip attention and(2) recurrent attention. While the clip attention selects materials
from the current clip vector 𝒉𝑛 , the recurrent attention selects
materials based on both the current and previous clips. Using thesemodules, the material selector computes a soft selection 𝒂𝑛 as the
1 We employ pre-trained 300D word embedding, which can be downloaded fromhttp://nlp.stanford.edu/data/glove.6B.zip

material probability for each material in the material list E𝑛−1 . Thenit outputs the selected material vector 𝒆¯𝑛 as a weighted sum of thematerial vectors E𝑛−1 and material probability 𝒂𝑛 .
(3) Updater. Based on the selected actions and materials, theupdater represents the state transition of materials by computing
a new material vector 𝒆ˆ𝑚 . To this end, it first calculates an actionaware proposal vector 𝒍 𝑛 of materials with a bilinear transformationof selected action and material vectors (𝒇¯𝑛 , 𝒆¯𝑛 ). Then, based on thematerial probability 𝒂𝑛 , it computes the new material vector 𝒆ˆ𝑚 byinterpolating the action-aware proposal vector 𝒍 𝑛 and the current
𝑛−1 . The 𝑚-th new material vector 𝒆ˆ is assigned
material vector 𝒆𝑚𝑚𝑛 , which is forwarded to the next (𝑛 + 1)-th step.
to E𝑚

3.4

Decoder

The decoder 𝐷 outputs a procedural text by recurrently generating
sentences from the 𝑛-th output vector 𝒖𝑛 of the visual simulator.
As our decoder 𝐷, we use the transformer, which achieves state-ofthe-art performance in video captioning tasks [37, 38, 48]. Our task
allows the model to refer to a material list, thus to encourage thedecoder to generate materials, we incorporate the copy mechanism[36] into the decoder 𝐷. When generating the 𝑘-th word in the𝑛 ∈ E𝑛 , the copy
𝑛-th sentence, given the updated materials 𝒆𝑚
mechanism first calculates the attention probability 𝜷 𝑚using the𝑛,𝑘bilinear dot product of the vectors of the decoder output 𝒐𝑛,𝑘 and𝑛 ∈ E𝑛 as:
each material 𝒆𝑚
𝜷𝑚
𝑛,𝑘

=

𝑛 }
exp {(𝒐𝑛,𝑘 ) T𝑾 𝑐 𝒆𝑚Í𝑛 ,T𝑖 exp {(𝒐𝑛,𝑘 ) 𝑾 𝑐 𝒆𝑖 }(1)where 𝑾 𝑐 represents a bilinear map.
Then it calculates the copying gate 𝑔𝑛,𝑘 (0 ≤ 𝑔𝑛,𝑘 ≤ 1), whichmakes a soft choice between selecting a material from the material
list and generating a word from the vocabulary:Õ𝑛𝑔𝑛,𝑘 = 𝜎 (𝑾 𝑔 [𝒐𝑛,𝑘 ;𝜷𝑚(2)𝑛,𝑘 𝒆𝑚 ] + 𝒃 𝑔 ),𝑚where [·], 𝜎 (·), 𝑾 𝑔 , and 𝒃 𝑔 represent the concatenation function,
sigmoid function, linear map, and bias, respectively. Based on 𝑔𝑛,𝑘 ,the final predicted word probability 𝑃𝑛,𝑘 (𝑤) is computed as theweighted sum of the copy probability and generation probability
as follows:
Õ
 voc
© 1
ª
𝑃𝑛,𝑘 (𝑤) = 1 − 𝑔𝑛,𝑘 𝑃𝑛,𝑘(𝑤) + 𝑔𝑛,𝑘 ­𝜷𝑖 ® ,|𝒈𝑚 | 𝑖:𝑤 ∈𝒈 𝑛,𝑘
𝑖
𝑚
«
¬

(3)

voc (𝑤) and |𝒈 | represent the probability of the 𝑛-th senwhere 𝑃𝑛,𝑘𝑚tence’s 𝑘-th word 𝑤 in the vocabulary and the number of words ofin the 𝑚-th material, respectively2 .

3.5

Textual re-simulator

Because the state transition of materials is identical in the visualand textual worlds, it should be consistently traceable from thegenerated procedural texts. Based on this assumption, we add anovel textual re-simulator 𝑅𝑡 , encouraging the model to generate aprocedural text even more accurately.
2 To consider multiple words of materials, we divide the probability by the number ofwords.

The textual re-simulator consists of two sub-modules: (1) a textual encoder and (2) a textual simulator. The textual encoder converts a generated procedural text into step-aware sentence vectors
S = (𝒔 1, . . . , 𝒔𝑛 , . . . , 𝒔 𝑁 ). First, it applies the straight-through version of Gumbel softmax resampling [16] to sample a procedural
text, preserving the differentiable chain. The sampled procedural
text is further converted into feature vectors by computing the
average vector of the word embedding at each step. Note that theword embedding is shared between the decoder and textual encoder. They are then converted into step-aware sentence vectors S
using a biLSTM encoder. Finally, based on S, the textual simulator,
another NPN described in Section 3.3, reasons the state transitionof materials again as 𝑅𝑡 (S, E 0 ).

3.6

Loss functions

To train the model, we compute three types of losses: (1) sentence
generation loss L𝑠𝑒𝑛𝑡 , (2) visual simulation loss L𝑣_𝑠𝑖𝑚 , and (3)textual re-simulation loss L𝑡 _𝑠𝑖𝑚 .
(1) Sentence generation loss L𝑠𝑒𝑛𝑡 . This loss aims to train the
decoder, and is computed as the summed negative log-likelihood
for all input/output pairs {(V, G), Y} in the training set.
(2) Visual simulation loss L𝑣_𝑠𝑖𝑚 . This loss aims to train thevisual simulator, and consists of two losses: (1) material selection
loss and (2) action selection loss. These losses are computed as thesummed binary cross-entropy loss based on whether the materials/actions are involved/executed in the clip. To avoid costly humanannotations, we compute the loss from distant supervision [26]following the original NPN training method [5]. For the material
selection loss, labels are obtained whether or not each step contains
materials in the material list; for the action selection loss, labels
are obtained whether or not each step contains actions in the 384
actions defined by [5]. For example, from the sentence “crack theeggs and stir,” “eggs” is extracted as a material label, and “crack”
and “stir” are extracted as an action label. As the action selection
loss, the simple binary cross-entropy does not work in our preliminary experiment because the ratio of positive to negative actions is
imbalanced; a few actions are positive and most of the actions arenegative. Thus, as the action selection loss, we use asymmetric loss[45], which is a weighted binary cross-entropy loss, to defuse theimbalanced problem.
(3) Textual re-simulation loss L𝑡 _𝑠𝑖𝑚 . To train the textual resimulator, we also compute the above visual simulation loss fromthe sampled procedural texts.
Total loss. Consequently, to train the entire model in an end-toend manner, the total loss is computed as
L𝑡𝑜𝑡𝑎𝑙 = L𝑠𝑒𝑛𝑡 + L𝑣_𝑠𝑖𝑚 + 𝜆L𝑡 _𝑠𝑖𝑚 ,(4)where 𝜆 is a hyper-parameter used for weighting the importanceof the textual re-simulation loss.

4

EXPERIMENTS

We test the proposed method in the cooking domain because procedural texts (= recipes) have a large variety of actions and materials (= ingredients). We compare it with two state-of-the-art video
captioning models on four evaluations: word-overlap evaluation(Section 4.2), ingredient prediction (Section 4.3), retrieval evaluation

Table 1: YouCook2-ingredient dataset statistics.
#recipes#steps / #recipes#ingredients / #recipes

train
1,233
7.7
10.4

val
100
7.9
10.2

test
329
7.6
10.3

(Section 4.4), and qualitative analysis (Section 4.5) with thoroughablation. We also visualize the learned embedding of ingredients,
demonstrating that the visual simulator effectively reasons the statetransition of ingredients (Section 4.6).

4.1

Experimental settings

Dataset. We use the YouCook2 dataset [47], which consists of 2,000
cooking videos from 89 recipe categories. All of the videos have
3–16 clips with a start/end timestamp annotated by humans, andeach clip is also annotated with an English sentence. Note that
ingredients are not annotated in the original dataset; thus, we prepared the YouCook2-ingredient dataset by annotating ingredientswith recipes. We downloaded the videos from the website, removed
invalid videos (e.g., corrupted or unavailable videos), and finallyobtained 1,664 videos. Then, we hired three annotators to write
ingredients that appear in the recipe (for annotation details, see
Appendix B). After this annotation process, we manually removed
recipe/video pairs that had no ingredient annotations (here, twovideos were removed). The remaining 1,662 valid videos were usedas the YouCook2-ingredient dataset and divided as follows: 1,233
for training, 100 for validation, and 329 for testing3 . Table 1 shows
the statistics of the YouCook2-ingredient dataset.
Data preprocessing. As clip features, we use concatenated features of appearance and optical flow provided by [47]. With respect to the appearance, 2,048D feature vectors extracted from the“Flatten-673” layer in ResNet-200 [14] are used, and for the opticalflow, 1,024D feature vectors extracted from the “global pool” layer
in BN-Inception [15] are used. As in [20], we truncated sequences
longer than 100 for the clip and 20 for the sentence and set the
maximum length of the clip sequence to 12. Finally, we built the
vocabulary based on words that occurred three times or more, andthe resulting vocabulary contained 951 words.
Hyper-parameter settings. For both the encoder and decodertransformers, we set the hidden size to 768, the number of layers totwo, and the number of attention heads to 12. We train the model
following the optimization method described in [10, 20]; we use
the Adam optimizer [19] with an initial learning rate of 0.0001,
𝛽 1 = 0.9, and 𝛽 2 = 0.999. The L2 weight decay is set to 0.01, andthe learning rate warmup is over the first five epochs. We set thebatch size to 16, and continue training at most 50 epochs using
early stopping with CIDEr-D. We tune 𝜆 with four different values
𝜆 ∈ {0.25, 0.5, 0.75, 1.0} and set 𝜆 to 0.5 in our experiments (fordetails, see Section 4.2).
Models. We test the proposed method by comparing it with twostate-of-the-art video captioning models, as described below.
• Transformer-XL [8] is a powerful transformer-based language model that was originally proposed for capturing longterm dependency in natural language. As in [20], we adapt
3 We will release annotated ingredients and the dataset split.

Table 2: Word-overlap metrics for the baseline and the proposed models with ablation studies. The scores in bold
are the best among the comparative models. “I” indicates
whether the model uses ingredient information or not.
B=BLEU, M=METEOR, C=CIDEr-D, RL=ROUGE-L.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

I

✓

B1
37.8
39.7
39.8
44.6

B4
6.7
8.8
7.7
10.2

M
14.8
16.2
15.6
18.3

C
25.0
38.5
35.9
49.7

RL
31.5
34.4
32.2
36.4

✓
✓
✓

39.4
45.3
48.3
49.4

8.0
11.2
11.9
12.1

15.3
19.3
21.2
21.6

32.7
61.7
73.8
77.3

32.1
37.4
40.0
39.9

✓

Table 3: Change in word-overlap metrics with controlled 𝜆.
𝜆 = 0 (VIV)
𝜆 = 0.25
𝜆 = 0.5
𝜆 = 0.75
𝜆 = 1.0

B1
48.3
48.8
49.4
50.1
49.2

B4
11.9
11.9
12.1
11.7
12.0

M
21.2
21.4
21.6
21.6
21.6

C
73.8
75.4
77.3
76.8
76.1

RL
40.0
40.0
39.9
39.6
40.1

it for our task; the model directly uses all of the previous
hidden states to generate a current sentence.
• MART [20] is a transformer-based video captioning modelthat achieves state-of-the-art performance on the video paragraph description. This model generates a sentence with agated recurrent memory module, which does not pass allof the previous hidden states, but effectively summarizes
important information in the previous step.
Note that these models originally have no ingredient list in the
inputs and copy mechanism in the decoder. Thus for a fair comparison, we prepare for additional baselines, baseline + ingredient (-I)
models, which incorporate the material encoder (Section 3.2) andthe copy mechanism into the models (for implementation detailsof the models, see Appendix C).
Ablations. To reveal the impact of the components in the proposed method, we conduct ablation studies on the following variations.
• Video only (V) encodes a clip sequence with the clip sequence encoder, then generates a procedural text.
• V + Ingredient (VI) incorporates the material encoder andcopy mechanism in the model.
• VI + Visual simulator (VIV) incorporates the visual simulator into the VI model.
• VIV + Textual re-simulator (VIVT) additionally incorporates the textual re-simulator into the VIV model.

4.2

Word-overlap evaluation

Scores. To evaluate the captioning performance of the proposed
method, we compute commonly used word-overlap metrics, such as
BLEU [29], ROUGE-L [21], METEOR [4], and CIDEr-D [43] in thetest set. Note that they are computed at the recipe (paragraph)-level
following [20, 30, 44], not at the step (sentence)-level.

Table 4: Results of ingredient prediction.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)

Recall
11.7
19.7
13.0
21.6

Precision
19.5
33.6
20.7
33.1

F1
14.6
24.8
16.0
26.2

10.9
24.8
34.7
35.5

19.3
40.3
50.0
51.5

13.9
30.7
40.8
42.0

Results. Table 2 shows the results of the word-overlap evaluation. We observe that the proposed method consistently outperforms the state-of-the-art captioning models by a significant
margin. Our ablation studies show that the VIV model performs
better than the VI model, and the VIVT model further improves the
VIV model. This indicates that both the visual simulator and the
textual re-simulator are effective for generating a recipe accurately.
Performance change of controlling the hyper-parameter𝜆. Table 3 shows the results by varying the 𝜆 ∈ {0.0, 0.25, 0.5, 0.75, 1.0}.
Note that 𝜆 = 0 is equivalent to the VIV model, which does not havea textual re-simulator. The results indicate that (1) the VIVT model
performs better than the VIV model for any 𝜆 values, and (2) 𝜆 = 0.5
performs the best among the three metrics (BLEU4, METEOR, and
CIDEr-D) and obtains competitive results in BLEU1 and ROUGE-L.
Thus, we set 𝜆 = 0.5 in our experiments.

4.3

Ingredient prediction

To evaluate whether the models use correct ingredients at each step
without missing and hallucinating them, we design the ingredientprediction, which measures the step-level overlap of ingredients
between generated and ground-truth recipes. To this end, we first
construct an ingredient dictionary from all unique ingredients in
the YouCook2-ingredient dataset. Then, at each step, we extract
ingredients that are exact-matched between generated recipes andthe ingredient dictionary. The same process is performed to extract
ingredients in the ground-truth recipes. Finally, based on the extracted ingredient sets, we compute the micro- recall, precision,and F1 scores, respectively.
Results. Table 4 shows the results of the ingredient prediction.
This result shows that the proposed method outperforms the stateof-the-art video captioning models. In our ablation, we observe
the same tendency of performance change to the word-overlap
evaluation. We note that the VIV model performs much better thanthe VI model by 10% in F1, indicating that not only the copy mechanism but also the visual simulator are important for generating
ingredients correctly. We also notice that the VIVT model improves
the VIV model by 1.2% in F1, demonstrating the effectiveness of the
textual re-simulator.

4.4

Retrieval evaluation

To evaluate whether the generated procedural texts are sufficiently
concrete to describe the input clips, we design a step-level zeroshot sentence-to-clip retrieval evaluation. As a retrieval model, we
employ the MIL-NCE model [23] pre-trained on the HowTo100M
dataset [24], achieving the state-of-the-art performance. In thisIngredientsflour, eggs, baking soda, salt, pepper, water, shrimp, batter, breadcrumbs, oilstep 1step 2step 3

Clip sequence
MART +
Ingredients(MART-I)

add flour salt and pepper to a bowl and mix (✗eggs, baking soda)

add milk egg and milk to the bowl and mix(✗ water)coat the dough in the batter (✗ shrimp,
breadcrumbs)

V + Ingredients(VI)mix flour salt pepper and breadcrumbs (✗baking soda, eggs)

mix flour salt pepper and breadcrumbs with theflour (✗ water)

coat the shrimp with the flour mixture (✗ batter,breadcrumbs)

VI + Visual
simulator (VIV)

mix flour eggs and salt together(✗ baking soda, pepper)

add salt pepper to the eggs and mix (✗ water)

coat the shrimp in the batter (✗ breadcrumbs)+ VIV + Textual
re-simulator(VIVT)

mix flour eggs baking soda salt and pepper and add water eggs breadcrumbs to a bowl of watercoat the shrimp in the batter (✗ breadcrumbs)
salt
and mix

Ground truth

add flour eggs baking soda salt and pepper tothe bowl and stir

add cold water to the bowl and stir

cover the shrimp in the batter andbreadcrumbs

step 4step 5

Clip sequence
MART + Ingredients(MART-I)
V + Ingredients(VI)
VI + Visual simulator(VIV)

fry the onion rings in oil (✗ shrimp)

remove the shrimp from the oilheat oil in a pan and add the shrimp and fry

remove the shrimp from the oilheat oil in a pan and fry the shrimp in it

remove the shrimp from the oil

VIV + Textual re-simulator
fry the shrimp in oil(VIVT)
Ground truth

remove the shrimp from the oilplace the shrimp into a pan of hot oilremove the shrimp from the pan

Figure 4: Examples of generated recipes. Here, we compare four models, MART-I (baseline), VI, VIV, and VIVT with the groundtruth. Green bold and red words represent semantically correct and incorrect ingredients, respectively. Words in parentheses
indicate missing ingredients, which should be included in the sentence. Note that parallel words in a sentence are not separated
from the commas in the YouCook2 dataset (see step 1 in the ground truth).
Table 5: Results of retrieval evaluation. ↓ indicates that lower
is better.
Baseline
Transformer-XL+ Ingredients (Transformer-XL-I)
MART+ Ingredients (MART-I)
Ours
Video only (V)V + Ingredients (VI)
VI + Visual simulator (VIV)
VIV + Textual re-simulator (VIVT)
Ground truth

MedR (↓)
214
144.5
179
103

R@1
1.1
1.9
1.5
2.3

R@5
4.9
7.0
5.9
9.5

R@10
9.0
11.3
9.8
14.6

244
90
67
66
10

1.2
3.8
4.0
4.2
15.2

4.5
11.6
14.3
13.9
39.3

7.3
18.1
20.9
21.4
51.8

task, given a generated step-level sentence as a query, the MIL-NCE
model embeds it and computes the cosine similarly as a score between the query vector and all the 2,493 clip vectors from the test
set. Then, we sort scores with clips in descending order and calculate the median rank (MedR) and recall rate at the top 𝐾 (R@𝐾).
The median rank represents the median ranking of retrieved corresponding clips, hence lower is better; in contrast, R@𝐾 represents
the percentage of all the step-level sentence queries where thecorresponding clip is retrieved in the top 𝐾, hence higher is better.
Results. Table 5 shows the results of the retrieval evaluation.
We observe that the proposed method significantly outperforms thestate-of-the-art video captioning models. In MedR, the VIVT model

achieves 66, which is marginally lower than that of Transformer-XLI 144.5 and MART-I 103. This indicates that the proposed method
generates a more concrete recipe based on clips than the state-ofthe-art video captioning models. In our ablation, the VIV model
dramatically improves the VI model, indicating that the visual simulator is essential for generating concrete recipes. In addition, the
VIVT model shows a steady improvement from the VIV model,
indicating the effectiveness of the textual re-simulator.

4.5

Qualitative analysis

Figure 4 shows an example of the generated recipes. Other examples
are presented in Appendix D.
Insights. For the VPC task, it is important to generate correct
ingredients that are manipulated in a clip. MART-I fails to generate
ingredients correctly; the model tends to miss and hallucinate ingredients (e.g., “eggs” and “milk” in steps 1 and 2). We can observea similar tendency to the VI model, indicating that these models
superfluously generate ingredients listed in the ingredient list.
The VIV model suppresses these problems (e.g., “batter” in step3). In addition, owing to the textual re-simulator, the VIVT model
can generate ingredients that are missed in the VIV model (e.g.,“baking soda” and “pepper” in step 1, and “water” in step 2).
Limitations. Although the proposed method generates recipes
more accurately than the baseline models, we still found some differences from the ground truth. For example, in step 2, the VIV

Clip

Recip
Predicte
ingredient category

Ingredient

ourshrimptempura(a)potatoes

our

corn
dogs

(b)

our

our

naan

(c)

bacon

our
Clip

Predicte
Recip
ingredient category

eggs
eggs

d

shrimptempurascrambled
eggs
eggs
benedict

d

e

e

fl

fl

fl

fl

eggs

eggs

Figure 5: Learned embedding of ingredients obtained by the
VIVT model. Note that only raw and updated (the attention
weight in the material selector is higher than 0.5) ingredientsare transformed by t-SNE [41]. Red and blue colors represent
the raw and updated ingredients, respectively.
and VIVT models refer to “salt,” “pepper,” “eggs,” and “breadcrumbs”
superfluously, but only “water” is added in the ground truth. In addition, in step 3, the VIV and VIVT models refer only to “batter,” but“breadcrumbs” are also used. To solve these problems, we believethat incorporating fine-grained ingredient recognition modules [7]would help the model to generate a recipe more precisely.

4.6

Discussion of the learned embedding

To investigate how the visual simulator represents the state transition of ingredients, we visualize the ingredient embedding byprojecting it to 2D space using t-SNE [41]. Figure 5 shows the
projected learned embedding of ingredients obtained by the VIVT
model. Note that only raw (red) and updated (blue)4 ingredientsare shown in this figure. This result shows that the raw and updated points are clearly divided into two main clusters in the vector
space5 .
We also investigate the ingredients’ trajectory with the retrieved
top-2 nearest ingredient vectors from updated ingredient vectors(see the zoomed parts of the figure). The retrieved ingredients indicate their state-awareness; that is, ingredients with the similarstates are embedded into the same cluster in the vector space regardless of the difference in their recipe categories defined by [47].
For example, the near vectors of updated “eggs” with the “beat”
state are also updated “eggs” with “beat”-like states (e.g., mix andstir). “flour” also has the same tendency.
Semantic vector arithmetic. To demonstrate the state-aware
ness of learned embedding, we attempt to apply simple arithmetic
operations as performed in the literature [25, 32, 34]. In the context
of our task, the state transition of ingredients is expected to be
4 The attention weight in the material selector was higher than 0.5.

5 The raw and updated ingredients correspond to an embedding E 0 by the material

encoder and an embedding E𝑛 updated from E 0 by the visual simulator, respectively.

Updated ingredientcut
tomatoes
ad
eg
fr
onio
fr
onio

Ra
ingredientUpdated ingredien(nearest vector)

tomatoescut potatoes

egg

added ouronion

fried bacon

onion

choppemeat (fail)

(d)

meat

(e)

choppedshallot

add
egg

egg

added chopped
shallot(f)

cut
shrimpcover
tortillatortilla

covered cut
shrimpcu
potatoes

add
egg

egg

mashedpotatoes (fail)

(g)
fl y
t

d

g

w

 

n

d

fl

t

Figure 6: Arithmetics using the learned embedding of ingredients. Examples (a) to (d) and (e) to (g) represent thefirst-order (raw-to-updated) and second-order (updated-toupdated) transformations, respectively. (d) and (g) show the
failure cases for each transformation.
computed as 𝑣 (cut potatoes) = 𝑣 (potatoes) + 𝑣 (cut tomatoes) −𝑣 (tomatoes), where 𝑣 represents the map into the embedding space.
Figure 6 shows seven examples of the arithmetic operations.
From (a) to (d), first-order transformations (raw-to-updated) are described, and from (e) to (g), second-order transformations (updatedto-updated) are described. We can see that the learned embedding
simulates the state transition of ingredients by specific actions inboth transformations. For example, in (a) and (e), “raw potatoes”
and “chopped shallot” are converted into “cut potatoes” and “added
chopped shallot,” respectively. However, we observe some failurecases, where (d) and (g), “raw meat” is transformed into “chopped
meat” via “fry” and “cut potatoes” into “mashed potatoes” via “add.”
In these examples, ingredients are consistent before and after, but
executed actions are different because of the failure in action selection. We believe that noisy action labeling causes this failure, and asophisticated action selection would ease this problem.

5

CONCLUSION

In this paper, we proposed a new VPC task for generating a procedural text from a clip sequence and material list. To generate aprocedural text accurately, it is essential for models to track material states in a clip sequence. To achieve this, we proposed a novel
VPC method, which modifies the existing simulator as a visual
simulator and incorporates it into the encoder-decoder architecture. Our experimental results with thorough ablation demonstrate
the effectiveness of the proposed method, which outperforms thestate-of-the-art video captioning models. The learned embeddingof materials demonstrates that the simulator effectively captures
their state transition.

ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Number JP21J20250
and JP20H04210, and partially supported by JP21H04910, JP17H06100,
JST-Mirai Program Grant Number JPMJMI21G2, and JST ACT-I
Grant Number JPMJPR17U5.

REFERENCES[1] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan
Laptev, and Simon Lacoste-Julien. 2016. Unsupervised learning from narrated
instruction videos. In Proc. CVPR. 4575–4583.
[2] Jean-Baptiste Alayrac, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. 2017.
Joint discovery of object states and manipulation actions. In Proc. ICCV. 2127–
2136.
[3] Mustafa Sercan Amac, Semih Yagcioglu, Aykut Erdem, and Erkut Erdem. 2019.
Procedural reasoning networks for understanding multimodal procedures. In
Proc. CoNLL. 441–451.
[4] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for
MT evaluation with improved correlation with human judgments. In Proc. ACL
Workshop IEEMMTS. 65–72.
[5] Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin
Choi. 2018. Simulating action dynamics with neural process networks. In Proc.
ICLR.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection withtransformers. In Proc. ECCV. 213–229.
[7] Jingjing Chen and Chong wah Ngo. 2016. Deep-based ingredient recognition forcooking recipe retrieval. In Proc. ACMMM. 32–41.
[8] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan
Salakhutdinov. 2019. Transformer-XL: attentive language models beyond a fixedlength context. In Proc. ACL. 2978–2988.
[9] Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen tau Yih, and Peter Clark. 2018.
Tracking state changes in procedural text: a challenge dataset and models forprocess paragraph comprehension. In Proc. NAACL. 1595–1604.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
pre-training of deep bidirectional transformers for language understanding. In
Proc. NAACL. 4171–4186.
[11] Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan,
Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent
convolutional networks for visual recognition and description. In Proc. CVPR.
2625–2634.
[12] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.
2016. DAPs: deep action proposals for action understanding. In Proc. ECCV. 768–784.
[13] Aditya Gupta and Greg Durrett. 2019. Tracking discrete and continuous entity
state for process understanding. In Proc. NAACL Workshop SPNLP. 7–12.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual
learning for image recognition. In Proc. CVPR. 770–778.
[15] Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: accelerating deep
network training by reducing internal covariate shift. In Proc. ICML. 448–456.
[16] Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization withgumble-softmax. In Proc. ICLR.
[17] Jermsak Jermsurawong and Nizar Habash. 2015. Predicting the structure ofcooking recipes. In Proc. EMNLP. 781–786.
[18] Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi.
2015. Mise en Place: unsupervised interpretation of instructional recipes. In Proc.
EMNLP. 982–992.
[19] Diederik P. Kingma and Jimmy Ba. [n.d.]. Adam: A method for stochastic optimization. In Proc. ICLR. USA.
[20] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara Berg, and Mohit Bansal. 2020.
MART: memory-augmented recurrent transformer for coherent video paragraphcaptioning. In Proc. ACL. 2603–2614.
[21] Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machinetranslation quality using longest common subsequence and skip-bigram statistics.
In Proc. ACL. 605–612.
[22] Hirokuni Maeta, Tetsuro Sasada, and Shinsuke Mori. 2015. A framework forprocedural text understanding. In Proc. IWPT. 50–60.
[23] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, andAndrew Zisserman. 2020. End-to-end learning of visual representations fromuncurated instructional videos. In Proc. CVPR. 9879–9889.

[24] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan
Laptev, and Josef Sivic. 2019. HowTo100M: learning a text-video embedding by
watching hundred million narrated video clips. In Proc. ICCV. 2630–2640.
[25] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013.
Distributed representations of words and phrases and their compositionality. In
NeurIPS. 3111–3119.
[26] Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proc. ACL-IJCNLP. 1003–1011.
[27] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, Yoko
Yamakata, and Shinsuke Mori. 2020. Structure-aware procedural text generationfrom an image sequence. IEEE Access 9 (2020), 2125–2141.
[28] Liangming Pan, Jingjing Chen, Jianlong Wu, Shaoteng Liu, Chong-Wah Ngo,
Min-Yen Kan, Yu-Gang Jiang, and Tat-Seng Chua. 2020. Multi-modal cooking
workflow construction for food recipes. In Proc. ACMMM. 1132–1141.
[29] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: amethod for automatic evaluation of machine translation. In Proc. ACL. 311–318.
[30] Jae Sung Park, Marcus Rohrbach, Trevor Darrell, and Anna Rohrbach. 2019.
Adversarial inference for multi-sentence video description. In Proc. CVPR. 6598–
6608.
[31] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe:
global vectors for word representation. In Proc. EMNLP. 1532–1543.
[32] Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv.
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time object detection with region proposal networks. In Proc. NeurIPS.
91–99.
[34] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar
Weber, and Antonio Torralba. 2017. Learning cross-modal embeddings for cooking recipes and food images. In Proc. CVPR. 3020–3028.
[35] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski,
Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy
Lillicrap. 2019. Relational recurrent neural networks. In Proc. NeurIPS. 7299–7310.
[36] Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point:summarization with pointer-generator networks. In Proc. ACL. 1073–1083.
[37] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming
Zhou. 2019. Dense procedure captioning in narrated instructional videos. In Proc.
ACL. 6382–6391.
[38] Botian Shi, Lei Ji, Zhendong Niu, Nan Duan, Ming Zhou, and Xilin Chen. 2020.
Learning semantic concepts and temporal alignment for narrated video procedural captioning. In Proc. ACMMM. 4355–4363.
[39] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.
2019. VideoBERT: a joint model for video and language representation learning.
In Proc. ICCV. 7464–7473.
[40] Ganchao Tan, Daqing Liu, Meng Wang, and Zheng-Jun Zha. 2020. Learning todiscretely compose reasoning module networks for video captioning. In Proc.
IJCAI. 745–752.
[41] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.
Journal of Machine Learning Research 9 (2008), 2579–2605.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Proc. NeurIPS. 5998–6008.
[43] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr:
consensus-based image description evaluation. In Proc. CVPR. 4566–4575.
[44] Yilei Xiong, Bo Dai, and Dahua Lin. 2018. Move forward and tell: a progressive
generator of video descriptions. In Proc. ECCV. 489–505.
[45] Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.
2020. Asymmetric loss for multi-label classification. arXiv.
[46] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus
Rohrbach. 2019. Grounded video description. In Proc. CVPR. 6578–6587.
[47] Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2018. Towards automaticlearning of procedures from web instructional videos. In Proc. AAAI. 7590–7598.
[48] Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong.
2018. End-to-end dense video captioning with masked transformer. In Proc. CVPR.
8739–8748.