SlowGuess/ACL_2020_OCR huggingface dataset:
Angela S. Lin * Sudha Rao* Asli Celikyilmaz* Elnaz Nouri* Chris Brockett* Debadeepta Dey* Bill Dolan*
\*Salesforce Research, Palo Alto, CA, USA
Microsoft Research, Redmond, WA, USA
angela.lin@salesforce.com {sudhra, aslicel, elnouri}  $@$  microsoft.com
{chrisbkt,dedey,billdol}  $@$  microsoft.com
Many high-level procedural tasks can be decomposed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many partially-overlapping text and video recipes (i.e. procedures) that describe how to make the same dish (i.e. high-level task). Aligning instructions for the same dish across different sources can yield descriptive visual explanations that are far richer semantically than conventional textual instructions, providing commonsense insight into how real-world procedures are structured. Learning to align these different instruction sets is challenging because: a) different recipes vary in their order of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text instructions. To address these challenges, we first use an unsupervised alignment algorithm that learns pairwise alignments between instructions of different recipes for the same dish. We then use a graph algorithm to derive a joint alignment between multiple text and multiple video recipes for the same dish. We release the MICROSOFT RESEARCH MULTIMODAL ALIGNED RECIPCE CORPUS containing  $\sim 150\mathrm{K}$  pairwise alignments between recipes across 4,262 dishes with rich commonsense information.
Although machine learning has seen tremendous recent success in challenging game environments such as Go (Schrittwieser et al., 2019), DOTA (OpenAI, 2019), and StarCraft (DeepMind, 2019), we have not seen similar progress toward algorithms that might one day help humans perform everyday tasks like assembling furniture, applying makeup,
Figure 1: Text recipe (left) and transcript of video recipe (right) for shrimp fried rice. Aligned instructions are highlighted in the same color. Ingredients that can be substituted are encircled in the same color.
repairing an electrical problem, or cooking a particular dish. In part this is because the relevant large-scale multimodal (language, video, audio) datasets are difficult to acquire, even with extensive crowdsourcing (Salvador et al., 2017; Sanabria et al., 2018). Unimodal data, though, is abundant on the web (e.g. instructional videos or textual instructions of tasks). Using language as the link between these modalities, we present an approach for learning large-scale alignment between multimodal procedural data. We hope our work, and the resulting released dataset, will help spur research on real-world procedural tasks.
Recipes in the cooking domain provide procedural instruction sets that are captured – in large volume – both in video and text-only forms. Instruction sets in these two modalities overlap sufficiently to allow for an alignment that reveals interestingly different information in the linguistic and visual realms. In Figure 1, for instance, the text recipe (left) and the transcribed video recipe (right) for shrimp fried rice vary in word usage, order of instructions and use of ingredients. Know-
*Work done when the author was an intern at Microsoft.  
<sup>1</sup>https://github.com/microsoft/multimodal-aligned-recipe-corpus
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4871-4884 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Dish level alignment between three text recipes and two video recipes for fried rice. Same colored text boxes (in text recipes) and image borders (in video recipes) indicate instructions that are aligned to each other.
ing that the highlighted instructions correspond to the same step is useful in understanding potential ingredient substitutions, how the same step can be linguistically described and physically realized in different ways, and how instruction order can be varied without affecting the outcome.
Motivated by this idea that aligned procedural data can be a powerful source of practical commonsense knowledge, we describe our approach for constructing the MICROSOFT RESEARCH MULTIMODAL ALIGNED RECIPCE CORPUS. We first extract a large number of text and video recipes from the web. Our goal is to find joint alignments between multiple text recipes and multiple video recipes for the same dish (see Figure 2). The task is challenging, as different recipes vary in their order of instructions and use of ingredients. Moreover, video instructions can be noisy, and text and video instructions include different levels of specificity in their descriptions. Most previous alignment approaches (Munteanu and Marcu, 2005) deal with pairwise alignments. Since our goal is to align multiple instruction sets, we introduce a novel two-stage unsupervised algorithm. In the first stage, we learn pairwise alignments between two text recipes, two video recipes, and between a text and a video recipe using an unsupervised alignment algorithm (§3.1). In the second stage, we use the pairwise alignments between all recipes within a dish to construct a graph for each dish and find a maximum spanning tree of this graph to derive joint
alignments across multiple recipes (§3.2).
We train our unsupervised algorithm on 4,262 dishes consisting of multiple text and video recipes per dish. We release the resulting pairwise and joint alignments between multiple recipes within a dish for all 4,262 dishes, along with commonsense information such as textual and visual paraphrases, and single-step to multi-step breakdown (§5).
We evaluate our pairwise alignment algorithm on two datasets: 1,625 text-video recipe pairs across 90 dishes from the YouCook2 dataset (Zhou et al., 2018a), and a small set of 200 human-aligned text-text recipe pairs across 5 dishes from Common Crawl. We compare our algorithm to several textual similarity baselines and perform ablations over our trained model (§4). Finally, we discuss how this data release will help with research at the intersection of language, vision, and robotics (§6).
2 Recipe Data Collection
We describe our approach for collecting large-scale text and video recipes; and constructing recipe pairs for training our unsupervised alignment algorithm.
2.1 Common crawl Text Recipes
We extract text recipes from Common Crawl, one of the largest web sources of text. We heuristically filter the extracted recipes to obtain a total of 48,852 recipes across 4,262 dishes. The number
$^{2}$ https://commoncrawl.org/
3Details in supplementary.
Figure 3: An example transcript of a video recipe with sentences marked as "chat" (non-instructional) or "content" (instructional).
of recipes per dish ranges from 3 to 100 (with an average of 6.54 and standard deviation of 7.22). The average recipe length is 8 instructions.
2.2 YouTube Video Recipes
For each dish in the text recipes, we use the dish name with 'recipe' appended, e.g. 'chocolate chip cookie recipe', as a query on YouTube and extract the top N videos where N is proportional to the number of text recipes for that dish<sup>4</sup> to obtain a total of 77,550 video recipes. We transcribe these videos using the Microsoft Speech-to-Text Cognitive service.<sup>5</sup>
Video recipes, unlike text recipes, contain noninstructional ("chat") information. For instance, the presenter may give an introduction either of themselves or of the dish at the beginning of the video before diving into the steps of the recipe. Figure 3 contains an example transcript with "chat" and "content" information marked. We hypothesize that it is useful to remove such chat information from the transcripts before aligning them to text recipes. We build a supervised chat/content classifier using the YouCook2 dataset (Zhou et al., 2018a), an existing instructional cooking video dataset where parts of video that correspond to instructions are annotated by humans. We assume that these parts correspond to content whereas the rest of the video corresponds to chat. We preprocess the transcriptions of all 77,550 videos using this chat/content classifier<sup>7</sup> to remove all sentences classified as chat.
Train	Val	Test
No. of dishes	4,065	94	103
Text-Text Pairs	46,054	5,822	11,652
Text-Video Pairs	56,291	3,800	5,341
Video-Video Pairs	19,200	274	514
Table 1: Statistics of our recipe pairs data (2.3)
2.3 Recipe Pairs for Training
Given N text recipes and M video recipes for a dish, we pair each text recipe with every other text recipe to get  $O(N^2)$  text-text recipe pairs. Similarly, we pair each text recipe with every video recipe to get  $O(N * M)$  text-video recipe pairs, and pair each video recipe with every other video recipe to get  $O(M^2)$  video recipe pairs. On closer inspection, we find that some of these pairs describe recipes that are very different from one other, making a reasonable alignment almost impossible. For example, one black bean soup recipe might require the use of a slow cooker, while another describes using a stove. We therefore prune these recipe pairs based on the match of ingredients and length<sup>8</sup> to finally yield a set of 63,528 text-text recipe pairs, 65,432 text-video recipe pairs and 19,988 videodvdoe recipe pairs. We split this into training, validation and test split at the dish level. Table 1 shows the number of dishes and pairs in each split.
3 Recipe Alignment Algorithm
We first describe our unsupervised pairwise alignment model trained to learn alignments between text-text, text-video, and video-video recipes pairs. We then describe our graph algorithm, which derives joint alignments between multiple text and video recipes given the pairwise alignments.
3.1 Pairwise Alignments between Recipes
Our alignment algorithm is based on prior work (Naim et al., 2014) that learns to align a sequence of natural language instructions to segments of video recording of the same wet lab protocol. They first identify the nouns in the text sentences and the blobs (i.e. objects) in video segments. Given the blobs from  $M$  video segments  $F = [\mathbf{f}^{(1)},\dots,\mathbf{f}^{(M)}]$  and the nouns from  $N$  sentences  $E = [\mathbf{e}^{(1)},\dots,\mathbf{e}^{(N)}]$ , the task is to learn alignments between video segments and text sentences. They propose a hierarchical generative model which first uses a Hidden Markov Model
Details in supplementary.
4 Details in supplementary.
5https://azure.microsoft.com/en-us/services/cognitive-servic speech-to-text/
${}^{7}$  Classifier achieves 85% F1-score on a held out test set.
Figure 4: A maximum span tree for fried rice dish with text instructions and transcript segments as nodes, alignments as edges, and alignment probabilities as edge weights. Nodes representing text instructions are labeled "T". Nodes representing transcript segments are labeled "V". Each color indicates a different recipe. The bounding box shows a magnified section of the tree with edge weights and the instruction/transcript associated with each node.
(HMM) (Rabiner, 1989; Vogel et al., 1996) to generate each video segment  $f^{(m)}$  from one of the text sentences  $e^{(n)}$ . They then use IBM1 model (Brown et al., 1993) emission probabilities to generate the blobs  $\{f_1^{(m)},\dots,f_J^{(m)}\}$  in  $f^{(m)}$  from the nouns  $\{e_1^{(n)},\dots,e_I^{(n)}\}$  in  $e^{(n)}$  as follows:
$$
P \left(\mathbf {f} ^ {(m)} \mid \mathbf {e} ^ {(n)}\right) = \frac {\epsilon}{(I) ^ {J}} \prod_ {j = 1} ^ {J} \sum_ {i = 1} ^ {J} p \left(f _ {j} ^ {(m)} \mid e _ {i} ^ {(n)}\right) \tag {1}
$$
The hidden state in the HMM model corresponds to the alignment between video segment and text sentence, and the state transition probabilities correspond to the jump between adjacent alignments. For computational tractability, a video segment can be aligned to only one sentence (multiple sentences can align to the same video segment)
We use this algorithm to learn pairwise alignments between text-text, text-video and video-video recipes. Given two recipes (source and target) of the same dish, we define our alignment task as mapping each text instruction (or video transcript sentence) in the source recipe to one or more text instructions (or video transcript sentences) in the target recipe.
We make two modifications to the alignment algorithm described above: First, our recipe pairs, unlike the wet lab protocol data, does not follow the
same temporal sequence. The alignment algorithm must thus learn to jump within a longer range. We set the window of jump probabilities at  $[-2, 2]$ .<sup>9</sup> Second, we use transcriptions to learn alignments rather than the objects detected in videos. We hypothesize that the richness of language used in instructional videos may facilitate better alignment with transcripts (as others have observed (Malmaud et al., 2015; Sener et al., 2015)). We use all words (except stop words) in video transcript sentences and all words in text instructions while learning the IBM1 word level probabilities. An instruction in one recipe can be aligned to multiple instructions in the other recipe.
3.2 Joint Alignment among Multiple Recipes
We use the pairwise alignments to derive a joint alignment at the dish level between multiple text and video recipes. For each dish, we construct a graph where each node represents an instruction from a text recipe or a transcript sentence from a video recipe. We use the pairwise alignments to draw edges between nodes, with alignment probabilities as the edge weights. We include only those edges that have alignment probability greater than 0.5. The pairwise alignments are directed since they go from the source recipe to the target recipe.
<sup>9</sup>We find that increasing the window beyond 5 decreases performance.
We first convert the directed graph into an undirected graph by averaging the edge weights between two nodes and converting directed edges into undirected edges. Note that the resultant graph can have multiple connected components as some recipe pairs may not have any instructions aligned with probability greater than the threshold of 0.5
Our goal is to find a set of jointly-alignable instructions across different recipes. We therefore convert the graph (with cycles) into a forest by running the maximum spanning tree algorithm on the graph. Figure 4 shows an example tree derived for one of the dishes. A path in this tree, that has at most one node from each recipe, constitutes a set of jointly-alignable instructions. For example, in the magnified section of the tree in Figure 4, all unique colored nodes in the path from the yellow node to the green node constitute a set of jointly-alignable instructions.
We describe how we evaluate our pairwise alignment algorithm (from §3.1). We answer the following research questions using our experimentation:
1. How does our alignment model perform when evaluated on human-aligned recipe pairs?
2. Does our unsupervised alignment model outperform simpler non-learning baselines?
3. How does performance differ when we use only nouns or nouns and verbs instead of all words to learn alignments?
4.1 Human Aligned Evaluation Set
We evaluate our pairwise alignment algorithm on the following two human annotated datasets:
YouCook2 text-video recipe pairs The YouCook2 dataset (Zhou et al., 2018a) consists of 1,625 cooking videos paired with human-written descriptions for each video segment. These span 90 different dishes. We transcribe all videos using the Microsoft Speech-to-Text Cognitive service $^{10}$  and separate it into sentences using a sentence tokenizer. Given a sequence of human-written descriptions and a sequence of transcript sentences, the alignment task is to align each transcript sentence to one of the human-written descriptions. We train our pairwise alignment model on the train split of our text-video recipe pairs (from
$\S 2.3)$  and evaluate on the YouCook2 dataset. An important difference between the text-video pairs in YouCook2 and in our data is that in YouCook2, the text instructions and the video segments are temporally aligned since the text instructions were specifically written for the videos. In our data, however, the text and the video recipes can differ in order.
CommonCrawl text-text recipe pairs We randomly choose 200 text-text recipes pairs (spanning 5 dishes) from the test split of our data ( $\S 2.3$ ) and collect alignment annotations for them using six human experts. We show annotators a numbered list of the instructions for the target recipe (along with its title and ingredients). We display instructions for the source recipe with input boxes besides them and ask annotators to write in the number(s) (i.e. labels) of one or more target instruction(s) with which it most closely aligns. Each recipe pair is annotated by three annotators. For  $65\%$  of the instructions, two or more annotators agree on a label. For only  $42\%$  of the instructions do all three annotators agree, suggesting that the difficulty level of this annotation task is high. We train our pairwise alignment model on the train split of our text-text recipe pairs ( $\S 2.3$ ) and evaluate on the 200 human-aligned pairs.
Baselines described below align each instruction in the source recipe to one or more instructions in the target recipe.
Random We align each instruction in the source recipe to a random instruction in the target recipe.
Uniform alignment Given  $N$  instructions in the target recipe, we divide the instructions in the source recipe into  $N$  equal chunks and align each instruction in the  $i^{th}$  chunk of the source recipe to the  $i^{th}$  instruction in the target recipe. For instance, given a source recipe  $[S1, S2, S3, S4]$  and a target recipe  $[T1, T2]$ , uniform alignment would align  $S1$  and  $S2$  to  $T1$  and  $S3$  and  $S4$  to  $T2$ . More generally, we align the  $i^{th}$  instruction in the source recipe to the  $\left[\left(\frac{N}{M} i\right)^{th} - \left(\frac{N}{M} (i + 1)\right)^{th}\right)$  instruction in the target recipe.
BM25 retrieval We use BM25 (Robertson et al., 2009) as our information retrieval baseline. Given
<sup>11</sup>We use the term "instruction" to mean both text instruction and transcript sentence.
10 https://azure.microsoft.com/en-us/services/cognitive-services/speech-to-text/
Methods	Precision	Recall	F1
Random	18.53	14.47	14.49
Uniform alignment	63.44	50.81	53.10
BM25 retrieval	48.86	39.85	38.91
Textual Similarity
Exact word match	46.75	40.70	40.06
TF-IDF	46.82	39.23	38.55
GloVe	46.13	38.74	37.14
BERT	48.83	41.48	40.89
RoBERTa	50.21	42.43	42.28
HMM+IBM1
Nouns	78.63	63.83	65.29
Nouns+Verbs	80.56	67.90	69.00
All words	81.39	69.27	70.30
a source and a target recipe pair, we construct a corpus using all instructions in the target recipe. We then use each source instruction as a query to retrieve the top most instruction from the target instruction corpus and align the source instruction to the retrieved target instruction.
Textual similarity Given a source recipe instruction and a target recipe instruction, we define a measure of textual similarity between the two instructions using the following five methods. For each source instruction, we compute its similarity score with every target instruction and align it to the target instruction with the highest score.
a. Exact word match: Given two instructions, we define exact word match as the ratio of the number of common words between the two divided by the number of words in the longer of the two. This gives us a measure of word match that is comparable across instructions of different lengths.
b. TF-IDF: We use all the recipes in our training set to create a term frequency (TF)-inverse document frequency (IDF) vectorizer. Given an instruction from the evaluation set, we compute the TF-IDF vector for the instruction using this vectorizer. Given two instructions, we define their TF-IDF similarity as the cosine similarity between their TF-IDF vectors.
c. GloVe: We train GloVe embeddings (Pennington et al., 2014) on an in-domain corpus of 3 million words put together by combining text recipes and video transcriptions. Given an instruction, we average the GloVe embeddings (Pennington
Table 2: Results for text-video recipe alignments on YouCook2 dataset.
Methods	Precision	Recall	F1
Random	14.26	14.00	12.69
Uniform alignment	41.38	31.85	33.22
BM25 retrieval	50.06	55.27	49.30
Textual Similarity
Exact word match	53.90	48.39	46.98
TF-IDF	52.78	46.82	45.12
GloVe	56.04	51.89	50.30
BERT	50.72	55.07	49.10
RoBERTa	52.49	55.86	50.44
HMM+IBM1
Nouns	62.11	48.99	50.73
Nouns+Verbs	64.72	50.76	52.97
All words	66.21	52.42	54.55
Table 3: Results for text-text recipe alignment on Common Crawl dataset.
et al., 2014) of nouns and verbs $^{12}$  to obtain its embedding vector. Given two instructions, we define their embedding similarity as the cosine similarity of their embedding vectors.
d. BERT: Given an instruction, we compute its embedding vector using BERT-based sentence embedding (Reimers and Gurevych, 2019). We experiment with different variants and find that the BERT-base model trained on AllNLI, then on STS benchmark training set<sup>13</sup> performed the best for us. Given two instructions, we define their BERT similarity as the cosine similarity between their sentence embedding vectors.
e. RoBERTa: We also experiment with a variant of the above baseline where we use RoBERTa (Liu et al., 2019) instead of BERT to compute the sentence embeddings. We use RoBERTa-large trained on AllNLI, then on STS benchmark training set.
4.3 Model Ablations
We experiment with the following ablations of our unsupervised pairwise alignment model (§3.1):
HMM+IBM1 (nouns) We use the NLTK $^{14}$  part-of-speech tagger to identify all the nouns in an instruction and only use those to learn the IBM1 word-level alignments. This ablation is similar to the model proposed by Naim et al. (2014) that align objects in videos to nouns in text.
12We find that using only nouns and verbs outperforms using all words.
13https://pypi.org/project/
sentence-transformers/
14 https://www.nltk.org/
HMM+IBM1 (nouns and verbs) We use both nouns and verbs to learn IBM1 word-level alignments. This ablation is similar to the method used in Song et al. (2016) that align objects and actions in videos to nouns and verbs in text.
HMM+IBM1 (all words) We use all words (except stop words) in the source and the target recipe instructions to learn the word-level alignments.[15]
4.4 Evaluation Metrics
Given  $M$  source recipe instructions and  $N$  target recipe instructions, the alignment task is to label each of the  $M$  source instructions with a label from  $[0,\dots,(N - 1)]$ . Given a predicted sequence of labels (from baseline or proposed model) and a reference sequence of labels (from human annotations) for a recipe pair, we calculate the weighted-average<sup>16</sup> precision, recall and F1 score. We average these scores across all alignment pairs to compute aggregate scores on the test set.
4.5 Results
On text-video alignments Table 2 shows results of our pairwise alignment algorithm compared with baselines on 1,625 human aligned text-video recipe pairs from YouCook2. The BM25 baseline outperforms two of the textual similarity baselines. Within the textual similarity baselines, RoBERTa outperforms all others suggesting that a pretrained sentence level embedding acts as a good textual similarity method for this alignment task. The uniform alignment baseline, interestingly, outperforms all other baselines. This is mainly because in the YouCook2 dataset, the text instructions and the transcript sentences follow the same order, making uniform alignment a strong baseline. Our unsupervised HMM+IBM1 alignment model significantly outperforms (with  $p < 0.001$ ) all baselines. Specifically, it gets much higher precision scores compared to all baselines. Under ablations of the HMM+IBM1 model, using all words to learn alignments works best.
On text-text alignments Table 3 shows results of our pairwise alignment algorithm compared with baselines on 200 human-aligned text-text recipe pairs from Common Crawl. Unlike text-video alignments, we find that the uniform alignment