SlowGuess/ACL_2020_OCR huggingface dataset:
baseline does not outperform textual similarity baselines, suggesting that the different re-orderings between text-text recipe pairs makes alignment more challenging. Within textual similarity baselines, similar to text-video alignment, RoBERTa outperforms all others. We believe this is because text recipes tend to share similar vocabulary, making it easier to find similar words between two textual instructions. Video narrators tend to use more colloquial language than the authors of text recipes, making it more difficult to learn alignments using word similarities. Interestingly, both BM25 and RoBERTa get higher recall than our best HMM+IBM1 model but they lose out on precision. This suggests that retrieval models are good for identifying more alignments, albeit with lower precision. Our unsupervised HMM+IBM1 model again significantly outperforms  $(p < 0.001)$  all baselines on F1 score. Under ablations of the HMM+IBM1 model, we again find that using all words to learn alignments performs best.
Comparing text-video and text-text alignment results On comparing Table 2 and Table 3, we find that textual similarity baselines have overall higher scores on the text-text alignments than the text-video alignments. Our HMM+IBM1 model, on the other hand, has overall higher scores on text-video alignments than on text-text alignments. We attribute this contrast to the fact that two text recipes have higher vocabulary similarities than a text and a video recipe, resulting in textual similarity baselines to perform well on text-text alignments. Our HMM+IBM1 unsupervised learning model is able to do better on text-video pairs where the word usage differences are higher. Furthermore, the text-video pairs from YouCook2 are temporally aligned whereas the text-text pairs from Common crawl have several re-orderings making the text-text evaluation set comparatively harder. The supplementary material includes an analysis of alignment outputs.
5 Data Release
We describe the data released in our MICROSOFT RESEARCH MULTIMODAL ALIGNED RECIPER CORPUS. In all our released data, for text recipes, we include the actual text of the instructions. Whereas, for video recipes, we release the URL to the YouTube video with timestamps corresponding to the aligned video segments.
$^{15}$ Experimental details of HMM+IBM1 model is in supplementary.  
 $^{16}$ Calculate metrics for each label, and find their average weighted by the number of true instances for each label.
Single Step	Multiple Steps
Beat eggs, oil vanilla and sugar together in a large bowl.	1. Beat eggs in large bowl until foamy.
2. Add sugar, oil and vanilla mix well.
Butter 2 loaf pans and bake 1 hour at 325 degrees.	1. Pour into greased muffin tins or loaf pans
2. Yields about 4 small loaves or 2 large.
3. Bake for 25 minutes.
Mix the zucchini, sugar, oil, yogurt and egg in a bowl.	1. Beat eggs, sugar, oil and vanilla.
2. Add zucchini.
Table 4: Three examples of single-step to multi-step breakdown from the pairwise alignments.
Figure 5: We plot the trade-off between the percentage of paraphrases extracted and the precision, recall and F1 score (as measured by human annotators) with increasing alignment probability threshold on 200 human-aligned text-text recipes pairs.
5.1 Pairwise and Joint Alignments
We release the pairwise alignments between recipes of the same dish (derived from § 3.1) for 4,262 dishes. This includes 63,528 alignments between text recipes, 65,432 alignments between text and video recipes; and 19,988 alignments between video recipes. We also release the joint alignments between multiple text and multiple video recipes within a dish (derived from § 3.2) for 4,262 dishes.
5.2 Textual and Visual Paraphrases
The pairwise alignment algorithm described in §3.1 gives alignment probabilities for each pair of instructions it aligns. We threshold on these alignment probabilities to retrieve textual and visual paraphrases. Since our goal is to extract large numbers of high quality paraphrases, we decide on the threshold value by looking at the trade-off between the percentage of paraphrases extracted and their quality as measured by human annotators on 200 human-aligned text-text recipe pairs from our evaluation set (§4.1).
Figure 5 shows the trade-off between the preci
sion, recall and F1 score and the percentage of paraphrases extracted with increasing threshold on instruction-level alignment probability. At 0.5 threshold, we extract  $60\%$  of the total alignments as paraphrases from our evaluation set. We use this threshold value of 0.5 on the pairwise alignments in the training, validation and test sets to extract a total of 358,516 textual paraphrases and 211,703 text-to-video paraphrases from 4,262 dishes and include it in our corpus.
5.3 Single-step to Multi-step breakdown
The pairwise alignments between text recipes include many instances where one instruction in one recipe is aligned to multiple instructions in another recipe with high alignment probability (greater than 0.9). Table 4 shows three such single-step to multi-step breakdown. We extract a total of 5,592 such instances from 1,662 dishes across the training, validation and test sets and include it in our corpus.
6 Applications of Our Corpus
We believe that our data release will help advance research at the intersection of language, vision and robotics. The pairwise alignment between recipes within a dish could be useful in training models that learn to rewrite recipes given ingredient or cooking method based constraints. The joint alignment over multiple text recipes within a dish should prove useful for learning the types of ingredient substitutions and instruction reordering that come naturally to expert cooks. The textual and visual paraphrases will, we believe, have implications for tasks like textual similarity, image and video captioning, dense video captioning and action recognition. The single-step to multi-step breakdown derived from our pairwise alignments may also prove useful for understanding task simplification, an important problem for agents performing complex actions.
Such multimodal data at scale is a crucial ingredient for robots to learn-from-demonstrations
of procedural tasks in a variety of environments. Collecting such large scale data is prohibitively expensive in robotics since it requires extensive instrumentation of many different environments. Other example applications are learning to ground natural language to physical objects in the environment, and catching when humans are about to commit critical errors in a complicated task and offering to help with corrective instructions.
Alignment Algorithms Our unsupervised alignment algorithm is based on Naim et al. (2014), who propose a hierarchical alignment model using nouns and objects to align text instructions to videos. Song et al. (2016) further build on this work to make use of action codewords and verbs. Bojanowski et al. (2015) view the alignment task as a temporal assignment problem and solve it using an efficient conditional gradient algorithm. Malmaud et al. (2015) use an HMM-based method to align recipe instructions to cooking video transcriptions that follow the same order. Our work contrasts with these works in two ways: we learn alignments between instructions that do not necessarily follow the same order; and our algorithm is trained on a much larger scale dataset.
Multi-modal Instructional Datasets Marin et al. (2019) introduce a corpus of 1 million cooking recipes paired with 13 million food images for the task of retrieving a recipe given an image. YouCook2 dataset (Zhou et al., 2018a) consists of 2,000 recipe videos with human written descriptions for each video segment. The How2 dataset (Sanabria et al., 2018) consists of 79,114 instructional videos with English subtitles and crowdsourced Portuguese translations. The COIN dataset (Tang et al., 2019) consists of 11,827 videos of 180 tasks in 12 daily life domains. YouMakeup (Wang et al., 2019) consists of 2,800 YouTube videos, annotated with natural language descriptions for instructional steps, grounded in temporal video range and spatial facial areas.
Leveraging Document Level Alignments Our work relies on the assumption that text recipes and instructional cooking videos of the same dish are comparable. This idea has been used to extract parallel sentences from comparable corpora to increase the number of training examples for machine translation (Munteanu and Marcu, 2005;
Abdul-Rauf and Schwenk, 2009; Smith et al., 2010; Grégoire and Langlais, 2018). Likewise, Talk-Summ (Lev et al., 2019) use the transcripts of scientific conference talks to automatically extract summaries. Zhu et al. (2015) use books and movie adaptations of the books to extract descriptive explanations of movie scenes.
Related Tasks A related task is localizing and classifying steps in instructional videos (Alayrac et al., 2016; Zhukov et al., 2019) where they detect when an action is performed in the video whereas we focus on describing actions. Dense event captioning of instructional videos (Zhou et al., 2018b; Li et al., 2018; Hessel et al., 2019) relies on human curated, densely labeled datasets whereas we extract descriptions of videos automatically through our alignments.
We introduce a novel two-stage unsupervised algorithm for aligning multiple text and multiple video recipes. We use an existing algorithm to first learn pairwise alignments and then use a graph-based algorithm to derive the joint alignments across multiple recipes describing the same dish. We release a large-scale dataset constructed using this algorithm consisting of joint alignments between multiple text and video recipes along with useful commonsense information such as textual and visual paraphrases; and single-step to multi-step breakdown.
Although our dataset focuses on the cooking domain, our framework should generalize to any domain with abundant volumes of unstructured-but- alignable multi-modal data. DIY (Do-It-Yourself) videos and websites, for instance, are an obvious next target. We also envision extending this work by including audio and video features to enhance the quality of our alignment algorithm. Ultimately, we believe this work will further the goal of building agents that can work with human collaborators to carry out complex tasks in the real world.
We would like to thank Harpreet Sawhney, Roshan Rao, Prasoon Goyal, Dilip Arumugam and Raymond J. Mooney for all their help. We would also like to thank the four anonymous reviewers for their useful comments and suggestions.
Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the use of comparable corpora to improve SMT performance. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 16-23, Athens, Greece. Association for Computational Linguistics.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. 2016. Unsupervised Learning from Narrated Instruction Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4575-4583.
Piotr Bojanowski, Rémi Lajugie, Edouard Grave, Francis Bach, Ivan Laptev, Jean Ponce, and Cordelia Schmid. 2015. Weakly-Supervised Alignment of Video with Text. In Proceedings of the IEEE international conference on computer vision, pages 4462-4470.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
DeepMind. 2019. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii.
Francis Grégoire and Philippe Langlais. 2018. Extracting parallel sentences with bidirectional recurrent neural networks to improve machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1442-1453, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jack Hessel, Bo Pang, Zhenhai Zhu, and Radu Soricut. 2019. A case study on combining ASR and visual features for generating instructional video captions. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 419-429, Hong Kong, China. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Guy Lev, Michal Shmueli-Scheuer, Jonathan Herzig, Achiya Jerbi, and David Konopnicki. 2019. Talk-Summ: A dataset and scalable annotation method for scientific paper summarization based on conference talks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2125-2131, Florence, Italy. Association for Computational Linguistics.
Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. 2018. Jointly Localizing and Describing Events for Dense Video Captioning. In Proceed
ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7492-7500.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nicholas Johnston, Andrew Rabinovich, and Kevin Murphy. 2015. What's cookin'? interpreting cooking videos using text, speech and vision. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 143-152, Denver, Colorado. Association for Computational Linguistics.
Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2019. Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477-504.
Iftekhar Naim, Young Chol Song, Qiguang Liu, Henry Kautz, Jiebo Luo, and Daniel Gildea. 2014. Unsupervised Alignment of Natural Language Instructions with Video Segments. In Twenty-Eighth AAAI Conference on Artificial Intelligence.
OpenAI. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv preprint arXiv:1912.06680.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-286.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4):333-389.
Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning Cross-modal Embeddings for Cooking Recipes and Food Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: A Large-scale Dataset for Multimodal Language Understanding. In NeurIPS.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. 2019. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model.
Ozan Sener, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. 2015. Unsupervised Semantic Parsing of Video Collections. In Proceedings of the IEEE International Conference on Computer Vision, pages 4480-4488.
Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 403-411, Los Angeles, California. Association for Computational Linguistics.
Young Chol Song, Iftekhar Naim, Abdullah Al Mamun, Kaustubh Kulkarni, Parag Singla, Jiebo Luo, Daniel Gildea, and Henry A Kautz. 2016. Unsupervised Alignment of Actions in Video with Text Descriptions. In IJCAI, pages 2025-2031.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1207-1216.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
Weiying Wang, Yongcheng Wang, Shizhe Chen, and Qin Jin. 2019. YouMakeup: A large-scale domain-specific multimodal dataset for fine-grained semantic comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5133-5143, Hong Kong, China. Association for Computational Linguistics.
Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018a. Towards Automatic Learning of Procedures from Web Instructional Videos. In Thirty-Second AAAI Conference on Artificial Intelligence, pages 7590-7598.
Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. 2018b. End-to-End Dense Video Captioning with Masked Transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8739-8748.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the IEEE International Conference on Computer Vision, pages 19-27.
Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberg Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3537-3545.
In this supplementary, we describe the details of our data collection process (§A.1), experimental details of our algorithm (§A.2) and provide analysis of our alignment outputs (§A.3).
A.1 Details of Data Collection
A.1.1 Common crawl Text Recipes
We use recipe data from Common Crawl  $^{17}$  that has metadata formatted according to the Schema.org Recipe schema  $^{18}$  including title, ingredients, instructions, and a URL to the recipe source. There were originally 3.2 million recipes extracted from Common Crawl. We filter the data by limiting the data to recipes with instructions written in English, removing recipes with titles that are longer than 5 words, removing duplicate recipes, removing recipes where the recipe title contains words that are not in the top  $50\%$  most common words
$^{17}$ https://commoncrawl.org/
$^{18}$ https://schema.org/Recipe
that occur in the recipe titles, and removing recipes with fewer than 2 steps. After filtering the data, we clustered the recipes into dishes using exact match on the recipe titles. We only retain recipes from dishes that have at least three recipes. The final dataset has a total of 4,262 dishes and 48,852 recipes with an average of 8 instructions per recipe.
A.1.2 YouTube Video Recipes
Given the dish names from the text recipes, we extract YouTube video recipes for each of the dishes. The number of videos extracted for each dish is proportional to the number of text recipes found for that dish. For instance, for a more popular dish like chocolate chip cookies, we would extract more text and video recipes than for a less popular dish like crème brulee. The number of videos extracted ranges from 3 to 100.
A.1.3 Chat/Content Classifier
Instructional cooking videos can contain a lot of non-instructional content ("chat"). For example, the person cooking the dish often introduces themselves (or their video channel) at the beginning of the video. They sometimes also introduce the dish they are going to prepare and suggest pairings for the dish. The non-instruction content are often found in the beginning and towards the end of the video but there are several instances of "chat" interspersed with instructional content as well. Since we wish to align these videos to text recipe instructions that do not contain non-instructional information, we need a way to remove non-instructional content. We train a supervised neural network based classifier for this task.
We train our classifier using the YouCook2 dataset (Zhou et al., 2018a) of 1,500 videos across 90 dishes. This dataset was created by asking humans to identify segments of a video that correspond to an instruction and annotate each segment with an imperative statement describing the action being executed in the video segment. We make the assumption that the transcript sentences that are included within an annotated video segment are instructional whereas those that are not included within an annotated video segment are noninstructional. We first transcribe all 1,500 videos in the dataset using a commercial transcription web service. We split the transcription into sentences using a sentence tokenizer. We label a transcript sentence with the label 1 if the corresponding video segment was annotated and with the label 0 if it
was not. We get a total of 90,927 labelled transcript sentences which we split by dishes into the training (73,728 examples), validation (7,767 examples) and test (9,432 examples) sets.
We use an LSTM (long-short term memory) model (Hochreiter and Schmidhuber, 1997) with attention (Luong et al., 2015) to train a binary classifier on this data. We initialize (and freeze) our 300-dimensional word embeddings using GloVe (Pennington et al., 2014) vectors trained on 330 million tokens that we obtain by combining all text recipes and transcript sentences. We use the validation set to tune hyperparameters of our LSTM classifier (hidden size: 64, learning rate: 0.00001, batch size: 64, number of layers: 1). Our chat/content classifier achieves 86.76 precision, 84.26 recall and 85.01 F1 score on the held out test set.
A.1.4 Recipe Pair Pruning Strategy
We define the following two pruning strategies to reduce the number of extracted recipe pairs:
Ingredient match: Each of our text recipes from Common Crawl contains an ingredients list. Video recipes from YouTube however do not contain ingredient lists. We therefore estimate the ingredients for video recipes using text recipes of the same dish. We construct a set of ingredients at the dish level by combining all ingredients of the text recipes within that dish. We then use this dish-level ingredients information to identify ingredient words from the words of video transcriptions. Given a recipe pair, we compare the ingredients of the two recipes and if the percentage of ingredients that match is below a threshold, we remove the pair. For text-text and text-video recipe pair, we set this threshold to be  $70\%$ , whereas for video-video recipe pair, we set this threshold to be  $90\%$  (since video-video recipe pairs tend to be more noisy).