SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Triantafyllos Afouras* Effrosyni Mavroudi* Tushar Nagarajan Huiyu Wang Lorenzo Torresani
{afourast, emavroudi, tusharn, huiyuw, torresani}@meta.com
We introduce HT-Step, a large-scale dataset containing temporal annotations of instructional article steps in cooking videos. It includes 116k segment-level annotations over 20k narrated videos (approximately 2.1k hours) of the HowTo100M dataset. Each annotation provides a temporal interval, and a categorical step label from a taxonomy of 4,958 unique steps automatically mined from wikiHow articles which include rich descriptions of each step. Our dataset significantly surpasses existing labeled step datasets in terms of scale, number of tasks, and richness of natural language step descriptions. Based on these annotations, we introduce a strongly supervised benchmark for aligning instructional articles with how-to videos and present a comprehensive evaluation of baseline methods for this task. By publicly releasing these annotations and defining rigorous evaluation protocols and metrics, we hope to significantly accelerate research in the field of procedural activity understanding.
1. In a large mixing bowl, whisk together the flour, salt, and baking soda. Mixing your dry ingredients together separately ensures that ...
2. Beat the butter in a separate bowl until it becomes light and fluffy. You can do this using a whisk or an electric mixer.
3. Beat the white and brown sugar for about 2 minutes. You might find it easier to mix if you add the sugar in a little bit at a time ...
4. Whisk in the vanilla extract and the milk. Keep whisking until everything is evenly combined.
5. Beat in the flour mixture. At this point, the mixture will start to stick together to form a ball. If you were using a whisk, ..
Figure 1: HT-Step is a dataset of annotations aimed towards instructional video-article alignment. Given an untrimmed how-to video and an instructional article containing a list of step descriptions, the goal is to temporally localize the steps shown in the video and reject the ones not demonstrated. Our dataset includes over 116k segment-level annotations for this task, as well as labels indicating whether a temporal segment is a full or a partial match to the step, i.e., whether the segment demonstrates all substeps and ingredients mentioned in the step headline (shown in **bold**).
Instructional videos have become a popular way to learn or improve our skills, providing an entertaining and dynamic alternative to traditional written manuals. For example, how-to videos are listed as one of the top-four watched categories on the YouTube platform [1], where viewers can access narrated visual demonstrations for a wide range of activities, from cooking to crafts and DIY projects. In addition to being a valuable learning medium for humans, how-to videos are a great source of training data for computer vision models [6, 25, 30]. Since many of these instructional videos focus on complex activities entailing a sequence of steps, these repositories offer great potential for learning task graphs [47] and training procedural activity models [42, 50]. However, although the videos are often accompanied by informative closed captions or ASR transcriptions, they lack structured step annotations that clearly denote which video segments correspond to what steps of the activity.
An alternative source of information for acquiring new skills is instructional article collections, such as wikiHow [20]. These typically provide a list of procedural steps and have the advantage of being more comprehensive, incorporating variations and detailed explanations. Although they often contain still-pictures or graphic illustrations, they lack the dynamic demonstrations offered by videos.
In this work we introduce HT-Step, the largest existing dataset with labeled step segments on instructional videos. We leverage wikiHow articles [20] to guide the annotation on a large subset of the open-sourced HowTo100M dataset [31]. These cross-dataset associations provide us with a taxonomy of steps well fitting the data and with an automatic mechanism to identify the set of potential steps represented in each individual video. Through a meticulous process of manual labeling by professional annotators we obtain 116k temporal step labels over 20k videos (approximately 2.1k hours). Our benchmark significantly surpasses the largest existing datasets of labeled steps [41, 50] in procedural videos along multiple axes: scale ( $2.6 \times$  the number of segments annotated,  $4.8 \times$  the number of video hours), taxonomy granularity ( $7 \times$  the number of steps), and scope ( $2 \times$  the number of activities).
We share these annotations with the research community and present a benchmark for the task of temporal article grounding. Given as input a how-to video and an instructional article listing a sequence of potential steps, the objective is to temporally ground the subset of steps that are demonstrated in the video and to detect outliers (i.e., the steps not executed). We note that this task differs from traditional temporal grounding task which require localizing a single query that is known to be present in the video. Instead, in our problem setting the system is given the complete sequence of potential steps. We believe that this formulation may encourage the design of methods that can perform global reasoning over all the steps and take full advantage of ordering constraints and sequence priors to disambiguate the grounding. Furthermore, on average about  $58\%$  of the steps listed in the associated wikiHow article are not demonstrated in the given video. This implies that the model must have strong capability to recognize the steps that are not boundable. Finally, we include in our benchmark a test set over unseen tasks, i.e., activities that are not represented in the training set. Traditional activity detection methods requiring a taxonomy known a priori cannot generalize to unseen activities. By introducing this setting we hope to promote the development of language-based temporal alignment models that can learn to ground the rich textual descriptions of steps in instructional articles even for never-seen tasks.
In summary, our contributions are the following: (1) We introduce a large collection of step annotations on videos from the popular HowTo100M dataset. (2) We propose a scalable pipeline for annotating procedural videos that leverages an automatic strategy for associating potential steps to video and for inferring the step taxonomy. (3) We introduce splits, evaluation protocols and metrics that evaluate the ability of models to predict the temporal extent of steps in videos, and reject steps that are not visually groundable. (4) We train and evaluate state-of-the-art approaches and baselines that tackle our task from three different angles: activity detection, single-sentence temporal grounding, and temporal article grounding. (5) We discuss properties and unique aspects of HT-Step through comprehensive analyses and experiments.
Procedural and Instructional Video Datasets. Procedural videos portray humans performing sequential steps in some constrained yet non-unique order, with the aim of achieving a specific goal, such as preparing a dish. Understanding the content of such videos has been an active area of
Table 1: Comparison between HT-Step and existing procedural video understanding (top), temporal grounding (middle), and instructional video (bottom) datasets. Note that HT-Step seen validation and test sets introduced in [28] are subsumed by HT-Step.
Dataset	Duration (h)	# Videos	# Segments	# Activities	Domain
Procedural Activity Datasets
MPII [35]	10	44	5.6k	-	cooking
50Salads [40]	4.5	50	899	-	cooking
TACoS [34]	~ 600	127	18.8k	-	cooking
Breakfast [22]	77	2k	8.5k	10	cooking
Ikea-FA [16]	4	101	1.9k	1	furniture
EPIC-KITCHENS [9]	100	700	90k	-	cooking
EgoProcel [3]	62	298	1k	16	multiple
Assembly101 [36]	513	4.3k	1M	101	assembly
Temporal Grounding Datasets
ActivityNet Captions [21]	849	20k	100k	-	multiple
Charades-STA [14]	~ 90	10k	18k	157	multiple
QVHighlights [23]	845	10.2	16.1k	-	multiple
Instructional Video Datasets
YouCook [10]	2	88	-	-	cooking
YouCook2 [48]	176	2k	13.8k	-	cooking
YouwikiHow [8]	-	47k	-	1398	multiple
CrossTask [50]	213	2.8k	21k	18	multiple
COIN [41]	476	11.8k	46.4k	180	multiple
HT-Step Seen Val+Test Set [28]	124	1.2k	7k	177	cooking
HT-Step (Ours, Full)	2.1k	19.7k	116k	433	cooking
HT-Step splits
HT-Step Train	1.9k	17.4k	103k	401	cooking
HT-Step Val seen	64	600	3.4k	120	cooking
HT-Step Test seen (S1)	61	600	3.6k	120	cooking
HT-Step Test unseen (S2)	116	1000	5.7k	32	cooking
research over the last decade [2, 6, 11, 13, 22, 34, 37, 47], largely driven by the release of benchmark datasets [22, 41, 50]. Early datasets such as Breakfast [22] and 50Salads [40] are manually recorded, small-scale (spanning at most 100 hours) and limited in activity diversity (e.g., covering only breakfast recipes [22] or IKEA furniture assembly [16]). More recent datasets, such as COIN [41] and CrossTask [50], have capitalized on public video collections, including videos sourced from YouTube, in order to substantially increase dataset scale and activity diversity. A key characteristic of these videos is that they are instructional, i.e., they typically involve an individual teaching a complex task by narrating the sequence of steps.
Our HT-Step dataset also provides temporal annotations for steps of instructional videos. However, it surpasses existing datasets in terms of scale and activity diversity, as the steps are sourced from real-world instructional articles (wikiHow [20]) rather than being manually defined. This approach leads to much richer step descriptions, including a paragraph providing detailed instructions. Despite its exclusive focus on the cooking domain, HT-Step is significantly larger in scale and step description complexity. Our work significantly expands the small-scale val/test set that was recently released [28] for evaluating step grounding approaches trained without annotated temporal segments. While the focus of that work was the design of weakly supervised methods leveraging narrations as "free" signal for step-to-video alignment, here we share the first large-scale dataset enabling strongly-supervised training of methods for temporal article grounding. Our annotations over a large-scale training set, and a new test set of unseen tasks (not included in the training set) provide almost  $20 \times$  more annotated videos and number of segments. We also include additional metadata per step (indicating full or partial segment-step matches). Furthermore, we include a comprehensive experimental evaluation of strongly supervised baselines on this task.
Instructional Video Understanding There exist various tasks related to instructional video understanding, such as step classification [25], step detection [41, 46], action segmentation [13], captioning [49], video retrieval [15], visual object grounding [29], and temporal grounding [26]. In this paper, we focus on the recently-introduced task of temporal article grounding, which has been tackled only with weakly-supervised approaches so far [8, 28]. In particular Chen et al. [8] is the first work to train and evaluate models for the temporal article grounding task. However this work was limited to weakly-supervised training due to the lack of a strongly labeled dataset, and used CrossTask as a proxy dataset for evaluation, with noisy mapping between article steps and CrossTask labels. In contrast, we introduce a large amount of strong temporal segment annotations
Figure 2: Sample frames for 12 videos spanning 4 tasks from the HT-Step training set. For each video, we show relevant frames for three steps. By annotating multiple videos of the same task, we obtain multiple instances of each step, which have high variations in appearance and viewpoints.
for training, and propose a metric that evaluates both the recall and precision of models, and is thus better suited to the task of temporal article grounding which involves multiple ungroundable steps. The temporal article grounding task is closely related to two other grounding tasks: (a) temporal grounding of a single natural language query [14, 23] (also known as moment retrieval) and (b) video paragraph grounding [4]. Benchmarks [14, 21, 23] for these tasks typically consist of video captions or questions that are loosely related to each other and which were obtained by human annotators based on the video contents. Thus, most approaches assume that each of the queries is groundable in the video. For example, video paragraph grounding aims at predicting a single temporal segment for each sentence of the paragraph. Instead, in our HT-Step benchmark the step queries originate from wikiHow articles, which might contain steps that are not shown in the video (non-groundable steps) or steps that are partially executed (partial matches). This makes it challenging to adapt temporal grounding approaches for temporal article grounding in our benchmark.
3 HT-Step
The creation of HT-Step involved pairing videos from HowTo100M with instructional articles from wikiHow and then annotating segments in each video with steps of the associated article. wikiHow $^2$  is an online knowledge base that houses thousands of articles with step-by-step instructions for a large variety of activities. Every article is typically structured as a series of steps, each step anchored by a descriptive headline and further supplemented with a detailed paragraph. HowTo100M, on the other hand, contains an extensive compilation of instructional videos that were collected using titles of wikiHow articles as search keywords for a subset of activities deemed to contain visual demonstrations. As a result of this sourcing procedure, each video in HowTo100M comes with a task label that corresponds to a specific wikiHow article. These pairings create a natural opportunity for annotating the HowTo100M videos with the step labels of the corresponding wikiHow articles, which we exploited for creating HT-Step. Next we describe in detail our annotation process.
3.1 Annotation workflow
Activity selection. HowTo100M spans a variety of domains, including cooking, DIY, crafts, sports, and gardening. As in previous works [17], we chose to focus on cooking activities as i) they account for approximately a third of the HowTo100M videos, and ii) they are relatively low-complexity tasks which can be annotated by non-experts. Using metadata, we selected 495 cooking activities (detailed list in the Appendix) for which HowTo100M contains at least 70 videos.
Step Taxonomy. We automatically sourced the step descriptions for all the activities from wikiHow. For every activity, we extracted the steps from the corresponding wikiHow article, forming a step taxonomy that includes 4,958 unique steps. We list the full taxonomy in the Appendix.
$^{2}$ https://www.wikihow.com/
Video validation. For every video, the annotators were given the corresponding activity title and step taxonomy. They were then asked to watch the whole video and determine whether it contains the specified activity. If the video was found not matching the task, it was rejected.
Temporal annotation. During the temporal annotation loop annotators were asked to provide temporal boundaries for the wikiHow steps represented in all the non-rejected videos. An example annotation is illustrated in Figure 1.
Task variations. wikiHow articles often list several variations for a given task. For example, the instructional recipe for "Make Taco Salad" includes three versions: basic, deluxe and vegetarian. For such activities, we asked annotators to choose the variation that best fits the video and to annotate only the steps listed in that variation.
Full vs partial matches. We asked the annotators to mark a step as a "full" match if (i) all the instructions and ingredients listed in the step headline are shown in the segment and (ii) the step is brought to full completion. Conversely, "partial" indicates that some sub-steps in the step headline are skipped, some ingredients are not used, or that a step is interrupted and continued later.
QA process. To ensure the quality of the annotations, we followed a rigorous multi-stage Quality Assurance (QA) process. In the first stage, the videos were annotated by a single annotator. These initial annotations were then reviewed by more experienced annotators, who either approved all the annotations on a given video (all the marked steps were correct and no steps were missing) or marked it for re-doing, with specific comments indicating the necessary correction. At the last stage of the QA process, the annotations that were marked as incorrect were redone by a third set of annotators. Due to budget constraints, the full QA was enforced on a sample of roughly  $13\%$  of the total annotations.
Annotation output. The annotation process took approximately 33k person-hours by 90 professional human annotators. Overall,  $34\%$  of the videos were rejected as not matching the specified activity, resulting in annotations for 433 of the initial 494 activities. We summarize the statistics of the resulting annotations and compare to existing datasets in Table 1.
3.2 Training, validation, and test splits
We formed two types of test sets: one for seen activities, i.e., activities that are also included in the training set, and one for unseen activities, i.e., novel activities not appearing in the training set. For seen activities we also formed a validation (val) set containing the same activities.
Seen val/test set (S1). We directly adopt the HT-Step validation and test splits [28] as the seen evaluation sets of our dataset, each containing 600 videos, 5 samples for each of 120 activities.
Unseen test set (S2). To create the unseen test set, we selected 32 activities that do not appear in the training set or the seen val/test set. We include all the annotations on videos of these activities that went through the QA review, resulting in a test set of 1000 annotated videos. This set is imbalanced w.r.t. activities, with the video count per activity ranging from 17 to 70.
Training set. All the other annotated videos are included in the final training set, which contains more than 17k videos and 100k annotated segments. We summarize the split statistics in the bottom part of Table 1.
Article steps. As previously mentioned, step descriptions in wikiHow articles have two parts: a headline and a paragraph. The headline is brief and may not reveal enough information for grounding; the paragraph contains additional details about the execution of the step. Annotators were asked to localize each step by considering both. This allowed us to obtain annotations where the paragraph description is essential for grounding the steps. We show some examples in Figure 3.
Composite steps. Many wikiHow steps are composite, i.e., include multiple sub-steps. Two examples of such steps are shown in Figure 4 (b,c). We introduced the labeling of partial vs full matches (discussed in Section 3.1) to account for the fact that composite steps are rarely fully represented in videos or they may be carried out in disjoint time intervals of the video.
Non-groundable steps. Some of the steps listed in the wikiHow article may not be represented at all in the video. An example is illustrated in Figure 1. Temporal grounding models that are trained/evaluated on this data must support handling queries with no associated temporal segments.
3.4 Statistics
In this section we provide insightful statistics about the dataset. See Appendix for more.
Dice the celery, carrot, and pickle. Use a sharp knife to carefully dice enough red onion to measure 1/4 cup (40 g). You'll also need to dice enough celery to measure 1/4 cup (55 g) and enough pickle to measure 1/4 cup (35 g). Transfer the vegetables to the bowl with the mashed chickpeas.
Make the cake ball dough. Place two-inch (5-cm) cubes of cake into the food processor and pulse until the cake is broken up into fine crumbs. Transfer the crumbs to a mixing bowl. [...] If you don't have a food processor, place the chunks of cake into a mixing bowl and mash them with a fork or break them up with your hands.
Figure 3: Qualitative examples where the paragraph is essential to localize the step. The paragraph sentences that are visually demonstrated in the video are highlighted in yellow.
Figure 4: Examples of composite steps and full/partial matches. Frames annotated as "full matches" are shown in green, "partial matches" are shown in orange. (a) An atomic step, fully matched. (b) A composite step: it is labeled as a full match as all the ingredients and the substeps mentioned in the paragraph are shown in the video. (c) Another composite step: here not all the described ingredients are utilized and thus it is labeled as a "partial match." (d) An example of step fragmentation: the step is executed over two disjoint segments which are consequently labeled as "partial matches."
Overview. Table 1 summarizes the statistics of HT-Step. It is larger and has a much richer step taxonomy than existing datasets. Compared to COIN, it has  $4.8 \times$  the duration,  $2.6 \times$  the number of annotated segments,  $7 \times$  the taxonomy size (4958 vs 778 unique steps), and  $2 \times$  the number of videos. Segment duration. According to Figure 5a, the average segment duration is approx. 15 seconds, with most of the annotated segments at 10 seconds or shorter and a long tail of rare longer segments. Video coverage. We define video coverage as the percentage of a video that is covered by annotations. Figure 5b shows the distribution of the average video coverage per activity, with a mean of  $22\%$ .
Step coverage. Step coverage denotes the fraction of unique steps from the associated wikiHow article that are annotated in a video and is on average  $42\%$ . Figure 5c shows that even for the activities with the highest step coverage, many article steps have no relevant temporal segments in each video. Distribution of segments over activities. Figure 5d shows how the number of annotations is distributed over the 4,958 steps. On average we get 24.6 segments per step. Since the choice of steps to be annotated was determined by the data, several steps have very few training samples, as indicated by the large concentration in the first bucket. We believe that this property will encourage the development of methods that can perform zero- or few-shot detection, as well as techniques that approach the task as language-based temporal grounding as opposed to class-based detection.
Length of text descriptions. We calculate the number of words as a proxy for the complexity of the step descriptions. In Figure 5e we provide the distribution of this metric averaged over activities, for HT-Step and COIN [41]. It is clear that HT-Step contains richer text descriptions, even when only the headlines are used (8.5 words vs 4.9 for COIN on average). When paragraph text is included in this calculation, the word count increases even further, to 17.1 words on average.
Step ordering variation. The steps in a wikiHow article are listed according to a recommended order of execution. In order to capture real-world deviations from these canonical orderings, we measure the Normalized Edit Distance (NED) between the annotated step sequence and the wikiHow listing, normalized over the length of the annotated sequence in each video. We calculate the mA-NED as the mean of the average NED per activity. We compute the same metric for the COIN dataset, using the ordering provided with the taxonomy of this dataset. Steps in our dataset are much more frequently executed out of order compared to COIN (mA-NED is 0.37 vs 0.27), as shown in Figure 5f. This presents a much more challenging temporal grounding setting.
(a) Distribution of segment durations. The average segment duration is about 15 seconds, with most of the annotated segments at 10 seconds or shorter, and a long tail of rare longer segments.
(b) Distribution of video coverage, averaged per activity. Video coverage denotes the percentage of a video that is covered by step annotations. It exhibits a normal distribution around the mean.