SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yale Song Eugene Byrne Tushar Nagarajan Huiyu Wang Miguel Martin Lorenzo Torresani Fundamental AI Research (FAIR), Meta os://github.com/facebookresearch/ego4d-goals
Human activities are goal-oriented and hierarchical, comprising primary goals at the top level, sequences of steps and substeps in the middle, and atomic actions at the lowest level. Recognizing human activities thus requires relating atomic actions and steps to their functional objectives (what the actions contribute to) and modeling their sequential and hierarchical dependencies towards achieving the goals. Current activity recognition research has primarily focused on only the lowest levels of this hierarchy, i.e., atomic or low-level actions, often in trimmed videos with annotations spanning only a few seconds. In this work, we introduce Ego4D Goal-Step, a new set of annotations on the recently released Ego4D with a novel hierarchical taxonomy of goal-oriented activity labels. It provides dense annotations for 48K procedural step segments (430 hours) and high-level goal annotations for 2,807 hours of Ego4D videos. Compared to existing procedural video datasets, it is substantially larger in size, contains hierarchical action labels (goals - steps - substeps), and provides goal-oriented auxiliary information including natural language summary description, step completion status, and step-to-goal relevance information. We take a data-driven approach to build our taxonomy, resulting in dense step annotations that do not suffer from poor label-data alignment issues resulting from a taxonomy defined a priori. Through comprehensive evaluations and analyses, we demonstrate how Ego4D Goal-Step supports exploring various questions in procedural activity understanding, including goal inference, step prediction, hierarchical relation learning, and long-term temporal modeling.
Recognizing complex patterns of human activities has been the subject of extensive research in computer vision and the broader AI community [2, 25, 46, 8, 23, 22]. However, progress has been relatively slow compared to object and scene understanding [59, 32, 36, 62]. One of the main obstacles has been the scarcity of large-scale datasets annotated with a comprehensive taxonomy representing complex human activities. While object recognition benefits from WordNet [15] that provides an extensive taxonomy of objects found in everyday scenarios, activity recognition is presented with unique difficulties because there is currently no established taxonomy in place that encompasses the broadly varying granularities of activities, from atomic actions (e.g., pick-up cup, sit down) to procedural sequences (e.g., make lasagna).
In our quest to build a new dataset for human activity recognition, we draw inspiration from the psychology literature. Studies have shown the inherent hierarchical nature of human behavior [5, 12], comprising the primary goals at the highest level, intermediate steps and their substeps in the middle, and atomic actions at the lowest level. Social cognitive theories [3] suggest that this hierarchy is formed by human agents deliberately setting goals, anticipating potential consequences of different
Figure 1: Ego4D Goal-Step offers hierarchical procedural activity labels with three distinct levels: goal - step - substep. Each annotation comes with a time interval, categorical label, and natural language description. It also provides auxiliary information including step summaries, task completion status (is Continued), task relevance (is Relevant), and procedural activity indicator (is_procedural).
actions, and planning out a sequence of steps and their substeps to achieve the desired goal in a hierarchical manner. Although the planned sequence of actions may not necessarily align with the actual execution order [5], inferring and reasoning over the hierarchical representations has been shown to be crucial for understanding human behavior [12, 41].
Most existing activity datasets have focused on the lowest levels of this hierarchy – i.e., atomic or low-level actions – often in trimmed videos and with annotations spanning only a few seconds [25, 46, 8, 23]. This focus on atomic actions has even raised questions about the necessity of temporal modeling in existing video tasks [40, 11, 44], and its suitability for studying real-world videos containing higher-level activities over longer temporal extents.
In response to this, procedural activities – those that involve performing a series of steps to achieve predetermined goals – have recently gained particular attention [29, 60, 49, 61, 48, 34, 4, 43, 55]. Recognizing goal-oriented steps that unfold over a long time horizon requires modeling long-term temporal context, making it a challenging long-form video understanding task. However, existing datasets are either small-scale [29, 3, 43], do not model high-level goals, or ignore the hierarchical relationship between steps [49, 61, 4]. Furthermore, the step taxonomy is commonly built from external sources detached from videos (e.g., text articles from wikiHow [60, 49, 61]), resulting in misalignment between the constructed label space and the observed data. Consequently, a significant portion of the video is left unlabeled, offering an incomplete record of activities.
To address these limitations, we introduce Ego4D Goal-Step, a new set of annotations on Ego4D [22] with a newly developed hierarchical taxonomy for procedural activities. It contains two main components: (1) The goal annotation set consists of 7,353 videos, totaling 2,807 hours, labeled with a taxonomy of 319 goals grouped into 34 scenarios. This is a focused set covering the  $72\%$  of Ego4D videos and is intended to provide a large-scale training and evaluation dataset for goal inference. (2) The step annotation set focuses on the cooking scenario portion of Ego4D, and is intended specifically for procedural activity recognition. It consists of 47,721 densely labeled step and substep segments, amounting to 430 hours in total, annotated based on a taxonomy of 86 goals and 514 fine-grained steps commonly performed in diverse home kitchen environments across many countries.
Ego4D Goal-Step stands out from existing procedural video datasets for its several unique features. (1) We develop our taxonomy in a data-driven manner to accurately represent the activities, rather than
Dataset	Hier	Video Statistics	Segment Statistics	Taxonomy
Count (total)	Duration (total / avg)	Count (total / avg)	Duration (total /avg)	Density (Fg ratio)	# Goals	# Steps
EgoProceL [4]	329	62h / 13m	1K / 8.7	24h / 16.3s	0.38	16	139
YouCook2 [60]	2,000	176h / 5m	15K / 7.7	84h / 19.6s	0.48	89	n/a
CrossTask [61]	2,750	375h / 5m	21K / 7.6	56h / 9.6s	0.15	18	133
COIN [49]	11,827	468h / 2m	46K / 3.9	192h / 14.9s	0.41	180	778
Breakfast [29]	✓	2,193	85h / 2m	62K / 28	108h / 6.3s	0.88	10	225
Assembly101 [43]	✓	362	43h / 7m	83K / 237	156h / 1.7s	0.81	101	1,582
Ego4D Goal-Step	✓	851	368h / 26m	48K / 56	430h / 32.5s	0.77	86	514
Ego4D Goal-Step (goal labels)	7,353	2,807h / 23m	7546 / 1	2,470h / 19.6m	0.88	319	-
Table 1: Dataset statistics. We report "Ego4D Goal-Step" — the subset that comes with hierarchical step labels, and "Ego4D Goal-Step (goal labels)" — the full set which includes videos with goal labels but no step labels. "Hier" indicates datasets with hierarchical label spaces. Breakfast [29] and Assembly101 [43] provide both coarse- and fine-level segments, analogous to our step and substep segments; we report their combined numbers. Assembly101 [43] provides 12 synchronized views per recording; we report single view statistics to make the numbers compatible with other datasets.
resorting to external resources such as wikiHow. As a result, the average density of annotated segments per video is  $77\%$ , i.e.,  $2 - 5 \times$  higher than existing datasets that rely on external taxonomies [61, 49]. (2) The annotations form a three-level hierarchy (goal - steps - substeps), a unique feature that is unavailable in most existing datasets. When combined with existing Ego4D labels that are action-centric (e.g., narrations, FHO, and moments), it creates an attractive multi-level hierarchical label space. (3) Every annotation includes a time interval, a categorical label, and a natural language description, enabling both detection and language grounding tasks. (4) Procedural segments also come with step summary descriptions, useful for video summarization [38]. (5) Additional goal-oriented information, such as task relevance and completion status signals, supports novel research directions such as task graph inference and progress tracking [33]. (6) Our dataset inherits the large scale and diversity of Ego4D, capturing immersive views of procedural activities from a first-person perspective, and featuring long untrimmed videos that reveal the unfolding of individual goals over time. Together, these strengths make Ego4D Goal-Step a significant step forward in procedural activity understanding. See Figure 1 for an illustration of these features.
In summary, we introduce Ego4D Goal-Step, the largest available egocentric dataset for procedural activity understanding, with 2,807 hours of videos with specific goal labels and 430 hours of segments with fine-grained step/substep labels. In what follows, we describe our annotation and taxonomy development process. We also provide a comprehensive analysis of new annotations and compare them with existing procedural video datasets. Finally, we demonstrate the value of our annotations for temporal detection and grounding tasks, and analyze the results in the context of the unique properties of our dataset.
Activity recognition has a two-decade history in computer vision. Early works have tackled atomic action classification using datasets of seconds-long video clips at relatively small scales [42, 20, 35, 30, 47]. Following the success of deep neural networks, several datasets have focused on scaling up by leveraging online videos [27, 1, 8, 28, 21, 37]. Recognizing the need for long-form video modeling, several datasets have also been proposed for action detection in untrimmed videos [6, 46, 26, 54, 23].
Recently, the community has expanded the scope by developing datasets for procedural activities. The typical dataset construction process involves selecting procedural tasks beforehand, e.g., various recipes in cooking, then collecting data for the pre-selected tasks through participant recordings or by mining online video repositories. Participant-recorded datasets like Breakfast [29] and Assembly101 [43] benefit from a controlled collection setting, enabling the development of a taxonomy aligned with the data, and resulting in dense annotations and hierarchical step segments similar to ours. However, they capture limited diversity (e.g., 10 cooking recipes) and are smaller in scale.
On the other hand, Cross-Task [61] and COIN [49] are Internet-mined datasets that benefit from the scalability. However, they rely on external resources to develop a taxonomy (e.g., wikiHow), leading to label spaces that do not capture precisely and comprehensively the activities represented in the videos. Consequently, a large portion of videos remains unlabeled, with densities of labeled segments in the low  $40\%$  (see Table 1). Furthermore, annotated segments often only partially match with step labels (e.g., due to subtle variations in objects used, step ordering, etc.), resulting in weakly-labeled data. These datasets are also often non-hierarchical and represent steps at a single level.
Compared to existing procedural video datasets, Ego4D Goal-Step provides dense annotations at scale, a well-aligned taxonomy, and a hierarchical label space - all at the same time - thanks to our hierarchical partitioning [10] approach to data annotation and taxonomy development. With a wide range of procedural videos sourced from Ego4D, accompanied by dense annotations with a comprehensive taxonomy, our dataset enables procedural activity understanding at scale.
3 Ego4D Goal-Step
3.1 Ego4D: The Status Quo
Ego4D is annotated in a variety of ways. All videos are annotated with scenarios and narrations which provide high-level and low-level descriptions of actions, respectively. Scenarios provide coarse categorization of activities (e.g., construction, arts and crafts), while narrations describe a camerewer's action at a specific time. The narrations are interaction-centric, focusing on individual, atomic actions that a camerewer performs over a short time period. For example "C picks up the spoon," "C pets the dog," "C unscrews the screw," where C represents the camera-wearer.
While the narrations offer valuable information to understand simple movements and hand-object interactions, in the broader context of activity understanding, they are limited. Human actions are not performed arbitrarily — they are intentional and are done to accomplish a particular goal [3]. For example, C picks up the spoon to add sugar to coffee; C unscrews the screw to detach a bicycle wheel. These goals are left hidden in existing narrations. Moreover, these goals themselves are part of more structured activities. “Adding sugar” is a step in the process of making coffee, and “Detaching the bicycle wheel” is a step towards replacing a punctured tire tube. While narrations explicitly capture what is immediately happening, they do not reveal why, or more broadly, to what end.
Narrations form the scaffolding for various other annotations on smaller subsets of Ego4D. Forecasting hands and objects (FHO) annotations involve atomic actions parsed into simpler (verb, noun) tuples. For example, the long-term forecasting task involves predicting the sequence of future atomic actions, without capturing the overarching goal. Moments query annotations go one step higher, representing composite actions that involve sequences of atomic actions like "wash dishes in the sink" or "put on safety equipment." While they are higher-level, they are still short-term activities and are not connected by their long-term task structure. Moreover, they cover a small set of categories (roughly 100), span an inconsistent set of granularities (e.g., the atomic action "cut dough", and the high level "operate the dough mixing machine"), and are not intended to cover complete activities/goals.
Collectively, these annotations inherit the narrow scope of narrations and offer only a short-term understanding of human activity, limiting their value for procedural activity understanding in intentional, long-form and structured human activity.
3.2 Ego4D Goal-Step: Annotation and Taxonomy Development
To address this gap, we annotate Ego4D for procedural video understanding. Ego4D videos are collected without prearranged scripts or step-by-step instructions. As a consequence, the complete set of activities present in the dataset is unknown and a taxonomy cannot be established beforehand. We overcome this in a data-driven manner, using a hierarchical partitioning approach for annotation and taxonomy development. In short, we first ask annotators to identify the primary goals depicted in each video. Next, they delve into each goal segment to identify the individual steps and their corresponding action sequences. Then, again, annotators recursively analyze each action segment to further annotate steps at lower levels to construct a complete step hierarchy. Throughout this process, we present the annotators with an incomplete and evolving taxonomy and encourage them to
suggest missing categories. We review them periodically and update our taxonomy over the course of annotation. The whole process involves five stages:
Stage 1: Goal taxonomy initialization We initialize a goal taxonomy with Ego4D scenario labels and manually subdivide them into specific goal categories. For example, we expand the "cooking" scenario into popular dishes (e.g., "make pasta," "make omelet") and the "construction site" scenario into specific construction tasks (e.g., "paint drywall," "build wooden deck").
Stage 2: Goal annotation and taxonomy update Annotators watch full-length videos (not clips cut to shorter lengths) and identify distinctive goal segments, assigning a goal category and providing free-form text description, e.g., make_omelet and "making omelet with toasted bread" as shown in Figure 1. We emphasize the importance of providing full-length videos to the annotators because short-length clips will inherently lack the overall goal of actions (e.g., a clip capturing boiling salted water will not give any clue that it was part of making pasta). For missing categories, annotators choose "other" and describe it in text. Auxiliary information, including procedural activity indicator (is_procedural), task continuity from previous segments (is Continued), and bullet-point summaries of procedural steps, are also annotated.
While the annotation is in progress, we iteratively refine our goal taxonomy. This process is largely manual. It involves mapping keywords in descriptions to existing goal categories and adding new categories to the taxonomy when necessary. We also manually verify the correctness of keyword mapping by visual inspection. This process is done periodically in batches of annotations.
Stage 3: Step taxonomy initialization We prompt a large language model, LLaMA-7B [50], to generate step-by-step instructions for each goal. This provides a concise, but potentially incomplete, list of step candidates per goal category. Moreover, it often misses routine steps performed in a given environment, e.g., washing hands in a home kitchen. To represent missing steps, we create a "catch-all" bucket for each scenario and use the same model to generate commonly occurring steps.
Stage 4: Step annotation and taxonomy update Annotators review full-length videos with goal annotations and identify step segments, assigning a step category and natural language description. To ensure annotators capture step-level granularity that reveals intentions and not low-level physical movements, we provide a specific template, "the camera wearer did  $X$  in order to  $Y$ ", and ask them to prioritize intentions (Y) over physical movements (X) in their annotations. As shown in Figure 1, this allows us to collect labels that reveal the functional objectives behind actions (e.g., "toast bread," "crack eggs," "beat eggs with chopsticks") rather than just the description of low-level actions.
Similar to stage 2, annotators indicate the procedural nature of step segments and describe substeps in bullet-points. Additionally, they provide task-relevance information (is Relevant) for each step segment using a multiple-choice question ("essential," "optional," and "irrelevant"). They are further requested to find a relevant wikiHow article and provide its URL, for annotation accuracy and educational resources. The taxonomy update follows the same process as stage 2.
Stage 5: Substep annotation and taxonomy update Finally, we ask annotators to further partition step segments into individual substeps. This process is largely the same as stage 4, and it shares the same step taxonomy. The only difference is that we provide annotators with both the corresponding goal and step annotations, and that we encourage annotators to focus on lower levels of granularity.
3.3 Dataset Analysis
Statistics Table 1 compares Ego4D Goal-Step with existing procedural activity datasets. As shown, Ego4D Goal-Step represents the largest available procedural video dataset in terms of total number of hours annotated with step labels. Goal annotations are available for a subset covering  $72\%$  of Ego4D after discarding non-procedural or uninteresting videos, totaling 2,807 hours of videos. At the segment level, there are a total of 47,721 segments for steps and substeps combined. On average, each goal segment has 23.82 step segments, and each step segment has 4.6 substep segments.
Annotation We contracted a third party vendor to manage the annotation process, and checked privacy and ethical compliance through rigorous reviews. We completed 10 iterations for goal annotation (stage 2), 10 iterations for step annotation (stage 4), and 8 iterations for substep annotation (stage 5). Each iteration involved annotators providing labels for a batch of videos, us reviewing them, and updating the taxonomy. On average, annotators took  $2 \times$  the video duration to annotate
Figure 2: Dataset statistics illustrating the distributions of segment durations (x-axis scaled to the square root for better display) and auxiliary labels across different levels of hierarchy.
goals, and  $7.5 \times$  to annotate steps/substeps because step annotation required fine-grained inspection. The total annotation time amounted to roughly 10,000 worker hours.
Granularity Figure 2 illustrates the distribution of goal, step, and substep segment durations. On average, goal segments span 19.64 minutes, while step and substep segments last 50.03 seconds and 19.49 seconds, respectively. The average substep duration aligns with datasets that derive their taxonomy from wikiHow [4, 60, 49], which suggests that our data-driven taxonomy has granularity similar to that found in annotations based on fixed step-by-step instructions. The average duration of our steps/substeps is 32.5 seconds, which is 6 times longer than Breakfast [29] and 22 times longer than Assembly101 [43]. This shows that most of the step/substep segments capture longer-duration actions than [30, 43] without sacrificing annotation density or capturing short-term atomic actions.
Taxonomy Our taxonomy contains 319 goal categories grouped into 34 scenarios. Among these, the cooking scenario provides procedural step annotations, comprising 86 goal categories and 514 step categories. We use these step categories from the taxonomy to annotate both step and substep segments. The dataset exhibits a long-tail distribution, with 125 goal categories representing  $90\%$  of labeled goal segments and 209 step categories covering  $90\%$  of labeled step/substep segments.
Labels Ego4D Goal-Step comes with various goal-oriented activity labels. About  $92\%$  of the goal segments are annotated with descriptions (3.03 words per sentence) and  $63\%$  with summaries (5.43 sentences per summary, 4.92 words per sentence).  $100\%$  of the step and substep segments come with descriptions (4.44 words per sentence) and  $15\%$  of them include summaries (4.38 sentences per summary, 3.1 words per sentence). Figure 2 shows the distributions of auxiliary labels.
Splits We provide data splits for training  $(70\%)$ , validation  $(15\%)$ , and testing  $(15\%)$  purposes. The splits were made at the video level, ensuring balanced inclusion of step categories across the splits. We release full annotations for the training and validation splits while withholding the test split.
4.1 Tasks
Goal/step localization Localization temporal action segments within a long, untrimmed video has numerous applications ranging from activity recognition to automatic chapterization for efficient navigation of instructional videos. Following temporal action localization [45], we formulate the task as predicting tuples of (start_time, end_time, class_category) encompassing goals and individual steps given a long untrimmed video. We use ActionFormer [56] and EgoOnly [51] as our baseline models for their superior performance demonstrated in various action localization benchmarks. We evaluate the results using the detection mAP averaged over goal or step categories, and over temporal IoUs  $\{0.1, 0.2, 0.3, 0.4, 0.5\}$ .
Online goal/step detection While the localization task above focuses on offline inference scenarios, here we consider its online counterpart. The task is formulated as predicting goal/step categories at each timestamp as they unfold in a streaming input video. Models must utilize information from the
Task	Metric	Model	Val	Test
Goal localization	Detection mAP	ActionFormer [56]	42.2 ± 3.8	45.9 ± 2.4
Step localization	Detection mAP	ActionFormer [56]	9.9 ± 0.3	8.7 ± 0.3
EgoOnly [51]	13.1 ± 0.3	14.0 ± 0.4
Online goal detection	Per-frame mAP	LSTR [53]	21.5 ± 0.5	24.5 ± 1.3
Online step detection	Per-frame mAP	LSTR [53]	8.7 ± 0.2	7.9 ± 0.2
EgoOnly [51]	10.2 ± 0.1	10.8 ± 0.1
Step grounding	Recall@1, mIoU=0.3	VSLNet [57]	11.7 ± 0.2	10.7 ± 0.3