SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Main results on tasks supported by our dataset. We report standard deviation over 8 runs.
past leading up to the current timestamp, without having access to future frames. This task holds particular significance in egocentric vision due to its practical applications in AR scenarios such as real-time step-by-step guidance for procedural activities. We adopt LSTR [53] and EgoOnly [51] as our baseline models, given their strong performance and the availability of open-source implementations. Following existing literature, we report per-frame mAP.
Step grounding Unlike the localization and detection tasks that assume a fixed set of categories, some scenarios require models that can recognize new, unseen steps from open vocabulary text descriptions. Thus, we study step grounding, where given a long, untrimmed egocentric video and a natural language description of a step (e.g., "beat eggs with chopsticks" in Figure 1), a model must predict its temporal extent (start_time, end_time) in the video (i.e., the green bar). We adopt VSLNet [57], a popular video query grounding model due to its strong performance on similar egocentric grounding tasks. We report Recall@1, IoU=0.3 following prior work [22]. Note that we do not consider goal grounding in this work since many videos contain a single goal spanning most of a video, for which recall measures are trivially high.
Implementation details For all experiments except for the EgoOnly [51] baseline, we use precomputed clip-level features extracted densely from each video using Omnivore [19], which are publicly available for download on the official Ego4D repository (we used "omnivore(video_swinl"). The Omnivore model has been pretrained on a combination of multiple modalities (images, videos, and 3D data) in a supervised fashion and has been demonstrated to achieve strong generalization ability across a wide variety of vision tasks. EgoOnly [51] pretrains the ViT [14] backbone from scratch on the raw frames of Ego4D [22] using the MAE [24] objective, then further finetunes it on a combination of four existing action recognition datasets (Kinetics-600 [7], Ego4D Moments [22], EPIC-Kitchens-100 [13], COIN [49]) in a supervised fashion. For EgoOnly on the online detection task, we attach a single-layer linear prediction head on top of the pretrained ViT backbone and train the entire model end-to-end. Each prediction is made on input frames with 2 second temporal context. For EgoOnly on the offline localization task, we take the ViT backbone finetuned on the online detection task and attach the ActionFormer head [56] on top, and train just the prediction head while keeping the ViT backbone frozen throughout training. For all tasks, we use open source baseline implementations<sup>1</sup> and adapt hyperparameters for the proposed dataset (details in Supp.).
Table 2 reports the main results for all the tasks on both the validation and test sets. EgoOnly [51] achieves strong performance on offline step localization and online step detection, demonstrating the effectiveness of its egocentric pretraining strategy and corroborating the strong empirical evidence presented in their paper. Note that, compared to ActionFormer [56] and LSTR [53] that leverage precomputed features from Omnivore [19], EgoOnly finetunes its ViT backbone directly on our dataset by solving the temporal segmentation task, providing further performance improvement over the two other baselines.
<sup>1</sup>Actionformer (https://github.com/happyharrycn/actionformer_release), LSTR (https://github.com/amazon-science/long-short-term-transformer), and VSLNet (https://github.com/EG04D/episodic-memory/tree/main/NLQ/VSLNet)
Train on	Avg. Length	Localization	Online detect.
Goal	SSteps	Goal	SSteps
Goals [583]	1500 s	42.2	-	21.5	-
SSteps [32k]	32 s	-	9.9	-	8.7
Goals + SSteps [32k]	59 s	37.1	9.7	21.2	8.1
* "SSteps" denotes Steps + Substeps
* Numbers in square bracket denotes number of training samples
Train on	Step	Substep
(A) Substeps [15k]	9.6	6.4
(B) Steps [12k]	13.9	5.8
(C) Substeps + Parent steps [15k]	7.5	8.1
(D) Substeps [15k] + Narr. [12k]	10.5	7.4
(E) Steps [12k] + Narr. [15k]	14.0	5.6
Steps [12k] + Substeps [15k]	15.8	8.7
Table 3: Validation set results on hierarchical goal-step relationship. Left: localization and online detection. Combining goal and step/substep instances does not help each other, likely due to the large difference in segment lengths and the limited number of goal samples. Metric: detection/per-frame mAP. Right: grounding. Training jointly on steps and substeps outperforms other alternatives, highlighting the benefit of the hierarchical nature of our step annotations. Metric: Recall@1, IoU=0.3. These results suggest that exploiting the step-substep hierarchy is clearly beneficial (right table), while effectively leveraging the goal-step hierarchy needs further investigation (left table).
Compared to the step prediction tasks, goal prediction requires a much longer temporal context. For instance, step segments have an average duration of 32.5 seconds, whereas goal segments have an average duration of 1946.9 seconds. This makes it a challenging long-form video understanding task. We report ActionFormer and LSTR baseline results but omit EgoOnly results due to its focus on relatively shorter-term temporal context. Specifically, EgoOnly's ViT backbone is fine-tuned with 2-second clips from our dataset, providing limited long-term context to the learned representations and, as a result, achieving inferior results compared to baselines that leverage Omnivore features. This highlights the challenging nature of our dataset and warrants further investigation into refinements in training and modeling approaches to achieve a more balanced performance across goal and step prediction tasks.
On the step grounding task, models achieve roughly similar recall scores on the validation and test splits. Note that a subset of text queries in each split belong to step classes that are not seen during training. For example, the step category label "Add the melted chocolate and blend until smooth" does not occur in the training set, but its corresponding natural language description "Add chocolate balls to blender jar" is still groundable. The results on this zero-shot subset are understandably lower - 4.3 and 3.8 Recall@1, mIoU=0.3 on validation and test splits, respectively.
4.3 How do models exploit the hierarchical goal-step-substep relationship?
Next, we study the hierarchical relationship among goals, steps, and substeps in our dataset. First, we explore the goal-step hierarchy by comparing models trained on goals only, steps and substeps only, and all combined. Table 3 (left) shows that simply combining all instances does not help the individual tasks of localization and online detection, likely due to the non-overlapping taxonomy, the large difference in segment lengths, as well as limited number of goal samples.
However, once we switch to step-substep hierarchy, in Table 3 (right), we find that jointly training on steps and substeps (last row) is superior to models trained on substeps only (A) or steps only (B). This joint training is significantly more effective than pairing steps with their parent descriptions (C).<sup>2</sup>
Importantly, the gain over (A) and (B) is not simply an effect of dataset size. In (D-E), we replace steps/substeps with an equivalent amount of Ego4D narration data from the same videos to match the total training set size of the result in the last row. The narrations capture low-level actions performed by the camera-wearer (e.g., "camera-wearer picks up a pan") and have been shown to improve performance in egocentric grounding models [39]. Our results show that while adding narrations can offer a small improvement, jointly training on steps and substeps remains a superior strategy.
Put together, the results suggest that jointly training with steps and substeps leads to models that are aware of the sequential and hierarchical relationships between steps, leading to stronger performance across both levels.
2This was done by concatenating the step and substep descriptions and using that as our hierarchy-aware query. More sophisticated, embedding-based step and substep fusion performed worse in our experiments.
Figure 3: Grounding performance breakdown by relevance type. Essential steps are hardest to recognize as they tend to involve the same set of objects in related tasks (e.g., different stages of preparing the dough in various baking scenarios), requiring fine-grained recognition of objects, scenes, and actions. Optional and irrelevant steps, despite being rare during training and not as related to the activity, are visually distinct and easier to recognize. Numbers above bars denote the number of training samples in each category.
Figure 4: Left: Goal/Step localization with varying effective temporal context. Longer temporal context is required to disambiguate goals, while steps favor shorter context for finer temporal details. Joint training with shorter context adversely affects goal localization due to the dominance of step segments in the training signal. Middle: Online goal detection with varying LSTR memory length. For goals, a larger short term memory consistently leads to better results. Right: Online step detection with varying LSTR memory length. For steps, capturing fine-grained information via short-term temporal context is more important than capturing long-term context.
4.4 Which types of steps are easier to recognize?
We evaluate grounding models based on the role of steps in the overall procedure: essential, optional, or irrelevant. Essential steps are necessary for completing the task (e.g., cracking eggs for cooking an omelet), optional steps are relevant but not essential (e.g., adding sriracha sauce to an omelet), and irrelevant steps are unrelated (e.g., answering phone while cooking). Figure 3 shows that essential steps are the most difficult to recognize. This is somewhat counterintuitive since they occur frequently within the task (as evidenced by the largest number of instances reported in the figure) and are contextual with the activity. However, we found that they are difficult to distinguish from other steps of the related tasks as they tend to involve the same set of objects, scenes, and actions, e.g., flatten dough on table, knead dough, coat dough with flour in the task "Make bread," requiring fine-grained recognition. Conversely, optional steps (e.g., dispose eggshells, wear apron) and irrelevant steps (e.g., use phone, drink tea) are visually distinct in the context of the procedure and thus are easier to recognize, overcoming the disadvantage of relatively low training data (number above each bar) and despite their loose (or missing) connection to the goal.
4.5 Can long-term temporal context benefit procedural activity recognition?
Recognizing procedural activities in egocentric videos, especially in the online inference regime, requires long-term temporal context to accumulate enough past history [52]. For example, recognizing different but similar recipes may become possible only after observing completion of certain steps.
We quantitatively study this property by exploiting the hierarchical structure in Ego4D Goal-Step. In the localization task, this is achieved by adjusting the feature sampling stride before feeding features into the ActionFormer [56] that localizes goals and/or steps. This changes the effective temporal context consumed, especially when the theoretical receptive field does not cover the full video. Results are visualized in Figure 4 (left) with error bars representing standard deviation of 8 runs. We find that a larger temporal context is indeed required to disambiguate goals, while steps favor a much smaller context for finer temporal details. Additionally, we find goal-step joint training with a small context adversely affects goal localization due to the dominance of shorter steps in the training signal. As a result, the step results with and without goal instances are almost overlapping.
In online goal/step detection tasks, we follow the experimental setting of LSTR [53] that varies both short-term and long-term memory lengths. For online goal detection (Figure 4 middle), we find that increasing the long-term memory to 256 seconds and longer leads to noticeable gains. Furthermore, a larger short term memory consistently leads to better results. These findings demonstrate that predicting goals indeed requires long-term temporal context. This aligns with our intuition that disambiguating the goals requires piecing together evidence that needs to be accumulated over a long temporal span. For instance, in the case of preparing a recipe, it may only become fully recognizable after several ingredients have been used and multiple steps have been completed.
On the other hand, for online step detection (Figure 4 right), performance increases when we switch the short term memory length from 4 seconds to 8 seconds, but plateaus when we further push to 16 seconds. Also, longer long-term memory does not seem to improve performance; in fact, when a 4 second short-term memory is used, performance decreases when the long-term memory exceeds 64 seconds, which is twice as long as the average step segment duration of 32.5 seconds. This pattern is consistent with our observation on step localization, suggesting that step detection favors fine-grained information captured in short-term temporal context over longer context.
We presented Ego4D Goal-Step, a new set of annotations with a hierarchical taxonomy of procedural activity labels on the recently released Ego4D. It is larger than existing procedural video datasets, contains hierarchical action labels (goals - steps - substeps), and provides various auxiliary information useful for procedural activity understanding. We demonstrated three distinct task scenarios supported by our dataset - temporal localization, online detection, and grounding - and analyzed how various research questions can be explored such as hierarchical learning and long-term video modeling.
The comprehensive nature of our dataset calls for future investigations into other aspects of procedural activity understanding. One promising direction is incorporating existing Ego4D annotations for more comprehensive analyses of procedural activities. For instance, narrations and FHO/Moments provide atomic action labels that can be combined with our dataset to form a 4-level hierarchy (goals - steps - substeps - actions). Furthermore, hand & object interaction annotations provide object state information that can enable object state-based progress monitoring. We eagerly anticipate the emergence of active research threads pursuing these directions.
Limitations and societal impact We acknowledge that Ego4D Goal-Step is intended for research purposes and should not be regarded as a comprehensive dataset encompassing the full range of daily human activities. Models trained on our dataset may exhibit biases towards the specific activities included in the dataset, resulting in a limited coverage of our everyday living scenarios.
[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016. 3
[2] J. K. Aggarwal and M. S. Ryoo. Human activity analysis: A review. *Acm Computing Surveys* (Csur), 43(3):1-43, 2011. 1
[3] A. Bandura. Social cognitive theory: An agentic perspective. Asian journal of social psychology, 2(1):21-41, 1999. 1, 2, 4
[4] S. Bansal, C. Arora, and C. Jawahar. My view is the best view: Procedure learning from egocentric videos. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII, pages 657-675. Springer, 2022. 2, 3, 6
[5] M. Botvinick and D. C. Plaut. Doing without schema hierarchies: a recurrent connectionist approach to normal and impaired routine sequential action. Psychological review, 111(2):395, 2004. 1, 2
[6] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-970, 2015. 3
[7] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. 7, 14
Appendix A Implementation details
For all the baselines except for EgoOnly [51], we use pre-computed clip-level features extracted densely from each video using Omnivore [19], which are publicly available for download on the official Ego4D repository (we used "omnivore(video_swinl"). These features are extracted with a stride of 16 frames, which is 0.533s or 1.875 features per second on Ego4D videos of 30 fps.
For EgoOnly [51], we use the checkpoint which was pretrained on the raw frames of Ego4D [22] using the MAE [24] objective and then further finetuned with the temporal segmentation objective on a combination of four popular action recognition datasets: Kinetics-600 [7], Ego4D MQ [22], EPIC-Kitchens-100 [13], COIN [49]. In our experiments, we take this checkpoint and further finetune it on the temporal segmentation objective with our step annotations, for 10 epochs<sup>3</sup> with a learning rate of 1e-5.
Goal/step localization. We use ActionFormer [56] and EgoOnly [51] as our baseline approaches. ActionFormer combines multi-scale feature representations with local self-attention, and a lightweight decoder to predict an action category at every moment in time. We use the official implementation of ActionFormer [56] for the EPIC-Kitchens-100 [13] dataset, unless noted otherwise, and tune the hyperparameters based on the best result on the validation set. We extend the FPN from 5 stages to 9 stages to learn from longer temporal context. Training takes less than 7 hours (32 epochs) on a single V100 GPU. We report the average and standard deviation of 8 runs for each result.
For EgoOnly [51], we attach ActionFormer [56] on top of the features generated from the EgoOnly backbone finetuned on temporal step segmentation. We follow the same hyperparameters in this stage as the ActionFormer baseline, i.e. 9-stage FPN, 32 epochs.
For step localization experiments ("SSteps" in Table 3), we train the models by combining step and substep instances and treating them equally. For evaluation, we average the mAP across all the step and substep categories because they share the same taxonomy. In the joint goal-step localization experiments ("Goals + Ssteps" in Table 3 and Figure 4 left), we merge the goal and step taxonomy categories and combine all of goal, step, substep instances. However, evaluation is still independent for each category, regardless of the category coming from the goal taxonomy or the step taxonomy, following the common detection per-class mAP measure. Then, the goal localization results are averaged as the goal mAP, and the step localization results are averaged as step mAP.
We use the base learning rate of 2e-4 and train the model for 32 epochs, with linear warm-up for 16 epochs. We denote the raw feature stride of 0.533s (i.e. 1.875 fps) by the  $1 \times$  "effective temporal context" (ETC) reported in Figure 4 left of the main paper. Based on the best results reported in Figure 4 (left), we set the ETC for step localization to be  $1 \times$ , for both step-only and goal-step joint experiments. For goal localization, we set it to  $8 \times$  (feature stride of 4.266s) for goal-only experiments and  $16 \times$  (feature stride of 8.533s) for goal-step joint experiments.
Online goal/step detection. We use LSTR [53] and EgoOnly [51] as our baseline approaches. LSTR combines an encoder that captures the long-term coarse-scale information and a decoder that models short-term information. We use the official implementation of LSTR [53], including the default short-term memory of 8 seconds and the long-term memory of 512 seconds, unless noted otherwise. In our experiments, we found that training the model for 4 epochs yields the best performance on the validation split. We train with the base learning rates of 4e-5 for goals, and 2e-4 for steps. We follow the default hyperparameters provided in the official implementation, and apply the linear learning rate warm-up for the first  $40\%$  of the training iterations. Training each model takes less than 4 hours (4 epochs) on a single V100 GPU.
For EgoOnly, we attach a single-layer classification head on top of the pretrained EgoOnly backbone and train the entire network end-to-end with the temporal segmentation objective with the default temporal context of 2 seconds. For online step detection inference and evaluation, we take only the last frame classification output as the prediction at the current timestamp. We use the same model to extract features for offline step localization experiments.
Step grounding. For this task, we use all annotated instances of steps and substeps. We ignore goal segment annotations as they do not inform model selection. This is because many videos contain
3We note that here the notion of epoch is different from ActionFormer/LSTR because of the differences in training settings; we refer to EgoOnly [51] for details.
a single goal that spans a large portion of the video, which results in trivially high recall scores (as mentioned in Section 4.1).
For our baseline model, we use VSLNet [57]. The model works as follows. First, a sequence of features are extracted for the video and text query. A cross-attention mechanism then aggregates information across modalities to produce one feature per time-step. Finally, two LSTM models aggregate these features to predict the start / end time probabilities at each time-step. We refer the reader to the original paper for full model details [57].
Each video is re-sampled to 128 features for training. We train models for 200 epochs with learning rate 1e-3, batch size 32, and select the model with the highest validation set performance for testing. The remaining hyperparameters follow the default for Ego4D NLQ [22]. Training takes approximately 6 hours on two V100 GPUs. We report an average of 8 runs for each result (Table 2 in the main paper).
Appendix B Additional dataset statistics
(a) Goal descriptions
Figure 5 illustrates the distribution of natural language descriptions for goals and steps/sub-steps combined. To create the plot, we removed stop words and stemmed the remaining words. For the goal descriptions, we display the first two words, while for the step/sub-step descriptions, we display the first three words.
(b) Step descriptions (cooking scenarios)
Figure 5: Distribution of natural language descriptions for goals and steps.
Appendix C Datasheet for Ego4D Goal-Step
We provide the datasheet following the format suggested by Gebru et al. [16]. We would like to note that Ego4D Goal-Step contains new annotations on existing videos in Ego4D. No new videos were collected or recorded during the annotation process. In all our responses below, the term "data" specifically refers to the annotations, not the videos associated with Ego4D Goal-Step, unless otherwise noted.
a) For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
Ego4D Goal-Step was created to support research on human activity understanding in long-form egocentric videos. Specifically, it provides procedural activity labels organized in a hierarchical
structure, capturing three distinct granularities of human actions (goals - steps - substeps). It supports various tasks in procedural activity understanding, including temporal action localization [56] and temporal grounding of step descriptions [58] - both of which we demonstrate in our paper - as well as other related tasks including hierarchical action segmentation [17], action anticipation [18], procedure planning [9], and task graph learning [31].
b) Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization?
Ego4D Goal-Step was created by Fundamental AI Research (FAIR) at Meta. The authors of this paper are also part of the Ego4D consortium.