SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kumar Ashutosh UT Austin and FAIR, Meta
Triantafyllos Afouras FAIR,Meta
Santhosh Kumar Ramakrishnan UT Austin
Kristen Grauman  
UT Austin and FAIR, Meta
Procedural activity understanding requires perceiving human actions in terms of a broader task, where multiple keysteps are performed in sequence across a long video to reach a final goal state—such as the steps of a recipe or a DIY fix-it task. Prior work largely treats keystep recognition in isolation of this broader structure, or else rigidly confines keysteps to align with a predefined sequential script. We propose discovering a task graph automatically from how-to videos to represent probabilistically how people tend to execute keysteps, and then leverage this graph to regularize keystep recognition in novel videos. On multiple datasets of real-world instructional videos, we show the impact: more reliable zero-shot keystep localization and improved video representation learning, exceeding the state of the art. Project Page: https://vision.cs.utexas.edu/projects/task_graph/
Instructional "how-to" videos online allow users to master new skills and everyday DIY tasks, from cooking to crafts to sports [51]. In the future, AR assistants able to parse such procedural activities could augment human skills by providing interactive guidance throughout the task in sync with the user's visual context [25, 56, 64], or by automatically creating video summaries of the most important information [2, 23, 54]. Similarly, human expert demonstration videos have the potential to steer robot behavior in the right direction for complex sequential tasks that have notoriously sparse rewards [52, 62, 81].
In a procedural activity, there is a single task goal that a person accomplishes by executing a series of keysteps, some of which have causal dependencies. For example, to make tiramisu, keysteps include whisk the eggs, lay out the ladyfingers, sprinkle the cocoa—and the cocoa must be sprinkled only after laying ladyfingers in the pan; to replace a bike tube, the wheel needs to be removed from the bicycle, then the old tube deflated and removed before the new one is fitted in. Thus, unlike mainstream video recognition tasks focused on naming an action in a short video clip [13, 14] like shaking hands, playing instruments, etc., procedural activity understanding requires perceiving actions in terms of the broader goal, breaking down a long-form video into its component keysteps.
To address keystep recognition, prior work has proposed creative ideas to either match video clips to keystep names based on the transcribed narrations [45, 55] or align visual steps to a rigid linear script (e.g., with dynamic time warping) [15, 18, 19, 36]. Other methods pose keystep recognition as a classification problem and label each fixed-sized chunk into one of the possible classes [48, 50, 77, 79, 80, 88]. However, these existing approaches face important limitations. Simple feature-keystep similarity matching and classification assume all actions are independent and can appear anywhere throughout the video, while the existing alignment models assume every keystep will match some
Figure 1: Our method uses a video-mined task graph as a prior to update the preliminary keystep assignments. The visual signals confuse a toaster with iron and bread with clothes, whereas our method focuses on the broader context using the task graph and assigns the correct keystep labels. When the visual signal predicts taking out bread, we instead correctly map it to running the toaster since taking out can only happen after putting the bread and running the toaster.
video segment—neither of which holds universally. Furthermore, the status quo is to rely on text-based knowledge bases to represent the likely order of keysteps [19, 45, 55], e.g., WikiHow, which fails to encapsulate the rich variety of ways in which the procedure may be executed in practice.
We propose to regularize keystep predictions on novel videos based on a task graph automatically mined from how-to videos. The task graph is a probabilistic data structure capturing the keysteps (nodes) and their dependency relationships (edges). First, we show how to automatically discover the graph structure directly from unlabeled, narrated how-to videos. Notably, video demonstrations are much richer than a scripted list of (text) steps like WikiHow; they show how to perform the task and, through different video instances, reveal the strength of dependencies between steps and the alternative ways in which the same goal can be reached. Next, we perform keystep recognition on novel videos using the task graph as a prior for task execution. Here we develop a beam-search approach in which keysteps confidently recognized from the visual and/or narration inputs are anchors, and subsequent less-confident steps are inferred leveraging the graph to find a high probability path. The task graph allows our model to "see the forest through the trees," since it can anticipate the overall arc of progress required to complete an instance of the full task.
Ours is the first work to use task graphs to enhance keystep prediction in instructional videos, and the first to discover a probabilistic task graph model directly from video. We first demonstrate that our novel approach improves zero-shot keystep localization for two challenging procedural activity datasets, COIN [67] and CrossTask [89], outperforming prior work [18, 19, 45, 79]. We also show that our graph learned from real-world videos surpasses the prior supplied by traditional written scripts. Next, we show that our task graph improves video representation learning for the large-scale HowTo100M dataset [51], where our corrected keystep pseudo-labels benefit pretraining compared to state-of-the-art keystep understanding work [45] and popular multimodal embeddings [50, 79]. In addition, we show the resulting pretrained video representations benefit multiple downstream tasks: keystep classification, keystep forecasting, and task classification. Finally, visualizing our learned task graphs, we show their ability to discover ties even across different tasks.
Keystep Recognition and Localization. Keystep recognition for instructional video is a focus for multiple recent influential datasets [51, 67, 89]. COIN [67] and CrossTask [89] offer manually annotated keysteps for more than 900 keystep labels and 198 diverse procedural tasks. The even larger HowTo100M dataset [51] covers tens of thousands of tasks; while not labeled for keysteps due to its massive size, recent work shows the promise of (noisily) localizing keysteps in HowTo100M using a text-based matching between keysteps from a knowledge base (e.g., WikiHow) and the words spoken in the how-to video narration [45, 55, 86]. Our approach builds on this idea when mining videos to form a task graph.
Grounding keysteps in instructional videos [6, 19, 20, 21, 50, 51, 79, 89] is crucial for procedural planning [9, 12, 16, 40, 61, 71, 83, 85] and learning task structure [55, 87]. Some prior work localizes the keysteps by finding a similarity score between keystep (text) embeddings and video features using a multimodal embedding [50, 79], while others learn an embedding to map corresponding keysteps close together [6]. Unlike our approach, this assignment ignores the broader context about
the keysteps and their sequence. Another line of work uses Dynamic Time Warping (DTW) to ground keysteps to video [11, 15, 19, 40]. These methods take as input an ordered list of keysteps and localize them in the video while preserving the given ordering. While this has the potential advantage of enforcing a known ordering, existing methods do so rigidly: the single linear order is required, and it is assumed that all the actions are guaranteed to be present in the video—a very strong assumption for unconstrained video. While Drop-DTW [18] removes the last assumption by introducing a mechanism that allows dropping outliers, it remains constrained by the monotonic ordering. The set-supervised action recognition method [47] proposes a more relaxed pairwise consistency loss that uses weakly labeled videos and encourages the attentions on actions to follow a similar ordering. In contrast to any of the above, our proposed method uses a probabilistic task graph to guide the localization process and correct predictions based on the keystep transition patterns discovered in in-the-wild video demonstrations.
Graphs for Video Understanding. Prior work uses graphs to understand spatio-temporal relations between objects and actors in videos [7, 53, 74, 82]. Such modeling helps to surface underlying relationships that may not be captured implicitly by models. Graph Convolutional Networks (GCN) are another promising method to incorporate graph structure in the training process [30, 38, 53, 58]. Earlier work in activity recognition has explored a variety of statistical models to represent complex sequential activities, such as Bayesian Logic Networks [68], dynamic Bayesian Networks [60], AND-OR graphs [35], latent SVMs [39], and graph-parsing neural networks [57]. Our work is distinct for generating data-driven graphs from large-scale video samples, rather than exploit a predefined statistical model, and for bootstrapping noisy activity labels for recognition.
Methods for unsupervised procedure learning aim to discover the common structure from a set of videos showing the same task [2, 22, 59, 86] in order to reveal a common storyline [2, 22, 59] or in Paprika [86] (a concurrent approach) to generate pseudo-labels for video feature learning [86]. The goal of discovering the latent procedural structure resonates with our task graph formation; however, unlike the prior methods, we show how the task graph serves as an effective prior for accurate keystep recognition from video. Paprika [86] uses task graph nodes as a pretraining signal—their constructed graph is non-probabilistic, cannot represent that some transitions are more likely than others, and cannot be directly used for keystep recognition. Additionally, our approach predicts keysteps by fusing supervisory signals from the graph and a weakly-supervised similarity metric between video features and candidate keysteps, eliminating the need for explicit graph annotation [19, 64].
Video Representation Learning. Pretraining to extract meaningful visual representations is useful for many downstream tasks including action recognition [26, 33, 42, 43, 76] and action anticipation [1, 27, 28, 32, 49]. The pretraining objective is either to map the visual representations to meaningful classes [8, 45] or to its equivalent text representation [3, 44, 50, 79]. Text-guided pretraining further enables tasks like text-to-video retrieval [17, 24, 48, 79, 84] or video captioning [31, 48, 72, 78]. It is desirable to pretrain on large video datasets [13, 14, 34, 51] for better generalization. In particular, HowTo100M [51] is the largest instructional video dataset (134,472 hours of video); its scale makes annotations for keysteps impractical. Recent work uses external WikiHow (text) steps to generate keystep classes for pretraining [45, 55]. However, the generated keystep classes are noisy due to misalignments [37] and non-visual narrations [4], which affects the resulting representations. We show how to use a task graph to improve keystep label assignment across the massive dataset, which in turn significantly improves the resulting video representation and downstream benchmarks.
We propose to discover task graphs from how-to videos and then use them as a prior for keystep recognition. Given a keystep vocabulary (in text only) and videos from various instructional tasks, our goal is to automatically localize any instances of those keysteps in each video.
To that effect, we first use external text corpora to obtain a keystep vocabulary. Next we obtain preliminary keystep labels by linking visual and/or narration representations between the video clips and candidate keystep names. Then we use those (noisy) keystep labels to construct a task graph, and finally we use the preliminary keystep labels and task graph prior to obtain high-quality keystep labels. The recognized and localized keysteps are themselves a useful output of the method. We also explore using the inferred keystep labels to support large-scale video representation learning for instructional videos. Fig. 2 contains an overview of the method.
Figure 2: Overview of the proposed keystep recognition using our video-mined task graph. Left: We obtain similarity-based keystep  $\hat{k}$  for each video clip in the dataset using similarity between keysteps in  $\mathcal{K}$  and text and/or video features. Top middle: We use the inferred keystep labels to learn a probabilistic task graph  $\mathcal{T}$ . Right: Finally, we keep confident video segments and use the task graph  $\mathcal{T}$  priors to obtain the final keystep predictions. See text.
Task Definition. We have two inputs: (1) an unannotated dataset of narrated instructional videos  $\mathcal{D} = \{\mathcal{V}_i\}_{i=1}^{|\mathcal{D}|}$  and (2) a keystep vocabulary  $\mathcal{K}$ . Each video  $\mathcal{V}_i$  in  $\mathcal{D}$  is comprised of a sequence of clips, each of which has an associated spoken narration provided by the how-to demonstrator (and converted to text with ASR):  $\mathcal{V}_i = \{(v_1, n_1), \dots, (v_{|\mathcal{V}_i|}, n_{|\mathcal{V}_i|})\}$  where  $v_t$  and  $n_t$  are the visual and text segment corresponding to  $t^{th}$  clip in the instructional video. The keystep vocabulary  $\mathcal{K}$  is an unordered set of natural language terms for all the relevant keysteps in the tasks depicted in  $\mathcal{D}$ . For example, if the dataset contains the tasks "make tiramisu" and "fix bike derailleur", then  $\mathcal{K}$  would contain terms like {whisk eggs, soak ladyfingers, ..., prop up the bike, shift bike to highest gear,...}. To give a sense of scale, datasets in our experiments contain 18 to 1,059 unique tasks and 105 to 10,588 unique keystep names. We do not assume the task labels are known per video. Note that words in the keystep vocabulary are not equivalent to the spoken narrations—the latter is unconstrained.
Our goal is to predict the correct keystep label  $k_{t} \in \mathcal{K}$  for every  $(v_{t}, n_{t})$ . For notation simplicity, we refer to localized keystep predictions as  $k_{t}$  and keystep names in the vocabulary as  $k_{i}$ , for  $i \neq t$ .
Sourcing a Keystep Vocabulary. To obtain a keystep vocabulary  $\mathcal{K}$ , we consult existing text knowledge bases, namely WikiHow and the vocabulary curated by experts for the COIN dataset [67]. WikiHow [75] contains written instructions of more than 240K how-to tasks. Each article has a list of steps that needs to be taken to achieve the desired task. For example, "Make Instant Pudding"1 has steps "Tear open the instant pudding mix and pour it into the bowl", "Pour the mixture into small serving bowls". Note that several tasks share common keysteps, e.g. "Pour the mixture into small serving bowls" can happen in many recipes. In our experiments, we either use keysteps provided in COIN [67] ( $|\mathcal{K}| = 749$ ) and CrossTask [89] ( $|\mathcal{K}| = 105$ ), or WikiHow keysteps for the HowTo100M [51] dataset ( $|\mathcal{K}| = 10,588$ ). See [45] for details on keystep set curation and Supp. for details.
Preliminary Keystep Assignment. First we make a preliminary estimate of each  $k_{t}$ . Following recent work [45, 79], we draw on the similarity between (1) language features for the keystep names  $k_{i}$  and (2) visual and/or language features for the video clip  $(v_{t}, n_{t})$ . Specifically, let  $f_{v} := \mathbb{R}^{H \times W \times C \times T} \to \mathbb{R}^{D}$  and  $f_{n} := \mathbb{R}^{L} \to \mathbb{R}^{D}$  be the  $D$ -dimensional visual encoder and text encoder, respectively. Here,  $(H, W, C, T)$  denotes (height, width, channels, time) and  $L$  is the maximum language token length. Correspondingly, for every segment  $(v_{t}, n_{t})$  we obtain feature vectors  $f_{v}(v_{t})$  and  $f_{n}(n_{t})$ . Since keystep names are themselves text sentences, we also use  $f_{n}(k_{i})$  to obtain each keystep embedding. Here we denote  $f(v_{t}, n_{t})$  as a generic feature extractor that can be either  $f_{v}$  or  $f_{n}$  or a combination of both (to be specified per experiment). We obtain the preliminary keystep assignments as  $\hat{k}_{t} = \operatorname*{argmax}_{k_{i} \in \mathcal{K}} f(v_{t}, n_{t})^{T} f_{n}(k_{i})$ . Next, we will use these (noisy) keystep estimates to construct our task graph, which in turn will allow us to improve our keystep assignments
<sup>1</sup>https://www.wikihow.com/Make-Instant-Pudding
Figure 3: Visualization of properties emerging from the video-mined task graph. Some of the keysteps are directional e.g., pouring gelatin into boiling water must be preceded by the keystep prepare gelatin (left). Some examples of top-4 next keysteps (center), and a visualization showing keystep sharing discovered across related HowTo100M tasks (right). Best viewed in zoom.
for all clips. These initial estimates  $\hat{k}_t$  will also serve as a baseline, corresponding to what is done in practice today by existing multimodal methods using either visual [79] or narration text [45] features.
Video-Mined Task Graphs. Having obtained a rough keystep assignment, we next construct a task graph to capture how keysteps are related to each other. Our task graph design is motivated by two key observations. First, keysteps can share different relationships. Some keystep pairs are directional, that is, one is the precursor to the other (e.g., whisking an egg must be preceded by breaking an egg); others are simply connected, meaning they often both appear in an activity but with less constrained temporal ordering (e.g., whisking egg and sifting flour), while others are unconnected, meaning they support different task goals altogether (e.g., mixing the salad and cutting wood with a saw). Second, multiple video demonstrations of the same task share the same high-level objective and hence similar keysteps, yet not necessarily in the same temporal order. For example, two how-to's for making pizza may use different ingredients and perform steps in a different order.
Hence, rather than hand-design the task graph [19, 64], we propose to mine it directly from video samples. In addition, rather than encode possible transitions uniformly [86], we propose to represent them probabilistically. Finally, rather than constrain the graph to be representative of a single task [19], we aim to discover a single task graph across all tasks, such that related tasks can share information (e.g., whisk eggs would be common to many different recipes, and any sequential patterns surrounding it could inform other tasks in the graph). See Fig. 3 (right).
Formally, the task graph  $\mathcal{T} = (V,E,w)$  has all the keysteps as nodes, i.e.,  $V = \mathcal{K}$  and all transitions as edges,  $E = \mathcal{K}\times \mathcal{K}$ . Naturally, we want the graph to be directed and weighted so that more probable keystep transitions have higher weights. For any pair of keysteps  $k_{i}$  and  $k_{j}$ , we define the weights of the edges as:
$$
w (k _ {i}, k _ {j}; \mathcal {T}) = \frac {\sum_ {\mathcal {V} \in \mathcal {D}} \sum_ {t \in | \mathcal {V} |} \mathbb {1} (\hat {k} _ {t} = k _ {i} , \hat {k} _ {t + 1} = k _ {j})}{\sum_ {\mathcal {V} \in \mathcal {D}} \sum_ {t \in | \mathcal {V} |} \sum_ {x \in | \mathcal {K} |} \mathbb {1} (\hat {k} _ {t} = k _ {i} , \hat {k} _ {t + 1} = k _ {x})}
$$
where  $\mathbb{1}$  is the indicator function. In other words, the weight of an edge between  $k_{i}$  and  $k_{j}$  is the count of transitions between the two keysteps normalized by total count of keysteps  $k_{i}$  being executed. The normalization converts the weight into probability distribution and the sum of all outgoing edges is 1, i.e.,  $\forall k_{i} \in \mathcal{K}, \sum_{k_{x} \in \mathcal{K}} w(k_{i}, k_{x}) = 1$ .
Fig 3 (left) shows interesting properties emerging from the task graph. Some pairs of keysteps are directional, e.g., only after preparing the gelatin can it be poured into the boiling water. Fig 3 (middle) shows example top-4 transitions—all of which show the expected trend. Finally, the keystone transition heatmap (right) shows that the task graph discovers transitions between keysteps of related tasks like (tuning ukulele, fixing violin string) or (kombucha tea, kombucha scoby), while also detecting that neither pair shares with the other. We stress that these inter-task relationships are a direct consequence of our video-mined, probabilistic keystone transitions, unlike [19, 64, 86].
Keystep Update Using the Video-Mined Task Graph. Next, we use the preliminary keystep labels  $\hat{k}_t$  and task graph  $\mathcal{T}$  to obtain corrected keystep labels  $k_{t}$ . Intuitively, we want the task graph to regularize the initial estimates: keep keystep labels with a strong signal of support from the visual and/or text signal in the video, but adapt those keystep labels with low confidence using the prior given by the graph structure  $\mathcal{T}$ .
Specifically, we estimate the confidence score for keystep  $\hat{k}_t$  as the similarity between the video feature(s)  $f(v_{t},n_{t})$  and the keystep name feature  $f_{n}(k_{i})$ :  $s(\hat{k}_t) = f(v_t,n_t)^T f_n(\hat{k}_t)$ . Given any high-confidence pair  $\hat{k}_{t^{-}}$  and  $\hat{k}_{t + }$ , where  $t^{-}$  and  $t^{+}$  are the closest high-confidence time instances before and after  $t$ , respectively, we find the highest probability path in  $\mathcal{T}$  between  $\hat{k}_{t^{-}}$  and  $\hat{k}_{t^{+}}$ . Formally,
$$
k _ {t} = \left\{ \begin{array}{l l} \hat {k} _ {t} & \text {i f} f (v _ {t}, n _ {t}) ^ {T} f _ {n} (\hat {k} _ {t}) \geq \gamma \\ \operatorname {P a t h S e a r c h} (\hat {k} _ {t ^ {-}}, \hat {k} _ {t ^ {+}}, t) & \text {o t h e r w i s e} \end{array} \right.
$$
where  $\gamma$  is a confidence threshold, and  $t^{-}$  and  $t^{+}$  are time instances with confidence more than  $\gamma$ . PathSearch finds the maximum probability path between  $\hat{k}_{t^{-}}$  and  $\hat{k}_{t^{+}}$  from the task graph  $\mathcal{T}$ . We convert all the probabilities to their negative log and use Dijkstra's algorithm to find the minimum weight path. To account for self-loops, we prepend and append  $\hat{k}_{t^{-}}$  and  $\hat{k}_{t^{+}}$ , respectively to the resulting path. Since the obtained path can have a variable number of keysteps, we assign the discovered keysteps uniformly between  $t^{-}$  and  $t^{+}$  (see Supp. for an example).
The idea of PathSearch is inspired from Beam Search algorithms, commonly used in Machine Translation [5, 10, 29, 65] for pursuing multiple possible sequential hypotheses in parallel, where we assume  $k_{t-}$  and  $k_{t+}$  as the start and end token. Instead of finding the maximum probable token (keystep in our context), Beam Search finds a sequence of tokens that maximize the overall likelihood of the selected token sequence. The above process yields corrected keysteps  $k_{t}$  that we use as our predicted keystep label. Fig. 4 shows some qualitative results. Note that PathSearch algorithm's time complexity is  $O(|\mathcal{K}|^2)$  that introduces a minimal overhead. In general, this overhead is even less than a forward pass to the model.
Keystep Localization and Instructional Video Representation Learning. We explore the effectiveness of our approach in two settings: zero-shot keystep localization and representation learning for instructional videos. For zero-shot keystep localization, we evaluate the accuracy of our model's final keystep predictions compared to withheld ground truth labels. We stress that the results are zero-shot, since we are provided no clips annotated with their keysteps.