SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For representation learning, we augment the original unannotated dataset  $\mathcal{D}$  with pseudo-labeled keysteps to create  $\mathcal{D}' = \{\mathcal{V}_i\}_{i=1}^{|\mathcal{D}|}$ , where  $\mathcal{V}_i = \{(v_1, n_1, k_1), \ldots, (v_{|\mathcal{V}_i|}, n_{|\mathcal{V}_i|}, k_{|\mathcal{V}_i|})\}$  are the clips, narrations, and our model's inferred pseudo-labels. Our objective is to learn a video representation  $F_V(v; \theta) := \mathbb{R}^{H \times W \times C \times T} \to \mathbb{R}^{|K|}$  such that  $\arg\max_{i \in |\mathcal{K}|} F(v_t; \theta) = k_t$ . Here  $(H, W, C, T)$  denote (height, width, channels, time) in a video segment. This is a standard classification problem and we choose cross entropy as the training loss.
To evaluate the quality of the resulting learned representation  $F_{V}$ , we consider several downstream tasks. First, we evaluate task classification on  $\mathcal{D}$  against the withheld ground truth task labels (e.g., is the video depicting "make tiramisu" or "make omelet"), a common task for feature learning in instructional video [3, 8, 45] that can be evaluated more readily on large-scale datasets given the availability of task labels (vs. unavailability of keystep labels). In addition, we evaluate keystep classification given the temporal region of a keystep and keystep forecasting, where we must anticipate the next keystep given the video observed so far.
Network Architecture and Implementation Details. For zero-shot keystep recognition  $f_{v}$  and  $f_{n}$  are 768-dimensional frozen visual and text encoders of VideoCLIP [79]. For representation learning, we use MP-Net [63] to compute sentence embeddings and a modified TimeSformer [8] with 10,588 output classes as the visual encoder. The weights of the visual encoder are initialized to Kinetics [14] pretraining. For downstream tasks, we freeze the backbone and only replace the last linear layer with the dimension of the classification task, namely 778 and 133 output classes for keystep classification and forecasting in COIN and CrossTask, respectively, and 180 and 18 classes for task classification.
For zero-shot keystep segmentation, we use  $\gamma = 0.5$  and  $\gamma = 0.3$  for text and video features, respectively, since video features offer stronger supervision. We observe that the performance is not sensitive for  $\gamma \in [0.3, 0.5]$ ; details in Supp. In experiments where we consider both the video and text modalities, during path search, in the case of conflicting keystep suggestions, we choose the video
2We also explored a variant using Bayes Recursive Filtering (BRF) [69] to update the keystep belief state, but found this model less effective, likely because BRF only looks backward in time thus using less temporal context. See Supp. for experimental comparisons.
Figure 4: Qualitative examples comparing our keystep recognition with DistantSupervision [45] (first row) and VideoCLIP [79] (second row). Red/green denote incorrect/correct keystep predictions; yellow shows ASR narrations. Our proposed method is effective in correcting preliminary keystep assignments and reasons about the overall task structure. Best viewed in color and zoom.
keystep since the visuals tend to be a stronger cue. See Supp. for experiments justifying this design. For representation learning, similar to [45], we train the video model for 15 epochs with SGD with learning rate  $5 \times 10^{-3}$  followed by 15 epochs with AdamW [46] with learning rate  $5 \times 10^{-5}$ . In both cases, the learning rate is decayed progressively by 10 times in epochs 11 and 14.
Datasets. We use three public datasets of instructional videos—COIN, CrossTask, and HowTo100M—all of which were compiled from in-the-wild data on YouTube, and are accompanied by ASR transcriptions of the YouTuber's spoken narrations  $(n_t)$ . COIN [67] and CrossTask [89] contain 11,827 and 2,750 instructional videos, respectively, and are annotated for 778 and 133 (749 and 105 unique keysteps, respectively) keystep labels spanning 180 and 18 tasks, respectively. We use clip labels for evaluation only. HowTo100M [51] contains 1.22M instructional videos (i.e. more than  $100 \times$  larger than the others) spread over 23,000 tasks. As noted above, we leverage WikiHow [75] to obtain the keystep vocabulary  $\mathcal{K}$  for HowTo100M. We use COIN and CrossTask for both zero-shot keystep recognition and downstream task evaluation of our pretrained features. We use HowTo100M for downstream tasks only, since it lacks ground truth keystep annotations.
Evaluation Metrics. For keystep segmentation, we evaluate Frame-wise Accuracy (Acc) and Intersection over Union (IoU), following [18, 19, 79]. For every keystep  $k_{i}$ , Frame-wise Accuracy is the fraction of frames with ground truth  $k_{i}$  that has the correct assignment. The overall Frame-wise Accuracy is the mean accuracy for every keystep. Likewise, IoU is the intersection over union of a keystep  $k_{i}$ , averaged for every  $i$ . Consistent with prior work [18, 19], we do not consider background frames. For keystep classification, task classification, and keystep forecasting, we use the standard accuracy metric.
4.1 Zero-Shot Keystep Recognition
In this task, we predict keystep  $k_{t}$  for each time instance  $t$  in a given video  $\mathcal{V}$  in the test set. Following [79], we make one keystep prediction per second. We compare the following competitive baselines:
DistantSupervision [45]. A state-of-the-art model for zero-shot keystep recognition that predicts keysteps based on the text feature similarity between narrations and keysteps, i.e.,  $f(v_{t},n_{t}) = f_{n}(n_{t})$ .
Table 1: Zero-shot keystep recognition on COIN and CrossTask for three modality choices—text, video and video-text. We outperform strong baselines on all tasks. ' -' means the method is n/a.
Method	Text-only	Video-only	Video-Text
COIN	CrossTask	COIN	CrossTask	COIN	CrossTask
Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU
Random	0.0	0.0	0.01	0.01	0.0	0.0	0.01	0.01	0.0	0.0	0.01	0.01
VideoCLIP [79]	-	-	-	-	13.2	4.0	28.5	6.5	13.3	4.0	28.5	6.5
DistantSup. [45]	9.8	3.0	16.1	3.7	-	-	-	-	-	-	-	-
Linear Steps	10.4	3.1	16.3	3.8	13.4	4.3	28.5	6.5	13.4	4.0	28.5	6.5
Auto-Reg [66]	10.2	3.1	16.4	3.7	13.6	4.3	28.5	6.5	13.7	4.2	28.5	6.5
Pruning Keysteps	11.3	3.4	16.4	3.8	13.4	4.2	28.5	6.5	13.5	4.1	28.5	6.5
Ours	16.3 ± 0.3	5.4 ± 0.1	20.0 ± 0.2	4.9 ± 0.1	15.4 ± 0.1	4.7 ± 0.1	28.6 ± 0.0	6.6 ± 0.0	16.9 ± 0.1	5.4 ± 0.1	28.9 ± 0.1	6.7 ± 0.0
Table 2: Task-level keystep recognition on CrossTask. We outperform all methods in accuracy despite lacking their privileged information.
Method	Acc	IoU
Random	11.0	4.7
Bag of Steps	20.5	13.7
Drop-DTW [18]	22.3	15.1
Graph2Vid [19]	24.8	16.8
Ours	30.5	16.0
±0.2	±0.3
Table 3: Task classification on HowTo100M. On both modalities we outperform all state-of-the-art methods.
Method	Modality	Acc@1	Acc@5
MIL-NCE [50]	-	6.2	19.7
VideoCLIP [79]	-	7.9	25.5
DistantSup. [45]	Video	9.5	30.1
DistantSup. [45]	Text	12.3	28.5
Ours	Video	14.0	33.5
±0.3	±0.2
Ours	Text	15.5	35.0
±0.2	±0.2
VideoCLIP [79]. A popular transformer-based contrastive pretraining approach for zero-shot video-text understanding. To apply it here, we use the cross-modal video feature similarity with the keysteps, i.e.,  $f(v_{t}, n_{t}) = f_{v}(v_{t})$ .
Bag of Steps. This baseline (devised in [19]) is similar to the first two, except here the keystep set only contains known steps in a given task. Naturally, the number of candidate keysteps in this case is much lower than the keystep set used above and by our model.
Auto-Regressive [66, 70]. Instead of constructing an explicit task graph, an implicit representation could also model dependencies between keysteps. We use a transformer network to revise the preliminary noisy keystep labels (previous baselines) based on their aggregation over time.
Pruning Keysteps. The keystep set  $\mathcal{K}$  is generally broad and contains unlikely keysteps for some tasks, e.g., unscrew the bolt is irrelevant in the cooking task. We first cluster keysteps into  $C$  semantically similar clusters using k-means on their keystep embeddings and assign each video to one of the clusters per average similarity of the video's clips with the cluster members. Then, we compute similarity between only the selected cluster's keysteps and the video features to infer the per-clip keystep labels.
Linear Steps. Instead of using task graph, this baseline uses a linear order of keysteps as given in the dataset annotations. We still use keystep set  $\mathcal{K}$  for preliminary assignment.
Drop-DTW [18]. A SotA DTW-based approach where a linear order of steps is assumed. It requires a known order for execution for each task in the dataset.
Graph2Vid [19]. A SotA DTW-based approach that parses a non-probabilistic graph for each task and then performs DTW-based matching on all possible paths and chooses the one with the highest matching score.
Note that the Bag of Steps, Drop-DTW, and Graph2Vid models all have privileged information during inference compared to our model, namely the known task and its (ordered) list of keysteps. Hence below we compare those models in a separate experiment called task-level keystep recognition [19], where candidate keysteps for all methods come only from the subset of keysteps  $\mathcal{K}_T$  per task  $T$  as mined from WikiHow ( $|\mathcal{K}_T| \ll |\mathcal{K}|$ ). In all other zero-shot experiments, we use the universal keystep vocabulary  $\mathcal{K}$ .
Table 4: Downstream evaluation on CrossTask and COIN for keystep recognition (SR), task recognition (TR), and keystep forecasting (SF), using either an MLP (left) or transformer (right).
Method	MLP	Transformer
CrossTask	COIN	CrossTask	COIN
SR	TR	SF	SR	TR	SF	SR	TR	SF	SR	TR	SF
TSN [73]	-	-	-	36.5	-	-	-	-	-	-	73.4	-
ClipBERT [41]	-	-	-	30.8	-	-	-	-	-	-	65.4	-
S3D [50]	39.9	87.4	20.1	37.5	68.5	19.9	45.3	87.8	21.7	37.3	70.2	28.1
SlowFast [26]	44.9	89.7	23.4	32.9	72.4	23.0	48.5	89.8	24.0	39.6	71.6	25.6
VideoCLIP [79]	51.3	94.7	24.2	39.4	82.9	30.0	60.1	92.3	26.0	51.2	72.5	34.6
TimeSformer [8]	55.5	95.0	25.9	48.3	87.0	32.7	60.9	93.8	27.1	54.6	88.9	38.2
DistantSup. [45]	58.4	96.1	28.3	54.1	88.2	35.5	64.2	95.2	29.7	57.0	90.0	39.4
Ours	59.5 ± 0.1	97.1 ± 0.1	29.5 ± 0.1	55.2 ± 0.1	89.4 ± 0.0	36.3 ± 0.1	64.5 ± 0.0	96.0 ± 0.1	30.2 ± 0.1	57.2 ± 0.0	90.5 ± 0.0	40.2 ± 0.1
Results. Table 1 shows the zero-shot results on COIN and CrossTask. Throughout, standard errors denote variation across different test data splits. In all the three variants (text-only, video-only, and video-text), our method outperforms strong baselines and prior work. We see a gain of up to  $6.5\%$  (relative  $66\%$ ). Importantly, our model's preliminary keystep assignments correspond to the VideoCLIP [79] and DistantSupervision [45] baselines for the video-only and text-only settings; our relative gains directly show the impact of our video-mined task graph in correcting those labels. In addition, our gains over Linear Steps show the impact of our full probabilistic graph structure, compared to a prior based on linear ordering of steps.
Fig. 4 shows some qualitative examples comparing our outputs with DistantSupervision [45] (top) and VideoCLIP [79] (bottom). The narration pushing for thirty beats is incorrectly mapped to using needle by DistantSupervision, whereas VideoCLIP confuses salmon with cloth. Our task graph prior improves recognition.
Table 2 shows the task-level keystep recognition results. We outperform all methods, including the state-of-the-art Graph2Vid [19] by  $5.7\%$  (relative  $23\%$ ) in accuracy. Our IoU is similar to Graph2Vid's. We observe that the baselines conservatively assign the background label whenever the confidence is low, which helps IoU since background is not counted in the union operation [18, 19]. In contrast, accuracy accounts equally for false negatives and false positives, making it a more complete metric. We also emphasize that unlike Drop-DTW [18] and Graph2Vid [19], we do not assume keystep ordering from WikiHow recipes since that restricts the applicability to general in-the-wild instructional videos. Overall, our method provides a significant advantage over all the strong baselines and achieves state-of-the-art keystep recognition.
4.2 Instructional Video Representation Learning for Downstream Tasks
Next we apply our model to learn features  $F_{V}(v;\theta)$  on the large-scale HowTo100M and assess their impact for multiple downstream tasks. We add comparisons to other popular pretrained representations, MIL-NCE [50], VideoCLIP [79], and TSN [73].
Table 3 shows the task classification results compared to multiple state-of-the-art methods on the validation split of HowTo100M, the dataset we use for pretraining  $F_V$ . Following [45], we infer task labels by predicting keysteps then mapping them back to the task category. Our approach outperforms all the existing methods. Again, our gain versus VideoCLIP and DistantSupervision shows the impact of our improved pseudo-labels.
Table 4 shows the results when we transfer the features pretrained on HowTo100M to downstream tasks on COIN and CrossTask. We deploy both MLPs and transformers fine-tuned for each task. $^3$
Note that the concurrent work Paprika [86] uses a setting different from the SotA DistantSupervision [45], which makes their numbers not comparable; DistantSupervision's reported results are higher than those re-implemented in [86] for all tasks, making it a stronger baseline to beat. Further, keeping our setting consistent with DistantSupervision allows us to compare with other SotA methods.
Figure 5: Task graph visualization of CrossTask (35/108 keysteps). We only show top 5 transitions (excluding self-loops) within this keystep subset for clear visualization. The edges are directed. Please zoom for best view.
Our results are strongest overall. We outperform the state-of-the-art DistantSupervision [45] on all downstream tasks for both datasets under both architectures.
4.3 Task Graph Visualization
We display a portion of the mined task graph for CrossTask in Figure 5. It contains 35 out of 108 keysteps with top 5 transitions labeled (to avoid clutter). We see some interesting properties. For example, "press coffee" happens only after "add coffee", "close lid" after "open lid" whereas "cut cucumber" and "cut onion" can both happen after each other. Again, this structure is discovered automatically from unannotated videos.
We introduced an approach to discover the structure of procedural tasks directly from video, and then leverage the resulting task graph to bolster keystep recognition and representation learning. Our model offers substantial gains over SotA methods, and our qualitative results also validate our improvements. In future work we plan to explore video-mined task graphs for other video understanding tasks including procedure planning and mistake detection.
UT Austin is supported in part by the IFML NSF AI Institute. KG is paid as a research scientist at Meta. We thank the authors of DistantSup. [45] and Paprika [86] for releasing their codebases.
recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022. 3
[77] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video-language model pre-training for video understanding. arXiv preprint arXiv:2105.09996, 2021. 1
[78] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. VIm: Task-agnostic video-language model pre-training for video understanding. arXiv preprint arXiv:2105.09996, 2021. 3
[79] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 1, 2, 3, 4, 5, 6, 7, 8, 9
[80] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11562-11572, 2021. 1
[81] Hejia Zhang and Stefanos Nikolaidis. Robot learning and execution of collaborative manipulation plans from youtube cooking videos. arXiv preprint arXiv:1911.10686, 2019. 1
[82] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. A structured model for action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9975-9984, 2019. 3
[83] He Zhao, Isma Hadji, Nikita Dvornik, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. P3iv: Probabilistic procedure planning from instructional videos with weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2938-2948, 2022. 2
[84] Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Centerclip: Token clustering for efficient text-video retrieval. arXiv preprint arXiv:2205.00823, 2022. 3
[85] Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure-aware video representation from instructional videos and their narrations. arXiv preprint arXiv:2303.17839, 2023.2
[86] Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2, 3, 5, 9, 10
[87] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 2
[88] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746-8755, 2020. 1
[89] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545, 2019. 2, 4, 7