SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(d) Distribution of annotations. Annotations follow a long-tail distribution which should encourage the development of zero/few-shot detection or text grounding methods.
(e) Comparison of text description lengths. HT-Step contains long step descriptions, almost twice as long as those of COIN, even when only the headline text is considered.
(c) Average number of unique steps annotated per video for the 20 activities with most unique steps.
(f) NED. Step sequences in HT-Step have a 0.37 NED compared to the canonical step ordering on average, which makes it more challenging compared to previous benchmarks.
Figure 5: HT-Step statistics. We provide various statistics for a qualitative overview of the dataset.
Full vs partial matches. Of the resulting annotated segments,  $27\%$  are full and  $73\%$  are partial matches. Some examples of full and partial match annotations are shown in Figure 4.
Variations. Most tasks (335 out of 433) involve a single listing of steps in wikiHow. However there is a significant number of activities (98) that was annotated according to more than one (and up to 6) sequences of steps. Refer to Appendix for an overview of the variations distribution.
4 Benchmark & Experiments
Table 2: Article grounding mAP. Comparison of state-of-the-art approaches on our benchmark. TS denotes the TimeSformer backbone.
Method	Training data	Video Backbone	Text Backbone	Seen (S1)	Unseen (S2)
↑ mAP@IOU	↑ mAP@IOU
@0.3	@0.5	@0.7	@[0.3-0.7]	@0.3	@0.5	@0.7	@[0.3-0.7]
Detection
ActionFormer [46]	HT-Step	TS [5, 25]	-	32.9	23.6	12.6	23.1	-	-	-	-
ActionFormer [46]	HT-Step	S3D [30, 44]	-	35.1	25.6	14.8	25.4	-	-	-	-
Grounding
UMT [26]	HT-Step	S3D [30, 44]	Word2Vec [32]	8.0	4.0	1.5	4.4	4.9	2.3	0.8	2.6
UMT [26]	HT-Step	TS [5, 25]	CLIP [33]	15.7	8.7	3.2	9.1	9.4	4.9	1.7	5.3
MT+BCE	HT-Step	S3D [30, 44]	Word2Vec [32]	31.5	19.6	8.1	19.7	18.6	10.3	3.7	10.6
MT+BCE	HT-Step	TS [5, 25]	Word2Vec [32]	26.2	16.5	6.0	16.1	13.5	6.8	2.5	7.4
ActionFormer-T	HT-Step	S3D [30, 44]	Word2Vec [32]	27.8	20.0	10.6	19.6	10.8	6.4	3.1	6.6
ActionFormer-T	HT-Step	S3D [30, 44]	MPNet [39]	36.9	26.5	15.3	26.3	27.4	18.3	9.3	18.4
VINA [28]	HT100M	S3D [30, 44]	Word2Vec [32]	12.6	4.7	1.2	5.9	8.2	3.1	0.6	3.8
MT+BCE (VINA)	HT100M+HT-Step	S3D [30, 44]	Word2Vec [32]	46.2	29.9	12.9	29.8	31.6	18.7	7.7	19.3
ActionFormer-T	HT100M+HT-Step	S3D [30, 44]	MPNet [39]	41.2	30.8	18.3	30.2	29.7	20.3	10.7	20.4
Task definition. The annotated data of HT-Step can be used to train and evaluate models for a plethora of tasks, from temporal article grounding to task-graph-based step localization [47], narration-based step grounding [28], captioning[21], and step forecasting [25]. In this paper, we focus on the temporal article grounding task: given an untrimmed video and an instructional article containing a sequence of steps, a model must predict temporal segments for the steps demonstrated in the video, along with a confidence score for every segment.
Evaluation protocol. For evaluating models, we introduce the article-grounding mAP - a variant of the popular mean Average Precision (mAP) metric from object detection [12, 24]. AP is separately computed for each activity by treating all steps within the activity as class-agnostic text queries. The per-activity APs are then averaged to obtain the article-grounding mAP. Following prior work [18], we evaluate AP under different temporal Intersection-over-Union (tIoU) thresholds.
We evaluate state-of-the-art approaches and baselines from step detection, single-step grounding, and multi-step grounding on our task. Full architecture, training, and implementation details can be found in Appendix.
ActionFormer [46] is a state-of-the-art, multi-scale action detection model that operates on a fixed taxonomy of steps (i.e., it does not use language), and as a result, can only detect seen steps (S1 split). ActionFormer-T is a variant of the ActionFormer model which we adapted to enable open-vocabulary step detection (i.e. unseen step grounding). It is constructed by replacing the classification head with a dot product between the textual embeddings of every step and the visual embeddings obtained from the temporal feature pyramid.
UMT [26] is a transformer-based model built for joint moment retrieval and highlight detection, and represents the state-of-the-art in single-step grounding.
VINA [28] is a model for temporal article grounding. Unlike traditional single-step grounding models, it can capture step ordering and the relationship between steps, and was trained with weak supervision from ASR transcripts and video-article pairs.
$\mathbf{MT} + \mathbf{BCE}$  is our modification of VINA for fully-supervised temporal article grounding. It embeds the full sequence of step headlines along with the video clips, and feeds them together to a multimodal transformer. We train it with a per-frame binary cross entropy loss (BCE) and produce segment predictions using a 1D blob detection routine [43].
Baselines comparison. Table 2 compares the performance of the aforementioned models for different choices of visual and textual backbones on the seen and unseen article test splits. Among all models trained from scratch on our training set, ActionFormer-T achieves the best performance on the seen test set (S1), while the performance of ActionFormer is slightly lower but comparable (26.3% vs 25.4% mAP). We conjecture that the structure of our training set, which has multiple video examples per activity, allows the ActionFormer to learn step representations without explicitly requiring language descriptions. However, modelling the language explicitly with ActionFormer-T yields a small boost, as it enables capturing similarities and subtle differences between step descriptions. More importantly, ActionFormer cannot be applied to detect steps of novel activities and articles in the unseen test set (S2). On this test set, our vanilla MT+BCE baseline that leverages the article structure (in contrast to grounding each step independently) performs reasonably well, outperforming UMT, even though the latter has been trained with a regression loss for more accurate temporal boundaries. This shows that leveraging the article step structure is very helpful for grounding novel articles in video. However, although ActionFormer-T has the disadvantage of performing single-step grounding, it outperforms all models in the unseen split too, presumably because it combines an architecture tailored to multi-scale detection with language modelling.
Weakly supervised pretraining. The state-of-the-art weakly supervised VINA model trained on HowTo100M ASR captions and wikiHow articles yields a low mAP metric since it was not trained with accurate temporal boundaries. However, using the same pretrained VINA model as initialization to finetune our  $\mathrm{MT + BCE}$  baseline on HT-Step leads to a remarkable performance boost, outperforming all previous baselines by a large margin, achieving 29.8 and  $19.3\mathrm{mAP}$  on S1 and S2 respectively. We perform a similar experiment by pre-training the ActionFormer-T (AF-T) model on the ASR captions of HowTo100M, then fine-tuning the model with the HT-Step annotations. The results are similar, with the model achieving very good performance in the high IOU evaluation settings, leading to the best overall results, i.e. 30.2 and  $20.4\mathrm{mAP}$  on S1 and S2 respectively. The VINA-pretrained model still shows better performance on the low-IOU evaluation, reflecting the strong localization abilities of the original model. These experiments clearly showcase the great
potential in combining strong step labels from HT-Step with noisy ASR captions from HowTo100M for training grounding models.
Table 3: Ablation of text used for step description.
Method	Text backbone	Text input	↑mAP @ [0.3-0.7]
Hdl.	Prg.	Act.	seen (S1)	unseen (S2)
ActionFormer-T	MPNet	✓	✗	✗	26.3	18.4
ActionFormer-T	MPNet	✓	✓	✗	26.1	18.0
ActionFormer-T	MPNet	✓	✓	✓	24.2	14.7
ActionFormer-T	MPNet	✓	✗	✓	24.0	13.7
ActionFormer-T	MPNet	✗	✓	✗	22.8	13.9
ActionFormer-T	MPNet	✗	✓	✓	21.1	11.3
MT + BCE	Word2Vec	✓	✗	✗	19.4	10.1
MT + BCE	Word2Vec	✓	✓	✗	19.2	10.3
MT + BCE	Word2Vec	✓	✗	✓	19.7	10.6
MT + BCE	Word2Vec	✗	✓	✗	17.0	8.3
MT + BCE	Word2Vec	✓	✓	✓	19.8	10.1
Table 4: Partial vs full matches.
Method	seen	unseen
partial	full	partial	full
Detection
ActionFormer [46]	20.4	29.6	-	-
Grounding
MT + BCE [17]	16.5	24.7	8.8	8.7
What text information is important for article grounding? We experiment with different combinations of a step's headline, paragraph and activity name to form its description and summarize the results in Table 3 (details for how every model combines text sources are provided in Appendix). It is evident that the headline contains the most valuable information. Paragraph information alone is enough for learning the task, however performance when using only paragraphs is lower than with headlines. Results from incorporating the activity name in the step descriptions vary depending on the model. Overall, although paragraph information is essential for accurate grounding, as shown in Figure 3, combining headline and paragraph provides marginal improvements. In addition, with all the approaches, there is a significant performance gap between the seen and the unseen test set. We conclude that temporal article grounding on our benchmark presents a challenging and exciting problem that will hopefully inspire the development of better models by the community.
What are the results on partial matches compared to full matches? To understand the differences between partial and full matches, we take models trained on the complete set and test them on subsets containing only partial-match or only full-match labels (details in Appendix). We present the results in Table 4. When evaluating on seen activities (S1), the performance of all models is significantly better on the full-match subset. This aligns with our intuition that full-match annotations should be easier to ground, as their step descriptions align better to what is visually demonstrated in the video. However, this gap disappears when evaluating on unseen activities (S2). We conjecture this may be due to full step matches being visually more consistent, resulting in models overfitting them on seen activities.
Table 5: Comparison to SOTA for zero-shot step grounding on the CrossTask dataset.
Method	↑Avg. R@1 (%)
Zhukov [50]	22.4
HT100M [31]	33.6
VideoCLIP [45]	33.9
MCN [7]	35.1
DWSA [38]	35.3
MIL-NCE [30]	40.5
VT-TWINS [19]	40.7
UniVL [27]	42.0
AF-T (HT100M)	37.1
AF-T (HT100M + HT-Step)	48.5
Table 6: Zero-shot article grounding evaluation on CrossTask.
Training data	↑Avg. R@1 (%)	mAP@[0.3-0.7]
HT100M	37.1	5.2
HT100M + COIN	41.4	6.7
HT100M + HT-Step	48.5	9.5
Do models trained on HT-Step generalize to other tasks? We showcase the potential of leveraging HT-Step for model pre-training by evaluating one of our models on zero-shot atomic step localization. In particular, we pre-trained the ActionFormer-T (AF-T) model on the ASR captions of HowTo100M (following prior work [27, 30, 45]), then fine-tuned the model with the HT-Step annotations. We perform evaluation on the CrossTask dataset, following standard protocol [50], i.e., we compute the recall per task based on a single predicted timestamp per step, computed over 20 random subsets of the training videos. As Table 5 demonstrates, it is clear that training on HT-Step results in a
huge boost in performance, namely a  $+11.4\%$  absolute improvement, but also surpasses the previous state-of-the-art on this benchmark by a large margin  $(+6.5\%)$ .
To further motivate the usefulness of the HT-Step annotations compared to existing datasets, we train the same model on the COIN dataset [41], and perform zero-shot evaluation on CrossTask, using both the standard recall metric (for single timestamp prediction evaluation) as well as mAP (for temporal segment and precision evaluation). The results are shown in Table 6. It is clear that fine-tuning on the HT-step annotations results in a much more substantial performance improvement compared to fine-tuning on COIN ( $+7.1\%$  R@1 and  $+2.8$  mAP).
We have introduced HT-Step, a dataset containing temporal annotations of instructional article steps in cooking videos. The dataset offers a large-scale training resource for the task of temporal article grounding, presenting new challenges and encouraging the development of alignment methods that leverage order and procedural structure. By releasing the annotations and providing benchmark protocols, we aim to spur new research in this domain and advance the field of procedural activity understanding.
Limitations and societal impact We acknowledge that HT-Step is intended for research purposes and should not be regarded as a comprehensive dataset encompassing the full range of human activities. Models trained on our dataset may exhibit biases towards the specific activities included in the dataset, resulting in a limited coverage of our everyday living scenarios.
Acknowledgements. We thank Mandy Toh, Yale Song, Gene Byrne, Fu-Jen Chu, Austin Miller, and Jiabo Hu for helpful discussions and invaluable engineering support.
[1] Top Content Categories on YouTube. https://www.thinkwithgoogle.com/consumer-insights/consumer-trends/top-content-categories-youtube. Accessed on June 1, 2023. 2
[2] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4575-4583, 2016. 3
[3] Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII, pages 657-675. Springer, 2022. 3
[4] Peijun Bao, Qian Zheng, and Yadong Mu. Dense events grounding in video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 920-928, 2021. 4
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[6] Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Locvtp: Video-text pre-training for temporal localization. In European Conference on Computer Vision, 2022. 2, 3
[7] Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. Multimodal clustering networks for self-supervised learning from unlabeled videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8012-8021, 2021. 9
[8] Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, and Shih-Fu Chang. Weakly-supervised temporal article grounding. In Conference on Empirical Methods in Natural Language Processing, pages 9402-9413, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. 3, 15
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV), 130:33-55, 2022. 3
[10] Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A thousand frames in just a few words: Linguual description of videos through latent topics and sparse object stitching. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 2634-2641, 2013. 3
[11] Nikita Dvornik, Isma Hadji, Konstantinos G. Derpanis, Animesh Garg, and Allan D. Jepson. Drop-dtw: Aligning common signal between sequences while dropping outliers. ArXiv, abs/2108.11996, 2021. 3
This Appendix provides more comprehensive statistics about the dataset (Section A), architecture and implementation details for the baselines discussed (Section B), a breakdown of model performance by activity (Section C), and extra details about the evaluation protocol (Section D).
A Extra statistics
Figures 6a-6i contain additional statistics about HT-Step.
(a) Distribution of step coverage across activities.
(b) Distribution of step coverage across videos.
(c) Distribution of annotated video durations.
(e) Distribution of step coverage across videos.
(d) Distribution of step coverage across activities.
(g) Distribution of unique steps annotated per video vs steps in taxonomy
(f) Distribution of annotated video durations.
(h) Distribution of the number of variations appearing in HT-step across activities.
(i) Distribution of annotated video durations.
Figure 6: Additional statistics of HT-Step.
B Baselines implementation details
We train all the baselines on top of the same feature sequences, extracted from frozen backbones.
TimeSformer (TS) features. The TimeSformer features are extracted using the public model<sup>3</sup> of [25] pre-trained with distant supervision on HowTo100M. We obtain 1 feature per second, by resampling the video at 8 fps and extracting features with a stride of 8 frames. The feature dimensionality is 768.
<sup>3</sup>https://dl.fbaipublicfiles.com/video-distant-supervision/TimeSformer_divST_8x32_224_HowTo100M_pretrained.pth
S3D features. The S3D features are extracted using the published model of [30] pre-trained with MIL-NCE [30]. We obtain 1 feature vector per second, by resampling the video at 16 fps and extracting features with a stride of 16 frames. The feature dimensionality is 1024.
ActionFormer. We use the official ActionFormer implementation<sup>5</sup> provided by the authors[46]. We set the number of classes to 4958 i.e., one detection output for each step in the taxonomy. We set the max sequence length to 512 and train for 20 epochs with a batch size of 16, using the AdamW optimiser with cosine learning rate schedule, base learning rate of  $1e - 4$  and 5 warm-up epochs on a single Nvidia Tesla V100 GPU with 32GB of memory.
ActionFormer-T. ActionFormer-T is trained with the exact same hyper-parameters, loss and labels as ActionFormer. We extract the text representations, using the MPNet implementation provided by the sentence_transformers library<sup>6</sup>. The text embeddings are frozen, so no gradients are backpropagated into the pre-trained language model. We use a 2-layer MLP with hidden side 512, ReLU activation and Layer Normalization, to project and transform the 768-dimensional text embeddings into the 512-dimensional video embedding space. Step text descriptions from different sources (e.g. headline, paragraph, activity, see also Table 3 of the main paper) are combined by simple concatenation at the text level, i.e. the combined sentences including all three have the form "Activity: Headline. Paragraph".
UMT. For UMT, we use the authors' official code<sup>7</sup>. We train models with learning rate  $1e - 3$  and batch size 64 and train for 200 epochs. We use only the unimodal encoder for video (we do not use the audio encoder, or the cross-attention modules). All remaining hyperparameters follow the configuration for the QVHighlights task provided by the authors.
$\mathbf{MT} + \mathbf{BCE}$ . The input to our temporal article grounding baseline is a temporal sequence of visual features extracted with a sliding window (using either the TimeSformer or S3D backbones as explained above), and a sequence of step sentences (consisting of the activity name and the article step headlines). We base our model on the VINA [28] architecture by removing the additional narrations modality, i.e., we do not use the narration unimodal encoders, positional encodings and the alignments of steps to narrations or narrations to video. We use the  $\mathrm{TAN}^8$  codebase for our implementation. All of the architecture hyperparameters (e.g., number of Multimodal Transformer layers, embedding dimensions etc.) are adopted from VINA. The only difference is the maximum length of the input video which we increase to 1200 seconds to account for the longer videos in the HT-Step training set.
To obtain temporal segment predictions for each article step from the Multimodal Transformer outputs, we: (1) compute the normalized dot product between each step contextual embedding and each video clip contextual embedding. This results in a  $T \times S$  alignment matrix, where  $T$  is the number of timesteps and  $S$  is the number of steps. (2) We pass these similarities through a sigmoid activation (with temperature 0.07) to obtain a confidence score about whether each timestep  $t$  is aligned with step  $s$ , (3) we post-process the temporal sequence of confidence scores for each step with an 1D blob detection routine to obtain temporal segments at multiple scales. In particular, we apply Laplacian of Gaussian filters at 13 scales, covering Gaussian standard deviations from 1 to 480 [43].
The model is trained with binary cross-entropy loss applied at each temporal timestep and for each article step. We train our model for 9 epochs using the same optimizer, learning rate and batch size as VINA [28].
Adding paragraph information: For our ablations in Table 3 of the main paper, we added paragraph information to the  $\mathrm{MT + BCE}$  model simply by interleaving step headlines with step paragraph sentences. In other words, we encode and feed that sequence of sentences to the Multimodal Transformer. We use the same positional encoding for all sentences associated with the same article step. In order to
4https://github.com/antoine77340/S3D_HowTo100M
<https://github.com/happyharrycn/actionformer_release
<https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2>
<sup>7</sup>https://github.com/TencentARC/UMT/tree/main
<sup>8</sup>https://github.com/TengdaHan/TemporalAlignNet
obtain a single contextual embedding for each step, we max pool the embeddings of the headline and paragraph sentences of that step.
Model weights initialization: We train all variants of MT+BCE from scratch, except for the model in the last row of Table 2, which was trained after initializing the unimodal encoders, positional encodings and Multimodal Transformer weights using a VINA model pretrained on the HTM370k [17] subset of HowTo100M using pseudo-labels for wikiHow steps (and no ASR narrations) [28]. For this experiment, we adopt the same maximum video length as VINA (1024 seconds).
C Per-activity predictions
In Table 7 we show the per-activity AP breakdown of the performance of the two best models. We show the 25 highest and 25 lowest scoring activities, ranked by the performance of the ActionFormer detection model. Note that activities that are challenging for the fixed taxonomy, detection model (such as Cook Pork Tenderloin for which the AP is  $0.62\%$ ) are handled better by the temporal grounding model (achieving  $28.9\%$  AP for Cook Pork Tenderloin). For this particular example of Cook Pork Tenderloin, this can be explained since this activity has only 4 examples in the training set. Therefore, the detection model does not have enough training samples to learn a good representation for the steps of this activity. On the other hand, the temporal article grounding model, that has been initialized with a model trained with weak-supervision on a much larger dataset (HTM370k) can perform better in this few-shot scenario. Another interesting observation is that for some activities the detection-based model outperforms language-based grounding.
D Evaluation protocol details
D.1 Article-grounding AP metric