HT-Step: Aligning Instructional Articles with How-To Videos
Authors: Triantafyllos Afouras
Effrosyni Mavroudi
Tushar Nagarajan
Huiyu Wang
Lorenzo Torresani
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: step annotations
temporal article grounding
instructional video
instructional articles
how-to
TL;DR: We introduce HT-Step, a large-scale dataset of step annotations on instructional videos.

Abstract: We introduce HT-Step, a large-scale dataset containing temporal annotations of instructional article steps in cooking videos. It includes 122k segment-level annotations over 20k narrated videos (approximately 2.3k hours) of the HowTo100M dataset.
Each annotation provides a temporal interval, and a categorical step label from a taxonomy of 4,958 unique steps automatically mined from wikiHow articles which include rich descriptions of each step.
Our dataset significantly surpasses existing labeled step datasets in terms of scale, number of tasks, and richness of natural language step descriptions. Based on these annotations, we introduce a strongly supervised benchmark for aligning instructional articles with how-to videos and present a comprehensive evaluation of baseline methods for this task.
By publicly releasing these annotations and defining rigorous evaluation protocols and metrics,
we hope to significantly accelerate research in the field of procedural activity understanding.

HT-Step: Aligning Instructional Articles with How-To Videos
Triantafyllos Afouras"
Effrosyni Mavroudi*
Tushar Nagarajan
Huiyu Wang Lorenzo Torresani
FAIR, Meta
fafourast, emavroudi, tusharn, huiyuw, torresani)@meta.ccm
Abstract
We introduce HT-Step, a large-scale dataset containing temporal annotations of instructional article steps in cooking videos. It includes 116k segment-level anno- tations over 20k narrated videos (approximately 2.1k hours) of the 1000 dataset. Each annotation provides temporal interval, and categorical step la- bel from taxonomy of 4, 958 unique steps automatically mined from wikiHow articles which include rich descriptions of each step. Our dataset significantly surpasses existing labeled step datasets in terms of scale, number of tasks, and richness of natural language step descriptions. Based on these annotations, we introduce strongly supervised benchmark for aligning instructional articles with how-to vidcos and present comprehensive evaluation of baseline methods for this task. By publicly releasing these annotations and defining rigorous evaluation protocols and metrics, we hope to significantly accelerate research the field procedural activity understanding.
wiki How
In large mixing bowl, whisk together the flour, sult, and baking soda. Mixing ingredients together sepurately
How-Ta Video
ensures
2. Beat the butter in separate buwl uniil becomes light fluffy. You do using whisk or an electric mixer..
Beat the white and brown sugar for
time
might
to mix if you add the sugar in little bit at time
66:00
86-ilt
90:00
104:00
106:00
125:00
130:00
Whisk in vanilla extract the milk, Keep whisking until everything evenly combined.
Non-groundable
steps:
5.
Beat in the flour mixture. At this point,
Step
the misture will start to stick logether
Step
form ball. you were using whisk,
full match? Yes
Is full match? No
Figure 1: HT-Step is dataset of annotations aimed towards instructional video-artee article alignment. Given an untrimmed how-to video and an instructional article containing list of step descriptions, the goal is to temporally localize the steps shown in the video and reject the ones not demonstrated. Our dataset includes over 11kk segment-level annotations for this task, as well as labels indicating whether temporal segment is full or partial match to the step, i.e., whether the segment demonstrates all substeps and ingredients mentioned in the step headline (shown in bold).
"equal contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Instructional videos have become a popular way to learn or improve our skills, providing an entertain- ing and dynamic alternative to traditional written manuals. For example, how-to videos are listed as one of the top-four watched categories on the YouTube platform [1], where viewers can access narrated visual demonstrations for a wide range of activities, from cooking to crafts and DIY projects. In addition to being a valuable learning medium for humans, how-to videos are a great source of training data for computer vision models [6, 25, 30J. Since many of these instructional videos focus on complex activities entailing a sequence of steps, these repositories offer great potential for learning task graphs [47] and training procedural activity models [42, 50J. However, although the videos often accompanied by informative closed captions or ASR transcriptions, they lack structured step annotations that clearly denote which video segments correspond to what steps of the activity.
An alternative source of information for acquiring new skills is instructional article collections, such as wikiHow [20]. These typically provide list of procedural steps and have the advantage of being more comprehensive, incorporating variations and detailed explanations. Although they often contain still-pictures or graphic illustrations, they lack the dynamic demonstrations offered by videos.
In this work we introduce HT-Step. the largest existing dataset with labeled step segments on instructional videos. We leverage wikiHow articles [20] to guide the annotation on a large subset of the open-sourced How NowToo00M dataset [31]. These cross-dataset associations provide us with a taxonomy of steps well fitting the data and with an automatic mechanism to identify the set of potential steps represented in each individual video. Through a meticulous process of manual labeling by professional annotators we obtain 116k temporal step labels over 20k videas (approximately 2.1k hours). Our benchmark significantly surpasses the largest existing datasets of labeled steps [41, 50] in procedural videos along multiple axes: scale (2.6x the number of segments annotated, 4.8x the number of video hours), taxonomy granularity (7x the number of steps), and scope (2x the number of activities).
We share these annotations with the research community and present benchmark for the task of temporal article grounding. Given as imput how-to video and an instructional article listing a sequence of potential steps, the objective to temporally ground the subset of steps that are demonstrated in the video and to detect outliers (i.e., the steps not executed). We note that this task differs from traditional temporal grounding task which require localizing single query that is known to be present in the video. Instead, in our problem setting the system is given the complete sequence of potential steps. We believe that this formulation may encourage the design of methods that can perform global reasoning over all the steps and take full advantage of ordering constraints and sequence priors to disambiguate the grounding. Furthermore, on average about 58% of the steps listed in the associated wikiHow article are not demonstrated in the given video. This implies that the model must have strong capability to recognize the steps that are not groundable. Finally, we include in our benchmark test set over unseen tasks, i.c.. activities that are not represented in the training set. Traditional activity detection methods requiring a taxonomy known a priori cannot generalize to unseen activities. By introducing this setting we hope to promote the development of language-bart based temporal alignment models that can learn to ground the rich textual descriptions of steps in instructional articles even for never-seen tasks.
In summary, our contributions are the following: (1) We introduce a large collection of step anno- tations on videos from the popular HowTo 1000 dataset. (2) We propose a scalable pipeline for annotating procedural videos that leverages an automatic strategy for associating potential steps to video and for inferring the step taxonomy. (3) We introduce splits, evaluation protocols and metrics that evaluate the ability of models to predict the temporal extent of steps in vidcos, and reject steps that are not visually groundable. (4) We train and evaluate state-of-the- approaches and baselines that tackle our task from three different angles: activity detection, hingl-ssentence temporal grounding. and temporal article grounding. (5) We discuss properties and unique aspects of HT-Step through comprehensive analyses and experiments.
2 Related Work
Procedural and Instructional Video Datasets. Procedural videos portray humans performing sequential steps in some constrained yet non-unique order, with the aim of achieving specific goal, such as preparing a dish. Understanding the content of such videos has been active area of

---

Table Comparison between Step and existing procedural video understanding (top), temporal grounding (middle), and instructional video (bottom) datasets. Note that HT-Step seen validation and test sets introduced in [28] are subsumed by HT-Step.
Dataset
Duration (b)
Videos
Segments
# Activities
Domain
Procedural Activiry Activiry Dunasets
MPI] [35]
10
44
5.6k
cooking
50Salads [40]
4.5
50
899
cooking
TACoS [34]
600
127
18.8k
cooking
Breakfast [22]
77
2k
8.5k
10
cooking
Ikea I.A [16]
101
1.9%
furniture
PPPC-KITCHENSSP [9]
90k
cooking
EgoProcel [3]
62
298
1k
16
multiple
Assembly 101 [36]
513
4.3k
1M
101
assembly
Temporal Grounding Datazets
AetivityNet Captions [21]
20k
multiple multiple multiple
Charades-STA [14]
90
10k
18k
157
QV-highlights [23]
10.2
16.1k
Instructional Video Datavets
YouCook [10]
88
cooking
YouCook2 [48]
176
2k
13.8k
cooking
YouwikiHow [8]
47k
1398
multiple
CrossTask [50]
213
2.8k
21k
18
multiple
COIN [41]
476
11.8k
46.4k
180
multiple
HT. SteStep Soen Val+Test Set [28]
124
1.2k
7k
177
cooking
HT-Step (Ours, Full)
2.1k
19.7k
116k
433
cooking
HT-Step splits
Step HT Step Val seen
cooking
64
600
3.4k
120
cooking
HT-Step Test seen (SI) IIT Step Text (S2)
cooking
116
1000
5.7k
32
cooking
research over the last decade [2, 6, 11, 13, 22, 34, 37, 47]. largely driven by the release of benchmark datasets [22, 41, 50]. Early datasets such Breakfast [22] and 50Salads [40] are manually recorded, small-scale (spanning at most 100 hours) and limited in activity diversity (e.g., covering only breakfast recipes [22] or IKEA furniture assembly [16]). More recent datasets, such as COIN [41] and CrossTask [50]. have capitalized on public video collections, including videos sourced from YouTube, in order to substantially increase dataset scale and activity diversity. A key characteristic of these videos is that they are instructional, i.e., they typically involve an individual teaching a complex task by narrating the sequence of steps.
Our HT-Step dataset also provides temporal annotations for steps of instructional videos. However, it surpasses existing datasets in terms of scale and activity diversity, the steps are sourced from real-world instructional articles (wikiHow (20]) rather than being manually defined. This approach leads to much richer step descriptions, including paragraph providing detailed instructions. Despite its exclusive focus on the cooking domain, HT-Step is significantly larger in scale and step description complexity. Our work significantly expands the small-scale val/test set that was recently released [28] for evaluating step grounding approaches trained without annotated temporal segments. While the focus of that work was the design of weakly supervised methods leveraging narrations "free" signal for step-to-video alignment, here we share the first large-scale dataset enabling strongly-superrisedd training of methods for temporal article grounding. Our annotations over large-scale training set, and a new test set of unseen tasks (not included in the training set) provide almost 20x more annotated videos and number of segments. We also include additional metadata per step (indicating full or partial segment-step matches), Furthermore, we include a comprehensive experimental evaluation of strongly supervised baselines on this task.
Instructional Video Understanding There exist various tasks related to instructional video un-
derstanding, such step classification [25], step detection [41, 46], action segmentation [13],
captioning [49], video retrieval [15], visual object grounding [29]. and temporal grounding [26]. In this paper, we focus on the recently-intoouuce task of temporal article grounding, which has been tackled only with weakly-supervised approaches so far [8, 28]. In particular Chen et al. [8] is the first work to train and evaluate models for the temporal article grounding task. However this work was limited to weakly-supervised training due to the lack of a strongly labeled dataset, and used Cross Task as proxy dataset for evaluation, with noisy mapping between article steps and Cross Task labels. In contrast, introduce a large amount of strong temporal segment annotations
3

---

Step headlnns
Apple Dumplings
Stop headinng
Crepes
mach parce.
eumporeee and small huohles forn the parn"
"Fola the cropics quarters "heat
Place We dartaingg on the prepared bosicg bay."
"Movo propes with monocen apuntule
"Bnnly oour the sauce now the sumplegge.
recoppoeee the vouce stilee healing the dropes.
Make Chicken Piccata
Make Naadan Kadala Curry
"Cout with pappor"
"Add orions the pan and case0 thoty
(redge complewy"
"Add the tomatoes and water iaam out an the pan and let cook 10 minitee.
both 3-4 AVtutes
"Put cooker onlelpeas and met.""
Figure 2: Sample frames for 12 videos spanning tasks from the HT-Step training set. For each video, we show relevant frames for three steps. By annotating multiple videos of the same task, we obtain multiple instances of each step, which have high variations in appearance and viewpoints.
for training, and propose a metric that evaluates both the recall and precision of models, and is thus better suited to the task of temporal article grounding which involves multiple ungroundable steps. The temporal article grounding task is closely related to two other grounding tasks: (a) temporal grounding of a single natural language query [14, 23] (also known as moment retrieval) and (b) video paragraph grounding [4]. Benchmarks [14, 21, 23] for these tasks typically consist of video captions or questions that are loosely related to each other and which were obtained by human annotators based on the video contents. Thus, most approaches assume that each of the queries is groundable in the video. For example, video paragraph grounding aims at predicting a single temporal segment for each sentence of the paragraph. Instead, in our HT-Step benchmark the step queries originate from wikiHow articles, which might contain steps that are not shown in the video (non- moon-grouddbble steps) or steps that are partially executed (partial matches). This makes challenging adapt temporal grounding approaches for temporal article grounding in our benchmark.
3 HT-Step
The creation of H'T-Step involved pairing videos from How with instructional articles from wikiHow and then annotating segments in each video with steps of the associated article. W an online knowledge base that houses thousands of articles with step-by-step instructions for large variety of activities. Every article is typically structured as a series of steps, each step anchored by descriptive headline and further supplemented with a detailed paragraph. HowTo100M on the other hand, contains an extensive compilation of instructional videos that were collected using titles of wikiHow articles as search keywords for a subset of activities deemed contain visual demonstrations. As result of this sourcing procedure, each video in HowTo comes with task label that corresponds to a specific wikiHow article. These pairings create natural opportunity for annotating the HowTo 100M videos with the step labels of the corresponding wikiHow articles, which we exploited for creating HT-Step. Next we describe in detail our annotation process.
3.1 Annotation workflow
Activity selection. HowTo 100M spans variety of domains, including cooking, DIY, crafts, sports, and gardening. As in previous works [17], we chose to focus on cooking activities as i) they account for approximately third of the HowTo100M videos, and ii) they are relatively low-complexity tasks which can be annotated by non-experts. Using metadata, we selected 495 cooking activities (detailed list the Appendix) for which HowTo100M contains at least 70 videos.
Step Taxonomy. We automatically sourced the step descriptions for all the activities from wokiHow. For every activity, we extracted the steps from the corresponding wikiHow article, forming step taxonomy that includes 4, 958 unique steps. We list the full taxonomy in the Appendix.
? hhttps://wwe wikihow.com com/

---

Video validation. For every video, the annotators were given the corresponding activity title and step taxonomy. They were then asked to watch the whole video and determine whether it contains the specified activity. If the video was found not matching the task, it was rejected.
Temporal annotation. During the temporal annotation loop annotators were asked to provide temporal boundaries for the wikiHow steps represented in all the non-rejected videos. An example annotation is illustrated in Figure 1.
Task variations. wikiHow articles often list several variations for a given task. For example, the instructional recipe for "Make Taco Salad" includes three versions: basic, deluxe and vegetarian. For such activities, we asked annotators to choose the variation that best fits the video and to annotate only the steps listed in that variation.
Full vs partial matches. We asked the annotators to mark a step as a "full" match if (i) all the instructions and ingredients listed in the step headline are shown the segment and (ii) the step is brought to full completion. Conversely, "partial" indicates that some sub-steps in the step headline are skipped, some ingredients are not used, or that step is interrupted and continued later.
QA process. To ensure the quality of the annotations, we followed rigorous multi-stage Quality Assurance (QA) process. In the first stage, the videos were annotated by a single annotator. These initial annotations were then reviewed by more experienced annotators, who either approved all the annotations on a given video (all the marked steps were correct and no steps were missing) or marked it for re-doing, with specific comments indicating the necessary correction. At the last stage of the QA process, the annotations that were marked as incorrect were redone by a third set of annotators. Due to budget constraints, the full QA enforced Annotation output. The annotation process took approximately 33k person-hours by 90 professional human annotators. Overall, 34% of the videos were rejected as not matching the specified activity, resulting in annotations for 433 of the initial 494 activities. We summarize the statistics of the resulting annotations and compare to existing datasets in Table
sample of roughly 13% of the total annotations.
3.2 Training, validation, and test splits
We formed two types of test sets: one for seen activilies, i.e., activities that are also included in the training set, and one for unseen activities, i.e., novel activities not appearing in the training set. For seen activities we also formed validation (val) set containing the same activities.
Seen val/test set (SI). We directly adopt the HT-Step validation and test splits [28] as the seen evaluation sets of our dataset, each containing 600 videos, 5 samples for each of 120 activities.
Unseen test set (S2). To create the unseen test set, we selected 32 activities that do not appear in the training set or the seen val/test set. We include all the annotations on videos of these activities that went through the QA review, resulting in test set of 1000 annotated videos. This set is imbalanced w.r.t. activities, with the video count per activity ranging from 17 to 70.
Training set. All the other annotated videos are included in the final training set, which contains more than 17k vidcos and 100k annotated segments, We summarize the split statistics in the bottom part of Table 1.
3.3 Properties
Article steps. As previously mentioned, step descriptions in wikiHow articles have two parts: a headline and a paragraph. The headline is brief and may not reveal enough information for grounding; the paragraph contains additional details about the execution of the step. Annotators were asked to localize each step by considering both. This allowed us to obtain annotations where the paragraph description is essential for grounding the steps. We show some examples in Figure 3.
Composite steps. Many wikiHow steps are composite, i.e., include multiple sub-steps. Two examples of such steps are shown in Figure 4 (b,c). We introduced the labeling of partial vs full matches (discussed in Section 3.1) account for the fact that composite steps are rarely fully represented in videos or they may be carried out in disjoint time intervals of the video.
-onnground steps. Some of the steps listed in the wikiHow article may not be represented at all in the video. An example is illustrated in Figure 1. Temporal grounding models that are trained/evaluated on this data must support handling queries with no associated temporal segments.
3.4 Statistics
In this section we provide insightful statistics about the dataset. See Appendix for more.
5

---

Dice the celery, carrot, and pickle. Use sharp knife to carefullt dice enough onion measure 1/4 cup (40 g). You'll also need to dice enough celery meusure 1/4 cup (55 g) and enough pickle messure 1/4 cup Tranzfor the vegetables to the houl with the masked chickpeus.
11
Make the cake ball dough. Place twa-inch (5-cm) cubes the food processcr pulse broken up into fine crumbs, Transfer the crumhs to mixing bowi, [...] you don't have food processor, placs the chunks of cake into a miking bowl and mash them with a fork or break them up with your hands.
cake
Figure 3: Qualitative examples where the paragraph essential to localize the step. The paragraph sentences that are visually demonstrated in the video are highlighted in yellow.
(a) Step: Fill the miini-Stromboli with meat, cheese, and seasoning. (b) Step: Pile the roast pork.
(c) Step: Add broccoli, carrots, and onions.
(c) Step: Cook chicken for minutes.
Figure 4: Examples of composite steps and full/partial matches. Frames annotated as "full matches" are shown in green, "partial matches" are shown in orange. (a) An atomic step, fully matched. (b) A composite step: it is labeled as full match as all the ingredients and the substeps mentioned in the paragraph are shown in the video. (c) Another composite step: here not all the described ingredients are utilized and thus it is labeled as "partial match." (d) An example of step fragmentation: the step executed over two disjoint segments which are consequently labeled as "partial matches."
Overview. Table 1 summarizes the statistics of HT-Step. It is larger and has much richer step taxonomy than existing datasets. Compared to COIN, has 4.8x the duration, 2.6x the number of annotated segments, 7x the taxonomy size (4958 vs 778 unique steps), and 2 the number of videos. Segment duration. According to Figure 5a, the average segment duration is approx. 15 seconds, with most of the annotated segments at 10 seconds or shorter and a long tail of rare longer segments. Video coverage. We define video coverage as the percentage of a video that is covered by annotations. Figure 5b shows the distribution of the average video coverage per activity, with a mean of 22%.
Step coverage. Step coverage denotes the fraction of unique steps from the associated wikiHow article that are annotated in a video and is on average 42%. Figure Sc shows that even for the activities with the highest step coverage, many article steps have no relevant temporal segments in each video. Distribution of segments over activities. Figure 5d shows how the number of annotations is distributed over the 4, 958 steps. On average we get 24.6 segments per step. Since the choice of steps to be annotated was determined by the data, several steps have very few training samples, as indicated by the large concentration the first bucket. We believe that this property will encourage the development of methods that can perform zero- or few-shot detection, as well as techniques that approach the task as language- temporal grounding as opposed to class-based detection.
Length of text descriptions. We calculate the number of words as a proxy for the complexity of the step descriptions. In Figure 5e we provide the distribution of this metric averaged over activities, for HT-Step and COIN [41]. It clear that HT-Step contains richer text descriptions, even when only the headlines are used (8.5 words vs 4.9 for COIN on average). When paragraph text is included in this calculation, the word count increases even further, to 17.1 1 words on average.
Step ordering variation. The steps a wikiHow article are listed according to a recommended order of execution. In order to capture real- world deviations from these canonical orderings. we measure the Normalized Edit Distance (NED) between the annotated step sequence and the wikiHow listing, normalized over the length of the annotated sequence in each video. We calculate the mA-NED as the mean of the average NED per activity. We compute the same metric for the COIN dataset, using the ordering provided with the taxonomy of this dataset. Steps in our dataset are much more frequently executed out of order compared COIN (mA-NED is 0.37 vs 0.27), as shown in Figure 5f. This presents much more challenging temporal grounding setting.

---

20KK
Moan: 14,94s
Moan: 0.22
# Seymenis
15K.
Activities
30-
55-
12K
25
# 12
20
XK
10
Mare puats Waat
0K
10
60
70
0.1
0.3 0.4
0.6
Sogment duration (seconds)
Video coverage
(a) Distribution of segment dura- tions. The average segment dura- tion about 15 seconds, with the annotated segments at 10 sec- onds or shorter, and a long tail of rare longer segments.
(b) Distribution of video coverage, averaged per activity. Video cov- erage denotes the percentage of video that is covered by step annota- tions. exhibits a normal distribu- tion around the mean.
Activity
(c) Average number of unique steps annotated per video for the 20 activities with most unique steps.
Mean:
Moan: 4,9 words
Moun: 0.23
Monn: words
# Activities
Stege
Steps
600
Mean: 17.1 words
400
*pp.. 00I
200
30
70
80
0.0 0.1
0.2
0.3
0.4 0.5 0.6 0.7
0.8
# Amnotations
#f Words
NED
(d) Distribution of annotations. Annotations follow long-tail distri- bution which should encourage the development of zeroffew-sht detec- tion or text grounding methods.
(e) Comparison of text description lengths. HT-Sttep contains long step descriptions, almost twice long as those COIN, even when only the headline text is considered.
(1) NED. Step sequences in HT-Step have 0.37 NED compared the canonical step ordering on average, which makes more challenging compared to previous benchmarks.
Figure 5: HT-Step statistics. We provide various statistics for a qualitative overview of the dataset.
Full vs partial matches. Of the resulting annotated segments, 27% are full and 73% are partial matches. Some examples of full and partial match annotations shown in Figure 4.
Variations. Most tasks (335 out of 433) involve single listing of steps in wikiHow. However there is significant number of activities (98) that was annotated according to more than one (and up to 6) sequences of steps. Refer to Appendix for an overview of the variations distribution.
4 Benchmark & Experiments
Table 2: Article grounding mAP. Comparison of state-of-the-art approaches on our benchmark. TS denotes the Tiorer backbone.
Method
Training data
Video Backhone Text Backhone
Seen (S1)
Unseen (S2)
mAP@1OU
mAP@10U
@0.3
@80.5 60.7 @ (0.3300.71
0.73 @0.3 60.5 @0.7 (0.3..0.7
Detection
ActionFormer [46]
IIT Step
[5]
32,9
23.6
12.6
23.1
Aotier [46]
HT-Step
S3D [40,44]
35.1
25.6
14.8
25.4
Grounding
UMI [26]
HT-Step
S3D 44]
Word2Vee [32]
8.0
4.0
1.5
4.4
4.9
2.3
0.8
2.6
UMT [26]
TS [5,
CLIP [33]
15.7
8.7
3.2
9.1
9.4
4.9
1.7
5.3
MT+RCE
ST-
S3D [40
Wond2Vec [32]
31.5
19.6
8.1
19.7
18.6
10.3
3.7
10.6
MT+BCE
HT-Step
TS [5, 25]
Word2Vee [32]
26.2
16.5
6.0
16.1
13.5
6.8
2.5
7,4
ActionFormer-T
HT-Step
S3D
Word2Vee [32]
6.4
3.1
ActionFonmer-1
HT-Step
S3D [40
MPNet [39]
36.9
26.5
15.3
26.3
27.4
18.3
9.3
18.4
VINA [28]
HT100M
S3D [30,
Word2Vec [32]
12.6
4.7
1.2
8.2
3.1
0.6
3.8
MT+BCE (VINA)
ST100M-HT-Step
S3D [30.44]
Word2Voe [32]
46.2
29.9
12.9
29.8
31.6
18.7
7.7
19.3
ActionFFrmer-T
HUTO0M+HT-Stp
S3D [40
MPNet
30.2

---

Task definition. The annotated data of HT-Step can be used to train and evaluate models for plethora of tasks, from temporal article grounding to task graph-baser step localization [47], carration-bases step grounding [28], captioning(2 [21]. and step forecasting [25]. In this paper, we focus on the temporal article grounding task: given an untrimmed video and an instructional article containing a sequence of steps, model must predict temporal segments for the steps demonstrated in the video, along with
a confidence score for every segment.
Evaluation protocol. For evaluating models, we introduce the article priice-grounding mAP- variant of the popular mean Average Precision (mAP) metric from object detection [12, 24j. AP is separately computed for each activity by treating all steps within the activity as class-agnostic text queries. The per-activity APs are then averaged to obtain the article-grounding mAP. Following prior work [18], we evaluate AP under different temporal Intersection-over Union (tIOU) thresholds.
4.1 Results
We evaluate state-of-the-art approaches and baselines from step detection, single-step grounding, and multi-step grounding on our task. Full architecture, training, and implementation details can be found in Appendix.
ActionFormer [46] is a state-of-thh-aa taxonomy of steps (i.e., it does not use language), and as a result, can only detect seen steps (S1 split), ActionFormer-TT is a variant of the ActionFormer model which we adapted to enable open-vocabulary step detection (i.e. unseen step grounding). It is constructed by replacing the classification head with a dot product between the textual embeddings of every step and the visual embeddings obtained from the temporal feature pyramid.
multi-scale action detection model that operates on fixed
UMT [26] is transformer-ban based model built for joint moment retrieval and highlight detection, and represents the satt--of-the-an in single-step grounding.
VINA [28] is model for temporal article grounding. Unlike traditional single-step grounding models, it can capture step ordering and the relationship between steps, and was trained with weak supervision from ASR transcripts and video-article pairs.
MT+BCE is our modification of VINA for fully-supervised temporal article grounding. It embeds the full sequence of step headlines along with the video clips, and feeds them together to a multimodal transformer. We train it with a per-frame binary cross entropy loss (BCE) and produce segment predictions using a ID blob detection routine [43].
Baselines comparison. Table 2 compares the performance of the aforementioned models for different choices of visual and textual backbones on the seen and unseen article test splits. Among all models trained from scratch on our training set, ActionFormer-T achieves the best performance on the seen test set (SI), while the performance of ActionFormet is slightly lower but comparable (26.3% vs 25.4% mAP). We conjecture that the structure of our training set, which has multiple video examples per activity, allows the ActionFormer to learn step representations without explicitly requiring language descriptions. However, modelling the language explicitly with ActionFormer-TT yields a small boost, as it enables capturing similarities and subtle differences between step descriptions. More importantly, ActionFormer cannot be applied to detect steps of novel activities and articles in the unseen test set (S2). On this test set, our vanilla MT+BCE baseline that leverages the article structure (in contrast to grounding each step independently) performs reasonably well, outperforming UMT, even though the latter has been trained with a regression loss for more accurate temporal boundaries. This shows that leveraging the article step structure is very helpful for grounding novel articles in video. However, although ActionFormer-T has the disadvantage of performing single-step grounding, it outperforms all models the unseen split too, presumably because it combines an architecture tailored to multi-scale detection with language modelling.
Weakly supervised pretraining. The state-of-the-art weakly supervised VINA model trained on HowTol00M ASR captions and wikiHow articles yields a low mAP metric since it was not trained with accurate temporal boundaries. However, using the same pretrained VINA model as initialization finetune our MT+BCE baseline on HT-Step leads to a remarkable performance boost, outperforming all previous baselines by a large margin, achieving 29.8 and 19.3 mAP on SI and S2 respectively. We perform a similar experiment by pre- pre-training the ActionFormer-T (AF-T) model on the ASR captions of How T0M. then fine-tuning the model with the HT-Step annotations. The results are similar, with the model achieving very good performance in the high IOU evaluation settings, leading to the best overall results, i.e. 30.2 and 20.4 mAP on S1 and S2 respectively. The VINA-pretrained. model still shows better performance on the low-IOU evaluation, reflecting the strong localization abilities of the original model. These experiments clearly showcase the great
8

---

potential in combining strong step labels from HT-Stepp with noisy ASR captions from HowTo for training grounding models.
Table 3: Ablation of text used for step description.
Table 4: Partial vs full matches.
Method
Text backbone
Text input
't mAP @ [0.3-0.7]
Method
seen
unseen
Hill Prg.
Act.
seen (SI)
unseen (S2)
partial
full partial full
ActionFormer-T
MPNet
26.3
18.4
Detection
ActionFormer-T
MPNet
/
/
x
26.1
18.0
ActionFormer [46]
20.4
29.6
ActionFornerr1 T
MPNet
14.7
ActionFormeer T
MPNet
/
X
24.0
13.7
Grounding
ActionFor
MPNet
22.8
13.9
MT BCE [17]
16.5 24.7
8.8
8.7
ActionFormer-T
MPNet
J
21.1
11.3
MIT
WordZVec
10.1
MT + BCE
Word2Vec
/
V
X
19.2
10.3
+
MT + BCE
Word2Vee
X
17.0
8.3
MT + BCE
WordZVec
19.8
10.1
What text information is important for article grounding? We experiment with different combi- nations of step's headline, paragraph and activity name to form its description and summarize the results in Table 3 (details for how every model combines text sources are provided in Appendix). It evident that the headline contains the most valuable information. Paragraph information alone is enough for learning the task, however performance when using only paragraphs is lower than with headlines. Results from incorporating the activity name in the step descriptions vary depending on the model. Overall, although paragraph information is essential for accurate grounding, as shown in Figure 3, combining headline and paragraph provides marginal improvements. In addition, with all the approaches, there is a significant performance gap between the seen and the unseen test set. We conclude that temporal article grounding on our benchmark presents a challenging and exciting problem that will hopefully inspire the development of better models by the community.
What are the results on partial matches compared to full matches? To understand the differences between partial and full matches, we take models trained on the complete set and test them subsets containing only partial-match or only full-match labels (details in Appendix). We present the results in Table 4. When evaluating on activities (S1), the performance of all models is significantly better on the full-match subset. This aligns with our intuition that full-match annotations should be easier to ground, as their step descriptions align better to what is visually demonstrated in the video. However, this gap disappears when evaluating on unseen activities (S2). We conjecture this may be due to full step matches being visually more consistent, resulting in models overfitting them activities.
Table 5: Comparison to SOTA for zero-shot step grounding on the CrossTask dataset.
Table 6: Zero-shot article grounding evaluation on Cross Task.
Method
Avg. R@ 1 (%)
Training data
1Avg. R@I (%)
mAP@(0.3-007]
Zhukow [50]
22.4
HT100M HT100M COIN
37.1
5.2
HTT00M [31]
33.6
41.4
6.7
VidcoCLIP [45]
33.9
HT100M HT-Step
48.5
9.5
MCN [7]
35.1
DWSA [38]
35.3
MIL-NCE [30]
40.5
VT-TWINS [19]
40.7
UniVL [27]
42.0
AF-T (HT100M)
37.1
AF-T (HT100M + HT-Stte)
48.5
Do models trained on HT-Step generalize to other tasks? We showcase the potential of leveraging HT-Step for model mre-training by evaluating one of our models on zero-shot atomic step localization. In particular, we praiteei the ActionFormer-T (AF-T) model on the ASR captions of How To100M (following prior work [27,30 30, 45]), then fine-tuned the model with the HT-Step annotations. We perform evaluation on the CrossTask dataset, following standard protocol [50], i.e., we compute the recall per task based on a single predicted timestamp per step, computed over random subsets of the training videos. As Table 5 demonstrates, is clear that training on HT-Step results in

---

huge boost in performance, namely a ++ 1.4% absolute improvement, but also surpasses the previous state-of-the-art on this benchmark by a large margin (+6.5%).
To further motivate the usefulness of the HT-Stepp annotations compared to existing datasets, we train the same model the COIN dataset [41], and perform zero-shot evaluation on Cross using both the standard recall metric (for single timestamp prediction evaluation) as well as mAP (for temporal segment and precision evaluation). The results are shown in Table 6. It is clear that fine-tuning on the HT-step annotations results in much more substantial performance improvement compared to fine-tuning on COIN (+7.1% 1% @ and +2.8 mAP).
Conclusion
We have introduced HT-Step, a dataset containing temporal annotations of instructional article steps in cooking videos. The dataset offers a large-scate training resource for the task of temporal article grounding, presenting new challenges and encouraging the development of alignment methods that leverage order and procedural structure. By releasing the annotations and providing benchmark protocols, we aim to spur new research in this domain and advance the field of procedural activity understanding.
Limitations and societal impact We acknowledge that HT-Step is intended for research purposes and should not be regarded as a comprehensive dataset encompassing the full range of human activities. Models trained on our dataset may exhibit biases towards the specific activities included in the dataset, resulting in a limited coverage of our everyday living scenarios.
Acknowledgements. We thank Mandy Toh, Yale Song, Gene Byrne, Fu-Jen Chu, Austin Miller, and Jiabo Hu for helpful discussions and invaluable engineering support.
References
[1] Top Content Categories on YouTube.
http://wwww.tiiikiicchhohgggggo com/ www.////ooceeeeeeeeec homm/coouumer-inne inneinuhssriisshsss seseuuer-treedde caap-content-cceee centeentccategoriesyyrrttt
Accessed on June 1, 2023. 2
[2] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste- Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4575-4583, 2016. 3
[3] Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In Computer VisionEECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XIII. pages 657 675. Springer, 2022.
[4] Peijun Bao, Qian Zheng. and Yadong Mu. Dense events grounding video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 920-928, 2021.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[6] Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Locvtp: Video-text pre-training for temporal localization. In European Conference Computer Vision, 2022. 2,
[7] Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. Multimodal clustering networks for self-supervised learning from unlabcled videos. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8012-8021, 2021.
[8] Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, and Shih-Fu Chang. Weakly-supervised temporal article grounding. In Conference on Empirical Methods in Natural Language Processing, pages 9402-9413, Abu Dhabi, United Arab Emirates, Dec.
2022. Association for Computational Linguistics. 3. 15
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma. Evangelos Kazakos. Davide Moltisanti, Jonathan Munro. Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric Collection, pipeline and challenges for epic-kitchens-10
vision:
International Journal of Computer
Vision (IJCV). 130:33-55, 2022. 3
[10] Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In 2013 IEEE Conference Computer Vision and Pattern Recognition, pages 2634-2641, 2013. 3
[11] Nikita Dvomik, Isma Hadji, Konstantinos G. Derpanis, Animesh Garg, and Allan D. Jepson. Drop-dtw: Aligning common signal between sequences while dropping outliers. ArXiv, abs/2108. 1996, 2021.
10

---

[12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (LJCV). 88(2):003-338
2010. 8
[13] Daniel Fried, Jean-Baptiste Alayrac, Phil Blunsom, Chris Dyer, Stephen Clark, and Aida Nematzadeh. Learning to segment actions from observation and narration. In Annual Meeting of the Association for Computational Linguistics, pages 2569-2588, Online, July 2020. Association for Computational Linguistics. 3
[14] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-5275, 2017. 3.4
[15] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox Coot: Cooperative hierarchical transformer for video-text representation learning. Advances in neural information processing
systems, 33:22605- 22618, 2020.
[16] Tengda Han, Jue Wang. Anoop Cherian, and Stephen Gould. Human action forecasting by learning task
grammars arXiv: 1709, 2017. 3
Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2906-2916, June 2022. 4, 9, 15 [18] Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alexander Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumas challenge action recognition for videos "in the wild". Computer Vision and Image Understanding. 155:1 23,, 2017. 8, 15
[17]
[19] Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung- -oon On, Eun-Sol Kim, and Hyunwoo J Kim. Video-text representation learning via diffcrentiable weak temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5016-5025,
2022. 9
[20] Mahnaz Koupace and William Yang Wang. Wikihow: A large scale text summarization dataset. ArXiv, 2018, wikiHow dataset license available at: https: / github. cow www..cooo/mma.ee com/maannakkoupaeee WikiHow- MitHHow-Dataset. 2,3
[21] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV), 2017. 4,
[22] Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semanties of goal-dirccted human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 780-787, 2014. 3
[23] Lei, Tamara Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via
natural language queries. In Neural Information Processing Systems, 2021. 3, 4, 15
[24] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro
Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in
context. European Conference Computer Vision (ECCV), 2014. 8
[25] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning recognize procedural activities with distant supervision. arXiv preprint arXi::2201. 10990,
2022. 2, 3, 7,8, 8, 13
[26] Ye Liu, Siyuan Li. Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). pages 3042-3051 1, 2022. 3, 7. 8
[27] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint ar arXiv:2002.063:
2020. 9
[28] Etfrosyni Mavroudi. Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations. In International Conference on Computer Vision (ICCV), 2023. 3,
7, 8, 14, 15
[29] Effrosyni Mavroudi and René Vidal. Weakly-supervisee generation and grounding of visual descriptions with conditional generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15544-15554, 2022. 3
[30] Antoine Miech, Jean-Baptiste Alayrae, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-End Learning of Visual Representations from Uncurated Instructional Videos. In CVPR, 2020. 2,
7, 9, 14
[31] Antoine Miech. Dimitri Zhukov, Jean- Baptiste Alayrac. Ivan Laptev, and Josef Sivic. Howto 100m:
Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019. HTT00M dataset license available at: https: //gi thub. com/ Stt.oine77400/hottt 1ertto100m/maaam wwlob//mtteeeeeeeeeeeeeeem / /LICENSE 2,9
[32] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations
in vector space. In International Conference on Learning Representations, 2013. 7
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning
transferahle visual models from natural language supervision, 2021. 7
[34] Michacla Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25 36, 2013. 3
11

---

[35] Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. A database for fine grained activity detection of cooking activities. In 2012 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1194-1201, 2012.
[36] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly100 101: large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21096-
21106. 2022. 3
[37] Ozan Sener, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Unsupervised semantic parsing of vidco collections. In Proceedings of the IEEE International conference Computer Vision, pages 1480-4488,
2015. 3
[38] Yuhan Shen, Lu Wang. and Ehsan Elhamifar. Learning to segment actions from visual and language instructions via differentiable weak sequence alignment. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 10156-1011 10661665,220 2021. 9
[39] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan I Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.092:
2020.
[40] Sebastian Stein and Stephen McKenna. User-adaptive models for recognizing food preparation activitics.
In Proceedings of the 5th international workshop on Multimedia for cooking & eating activities, pages
39 44, 2013. 3
[41] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng. Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In IEEE Conference on
Computer Vision and Puttern Recognition (CVPR). 2019. 2, 3, 6. 10, 15
[42] Yansong Tang, Jiwen Lu,, and Jie Zhou. Comprehensive instructional video analysis: The coin datasct and performance evaluation. IEEE transactions on pattern analysis and machine intelligence, 43(9))3138-333
2020. 2
[43] Huiyu Wang. Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without
exocentric pretraining, 2023, 8, 14
[44] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-off's in video classification. In European Conference on Computer
Vision. 2018.
[45] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online, Nov. 2021. Association for Computational Linguistics. 9 [46] Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers.
In European Conference on Computer Vision, volume 13664 of LNCS, pages 192-510, 2022. 3, 7, 8, 9, 14 [47] Honglu Zhou, Roberto Martin- Martin,, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, 2023. 2,8
[48] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18//AATS/// 18. AAAI Press, 2018. 3
[49] Wanrong Zhu, Bo Pang, Ashish V. Thapliyal, William Yang Wang. and Radu Soricut. End-to-end dense video captioning as sequence generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5651-5666 Gyeongju, Republic of Korca, Oct. 2022. International Committee on Computational Linguistics. 3
[50] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross task weakly supervised learning from instructional videos. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2019. 2, 3, 9, 15
12

---

Appendix
This Appendix provides more comprehensive statistics about the dataset (Section A), architecture and implementation details for the baselines discussed (Section B), breakdown of model performance by activity (Section C), and extra details about the evaluation protocol (Section D).
A
Extra statistics
Figures 6a-6i contain additional statistics about HT-Stepp.
40
Mean: 0.40
1750
Mean: 0.41
2500
Moan: 0.23
#AAtiiittes
30-
35
1500
2000
25-
Stops
1500
15
500
250
0
0.1
0.2
0.3
0.4 0.5 0.6 Step coverage
0.7
0.8
0.0
0.4
0.5
Step coverage
Vidoo coverage
(a) Distribution of step coverage across activitics.
(b) Distribution of step coverage
(c) Distribution of annotated
across videos.
video durations.
Cumulative frequency (%)
3000
Monn:
50
Moar: 281.5
L.0
Activittes
40
0.8
Vidoos
2000
30
0.6
20
# 1600
0.
0
1.5
30
200
400
600
800
1000
0.0
200
100
600
800
1000
Video duration (minules)
Annotations
Annotations
(d) Distribution of step coverage
(c) Distribution of step coverage
(f) Distribution of annotated video durations.
across activities.
across videos.
550
# Activiitis
50
40
Annotations
Taxonomy
300
250
40
035
30
200
# Activities
30-
150
5o 30
100
50
15
35
# Unique steps
Variations
0.0
0.2
0.4
0.5
0.8
1.0
Fraction of full-uuate anmotations
(g) Distribution of unique
(h) Distribution of the number of variations appearing HT-step
notated per video vs steps in tax- onomy
(i) Distribution annotated video durations.
across activities.
Figure 6: Additional statistics of HT-Step.
B Baselines implementation details
We train all the baselines on top of the same feature sequences, extracted from frozen backbones.
TimeSformer (TS) features. The TimeSformer features extracted using the public model? of [25] pre-trained with distant supervision on How 10000.00.. We obtain feature per second, by resam- pling the video at fps and extracting features with stride of 8 frames. The feature dimensionality 768.
htttps://d1.foo www.pbbiccilles......opbbb htapppplleei..e...ooioddddddoded www.://aaat--ppprrssssootttaaaae www.:oorrmmm.d.v..eeee___. _x32
224 Lotooooo10._preerrriiee pth
13

---

S3D features. The S3D features are extracted using the published model* of [30] pre-trained with MIL-NCE [30]. We obtain 1 feature vector per second, by resampling the video at 16 fps and extracting features with stride of 16 frames. The feature dimensionality is 1024.
ActionFormer. We use the official ActionFormer implementation" provided by the authors[46] We set the number of classes to 4958 i... one detection output for each step in the taxonomy. We set the max sequence length to 512 and train for 20 epochs with a batch size of 16, using the AdamW optimiser with cosine learning rate schedule, base learning rate of 1e - 4 and 5 warm-up epochs on single Nvidia Tesla V100 GPU with 32GB of memory.
ActionFormer-T. ActionFormer-T is trained with the exact same hyper-parameterr, loss and labels as ActionFormer. We extract the text representations, using the MPNet implementation provided by the sentence_ backpropagated into the pre-trained langauge model. We use 2-layer MLP with hidden side 512, ReLU activation and Layer Normalization, to project and transform the 768 dig-dimeesional text embeddings into the di2-dimensoonal video embedding space. Step text descriptions from different sources (e.g. headline, paragraph, activity, see also Table 3 of the main paper) are combined by simple concatenation at the text level, i.e. the combined sentences including all three have the form "Activity: Headline. Paragraph".
library". The text embeddings are frozen, so no gradients are
UMT. For UMT, we use the authors' official code We train models with learning rate 1e and batch size 64 and train for 200 epochs. We use only the unimodal encoder for video (we do not use the audio encoder, or the cross-attention attentieeno modules). All remaining hyperparameters follow the configuration for the QVHighlights task provided by the authors.
The input to our temporal article grounding baseline is temporal sequence of visual features extracted with a sliding window (using either the TimeSformer or S3D backbones as explained above), and sequence of step sentences (consisting of the activity name and the article step headlines). We base our model on the VINA [28] architecture by removing the additional narrations modality, i.e., we do not use the narration unimodal encoders, positional encodings and the alignments of steps to narrations or narrations to video. We use the TAN$ codebase for our implementation. All of the architecture hyperparameters (e.g., number of Multimodal Transformer layers, embedding dimensions etc.) are adopted from VINA. The only difference is the maximum length of the input video which we increase to 1200 seconds to account for the longer videos in the HT-Step training set.
MT+BCE.
To obtain temporal segment predictions for each article step from the Multimodal Transformer outputs, we: (I) compute the normalized dot product between each step contextual embedding and each video clip contextual embedding. This results in T S alignment matrix, where T is the number of timesteps and S is the number of steps. (2) We pass these similarities through sigmoid activation (with temperature 0.07) to obtain a confidence score about whether each timestep is aligned with step s, (3) we post- pest-process the temporal sequence of confidence scores for each step with an ID blob detection routine obtain temporal segments at multiple scales. In particular, we apply Laplacian of Gaussian filters at 13 scales, covering Gaussian standard deviations from 1 to 480 [43].
The model is trained with binary cross-entropy loss applied at each temporal timestep and for each article step. We train our model for 9 epochs using the same optimizer, learning rate and batch size as VINA [28].
Adding paragraph information: For our ablations in Table 3 of the main paper, we added paragraph information to the MT+BCE model simply by interleaving step headlines with step paragraph sentences. In other words, we tokenize the article into sentences (with a maximum of 28 sentences per step) and we encode and feed that sequence of sentences to the Multimodal Transformer. We use the same positional encoding for all sentences associated with the same article step. In order to
htttpp///.ttaa.occo//aaaaar////p 7 http:///wggiigteeeeaaaeaeeeeeata
14

---

obtain a single contextual embedding for each step, we max pool the embeddings of the headline and paragraph sentences of that step.
Model weights initialization: We train all variants of MT+ from scratch, except for the model in the last row of Table 2, which was trained after initializing the unimodal encoders, positional encod- ings and Multimodal Transformer weights using a VINA model pretrained on the HTM370K [17] subset of HowTo 100M using pseudo-labels for wikiHow steps (and no ASR narrations) [28]. For this experiment, we adopt the same maximum video length as VINA (1024 seconds).
C Per-activity predictions
In Table 7 we show the per-activity AP breakdown of the performance of the two best models. We show the 25 highest and 25 lowest scoring activities, ranked by the performance of the ActionFormer detection model. Note that activities that are challenging for the fixed taxonomy, detection model (such as Cook Pork Tenderloin for which the AP is 0.62%) are handled better by the temporal grounding model (achieving 28.9% AP for Cook Pork Tenderloin). For this particular example of Cook Pork Tenderloin, this can be explained since this activity has only 4 examples in the training set. Therefore, the detection model does not have enough training samples to learn good representation for the steps of this activity. On the other hand, the temporal article grounding model, that has been initialized with a model trained with weak- suee-suuprvision on much larger dataset (HTM370K) can perform better in this few-shot scenario. Another interesting observation is that for some activities the eetection-based model outperforms language-basee grounding.
D Evaluation protocol details
D.1 Article-grounding AP metric
Approaches in our proposed temporal article grounding benchmark are evaluated using Article Grounding mean Average Precision (AGrd. mAP) over temporal IoU thresholds from 0.3 to 0.7 with a step size set to 1 (as in existing benchmarks [18]), and using three fixed tloU thresholds at 0.3, 0.5 and 0.7. As explained in the main paper, our proposed metric computes an AP per activity (which might be associated with multiple articles if is has variations) by treating all article steps associated with that activity as class-agnostic text queries (similar to the temporal grounding Average Precision introduced [23]. The per-activity AP only computed on videos demonstrating each particular activity. The final article-grounding
mAp is computed by averaging the per-activity APs. Our maP-based metric is more suitable for the temporal article grounding task than existing recall-based metrics for grounding [8, 50] which ignore non- -on-grounddble steps, or frame- wise metrics for step detection [41], which ignore the temporal extent of each segment.
D.2 Breakdown of article-grounding mAP per match type (full vs partial)
In Table 4 of our main paper. we report article-grounding mAP computed per step match type (full vs partial). The mAP for full matches was computed separately for step queries that only have fully-matching temporal segments (or no matching segments) in their corresponding video. Step queries that have both full and partial matches in the same video were ignored from the computation of the mAP on full matches. Furthermore, APs are only computed for activities that have ground-truth step queries with full matches and averaged over those. Overall, the mAP for full matches was computed based on 78 activities, with 1176 ground-truth instances. The mAP for partial matches was computed in a corresponding manner, covering 79 activities and 2375 ground-truth instances.
E
Training, validation, and test splits
We have included details about the training, validation and test splits in Section 3.2 of the main paper. Here add some comments.
Seen val/test set (S1). We note that these sets are balanced, each containing 600 videos in total, videos per each of 120 activities, with an overlap between validation and test amounting to 63 activities. Labels are released for the val set, while labels for the seen test set are withheld and
15

---

Table 7: Breakdown of AP performance per activity on the seen test set (S1), We show the 25 highest and 25 lowest scoring actívities, ranked by the performance of the ActionFormer model.
Model
ActionFormer
MT+BCE(VINA)
Activity
Make Lunch Box Oatmeal Cookies
55.02
66.65
Make Chicken Liver Pate
51.94
45.56
Deep Fry Turkey
51.56
43.02
Make Tomato Pie
47.97
38.43
Make Buttermilkk Fried Chicken
45.53
39.40
Bake Sweet Potato Pie
43.62
25.70
Make Scotch Eggs
42.90
41.82
Make Pecan Crusted Blackened Catfish
42.37
26.54
Make Vegetable Paniyaram
42,30
39,91
Cook Arepas
42.16
43.75
Prepare Mexican Chilaquiles
41.15
40.09
Make Chiles Rellenos
40.82
35.48
Make Beef Stroganoff
40.59
27.03
Clarify Butter
40,38
39.71
Make Toad in the Hole
38.56
43.81
Make Focaccia
38.54
39.11
Clean Flounder
37,95
15.60
Make Chicken Piccata
37.77
44.94
Brine, Truss, and Roast Turkey
36,15
49.03
Grill Bacon
35,89
55.26
Make Eggplant Pasta Sauce
35.31
30.93
Make Mofongo
35.05
43.49
Make Saltimbocca
34.84
34.21
Cook Brussels Sprouts with Chestnuts
34,52
42.86
Make Beignets
34.34
40.71
Make White Chili
15.79
Make Fairy Cakes with Self Raising Flour
15.51
36.88
Make Chicken Cacciatore
15.42
28.50
Make Grilled Artichokes
15.19
27.35
Make Healthier Fish Sticks
14.98
23.45
Bake Queen Elizabeth Cake
14,91
27.57
Make Coconut Rice
14,82
31.91
Make Hostess Twinkies
14.54
24.67
Cook Cube Steak
13.20
34.79
Make Bannock
12.75
17.94
Make Mango Chutney
12.74
11.38
Make Overnight Caramel Pecan Rolls
12.47
18.48
Make Vegan Ceviche
11.96
4.85
Cook Black Eyed Peas
11.44
19.02
Make a Cheese Crisp
9.32
14.16
Cook Bacon in the Microwave
9.14
40,50
Make Italian Ice
9.01
19.73
Make Quick and Easy Sausage Rolls
8.27
16.67
Braai Steak
8.08
9.27
Make a Hearty Stew
7.73
31.03
Make Mediterranean Vegetable Cheese Pie
6.19
16.37
Make Hungarian Goulash
6,00
18.43
Make Bacon Toffee
3.92
8.15
Make Blucberry Strudel
1,96
9.29
Cook Pork Tenderloin
0.62
28.94
MAP
25.4
29.8
16

---

fair evaluation protocol on this set is supported via a test server that will be made available to the community.
Unseen val/test set (S2). Note that the headlines paragraphs of some steps in the unseen val/test sets may be very similar to steps of the activities included in the training set, due to the compositionality of recipes. For example, the unseen activity of Make Poutine contains the step "Add the garlic and shallot" which similar to steps such as "Add the garlic and cook for 30 seconds" from the seen activity Make Hearty Stew and "add the garlic slices and cook for minute." from Make Tumber. Evaluation on the unseen test set will be made possible through the test server.
17


Reviews:
[{"review_id": "ZC2FPjeMQO", "reviewer": "Reviewer_6wqD", "summary": "", "questions": "", "limitations": "The authors addressed the limitations in the paper. They claimed that  models trained on the dataset may exhibit biases towards the specific activities included in the dataset. Therer is no potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed dataset is large-scale. The step annotations  are much larger than pervious datasets.\n\n2. The paper proposes a challenging benchmark for the task of temporal article grounding, and evaluate SOTA methods on it.\n\n3. The details of dataset collection, annotation and statistics are well demostrated.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EK8ppYj0Gm", "reviewer": "Reviewer_XtLe", "summary": "", "questions": "", "limitations": "I believe the contribution of this work is sufficient. I do not have comments for the limitations.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Large-scale dataset: The paper introduces a large-scale dataset of instructional videos and instructional articles, which significantly surpasses existing labeled step datasets in terms of scale, number of tasks, and richness of natural language step descriptions. This dataset provides a valuable resource for training and evaluating models for aligning instructional articles with how-to videos.\n\n2. Benchmark for temporal article grounding: The paper proposes a benchmark for the task of temporal article grounding, where the goal is to temporally localize the steps demonstrated in a video and reject the ones not shown. By providing this benchmark, the authors enable the development and evaluation of language-based temporal alignment models that can effectively ground textual descriptions of steps in instructional articles, even for unseen tasks.\n\n3. Taxonomy of steps: The authors leverage wikiHow articles to guide the annotation process and create a taxonomy of steps fitting the data. This taxonomy provides a structured framework for organizing and categorizing the steps described in instructional articles, enhancing the understanding and alignment of instructional content with corresponding video segments.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JLcC976Vpw", "reviewer": "Reviewer_rjWS", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper leverages the rich how-to instructions from wikihow, i.e. Activity, headline, and paragraph to annotate the instructional videos. HowTo100m has been widely studied by the community, but the poor ASR quality has limited the research. Recent works such as TAN, DistantSup attempted to improve the dataset by either aligning video with ASR or retrieving the most similar step description regarding ASR. HT-Step takes a step further by providing more detailed step descriptions and temporal segments. This dataset outperforms most procedure activity datasets in terms of videos, number of segments, and activities. \n2. On various video backbones (S3D, TimeSformer), text backbones (Word2Vec, MPNet) and task formulations (activity detection, single-sentence grounding, article grounding), the proposed benchmark dataset as well as challenging subsets (i.e. Unseen activities) are fairly evaluated. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "z5Eye24WsE", "reviewer": "Reviewer_ssNp", "summary": "", "questions": "", "limitations": "No obvious limitations.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The scale of the proposed dataset is relatively large.\n\n2. There are no such segment-instruction dataset for now. Thus, it is beneficial for the video & language research community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper receives mostly positive reviews except one. The positive reviews acknowledges the size and clever use of wikiHow data. The only one negative reviewer raised questions about novelty of the task, not only the authors provide comprehensive answers to them and the reviewer did not actively respond to the authors' response but also the concern is not quite valid -- the task that the dataset is addressing is not necessarily novel. Given the value of the proposed dataset, the AC recommends to accept the submission to NeurIPS 23 DB track.
Decision: Accept (Poster)