Title: Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos
Venue: NeurIPS 2024
Accepted: yes
Keywords: Task Graph

Procedural Sequences

Online Mistake Detection

Video Understanding
Abstract: Procedural activities are sequences of key-steps aimed at achieving specific goals. They are crucial to build intelligent agents able to assist users effectively. In this context, task graphs have emerged as a human-understandable representation of procedural activities, encoding a partial ordering over the key-steps. While previous works generally relied on hand-crafted procedures to extract task graphs from videos, in this paper, we propose an approach based on direct maximum likelihood optimization of edges' weights, which allows gradient-based learning of task graphs and can be naturally plugged into neural network architectures. Experiments on the CaptainCook4D dataset demonstrate the ability of our approach to predict accurate task graphs from the observation of action sequences, with an improvement of +16.7% over previous approaches. Owing to the differentiability of the proposed framework, we also introduce a feature-based approach, aiming to predict task graphs from key-step textual or video embeddings, for which we observe emerging video understanding abilities. Task graphs learned with our approach are also shown to significantly enhance online mistake detection in procedural egocentric videos, achieving notable gains of +19.8% and +7.5% on the Assembly101-O and EPIC-Tent-O datasets. Code for replicating the experiments is available at https://github.com/fpv-iplab/Differentiable-Task-Graph-Learning.

Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos

Abstract

Procedural activities are sequences of key-steps aimed at achieving specific goals. They are crucial to build intelligent agents able to assist users effectively. In this context, task graphs have emerged as a human-understandable representation of procedural activities, encoding a partial ordering over the key-steps. While previous works generally relied on hand-crafted procedures to extract task graphs from videos, in this paper, we propose an approach based on direct maximum likelihood optimization of edges' weights, which allows gradient-based learning of task graphs and can be naturally plugged into neural network architectures. Experiments on the CaptainCook4D dataset demonstrate the ability of our approach to predict accurate task graphs from the observation of action sequences, with an improvement of +16.7% over previous approaches. Owing to the differentiability of the proposed framework, we also introduce a feature-based approach, aiming to predict task graphs from key-step textual or video embeddings, for which we observe emerging video understanding abilities. Task graphs learned with our approach are also shown to significantly enhance online mistake detection in procedural egocentric videos, achieving notable gains of +19.8% and +7.5% on the Assembly101-O and EPIC-Tent-O datasets. Code for replicating the experiments is available at https: //github.com/fpv-iplab/Differentiable-Task-Graph-Learning .

1 Introduction

Procedural activities are fundamental for humans to organize tasks, improve efficiency, and ensuring consistency in the desired outcomes, but require time and effort to be learned and achieved effectively. This makes the design of artificial intelligent agents able to assist users to correctly perform a task appealing [21, 31]. Achieving these abilities requires building a flexible representation of a procedure, encapsulating knowledge on the partial ordering of key-steps arising from the specific context at hand. For example, a virtual assistant needs to understand that it is necessary to break eggs before mixing them or that the bike's brakes need to be released before removing the wheel. Importantly, for a system to be scalable, this representation should be automatically learned from observations (e.g., humans making a recipe many times) rather than explicitly programmed by an expert.

Previous approaches focused on directly tackling tasks requiring procedural knowledge such as action anticipation [16, 14, 34] and mistake detection [37, 13, 7, 41, 15] without developing explicit representations of the procedure. Other works proposed neural models able to develop implicit representations of the procedure by learning how to recover missing actions [43, 28], discover keysteps [11, 4, 5], or grounding them to video [10, 25]. A different approach [3, 10, 18] consists in representing the structure of a procedure in the form of a task graph , i.e., a Directed Acyclic Graph (DAG) in which nodes represent key-steps, and directed edges impose a partial ordering over key-steps,

Graph

positive gradients

negative gradients

Get a

Bowl

Crack

Egg

Add

Water

Add

Milk

Mix

Pour

Mixture

(a) Example Task Graph

Current key-step

Task Graph Maximum Likelihood Loss

(b) Task Graph Learning as Maximum Likelihood Estimation

encoding dependencies between them (see Figure 1(a)). 1 Graphs provide an explicit representation which is readily interpretable by humans and easy to incorporate in downstream tasks such as detecting mistakes or validating the execution of a procedure. While graphs have been historically used to represent constraints in complex tasks and design optimal sub-tasks scheduling [38], graph-based representations mined from videos [3], key-step sequences [39, 20] or external knowledge bases [44] have only recently emerged as a powerful representation of procedural activities able to support downstream tasks such as key-step recognition or forecasting [3, 44]. Despite these efforts, current methods rely on meticulously crafted graph mining procedures rather than setting graph generation in a learning framework, limiting the inclusion of task graph representations in end-to-end systems.

In this work, we propose a novel approach to learn task graphs from demonstrations in the form of sequences of key-steps performed by real users in a video while executing a procedure. Given a directed graph represented as an adjacency matrix and a set of key-step sequences, we provide an estimate of the likelihood of observing the set of sequences given the constraints encoded in the graph. We hence formulate task graph learning under the well-understood framework of Maximum Likelihood (ML) estimation, and propose a novel differentiable Task Graph Maximum Likelihood (TGML) loss function which can be naturally plugged into any neural-based architecture for direct optimization of task graph from data. Intuitively, our TGML loss generates positive gradients to strengthen the weights of directed edges B → A when observing the < . . . , A, . . . , B, . . . > structure, while pushing down the weights of all other edges in a contrastive manner (see Figure 1(b)). To evaluate the effectiveness of the proposed framework, we propose two approaches to task graph learning. The first approach, called 'Direct Optimization (DO)', uses the proposed TGML loss to directly optimize the weights of the adjacency matrix, which constitute the only parameters of the model. The output of the optimization procedure is the learned graph. The second approach, termed Task Graph Transformer (TGT) is a feature-based model which uses a transformer encoder and a relation head to predict the adjacency matrix from either text or video key-step embeddings.

We validate the ability of our framework to learn meaningful task graphs on the CaptainCook4D dataset [30]. Comparisons with state-of-the-art approaches show superior performance of both proposed approaches on task graph generation, with boosts of up to +16 . 7% over prior methods. On the same dataset, we show that our feature-based approach implicitly gains video understanding abilities on two fundamental tasks [46]: pairwise ordering and future prediction. We finally assess the usefulness of the learned graph-based representation on the downstream task of online mistake detection in procedural egocentric videos. To tackle this task, we observe that procedural errors mainly arise from the execution of a given key-step without the correct execution of its pre-conditions. We hence design an approach which uses the learned graph to check whether pre-conditions for the current action are satisfied, signaling a mistake when they are not, obtaining significant gains of +19.8% and +7.5% in the online mistake detection benchmark recently introduced in [13] on Assembly101 [37] and EPIC-Tent [19], showcasing the relevance and quality of the learned graph-based representations.

The contributions of this work are the following: 1) We introduce a novel framework for learning task graphs from action sequences, which relies on maximum likelihood estimation to provide a differentiable loss function which can be included in end-to-end models and optimized with gradient descent; 2) We propose two approaches to task graph learning based on direct optimization of the adjacency matrix and processing key-step text or video embeddings, which offer significant improvements over previous methods in task graph generation and shows emerging video understanding abilities; 3) We showcase the usefulness of task graphs in general, and the learned graph-based representations in particular, on the downstream task of online mistake detection from video, where we improve over competitors. The code to replicate the experiments is available at https://github.com/fpv-iplab/Differentiable-Task-Graph-Learning .

2 Related Work

Procedure Understanding Previous investigations considered different tasks related to procedure understanding, such as inferring key-steps from video in an unsupervised way [45, 47, 12, 4, 5, 11], grounding key-steps in procedural video [25, 9, 10, 27], recognizing the performed procedure [24], inferring key-step orderings [4, 5, 25, 10, 43], and procedure structure verification [28]. Recently, task graphs, mined from video or external knowledge such as WikiHow articles, have been investigated as a powerful representation of procedures and proved advantageous for learning representations useful for downstream tasks such as key-step recognition and forecasting [44, 3].

Differently from previous works [28, 43], we aim to develop an explicit and human readable representation of the procedure which can be directly plugged in to enable downstream tasks [3], rather than an implicit representation obtained with pre-training objective [44, 28]. As a departure from previous paradigms which carefully designed task graph construction procedures [3, 44, 39, 20], we frame task prediction in a general learning framework, enabling models to learn task graphs directly from input sequences, and propose a differentiable loss function based on maximum likelihood.

Task Graph Construction A line of works investigated the construction of task graphs from natural language descriptions of procedures (e.g., recipes) using rule-based graph parsing [36, 10], defining probabilistic models [23], fine-tuning language models [35], or proposing learning-based approaches [10] involving parsers and taggers trained on text corpora of recipes [8, 42]. While these approaches do not require any action sequence as input, they depend on the availability of text corpora including procedural knowledge, such as recipes, which often fail to encapsulate the variety of ways in which the procedure may be executed [3]. Other works proposed hand-crafted approaches to infer task graphs observing sequences of actions depicting task executions [20, 39]. Recent work designed procedures to mine task graphs from videos and textual descriptions of key-steps [3] or cross-referencing visual and textual representations from corpora of procedural text and videos [44].

Differently from previous efforts, we rely on action sequences, grounded in video, rather than natural language descriptions of procedures or recipes [35, 10] and frame task graph generation as a learning problem, providing a differentiable objective rather than resorting to hand-designed algorithms and task extraction procedures [20, 39, 3, 44].

Online Mistake Detection in Procedural Videos Despite the interest in procedural learning, mistake detection has been systematically investigated only recently. Some methods considered fully supervised scenarios in which mistakes are explicitly labeled in video and mistake detection is performed offline [37, 41, 30]. Other approaches considered weak supervision, with mistakes being labeled only at the video level [15]. Finer-grade spatial and temporal annotations are exploited in [7] to build knowledge graphs, which are then leveraged to perform mistake detection. Recently, the authors of [13] proposed an online mistake detection benchmark incorporating videos from the Assembly101 [37] and EPIC-Tent [19] datasets, as well as PREGO, an approach to online mistake detection in procedural egocentric videos.

Rather than addressing online mistake detection with implicit representations [13] or carefully designed knowledge bases [37], we design a simple approach which relies on learned explicit task graph representations. As we show in the experiments, this leads to obtain significant performance gains over previous methods, even when the predicted graphs are suboptimal, while best results are obtained with task graphs learned within the proposed framework.

3 Technical Approach

3.1 Task Graph Maximum Likelihood Learning Framework

Preliminaries Let K = { K 0 = S, K 1 , . . . , K n , K n +1 = E } be the set of key-steps involved in the procedure, where S and E are placeholder 'start' and 'end' key-steps denoting the start and end of the procedure. We define the task graph as a directed acyclic graph, i.e., a tuple G = ( K , A , ω ) , where K is the set of nodes (the key-steps), A = K×K is the set of possible directed edges indicating ordering constraints between pairs of key-steps, and ω : A → [0 , 1] is a function assigning a score to each of the edges in A . An edge ( K i , K j ) ∈ A (also denoted as K i → K j ) indicates that K j is a pre-condition of K i (for instance mix → crack egg) with score ω ( K i , K j ) . We assume normalized weights for outgoing edges, i.e., ∑ j w ( K i , K j ) = 1 ∀ i . We also represent the graph G as the adjacency matrix Z ∈ [0 , 1] ( n +2) × ( n +2) , where Z ij = ω ( K i , K j ) . For ease of notation, we will denote the graph G = ( K , A , ω ) simply with its adjacency matrix Z in the rest of the paper. We assume that a set of N sequences Y = { y ( k ) } N k =1 showing possible orderings of the key-steps K is available, where the generic sequence y ∈ Y is defined as a set of indexes to key-steps K , i.e., y = < y 0 , . . . , y t , . . . , y m +1 > , with y t ∈ { 0 , . . . , n +1 } . We further assume that each sequence starts with key-step S and ends with key-step E , i.e., y 0 = 0 and y m +1 = n + 1 2 and note that different sequences y ( i ) and y ( j ) have in general different lengths. Since we are interested in modeling key-step orderings, we assume that sequences do not contain repetitions. 3 We frame task graph learning as determining an adjacency matrix ˆ Z such that sequences in Y can be seen as topological sorts of ˆ Z . A principled way to approach this problem is to provide an estimate of the likelihood P ( Y| Z ) and choose the maximum likelihood estimate ˆ Z = arg max Z P ( Y| Z ) .

Modeling Sequence Likelihood for an Unweighted Graph Let us consider the special case of an unweighted graph, i.e., ¯ Z ∈ { 0 , 1 } ( n +2) × ( n +2) . We wish to estimate P ( y | Z ) , the likelihood of the generic sequence y ∈ Y given graph Z . Formally, let Y t be the random variable related to the event 'key-step K y t appears at position t in sequence y '. We can factorize the conditional probability P ( y | Z ) as:

We assume that the probability of observing a given key-step K y t at position t in y depends on the previously observed key-steps ( K y t -1 , . . . , K y 0 ), but not on their ordering, i.e., the probability of observing a given key-step depends on whether its pre-conditions are satisfied, regardless of the order in which they have been satisfied. Under this assumption, we write P ( Y t | Y t -1 , . . . , Y 0 , Z ) simply as P ( K y t | K y t -1 , . . . , K y 0 , Z ) . Without loss of generality, in the following, we denote the current key-step as K i = K y t , the indexes of key-steps observed at time t as J = O ( y, t ) = { y t -1 , . . . , y 0 } , and the corresponding set of observed key-steps as K J = { K i | i ∈ J} . Similarly, we define ¯ J = O ( y, t ) = { 0 , . . . , n + 1 } \ O ( y, t ) and K ¯ J as the sets of indexes and corresponding keysteps unobserved at position t , i.e., those which do not appear before y t in the sequence. Given the factorization above, we are hence interested in estimating the general term P ( K y t | K y t -1 , . . . , K y 0 ) = P ( K i | K J ) . We can estimate the probability of observing key-step K i given the set of observed keysteps K J and the constraints imposed by ¯ Z , following Laplace's classic definition of probability [26] as 'the ratio of the number of favorable cases to the number of possible cases'. Specifically, if we were to randomly sample a key-step from K following the constraints of ¯ Z , and having observed key-steps K J , sampling K i would be a favorable case if all pre-conditions of K i were satisfied, i.e., if ∑ j ∈ ¯ J Z ij = 0 (there are no pre-conditions in unobserved key-steps K ¯ J ). Similarly, sampling a key-steps K h is a 'possible case' if ∑ j ∈ ¯ J Z hj = 0 . We can hence define the probability of observing key-step K i after observing all key-steps K J in a sequence as follows:

where 1 ( · ) denotes the indicator function, and in the denominator, we are counting the number of key-steps that have not appeared yet are 'possible cases' under the given graph Z . Likelihood P ( y | Z ) can be obtained by plugging Eq. (2) into Eq. (1).

A

D

B

C

S

E

Graph

Observed

Key-steps

0.1

0.55

0.95

Feasibility

=

Goal: Estimate

Observed sequence

A

B

C

D

S

E

A

B

C

D

S

E

Observed

Future

Example: estimate

0.7

0

0.1

0.1

0.05 0.45

0.4

0

0.7

0.2

0

0.05

A

A

B

B

C

C

D

D

0.1

0.1

0.05

0.05

0.1

0.35

0.4

0

0

0.05

0.05 0.05 0.85

S

E

0

0

0

0

0

0

0

0

0

0

0

E

S

Modeling Sequence Likelihood for a Weighted Graph To enable gradient-based learning, we consider the general case of a continuous adjacency matrix Z ∈ [0 , 1] ( n +2) × ( n +2) . We generalize the concept of 'possible cases' discussed in the previous section with the concept of 'feasibility of sampling a given key-step K i , having observed a set of key-steps K J , given graph Z ', which we define as the sum of all weights of edges between observed key-steps K J and K i : f ( K i | K J , Z ) = ∑ j ∈J Z ij . Intuitively, if key-step k i has many satisfied pre-conditions, we are more likely to sample it as the next key-step. We hence define P ( K i | K J , Z ) as 'the ratio of the feasibility of sampling K i to the sum of the feasibilities of sampling any unobserved key-step':

Figure 2 illustrates the computation of the likelihood in Eq. (3). Plugging Eq. (3) into Eq. (1), we can estimate the likelihood of a sequence y given graph Z as:

Where we set P ( K y 0 | Z ) = P ( S | Z ) = 1 as sequences always start with the start node S .

Task Graph Maximum Likelihood Loss Function Assuming that sequences y ( i ) ∈ Y are independent and identically distributed, we define the likelihood of Y given graph Z as follows:

We can find the optimal graph Z by maximizing the likelihood in Eq. (5), which is equivalent to minimizing the negative log-likelihood -log P ( Y , Z ) , leading to formulating the following loss:

where β is a hyper-parameter. We refer to Eq. (6) as the Task Graph Maximum Likelihood (TGML) loss function. Since Eq. (6) is differentiable with respect to all Z ij values, we can learn the adjacency matrix Z by minimizing the loss with gradient descent to find the estimated graph ˆ Z = arg Z max L ( Y , Z ) . Eq. (6) works as a contrastive loss in which the first logarithmic term aims

Concat. all

combinations

MLP

MLP

Trasformer Layer

Trasformer Layer

Relation transformer

with dim. reduction

Distinctiveness

Cross-Entropy Loss

pairwise cosine

similarities

...

Sequences

Relation Head

Transformer

Encoder

s

e

s

e

s

s

e

e

e

e

e

e

s

s

s

...

...

...

...

...

...

...

TGML

Loss

s

s

...

e

s

...

e

Take bowl

Take Eggs

Break Eggs

Mix Eggs

Video Embeddings

Text Embeddings

RS

✏

✏

✏

✏

EgoVLPv2

RS

Learned Start Embedding

Learned End Embedding

RS

RS

OR

EgoVLPv2

to maximize , at every step t of each input sequence, the weights Z y t j of edges K y t → K j going from the current key-step K y t to all previously observed key-steps K j , while the second logarithmic term (contrastive term) aims to minimize the weights of edges K h → K j between key-steps yet to appear K h and already observed key-steps K j . The hyper-parameter β regulates the influence of the summation in the contrastive term which, including many more addends, can dominate gradient updates. As in other contrastive learning frameworks [29, 33], our approach only includes positives and negatives and it does not explicitly consider anchor examples.

3.2 Models

Direct Optimization (DO) The first model aims to directly optimize the parameters of the adjacency matrix by performing gradient descent on the TGML loss (Eq. (6)). We define the parameters of this model as an edge scoring matrix A ∈ R ( n +2) × ( n +2) , where n is the number of key-steps, plus the placeholder start ( S ) and end ( E ) nodes, and A ij is a score assigned to edge K i → K j . To prevent the model from learning edge weights eluding the assumptions of directed acyclic graphs, we mask black cells in Figure 2 with -∞ . To constrain the elements of Z in the [0 , 1] range and obtain normalized weights, we softmax-normalize the rows of the scoring matrix to obtain the adjacency matrix Z = softmax ( A ) . Note that elements masked with -∞ will be automatically mapped to 0 by the softmax function similarly to [40]. We train this model by performing batch gradient descent directly on the score matrix A with the proposed TGML loss. We train a separate model per procedure, as each procedure is associated to a different task graph. As many applications require an unweighted graph, we binarize the adjacency matrix with the threshold 1 n , where n is the number of nodes. We also employ a post-processing stage in which we remove redundant edges, loops, and add obvious missing connections to S and E nodes. 4

Task Graph Transformer (TGT) Figure 3 illustrates the proposed model, which is termed Task Graph Transformer (TGT). The proposed model can take as input either D -dimensional embeddings of textual descriptions of key-steps or D -dimensional video embeddings of key-step segments extracted from video. In the first case, the model takes as input the same set of embeddings at each forward pass, while in the second case, at each forward pass, we randomly sample a video embedding per key-step from the training videos (hence each key-step embedding can be sampled from a different video). We also include two D -dimensional learnable embeddings for the S and E nodes. All key-step embeddings are processed by a transformer encoder, which outputs D -dimensional vectors enriched with information from other embeddings. To prevent representation collapse, we apply a regularization loss encouraging distinctiveness between pairs of different nodes. Let X be the matrix of embeddings produced by the transformer model. We L2-normalize features, then compute

Table 2: We compare the abilities of our TGT model trained on visual features to generalize to two fundamental video understanding tasks, i.e., pairwise ordering and future prediction. Despite not being explicitly trained for these tasks, our model exhibits video understanding abilities, surpassing the baseline.

pairwise cosine similarities Y = X · X T · exp( T ) as in [33]. To prevent the transformer encoder from mapping distinct key-step embeddings to similar representations, we enforce the values outside the diagonal of Y to be smaller than the values in the diagonal. This is done by encouraging each row of the matrix Y to be close to a one-hot vector with a cross-entropy loss. Regularized embeddings are finally passed through a relation transformer head which considers all possible pairs of embeddings and concatenates them in a ( n +2) × ( n +2) × 2 D matrix R of relation vectors. For instance, R [ i, j ] is the concatenation of vectors X [ i ] and X [ j ] . Relation vectors are passed to a transformer layer which aims to mine relationships among relation vectors, followed by a multilayer perceptron to reduce dimensionality to 16 units and another pair of transformer layer and multilayer perceptron to map relation vectors to scalar values, which are reshaped to size ( n +2) × ( n +2) to form the score matrix A . We hence apply the same optimization procedure as in the DO method to supervise the whole architecture.

4 Experiments and Results

4.1 Graph Generation

Problem Setup We evaluate the ability of our approach to learn task graph representations on CaptainCook4D [30], a dataset of egocentric videos of 24 cooking procedures performed by 8 volunteers. Each procedure is accompanied by a task graph describing key-steps constraints. We tackle task graph generation as a weakly supervised learning problem in which models have to generate valid graphs by only observing labeled action sequences (weak supervision) rather than relying on task graph annotations (strong supervision), which are not available at training time. All models are trained on videos that are free from ordering errors or missing steps to provide a likely representation of procedures. We use the two proposed methods in the previous section to learn 24 task graph models, one per procedure, and report average performance across procedures.

Compared Approaches Wecompare our methods with previous approaches to task graph generation, and in particular with MSGI [39] and MSG 2 [20], which are approaches for task graph generation based on Inductive Logic Programming (ILP). We also consider the recent approach proposed in [3] which generates a graph by counting co-occurrences of matched video segments. Since we assume labeled actions to be available at training time, we do not perform video matching and use ground truth segment matching provided by the annotations. This approach is referred to as 'Count-Based'. Given the popularity of large language models as reasoning modules, we also consider a baseline which uses a large language model 5 to generate a task graph from key-step descriptions, without any access to key-step sequences. 6 We refer to this model as 'LLM'.

Graph Generation Results Results in Table 1 highlight the complexity of the task, with classic approaches based on inductive logic, such as MSGI, achieving poor performance ( 12 . 8 F 1 ), language models and count-based statistics reconstructing only basic elements of the graph ( 55 . 0 and 60 . 6 F 1 for LLM and Count-Based respectively), and even more recent methods based on inductive logic and heuristics only partially predicting the graph ( 71 . 1 F 1 of MSG 2 ). The proposed Direct Optimization

(DO) approach outperforms all other methods, achieving the highest scores across all measures, with improvements in the [+15 . 5 , +18 . 1] range with respect to the best competitor MSG 2 . This result highlights the effectiveness of the proposed framework to learn task graph representations from key-step sequences, especially considering the simplicity of the DO method, which performs gradient descent directly on the adjacency matrix. We obtain a slightly higher recall as compared to the precision ( 89 . 7 vs 86 . 4 ), showing that our approach tends to retrieve most ground truth edges, while hallucinating some pre-conditions, probably due to the dataset being unbalanced towards the most common ways of completing a procedure. Second best results are consistently obtained by our feature-based TGT approach, showing the generality of our learning framework and the potential of integrating it into complex neural architectures. Tight confidence intervals for DO highlight the stability of the proposed loss. The lower performance of TGT, as compared to DO, may be due to the relatively small size of the dataset, which makes it hard for complex architecture to generalize.

Video Understanding Results Table 2 reports the performance of TGT trained on videos on two fundamental video understanding tasks [46] of pairwise clip ordering and future prediction. 6 For pairwise ordering, we feed our TGT model with video embeddings of two clips and sort them according to the predicted adjacency matrix, placing first the clip identified as a pre-condition. For future predictions, given an anchor clip, we have to choose which among two other clips is the correct future. Despite TGT not being explicitly trained for pairwise ordering and future predictions, it exhibits emerging video understanding abilities, surpassing the random baseline.

4.2 Online Mistake Detection

Problem Setup We follow the PREGO benchmark and used the datasets (Assembly101-O and EPICTent-O) recently proposed in [13], in which models are tasked to perform online action detection from procedural egocentric videos. To evaluate the usefulness of task graphs on this downstream task, we design a system which flags the current action as a mistake if its pre-conditions in the predicted graph do not appear in previously observed actions. 6

Competitors We compare our approach with respect to the PREGO model proposed in [13], which detects mistakes based on the comparison between the currently observed action and an action predicted by a forecasting module. We note that PREGO is based on an implicit representation of the procedure (the forecasting module), while our approach is based on the explicit task graph representation, learned with the proposed framework. We also compare our approach with respect to baselines based on all graph prediction approaches compared in Table 1 to assess how the ability to predict accurate graphs affects downstream performance. For all methods, we report results based on ground truth action segments and on action sequences predicted by a MiniRoad [2] instance, a state-of-the-art online action detection module trained on each target dataset.

Results Results in Table 3 highlight the usefulness of the learned task graphs for downstream applications. The proposed DO method achieves significant gains over prior art with improvements of +19 . 8 and +7 . 5 in average F 1 score on Assembly101-O and EPIC-Tent-O respectively when ground truth action sequences are considered to make predictions. While TGT is the second-best performer on Assembly101-O, it obtains best results on EPIC-Tent-O ( 64 . 1 vs 58 . 3 in average F 1 score). This is due to the nature of action annotations in the two datasets. Indeed, while key-step names are informative in EPIC-Tent (e.g., 'Place Vent Cover', 'Open Stake Bag', or 'Spread Tent'), they are less distinctive in Assembly101 (e.g., 'attach cabin', 'attach interior', or 'screw chassis'). This highlights the flexibility of the proposed learning framework which can work in purely abstract, symbolic settings, with the DO approach, but can also leverage semantics with TGT when beneficial. Interestingly, the second best performers are graph-based approaches, with MSG 2 achieving an average F 1 of 56 . 1 on Assembly101-O and the simple Count-Based approach obtaining an average F 1 score of 56 . 6 on EPIC-Tent-O. In contrast, PREGO obtains average F 1 scores of 39 . 4 and 32 . 1 on Assembly101-O and EPIC-Tent-O respectively, suggesting the potential of explicit graph-based representations for mistake detection, versus the implicit one of PREGO. Breaking down performance into correct and mistake F 1 scores reveal some degree of unbalance of our approaches and the main competitor MSG 2 towards identifying correct actions rather than mistakes. This suggests that the related graph-based representations tend to detect some spurious pre-conditions, probably due to the limited demonstrations included in the videos, while the implicit PREGO model exhibits a skew with respect to mistakes. Further breaking down F 1 scores into related precision and recall values

highlights that the main failure modes are due to large imbalances between precision and recall. For instance, the Count-Based method achieves a precision of only 4 . 8 with a recall of 85 . 7 in predicting correct segments on Assembly101-O. In contrast, the proposed approach obtains balanced precision and recall values in detecting correct segments in Assembly101-O ( 98 . 2 / 83 . 4 ) and EPIC-Tent-O ( 94 . 1 / 93 . 5 ), and detecting mistakes in EPIC-Tent-O ( 33 . 3 / 35 . 7 ), while the prediction of mistakes on Assembly101-O is more skewed ( 46 . 7 / 90 . 4 ). Results based on action sequences predicted from video (bottom part of Table 3) highlight the challenging nature of the task when considering noisy action sequences (see Figure 4). While the explicit task graph representation may not accurately reflect the predicted noisy action sequences, we still observe improvements over previous approaches of +7 . 3 and +1 . 3 in average F 1 score in Assembly101-O and EPIC-Tent-O. Remarkably, best competitors are still graph-based methods, such as MSG 2 and the Count-Based approach, with significant improvements over the implicit representation of the PREGO model ( 32 . 5 average F 1 versus 53 . 5 of the proposed DO model). Also, in this case, we observe that graph-based methods tend to be skewed towards detecting correct action sequences. In this regard, our TGT model only achieves 38 . 2 in mistake F 1 score, a drop in 5 . 7 points over the best performer, the Count-Based method, which, on the other hand, only achieves an F 1 score of 2 . 6 when predicting correct segments.

5 Limitations

The proposed approach requires the availability of key-step sequences, a common assumption of works addressing other video understanding tasks [6, 22, 19, 17, 18]. While our method is applicable to any fully supervised video understanding dataset, future works should focus on overcoming such limitation and taking advantage of the vast amount of unlabeled video and textual data sets. While the proposed TGT method has shown promising results when trained directly on video features, the investigation of task graph learning in the absence of labeled key-step sequences is beyond the scope of this paper. We noted a reduced ability of our approach to work with noisy action sequences and a tendency to hallucinate pre-conditions, likely due to the limited expressivity of key-step sequences arising from videos showing the most common ways to perform a procedure. The performance of our designed system to detect mistakes is influenced by the quality of action recognition (see Figure 4). If the action recognition module fails to detect an action, the method may incorrectly signal a missing pre-condition. Conversely, if an action is falsely detected as performed, the method may fail to signal an actual mistake. Future improvements in online action recognition will enhance the robustness of our method. Furthermore, our approach does not explicitly model 'optional'

0.0

0.2

0.4

0.6

0.8

1.0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

F1

0.76

0.73

0.68

0.66

0.63

0.62

0.58

0.55

0.52

0.50

0.46

0.90

0.88

0.84

0.82

0.80

0.77

0.75

0.71

0.68

0.64

0.59

0.62

0.58

0.52

0.49

0.47

0.46

0.42

0.40

0.37

0.36

0.33

Avg

Correct

Mistake

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

F1

0.64

0.54

0.50

0.47

0.45

0.42

0.40

0.39

0.37

0.35

0.36

0.94

0.86

0.82

0.76

0.74

0.69

0.66

0.65

0.62

0.59

0.60

0.34

0.21

0.18

0.18

0.16

0.14

0.14

0.14

0.13

0.11

0.12

Avg

Correct

Mistake

Perturbation rate

Perturbation rate

key-steps, which can lead to incorrect error signaling if optional steps are treated as mandatory. This issue could potentially be addressed through the integration of specialized modules capable of detecting optional nodes. Another limitation of task graph representations, both in this work and in prior approaches [3, 44, 39, 20], is their inability to explicitly model repeatable key-steps. Recent advancements such as [18] have introduced a 'repeatable' node attribute to task graphs, but this extension is based on manual annotations, and the automatic learning of such attributes from data remains an open problem. Despite this limitation, the proposed error detection model demonstrates an ability to handle cases where key-steps may recur (e.g., spreading peanut butter). At test time, pre-conditions of key-steps are verified via the predicted task graph, even if a key-step has appeared earlier in the sequence. Nevertheless, more effective modeling of repeatable key-steps, especially in contexts where specific repetitions are required (e.g., 'cut three slices of bread'), remains an important area for future research. Future work should explore methods for incorporating these requirements into task graph learning frameworks. Our method follows the setup of PREGO [13], which defines the Assembly101-O and EPIC-Tent-O datasets as curated versions of their originals to account for open-set procedural errors such as 'order', 'omission', 'correction', and 'repetition' mistakes. These are procedural mistakes, as distinguished from 'proficiency errors' described in prior works [18]. The proposed method focuses on procedural mistakes at the abstract level of executed actions, and thus, would not be directly applicable to proficiency error detection. In real-world systems, this limitation could be mitigated by integrating subsystems that specialize in detecting different types of errors. Developing an integrated approach that addresses both procedural and proficiency errors is a promising direction for future research.

6 Conclusion

We considered the problem of learning task graph representations of procedures from video demonstrations. Framing task graph learning as a maximum likelihood estimation problem, we proposed a differentiable loss which allows direct optimization of the adjacency matrix through gradient descent and can be plugged into more complex neural network architectures. Experiments on three datasets show that the proposed approach can learn accurate task graphs, develop video understanding abilities, and improve the downstream task of online mistake detection surpassing state of the art methods. We release our code at the following URL: https: //github.com/fpv-iplab/Differentiable-Task-Graph-Learning .

7 Acknowledgments

This research is supported in part by the PNRR PhD scholarship 'Digital Innovation: Models, Systems and Applications' DM 118/2023, by the project Future Artificial Intelligence Research (FAIR) PNRR MUR Cod. PE0000013 - CUP: E63C22001940006, and by the Research Program PIAno di inCEntivi per la Ricerca di Ateneo 2020/2022 - Linea di Intervento 3 'Starting Grant' EVIPORES Project - University of Catania.

We thank the authors of [13] and in particular Alessandro Flaborea and Guido D'Amely for sharing the code to replicate experiments in the PREGO benchmark.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Joungbin An, Hyolim Kang, Su Ho Han, Ming-Hsuan Yang, and Seon Joo Kim. Miniroad: Minimal rnn framework for online action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10341-10350, 2023.

Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras, and Kristen Grauman. Videomined task graphs for keystep recognition in instructional videos. Advances in Neural Information Processing Systems , 36, 2024.

Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In European Conference on Computer Vision , pages 657-675. Springer, 2022.

Siddhant Bansal, Chetan Arora, and CV Jawahar. United we stand, divided we fall: Unitygraph for unsupervised procedure learning from videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6509-6519, 2024.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 961-970, 2015.

Guodong Ding, Fadime Sener, Shugao Ma, and Angela Yao. Every mistake counts in assembly. arXiv preprint arXiv:2307.16453 , 2023.

Lucia Donatelli, Theresa Schmidt, Debanjali Biswas, Arne Köhn, Fangzhou Zhai, and Alexander Koller. Aligning actions across recipe graphs. In Proceedings of the 2021 conference on empirical methods in natural language processing , pages 6930-6942, 2021.

Mikita Dvornik, Isma Hadji, Konstantinos G Derpanis, Animesh Garg, and Allan Jepson. Drop-dtw: Aligning common signal between sequences while dropping outliers. Advances in Neural Information Processing Systems , 34:13782-13793, 2021.

Nikita Dvornik, Isma Hadji, Hai Pham, Dhaivat Bhatt, Brais Martinez, Afsaneh Fazly, and Allan D Jepson. Graph2vid: Flow graph to video grounding for weakly-supervised multi-step localization. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. Stepformer: Self-supervised step discovery and localization in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18952-18961, 2023.

Ehsan Elhamifar and Dat Huynh. Self-supervised multi-task procedure learning from instructional videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16 , pages 557-573. Springer, 2020.

Alessandro Flaborea, Guido Maria D'Amely di Melendugno, Leonardo Plini, Luca Scofano, Edoardo De Matteis, Antonino Furnari, Giovanni Maria Farinella, and Fabio Galasso. Prego: online mistake detection in procedural egocentric videos. In International Conference on Computer Vision and Patter Recognition (CVPR) , 2024.

Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence , 43(11):4021-4036, 2020.

Reza Ghoddoosian, Isht Dwivedi, Nakul Agarwal, and Behzad Dariush. Weakly-supervised action segmentation and unseen error detection in anomalous instructional videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10128-10138, 2023.

Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 13505-13515, 2021.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259 , 2023.

Youngkyoon Jang, Brian Sullivan, Casimir Ludwig, Iain Gilchrist, Dima Damen, and Walterio MayolCuevas. Epic-tent: An egocentric video dataset for camping tent assembly. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0-0, 2019.

Yunseok Jang, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Moontae Lee, and Honglak Lee. Multimodal subtask graph generation from instructional videos. arXiv preprint arXiv:2302.08672 , 2023.

Takeo Kanade and Martial Hebert. First-person vision. Proceedings of the IEEE , 100(8):2442-2453, 2012.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 982-992, 2015.

Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13853-13863, 2022.

Zijia Lu and Ehsan Elhamifar. Set-supervised action learning in procedural task videos via pairwise order consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19903-19913, 2022.

Pierre Simon Marquis de Laplace. Théorie analytique des probabilités , volume 7. Courcier, 1820.

Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9879-9889, 2020.

Medhini Narasimhan, Licheng Yu, Sean Bell, Ning Zhang, and Trevor Darrell. Learning and verification of task structure in instructional videos. arXiv preprint arXiv:2303.13519 , 2023.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Jikai Wang, Qifan Zhang, Vasundhara Komaragiri, Eric Ragan, Nicholas Ruozzi, et al. Captaincook4d: A dataset for understanding errors in procedural activities. arXiv preprint arXiv:2312.14556 , 2023.

Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocentric vision. International Journal fn Computer Vision , 2023.

Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5285-5297, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando. Interaction region visual transformer for egocentric action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6740-6750, 2024.

Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. proScript: Partially ordered scripts generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2138-2149, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Pol Schumacher, Mirjam Minor, Kirstin Walter, and Ralph Bergmann. Extraction of procedural knowledge from the web: A comparison of two workflow extraction approaches. In Proceedings of the 21st International Conference on World Wide Web , pages 739-747, 2012.

Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2109621106, 2022.

Steven S Skiena. The algorithm design manual , volume 2. Springer, 1998.

Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. arXiv preprint arXiv:2001.00248 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 20270-20281, 2023.

Yoko Yamakata, Shinsuke Mori, and John A Carroll. English recipe flow graph corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 5187-5194, 2020.

Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure-aware video representation from instructional videos and their narrations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14825-14835, 2023.

Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10727-10738, 2023.

Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Yipin Zhou and Tamara L Berg. Temporal perception and prediction in ego-centric video. In Proceedings of the IEEE International Conference on Computer Vision , pages 4498-4506, 2015.

Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3537-3545, 2019.

Get a

Bowl

Crack

Egg

Add

Water

Add

Milk

Mix

Eggs

Pour

Mixture

A Task Graph

An example of a task graph is illustrated in Figure 5. A task graph is a Directed Acyclic Graph (DAG) where nodes represent key-steps and directed edges impose a partial order on these steps, indicating the necessary preconditions for each node. For example, the key-step 'Mix' has preconditions such as 'Add Water', 'Add Milk', and 'Crack Egg'. This formulation of task graphs is not a novel contribution of this paper but was originally introduced in [18].

B Evaluation Measures

This appendix details the evaluation measures used to assess performance experimentally for the two considered tasks of task graph generation and online mistake detection.

Task Graph Generation Task graph generation is evaluated by comparing a generated graph ˆ G = ( ˆ K , ˆ A ) with a ground truth graph G = ( K , A ) . Since task graphs aim to encode ordering constraints between pairs of nodes, we evaluate task graph generation as the problem of identifying valid pre-conditions (hence valid graph edges) among all possible ones. We hence adopt classic detection evaluation measures such as precision, recall, and F 1 score. In this context, we define True Positives (TP) as all edges included in both the predicted and ground truth graph (Eq. (7)), False Positives (FP) as all edges included in the predicted graph, but not in the ground truth graph (Eq. (8)), and False Negatives (FN) as all edges included in the ground truth graph, but not in the predicted one (Eq. (9)). Note that true negatives are not required to compute precision, recall and F 1 score.

Online Mistake Detection We follow previous works on mistake detection from procedural egocentric videos [37, 41, 13] and evaluate online mistake detection with standard precision, recall, and F 1 scores. We break down metrics by the 'correct' and 'mistake' classes, as well as report average values.

C Implementation Details

This appendix provides implementation details to replicate the experiments discussed in Section 4.

C.1 Data Augmentation

In procedural tasks, it is common for certain actions to be repeated multiple times throughout the execution of a task. For example, in the EPIC-Tent dataset [19], an operation such as "reading the instructions" may be performed repeatedly at any point during the task. To model key-step orderings within the framework of topological sorts, our approach assumes that sequences should not contain such repetitions. Since repetitions denote that a specific action can appear at different stages of a procedure, we expand each sequence with repetitions to all distinct sequences obtained by dropping repeated actions. This data augmentation strategy enhances the robustness of our model on Assembly101 [37] and EPIC-Tent [19], while it was not necessary for the CaptainCook4D dataset [30], as sequences do not contain any repetitions.

C.2 Early Stopping

The learning process was conducted without the use of a validation set. To avoid overfitting and saving computation we defined a 'Sequence Accuracy (SA)' score used to determine when the model reaches a learning plateau. We early stop models when an SA value of at least 0 . 95 is reached, and if the model shows no SA improvement for 25 consecutive epochs. The SA score is as follows:

where Y defined sequences in the training set, y is a sequence from Y , y i is the i -th element of sequence y , y [: i ] are the predecessors of the i -th element in the sequence y , and pred ( y i , Z ) are the predicted predecessors for y i from the current binarized adjacency matrix Z . The function c is defined as:

The SA score measures the compatibility of each sequence with the current task graph based on the ratio of correctly predicted predecessors of the current symbol y i of the sequence to the total number of predicted predecessors for y i in the current task graph.

C.3 Hyperparameters

Table 4 details the hyperparameters employed in the experiments for task graph generation on the CaptainCook4D dataset [30]. During the training of TGT, we utilized a pre-trained EgoVLPv2 [32] on Ego-Exo4D [18] to extract text and video embeddings. The temperature value T used in the cross-entropy distinctiveness loss was set to 0 . 9 as in [33]. The β parameter was linearly annealed from an initial value of 1.0 to a final value of 0.05, with updates occurring every 100 epochs. This gradual decrease in β mimics the warm-up strategy of [40], enabling smoother optimization early in training and leading to improved convergence as training progresses.

Table 5 details the hyperparameters employed in the experiments for task graph generation on the Assembly101-O and EPIC-Tent-O datasets. For the downstream task of online mistake detection within the DO model framework, we extended the maximum training epochs to 1200, particularly for Assembly101-O. This change was necessary because, even after 1000 epochs, the model continued to exhibit many cycles among its 86 nodes. Extending the number of epochs allows the model additional time to learn and minimize these cycles, which is crucial given the complexity of the graph. In the TGT configuration, we reduced the dropout rate, while the β parameter was gradually annealed from an initial value of 1.0 to 0.55 to prevent overfitting.

The reader is referred to the code for additional implementation details.

Table 5: List of hyper-parameters used in the models training process for task graph generation using Assembly101-O and EPIC-Tent-O.

C.4 LLMPrompt

Below is the prompt that was employed to instruct the model on its task, which involves identifying pre-conditions for given procedural steps.

I would like you to learn to answer questions by telling me the steps that need to be performed before a given one.

The questions refer to procedural activities and these are of the following type:

Q -Which of the following key steps is a pre-condition for the current key step "add brownie mix"?

-pour the mixture in the tray

-add oil -add water -break eggs -mix all the contents -mix eggs -spray oil on the tray

-None of the above

Your task is to use your immense knowledge and your immense ability to tell me which preconditions are among those listed that must necessarily be carried out before the key step indicated in quotes in the question.

You have to give me the answers and a very brief explanation of why you chose them.

Provide the correct preconditions answer inside a JSON format like this: { eggs"]

C.5 Data Split

The CaptainCook4D dataset [30] comprises various error types, including order errors, timing errors, temperature errors, preparation errors, missing steps errors, measurement errors, and technique errors. Of these, missing steps and order errors directly impact the sequence integrity. Consequently, for our task graph generation, we utilized only those sequences of actions free from these specific types of errors. Table 6 shows statistics on the CaptainCook4D subsets used for task graph generation.

For Online Mistake Detection, we considered the datasets defined by the authors of PREGO [13].

In the context of pairwise ordering and forecasting, we employed the subset of the CaptainCook4D dataset designated for task graph generation (refer to Table 6) and divided it into training and

testing sets. This division was carefully managed to ensure that 50% of the scenarios were equally represented in both the training and testing sets.

C.6 Pairwise ordering and future prediction

We setup the pairwise ordering and future prediction video understanding tasks following [46].

Pairwise Ordering Models take as input two randomly shuffled video clips and are tasked with recognizing the correct ordering between key-steps. We sample all consecutive triplets of labeled segments from test videos, discard the middle one, and consider the first and third ones as input pair. We evaluate models using accuracy.

Future Prediction Models take as input an anchor video clip and two randomly shuffled video clips and are tasked to select which of the two clips is the correct future of the anchor clip. We sample all consecutive triplets of labeled segments from test videos and consider the middle clip as the anchor and the remaining two clips as the two options. We evaluate models using accuracy.

Model We trained our TGT model using video embeddings extracted with a pre-trained EgoVLPv2 [32] on Ego-Exo4D [18]. During the training process, if multiple video embeddings are associated with the same key-step across the training sequences, one embedding per key-step is randomly selected. The model is trained for task graph generation on the training video and tested for pairwise ordering and future prediction on the test set.

For pairwise ordering, we feed our model with two clips and obtain a 4 × 4 adjacency matrix, where the nodes represent START , A , B , END . We establish the order between A and B based on the fulfillment of at least one of the following conditions: (a) if the weight of the edge A → B is greater than the weight of the edge B → A , we conclude that A precedes B ; (b) by analyzing the sequences

< START , A, B, END > and < START , B, A, END > , we calculate their probabilities using Eq. (4). If P ( < START , A, B, END > | Z ) is greater than P ( < START , B, A, END > | Z ) , we infer that A precedes B ; (c) if the weight of the edge END → B is greater than that of END → A , it implies that B is a necessary precondition for concluding the procedure, indicating that B follows A , and consequently, A precedes B . If none of these conditions hold, we determine that B precedes A .

For future prediction, we feed three clips and obtain a 5 × 5 adjacency matrix, where the nodes represent START , A , anchor , B , and END . We hence inspect the weights of edges anchor → A and anchor → B and choose as the future clip, the one related to the smallest weight (a small weight indicates that the selected clip is not a precondition). Another method to determine the future clip is by calculating the probabilities of the sequences < START , A, anchor, B, END > and < START , B, anchor, A, END > using Eq. (4). If P ( < START , A, anchor, B, END > | Z ) is greater than P ( < START , B, anchor, A, END > | Z ) , we infer that the sequence involving A before B is more probable, indicating that B is the future clip for anchor . Conversely, if the probability of the second sequence is greater, then A is deemed the future clip for anchor .

C.7 Scalability of Task Graph Transformer (TGT)

The Direct Optimization (DO) approach requires a separate training session for each procedure. Task Graph Transformer (TGT) offers more flexibility by allowing different sets of key-step embeddings at each forward pass, ideally enhancing scalability. Leveraging this capability, we trained a single TGT text model for all CaptainCook4D procedures. This was achievable due to TGT's ability to handle varying embeddings per forward pass, enabling simultaneous optimization across multiple procedures during training. As shown in Table 7, the confidence intervals of both the single and unified models highlight some performance variance. The unified model exhibits lower average precision, recall, and F1 scores compared to the individually trained models, with a larger confidence interval. However, the results suggest that TGT-text models can still generalize across diverse procedures, reducing training complexity while maintaining reasonable performance.

C.8 Graph Post-processing

We binarize the adjacency matrix with the threshold 1 n , where n is the number of nodes. After this thresholding phase, it is possible to encounter situations like the one illustrated in Figure 6, where node A depends on nodes B and C, and node B depends on node C. Due to the transitivity of the pre-conditions, we can remove the edge connecting node A to node C, as node B must precede node A. Sometimes, it may occur that a node does not serve as a pre-condition for any other node; in such cases, the END node should be directly connected to this node. Conversely, if a node has no pre-conditions, an edge is added from the current node to the START node.

At the end of the training process, obtaining a graph containing cycles is also possible. In such cases, all cycles within the graph are considered, and the edge with the lowest score within each cycle is removed. This method ensures that the graph remains a Directed Acyclic Graph (DAG).

C.9 Details on Online Mistake Detection

Given the noisy sequences in Assembly101 [37] and EPIC-Tent [19], a distinct approach was adopted during the post-processing phase of task graph generation. Specifically, if a key-step in the task graph has only two pre-conditions and one is the START node, the other pre-condition will be removed regardless of its score, otherwise we apply the transitivity dependences reduction aforementioned. This approach allows for a graph with fewer pre-conditions in the initial steps.

C

B

A

(a)

C

B

A

C

B

A

(b)

Read

Instruction

Pickup/Open

Pickup/Open

Supportbag

START

Read Instruction

Pickup/Open

Pickup/Open

Supportbag

Assemble

Support

Pickup/Place

Ventcover

Pickup/Open

Place Guyline

Tie Top

END

Assemble

Support

Support

Support Tab

Pickup/Open

Past key-steps

Current key-step

START

Read Instruction

Pickup/Open

Pickup/Open

Supportbag

Assemble

Support

...

GT:

correct

Past key-steps

Current key-step

Read

Instruction

Pickup/Open

Pickup/Open

Supportbag

correct

correct

GT:

correct

correct

correct

correct

correct

correct

correct

correct

mistake

In the case of Assembly101, which includes multiple procedural tasks, we opted to consider a single task graph that summarizes all the procedures, rather than generating individual graphs for each.

C.10 Qualitative Examples

Figures 8 - 31 report qualitative examples of prediction using our Direct Optimization (DO) method on the procedures of CaptainCook4D. The task graphs must be read in a bottom-up manner, where the START node (bottom) is at the lowest position and represents the first node with no preconditions, while the END node (up) is the final step of the procedure.

Figure 7 reports a qualitative analysis of the generated task graph for detecting the mistakes on EPIC-Tent-O.

C.11 Experiments Compute Resources

The experiments involving the training of the DO model on symbolic data from the CaptainCook4D dataset proved to be highly efficient. We were able to generate all the task graphs in approximately half an hour using a Tesla V100S-PCI GPU. This GPU allowed us to run up to 8 training processes simultaneously. In contrast, training the TGT models for all scenarios in the CaptainCook4D dataset required about 24 hours, with the same GPU supporting the concurrent training of up to 2 models. Additionally, once the task graphs were obtained, executing the PREGO benchmarks for mistake detection was significantly faster, requiring online action prediction, which could be performed in real-time on a Tesla V100S-PCI GPU.

D Societal Impact

Reconstructing task graphs from procedural videos may enable the construction of agents able to assist users during the execution of the task. Learning task graphs from videos may be affected by geographical or cultural biases appearing in the data (e.g., specific ways of performing given tasks), which may limit the quality of the feedback returned to the user, potentially leading to harm. We expect that training data of sufficient quality should limit such risks.

START

Add-1/2 tsp baking powder to a blender

Serve-Serve the pancakes with chopped strawberries

Chop-Chop 1 strawberry

Transfer-Transfer to a plate

cook-cook for 20-30 seconds more

Flip-Flip the pancakes with a fork or a

fi

sh slice spatula

Cook-Cook for 1 min or until the tops start to bubble

Melt-Melt a small knob of butter in a non-stick frying pan over low-medium heat

blitz-blitz the blender for 20 seconds

Add-Add 1 banana to a blender

Add-1 egg to a blender

Add-1 heaped tbsp

fl

our to a blender

splash-splash maple syrup on plate

Pour-Pour three little puddles straight from the blender into the frying pan

END

(a)

START

Add-1/2 tsp baking powder to a blender

Serve-Serve the pancakes with chopped strawberries

Transfer-Transfer to a plate

cook-cook for 20-30 seconds more

Flip-Flip the pancakes with a fork or a

fi

sh slice spatula

Cook-Cook for 1 min or until the tops start to bubble

Melt-Melt a small knob of butter in a non-stick frying pan over low-medium heat

blitz-blitz the blender for 20 seconds

Add-Add 1 banana to a blender

Add-1 egg to a blender

Add-1 heaped tbsp

fl

our to a blender

splash-splash maple syrup on plate

Pour-Pour three little puddles straight from the blender into the frying pan

Chop-Chop 1 strawberry

END

(b)

START

Whisk-Whisk the egg

add-Extract and add contents of an egg to a microwave-safe bowl

Microwave-Microwave for 3 minutes, stirring in between

Add-Add 1/2 tbsp sweet and sour sauce to the bowl

Sprinkle-Sprinkle 1 tbsp shredded cheddar cheese on top of the egg

Pour-Pour egg mixture on top of the tortilla

Mix-Mix the contents of the bowl well

Place-Place 8 inch tortilla on a cutting board

Add-Add 1 tbsp salsa to the bowl

Sprinkle-Sprinkle oregano in the bowl

Roll-Roll the tortilla from one end to another into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps but not so tight that the

fi

lling leaks

END

(a)

START

Whisk-Whisk the egg

add-Extract and add contents of an egg to a microwave-safe bowl

Microwave-Microwave for 3 minutes, stirring in between

Add-Add 1/2 tbsp sweet and sour sauce to the bowl

Sprinkle-Sprinkle 1 tbsp shredded cheddar cheese on top of the egg

Pour-Pour egg mixture on top of the tortilla

Mix-Mix the contents of the bowl well

Add-Add 1 tbsp salsa to the bowl

Place-Place 8 inch tortilla on a cutting board

Sprinkle-Sprinkle oregano in the bowl

Roll-Roll the tortilla from one end to another into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps but not so tight that the

fi

lling leaks

END

(b)

START

add-add 1/2 tbsp softened butter to the bowl

Mix-Mix the cheese and red bell pepper in the bowl

Microwave-Microwave the bowl, covered, for 2 minutes

Melt-Melt the cheese by microwaving cup for 30 sec. (Check after 30 seconds and microwave for 10 seconds more if needed)

Add-Add 1 tablespoons of water to the bowl

Place-Place the chopped pepper in the microwave-safe bowl

Add-Add 1/3 cup cheddar cheese to a microwave-safe cup

Chop-Chop 1/4 red bell pepper into tiny bits

Add-Add 1/4 teaspoon salt to the bowl

Add-Add 1/4 teaspoon pepper to the bowl

Mix-Mix all the ingredients of the bowl well

END

(a)

START

add-add 1/2 tbsp softened butter to the bowl

Add-Add 1/4 teaspoon salt to the bowl

Mix-Mix the cheese and red bell pepper in the bowl

Microwave-Microwave the bowl, covered, for 2 minutes

Melt-Melt the cheese by microwaving cup for 30 sec. (Check after 30 seconds and microwave for 10 seconds more if needed)

Add-Add 1 tablespoons of water to the bowl

Place-Place the chopped pepper in the microwave-safe bowl

Add-Add 1/3 cup cheddar cheese to a microwave-safe cup

Chop-Chop 1/4 red bell pepper into tiny bits

Add-Add 1/4 teaspoon pepper to the bowl

Mix-Mix all the ingredients of the bowl well

END

(b)

START

Wait-Wait about 30 seconds for the co

ff

ee to bloom. (You wil  see smal  bubbles or foam on the co

ff

ee grounds during this step.)

Pour-Pour a smal  amount of water into the

fi

lter to wet the grounds

Transfer-Transfer the grounds to the

fi

check-Once the water has boiled, check the temperature of the water. (The water should be between 195-205 degrees Fahrenheit or between 91-96 degrees Celsius. If the water is too hot, let it cool brie

fl

y.)

spread-spread open

fi

lter in dripper to create a cone

Grind-Grind the co

ff

ee beans until the co

ff

ee grounds are the consistency of coarse sand, about 20 seconds

Boil-Boil the water. (While the water is boiling, assemble the

fi

Place-Place the paper

fi

lter in the dripper

Weigh-Weigh the co

ff

ee beans (0.8oz-0.12 oz)

transfer-transfer water to a ket le

Prepare-Prepare the

fi

fi

lter in half to create a semi-circle, and in half again to create a quarter-circle

Place-Place the dripper on top of a co

ff

ee mug

Measure-Measure 12 ounces of cold water

pour-Slowly pour the rest of the water over the grounds in a circular motion. Do not over

fi

l  beyond the top of the paper

fi

fi

ff

ee grounds

drain-Let the co

ff

ee drain completely into the mug before removing the dripper

END

(a)

START

Wait-Wait about 30 seconds for the co

ff

ee to bloom. (You wil  see smal  bubbles or foam on the co

ff

ee grounds during this step.)

Pour-Pour a smal  amount of water into the

fi

lter to wet the grounds

Transfer-Transfer the grounds to the

fi

check-Once the water has boiled, check the temperature of the water. (The water should be between 195-205 degrees Fahrenheit or between 91-96 degrees Celsius. If the water is too hot, let it cool brie

fl

y.)

spread-spread open

fi

lter in dripper to create a cone

Grind-Grind the co

ff

ee beans until the co

ff

ee grounds are the consistency of coarse sand, about 20 seconds

Boil-Boil the water. (While the water is boiling, assemble the

fi

Weigh-Weigh the co

ff

ee beans (0.8oz-0.12 oz)

Place-Place the paper

fi

lter in the dripper

Measure-Measure 12 ounces of cold water

Prepare-Prepare the

fi

fi

lter in half to create a semi-circle, and in half again to create a quarter-circle

Place-Place the dripper on top of a co

ff

ee mug

transfer-transfer water to a ket le

pour-Slowly pour the rest of the water over the grounds in a circular motion. Do not over

fi

l  beyond the top of the paper

fi

fi

ff

ee grounds

drain-Let the co

ff

ee drain completely into the mug before removing the dripper

END

(b)

START

Add-1/4 teaspoon of red chil i powder to the bowl

whisk-In a mixing bowl, whisk 1 cup of chil ed curd until smooth. Use fresh homemade or packaged curd

peel-peel the cucumber

Rinse-Rinse 1 medium sized cucumber

Add-1/4 teaspoon salt to the bowl

chop or grate-chop or grate the cucumber

add-add 1 tablespoon of chopped cilantro leaves to the bowl

Add-Add 1 teaspoon of cumin powder to the bowl

Add-Add the chopped or grated cucumber to the whisked curd

END

(a)

START

Add-1/4 teaspoon of red chilli powder to the bowl

whisk-In a mixing bowl, whisk 1 cup of chilled curd until smooth. Use fresh homemade or packaged curd

peel-peel the cucumber

Rinse-Rinse 1 medium sized cucumber

Add-1/4 teaspoon salt to the bowl

chop or grate-chop or grate the cucumber

Add-Add the chopped or grated cucumber to the whisked curd

add-add 1 tablespoon of chopped cilantro leaves to the bowl

Add-Add 1 teaspoon of cumin powder to the bowl

END

(b)

START

Pour-Pour the sauces over the meatballs

Mix-Mix 1/4 cup sweet-and-sour sauce and 1/2 teaspoon soy sauce in a small bowl

Top-Top the plate with the carrots, onion, garlic and 1/4 tsp pepper powder

Slice-Slice 1/8 medium onion

Place-Place 5 meatballs in a Microwave-safe plate

Cut-Cut 1/4 medium carrot into short, thin strips

Mince-Mince 1/8 garlic clove

cut-Cut onion into two pieces

Cut-Cut 1/8 garlic clove

Stir-Stir the contents in the microwave with a spoon

Microwave-Microwave the plate, covered, on high for 1.5 minutes

Microwave-Microwave for 1 more minute

END

(a)

START

Pour-Pour the sauces over the meatballs

Mix-Mix 1/4 cup sweet-and-sour sauce and 1/2 teaspoon soy sauce in a small bowl

Top-Top the plate with the carrots, onion, garlic and 1/4 tsp pepper powder

Slice-Slice 1/8 medium onion

Place-Place 5 meatballs in a Microwave-safe plate

Cut-Cut 1/4 medium carrot into short, thin strips

Mince-Mince 1/8 garlic clove

cut-Cut onion into two pieces

Cut-Cut 1/8 garlic clove

Stir-Stir the contents in the microwave with a spoon

Microwave-Microwave the plate, covered, on high for 1.5 minutes

Microwave-Microwave for 1 more minute

END

(b)

START

Add-Add 1/8 cup soy sauce to the bowl

Set-Set aside the sauce mixture

Whisk-Whisk the contents of bowl

Add-Add 1 tablespoon honey to the bowl

Add-Add 1 teaspoon cornstarch to the bowl

Add-Add 2 cloves minced garlic to the bowl

Add-Add 1/2 tablespoon minced ginger to the bowl

Add-Add 1/8 teaspoon black pepper to the bowl

Add-Add 1/6 cup water the bowl

mince-mince garlic

cook-cook, stirring often, for 4 minutes. If the pan gets too hot on medium-high, turn the heat down to medium

Add-add sliced mushrooms to the skil et

Add-Add broccoli to the skil et

Heat-Heat 2 tablespoons olive oil in a skil et over medium-high heat

slice-slice mushrooms

slice-slice 1/3 of the bel  pepper

Take-Take 5 in number broccoli

fl

orets

Take-Take 2 cremini mushrooms

Take-Take 1 bel  pepper

Pour-Pour the sauce into the skil et

Whisk-Whisk the sauce again to recombine the ingredients

continue cooking-continue cooking, stirring often, for 2-3 minutes, until vegetables are crisp-tender

Add-Add bel  pepper to the skil et

cook-cook, stirring, for 1 minute until the sauce thickens

END

(a)

START

Add-Add 1/8 cup soy sauce to the bowl

Set-Set aside the sauce mixture

Whisk-Whisk the contents of bowl

Take-Take 2 cremini mushrooms

Add-Add 1 tablespoon honey to the bowl

Add-Add 2 cloves minced garlic to the bowl

Add-Add 1 teaspoon cornstarch to the bowl

mince-mince garlic

Add-Add 1/2 tablespoon minced ginger to the bowl

Add-Add 1/8 teaspoon black pepper to the bowl

Add-Add 1/6 cup water the bowl

Take-Take 5 in number broccoli

fl

orets

cook-cook, stirring often, for 4 minutes. If the pan gets too hot on medium-high, turn the heat down to medium

Add-add sliced mushrooms to the skil et

Add-Add broccoli to the skil et

Heat-Heat 2 tablespoons olive oil in a skil et over medium-high heat

slice-slice mushrooms

slice-slice 1/3 of the bel  pepper

Take-Take 1 bel  pepper

Pour-Pour the sauce into the skil et

Whisk-Whisk the sauce again to recombine the ingredients

continue cooking-continue cooking, stirring often, for 2-3 minutes, until vegetables are crisp-tender

Add-Add bel  pepper to the skil et

cook-cook, stirring, for 1 minute until the sauce thickens

END

(b)

START

add-1/8 cup shredded mozzarel a to a bowl

add-1/4 tsp salt to a bowl

Slice-Slice two 1/2 inch thick rounds from a baguette (slice slanted)

Spoon-Spoon the mixture from the bowl onto the bread

Toast-Toast both sides of the slices on the pan for 2 to 3 minutes until lightly charred and crispy and transfer the slices to a plate

add-1/4 tsp pepper to a bowl

add-1/16 cup basil to a bowl

add-In a bowl, add the cut cherry tomatoes

Brush-Brush 2 slices of baguette with olive oil on both sides

Cut-Cut 1/4 cup of cherry tomatoes into halves

END

(a)

START

add-1/8 cup shredded mozzarella to a bowl

add-1/4 tsp salt to a bowl

Slice-Slice two 1/2 inch thick rounds from a baguette (slice slanted)

Spoon-Spoon the mixture from the bowl onto the bread

add-1/4 tsp pepper to a bowl

Brush-Brush 2 slices of baguette with olive oil on both sides

add-In a bowl, add the cut cherry tomatoes

Cut-Cut 1/4 cup of cherry tomatoes into halves

add-1/16 cup basil to a bowl

Toast-Toast both sides of the slices on the pan for 2 to 3 minutes until lightly charred and crispy and transfer the slices to a plate

END

(b)

START

Cook-Cook for 2 minutes or until the zoodles are done

Add-1/6 cup grated parmesan cheese

season-pepper to taste

season-season with salt

Add-Add the zucchini noodles

Cook-Cook garlic until fragrant (about 1 minutes). Be careful not to burn garlic

Add-Add 1 large minced garlic cloves to the pan

Melt-Melt 1 tablespoons of softened butter

Heat-Heat a large pan on medium heat

Top-Top with more parmesan if desired

Spiralize-Spiralize 1 medium zucchini into thin noodles using a spiralizer

END

(a)

START

Cook-Cook for 2 minutes or until the zoodles are done

Add-1/6 cup grated parmesan cheese

season-pepper to taste

season-season with salt

Add-Add the zucchini noodles

Cook-Cook garlic until fragrant (about 1 minutes). Be careful not to burn garlic

Add-Add 1 large minced garlic cloves to the pan

Melt-Melt 1 tablespoons of softened butter

Top-Top with more parmesan if desired

Heat-Heat a large pan on medium heat

Spiralize-Spiralize 1 medium zucchini into thin noodles using a spiralizer

END

(b)

START

add-Measure 1/8 teaspoon of salt and add it to the mug

Take-Take a microwavable mug

Sprinkle-Sprinkle dried Italian herbs inside the mug

Sprinkle-Sprinkle 1 generous tablespoon of mozzarel a cheese on top of the sauce

spread-spread marinara sauce around the surface of the batter

Take-Take 1 tablespoon of marinara sauce

Mix-Mix the contents of the mug thoroughly. (There might be some lumps, but that is ok.)

Add-1 tablespoon of olive oil to the mug

Add-Add in 3 tablespoons of milk to the mug

Stir-Stir the contents in the mug wel

add-Measure 4 tablespoons of

fl

our and add it to the mug

add-Measure 1/16 teaspoon of baking soda and add it to the mug

add-Measure 1/8 teaspoon of baking powder and add it to the mug

Microwave-Microwave for 1 minute 20 seconds, or until it rises and the toppings are bubbling

END

(a)

START

add-Measure 1/8 teaspoon of salt and add it to the mug

Take-Take a microwavable mug

Sprinkle-Sprinkle dried Italian herbs inside the mug

Sprinkle-Sprinkle 1 generous tablespoon of mozzarel a cheese on top of the sauce

spread-spread marinara sauce around the surface of the batter

Take-Take 1 tablespoon of marinara sauce

Mix-Mix the contents of the mug thoroughly. (There might be some lumps, but that is ok.)

Add-1 tablespoon of olive oil to the mug

Add-Add in 3 tablespoons of milk to the mug

Stir-Stir the contents in the mug wel

Microwave-Microwave for 1 minute 20 seconds, or until it rises and the toppings are bubbling

add-Measure 4 tablespoons of

fl

our and add it to the mug

add-Measure 1/16 teaspoon of baking soda and add it to the mug

add-Measure 1/8 teaspoon of baking powder and add it to the mug

END

(b)

START

Heat-Heat 1 tbsp oil in a non-stick frying pan

Cut-Cut tomato into two pieces

Beat-Beat the contents of the bowl

Take-Take a tomato

crack-crack one egg in a bowl

add-add the chopped cilantro to the bowl

add-1/2 tsp ground black pepper to the bowl

Chop-Chop 2 tbsp cilantro

put-put tomatoes on a serving plate

Scoop-Scoop the tomatoes from the pan

cook-cook the tomatoes cut-side down until they start to soften and colour

stir-stir gently with a wooden spoon so the egg that sets on the base of the pan moves to enable the uncooked egg to

fl

ow into the space

Pour-Pour the egg mixture into the pan

Transfer-Transfer omelette to the plate and serve with the tomatoes

Stop-Stop stirring when it's nearly cooked to allow it to set into an omelette

END

(a)

START

Heat-Heat 1 tbsp oil in a non-stick frying pan

crack-crack one egg in a bowl

add-add the chopped cilantro to the bowl

Cut-Cut tomato into two pieces

Chop-Chop 2 tbsp cilantro

Take-Take a tomato

put-put tomatoes on a serving plate

Scoop-Scoop the tomatoes from the pan

Beat-Beat the contents of the bowl

cook-cook the tomatoes cut-side down until they start to soften and colour

add-1/2 tsp ground black pepper to the bowl

stir-stir gently with a wooden spoon so the egg that sets on the base of the pan moves to enable the uncooked egg to

fl

ow into the space

Pour-Pour the egg mixture into the pan

Transfer-Transfer omelette to the plate and serve with the tomatoes

Stop-Stop stirring when it's nearly cooked to allow it to set into an omelette

END

(b)

START

Pour-Pour 1 egg into the ramekin cup

Coat -Coat a 6-oz. ramekin cup with cooking spray

Place -Place the egg from the cup over the lettuce

Line -Line the bottom piece of the English mu

ffi

n with lettuce

Microwave -Microwave just until cheese melts, about 10 seconds

Cut -Cut the English mu

ffi

n into two pieces with a knife

sprinkle -sprinkle 1 tablespoon of cheese on cup

Top -Top cup with 1 tablespoon of salsa

Microwave -Continue to Microwave for 15-30 more seconds or until the egg is almost set

Microwave-Microwave the ramekin cup uncovered on high for 30 seconds

stir-stir the ramekin cup

replace -replace the top of the English mu

ffi

n

END

(a)

START

Pour-Pour 1 egg into the ramekin cup

Coat -Coat a 6-oz. ramekin cup with cooking spray

Place -Place the egg from the cup over the lettuce

Line -Line the bottom piece of the English mu

ffi

n with lettuce

Microwave -Microwave just until cheese melts, about 10 seconds

Microwave-Microwave the ramekin cup uncovered on high for 30 seconds

Cut -Cut the English mu

ffi

n into two pieces with a knife

sprinkle -sprinkle 1 tablespoon of cheese on cup

Top -Top cup with 1 tablespoon of salsa

Microwave -Continue to Microwave for 15-30 more seconds or until the egg is almost set

stir-stir the ramekin cup

replace -replace the top of the English mu

ffi

n

END

(b)

START

Microwave-Microwave on high for 90 seconds until the egg is cooked through

add-add bread pieces to the egg mixture in the mug, pressing the bread down into the egg

stir-stir the mug

Cut or tear-Cut or tear 1 slices of bread into bite-size pieces

Sprinkle-Sprinkle 1/4 teaspoon cinnamon over the egg

Add-Add 1/4 teaspoon vanilla extract to the mug

whisk-In the mug, whisk one egg with a fork until well blended

Roll-Roll the butter around in the mug to coat it

melt-In a large mug, melt 1 tablespoon of softened butter in the microwave for about 30 seconds

cut-cut the contents on plate, and serve

Put-Put the mug's contents on a plate

END

(a)

START

Microwave-Microwave on high for 90 seconds until the egg is cooked through

add-add bread pieces to the egg mixture in the mug, pressing the bread down into the egg

stir-stir the mug

Cut or tear-Cut or tear 1 slices of bread into bite-size pieces

Sprinkle-Sprinkle 1/4 teaspoon cinnamon over the egg

Add-Add 1/4 teaspoon vanilla extract to the mug

whisk-In the mug, whisk one egg with a fork until well blended

Roll-Roll the butter around in the mug to coat it

melt-In a large mug, melt 1 tablespoon of softened butter in the microwave for about 30 seconds

cut-cut the contents on plate, and serve

Put-Put the mug's contents on a plate

END

(b)

START

Whisk-Whisk bat er until no lumps remain

Measure and add-2 tbsp water to the bowl

Measure and add-Measure and add 2 tsp vegetable oil to the bowl

Measure and add-1/4 tsp vanil a extract to the bowl

Whisk-Whisk to combine mixture of

fl

our, sugar and baking powder in the bowl

Measure and add-1.5 tbsp sugar to the mixing bowl

Measure and add-Measure and add 2 tbsp

fl

our to the mixing bowl

Measure and add-1/4 tsp baking powder to the bowl

Measure and add-a pinch of salt to the mixing bowl

remove-then careful y remove the paper liner

Al ow-Al ow to cool until it is no longer hot to the touch

Invert-Invert the mug to release the cake onto a plate

Microwave-Microwave the mug and bat er on high power for 60 seconds. Check if the cake is done by inserting and toothpick into the center of the cake and then removing it. If wet bat er clings to the toothpick, microwave for an additional 5 seconds. If the toothpick comes out clean, continue

Pour-Pour bat er into prepared mug

Scoop-While the cake is cooling, prepare to pipe the frosting. Scoop 4 spoonfuls of chocolate frosting into a zip-top bag

Set-Set aside the lined mug

Place-Place the paper cupcake liner inside the mug

Squeeze-Squeeze the frosting through the opening to apply smal  dol ops of frosting to the plate in a circle around the base of the cake

cut-Use scissors to cut one corner from the bag to create a smal  opening 1/4 inch in diameter

seal-seal zip top bag, removing as much air as possible

END

(a)

START

Whisk-Whisk bat er until no lumps remain

Measure and add-2 tbsp water to the bowl

Measure and add-Measure and add 2 tsp vegetable oil to the bowl

Measure and add-1/4 tsp vanil a extract to the bowl

Place-Place the paper cupcake liner inside the mug

Whisk-Whisk to combine mixture of

fl

our, sugar and baking powder in the bowl

Measure and add-1.5 tbsp sugar to the mixing bowl

Measure and add-Measure and add 2 tbsp

fl

our to the mixing bowl

Measure and add-1/4 tsp baking powder to the bowl

Measure and add-a pinch of salt to the mixing bowl

remove-then careful y remove the paper liner

Al ow-Al ow to cool until it is no longer hot to the touch

Invert-Invert the mug to release the cake onto a plate

Microwave-Microwave the mug and bat er on high power for 60 seconds. Check if the cake is done by inserting and toothpick into the center of the cake and then removing it. If wet bat er clings to the toothpick, microwave for an additional 5 seconds. If the toothpick comes out clean, continue

Pour-Pour bat er into prepared mug

Scoop-While the cake is cooling, prepare to pipe the frosting. Scoop 4 spoonfuls of chocolate frosting into a zip-top bag

Set-Set aside the lined mug

Squeeze-Squeeze the frosting through the opening to apply smal  dol ops of frosting to the plate in a circle around the base of the cake

cut-Use scissors to cut one corner from the bag to create a smal  opening 1/4 inch in diameter

seal-seal zip top bag, removing as much air as possible

END

(b)

START

Cut-Cut 1/4 block or 3 ounces of fresh tofu into large cubes (about 1 in x 1 in)

Turn-Turn on the heat to medium

add-add the tofu cubes to the pan

add-add 1/4 tsp salt to the pan

Add-Add 1 tablespoon of olive oil to a non-stick pan

drizzle-drizzle 1 tablespoon soy sauce (watch for spitting) on the pan

fl

y remove from the heat again

cook-cook until tofu turns brown

Return-Return the heat to medium

Flip-Flip the tofu with tongs

drizzle-drizzle with the 1 tablespoons sesame oil on the pan

fl

y remove the pan from the heat to reduce spitting

Cook-Cook 5 to 6 minutes until tofu cubes are lightly browned on the bottom

fl

ip-

fl

ip tofu on the pan

cook-cook pan for 2 minutes

Return-Return to low heat

Transfer-Transfer to a serving dish

cook-cook pan for 2 minutes until the colour is darkened

END

(a)

START

Cut-Cut 1/4 block or 3 ounces of fresh tofu into large cubes (about 1 in x 1 in)

Turn-Turn on the heat to medium

drizzle-drizzle 1 tablespoon soy sauce (watch for spitting) on the pan

fl

y remove from the heat again

cook-cook until tofu turns brown

Return-Return the heat to medium

Flip-Flip the tofu with tongs

drizzle-drizzle with the 1 tablespoons sesame oil on the pan

add-add the tofu cubes to the pan

Add-Add 1 tablespoon of olive oil to a non-stick pan

fl

y remove the pan from the heat to reduce spitting

Cook-Cook 5 to 6 minutes until tofu cubes are lightly browned on the bottom

add-add 1/4 tsp salt to the pan

fl

ip-

fl

ip tofu on the pan

cook-cook pan for 2 minutes

Return-Return to low heat

Transfer-Transfer to a serving dish

cook-cook pan for 2 minutes until the colour is darkened

END

(b)

START

Trim-Trim the ends of the tortilla roll with the butter knife, leaving 1/2 inch margin between the last toothpick and the end of the roll

Secure-Secure the rolled tortilla by inserting 5 toothpicks about 1 inch apart

Roll-Roll the tortilla from one end to the other into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps, but not so tight that the

fi

lling leaks

Clean-Clean the knife by wiping with a paper towel

slicing-Continue slicing with

fl

oss to create 1 more pinwheel

pull-pull the

fl

oss ends in opposite directions to slice

Cross-Cross the

fl

oss's two ends over the tortilla roll's top

Place-Place the

fl

oss halfway between toothpicks

Slide-Slide

fl

oss under the tortilla, perpendicular to the length of the roll

Spread-Spread jelly over the nut butter

scoop-Use the knife to scoop jelly from the jar

Clean-Clean the knife by wiping it with a paper towel

Spread-Spread nut butter onto the tortilla, leaving 1/2-inch uncovered at the edges

scoop-Use a butter knife to scoop nut butter from the jar

Place-Place the pinwheels on a plate

Place-Place 8-inch

fl

our tortilla on cutting board

END

(a)

START

Trim-Trim the ends of the tortilla roll with the butter knife, leaving 1/2 inch margin between the last toothpick and the end of the roll

Secure-Secure the rolled tortilla by inserting 5 toothpicks about 1 inch apart

Roll-Roll the tortilla from one end to the other into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps, but not so tight that the

fi

lling leaks

Clean-Clean the knife by wiping with a paper towel

slicing-Continue slicing with

fl

oss to create 1 more pinwheel

pull-pull the

fl

oss ends in opposite directions to slice

Cross-Cross the

fl

oss's two ends over the tortilla roll's top

Place-Place the

fl

oss halfway between toothpicks

Slide-Slide

fl

oss under the tortilla, perpendicular to the length of the roll

Spread-Spread jelly over the nut butter

scoop-Use the knife to scoop jelly from the jar

Clean-Clean the knife by wiping it with a paper towel

Spread-Spread nut butter onto the tortilla, leaving 1/2-inch uncovered at the edges

scoop-Use a butter knife to scoop nut butter from the jar

Place-Place the pinwheels on a plate

Place-Place 8-inch

fl

our tortilla on cutting board

END

(b)

START

place-place avocado slices on each leaf

Lay-Lay out 2 large lettuce leaves

cut-cut avocado into thin slices

Take-Take 1 ripe avocado

Season-season 1/4 tsp pepper on the bowl

Mix-Mix the contents of the bowl

Add-1/4 cup mayonnaise to the bowl

Add-add chopped scallion to the bowl

Add-Add 1 can drained tuna to the bowl

Add-1 tsp Sriracha sauce to the bowl

Chop-Chop 1 scallion

drain-drain excess water from can

Open-Open a can of tuna

Roll-Roll up the lettuce wraps

top-top lettuce leaves with the tuna mixture

Season-Season bowl with 1/4 tsp salt

secure-secure the wrap with a toothpick

END

(a)

START

place-place avocado slices on each leaf

Add-1/4 cup mayonnaise to the bowl

drain-drain excess water from can

Lay-Lay out 2 large lettuce leaves

Chop-Chop 1 scallion

cut-cut avocado into thin slices

Open-Open a can of tuna

Take-Take 1 ripe avocado

Season-season 1/4 tsp pepper on the bowl

Mix-Mix the contents of the bowl

Add-add chopped scallion to the bowl

Add-Add 1 can drained tuna to the bowl

Add-1 tsp Sriracha sauce to the bowl

Roll-Roll up the lettuce wraps

top-top lettuce leaves with the tuna mixture

Season-Season bowl with 1/4 tsp salt

secure-secure the wrap with a toothpick

END

START

Heat-Heat the contents of the mug for 1 minute and serve

Mix-Mix the contents of the mug

Add-Add 1/5 teaspoon cinnamon to the mug

Add-Add 1 teaspoon of white sugar to the mug

Add-Add 2 pieces of chocolate to the mug

Microwave-Microwave the contents of the mug for 1 minute

Fill-Fill a microwave-safe mug with skimmed milk

END

(a)

START

Heat-Heat the contents of the mug for 1 minute and serve

Mix-Mix the contents of the mug

Add-Add 1/5 teaspoon cinnamon to the mug

Add-Add 1 teaspoon of white sugar to the mug

Add-Add 2 pieces of chocolate to the mug

Microwave-Microwave the contents of the mug for 1 minute

Fill-Fill a microwave-safe mug with skimmed milk

END

(b)

START

Slice-Slice one tomato into about 1/2 inch thick slices

dry-gently dry it with a paper/tea towel

Rinse-Rinse a tomato

Place-Place the thick slices of tomatoes on a platter, ensuring they only make a single layer

Add-Add a drizzle of extra-virgin olive oil, about 1 tablespoon, over the entire platter

Season-Season the tomato slices with salt

Sprinkle-Sprinkle mozzarel a cheese on top of the tomato throughout the platter

Season-Season platter with 1/4 teaspoon black pepper

Garnish-Garnish platter with italian seasoning

END

(a)

START

Slice-Slice one tomato into about 1/2 inch thick slices

dry-gently dry it with a paper/tea towel

Rinse-Rinse a tomato

Place-Place the thick slices of tomatoes on a platter, ensuring they only make a single layer

Add-Add a drizzle of extra-virgin olive oil, about 1 tablespoon, over the entire platter

Season-Season the tomato slices with salt

Sprinkle-Sprinkle mozzarel a cheese on top of the tomato throughout the platter

Season-Season platter with 1/4 teaspoon black pepper

Garnish-Garnish platter with italian seasoning

END

(b)

START

pat-pat rinsed mushrooms dry with a paper towel

Rinse-Rinse 3 mushrooms under cold water

cook-cook the pan, often stirring, for 1 minute

Add-Add chopped shallot to the pan

cook-cook for 3-5 minutes, stirring often, until mushrooms start to soften and brown

add-Once the pan is hot, add the mushrooms

Heat-Heat 1 tbsp olive oil in a large skillet over medium-high heat

Slice-Slice the mushrooms

mince-mince garlic cloves

Chop-Chop 1 shallot

Pull-Pull out mushroom stems

Transfer-Transfer the contents of the pan to a serving dish

Add-1/4 tbsp balsamic vinegar to the pan

Add-Add 2 cloves of minced garlic to the pan

Season-pepper on pan to taste

Season-Season pan with salt

END

(a)

START

pat-pat rinsed mushrooms dry with a paper towel

Rinse-Rinse 3 mushrooms under cold water

cook-cook the pan, often stirring, for 1 minute

Add-Add chopped shallot to the pan

cook-cook for 3-5 minutes, stirring often, until mushrooms start to soften and brown

add-Once the pan is hot, add the mushrooms

Heat-Heat 1 tbsp olive oil in a large skillet over medium-high heat

Slice-Slice the mushrooms

mince-mince garlic cloves

Chop-Chop 1 shallot

Pull-Pull out mushroom stems

Transfer-Transfer the contents of the pan to a serving dish

Season-Season pan with salt

Season-pepper on pan to taste

Add-1/4 tbsp balsamic vinegar to the pan

Add-Add 2 cloves of minced garlic to the pan

END

(b)

START

Mix-Mix in the

fl

avour packet to the bowl

Let-Let the noodles sit for about 1 minute after the microwave stops

Add-Add basil to the bowl

Add-Add chopped cilantro to the bowl

Microwave-Microwave the ramen for 4 minutes

Cover-Cover with a lid (or paper towel) to prevent splattering

cover-cover the noodles with water

Stir-Stir noodles with a spoon or fork until the

fl

avouring dissolves

Remove-Remove the noodles from the package(Break Noodles / Keep them as a block)

Add-Add the noodles to the bowl

Put-Put all the Vegetables in a microwave-safe bowl

slice-slice 1/4 medium onion into pieces

Chop-Chop 1 garlic clove on a cutting board

END

(a)

START

Mix-Mix in the

fl

avour packet to the bowl

Let-Let the noodles sit for about 1 minute after the microwave stops

Add-Add basil to the bowl

Add-Add chopped cilantro to the bowl

Microwave-Microwave the ramen for 4 minutes

Cover-Cover with a lid (or paper towel) to prevent splattering

cover-cover the noodles with water

Stir-Stir noodles with a spoon or fork until the

fl

avouring dissolves

Remove-Remove the noodles from the package(Break Noodles / Keep them as a block)

Add-Add the noodles to the bowl

Put-Put all the Vegetables in a microwave-safe bowl

slice-slice 1/4 medium onion into pieces

Chop-Chop 1 garlic clove on a cutting board

END

(b)

START

add-add lime juice to the bowl

Extract-Extract lime juice from 1/3 lime

Microwave-Microwave the corn for 3 more minutes

stir-then stir the bowl

Add-1 teaspoon of pepper powder to the bowl

Add-Add 1 teaspoon of softened butter

Microwave-Microwave the corn for 2 minutes

Measure-Measure 2 cups of frozen corn

Add-Add the corn into a microwave-safe bowl

Thaw-Thaw the frozen corn by putting it in a sieve and running it under cold water

Mix-Mix the contents of the bowl well

Add-Add 1 teaspoon salt to the bowl

END

(a)

START

add-add lime juice to the bowl

Extract-Extract lime juice from 1/3 lime

Microwave-Microwave the corn for 3 more minutes

stir-then stir the bowl

Add-1 teaspoon of pepper powder to the bowl

Add-Add 1 teaspoon of softened butter

Measure-Measure 2 cups of frozen corn

Microwave-Microwave the corn for 2 minutes

Add-Add the corn into a microwave-safe bowl

Thaw-Thaw the frozen corn by putting it in a sieve and running it under cold water

Mix-Mix the contents of the bowl well

Add-Add 1 teaspoon salt to the bowl

END

(b)

START

Chop-Chop 1 tsp cilantro

Whisk-Whisk the egg mixture in the bowl

add-add 1/3 tsp salt to the bowl

Crack-Crack one egg in the bowl

add-add 1 tbsp milk to the bowl

Add-Add garlic to the pan

Saute-Saute the onions on medium heat until they are soft and translucent

add-add 1/3 tsp salt to the pan

add-add chopped onions to the pan

Heat-Heat 2 tbsp oil in a heavy-bottomed or nonstick pan on medium heat

Chop-Chop 1 green chilli

Chop-Chop 1/4 medium onion

Chop-Chop 1/4 tomato

Mince-Mince peeled garlic cloves

Add-Add chilli to the pan

Cook-Cook for 1 minute, mixing everything

Add-Add 1/8 tsp of turmeric to the pan

Cook-Cook covered for 1 minute or until the tomatoes soften

Add-Add tomatoes to the pan

mixing-Keep mixing with a spatula for 3 minutes or until the eggs are almost cooked

pour-Slowly pour the whisked eggs into the pan

Garnish-Garnish with 1 tbsp chopped cilantro and serve

END

(a)

START

Chop-Chop 1 tsp cilantro

Whisk-Whisk the egg mixture in the bowl

Chop-Chop 1 green chilli

add-add 1/3 tsp salt to the bowl

add-add 1 tbsp milk to the bowl

Crack-Crack one egg in the bowl

Add-Add garlic to the pan

Saute-Saute the onions on medium heat until they are soft and translucent

add-add 1/3 tsp salt to the pan

add-add chopped onions to the pan

Heat-Heat 2 tbsp oil in a heavy-bottomed or nonstick pan on medium heat

Chop-Chop 1/4 medium onion

Chop-Chop 1/4 tomato

Mince-Mince peeled garlic cloves

Add-Add chilli to the pan

Cook-Cook for 1 minute, mixing everything

Add-Add 1/8 tsp of turmeric to the pan

Cook-Cook covered for 1 minute or until the tomatoes soften

Add-Add tomatoes to the pan

mixing-Keep mixing with a spatula for 3 minutes or until the eggs are almost cooked

pour-Slowly pour the whisked eggs into the pan

Garnish-Garnish with 1 tbsp chopped cilantro and serve

END

(b)

START

Add-Add 1/4 tsp mustard to the pan

Heat-Heat 3 tbsp oil in a pan over medium heat

mince-mince the garlic

puree-puree tomatoes without any water in a blender/mixer

Chop-Chop tomato roughly (anysize chunks are

fi

ne)

Take-Take 1 tomato

Add-1/2 tsp cumin seeds to the pan

Add-Add tomato puree to the pan

mix-mix well contents of the pan

Add-Add 2 tbsp red chili powder to the pan

Saute-Saute the garlic for 2-3 minutes

Lower-Lower the heat

Take-Take the pan o

ff

the heat

simmer-Allow the mixture to simmer over low heat for 5 minutes or until the mixture becomes thick

Mix-Mix well tomato puree with contents in the pan

Add-1/2 tsp salt to the pan

add-When mustard and cumin seeds begin to sizzle, add minced garlic

Transfer-Transfer it to a serving bowl

END

(a)

START

Add-Add 1/4 tsp mustard to the pan

Heat-Heat 3 tbsp oil in a pan over medium heat

mince-mince the garlic

puree-puree tomatoes without any water in a blender/mixer

Chop-Chop tomato roughly (anysize chunks are

fi

ne)

Take-Take 1 tomato

Add-1/2 tsp cumin seeds to the pan

Add-Add tomato puree to the pan

mix-mix well contents of the pan

Add-Add 2 tbsp red chili powder to the pan

Saute-Saute the garlic for 2-3 minutes

Lower-Lower the heat

Take-Take the pan o

ff

the heat

simmer-Allow the mixture to simmer over low heat for 5 minutes or until the mixture becomes thick

Mix-Mix well tomato puree with contents in the pan

Add-1/2 tsp salt to the pan

add-When mustard and cumin seeds begin to sizzle, add minced garlic

Transfer-Transfer it to a serving bowl

END

(b)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: we introduce a novel approach to build task graphs using a differentiable loss function. The usefulness of the learned representation is assessed on three datasets on the tasks of task graph generation, and online mistake detection. Technical descriptions are reported in Section 3 and experiments are reported in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: the limitations are discussed in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: the paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 3 presents the description of our differentiable loss function and describes the proposed models. Experiments in Section 4 and the supplementary material contain descriptions for reproducibility. We release a preliminary version of our code in the supplementary and we will publicly release the final code to replicate the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We use publicly available datasets and provide details on data splits in C.5. We share a preliminary version of our code in the supplementary material and plan to release the final code to replicate all experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We report implementation details and hyper-parameters in Section C of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Results in Table 1 report the bounds of confidence intervals computed at a 90% confidence level. These intervals are derived using bootstrapping, where we resample the results from 5 runs with different random initializations to estimate the distribution of the performance metrics. We report the average performance ¯ x across the 5 runs with the corresponding standard deviation σ . The confidence bounds are obtained by repeatedly resampling the data and calculating the desired percentiles from the empirical distribution. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report details on computational requirements to run the experiments in Section C.11 of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We used public datasets which have been collected by the related authors following the recommendations provided by their institutions. The datasets have not been deprecated. We do not re-distribute any of the used data.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss societal impact in Section D of the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have referenced the data used for the experiments and followed the related licenses. Licenses are available at the respective authors' pages.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduce a new loss function for task graph generation and two new models (see Section 3).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to learning task graphs from procedural activities observed in egocentric videos. Task graphs represent the partial ordering of key-steps needed to complete a task and are crucial for developing intelligent agents that can assist users. The proposed method utilizes direct maximum likelihood optimization of edge weights, enabling gradient-based learning that can be integrated into neural network architectures. The study demonstrates significant improvements in task graph prediction accuracy and online mistake detection using datasets like CaptainCook4D, Assembly 101, and EPIC-Tent.\n\nThe main motivation behind this work is twofold. Firstly, it aims to enhance the ability of intelligent systems to assist users in performing complex tasks by automatically learning flexible and accurate task graph representations from video observations, rather than relying on manually crafted procedures. Secondly, the authors seek to explicitly model task graphs to provide a more interpretable and human-understandable representation of procedural activities.\n\nThe authors introduce two primary approaches for task graph learning: Direct Optimization (DO) and Task Graph Transformer (TGT). The DO method uses the Task Graph Maximum Likelihood (TGML) loss to directly optimize the adjacency matrix representing the task graph. This optimization is performed using gradient descent, making the process straightforward and effective. On the other hand, the TGT model employs a transformer encoder to process key-step text or video embeddings and predict the adjacency matrix from these embeddings. This model includes a regularization loss to maintain distinctiveness among embeddings, ensuring accurate and meaningful task graph generation.\n\nSTRENGTHS:\nThe main contributions of the paper are as follows:\n\n1. **Approach:** This paper presents an new error detection approach via learning task graphs by directly optimizing a maximum likelihood objective. This approach enables gradient-based task graph learning, allowing task graph methods to be applied to the problem of mistake detection in programs.\n2. **Performance:** The proposed methods achieve superior performance in task graph generation and mistake detection compared to existing methods.\n3. **Code Availability**: The availability of code for replication enhances the study's transparency and usability.\n\nWEAKNESSES:\n1. **Dependence on Labeled Key-Step Sequences:** The method proposed in this work relies on action recognition models or Ground-truth Key-Step Labels. On the one hand, this will bring additional performance overhead, and on the other hand, relying on other action recognition models may introduce noise when errors occur during recognition. It's uncertain whether it will also affect the results of mistake detection. More discussions on these issues are needed.\n\n2. **Limitation in Versatility and Scalability:** In this approach, a new model (task graph) needs to be trained for each new sequence of actions in order to achieve mistake detection. This method to some extent limits the versatility and scalability of the model, as retraining is required for each new task. Additionally, it seems that the proposed method has no versatility on order-unrelated mistakes, such as technique/measurement errors, which are highlighted in latest datasets (e.g., HoloAssist-ICCV23[1], CaptainCook4D-ICML23[2], EgoPER-CVPR24[3])\n\n3. **Insufficient Qualitative Analysis**: The paper only provides one qualitative result on generated task graph in Figure 5. However, there is no qualitative analysis of the generated task graphs for detecting the mistakes. Also, it is suggested to provide discussions on the corner cases that may be challenging for the proposed method, which would bring insightful observations for the error detection community.\n\n[1] Wang, Xin, et al. \"Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[2] Peddi, Rohith, et al. \"CaptainCook4D: A dataset for understanding errors in procedural activities.\" arXiv preprint arXiv:2312.14556 (2023).\n[3] Lee, Shih-Po, et al. \"Error detection in egocentric procedural task videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nSuggestions for authors: \nMy main concerns are about:\n- The versatility of the proposed method for various types of mistakes.\n- The detailed qualitative analysis on generated graphs when used to detect mistakes.\nPlease do read the weakness carefully, I would consider to raise my rating if my concerns are well addressed.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a learning-based task graph generation for procedural actions and mistake detection. Compared to previous approaches which usually consider natural language based descriptions, the proposed approach aims to use backward propagation to adaptively optimize procedural action sequences. To the reviewer, the introduced approach is much more flexible compared to other \"hard-coded\" language-based descriptions. In summary, this papers introduced a simple yet inspiring approach for graph generation.\n\nSTRENGTHS:\n1. The proposed optimization based graph generation seems inspiring and novel. Unlike other existing hand-crafted methods, learning-based graph generation can be end-to-end trained with task model parameters. This has the potentials in significantly boosting model performances, since the graph is no longer hard-coded, which makes models able to automatically search for the optimal correlations.\n2. The methodology is mostly clear and easy to follow. The authors well demonstrate how to establish and end-to-end optimize trainable graphs. \n3. The experimental results are quite promising in the experiment sections, and the improvements are non-trivial and encouraging.\n\nWEAKNESSES:\n1. The explanations of contrastive loss are somehow hard to follow. For typical contrastive learning, to push away or to pull close the distances always require positive/negative and anchor samples. The authors are expected to provide more detailed discussions to help readers understand.\n2. In line 140, the authors make an assumption that no repetitive actions exists in sequences, without any further discussions. It is suggested that authors could provide more explanations on why they have this assumption and what impact this assumption causes to the proposed approach.\n3. It is suggested that how the video and text embeddings are sent to the overall framework should be clearly demonstrated in the Figure 3. In the current submitted version, it is somehow confusing to the readers how the video features are considered into the training process.\n\nQUESTIONS:\nPlease refer to weakness for details.\n\nLIMITATIONS:\nThe limitations that the authors introduced are reasonable considering current submission, and they do not have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a differentiable loss function based on the maximum likelihood of generating task graphs from video features. The two proposed methods (DO and TGT) show strong performance, verifying their effectiveness in generating task graphs. Moreover, the predicted graphs are evaluated on downstream tasks to demonstrate the influence of explicit graph representations for general video understanding.\n\nSTRENGTHS:\nIn general, I like the paper from every perspective. It advances the progress of using task graphs for video understanding.\n\n- The paper is well-written and easy to understand.\n- The literature survey is comprehensive enough, which helps readers easily understand the motivation and the problem setting.\n- The experimental comparison is extensive enough, including a rich set of recent baselines that vary from different types of task graph generation.\n- The two proposed methods show strong performance on the evaluated datasets, verifying the effectiveness of the proposed differentiable task graph generation.  \n- The generated explicit task graph representation is also evaluated on the downstream tasks.\n- The discussed limitations, e.g., the assumption of availability of key-step sequences, are insightful and are critical to the future development of task graphs for video understanding.\n\nWEAKNESSES:\n- L140 describes that the authors separate sequences if the key-step repetition is available. I am wondering if this will limit the applications for some daily scenarios where a key step needs to be conducted many times, e.g., spreading peanut butter multiple times.\n\n- In L329, it would be nice to have a reference of those action recognition works that assume key-step sequences are available.\n\nQUESTIONS:\nSee weaknesses. My concern is the potential limitation for applications to long-form daily-life procedure activities where repetition is crucial.\n\nLIMITATIONS:\nYes, the authors have discussed the limitations of this work, which is insightful and can facilitate future works of task graphs for video understanding.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}