OPEL: Optimal Transport Guided ProcedurE Learning

Authors: shafayet  sayeed
soumyadeep  chowdhury
kaushik  chandra
  roy
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:38:51

Abstract: Procedure learning refers to the task of identifying the key-steps and determining their logical order, given several videos of the same task. For both third-person and first-person (egocentric) videos, state-of-the-art (SOTA) methods aim at finding correspondences across videos in time to accomplish procedure learning. However, to establish temporal relationships within the sequences, these methods often rely on frame-to-frame mapping, or assume monotonic alignment of video pairs, leading to sub-optimal results. To this end, we propose to treat the video frames as  samples from an unknown distribution, enabling us to frame their distance calculation as an optimal transport (OT) problem. Notably, the OT-based formulation allows us to relax the previously mentioned assumptions. To further improve performance, we enhance the OT formulation by introducing two regularization terms. The first,  inverse difference moment regularization, promotes transportation between instances that are homogeneous in the embedding space as well as being temporally closer. The second, regularization based on the KL-divergence with an exponentially decaying prior smooths the alignment while enforcing conformity to the optimality (alignment obtained from vanilla OT optimization) and temporal priors. The resultant optimal transport guided procedure learning framework (`OPEL') significantly outperforms the SOTA on benchmark datasets. Specifically, we achieve 22.4\% (IoU) and 26.9\% (F1) average improvement compared to the current SOTA on large scale egocentric benchmark, EgoProceL. Furthermore, for the third person benchmarks (ProCeL and CrossTask), the proposed approach obtains 46.2\% (F1) average enhancement over SOTA.

Main paper: 
 introduction: 
 The development of autonomous agents capable of reliably replicating human actions to accomplish certain end goals presents significant challenges. Traditional approaches would necessitate hardcoding tedious explicit instructions for each sub-task of the process (thus difficult to scale and generalize). A more efficient solution would involve the agent learning directly from observing multiple demonstrations of the assembly, without the need of any label. This motivates us to explore unsupervised procedure learning from videos. In the context of this work, procedure learning (PL) is conceptualized as the process of determining the key-steps and their correct sequential order to accomplish an overall task, as demonstrated across multiple video demonstrations [1, 2, 3] .PL analyzes multiple videos of a task as illustrated in Figure 1 , in contrast to action-based tasks [4] , which focus on a single video. The single-video approach is inapplicable to the identification of repetitive key-steps across videos. Moreover, action-based tasks typically neglect the sequencing of events, crucial for discerning an overall expected procedure composed of the sub-tasks. For instance, they fail to capture variations in the sequence of key-steps between videos V1 and V2 (Fig. 1 ). Other research efforts in video understanding that employ instructional videos include procedure planning [5] , verifying sequences of procedures [6] , and summarizing instructional content [7] . Additionally, unlike video alignment tasks [8] , PL specifically aims to localize these essential steps within videos.Figure 1 : Key-steps required to prepare a brownie [17] . The sequences showcase temporal variations and corresponding key-step alignment challenges, namely (i) background frames (depicted as gray blocks), (ii) non-monotonic frames. OPEL aims to learn an embedding space where corresponding key-steps have similar embeddings while tackling the above challenges.Much of the research on PL till now has been performed within the frameworks of supervised [9, 10, 11] and weakly supervised learning [12, 13, 14] . In a supervised setting, the reliance on per-frame annotations demands extensive manual labor. Conversely, weakly supervised methods involve using either ordered or unordered lists of key-steps. The generation of these lists requires either direct observation of the videos or specific heuristics, both of which pose significant scalability challenges [3] . Consequently, recent studies [1, 8] have shifted focus towards self-supervised learning, which do not require frame-wise labeling. Such a learning paradigm leverages the structured nature of accomplishing a complex task, which typically unfold in a predictable sequence of steps. For example, the act of preparing a "brownie" might involve breaking an egg, adding water, oil, mixing the contents, and then baking in the oven. The alignment of video frames is commonly performed in a monotonic manner [15] , which presupposes a consistent order of actions across sequences. However, real-world sequences frequently deviate from this pattern, exhibiting temporal non-uniformities as depicted in Figure 1 . These deviations can be categorized as follows: (i) background frames: frames irrelevant to the primary activity and should thus be excluded from alignment; (ii) redundant frames: these frames appear only in one sequence but not in others and do not contribute to the task; (iii) non-monotonic frames: these frames are characterized by a non-monotonic sequence of actions. Such frames challenge the assumption of monotonic progression and highlight the complexity of real-world activities. State-of-the-art (SOTA) methods adopt custom approaches to counter these irregularities such as removing background frames from processing [2] , using extra information (e.g. gaze, depth) [16] , or simply ignore them leading to suboptimal results [1] .To address the limitations of previous approaches, we relax the strict assumptions about the temporal sequence of actions and introduce a novel PL framework designed to learn temporal correspondences across videos. By treating instances of the sequences as samples from an unknown distribution, we formulate the task of computing the distance between them as an optimal transport (OT) [18] problem. The differentiable OT loss facilitates the alignment of non-monotonic sequences through frame-wise matching based on individual frame features. However, it typically overlooks the temporal smoothness and the inherent ordering relationships within the videos. To overcome this deficiency, we integrate two priors into the transportation matrix. First, the optimality prior favors the positions dictated by the OT, whereas the temporal prior discourages transport between temporally distant frames. Both these priors are modeled using a Laplace distribution with exponentially decaying probability from the corresponding centers. We introduce an additional virtual frame into the OT matrix to address background and redundant frames. Furthermore, to avoid the common issue of converging to trivial solutions in temporal video alignment [15] , we employ a novel inter-video contrastive loss, which acts as a regularizer. Finally, the sub-tasks of each video are clustered in the embedding space using graphcut segmentation [19] . The overall framework, termed optimal transport guided procedure learning ('OPEL'), achieves SOTA results on both the ego and exocentric benchmarks. To summarize, our main contributions are-• We propose a novel optimal transport based procedure learning framework that aligns frames with similar semantics together in an embedding space. ensuring optimality, separability, and completeness. Assume, the inputs are two sequences of video frames, P = [p 1 , p 2 , . . . , p N ] and Q = [q 1 , q 2 , . . . , q M ]. We pass these through a deep encoder network (as illustrated in Fig. 2 β j δ yj , where δ x denotes the Dirac mass at x, and α and β are the weights for the distributions f and g, respectively. Since there is no justification for assigning greater importance to one frame over another, initially we set α i = 1N and β j = 1 M for all i, j, leading to a feasible set of weight matrices defined as the transportation polytope [48] , U (α, β):= {T ∈ R N ×M + : T 1 M = α, T ⊤ 1 N = β}.Here, t ij can be interpreted to be proportional to the probability that x i will be aligned to y j . We start by computing the pairwise Euclidean distances between embedding vectors, d(x i , y j ) = ∥x i -y j ∥ to form the N × M distance matrix, D. The cost of transporting mass from f to g with a transport plan T is quantified by the Frobenius inner product ⟨T , D⟩. Thus, the Wasserstein distance raised to the power p is: W p p (f, g) = l W (α, β, D) = min T ∈U (α,β) ⟨T , D⟩. We only consider p = 1, and drop p henceforth. To simplify the above optimization and make training feasible, Cuturi [48] introduced an entropy regularization, leading to the Sinkhorn distance,EQUATIONwhere h(T ) = -N i=1 M j=1 t ij log t ij denotes the entropy of T , and λ is the regularization parameter. The optimal solution for Eqn. (1) has the form [48] , T λ = diag(κ 1 ) exp(-λD)diag(κ 2 ), where exp(-λD) is the element-wise exponential of the matrix -λD, and κ 1 ∈ R N and κ 2 ∈ R M are the non-negative left and right scaling vectors to be obtained by the Sinkhorn fixed point iterations.Regularization with Priors. The above formulation minimizes the cost of aligning two sequences, however it totally neglects the temporal ordering relationships inherent in videos, failing to leverage the temporal consistency. Typically, the alignment of multiple videos depicting the same activity should constrain the temporal position of one sequence to correspond closely with adjacent temporal positions of another sequence. Perfect alignment would render the transport matrix T diagonal, but this strict requirement is impractical for real-world applications. As illustrated in Fig. 2 (B), variations such as earlier commencement of activities, differing action speeds, or non-monotonic sequences complicate alignment. To address these challenges and achieve optimal alignment while accounting for temporal variations, we introduce two priors into the OT framework. Essentially, there are 2 factors in play -(i) optimality which tries to find the best match between frames irrespective of their temporal distance (which may result in temporally incoherent alignment), and (ii) the temporal factor which promotes transport between nearby frames only without considering their feature matching. We hypothesize that the optimal solution requires striking a balance between both, and thus propose to enhance the OT formulation by incorporating two specific priors addressing the above factors [46] .The first prior, termed the 'Optimality Prior', is introduced to effectively manage non-monotonic sequences. This prior leverages the transport matrix T as delineated in Eqn. (1) , which provides a preliminary indication of alignment between two video sequences. This matrix adapts dynamically to reflect the temporal variations observed across the sequences. Our approach uses this dynamic behavior to establish the optimality prior. We want the point representing the most likely alignment according to T to have the highest likelihood, while the assignment probability decays along any perpendicular direction from this center. Specifically, we model this as a Laplace distribution,EQUATIONrepresents the average distance from (i, j) to the frame locations (i, j o ) and (i o , j) that correspond to the optimal alignment as indicated by the transport matrix, and b is a scale parameter. Motivated by [46] , we incorporate a second prior, termed the 'Temporal Prior', which promotes alignment of one sequence with elements in proximal temporal positions of the other sequence, thereby preserving the overall temporal structure and maintaining consistency in action order. This prior results in an assignment matrix characterized by peak values along the diagonal, with values diminishing perpendicular to the diagonal. Again, this scenario is modeled using a two-dimensional Laplace distribution, where the distribution along any line perpendicular to the diagonal is exponentially decaying, centered along the diagonal itself:EQUATIONis the distance from (i, j) to the diagonal. Inspired by [46] , we merge these priors as,EQUATIONwhere, ϕ serves as a dynamic weight, initially set to 1.0, and progressively reduced to 0.5 during training. This gradual adjustment of ϕ allows the model to adaptively improve its alignment based on the increasing fidelity of the OT predictions. Optimal alignment based on these priors is pictorially depicted in Fig. 2(C, D ). Note, the 1-dimensional alignment in Fig. 2(C ) is for demonstration only, our actual implementation is based on 2-dimensional distributional priors as shown in Fig. 2 Background and Redundant Frames. To effectively manage background and redundant frames, we integrate an additional 'virtual frame' within the transport matrix, following [46] . This serves as a placeholder for aligning any frame that do not match with the primary sequence, and allows OPEL to explicitly assign these non-contributing frames to the virtual frame, as shown in Fig. 4(B) . The augmented transport matrix, now denoted as T ∈ R (N +1)×(M +1) , includes an extra row and column to accommodate the virtual frame. Note, if the likelihood of a frame aligning with any salient frame falls below a predefined threshold, ζ, we assign that frame to the virtual frame.Training Methodology. While the above formulation sounds promising, devising a differentiable framework to leverage these during training is pivotal. To that effect, following [46] , we define 2 terms to effectively regularize T . To capture the essence of Q o , T needs to be structured to highlight prominent values at locations corresponding to the most probable alignments,EQUATIONSimilarly, to conform to Q t , T is expected to exhibit prominent values along its diagonal, reflecting temporally closely aligned frames; while off-diagonal elements should ideally possess diminished magnitudes. This sort of structural arrangement can be quantitatively assessed using:EQUATIONSimilar to Eqn. (4), the above 2 equations are combined as a regularizer on the transport matrix,M ( T ) = ϕ M t ( T ) + (1 -ϕ) M o ( T ). Such a structure is known as the inverse difference moment (IDM) [49, 46] . To encourage optimal alignment, M ( T ) of the learned T should be maximized. In order to facilitate this and to ensure the smooth assignment of such matches, we define a modified feasible set for T by incorporating two additional constraints into the set U (α, β),EQUATIONwhere KL(T ∥ Q) = N +1 i=1 M +1 j=1 t ij log tij qijis the Kullback-Leibler (KL) divergence between T and Q, and Q is same as Eqn. (4) but augmented with the virtual frame. So, the regularized Wasserstein distance between X and Y now becomesl R ξ1,ξ2 (X, Y ) = minEQUATIONThe above optimization can be efficiently solved by considering its dual. As such, we incorporate two Lagrange multipliers, λ 1 > 0 and λ 2 > 0, to obtain the dual of Eqn. (8) asl R λ1,λ2 (X, Y ) := ⟨ Tλ1,λ2 , D⟩, s.t. Tλ1,λ2 = arg minEQUATIONThe optimal Tλ1,λ2 that optimizes Eqn. ( 9) is e diag(-1 2 -µ λ 2 ) Ke diag(-1 2 -ν λ 2 ) , where K = [q ij e 1 λ 2 (s λ 1 ij -dij ) ] ij , s λ1 ij = λ 1 1 ( i N +1 -j M +1 ) 2 +1 + 1 1 2 dm+1, d m given by Eqn. (5) , and µ and ν are the dual variables for the two equality constraints T 1 M +1 = α, and T T 1 N +1 = β, respectively. The detailed derivation of this optimal Tλ1,λ2 is provided in appendix section A.1.Contrastive Regularization. Incorporating temporal priors into the video alignment processes often leads to trivial solutions [41, 46] . So, following [41, 1] , we utilize the Contrastive-Inverse Difference Moment (C-IDM) loss to further regularize the training. This loss is characterized by,EQUATIONwhere γ(i, j) = (i -j) 2 + 1, d(i, j) = ∥x i -x j ∥, N (i, j) is a neighborhood function defined as: N (i, j) = 1, if |i-j| ≤ δ and 0 otherwise, δ is a predefined window size, λ 3 is a margin parameter. The preceding C-IDM loss is an intra-video loss. Additionally, we incorporate an inter-video contrastive loss guided by OT to further regularize the training process. Specifically, this novel loss component contrasts pairs of videos based on their similarity as quantified by the OT matrix. We find, x best (i) = arg max j Tλ1,λ2 and x worst (i) = arg min j Tλ1,λ2 . Likewise, y best (j) = arg max i Tλ1,λ2 and y worst (j) = arg min i Tλ1,λ2 are calculated. Then, the best distance is computed as the average of squared differences between matched pairs, scaled by a temperature factor: best_distance =1 temperature • 1 N N i=1 ∥x i -y xbest(i) ∥ 2 + 1 M M j=1 ∥y j -x ybest(j) ∥ 2 .Similarly, the worst distance is: worst_distance= 1 temperature • 1 N N i=1 ∥x i -y xworst(i) ∥ 2 + 1 M M j=1 ∥y j -x yworst(j) ∥ 2 .Finally, the inter-sequence loss is computed using the cross-entropy over the best and worst distances:EQUATIONIdeally, we want each frame embedding x i , to align highly to its best match from Y . So the best distance should be as close to 0 as possible, at the same time, we maximize its distance from the unmatched frame embeddings, and the same holds true for y j s. As a result, our proposed inter-video loss (Eqn. ( 11)) promotes learning disentangled representations. So, the overall loss for OPEL combines the regularized OT loss (Eqn.( 9)) with the contrastive regularization terms,EQUATION)Clustering and Key-step Ordering. After learning the embeddings, our goal is to localize the key-steps required for PL. We frame this problem as multi-label graph-cut segmentation [50] . The node set V of the graph includes k terminal nodes representing the key-steps and non-terminal nodes corresponding to the number of frames, which are derived from the embeddings produced by the embedder network. Upon constructing the graph, we apply α-Expansion [19] to identify the minimum cost cut, utilizing the results to assign frames to k labels. To deduce the sequential order of key-steps, we first compute the normalized time for each frame in a video, following [1] . Subsequently, the temporal instant for each cluster is determined by calculating the average normalized time for frames allocated to that cluster. Clusters are then sequenced in ascending order of their average time, thus outlining the sequence of key-steps of a video. Upon establishing the key-step order for all videos associated with the same task, we generate a ranked list based on the frequency at which subjects adhere to a specific sequence. The most commonly observed order is placed at the top of this list. This methodological approach allows us to discern various sequential orders of key-steps of a task. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discusses limitations of the proposed OPEL framework, notably the assumption that subjects utilize similar objects for identical key-steps, which may introduce inaccuracies when dissimilar objects are employed. Additionally, the paper mentions the need for future work to explore the integration of additional contextual and semantic features within the OT framework to refine the procedure learning process further. These discussions demonstrate the authors' awareness of the limitations of their work and their intention to address them in future research. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The paper includes theoretical results, such as the introduction of the OPEL framework and its associated assumptions. Additionally, the empirical results are supported by theoretical underpinnings regarding optimal transport theory. The assumptions are clearly stated in the text, and the proofs, if any, are likely provided in either the main paper or supplementary materials, ensuring completeness and correctness.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments and results: 
 Datasets. In contrast to previous research that predominantly utilized either 1 st person or 3 rd person viewpoints for PL, we incorporate datasets from both perspectives. For 3 rd person view, we utilize established benchmark datasets, namely CrossTask [11] and ProceL [3] . CrossTask features 213 hours of video footage spanning 18 primary tasks, totaling 2763 videos. ProceL includes 47.3 hours of video from 12 varied tasks, comprising 720 videos. To evaluate the effectiveness of our proposed OPEL framework, we apply it to the 1 st -person EgoProceL benchmark [1] , which contains 62 hours of egocentric video recordings from 130 subjects engaged in 16 tasks. Detailed information on individual datasets is provided in Table A2 of appendix.Evaluation. Unless specified differently, we assess OPEL as per the current SOTA [1, 2] . We compute the framewise scores for each key-step separately and then take the mean of the scores over all the key-steps, reporting both the F1-score and the Intersection over Union (IoU). The F1-score is defined as the harmonic mean of precision and recall. Precision is calculated as the ratio of the number of frames correctly predicted as key-steps to the total number of frames labeled as key-steps.Recall is determined by the ratio of correctly predicted key-step frames to the total number of actual key-step frames. Following the methodology in [1, 2, 31, 3, 47] , we employ the Hungarian algorithm [51] to derive a one-to-one mapping between the ground truth and the predictions. Experimental Setup. We employ ResNet-50 (pretrained on ImageNet) as the embedder network.Inspired by [1] , we train the embedder using pairs of training videos. Within these videos, we randomly select frames and optimize the proposed L OPEL until convergence. The feature extraction is conducted from the Conv4c layer, and we subsequently create a stack of 2 context frames along the temporal dimension. Our video frames are resized to 224×224. The aggregated features are processed through two 3D convolutional layers, followed by a 3D global max pooling layer, two fully-connected layers, and a linear projection layer that outputs embeddings of 128 dimensions. All hyper-parameters are listed in Table A1 . Our code is provided as part of the supplementary material. Results on Egocentric View. Table 1 presents a comparative analysis between the SOTA techniques and OPEL applied to the large scale egocentric benchmark, EgoProceL. Results from tasks within CMU-MMAC and EGTEA G. have been aggregated and presented (detailed task-wise results are given in Table A4 ).It is important to highlight that EgoProceL represents a contemporary dataset specifically designed for egocentric procedure learning, thereby limiting the number of applicable approaches for fair comparison. Notably, OPEL outperforms the SOTA across most tasks. This superiority underscores the efficacy of the video representation learning through OT. Specifically, we achieve 22.4% (IoU) and 26.9% (F1) average improvement compared to current SOTA.Results on Third-person View. We provide comparison between SOTA and OPEL on two distinct third-person datasets [3, 11] in Table 2 . To ensure consistency in evaluation, we follow the evaluation protocol outlined in the SOTA prior arts [1, 2] . Once again, we perform better in almost all cases, with Figure 3 : Qualitative results from MECCANO [53] and PC Assembly [1] tasks. Each sub-task is color-coded to represent different key-steps, while gray areas signify background elements. Notably, OPEL's performance surpasses that of the SOTA networks, attributed to its ability to handle unmatched frames through the integration of a virtual frame, thus enhancing alignment accuracy.46.2% (F1) average enhancement over SOTA. Note, [32, 3] predominantly allocate frames to a single key-step, resulting in elevated recall rates but concomitantly diminishing precision, consequently impacting the overall F-score. Additional detailed third-person results from CMU-MMAC [17] , ProceL [3] and CrossTask [11] are given in Table A3 and Table A5 , respectively.Qualitative Results. Fig. 3 illustrates the qualitative PL outcomes of the baselines and OPEL.Higher match with the ground truth in our case (the bottom row) depicts the usefulness of OPEL. Additionally, we depict the alignment of two sequences in Fig. 4 (B), showcasing accurate alignment despite temporal variations, with correct matches indicating consistent action frame alignment and redundancy handling, affirming the reliability of our model. Comparison with Multimodal Models. While we only use videos for training, our results are competitive with models using multiple modalities. On the egocentric EgoProcel dataset, we perform comparably or even better (4 out of 6 datasets) compared to the multimodal SOTA model, STEPS [16] , as shown in Table 3 . Note, STEPS uses gaze and depth data during training, thus enhancing its results on EPIC-Tents. We also outperform [47, 34] (Table 2 ), which use narrations with video. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper includes detailed descriptions of the experimental setup, including the datasets used, evaluation metrics, and any preprocessing steps applied. Additionally, it describes the methodology and hyperparameters used to conduct the experiments, ensuring that other researchers can replicate the results. Our code is provided as part of the supplementary material. 
ablation study: 
 Effectiveness of L OPEL We analyze the effectiveness of the proposed loss by replacing or combining with other SOTA losses used for PL -TCC [8] , LAV [41] , and CnC [1] . Overall, the proposed L OPEL outperforms previous approaches as shown in Table 4 . This enhancement can be attributed to the flexibility in modeling sequences provided by OT. Furthermore, ablation results on all the loss components of L OPEL are provided in Table 5 , where we show the contribution of each factor individually to analyze their effect on the overall result. Comparing row 3 with row 9, we observe, the priors jointly play a critical role; as without them (row 3), the F1 and IoU scores drop by ∼5 points.Specifically, the optimality prior has a significant impact (∼2 point), while the temporal prior affects the score by ∼1 point. Similar to the combined priors, the intra and inter-video contrastive losses together (row 6 vs row 9) have a significant effect (∼3.5 points) on the overall performance. The individual effect of virtual frame is negligible as it only plays a role in case of excessive background frames -a scenario that is not prevalent in most datasets. Furthermore, due to the IDM structure of M ( T ), the T and Q are similar by formulation. This results in KL( T ∥ Q) to be already small. As a consequence, adding the KL divergence as a standalone loss component in the proposed pipeline has a minimal impact. Overall, while some loss components may have a smaller individual impact, they do contribute to performance improvements, even if incrementally. Therefore, our proposed approach incorporates all of them to achieve the best possible results. Choice of clustering algorithm. We replace the proposed clustering approach with K-means and subset selection (SS). The results in Table 6 show that OPEL performs the best, highlighting the effectiveness of OT with graphcut segmentation. 7 , we present the results of OPEL alongside baseline models, with varying k. Note, we obtain best results with k=7, and the performance drops sharply as k goes from 7 to 10 or higher. This observation is consistent with all the other SOTA methods on the same datasets [1, 2, 16] . We hypothesize that k=7 works best as it is the optimal number of clusters considering the average number of distinct key-steps (subtasks) of the datasets. For example, for PC Disassembly, although the ground-truth (GT) number of steps is 9, 3 steps are quite similar (remove hard disk, remove motherboard, remove RAM), effectively making them quite close in the feature space. This results in k=7 being a better estimation of the cluster number with distinct steps. Note, this demarcation of subtasks (hence, number of clusters) is subjective and varies from dataset to dataset as well as from task to task; as some may consider semantically similar tasks (e.g. pouring oil vs water) to be one subtask, while others may consider it different. As k becomes larger than the actual distinctive number of clusters, each subtask gets split into multiple clusters with very similar embeddings, which upon comparison with GT leads to inferior results.Impact of Training data Quantity. Fig. 4 (A) presents the results from varying the number of training videos on MECCANO, aiming to evaluate OPEL's performance with respect to video count. We consistently outperform other SOTA methods. Overall, the performance improves with more training data, however, even with just few (2-5) videos of a task, we reach the upper-limit of other methods using full dataset, as shown in Fig. 4(A) . Additional ablation results including choice of distribution as priors and hyperparameters λ 1 , λ 2 are provided in appendix A.8.Comparison with AS methods. PL and action segmentation (AS) are related but not the same. PL, when applied to a set of instructional videos depicting the same task, involves two primary steps: (i) assigning each video frame to one of the k key-steps (including background elements), and (ii) determining the logical sequence of these key-steps necessary to complete the task. As illustrated in Fig. 1 , PL addresses multiple videos of a given task, enabling the identification of repetitive key-steps across these videos [1, 2] . In contrast, AS [4] focuses on a single video, thereby lacking the ability to discern repetitive key-steps across different videos. Despite the differences between PL and AS, we compare our approach against existing SOTA unsupervised AS models and present the results in Table 8 . Our model demonstrates a significant performance improvement compared to these works. In [3, 56] , authors report a high recall score for CrossTask as it assigns majority of the frames to a single key-step -a phenomenon also reported by [2] . While achieving high recall is important for ensuring that most positive instances are correctly identified, it can result in a greater number of false positives, which in turn lowers precision and leads to undesirable results. Therefore, it is crucial to balance recall with precision to develop an effective model. This balance is reflected in the superior performance of our model, as evidenced by the F1-score results across various benchmarks. Note in the Table 8 , our approach is compared with SOTA unsupervised AS methods for only third-person datasets, as these do not report any result on egocentric datasets.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The supplementary material accompanying the paper includes the necessary code for reproducing the main experimental results. Detailed instructions provided within the folder guide users on executing the code to replicate the experiments, including accessing and preparing the necessary data. This encompasses instructions on accessing raw data and preprocessed data utilized in the experiments. Additionally, the scripts are provided to reproduce all experimental results for both the new proposed method and baselines.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We anticipate no negative societal impact of the work, however it could have positive impact by improving robotic applications (which we discuss as motivation in the introduction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this study, we have introduced a novel approach for procedure learning leveraging optimal transport, enhanced by temporal and distributional regularizations to improve the alignment of key-steps across multiple video instances. Our method addresses inherent limitations in current SOTA techniques that primarily rely on frame-to-frame mappings and assumptions of monotonic alignment, which do not optimally utilize temporal information. We observe an improvement of 22.4% in IoU and 26.9% in F1 scores on the EgoProceL dataset, outperforming the current state-of-the-art methods.Similarly, in third-person video benchmarks, such as ProCeL and CrossTask, our framework achieves an average F1 score enhancement of 46.2% over existing methods. These advancements underscore the potential of OT guided learning in handling complex video procedure learning tasks. A limitation of the proposed OPEL framework is the assumption that subjects utilize similar objects for identical key-steps, which may introduce inaccuracies when dissimilar objects are employed in the execution of these steps. Future work will focus on exploring the integration of additional contextual and semantic features within the OT framework to further refine the procedure learning process. Moreover, extending this framework to other domains of video understanding could provide valuable insights into the general applicability of optimal transport in video analysis tasks. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?• The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Derivation of the Optimal Transport Matrix ( Tλ1,λ2 )In this section, we obtain the optimal transport matrix Tλ1,λ2 that optimizes the OT Eqn. 9. Note, all the notations used here are same as section 3 of the main manuscript. We start with Eqn. 9 of section 3l R λ1,λ2 (X, Y ) := ⟨ Tλ1,λ2 , D⟩, s.t. Tλ1,λ2 = arg minT ∈U (α,β) ⟨ Tλ1,λ2 , D⟩-λ 1 M ( T )+λ 2 KL( T ∥ Q),From the duality theory, we know, for each pair ξ 1 , ξ 2 in Equation 8, a corresponding pair λ 1 > 0, λ 2 > 0 exists, such that l R ξ1,ξ2 (X, Y ) = l R λ1,λ2 (X, Y ) for the pair (X, Y ). Tλ1,λ2 is the optimal transport matrix, so it optimizesminT ∈R N +1×M +1 + ⟨ T , D⟩ -λ 1 M ( T ) + λ 2 KL( T ∥ Q) subject to T 1 M +1 = α, T ⊤ 1 N +1 = β, (A1)Given that both the objective function and the feasible set defined in Equation A1 are convex, the existence and uniqueness of the optimal transport matrix Tλ1,λ2 are guaranteed. To derive this optimal matrix, the analysis begins by taking the Lagrangian of Equation A1 as-L( T , µ, ν) = N +1 i=1 M +1 j=1   dijtij -λ 1 t ij    1 i N +1 -j M +1 2 + 1 + 1 1 2 d m + 1    + λ 2 t ij log t ij q ij    + µ T ( T 1 M +1 -α) + ν T ( T ⊤ 1 N +1 -β), (A2)whereEQUATION, i o and j o have their same meaning as the main paper, i.e. they correspond to the optimal assignment locations (i, j o ) and (i o , j) as provided by the transport matrix, and µ and ν are the dual variables for the two equality constraints T 1 M +1 = α, and T T 1 N +1 = β, respectively. Taking the derivative of L( T , µ, ν) w.r.t.t ij yields- ∂L( T , µ, ν) ∂t ij = d ij -λ 1     1 i N +1 -j M +1 2 + 1 + 1 1 2 i-io N +1 2 + j-jo M +1 2 + 1     + λ 2 log t ij q ij + λ 2 + µ i + ν j . (A3) Let, s λ1 ij = λ 1 1 ( i N +1 -j M +1 ) 2 +1 + 1 1 2 ( i-io N +1 ) 2 +( j-jo M +1 ) 2 +1.Setting L( T , µ, ν) = 0, we get-t ij = q ij e -1 2 - µ i λ 2 e 1 λ 2 (s λ 1 ij -dij ) e -1 2 - ν j λ 2 . (A4) Let, K = [q ij e 1 λ 2 (s λ 1 ij -dij )] ij , then we get,t ij = e -1 2 - µ i λ 2 K ij e -1 2 - ν j λ 2 ,Thus,Tλ1,λ2 = e diag(-1 2 -µ λ 2 ) Ke diag(-1 2 -ν λ 2) . (A5)Every element of the matrix K is strictly positive, as we take the element-wise exponential to obtain each K ij and q ij > 0. As per the Sinkhorn's theorem (Theorem A), there exist diagonal matrices diag(κ 1 ) and diag(κ 2 ) with strictly positive diagonal elements, such that diag(κ 1 )Kdiag(κ 2 ) is a member of the set U (α, β), with κ 1 ∈ R N +1 and κ 2 ∈ R M +1 . This product matrix is unique, and the diagonal matrices are also uniquely determined, up to a scalar factor.Theorem A [57, 58] : For any (N + 1) × (M + 1) matrix A with all positive elements, diagonal matrices B 1 and B 2 exist such that B 1 AB 2 belongs to U (α, β). Both B 1 and B 2 possess strictly positive diagonal elements and are unique up to a positive scalar factor.The optimal Tλ1,λ2 of Equation (A5) in U (α, β) mirrors the form of diag(κ 1 )Kdiag(κ 2 ), thereby constituting the unique matrix in U (α, β) that represents a rescaled version of K. We efficiently compute the scaling vectors κ 1 and κ 2 , also unique up to a scaling factor, using the Sinkhorn-Knopp iterative matrix scaling algorithm-κ 1 ← α Kκ 2 , κ 2 ← β K T κ 1 .In this paper, only 20 iterations are used, as a limited number of iterations has been shown to effectively converge in previous studies [48] . For our experiment, we require adequate computing resources to effectively train our models. We utilize a single Nvidia A40 GPU, but its full RAM is not required. The GPU memory is dependent on batch size (bs). For a bs of 2, a GPU equipped with approximately 12GB of memory is sufficient for our purposes. Training time depends on dataset size and number of epochs (we used 10000). The above configuration allows us to process a dataset consisting of 15-20 videos (e.g. PC assembly/MECCANO domain) in around 12 hours. With these computing resources in place, we conducted our experiments effectively, ensuring optimal performance and reliable outcomes.In Table A2 , we provide the statistical analysis for each of the 16 tasks within the EgoProceL dataset [1] . N denotes the total number of videos, while K represents the number of key-steps for each task. u n signifies the count of unique key-steps, and g n denotes the number of annotated key-steps for the n th video. Following the approach outlined in reference [31] , we compute the subsequent metrics:Foreground Ratio: This measure indicates the proportion of the total duration occupied by key-steps in relation to the overall video duration. It assists in gauging the prevalence of background actions within a task. The foreground ratio is inversely correlated with the presence of background activity and is determined as:F = N n=1 t n k t n v N (A6)Here, t n k and t n v represent the duration of key-steps and the video for the n th instance, respectively. The foreground ratio F ranges from 0 to 1. The higher the value of F indicates minimal background actions. Missing Key-steps: This metric quantifies the number of omitted key-steps in each video. It is defined as:M = 1 - N n=1 u n KN (A7)The range of M is between 0 and 1. It aids in assessing the feasibility of completing a task despite certain steps being skipped.Repeated Key-steps: This metric assesses the occurrence of repeated key-steps across multiple videos.It is expressed as:R = 1 - N n=1 u n N n=1 g n (A8)The range of R varies between 0 and 1. Higher values of R indicate a greater recurrence of key-steps across videos. OPEL considers these repetitions, to demonstrate better performance.In this comparison, we assess the outcomes of training OPEL on diverse perspectives from CMU-MMAC [17] . Table A3 depicts the F1-Score and IoU scores per frame for exocentric views. We conducted our experiments on exocentric videos and achieved promising results. Through rigorous testing and analysis, our model demonstrated strong performance when trained and evaluated on this particular perspective. The obtained outcomes not only validate the effectiveness of our approach but also underscore its applicability to real-world scenarios involving both egocentric and exocentric video data. We report the results for all subtasks within the egocentric datasets in Table A4 such as CMU-MMAC [17] and EGTEA-GAZE+ [52] , as well as for various third-person exocentric videos in datasets like ProceL [3] and CrossTask [11] in Table A5 . Our analysis encompasses comprehensive evaluations across these diverse datasets, providing insights into the performance of our model across different perspectives and scenarios. These results offer a holistic understanding of the capabilities and effectiveness of our approach in handling varied video types and tasks, contributing significantly to the advancement of research in procedure learning and related domains. Learning from multiple videos of the same task opens up numerous potential applications. Firstly, in monitoring procedures, a system trained to recognize key steps can identify deviations or variations when a new person performs the task. Secondly, in guidance systems, such a system can detect the current step and suggest the next steps needed for task completion. Thirdly, in automated systems, this framework enables robotic systems to autonomously learn key steps through observation, allowing them to perform tasks independently in future instances without human assistance.In terms of cross-modal transfer within videos, the capability to align related videos without supervision allows for the transfer of annotations or other modalities from one video to another. For example, text annotations can be applied to an entire dataset of related videos by labeling just one. Additionally, temporal modalities like sound can be transferred between videos; for instance, the sound of pouring liquids can be transferred purely based on visual cues.Moreover, fine-grained retrieval within videos can be achieved by using nearest neighbors, enabling the retrieval of specific frames that depict various scenarios. Anomaly detection is possible by observing deviations in the video trajectories within the embedding space, helping to identify unusual activities. This ensures the proper sequence of tasks, such as jacking up a car before accessing the wheel during a tire change.Instead of using the Laplace distribution, we also tested a Gaussian mean distribution as described in Eqn. A9, with mean µ and variance σ to accommodate temporal variations. Additionally, we evaluated a Uniform distribution as described in Eqn. A10. We conducted ablation experiments with various hyperparameters and summarize the best performances in Table A6 . However, the Laplace distribution consistently outperformed these alternatives, leading us to adopt it for our experiments. To analyze the superior performance of Laplace as a prior, we plot the distributions in Fig. A1 . Note, we use the same distribution for optimality as well as temporal priors.Q(i, j) = N (x; µ, σ 2 ) = 1 √ 2πσ 2 exp - (x -µ) 2 2σ 2 (A9) Q(i, j) = f (x; a, b) = 1 b-a if a ≤ x ≤ b, 0 otherwise. (A10)For the optimality prior, the x-axis the difference of frames in the feature space (1-d representation for illustration purposes), and the y-axis denotes the corresponding probability of alignment. We want the point representing the most likely alignment (as per T ) to have the highest likelihood, while the assignment probability should exponentially decay further away. The graph clearly shows that the Laplace distribution captures this behavior suitably compared to Uniform and Gaussian. Similarly, for the temporal prior, the x-axis denotes the temporal distance between the frames, and the y-axis denotes the corresponding probability of alignment. The graph shows that Laplace distribution facilitates alignment of the frames when they are temporally aligned (close to center), and its long tail distribution enables better correlation of non-monotonic frames compared to Gaussian or Uniform. As a result, as shown in Fig. A1 , even at far away locations from the center (temporally distant frames), alignment is possible if indeed these frames have a high match feature-wise. In this case, the Laplace temporal prior provides non-zero probability to that far away frame (due to its long tail) unlike other distributions and the optimality prior gives a large score (due to feature match), resulting in improved handling of non-monotonicity.A.8.2 Hyperparameters λ 1 and λ 2We present additional experiments concerning the hyperparameters used in our model, specifically λ 1 and λ 2 in Eqn. 9. We assess their impact on the EgoProceL dataset. According to Table A7 , we find1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state the main claims of the paper, including the introduction of a novel approach, OPEL, for procedure learning, addressing limitations in current techniques, and demonstrating remarkable efficacy through empirical results on benchmark datasets. The claims are supported by empirical results showing significant improvements over existing methods, and the limitations and future directions are also mentioned, providing a comprehensive overview of the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper received active discussions between authors and reviewers, with the outcome that the reviewer reached an accept consensus (5, 5, 6, 7). The reviewers appreciate the novelty of the method, the new insights that come with bringing optimal transport in the video recognition domain, and the strong empirical performance. The reviewers have pointed out limitations when it comes to writing/explanations as well as the overlap with action segmentation. In the rebuttals, the authors have performed additional experiments with comparisons on action segmentation, which further strengthens the paper. The AC finds that the strengths of the paper outweigh the limitations given by the reviewers, especially after the rebuttal stage. Hence the recommendation to accept.

Reviews: review: paper_summary: paper_summary: OPEL is a novel technique for Procedure Learning. Procedure learning is the task of finding key steps in an action (such as cooking brownie) and aligning the videos based on these key steps. 
OPEL proposes to use the optimal transport distance between the two videos as the similarity metric rather than direct frame by frame mapping or assuming a strict monotone mapping.
This work also proposes two regularizers to incorporate priors about monotone mapping and about increasing the correspondence between nearby frames.
The results on both egocentric and 3rd person videos show significant improvement upon the previous SOTA tasks. 
The ablation studies analyze the effect of each of the losses and regularizers. 
It seems having the temporal prior and the inter-intra cross entropy are the most important factors in terms of f1 accuracy.
strength_weakness: strength_weakness: There are many factors added together to make this method work. Although many of them are common in other methods as well, but the method and also the ablation seem to suggest that some of the losses may be redundant. It is not clear if the difference between some of the lines in table. 5 is statistically significant.

Page 6. is written in such a compact way that it's not easy to read and follow. The indices such as i,j can be probably dropped. Words such as temperature in the formula can be probably replaced with tau.
questions: questions: Why do you think the number of steps makes such a large difference in table 7? going from 7 to 10 drops the performance significantly. another anomaly is 12 being worse than both 10 and 15.
limitations: limitations: not explicitly.
paper_summary: paper_summary: The paper proposes an unsupervised method for procedure learning that identifies the key steps and their orders in several videos of the same task.
The paper formulates the distribution of video frames as an optimal transport (OT) problem to compute the distances between the key steps. 
To handle the variations of the videos, the paper introduces a regularization with prior distribution.
The proposed method achieves SOTA on third-person and first-person video benchmarks.
strength_weakness: strength_weakness: - The explanation of using priors to mitigate the variation of the videos, i.e., action speeds, non-monotonic sequences, or starting of actions, is not clear.
- Besides the ablation studies of different prior distributions, the paper should explain why Laplacian outperforms other distributions.
questions: questions: The authors should give a clear explanation of the regularization usage as mentioned in the weakness.
limitations: limitations: The authors have mentioned the limitations.
paper_summary: paper_summary: The authors propose a novel approach for procedure learning leveraging optimal transport, OPEL. OPEL integrates optimality and temporal priors, and incorporates a novel inter-video contrastive loss. OPEL achieves significant improvements on egocentric and third-person benchmarks.
strength_weakness: strength_weakness: 1. The presentations of the paper needs improvement, especially in Section 3, where an excessive number of formulas obstruct readability.
2. Some figures are unclear, such as Figure 1. Are V1, V2, V3, V4 from four different videos, or do they appear to be from the same video? Additionally, what distinguishes V3 from V4, where one represents background frames and the other represents redundant frames?
questions: questions: 1. I still have some doubts regarding the difference between procedure learning and action segmentation. I believe their final output forms are same. From related work, I believe that action segmentation involves classifying each frame, while procedure learning only needs to identify key steps. Is this correct? Could you please explain in detail?
2.I also feel that the paper doesn't clearly present the problem formulation, including the input and output, which makes it somewhat confusing to understand. Does the output video contain a fixed number of K key steps? According to the paper, setting K to 7 is optimal. However, based on the dataset, the average number of steps for each task may be greater than or less than K. When it exceeds K, are some steps merged? And what happens when it's less than K? Is it reasonable to pre-define K?
3. Could you provide more results of ablation training on K for additional tasks?
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents OPEL, a novel framework for procedure learning from videos that leverages optimal transport (OT) to align key steps across different video instances. OPEL treats video frames as samples from an unknown distribution and formulates the distance calculation between them as an optimal transport problem, allowing for more flexible alignment compared to frame-to-frame mappings. The authors introduce two regularization terms to improve the OT formulation and experiments significantly outperforms state-of-the-art methods on benchmark datasets.
strength_weakness: strength_weakness: 1. I believe the authors can compare with recent action segmentation methods to further strengthen the experiments.
2. I believe the authors can also compare with methods that use temporal alignment methods like dynamic time warping to strengthen their contributions.
questions: questions: I do not have specific questions on the model design. I only have some experiment suggestions which are the same as I listed in "weaknesses".
limitations: limitations: N/A

Comments: title: Thank you
Discussion period
Discussion period
Discussion period
Consistent results across multiple runs
Acknowledged
comment: We thank the reviewer for the consideration and are glad that the reviewer liked our responses.
Thank the authors for their responses. I am satisfied with the answers. After reading other reviews, I will raise my score to 6.
Dear Reviewer, thanks again for your feedback. We hope you consider our responses based on your comments, looking forward to the discussion.
Dear Reviewer, thanks again for your feedback, hope you consider our responses based on your comments. Please let us know if you have any further queries, looking forward to the discussion.
Dear Reviewer, thanks again for your feedback, hope you consider our responses. Please let us know if you have any further queries, looking forward to the discussion.
Thanks for your response. In our paper, we reported just the **mean values** obtained over multiple runs and not the standard deviations as the results did not vary significantly. Also, previous SOTA works [a, b, c] do not report the standard deviation across runs either. However, as per the reviewer's suggestion, we report the results of the mean ± standard deviation (SD) over 5 separate runs in **Table A** below. Note, in all cases, we find that the SD is quite low, and we get consistent results over the multiple runs, further demonstrating the statistical significance of the results.    

  
**Table A:** Results showing mean ± SD over 5 runs for all the datasets

| Dataset         | F1 (mean ± SD)    | IoU (mean ± SD)    |
|:----------------|:-----------------:|:------------------:|
| CMU-MMAC        | 36.5 ± 0.138      | 18.8 ± 0.106       |
| EGTEA-GAZE+     | 29.5 ± 0.147      | 13.2 ± 0.145       |
| MECCANO         | 39.2 ± 0.319      | 20.2 ± 0.258       |
| EPIC-Tents      | 20.7 ± 0.165      | 10.6 ± 0.101       |
| PC Assembly     | 33.7 ± 0.311      | 17.9 ± 0.184       |
| PC Disassembly  | 32.2 ± 0.317      | 16.9 ± 0.203       |
| ProceL          | 34.9 ± 0.095      | 21.3 ± 0.037       |
| CrossTask       | 35.1 ± 0.142      | 21.5 ± 0.111       |

Please let us know if you have any further queries. Thanks again and hope you reconsider your score based on our responses.

**Refs.**:

[a] Bansal et al. My view is the best view: Procedure learning from egocentric videos. ECCV 2022.

[b] Bansal et al. United we stand, divided we fall: Unitygraph for unsupervised procedure learning from video. WACV 2024.

[c] Shah et al. Steps: Self-supervised key step extraction and localization from unlabeled procedural videos. ICCV, 2023.
Thank you for your response. In terms of statistical significance, what is the standard deviation for a set of 5 runs for example?