OPEL: Optimal Transport Guided ProcedurE Learning
Authors: Sayeed Shafayet Chowdhury
Soumyadeep Chandra
Kaushik Roy
Venue: NeurIPS 2024 poster (2024)
Keywords: Procedure learning
Egocentric vision
EgoProceL
Optimal Transport
Primary area: machine_vision
TL;DR: An unsupervised optimal transport guided method for procedure learning, which achieves state-of-the-art results

Abstract: Procedure learning refers to the task of identifying the key-steps and determining their logical order, given several videos of the same task. For both third-person and first-person (egocentric) videos, state-of-the-art (SOTA) methods aim at finding correspondences across videos in time to accomplish procedure learning. However, to establish temporal relationships within the sequences, these methods often rely on frame-to-frame mapping, or assume monotonic alignment of video pairs, leading to sub-optimal results. To this end, we propose to treat the video frames as  samples from an unknown distribution, enabling us to frame their distance calculation as an optimal transport (OT) problem. Notably, the OT-based formulation allows us to relax the previously mentioned assumptions. To further improve performance, we enhance the OT formulation by introducing two regularization terms. The first,  inverse difference moment regularization, promotes transportation between instances that are homogeneous in the embedding space as well as being temporally closer. The second, regularization based on the KL-divergence with an exponentially decaying prior smooths the alignment while enforcing conformity to the optimality (alignment obtained from vanilla OT optimization) and temporal priors. The resultant optimal transport guided procedure learning framework (`OPEL') significantly outperforms the SOTA on benchmark datasets. Specifically, we achieve 22.4\% (IoU) and 26.9\% (F1) average improvement compared to the current SOTA on large scale egocentric benchmark, EgoProceL. Furthermore, for the third person benchmarks (ProCeL and CrossTask), the proposed approach obtains 46.2\% (F1) average enhancement over SOTA.

OPEL: Optimal Transport Guided ProcedurE Learning
Sayeed Shafayet Chowdhury, Soumyadeep Chandra, and Kaushik Roy Elmore Family School of Electrical and Computer Engineering Purdue University, West Lafayette, IN 47907, USA
chowdh23, chand133, kaushik @purdue edu
Abstract
Procedure learning refers to the task of identifying the key-steps and determining their logical order, given several videos of the same task. For both third-person and first-person (egocentric) videos, state-of-the the--f-tthh-rttt (SOTA) methods aim at finding correspondences across videos in time to accomplish procedure learning. However, to establish temporal relationships within the sequences, these methods often rely on frame-to-frame mapping, or assume monotonic alignment video pairs, leading to sub-optimal results. To this end, propose treat the video frames as samples from an unknown distribution, cnabling us to frame their distance calculation based formulation allows us to relax the previously mentioned assumptions. To further improve performance, we enhance the OT formulation by introducing two regularization terms. The first, inverse difference moment regularization, promotes transportation between instances that are homogeneous in the embedding space as well as being temporally closer. The second, regularization based on the KL-divergence with exponentially decaying prior smooths the alignment while enforcing conformity to the optimality (alignment obtained from vanilla OT optimization) and temporal priors. The resultant optimal transport guided procedure learning framework ("OPEL') significantly outperforms the SOTA on benchmark datasets. Specifically, we achieve 22.4% (IoU) and 26.9% (FI) average improvement compared to the current SOTA on large scale egocentric benchmark, EgoProceL. Furthermore, for the third person benchmarks (ProCeL. and CrossTask), the proposed approach obtains 46.2% (F1) average enhancement over SOTA.
optimal transport (OT) problem. Notably, the OT-
Introduction
The development of autonomous agents capable of reliably replicating human actions to accomplish certain end goals presents significant challenges. Traditional approaches would necessitate hard- coding tedious explicit instructions for each sub-task of the process (thus difficult scale and generalize). A more efficient solution would involve the agent learning directly from observing multiple demonstrations of the assembly, without the need of any label. This motivates us to explore unsupervised procedure learning from videos. In the context of this work, procedure learning (PL) conceptualized as the process determining the key-steps and their correct sequential order to accomplish an overall task, as demonstrated across multiple video demonstrations 212
PL analyzes multiple videos of a task as illustrated in Figure ! contrast to action-based tasks 4 which focus on single video. The single-video approach is inapplicable to the identification of repetitive key-steps across videos. Moreover, action-based tasks typically neglect the sequencing of events, crucial for discerning overall expected procedure composed of the sub-tasks. For instance, they fail to capture variations the sequence of key-steps between videos V1 and V2 (Fig. [T. Other research efforts in video understanding that employ instructional videos include procedure planning [5), verifying sequences of procedures (6), and summarizing instructional content (7]. Additionally, unlike video alignment tasks [8. PL. specifically aims to localize these essential steps within videos.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Background
Ejs
Background Frames
tomparal
V
Tosk Add
V
Non-monotanic Frames
Tack 5 Kic contents
Ideal Frames
Task
Egu
Sequential progression proparation brownies
Embedding Spare
Figure 1: Key-steps required to prepare a brownie 17 and corresponding key-step alignment challenges, namely (i) background frames (depicted gray blocks), (ii) non-monotonic frames. OPEL aims to learn an embedding space where corresponding key-steps have similar embeddings while tackling the above challenges.
The sequences showcase temporal variations
Much of the research on PL till now has been performed within the frameworks of supervised [9, 10 I and weakly supervised learning [12 13 14] In a supervised setting, the reliance on per-frame annotations demands extensive manual labor. Conversely, weakly supervised methods involve using either ordered or unordered lists of key-steps. The generation of these lists requires either direct observation of the videos specific heuristics, both of which pose significant scalability challenges [3. Consequently, recent studies [ g have shifted focus towards self-supervised learning, which do not require frame-wise labeling. Such learning paradigm leverages the structured nature of accomplishing a complex task, which typically unfold in a predictable sequence of steps. For example, the act of preparing "brownie" might involve breaking the contents, and then baking in the oven. The alignment of video frames is commonly performed in a monotonic manner [5], which presupposes consistent order of actions across sequences. However, real-world sequences frequently deviate from this pattern, exhibiting temporal non- non-uniformities as depicted in Figure I These deviations can be categorized as follows: (i) background frames: frames irrelevant to the primary activity and should thus be excluded from alignment; (ii) redundant frames: these frames appear only one sequence but not in others and do not contribute the task; (iii) no-mooootonic frames: these frames are characterized by non- monootooo sequence of actions. Such frames challenge the assumption of monotonic progression and highlight the complexity of real-world activities. State-of-the-aae ahe-art (SOTA) methods adopt custom approaches to counter these irregularities such as removing background frames from processing (2], using extra information (e.g. gaze, depth) [16). or simply ignore them leading to suboptimal results 1
egg, adding water, oil, mixing
To address the limitations of previous approaches, we relax the strict assumptions about the temporal sequence of actions and introduce a novel PL framework designed to learn temporal correspondences across videos. By treating instances of the sequences as samples from an unknown distribution, we formulate the task of computing the distance between them as an optimal transport (OT) [18] problem. The differentiable OT loss facilitates the alignment of non mon-monoto sequences through frame-wird wise matching based on individual frame features. However, it typically overlooks the temporal smoothness and the inherent ordering relationships within the videos. overcome this deficiency, we integrate two priors into the transportation matrix. First, the optimality prior favors the positions dictated by the OT, whereas the temporal prior discourages transport between temporally distant frames. Both these priors are modeled using a Laplace distribution with exponentially decaying probability from the corresponding centers. We introduce an additional virtual frame into the OT matrix to address background and redundant frames. Furthermore, to avoid the common issue of converging to trivial solutions temporal video alignment (5), we employ a novel inter-video contrastive loss, which acts as a regularizer. Finally, the sub-tasks of each video are clustered in the embedding space using graphcut segmentation [19]. The overall framework, termed optimal transport guided procedure learning ('OPEL'), achieves SOTA results on both the ego and exocentric benchmarks. To summarize, our main contributions are-
We propose a novel optimal transport based procedure learning framework that aligns frames with similar semantics together in embedding space.

---

To enhance the OT-based learning, we integrate optimality and temporal priors, both modeled using the Laplace distribution. These two priors also serve regularizers. Furthermore, OPEL incorporates novel inter-video contrastive loss for additional improvement.
OPEL demonstrates substantial performance gains, achieving an average improvement of 22.4% in IoU and 26.9% in Fi-score compared to the current SOTA on the EgoProceL. benchmark.
2 Related Works
Representation Learning for Videos. Recent studies have explored various pretext tasks facilitate representation learning through self-supervised or unsupervised approaches. Examples include
temporal coherence and sequence ordering [20] 21 22 23 24], predicting frames [25 26 27 28 29).
and determining the directionality of time 30 These methods typically derive signals from a
constrained set of videos. In contrast, our objective is discern and characterize key-steps of certain task across multiple videos, expanding the scope and applicability of representation learning.
Self-Supervised Representations for Procedure Learning. Previous studies PL have focused on developing methods for learning frame-level features [1 3] 32 33
For instance, Kukleva et al. [32) enhance the representation space by utilizing relative timestamps of frames, while Vidal et al. 333 engage in predicting future frames along with their timestamps. Elhamifar et al. 3 apply attention mechanisms to individual frames enhance feature learning. Similarly, Bansal et al. [ leverage temporal correspondences across videos to generate signals and learn frame-level embeddings. The current SOTA model for egocentric PL (2) utilizes task-level graph representation to cluster semantically similar and temporally close frames. Despite these advancements, these methods often exhibit limitations in adequately modeling either temporal or spatial relationships within video sequences, especially in the presence of background and redundant frames. As result, extra curated processing steps are required, resulting in additional layer of complexity and computation e.g. 2 depends on background frame removal to improve performance.
Multi-modal Procedure Learning. PL has also been used with multi-modal data such as (a) narrated text and videos [34] 35
36
37
38 [3], (b) optical flow, depth and gaze information []
These studies typically rely on the assumption of reliable alignment between video content and corresponding supporting modalities [34 38 33), an assumption that often proves inaccurate B
due to lack of synchrony among the modalities. Additionally, the dependence on imperfect Automatic Speech Recognition (ASR) systems necessitates subsequent manual corrections. Moreover, multiple modalities require additional memory and compute. In contrast, our framework exclusively leverages visual data, thereby circumventing the inaccuracies associated with multimodal alignment and enhancing scalability by eliminating the need for extra data modalities.
Video Alignment. can be efficiently addressed in synchronized settings using established methods like Canonical Correlation Analysis (CCA) 40 and -Dtammi Time Warping (DTW) M. A recent work [2 aligns videos by learning self-supervised representations from multiple viewpoints. However, the requirement for synchronized multi-view recordings limits its applicability. To tackle this challenge, [8] proposes a cycle consistency loss to establish frame correspondences, focusing primarily on local matches and not on the global temporal structure of the videos. Perhaps, works leveraging OT for visual analysis 43 44 45] are most related to our approach. But, such approaches do not address sequence alignment as we do. An exception is [6 which employs OT for videos, however their setup for evaluation is supervised fine-tuning for action segmentation, fundamentally different task than unsupervised PL. As a result, [46] does not deal with temporal localization of the key-steps of a task nor their ordering, unlike us. Morcover, distribution and inter-video contrastive loss formulation are different from [46].
modeling of priors using Laplace
Learning Key-step Ordering. Most existing studies fail to account for variations in the ordering of key-steps required to complete a task, often assuming either strict sequential order [3 32| (3] or neglecting to model the sequence altogether 3 47 However, illustrated in Figure I individuals
frequently execute the same task in diverse manners, underscoring the need for a more flexible approach. To that end, OPEL is designed to identify and construct unique key-step sequence for each video, thereby adapting to and inferring the specific ordering of the task.
3 OPEL Framework
Optimal Transport Formulation. OT provides metric for assessing the dissimilarity between two probability distributions within a metric space [18]. By using the feature vectors from each entity and distance matrix between them, it establishes correspondences that minimize total distance, also
3

---

(A)
Video Embeddings
(8)
Subtasks Assignment Variations
Video Sequences
N
N
ENCOOEE
X
A
M
Y
Sequential Nignment
Temporal Offset
Speed Variation
Mon-Monotanic
Laplace Distribution
Video [iaan (D0)
(D)
M
Video En
Video
Temporal Prize
OptimalityPin Prioe
Opinnality Price
Cennet Alganen
Coctiied Prio
Combired add Prior
Alggnment
Figure 2: (A) The encoder generates frame-wise embeddings from videos, facilitating subsequent OT calculations. (B) Pair-wise scenarios captured through the assignment matrix- from strictly synchronized actions to temporal shifts and differing action speeds, to non-monotonicity. (C) 1-D depiction of alignment of single frame (i-th) of Video with its best match frame (j-th) of Video 1, based on the proposed priors. (D) 2-D representation of the optimal alignment of frame sequences.
ensuring optimality, separability. and completeness. Assume, the inputs are two sequences of video frames, P (Pi. P2. ... P.v and Q 441 r.r).. We pass these through deep encoder network (as illustrated in Fig. ({)) to obtain their respective embeddings, X = [I1, .2..... IN] and Y [3/1 ........ BM.) ]. Let. (2,,1) is metric space. where Ω S2 + denotes the distance in Ω and P(2)) represents all Borel probability measures on Ω. Considering the elements of X and Y independent samples, their probability measures can be written as, Σ a, and g ΣØe Bydn,, where 0g denotes the Dirac mass at æ, and a and B are the weights for the distributions and g. respectively. Since there is no justification for assigning greater importance to one frame over another, initially we a; N and B 1 for all i, j. leading
feasible set of weight matrices U (a, .) == {T N TIy = a, T 1N 3).
defined as the transportation polytope Here, &jj can be interpreted to be proportional to the probability that æ; will be aligned to yy. We start by computing the pairwise Euclidean distances between embedding vectors, d(x;; y3) = | y to form the N x M distance matrix, D. The cost of transporting mass from to with transport plan T' quantified by the Frobenius inner product (T, D). Thus, the Wasserstein distance raised to the power p is: W (f. Iw (a, ,3, D) minret ((x, .,9) (T. D). We only consider p and drop henceforth. To simplify the above optimization and make training feasible, Cuturi [48 introduced entropy regularization, leading the Sinkhorn distance,
1 (T, (T))..
(a, B, D) D) S.L. T3 = arg min (EU((.
where 1(T) = - Et Σx Lig log big denotes the entropy of T. and À is the regularization diag(ass
parameter. The optimal solution for Eqn. ① has the form [48). T3 diag(rs, where
the element-ue wise exponential of the matrix AD. and K1 E R and K2 R are the non neg-tieetive left and right scaling vectors to be obtained by the Sinkhorn fixed point iterations.
Regularization with Priors. The above formulation minimizes the cost of aligning two sequences, however it totally neglects the temporal ordering relationships inherent in videos, failing to leverage the temporal consistency. Typically, the alignment of multiple videos depicting the should constrain the temporal position of one sequence to correspond closely with adjacent temporal positions of another sequence. Perfect alignment would render the transport matrix diagonal, but this strict requirement is impractical for real-world applications. As illustrated in Fig. B)), variations such as earlier commencement of activities, differing action speeds, or non- mon-monttoiic sequences complicate alignment. To address these challenges and achieve optimal alignment while accounting for temporal variations, we introduce two priors into the OT framework. Essentially, there are factors in play (i) optimality which tries to find the best match between frames irrespective of their temporal distance (which may result temporally incoherent alignment), and (ii) the temporal factor which promotes transport between nearby frames only without considering their feature matching. We hypothesize that the optimal solution requires striking a balance between both, and thus propose to enhance the OT formulation by incorporating two specific priors addressing the above factors [46).
activity

---

The first prior, termed the "Optimality Prior', is introduced to effectively manage non monotoni sequences. This prior leverages the transport matrix T as delineated in Eqn. Q which provides a preliminary indication of alignment between two video sequences. This matrix adapts dynamically to reflect the temporal variations observed across the sequences. Our approach uses this dynamic behavior to establish the optimality prior. We want the point representing the most likely alignment according to T to have the highest likelihood, while the assignment probability decays along any perpendicular direction from this center. Specifically, we model this as a Laplace distribution,
i/N - io/NI+ + I - Ja/M
(i( 2b
where do((.))
(2)
2/1/N2 + 1/M2
represents the average distance from (i, to the frame locations (i, and (fo: that correspond to the optimal alignment as indicated by the transport matrix, and b is a scale parameter. Motivated by [46. we incorporate a second prior, termed the 'Temporal Prior', which promotes alignment of one sequence with elements in proximal temporal positions of the other sequence, thereby preserving the overall temporal structure and maintaining consistency in action order. This prior results in an assignment matrix characterized by peak values along the diagonal, with values diminishing perpendicular to the diagonal. Again, this scenario is modeled using a wio-dimensional Laplace distribution, where the distribution along any line perpendicular to the diagonal is exponentially decaying, centered along the diagonal itself::
Ide(6.2)
where de (1, = W - J/MI
Qi 2b
(3)
VI/N² + 1/M2
is the distance from (1,5) to the diagonal. Inspired by (46) we merge these priors as,
(i. & (i.)) + x) Qo(i.j),
where, & serves as a dynamic weight, initially set to 1.0, and progressively reduced to 0.5 during training. This gradual adjustment of e allows the model to adaptively improve its alignment based on the increasing fidelity of the OT predictions. Optimal alignment based on these priors is pictorially depicted in Fig C., D). Note, the I-dimensional alignment in Fig Z C) is for demonstration only, our actual implementation is based on 2-dimensional distributional priors as shown in Fig D))
Background and Redundant Frames. To effectively manage background and redundant frames, we integrate an additional 'virtual frame' within the transport matrix, following [46). This serves as a placeholder for aligning any frame that do not match with the primary sequence, and allows OPEL to explicitly assign these con-contributing frames the virtual frame, as shown in Fig. BB) The augmented transport matrix, now denoted as T E I (N)) (M+1), includes an extra row and column to accommodate the virtual frame. Note, if the likelihood of a frame aligning with any salient frame falls below a predefined threshold, Ç, we assign that frame to the virtual frame.
Training Methodology. While the above formulation sounds promising, devising a differentiable framework to leverage these during training is pivotal. To that effect, following [46], we define terms effectively regularize T. To capture the essence of Qo. I needs be structured to highlight prominent values at locations corresponding to the most probable alignments.
Mo Ma(T) = - ΣΣ
N+1 M+1
Lij
1 is
2
j M
j0
2
where dm =
(5)
dmm
N
+1
+1
5=1
Similarly, to conform to Qt, T is expected to exhibit prominent values along its diagonal, reflecting temporally closely aligned frames; while dfg-diago elements should ideally possess diminished magnitudes. This sort of structural arrangement can be quantitatively assessed using:
N+1M+1
lij
M, = -
(6)
2
M+1 +1
N+
N+I
4 the above 2 equations are combined as a regularizer on the transport matrix, M(T) = e (T) (1 - Φ) MT')). Such a structure is known as the inverse difference moment (IDM) 49 46 To encourage optimal alignment, M()) of the learned î should be maximized. In order to facilitate this and to ensure the smooth assignment of such matches, we define a modified feasible set for T by incorporating two additional constraints into the set U(ar
Similar to Eqn.
(G..c.(a,3 3) { E R N+1
T1M+1 az T 1 IN+1 = (T) 2 &, KL Q) S E2
5

---

where KL(T)) |1 = = ΣØ=1 bsy log fst is the Kullback- Leibler (KL) divergence between T and Q and Q is same as Eqn. (4) but augmented with the virtual frame. So, the regularized Wasserstein distance between X and Y now becomes
4i
E2 [X Y)
min
(T. (T.D).
(8)
TEUL (2
The above optimization can be efficiently solved by considering its dual. As such, we incorporate two Lagrange multipliers, A > 0 and X2 0, to obtain the dual of Eqn. (8) as-
1 A1 (x, Y) (Txinaa D): s.t. Tand = arg min (Txida -) M(T) +22 Q). (8
The optimal Tanna that optimizes Eqn. 9 is diagl- 2-- #. Ke diagl- x2) where K (gise (2 (e]) -uo)) Jyg. = An (xxi)-
3dm dm given by Eqn. and l and v
the dual variables for the two equality constraints T1M+1 a, and T 1N+1 B, respectively. The detailed derivation of this optimal Ax is provided in appendix section A.1
Contrastive Regularization. Incorporating temporal priors into the video aligmment processes often leads to trivial solutions 41 So. following [I we utilize the inntrastive-Inverss Difference
Moment (C-IDM) loss to further regularize the training. This loss is characterized by,
N+1 M+1
(1 - N (i) (1, max (0, As d(i.))) + + d(i (1,5)*
1(x)= = Σ
Σ
(10)
i=1 1=1
where (1. = ()- d(i.j) N(i.,) 1, if i- 5 and 0 otherwise, 6 is predefined window size, À is margin parameter. The preceding C-IDM loss is an intra-video loss. Additionally, we incorporate an inter-video contrastive loss guided by OT to further regularize the training process. Specifically, this novel loss component contrasts pairs of videos based on their similarity as quantified by the OT matrix. We find, (1) (1) = arg max, Tx,d2 and (1) = arg miny . Likewise, Dbes( = arg max; Xx and Ywora (j) = arg min; Ax calculated. Then, the best distance is computed as the average of squared differences between matched pairs, scaled by a temperature factor: dest__dstance =
||= d(
N(.,)) is neighborhood function defined as:
M j=1 |1y, =mea)s))l?). Similarly, the worst distance (+ Ei>1 |2: (1) + To Σタz1 |1y; - a your (1) 112) Finally, the inter-sequence loss is computed using the cross-entropy over the best and worst distances:
+
Σ [ ym 112 M
N
temporature
is: worst_distance
temperature
|eest distance bestt_distancee [worst_ joosst_dstancee
[0]
loss_inter = Feross crooss_nttrop
(11)
Ideally, we want each frame embedding 2;, to align highly to its best match from Y. So the best distance should be as close to 0 possible, the same time, we maximize its distance from the unmatched frame embeddings, and the same holds true for yjs. As a result, our proposed inter-video loss (Eqn. (II) promotes learning disentangled representations. So, the overall loss for OPEL combines the regularized OT loss (Eqn. (9)) with the contrastive regularization terms,
LXPEL(X, Y) = C1 * 2g (X, Y) + C2 * (I(X)= + I(Y)) + C3 * loss jnser.
(12)
Clustering and Key-step Ordering. After learning the embeddings, our goal is to localize the key-steps required for PI.. We frame this problem multi-label graph-cut segmentation (50) The node set V of the graph includes l terminal nodes representing the key-steps and non-terminal nodes corresponding to the number of frames, which are derived from the embeddings produced by the embedder network. Upon constructing the graph, we apply a- e-Exxansion [19] to identify the minimum cost cut, utilizing the results to assign frames we first compute the normalized time for each frame in a video, following []] Subsequently, the temporal instant for each cluster is determined by calculating the average normalized time for frames allocated to that cluster. Clusters are then sequenced in ascending order of their average time, thus outlining the sequence of key-steps of a video. Upon establishing the key-step order for all videos associated with the same task, we generate ranked list based on the frequency at which subjects adhere to a specific sequence. The most commonly observed order is placed at the top of this list. This methodological approach allows us to discern various sequential orders of key-steps of a task.
labels. To deduce the sequential order of key-steps,

---

Experiments and Results
Datasets. In contrast to previous research that predominantly utilized either 1* person or 3rd person viewpoints for PL, we incorporate datasets from both perspectives. For 3rd person view, we utilize established benchmark datasets, namely CrossTask [TT] and ProceL [3]. CrossTask features 213 hours of video footage spanning 18 primary tasks, totaling 2763 videos. ProceL includes 47.3 hours of video from 12 varied tasks, comprising 720 videos. To evaluate the effectiveness of our proposed OPEL framework, we apply it to the 1st -peersn EgoProceLL benchmark
[] which contains 62 hours of egocentric video recordings from 130 subjects engaged in 16 tasks. Detailed information on individual datasets is provided in Table A2 of appendix.
We compute the framewise scores for each key-step separately and then take the mean of the scores over all the key-steps, reporting both the Fl Fl-score and the Intersection over Union (loU). The Fl-score is defined as the harmonic mean of precision and recall. Precision is calculated as the ratio of the number of frames correctly predicted as key-steps to the total number of frames labeled as key-steps. Recall is determined by the ratio of correctly predicted key-step frames to the total number of actual key-step frames. Following the methodology in 2 31 47], we employ the Hungarian algorithm [5 to derive one-to-one mapping between the ground truth and the predictions.
Evaluation. Unless specified differently, we assess OPEL per the current SOTA 1 2
Table I: Results using EgoProcel [1 demonstrate the superior performance of OPEL. The results in bold and underline denote the highest and second-Hi highhsst values in a column, respectively.
EgoProceL
CMU-MMAC 17
EGTRA-GAZZ+ 52
MECCANO EPIC-Tene
54
PC Assembly
PC Disassemhly
Fi
IaU
F1
loU
F1
IoU
F1
loU
F1
InU
F1
loUl
Random
15.7
5.9
15.3
4.6
13.4
5.3
14.1
6.5
15.1
7.2
15.3
7.1
Uniforı
18.4
6.1
20.1
6.6
16.2
6.7
16.2
7.9
17.4
8.9
18.1
9.1
CnC [
22.7
11.1
21.7
9.5
18.1
7.8
17.2
8.3
25.1
12,8
27.0
14.8
GPL-2D 2
21.8
11.7
23.6
14.3
18.0
8.4
17.4
8.5
24.0
12.6
27.4
15.9
UG-13D 2
28.4
15.6
25.3
14.7
18.3
8.0
16.8
8.2
22.0
11.7
24.2
13.8
GPL-w BG 2
30.2
16.7
23.6
14.9
20.6
9.8
18.3
8.5
27.6
14.4
26.9
15.0
GPL-w/o BG 2
31.7
17.9
27.1
16.0
20.7
10.0
19.8
9.1
27.5
15.2
26.7
15.2
OPEL 36.5
18.8
29.5
13.2
39.2
20.2
20.7
10.6
33.7
17.9
32.2
16.9
Experimental Setup. We employ ResNet-50 (pretrained on ImageNet) the embedder network. Inspired by [. we train the embedder using pairs of training vidcos. Within these videos, we randomly select frames and optimize the proposed LOPEL until convergence. The feature extraction is conducted from the Conv4c layer, and we subsequently create a stack of 2 context frames along the temporal dimension. Our video frames are resized to 224x224. The aggregated features are processed through two 3D convolutional layers, followed by a 3D global max pooling layer, two fully-connected layers, and linear projection layer that outputs embeddings of 128 dimensions. All hyper-parameters are listed in Table A1 Our code is provided as part of the supplementary material.
Results on Egocentric View. Table ! presents a comparative analysis between the SOTA techniques and OPEL applied to the large scale egocentric benchmark, EgoProceL. Results from tasks within CMU-MMAC and EGTEA G. have been aggregated and presented (detailed task-wise results are given in Table A4). It is important to highlight that EgoProceL. represents contemporary dataset specifically designed for egocentric procedure learning, thereby limiting the number of applicable approaches for fair comparison. Notably, OPEL outperforms the SOTA across most tasks. This superiority underscores the efficacy of the video representation learning through OT. Specifically, we achieve 22.4% (IoU) and 26.9% (F1) average improvement compared to current SOTA.
Table 2: R, and Fl represent precision, recall, Fl Fcorc.
PL results on third-person datasets B
PruceL
3
Cross Task
[1]
P
R
R
Uniform
12.4
9.4
10.3
8.7
9.8
9.0
Alayre er al. 34
12.3
3.7
5.5
6.8
3.4
4.5
Kukleva et al.
11.7
30.2
16.4
9.8
35.9
15.3
Elhamifar al.
9.5
26.7
14.0
10.1
41.6
16.3
Fried et al. 67 Shen et al.
28.8
47
16.5
31.8
21.1
15.2
35.5
21.0
CnC !
20.7
22.6
21.6
22.8
22.5
22.6
GPL-2D
21.7
23.8
22.7
24.1
23.6
23.8
UG-I3D 2
21.3
23.0
22.1
23.4
23.0
23.2
GPL 2
22.4
24.5
23.4
24.9
24.1
24.5
STEPS 16
23.5
26.7
24.9
26.2
25.8
25.9
OPEL (Ours)
33.6
36.3
34.9
35.6
34.8
35.1
Results on Third-person View. We provide comparison between SOTA and OPEL on two distinct third-person datasets I in Table To ensure consistency in evaluation, we follow the evaluation 22. Once again, we perform better in almost all cases, with
protocol outlined in the SOTA prior arts
7

---

MECCANO Bike Assembly
PC Assembly
Ground
Truth
Random
CnC
GPL+BD
GPL
OPEL
Figure 3: Qualitative results from MECCANO 53 and PC Assembly [ tasks. Each sub-task is color-coded to represent different key-steps, while gray areas signify background elements. Notably, OPEL's performance surpasses that the SOTA networks, attributed to its ability to handle unmatched frames through the integration of virtual frame, thus enhancing alignment accuracy.
SOTA. Note, 32 predominantly allocate frames single key-step, resulting in elevated recall rates but concomitantly diminishing precision, consequently impacting the overall F-score. Additional detailed third-person results from CMU-MMAC (17).
46.2% (F1) average enhancement
ProceL 3
and CrossTask M are given in Table A3 and Table A5 respectively.
Qualitative Results. Fig. illustrates the qualitative PL outcomes of the baselines and OPEL. Higher match with the ground truth our case (the bottom row) depicts the usefulness of OPEL. Additionally, depict the alignment of two sequences in Fig. BB), showcasing accurate alignment despite temporal variations, with correct matches indicating consistent action frame ali alignment and redundancy handling, affirming the reliability of our model.
(A)
CsC
CPL
STEPS
CPEL
[8)
Tast Break Egp
Taat Mo
Cleter
Tol
Taus d.Add
S.ccrre
Tosk Hix contires
Vintal Frana
15-
Whong Alggnnnnt
Cerrect Algemmet
10
Number Training Videos (MECCANO)
Qualeer
Figure 4: (A) Impact of training data quantity on encoder training. (B) Example alignment of two videos with corresponding key-step clusters from the Brownie task (17).
Comparison with Multimodal Models. While we only use videos for training, our results are competitive with models using multiple modalities. On the egocentric EgoProcel dataset, we perform comparably or even better (4 out of datasets) compared the multimodal SOTA model, STEPS [16]. as shown in Table 3 Note, STEPS uses gaze and depth data during training, thus enhancing its
results on EPIC-Tents. We also outperform [47] 34 (Table 2) which use narrations with video.
Table 3: Comparison with models with multimodal input. Note, STEPS [66] uses additional data
(optical flow, gaze, depth) for training, while we use just the visual modality.
CMU-MMAC
EGTEA-GAZEE
MECCANO
EPIC-Tents
ProceL
CrossTask
F1
ToU
Fl
IoU
Fl
IoU
F1
loU
Fl
loU
F1
loU
STEPS [6]
28.3
11.4
30.8
12.4
36.4
18.0
42,2
21.4
24.9
15.4
25.9
14.6
OPEL.
36.5
18.8
29.5
13.2
39.2
20.2
20.7
10.6
34.9
21.3
35.1
21.5
5
Ablation Study
Effectiveness of LOrEL We analyze the effectiveness of the proposed loss by replacing or combining with other SOTA losses used for PL TCC [ LAV 41 and CnC M. Overall, the proposed
LOPEL outperforms previous approaches as shown in Table 4 This enhancement can be attributed
to the flexibility in modeling sequences provided by OT. Furthermore, ablation results on all the loss components of LOPEI. are provided in Table 5 where we show the contribution of each factor individually to analyze their effect on the overall result. Comparing row 3 with row 9, we observe, the priors jointly play a critical role; without them (row 3), the Fl and IoU scores drop by 5 points.

---

Specifically, the optimality prior has a significant impact (~22 point), while the temporal prior affects the score by ~1 point. Similar the combined priors, the intra and inter-video contrastive losses together (row 6 vs row 9) have significant effect (~3.5 points) on the overall performance. The individual effect virtual frame is negligible as only plays a role in case of excessive background frames scenario that is not prevalent in most datasets. Furthermore, due to the IDM structure of (T), the T and Q are similar by formulation. This results in KL(T Q) to be already small. As a consequence, adding the KL divergence as a standalone loss component in the proposed pipeline has minimal impact. Overall, while some loss components may have smaller individual impact, they do contribute to performance improvements, even incrementally. Therefore, our proposed approach incorporates all of them to achieve the best possible results.
Table 4: Comparison of effectiveness of LOFEL with other losses.
CMU-MMAC 17
MECCANO
53
AGTEA-GAZE+
[2
PC Assembly
P
Fl
loU
P
F1
IoU
P
F1
IoU
P
Fl
IoU
TCC + PCM 8
18.5
19.7
9.5
15.1
17.9
8.7
17.5
19.7
8.8
19.9
21.7
11.6
LAV TCC PCM
18.8
19.7
9.0
13.4
15.6
7.3
16.4
18.6
7.5
21.6
21.1
10.8
LAV PCM AI
20.6
21.1
9.4
14.6
17.4
7.1
17.4
19.1
8.0
21.5
22.7
11.7
TC3I+ + PCM (CnC) I
21.6
22.7
11.1
15.5
18.1
7.8
19.6
21.7
9.5
25.0
25.1
12.8
OT TCC
28.8
32.6
15.6
25.2
34.5
17.5
22.6
26.7
11.2
27.8
28.2
15.6
OT + LAV
30.2
34.7
16.8
26.7
36.2
18.8
23.1
27.8
12.4
30.2
30.9
16.8
OT + TCC LAV
27.6
31.2
15.3
23.8
33.6
16.1
21.8
25.4
10.5
28.1
28.4
14.7
OPEL (Ours)
32.8
36.5
18.8
28.9
39.2
20.2
24.3
29.5
13.2
32.5
33.7
17.9
Table 5: Analysis the impact of each term in LOPEI. on the overall performance.
Intra-
Inter-
KL
Temporal Prior
Optimality Prior
Virtual Frame
MECCANO [53]
CMU-MMAC [17]
Video
Video
Divergence
IoU
IoU
34.1
14.2
30.5
12.9
33.3
13.5
29.6
12.3
34.6
14.9
31.3
13.7
36.1
18.4
33.8
16.4
38.6
19.6
18.2
V
4
35.8
16.1
32.6
14.4
V
37.0
18.3
34.1
16.5
4
38.1
19.1
35.2
17.3
/
39.2
20.2
36.5
18.8
Table 6: Analysis of different clustering algorithms.
CMU-MMAC
ACTEA-GAZE-
MECCANO
EPIC-Tents
ProceL
Cross
F1
loU
loU
F1
IoU
F1
loU
loU
loU
Random
15.7
5.9
15.3
4.6
13.4
5.3
14.1
6.5
15.1
7.2
15.3
7.1
OT + K-means
34.2
13.5
23.9
8.8
31.8
19.6
16.2
7.9
24.8
12.5
27.4
14.4
+ SS
34.8
13.2
23.7
19.5
17.2
12.8
28.0
OPEL
36.5
18.8
29.5
13.2
39.2
20.2
20.7
10.6
33.7
17.9
32.2
16.9
Choice of clustering algorithm. We replace the proposed clustering approach with K-means and subset selection (SS). The results Table show that OPEL performs the best, highlighting the effectiveness of OT with graphcut segmentation.
present the results of OPEL alongside baseline models, with varying k. Note, we obtain best results with k=7, and the performance drops sharply as k goes from 7 to 10 or higher. This observation consistent with all the other SOTA methods on the same datasets [ 16). We hypothesize that k=7 works best as it is the optimal number of clusters considering the average number of distinct key-steps (subtasks) of the datasets. For example, for PC Disassembly, although the ground-truth (GT) number of steps is 9, steps are quite similar (remove hard disk, remove motherboard, remove RAM), effectively making them quite close in the feature space. This results in k=7 being better estimation of the cluster number with distinct steps. Note,
Number of key-steps. In Table
Table 7: Results obtained for different k.
PC Assembly
PC Disassembly
k
R
F1
loU
R
F1
loU
7
35.0
33.7 18.0
35.4
32.2
16.7
10
27.8
24.3
12.1
28.5
24.8
10.5
12
25.2
24.1
11.8
26.7
24.2
9.7
15
27.6
25.8
12.2
25.2
23.6 9.1

---

this demarcation of subtasks (hence, number of clusters) is subjective and varies from dataset to dataset as well as from task to task; as some may consider semantically similar tasks (e.g. pouring oil vs water) to be one subtask, while others may consider it different. As k becomes larger than the actual distinctive number of clusters, each subtask gets split into multiple clusters with very similar embeddings, which upon comparison with GT leads to inferior results.
Impact of Training data Quantity. Fig. ((A) presents the results from varying the number of training videos on MECCAÑO, aiming to evaluate OPEL's performance with respect to video count. We consistently outperform other SOTA methods. Overall, the performance improves with more training data, however, even with just few (2-5) videos of a task, we reach the upper-limit of other methods using full dataset, as shown in Fig. [(A). Additional ablation results including choice of distribution as priors and hyperparameters \1. 22 are provided in appendix [A.8]
Comparison with AS methods. PL. and action segmentation (AS) are related but not the same. PI., when applied to a set of instructional videos depicting the same task, involves two primary steps: (i) assigning each video frame to one of the k key-steps (including background elements), and (ii) determining the logical sequence of these key-steps necessary to complete the task. As illustrated in Fig. T PL addresses multiple videos of a given task, enabling the identification of repetitive key-steps across these videos 2 In contrast, AS 4 focuses on single video, thereby lacking the ability to discern repetitive sey-steps acToss different videos.
Despite the differences between PL and AS, we compare our approach against existing SOTA unsupervised AS models and present the results in T Our model demonstrates significant performance improvement compared to these works. In E (56). authors report high recall score for Cross Task as it assigns majority of the phenomenon also reported by [2). While achieving high recall is important for ensuring that most positive instances are correctly identified, it can result in greater number of false positives, which in turn lowers precision and leads to undesirable results. Therefore, it is crucial to balance recall with precision to develop an effective model. This balance is reflected in the superior performance of our model, as evidenced by the Fl-score results across various benchmarks. Note in the Table S our approach is compared with SOTA unsupervised AS methods for only third-person datasets, as these do not report any result on egocentric datasets.
Table 8: Comparison with SOTA unsupervised AS methods. Note '_' denotes that the authors have not provided any data on those metrics,
AS benchmark
ProceL [3]
Cross
[
P
R
F1
R
Fl
JointSeyFL [3
29.8
frames to a single key-step-
Elhamifar al.
9.5
26.7
14.0
10.1
41.6
16.3
Fried et al. 37
28.8
Shen ef al.
47
16.5
31.8
21.1
15.2
35.5
21.0
Dvomik a/.
155
25.3
StepFormer [56]
18.3
28.1
21.9
22.1
42
28.3
OPEL (Ours)
33.6
36.3
34.9
35.6
34.8
35.1
6 Conclusion
In this study, we have introduced a novel approach for procedure learning leveraging optimal transport, enhanced by temporal and distributional regularizations to improve the alignment of key-steps across multiple video instances. Our method addresses inherent limitations in current SOTA techniques that primarily rely on frame-to- trame-to-ramme mappings and assumptions of monotonic alignment, which do not optimally utilize temporal information. We observe improvement of 22.4% in loU and 26.9% in Fl scores on the EgoProceL dataset, outperforming the current maate-of-hhe- methods. Similarly, in third-person video benchmarks, such ProCel. and CrossTask, our framework achieves an average F1 score enhancement of 46.2% over existing methods. These advancements underscore the potential of OT guided learning in handling complex video procedure learning tasks. A limitation of the proposed OPEL. framework is the assumption that subjects utilize similar objects for identical key-steps, which may introduce inaccuracies when dissimilar objects are employed in the execution of these steps. Future work will focus on exploring the integration of additional contextual and semantic features within the OT framework to further refine the procedure learning process. Moreover, extending this framework to other domains of video understanding could provide valuable insights into the general applicability of optimal transport in video analysis tasks.
A Acknowledgments
This work was supported in part by the Center for Co-design of Cognitive Systems (CoCoSys), one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, by the SRC, the National Science Foundation, Intel Corporation, the DoD Vannevar Bush Fellowship, and by the U.S. Army Research Laboratory.
10

---

References
[1] Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In European Conference on Computer Vision, pages 657-675, Springer, 2022.
[2] Siddhant Bansal, Chetan Arora, and CV Jawahar. United we stand, divided we fall: Unitygraph for unsupervised procedure learning from videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6509-65005 6519, 2024.
[3] Ehsan Elhamifar and Dat Huynh. Self-supervised multi-task procedure learning from instructional videos. In Computer Vision ECcon 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XVII 16, pages 557-573, Springer, 2020.
[4] Sateesh Kumar, Sanjay Haresh, Awais Ahmed, Andrey Konin, M Zeeshan Zia, and Quoc-Hup Huy Tran. Unsupervised action segmentation by joint representation learning and online clustering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20174-20185, 2022.
[5] He Zhao, Isma Hadji, Nikita Dvoriik, Konstantinos Derpanis, Richard P Wildes, and Allan Jepson. P3iv: Probabilistic procedure planning from instructional videos with weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2938-2948, 2022.
[6] Yicheng Qian, Weixin Luo, Dongze Lian, Xu Tang, Peilin Zhao, and Shenghua Gao. Svip: Sequence verification for procedures in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19890-19902, 2022.
[7] Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid. TI; dw? summarizing instructional videos with task relevance and cross-modal saliency. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
[8] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1801-1810, 2019.
[9] Zwe Naing and Ehsan Elhamifar. Procedure completion by learning from partial summaries. In British Machine Vision Conference, 2020.
[10] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[11] Dimitri Zhukov, Jean- Bant-Ba Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545, 2019.
[12] Jun Li and Sinisa Todorovic. Set-constrained viterbi for set-supervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10820-10829, 2020.
[13] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetwork-vitee A framework for weakly supervised video learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 7386-7395, 2018.
[14] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEFE/VF Conference on Computer Vision and Puttern Recognition, pages 3546-3555, 2019.

---

[15] Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal alignment and sycle-consisteney. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11068-11077 1007, 2021.
[16] Anshul Shah, Benjamin Lundell, Harpreet Sawhney, and Rama Chellappa. Steps: Self- supervised key step extraction and localization from unlabeled procedural videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10375-10387, 2023.
[17] Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the carnegie mellon university multimodal activity (cmu- mmac) database. 2009.
[18) Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[19] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on puttern analysis and muchine intelligence, 23(11):1222-1239 2001.
[20] Basura Fernando, Hakan Bilen, E. Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[21] Hsin- Ying Lee, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. International Conference on Computer Vision (ICCV), 2017.
[22] Ishan Misra, C. I. Zitnick, and M. Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In European Conference on Computer Vision (ECCV), 2016.
[23] Jin woo Choi, Gaurav Sharma, S. Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. In European Conference on Computer Vision (ECCV), 2020.
[24] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Computer Vision and Puttern Recognition (CVPR). 2019.
[25] Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In International Conference on Computer Vision (ICCV), 2019.
[26] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Workshop on Large Scale Holistic Video Understanding, ICCV, 2019.
[27] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. In International Conference on Machine Learning (ICML), 2015.
[28] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Neural Information Processing Systems, 2016.
[29] Dahun Kim, Donghyeon Cho, and In-So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI Conference on Artificial Intelligence, 2019.
[30] Donglai Wei, Joseph Lim, Andrew Zisserman, and William Freeman. Learning and using the arrow of time. In Computer Vision and Patlern Recognition (CVPR), 2018.
[31] Ehsan Elhamifar and Zwe Naing. Unsupervised procedure learning via joint dynamic summarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6341-6350, 2019.
[32] Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall. Unsupervised learning of action classes with continuous temporal embedding. In Proceedings of the IEEE/VE Conference on Computer Vision and Pattern Recognition, pages 12066-12074, 2019.
[33] Rosaura G. VidalMata, Walter J. Scheirer, Anna Kukleva, David Cox, and Hilde Kuehne. Joint visual-temporal embedding for unsupervised learning of actions in untrimmed sequences. In IEEE/CVF Winter Conference Applications of Computer Vision (WACV), 2021.
12

---

[34] Bean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivie, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEFE conference computer vision and pattern recognition, pages
2016.
[35] Dima Damen, Teesid Leclasawassuk, Osian Haines, Andrew Calway, and Walterio Mayol- Cuevas. You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In British Machine Vision Conference (BMVC), 2014.
[36] Hazel Doughty, Ivan Laptev, Walterio Mayol-Cuevas, and Dima Damen. Action modifiers: Learning from adverbs in instructional videos. In Computer Vision and Pattern Recognition (CVPR), 2020.
[37] Daniel Fried, Jean-Baptiste Alayrac, Phil Blunsom, Chris Dyer, Stephen Clark, and Aida Nematzadeh. Learning segment actions from observation and narration. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2569-2588, 2020.
[38] Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. What's cookin'? ? interpreting cooking videos using text, speech and vision. In HLT-NAACL, 2015.
[39] Shoou-I Yu, Lu Jiang, and Alexander Hauptmann. Instructional videos for unsupervised harvesting and learning of action examples. In ACM International Conference on Multimedia, 2014.
[40] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255, 2013.
[41] Sanjay Haresh, Sateesh Kumar, Huseyin Coskun, Shahram N Syed, Andrey Konin, Zeeshan Zia, and Quoc-Huy Tran. Learning by aligning videos time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5548-54 5558, 2021.
[42] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1 1134 1141. IEEE, 2018.
[43] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In European conference on computer vision, pages 527-544. Springer, 2020.
[44] Paul-Edo -duuard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4938 4947, 2020.
[45] Mathieu Serrurier, Franck Mamalet, Alberto González-Sanz, Thibaut Boissin, Jean-Michel Loubes, and Eustasio Del Barrio. Achieving robustness in classification using optimal transport with hinge regularization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 505-514, 2021.
[46] Weizhe Liu, Bugra Tekin, Huseyin Coskun, Vibhav Vineet, Pascal Fua, and Marc Pollefeys. Learning to align sequential actions in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2181-2191, 2022.
[47] Yuhan Shen, Lu Wang, and Ehsan Elhamifar. Learning to segment actions from visual and language instructions via differentiable weak sequence alignment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10156-10165. 2021.
[48] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[49] Fritz Albregtsen et al. Statistical texture measures computed from gray level coocurrence matrices. Image processing laboratory, department of informatics, university of oslo, 5(5), 2008.
13

---

[50] Dorothy M Greig, Bruce T Porteous, and Allan H Seheult. Exact maximum a posterioni estimation for binary images. Journal of the Royal Statistical Society Series Statistical Methodology, 51(2):27-277, 1989.
[51] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-99)) 97, 1955.
[52] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference computer vision (ECCV), pages 619-635, 2018.
[53] Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1569-1578. 2021.
[54] Youngkyoon Jang. Brian Sullivan, Casimir Ludwig, lain Gilchrist, Dima Damen, and Walterio Mayol-Cuevas. Epic-tent: An egocentric video dataset for camping tent assembly. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[55] Nikita Dvomik, Isma Hadji, Hai Pham, Dhaivat Bhatt, Brais Martinez, Afsaneh Fazly, and Allan Jepson. Flow graph to video grounding for weakly-supervised multi-step localization. In European Conference Computer Vision, pages 319 -3-3... Springer, 2022.
[56] Nikita Dvomik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard Wildes, and Allan D Jepson. Stepformer: Self-supervised step discovery and localization in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18952-18961, 2023.
[57] R. Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 74(4):402 405, 1967.
[58] A. Borobia and R. Cantó. Matrix scaling: A geometric proof of sinkhorn's 's theorem. Linear Algebra and its Applications, 168:1-8, 1998.
14

---

A Appendix
A.1 Derivation of the Optimal Transport Matrix (Tx,.)a)
In this section, obtain the optimal transport matrix Tx Xx that optimizes the OT Eqn. Note, all the notations used here are same as section of the main manuscript. We start with Eqn. of section
A1 (X Y) == (TxiAa, D), s.t. Thid = arg min (a.(
- Q),
From the duality theory, we know, for each pair Eı, E2 in Equation 8 corresponding pair a > 0, 2 0 exists, such that 18 Ei.ez (X, Y) 22 (X, Y) for the pair (X
Tan is the optimal transport matrix, so it optimizes-
- AT Q) subject to T1M+1 = a, 1N-1 =B, TER. M+1x4+1
min
(A1)
Given that both the objective function and the feasible defined in Equation Al are convex, the existence and uniqueness of the optimal transport matrix Ax, are guaranteed. To derive this optimal matrix, the analysis begins by taking the Lagrangian of Equation Al as-
N+1 M+1 Σ
L(T
Σ
dijtys Artss
1
+ Aatis log
ty Gis
1=1
F=1
( i
N+T
M+T +1
j
2
dm +1
+ 1 T (T1M+1 - a) U.
(T
1N+1 - B), (A2)
where dm 1of N+1
io and jo have their same meaning the main paper, i.e. they correspond to the optimal assignment locations jo, and (i.:j) as provided by the transport matrix, and μ and v are the dual variables for the two equality constraints fl and TT 1 1 B, respectively. Taking the derivative of LT,, pa, v) w.r.. try yields-
(--1
OL v,,) dij A1 Otig
+
(N
(i-ig 2
M41
N+1
+
M-11
+1
tij
+ A2 log
+ A2 + us + + (A3)
9i5
Let, 1/ = ~1 ( xhr +1
Setting L(T,, p, O, we get-
hi Dije - i #3 (5 dis) -
(A4)
Let, K = =9 (52) -ds) then we get,
di by Kije - 1-
tg =
15

---

Thus,
Tx, dbag(= x/2 Ke ding(- - xz)
(A5)
Every element of the matrix is strictly positive, as we take the element- wiie exponential to obtain each Ki and (ij 0. As per the Sinkhorn's theorem (Theorem A), there exist diagonal matrices (Ka) and (iag((g) with strictly positive diagonal elements, such that diag (K) K diag(K2) is member of the set U (a, (3), with K1 IR++1 and K2 RM++1 This product matrix is unique, and the diagonal matrices are also uniquely determined, up to a scalar factor.
Theorem A 157 5 For any (N + 1) (M + 1) matrix A with all positive elements, diagonal matrices B1 and B2 exist such that B belongs to (a. (3). Both B1 and Bg possess strictly positive diagonal elements and unique up positive scalar factor.
The optimal A2 of Equation (S in (ox, (3) mirrors the form of diag (k) 1) K diag(K2). thereby constituting the unique matrix in U(a (3) that represents a rescaled version K. We efficiently compute the scaling vectors Ni and Kig, also unique up to scaling factor, using the Sinkhorn-Knopp iterative matrix scaling algorithm-
K
3
f-
KTKi In this paper, only 20 iterations used, as limited number of iterations has been shown to effectively converge in previous studies [48
A.2
Hyper-parameter Settings
Table A1 lists the hyper-parameters used for OPEL.
Table Al: Hyper-parameter settings for OPEL.
papee-parrmeter
Value
No. of key-steps (k)
No. of sampled frames (N, M)
32
No. of epochs
10000
Batch Size
2
Learning Rate
10
Weight Decay
10
Window size (6)
15
Laplace scale parameter (b)
3.0 (MECCANO, EPIC-Tents PC Assembly)
Laplace scale parameter (b)
2.0 (for all other datasets)
Temperature
0.5
A
N+M
X²
N.14N M
4.0
Margin (Λa)
2.0
Threshold for virtual frame (
2
N-
No. of context frames
2
Context stride
15
Embedding Dimension Optimizer
128
Adam
C1
Now 0.5
C2
Coefficient for loss inter (cs) Maximum Sinkhom Iterations
0.0001
20
A.3 Compute Resources for Experiments
For our experiment, we require adequate computing resources to effectively train our models. We utilize a single Nvidia A40 GPU, but its full RAM is not required. The GPU memory is
16

---

dependent on batch size (bs). For a bs of 2, GPU equipped with approximately 12GB of memory is sufficient for our purposes. Training time depends dataset size and number of epochs (we used 10000). The above configuration allows us to process a dataset consisting of 15-20 videos (e.g. PC assembly/MBCCANO domain) in around 12 hours. With these computing resources in place, we conducted our experiments effectively, ensuring optimal performance and reliable outcomes.
A.4 Detailed Statistics of Dataset
In Table A2 we provide the statistical analysis for each of the 16 tasks within the EgoProceL dataset M. NN denotes the total number of videos, while represents the number of key-steps for each task. My signifies the count of unique key-steps, and In denotes the number of annotated key-steps for the nh video. Following the approach outlined in reference
we compute the subsequent metrics:
Foreground Ratio: This measure indicates the proportion of the total duration occupied by key-steps in relation the overall video duration. It assists in gauging the prevalence of background actions within a task. The foreground ratio inversely correlated with the presence of background activity and is determined as:
N m=
F
(A6)
Here, & and t represent the duration of key-steps and the video for the nh instance, respectively. The foreground ratio F ranges from 0 to 1. The higher the value of F indicates minimal background actions.
Table A2: Statistics of the EgoProceL dataset across different tasks,
Videos
Key-steps
Foreground
Missing
Repeated
Task
Count
Count
Ratio
Key-steps
Key-steps
PC Assembly [
14
9
0.79
0.02
0.65
PC Disassembly
15
9
0.72
0.00
0.60
MECCANO (Toy Bike Assembly) [53
20
17
0.50
0.06
0.32
Epic-Tents (Tent Assembly) [54]
29
0.63
0.14
0.73
CMU-MMAC [17
Brownie
34
9
0.44
0.19
0,26
Eggs
33
0.26
0.05
0.26
Pepperoni Pizza
33
5
0.53
0.00
0,26
Salad
34
0.32
0.30
0.14
Sandwich
31
4
0.25
0.03
0.37
EGTEAGAZE+ [52
Bacon and Eggs
16
11
0.15
0.22
0.51
Cheese Burger
10
10
0.22
0.22
0.65
Continental Breakfast
12
0.23
0.20
0.36
Greek Salad
10
4
0.25
0.18
0.77
Pasta Salad
19
0.25
0.19
0.86
Hot Box Pizza
6
8
0.31
0.13
0.62
Turkey Sandwich
13
0.21
0.01
0.52
Missing Key-steps: This metric quantifies the number of omitted key-steps in each video. is defined as:
Σ m=
M =
(A7)
KN
The range of M between 0 and It aids in assessing the feasibility of completing task despite certain steps being skipped.
17

---

Repeated Key-steps: This metric assesses the occurrence of repeated key-steps across multiple videos. is expressed as:
R
m=
Un
(A8)
2x=1 In
The range of R varies between 0 and 1. Higher values of R indicate greater recurrence of key-steps across videos. OPEL considers these repetitions, to demonstrate better performance.
A.5 Third-Person Video Perspective
In this comparison, we assess the outcomes of training OPEL on diverse perspectives from CMU- MMAC [17) Table A3 depicts the F1-Score and loU scores per frame for exocentric views. We conducted our experiments on exocentric videos and achieved promising results. Through rigorous testing and analysis, our model demonstrated strong performance when trained and evaluated this particular perspective, The obtained outcomes not only validate the effectiveness of our approach but also underscore its applicability real-world scenarios involving both egocentric and exocentric video data.
Table A3: Third-person view results using diverse perspectives from CMU-MMAC [7]. Our findings demonstrate improved outcomes utilizing OT egocentric well as third-person videos, emphasizing their efficacy. P, R, and F denote precision, recall, and F-score, respectively.
View
P
R
F1
IoU
TP (Top)
29.0
42.0
34.0
17.5
TP (Back)
30.7
43.9
35.9
19.6
TP (LHS)
38.3
52.7
44.0
24.3
TP (RHS) 31.8
42.8
36.2
18.4
A.6
Quantitative results of OPEL on different subtasks across both ego- and exo-datasets
We report the results for all subtasks within the egocentric datasets in Table A4 such as CMU-MMAC 17 and EGTEA-GAZE+
[52]. as well as for various third-person exocentric videos datasets like ProceL. 3 and CrossTask [11 in Table A5 Our analysis encompasses comprehensive evaluations across these diverse datasets, providing insights into the performance of our model across different perspectives and scenarios. These results offer holistic understanding of the capabilities and effectiveness of our approach in handling varied vidco types and tasks, contributing significantly to the advancement of research in procedure learning and related domains.
Table A4: Results on individual subtasks of egocentric datasets.
(a) EGTEA-GAZE+ 52
Bacon Eggs
Cheeseburger
Breakfast
Greek Salad
Pasta Salad
Pizza
Turkey
F1
IoU
F1
loU
Fl
loU
Fl
IoU
F1
loU
Fl
loU
Fl
IoU
26.7
10.7
33.6
14.3
31.4
14.1
33.5
17.7
26.1
10.7
31.7
14.4
23.6
10.2
(b) CMU-MMAC 17
Brownie
Eggs
Pizza
Salad
Sandwich
F1
loU
F1
IoU
F1
loU
F1
loU
F1
loU
34.0
17.1
28.9
13.3
37.1
20.6
41.6
22.1
41.0
20.9
18

---

Table A5: Results on individual subtasks of Third Third-person exocentric datasets.
(a) Procel.
3
Clarinet
PB&J Sandwich
Salmon
Jump Car
Toilet
Tire Change
IoU
F1
IoU
IoU
F1
IoU
IoU
F1
IoU
34.5
21.0
36.6
22.7
37.3
23.2
31.3
18.7
31.6
18.8
35.5
21.7
Tie-Tie Tie
Coffee
iPhone Battery
Repot
Plant
Chromescast
CPR
F1
IoU
F1
IoU
F1
IoU
F1
IoU
F1
loU
F1
IoU
38.3
23.9
32.6
19.7
34.4
20.8
34.4
20.9
35.8
22.0
36.7
22.7
(h) CrossTask
40567
16815
23521
44047
44789
77721
F1
IoU
F1
IoU
F1
IoU
F1
IoU
F1
IoU
F1
IoU
35.0
21.8
37.4
23.1
34.4
20.9
35.1
21.5
32.6
19.7
38.7
24.3
87706
71781
94276
53193
76400
91515
F1
loU
F1
IoU
F1
loU
loU
F1
IoU
F1
loU
33.4
20.2
34.6
21.1
37.5
23.3
35.5
21.8
37.0
22.9
34.2
20.8
59684
95603
105253
105222
109972
113766
F1
loU
F1
IoU
F1
loU
F1
loU
F1
IoU
F1
loU
35.6
21.9
30.9
18.4
34.1
20.7
35.2
21.6
35.7
22.0
34.9
21.3
A.7
Additional Applications
Learning from multiple videos the same task opens up numerous potential applications. Firstly, in monitoring procedures, a system trained to recognize key steps identify deviations variations when a new person performs the task. Secondly, in guidance systems, such a system can detect the current step and suggest the next steps needed for task completion. Thirdly, in automated systems, this framework enables robotic systems to autonomously learn key steps through observation, allowing them to perform tasks independently in future instances without human assistance.
In terms of cross-modal transfer within videos, the capability align related videos without supervision allows for the transfer of annotations other modalities from one video to another. For example, text annotations can applied to an entire dataset related videos by labeling just one. Additionally, temporal modalities like sound can be transferred between videos; for instance, the sound of pouring liquids can transferred purely based on visual cues.
Moreover, fine-grained retrieval within videos can be achieved by using nearest neighbors, enabling the retrieval of specific frames that depict various scenarios. Anomaly detection is possible by observing deviations in the video trajectories within the embedding space, helping to identify unusual activities. This ensures the proper sequence of tasks, such as jacking up car before accessing the wheel during tire change.
A.8
Additional Ablation Studies
A.8.1
Distribution for Optimality and Temporal Priors
Instead of using the Laplace distribution, we also tested Gaussian mean distribution as described Eqn. A9 with mean μ and variance to accommodate temporal variations. Additionally, we evaluated a Uniform distribution as described in Eqn. A10 We conducted ablation experiments with various hyperparameters and summarize the best performances Table A6 However, the Laplace distribution consistently outperformed these alternatives, Icading us adopt for our experiments.
19

---

y=f(x)
0.
Laplace Gausssian
Uniform
0.2
data
-2
0
2
Figure A1: Importance choosing Laplace distribution as a prior.
(x (x-1)²
11)2
Q(i,)) - -N(;;
exp
(A9)
V2n0²
20
((.,)) (z;
-10-
P-t
f bb
(A10)
otherwise.
Table A6: Ablation on the choice of distribution function for optimality and temporal priors
EgoProceL
Distribution CMU-MMAC
MECCANO
PC Assembly
PC Disassembly
F1
IoU
F1
IoU
F1
IoU
IoU
Uniform
31.3
15.2
28.9
13.8
26.3
13.5
27.4
14.2
Gaussian
35.1
18.3
33.8
17.3
29.0
15.3
30.1
16.5
Laplace
36.5
18.8
39.2
20.2
33.7
17.9
32.2
16.9
To analyze the superior performance of Laplace as prior, we plot the distributions in Fig. A1 we use the same distribution for optimality as well as temporal priors.
Note,
For the optimality prior, the x-axis is the difference of frames in the feature space (I-d representation for illustration purposes), and the y-axis denotes the corresponding probability of alignment. We want the point representing the most likely alignment (as per T') have the highest likelihood, while the assignment probability should exponentially decay further away. The graph clearly shows that the Laplace distribution captures this behavior suitably compared Uniform and Gaussian. Similarly, for the temporal prior, the x-axis denotes the temporal distance between the frames, and the y-axis alignment. The graph shows that Laplace distribution facilitates alignment of the frames when they are temporally aligned (close center), and its long tail distribution enables better correlation of non-monotonic frames compared to Gaussian or Uniform. As result, shown in Fig. Al even far away locations from the center (temporally distant frames). alignment is possible if indeed these frames have a high match feature-wiss the Laplace temporal prior provides non probability that far away frame (due to its long tail) unlike other distributions and the optimality prior gives a large score (due to feature match). resulting in improved handling of non-monotonicity.
denotes the corresponding probability
this case,
A.8.2 Hyperparameters X and \/2
We present additional experiments concerning the hyperparameters used in our model, specifically À and Xg in Eqn. 9 We assess their impact on the EgoProceL dataset. According to Table A7 we find
20

---

that setting À (N+M) and X2 = M. achieves the best overall performance, where N and M 4.0
denotes the number of sampled frames. Therefore, adopt these values for the experiments reported our paper. Additionally, the results indicate that our approach maintains similar accuracy across different combinations of \ and X2, demonstrating robustness to the choice of hyperparameters.
Table A7: Ablation study for hyperparameters and X2
EgoProceL
Hyperparameter
Value
CMU-MMAC
MECCANO
PC Assembly
PC Disassembly
F1
loU
F1
loU
F1
loU
F1
IoU
0.2
35.2
17.7
38.1
18.8
32.7
17.1
31.6
15.7
(N+M)
X1
36.5
18.8
39.2
20.2
33.7
17.9
32.2
16.9
(N+M)
35.8
18.1
38.6
19.3
32.5
16.7
31.3
15.6
(N+M)
M. 4.0
34.6
16.9
36.9
17.8
31.1
15.4
29.8
14.2
X2
0.I+N=M
36.5
18.8
39.2
20.2
33.7
17.9
32.2
16.9
4.0
N.M 4.0
35.3
17.6
37.6
18.9
32.5
16.7
30.8
15.6
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main claims of the paper, including the introduction of a novel approach, OPEL, for procedure learning, addressing limitations in current techniques, and demonstrating remarkable efficacy through empirical results on benchmark datasets. The claims are supported by empirical results showing significant improvements over existing methods, and the limitations and future directions are also mentioned, providing comprehensive overview of the paper's ' contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses limitations of the proposed OPEL framework, notably the assumption that subjects utilize similar objects for identical key-steps, which may introduce inaccuracies when dissimilar objects are employed. Additionally, the paper mentions the need for future work to explore the integration of additional contextual and semantic features within the framework to refine the procedure learning process further. These discussions demonstrate the authors' awareness of the limitations of their work and their intention to address them in future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
22

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper includes theoretical results, such as the introduction of the OPEL framework and its associated assumptions. Additionally, the empirical results are supported by theoretical underpinnings regarding optimal transport theory. The assumptions are clearly stated in the text, and the proofs, if any, are likely provided in either the main paper or supplementary materials, ensuring completeness and correctness.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper includes detailed descriptions of the experimental setup, including the datasets used, evaluation metries, and any preprocessing steps applied. Additionally, describes the methodology and hyperparameters used to conduct the experiments, ensuring that other researchers can replicate the results. Our code is provided as part of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
23

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The supplementary material accompanying the paper includes the necessary code for reproducing the main experimental results. Detailed instructions provided within the folder guide users on executing the code to replicate the experiments, including accessing and preparing the necessary data. This encompasses instructions on accessing raw data and preprocessed data utilized in the experiments. Additionally, the scripts are provided to reproduce all experimental results for both the new proposed method and baselines.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (htttss://nips.. cc/
pub Dublic//n gwwdgguidccecooo Caalee//oooSSSuii
ssionPon aisionnooicy olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/ publ lllllic////
Codee///ooeSubrrr ss Peiiionolicee for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper provides all necessary training and test details, including datasets, data splits, hyperparameters, optimizers, etc. The hyperparameter settings for understanding
24

---

the experimental setup and interpreting the results are mentioned in Table 8 in the supplementary section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We didn't perform any specific significance tests. However, inline with the SOTA works for procedure learning which we compare to 2 and for the experimental setup used in this paper. Note, we repeated experiments for multiple runs and obtained consistent results across all datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma errоr bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In the appendix section A.3 detailed information regarding the computational resources utilized is provided, including CPU and GPU hours. Specifically, we utilize single Nvidia A40 GPU, but its full RAM is not required. The GPU memory is dependent on batch size (bs). For of a GPU equipped with approximately 12GB of memory is sufficient for our purposes. Training time depends on dataset size and number of epochs (we used 10000). The above configuration allows us to process a dataset consisting of 15-20 videos (e.g. PC assembly/MECCANO domain) in around 12 hours.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
nnes?
Answer: [Yes]
Justification: Our research adheres to the NeurIPS Code of Ethics as we utilize publicly available datasets for our experiments. We ensure that all data used in study complies with ethical standards and has been obtained following appropriate protocols. Additionally, we maintain anonymity and confidentiality in accordance with applicable laws and regulations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: We anticipate negative societal impact of the work, however it could have positive impact by improving robotic applications (which we discuss as motivation in the introduction).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
26

---

Answer: [NA]
The paper does not involve the release of data or models that have high risk for misuse. Therefore, no safeguards were necessary for the responsible release of such resources.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper properly credits the creators or original owners of assets such as code, data, and models. All relevant citations are provided, and the license and terms of use for each asset are explicitly mentioned and respected. Additionally, the authors have ensured that any usage assets complies with the specified licenses and terms of use.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers saperswitheode com. dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pan se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does introduce any assets, therefore documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
27

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
The paper does not involve crowdsourcing experiments or research with human subjects. Therefore, there are no instructions given to participants, screenshots, or details about compensation to include.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
The paper does not involve crowdsoureing experiments or research with human subjects. Therefore, there are potential risks incurred by study participants, disclosure of risks, or need for Institutional Review Board (IRB) approvals.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "33Vw6G4gVR", "reviewer": "Reviewer_XDhK", "summary": "OPEL is a novel technique for Procedure Learning. Procedure learning is the task of finding key steps in an action (such as cooking brownie) and aligning the videos based on these key steps. \nOPEL proposes to use the optimal transport distance between the two videos as the similarity metric rather than direct frame by frame mapping or assuming a strict monotone mapping.\nThis work also proposes two regularizers to incorporate priors about monotone mapping and about increasing the correspondence between nearby frames.\nThe results on both egocentric and 3rd person videos show significant improvement upon the previous SOTA tasks. \nThe ablation studies analyze the effect of each of the losses and regularizers. \nIt seems having the temporal prior and the inter-intra cross entropy are the most important factors in terms of f1 accuracy.", "questions": "Why do you think the number of steps makes such a large difference in table 7? going from 7 to 10 drops the performance significantly. another anomaly is 12 being worse than both 10 and 15.", "limitations": "not explicitly.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a sound and novel technique for procedure learning. Using optimal transport between frames significantly improves the results and add leniency when the frames are not exact matches temporally. This is a common happenstance in real world scenarios since not all the steps of a procedure need to happen in chronological order. Two sub procedures can be interchangeable and using optimal transport with only a regularizer on temporal monotony is can account for this. \n\nThe experiments are sufficiently evaluated on several datasets. Over all the datasets this method out performs the prior work. \n\nThe manuscript is written in a detailed fashion and the full information is provided for sake of replicability. Also the ablation study provided improves the clarity of what is important in this method.", "weaknesses": "There are many factors added together to make this method work. Although many of them are common in other methods as well, but the method and also the ablation seem to suggest that some of the losses may be redundant. It is not clear if the difference between some of the lines in table. 5 is statistically significant.\n\nPage 6. is written in such a compact way that it's not easy to read and follow. The indices such as i,j can be probably dropped. Words such as temperature in the formula can be probably replaced with tau.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging and insightful feedback. Please find the answers to your specific comments. Also please refer to our overall rebuttal response to all reviewers and the corresponding pdf containing additional figures and tables to support our claims.\n\n**Method factors:** The reviewer correctly points out that all the components are not equally critical for OPEL’s performance. We show the contributions of each factor individually to analyze their effect on the overall result in Table 5. Comparing row 3 with row 9, we observe, the priors jointly play a critical role; as without them (row 3), the F1 and IoU scores drop by *\\~5* points. Specifically, the optimality prior has a significant impact (*\\~2* point), while the temporal prior affects the score by *\\~1* point. Similar to the combined priors, the intra and inter-video contrastive losses together (see row 6 vs row 9) have a significant effect (*\\~3.5* points) on the overall performance. The individual effect of virtual frame is negligible as it only plays a role in case of excessive background frames - a scenario that is not prevalent in most datasets. Furthermore, due to the IDM structure of $M\\left(\\hat{T}\\right)$, the $\\hat{T}$ and $\\hat{Q}$ are similar by formulation. This results in $KL\\left(\\hat{T}\\ \\parallel\\hat{Q}\\right)$ to be already small. As a consequence, adding the KL divergence as a standalone loss component in the proposed pipeline has a minimal impact. Overall, while some loss components may have a smaller individual impact, they do contribute to performance improvements, even if incrementally. Therefore, our proposed approach incorporates all of them to achieve the best possible results. We are encouraged with the reviewer's note - the provided ablation study improves the clarity of what is important in this method.\n\n**Writing:** The equations are an integral part in explaining the proposed approach and the different loss terms. But we agree that the page 6 is too dense, and the formulations can be simplified to improve readability. We will update the notations and improve the overall writing as per the reviewer’s suggestion during revision. \n\n**Effect of k:** Note, we obtain best results with *k*=7 (Table 7), and performance drops sharply as *k* goes from 7 to 10. This observation is consistent with all the other SOTA methods on the same datasets [a, b, c]. We hypothesize that *k*=7 works best as it is the optimal number of clusters considering the average number of distinct key-steps (subtasks) of the datasets. For example, for PC Disassembly, although the ground-truth (GT) number of steps is 9, 3 steps are quite similar (remove hard disk, remove motherboard, remove RAM), effectively making them quite close in the feature space. This results in *k*=7 being a better estimation of the cluster number with distinct steps. Note, this demarcation of subtasks (hence, number of clusters) is subjective and varies from dataset to dataset as well as from task to task;  as some may consider semantically similar tasks (e.g. pouring oil vs water) to be one subtask, while others may consider it different. As *k* becomes larger than the actual distinctive number of clusters, each subtask gets split into multiple clusters with very similar embeddings, which upon comparison with GT leads to inferior results. This phenomenon is illustrated in **Fig. R2** (see rebuttal pdf), where a larger *k* might result in several erroneous clusters with very small windows (blue and red boxes in **Fig. R2**). This leads to large fluctuations (jittery predictions) within a single GT phase, thus deteriorating the overall performance.\n\nSecondly, the anomalous trend (*k* = 12 being slightly worse than 10 and 15) might be a dataset (PC Assembly) specific issue and not unique to our approach, as a similar trend for this dataset has been reported in [a, b]. However, for PC Disassembly and other datasets (reported as additional results in the rebuttal pdf, **Table R1**), we consistently find that the performance degrades as *k* is increased from 10 $\\to$ 12 $\\to$ 15. Note, that our contribution is on the optimal transport (OT)-based representation learning, the clustering is only used as an inference post-processing step. Overall, we achieve similar trends with respect to the SOTA works but consistently outperform them across all *k* and datasets.\n\n**Refs.:**\n\n[a] Bansal *et al.* My view is the best view: Procedure learning from egocentric videos. ECCV 2022.\n\n[b] Bansal *et al.* United we stand, divided we fall: Unitygraph for unsupervised procedure learning from video. WACV 2024.\n\n[c] Shah *et al.* Steps: Self-supervised key step extraction and localization from unlabeled procedural videos. ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8i3DGGDeJl", "reviewer": "Reviewer_pvnX", "summary": "The paper proposes an unsupervised method for procedure learning that identifies the key steps and their orders in several videos of the same task.\nThe paper formulates the distribution of video frames as an optimal transport (OT) problem to compute the distances between the key steps. \nTo handle the variations of the videos, the paper introduces a regularization with prior distribution.\nThe proposed method achieves SOTA on third-person and first-person video benchmarks.", "questions": "The authors should give a clear explanation of the regularization usage as mentioned in the weakness.", "limitations": "The authors have mentioned the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Different from prior methods having ordering constraints assumption, the paper relaxes this by formulating OT.\n\n- The paper addresses the variation between videos and then introduces a regularization with priors to enhance the OT.\n\n- The paper shows SOTA results in both first-person and third-person evaluations.", "weaknesses": "- The explanation of using priors to mitigate the variation of the videos, i.e., action speeds, non-monotonic sequences, or starting of actions, is not clear.\n- Besides the ablation studies of different prior distributions, the paper should explain why Laplacian outperforms other distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback.  Please find the answers to your specific comments:\n\n**Explanation of priors:** The concept of the **Optimality Prior** is crucial when dealing with video alignment, especially in challenging scenarios. When two videos are perfectly aligned (case 1 of Fig. 2B), the $\\hat{T}$ becomes strictly diagonal. However, the real value of the optimality prior emerges in more complex situations, such as speed variation, non-monotonicity, etc.\n\n**Speed variation:** Consider the scenario in Fig. 2B, case 3: video 1, shown at the top, takes two frames to complete task A, while video 2, displayed on the left, completes task A in one frame. Here, both frames 1 and 2 from video 1 should be aligned with frame 1 of video 2. The optimality prior enables this alignment, as depicted by the blue curve in Fig. 2C and explained by Eqn. 2. The procedure involves finding, for each frame $i$ from video 2, the closest matching frame from video 1 based on feature similarity, even if the corresponding frame $j$ from video 1 has a different index. In Fig. 2C, for example, frame $j$ from video 1 has the highest likelihood of aligning with frame $i$ from video 2, with the assignment likelihood decaying exponentially for frames further away from $j$.\n\nThe optimality prior also addresses **non-monotonic** cases (Fig. 2B, case 4). Here, though the first two frames from video 1 align with those in video 2, the third frame of video 1 reverts to an earlier task A. To correctly align this frame, it must be matched with its closest counterpart in video 2, which is frame 1 in this case. The optimality prior facilitates this by aligning non-monotonic frames that share a higher feature-wise match (similar to Fig. 2C).\n\nHowever, relying solely on the optimality prior is insufficient because it overlooks the temporal ordering inherent in videos, which is crucial for maintaining temporal coherence. As discussed in the regularization section on page 4 of our paper and illustrated in **Fig. R4** (rebuttal pdf), optimizing only for optimality can result in temporal incoherence. Proper alignment should account for the temporal relationship between frames, ensuring that corresponding frames in one sequence align closely with adjacent frames in the other. \n\nThis necessity leads to a second critical factor: the **Temporal Prior**. Unlike the optimality prior, which seeks the best feature match regardless of temporal distance, the temporal prior promotes alignment between frames that are temporally adjacent, thereby preserving the overall temporal coherence. Similar coherence based concepts have been utilized in other temporal alignment works, such as TCC [a], and contrastive regularization [b]. This temporal prior encourages the alignment matrix to exhibit peak values along the diagonal, with diminishing values away from the diagonal; we model this with a Laplace distribution (Eqn. 3 and the red curve in Fig. 2C).\n\nEssentially, there are 2 factors in play- (i) *optimality* which tries to find the best match between frames irrespective of their temporal distance (which may result in temporally incoherent alignment), (ii) the *temporal* factor which promotes transport between nearby frames without considering their feature matching. We hypothesize (and later validate with results) that the optimal solution requires a balance between both, and therefore propose to merge these two priors addressing the above factors, as expressed in Eqn. 4. **This combined prior** ensures accurate alignment between videos considering all the factors, as further illustrated in Fig. 2D and supported by the results in Table 5 of the main paper.\n\nWhen the **starting of actions** is different (case 2, Fig. 2B), video 2 has a background frame to start when video 1 starts doing task A. As mentioned in page 5 of our paper, we introduce a \\'virtual frame\\' to handle such cases. Even in this case, the combined prior comes into play as we assign any frame to the virtual frame if the likelihood of that frame aligning with any other task-related frames from the other video falls below a predefined threshold. This phenomenon is also illustrated in Fig. 4(B) of the main paper.\n\n**Laplace distribution:** As shown in appendix Table A6 (main paper), Laplace as a prior outperforms other distributions. To analyze this, we plot the distributions in **Fig. R3** (rebuttal pdf). Note, we use the same distribution for optimality as well as temporal priors. \n\nFor the *optimality prior*, the x-axis is the difference of frames in the feature space (1-d representation for illustration purposes), and the y-axis denotes the corresponding probability of alignment. We want the point representing the most likely alignment (as per $\\hat{T}$) to have the highest likelihood, while the assignment probability should exponentially decay further away. The graph clearly shows that the Laplace distribution captures this behavior suitably compared to Uniform and Gaussian. Similarly, for the *temporal prior*, the x-axis denotes the temporal distance between the frames, and the y-axis denotes the corresponding probability of alignment. The graph shows that Laplace distribution facilitates alignment of the frames when they are temporally aligned (close to center), and its **long tail distribution** enables better correlation of non-monotonic frames compared to Gaussian or Uniform. As a result, as shown in **Fig. R3**, even at far away locations from the center (temporally distant frames), alignment is possible if indeed these frames have a high match feature-wise. In this case, the Laplace temporal prior provides non-zero probability to that far away frame (due to its long tail) unlike other distributions and the optimality prior gives a large score (due to feature match), resulting in improved handling of non-monotonicity. \n\n**Refs:**\n\n[a] Dwibedi *et al.* Temporal cycle-consistency learning. CVPR 2019.\n\n[b] Haresh *et al.* Learning by aligning videos in time. CVPR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KA1Qs11lxX", "reviewer": "Reviewer_Yy5N", "summary": "The authors propose a novel approach for procedure learning leveraging optimal transport, OPEL. OPEL integrates optimality and temporal priors, and incorporates a novel inter-video contrastive loss. OPEL achieves significant improvements on egocentric and third-person benchmarks.", "questions": "1. I still have some doubts regarding the difference between procedure learning and action segmentation. I believe their final output forms are same. From related work, I believe that action segmentation involves classifying each frame, while procedure learning only needs to identify key steps. Is this correct? Could you please explain in detail?\n2.I also feel that the paper doesn't clearly present the problem formulation, including the input and output, which makes it somewhat confusing to understand. Does the output video contain a fixed number of K key steps? According to the paper, setting K to 7 is optimal. However, based on the dataset, the average number of steps for each task may be greater than or less than K. When it exceeds K, are some steps merged? And what happens when it's less than K? Is it reasonable to pre-define K?\n3. Could you provide more results of ablation training on K for additional tasks?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. An interesting and relevant topic regarding the treatment of procedure learning as an optimal transport problem.\n2. The authors thoroughly consider deviations in real-world sequences, including background frames, redundant frames, and non-monotonic frames.\n3. OPEL achieves excellent results across several benchmarks.", "weaknesses": "1. The presentations of the paper needs improvement, especially in Section 3, where an excessive number of formulas obstruct readability.\n2. Some figures are unclear, such as Figure 1. Are V1, V2, V3, V4 from four different videos, or do they appear to be from the same video? Additionally, what distinguishes V3 from V4, where one represents background frames and the other represents redundant frames?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback.  Please find answers to your specific comments:\n\n**Presentation:** We agree that the equations are too dense in Section 3, and the formulations can be simplified. We will update accordingly as per the reviewer’s suggestion during revision.\n\nThe V1 to V4 in Fig. 1 are from four different videos, while the frames from the same video are shown in x-axis. We apologize for the ambiguity of background frames vs redundant frames; the Fig.1 has been updated as **Fig. R1** (rebuttal pdf). Both refer to frames not directly related to any key-step (hence, do not belong to any cluster) and thus are clustered together as a background cluster (**Fig. R1**, right). So, we remove the term redundant frames for clarity.\n\n**PL vs AS:** Procedure learning (PL) and action segmentation (AS) are related but not the same. PL, when applied to a set of videos of the same task, involves 2 primary steps: (a) assigning each frame to 1 of the *k* key-steps (including background elements), (b) determining the logical sequence of the key-steps to complete the task. As illustrated in Fig. 1, PL addresses multiple videos of a given task, enabling the identification of repetitive key-steps across these videos [a, b]. In contrast, AS [c] focuses on a single video, thereby lacking the ability to discern repetitive key-steps across videos. Additionally, AS does not consider the sequence of individual events, which is often crucial for accurately identifying procedures. For instance, AS fails to capture the variations in the order of key-steps, such as those observed between V1 and V2 in Fig. 1. \n \nIn our work, the **OPEL** loss is used to train a representation learning algorithm to obtain an embedding space where similar frames are located close by. Then, as described in the ‘Clustering and Key-step Ordering’ part of page 6 of the main paper, these key-steps are clustered, and their sequential order is determined. A PyTorch function (with a toy example) is provided in **codeblock R1** of the rebuttal pdf to illustrate this process of finding the sequential order (part b from the definition of PL given above) of the frame-wise key-step predictions (output from part a from the above PL definition). \n\n**Problem formulation:** Given a set of videos of the same task (e.g. making brownies), PL aims to find the constituent key-steps (e.g. break egg, mix contents, add oil, mix egg) and their sequential order (break egg --> mix egg --> add water --> add oil --> mix contents).\n\nDuring training: Inputs are untrimmed and unlabeled videos. We train with pairs of videos at a time (formulation detailed in Section 3 of the main paper). Output is the frames clustered into different key steps and their sequential order. For evaluation metrics, we use framewise cluster predictions, following other SOTA works [a,b,d].\n\nDuring inference: \nInput: a single video, $P = \\left[p_1,\\ p_2,\\ .\\ .\\ .\\ ,\\ p_N\\right]$\nOutput: \nAssign each frame to a phase using trained model $f_{\\theta}∶ f_{\\theta}(p_i)= l_i$, where $l_i ∈ {c}^k_{c=1}$, represents the key-step corresponding to the phase of each frame $p_i$ in the video P.\n\nFrom all $l_i$, we determine the sequential order of the sub-tasks (eg: 0,3,2,4,1...). Note, the cluster number is arbitrary, each of them denoting a key-step. We provide a PyTorch code to illustrate this process of finding the sequential order in **codeblock R1** of the rebuttal pdf.\n\n**Choice of *k*:** In unsupervised PL, *k* is not set during training (learning of the embeddings), rather *k* is used during inference only to fix the no. of output clusters, and each frame of the video being inferred is mapped to one of those *k* clusters. Now, an output video may not contain that exact *k* number of key-steps (e.g. in making a sandwich, steps such as adding jelly, butter etc. may be present in some videos, while absent in others).   \n\nWe obtain best results with *k=7* (Table 7), consistent with other SOTA methods on the same datasets [a,b,d]. We agree that based on the dataset, the average number of steps for each task may be > or < *k*. If it exceeds *k*, some steps get merged, e.g., for PC Disassembly, though the ground-truth number of steps is 9, 3 are quite similar (remove hard disk, remove motherboard, remove RAM); effectively making them quite close in the feature space. As a result, choosing *k=7* results in these 3 key-steps getting merged. When the average number of key-steps becomes < *k*, each subtask gets split into multiple smaller clusters with very similar embeddings. This phenomenon is illustrated in **Fig. R2** (rebuttal pdf), where a larger k might result in several split clusters of small windows (blue and red boxes in **Fig. R2**). \n\nNote, this demarcation of subtasks (no. of clusters) varies from task to task, dataset to dataset and is subjective as some may consider semantically similar tasks (e.g. pouring oil vs water) to be same subtask, while others may consider it different. So, for best performance, *k* might be adjusted task-wise (using methods such as the elbow method, AIC criterion). However, optimizing for task-wise *k* is not the goal of this work; our contribution is on the OT-based learning and the clustering is only used as an inference post-processing step. So for fair comparison, we experiment with some reasonable values of *k* (following prior art [a,b,d]) and consistently outperform them across all *k* and all datasets with similar trends.\n\nWe report **additional ablation** on varying *k* in the rebuttal pdf, **Table R1**. We find that the performance degrades as *k* is increased from $7\\to10\\to12\\to15$, thus overall *k=7* provides the best performance. Note, our contribution is on the OT-based learning, the clustering is only used as an inference post-processing step. Overall, we achieve similar trends for varying *k* w.r.t. SOTA works [a,b,d].\n\n[a] Bansal et al. ECCV 2022\n\n[b] Bansal et al. WACV 2024\n\n[c] Kumar et al. CVPR, 2022\n\n[d] Shah et al. ICCV, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UJLS1h4bJd", "reviewer": "Reviewer_3Xtn", "summary": "The paper presents OPEL, a novel framework for procedure learning from videos that leverages optimal transport (OT) to align key steps across different video instances. OPEL treats video frames as samples from an unknown distribution and formulates the distance calculation between them as an optimal transport problem, allowing for more flexible alignment compared to frame-to-frame mappings. The authors introduce two regularization terms to improve the OT formulation and experiments significantly outperforms state-of-the-art methods on benchmark datasets.", "questions": "I do not have specific questions on the model design. I only have some experiment suggestions which are the same as I listed in \"weaknesses\".", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel optimal transport-based procedure learning framework that aligns frames with similar semantics together in an embedding space.\n2. The paper enhances the OT formulation with two regularization terms that address temporal and semantic relationships, contributing to better alignment and learning.\n3. OPEL demonstrates effective performance improvements over previous state-of-the-art methods on benchmark datasets.", "weaknesses": "1. I believe the authors can compare with recent action segmentation methods to further strengthen the experiments.\n2. I believe the authors can also compare with methods that use temporal alignment methods like dynamic time warping to strengthen their contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging and insightful feedback. Please find the answers to your specific comments:\n\n**Comparison with AS methods:** Procedure learning (PL) and action segmentation (AS) are related but not the same. PL, when applied to a set of instructional videos depicting the same task, involves two primary steps: (i) assigning each video frame to one of  the *k* key-steps (including background elements), and (ii) determining the logical sequence of these key-steps necessary to complete the task. As illustrated in Fig. 1 of the main paper, PL addresses multiple videos of a given task, enabling the identification of repetitive key-steps across these videos [i, j]. In contrast, AS [k] focuses on a single video, thereby lacking the ability to discern repetitive key-steps across different videos. \n\nDespite the differences between PL and AS, as per the reviewer's suggestion, we compare our approach against existing SOTA unsupervised AS models [a-f] and present the results in **Table X** below. Our model demonstrates a significant performance improvement compared to these works. In [b, f], authors report a high recall score for CrossTask as it assigns majority of the frames to a single key-step - a phenomenon also reported by [j]. While achieving *high recall* is important for ensuring that most positive instances are correctly identified, it can result in a greater number of *false positives*, which in turn lowers precision and leads to undesirable results. Therefore, it is crucial to balance recall with precision to develop an effective model. This balance is reflected in the superior performance of our model, as evidenced by the F1-score results across various benchmarks. Note in the Table X, our approach is compared with SOTA **Unsupervised AS** methods for only third-person datasets, as these do not report any result on egocentric datasets.\n\n**Table X:** Comparison of our approach with SOTA **Unsupervised Action Segmentation** methods. Note '-' denotes authors have not provided any data on those metrics. \n|Action Segmentation Papers| |ProceL | | | |CrossTask| |\n|:----|:---:|:---:|:---:|:-:|:---:|:---:|:---:|\n| |P|R|F1||P|R|F1|\n|JointSeqFL (2019) [a]|-|-|29.8||-|-|-|\n|Elhamifar et al. (2020) [b]|9.5|26.7|14.0||10.1|41.6|16.3|\n|Fried et al. (2020) [c]|-|-|-||-|28.8|-|\n|Shen et al. (2021) [d]|16.5|31.8|21.1||15.2|35.5|21.0|\n|Dvornik et al. (2022) [e]|-|-|-||-|-|25.3|\n|StepFormer (2023) [f]|18.3|28.1|21.9||22.1|**42**|28.3|\n|OPEL|**33.6**|**36.3**|**34.9**||**35.6**|34.8|**35.1**|\n\n\n**Temporal Alignment methods:** According to the reviewer's suggestion, we have included additional comparisons with temporal alignment techniques in **Table Y**. Specifically, we have compared our model with methods like **TCC** [g] and **LAV** [h], which incorporate temporal cycle consistency (TCC) and dynamic time warping (DTW), respectively. Other methods like **CnC** [i] uses TC3I (TCC + contrastive inverse difference moment, C-IDM) as the loss function while **GPL** [j] uses graph-based representation for temporal alignment. Our results clearly demonstrate the efficacy of our approach compared to these established methods, further validating the effectiveness of our model in maintaining temporal alignment while delivering superior PL performance. Note, in Table Y, our approach is compared with SOTA **Unsupervised Temporal alignment** methods for only egocentric (first-person) datasets, as these do not report any result on third-person datasets.\n\n**Table Y:** Comparison of our approach with existing **Unsupervised Temporal Alignment** methods.\n|Temporal Alignment Methods | CMU-MMAC| | | | MECCANO| | | |EPIC-Tent| | | |EGTEA-GAZE+| | | |PC Assembly| | | |PC Disassembly| | |\n|:----|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| |Precision|F1|IoU| |Precision|F1|IoU| |Precision|F1|IoU| |Precision|F1|IoU| |Precision|F1|IoU| |Precision|F1|IoU|\n|TCC [g]|18.5|19.7|9.5| |15.1|17.9|8.7| |14.2|14.9|7.8| |17.5|19.7|8.8| |19.9|21.7|11.6| |22.0|23.4|12.2|\n|LAV (DTW) [h]|20.6|21.1|9.4| |14.6|17.4|7.1| |15.2|15.8|8.3| |17.4|19.1|8.0| |21.5|22.7|11.7| |26.4|26.5|12.9|\n|LAV + TCC|18.8|19.7|9.0| |13.4|15.6|7.3| |16.0|16.7|8.5| |16.4|18.6|7.5| |21.6|21.1|10.8| |21.0|24.3|12.3|\n|CnC(TC3I) [i]|21.6|22.7|11.1| |15.5|18.1|7.8| |17.1|17.2|8.3| |19.6|21.7|9.5| |25.0|25.1|12.8| |28.4|27.0|14.8|\n|GPL [j]|30.3|31.7|17.9| |18.8|20.7|10.0| |17.9|19.8|9.1| |23.8|27.1|16.0| |27.1|27.5|15.2| |28.1|26.7|15.2|\n|OPEL|**32.8**|**36.5**|**18.8**| |**28.9**|**39.2**|**20.2**| |**18.8**|**20.7**|**10.6**| |**24.3**|**29.5**|**13.2**| |**32.5**|**33.7**|**17.9**| |**29.6**|**32.2**|**16.7**|\n\nThanks again for the suggestions to strengthen our paper, looking forward to your kind consideration.\n\n**Refs.:**\n\n[a] Elhamifar *et al.* Unsupervised Procedure Learning via Joint Dynamic Summarization, ICCV 2019.\n\n[b] Elhamifar *et al.* Self-supervised multi-task procedure learning from instructional videos, ECCV 2020.\n\n[c] Fried *et al.* Learning to segment actions from observation and narration, ACL 2020.\n\n[d] Shen *et al.* Learning to segment actions from visual and language instructions via differentiable weak sequence alignment, CVPR 2021.\n\n[e] Dvornik *et al.* Flow Graph to Video Grounding for Weakly-Supervised Multi-step Localization, ECCV 2022.\n\n[f] Dvornik *et al.* StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos, CVPR 2023.\n\n[g] Dwibedi *et al.* Temporal cycle-consistency learning, CVPR 2019.\n\n[h] Haresh *et al.* Learning by aligning videos in time, CVPR 2021.\n\n[i] Bansal *et al.* My view is the best view: Procedure learning from egocentric videos. ECCV 2022.\n\n[j] Bansal *et al.* United we stand, divided we fall: Unitygraph for unsupervised procedure learning from video. WACV 2024.\n\n[k] Kumar *et al.* Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering. CVPR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and feedback. We are encouraged that the reviewers like the soundness and novelty of our approach for procedure learning along with comprehensive evaluation (Reviewer XDhK), and enhancement of results over current state-of-the-art (SOTA) works (Reviewer pvnX). Additionally, Reviewer Yy5N finds our work interesting and relevant as we thoroughly consider deviations in real-world sequences, including background and non-monotonic frames. Lastly, Reviewer 3Xtn appreciates the novelty of our work and its SOTA performance. 

The reviewers have also raised some concerns that we have addressed in their individual rebuttal responses. Furthermore, as per the reviewers' suggestions, we have added additional relevant figures and results to support our claims in the attached 1 page pdf and referred to them in our responses. Specifically,

1. Figure R1 is a modified version of Figure 1 of the main paper.
2. Figure R2 illustrates the qualitative effect of varying *k* for clustering the key-steps.
3. Figure R3 shows the reasoning for the choice of Laplace distribution as a prior over other distributions.
4. Figure R4 emphasizes of the importance of both the optimality and temporal priors to combat real-world non-idealities like non-monotonic frames, speed variation, etc., while maintaining temporal coherence.
5. Table R1 shows additional ablation study on the effect of *k* on MECCANO and EPIC-Tents datasets.
6. Codeblock R1 depicts a Pytorch function to determine the sequential ordering of tasks from frame-wise key-step predictions.

In general, we agree that the formulations in Section 3 can be simplified to improve readability. We will update accordingly as per the reviewers' suggestions during the camera-ready version, if accepted. Please note, in our rebuttal responses, Fig./Table X (e.g. Fig. 2/Table 5) refers to the main submitted manuscript, while **Fig./Table RX** (e.g. Fig. R2/Table R1) refers to the **rebuttal pdf** attached herewith.

Once again, we sincerely appreciate your time and consideration. Please let us know if you have any further queries. We look forward to your responses.

Meta-review: This paper received active discussions between authors and reviewers, with the outcome that the reviewer reached an accept consensus (5, 5, 6, 7). The reviewers appreciate the novelty of the method, the new insights that come with bringing optimal transport in the video recognition domain, and the strong empirical performance. The reviewers have pointed out limitations when it comes to writing/explanations as well as the overlap with action segmentation. In the rebuttals, the authors have performed additional experiments with comparisons on action segmentation, which further strengthens the paper. The AC finds that the strengths of the paper outweigh the limitations given by the reviewers, especially after the rebuttal stage. Hence the recommendation to accept.
Decision: Accept (poster)