## 3 The Proposed Encoder-Decoder Network
### Spatial Encoder
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Trained with** & **MPII** & **MERL** & **50 Salads [18]** & **GETA [19]** & **JIGSAWS [20; 21]** \\
**Frame-Wise** & **Cooking2 [17]** & **Shopping [11]** & **50 Salads [18]** & **GETA [19]** & **JIGSAWS [20; 21]** \\ \hline MSB-RNN [11] (2016) & 41.22\% (\(mAP\)) & 80.31\% (\(mAP\)) & & & & \\ ED-TCN [12] (2017) & & 74.1\% (\(F.Acc\)) & 64.7\% (\(F.Acc\)) \({}^{*}\) & & & 80.8\% (\(F.Acc\)) \\  & & 64.2\% (\(mAP\)) & 59.8\% (\(Edit\)) \({}^{*}\) & & & 84.7\% (\(Edit\)) \\  & & & 66.3\% (\(F.Acc\)) \({}^{*}\) & & & \\ TCED [14] (2019) & & & 62.5\% (\(Edit\)) \({}^{*}\) & 63.4\% (\(F.Acc\)) & 82.2\% (\(F.Acc\)) \\  & & & 75.9\% (\(F.Acc\)) \({}^{*}\) & 70.9\% (\(Edit\)) & 87.7\% (\(Edit\)) \\  & & & 71.3\% (\(Edit\)) \({}^{**}\) & & & \\  & & & 84.5\% (\(F.Acc\)) \({}^{*}\) & 77.3\% (\(F.Acc\)) & & \\  & & & 79.3\% (\(Edit\)) \({}^{*}\) & 83.7\% (\(Edit\)) & & \\ ASRF [15] (2021) & & & 84.9\% (\(\text{F100.10}\)) \({}^{*}\) & 89.4\% (\(\text{F100.10}\)) & & \\  & & & 83.5\% (\(F\)[EMAIL_ADDRESS]) \({}^{*}\) & 87.8\% (\(F\)[EMAIL_ADDRESS]) & & \\  & & & 77.3\% (F\)[EMAIL_ADDRESS]) \({}^{*}\) & 79.8\% (\(F\)[EMAIL_ADDRESS]) & & \\ \hline
**Trained with Only** & **Breakfast [22]** & **Hollywood** & **50 Salads** & **YouTube** & **YouCook2 [25]** \\
**Order of Actions:** & & **Ex. [23]** & **50 Salads** & **Instructions [24]** & **YouCook2 [25]** \\ \hline GRU + HMM [1] & 33.3\% (\(MoF\)) & 11.9\% (\(IoU\)) \({}^{*}\) & & & & \\
**(2017)** & 47.3\% (\(IoD\)) \({}^{\circ}\) & 51.1\% (\(IoD\)) \({}^{\circ}\) & & & & 50.6\% (\(Jac.\)) \({}^{\circ\circ}\) \\  & & & & & & 37.0\% (\(IoU\)) \({}^{\circ}\) \\  & & & & & & 37.1\% (\(Rec.\)) \({}^{\circ\circ}\) \\  & & & & & & 30.4\% (\(Prec.)) \({}^{\circ\circ}\) \\  & & & & & & 33.4\% (\(F1\)) \({}^{\circ}\) \\ Unsupervised [26] & & & & & & \\
**(2019)** & 41.8\% (\(MoF\)\({}^{\circ\circ}\)) & & & & & \\  & & & & & 35.5\% (\(MoF\)\({}^{\circ\circ}\)) & & \\  & & & & & & \\ Duration [16] (2021) & 55.7\% (\(F.Acc\)) & 50.1\% (\(F.Acc\)) & & & & \\  & & 36.3\% (\(IoU\)) & 31.4\% (\(IoU\)) & & & \\ \hline \hline \end{tabular} All results are reported without any form of supervision on test data, with only the temporal ordering of actions available for training, unless marked with \({}^{\circ}\). \({}^{\circ}\) denotes that additional information on the temporal ordering of actions is also provided in the test video. \({}^{\circ\circ}\) denotes even weaker supervision, where even the order of actions is not available, only an-ordered sets of the actions contained are available. All results are performed with low-level granularity with 48 sub-action classes, where each mid-level granularity is further divided into pre-, core-, and post-phases, unless marked with \({}^{\circ}\). \({}^{*}\) denotes that evaluations were performed at mid-level action granularity with 18 sub-action classes. The mid-level labels differentiate actions like “cut tomato” from “cut cucumber”, whereas the higher-level labels combine these into a single class, “cut. \({}^{*}\) denotes that evaluations were performed at eval-level action granularity with 9 sub-action classes such as “cut.”, “peel,” and “add dressing”. Metrics definitions: (1) _MoF_ (mean-over-frames) is equivalent to \(F.Acc\), and refers to the average percentage of correctly labeled frames. (2) _IoU_, _IoD_, and _Jac._ (intersection over union/detection/prediction) refer to \(\frac{IoU}{IoU}\), \(\frac{IoU}{IoU}\), and \(\frac{IoU}{IoU}\), respectively, where \(l\) and \(l\) are ground-truth and prediction intervals, respectively. _IoU_ penalizes all the misalignment of proposal while _Jaccard_ only penalizes the partition of segments beyond the ground truth. (3) _Edit_, _F1_, _Prec._, and _Rec._; see Equations (10)-(12) in Section 4.3 for definitions.  
\end{table} 
Table 1: State-of-the-art performance under different training conditions.