SlowGuess/ACL_2020_OCR huggingface dataset:
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaou Tang, and Dahua Lin. 2017. Temporal action detection with structured segment networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Luowei Zhou, Xu Chenliang, and Jason J. Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings of the Conference on Artificial Intelligence (AAAI).
Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
A Optimization
For both training conditions for our semi-Markov models that require gradient descent (generative unsupervised and discriminative supervised), we initialize parameters randomly and use Adam (Kingma and Ba, 2015) with an initial learning rate of 5e-3, a batch size of 5 videos, and decay the learning rate when training log likelihood does not decrease for more than one epoch.
B Features
For our features  $x_{1:T}$ , we use the same base features as Zhukov et al. (2019). There are three feature types: activity recognition features, produced by an I3D model (Carreira and Zisserman, 2017) trained on the Kinetics-400 dataset (Kay et al., 2017); object classification features, from a ResNet-152 (He et al., 2016) trained on ImageNet (Russakovsky et al., 2015), and audio classification features $^{10}$  from the VGG model (Simonyan and Zisserman, 2015) trained by Hershey et al. (2017) on a preliminary version of the YouTube-8M dataset (Abu-El-Haija et al., 2016). $^{11}$
For the generative mdoels which use Gaussian emission distributions, we apply PCA to the base features above to reduce the feature dimensionality and decorrelate dimensions. We perform PCA separately for features within task and within each feature group (I3D, ResNet, and audio features), but on features from all videos within that task. We use 100 components for each feature group, which explained roughly  $70 - 100\%$  of the variance in the features, depending on the task and feature group. The 100-dimensional PCA representations for the I3D, ResNet, and audio features for each frame, at timestep  $t$ , are then concatenated to give a 300-dimensional vector for the frame,  $x_{t}$ .
C Unsupervised Evaluation
The HSMM model, when trained in a fully unsupervised setting, induces class labels for regions in the video; however while these class labels are distinct, they do not correspond a priori to any of the actual region labels (which can be step types, or background) for our task. Just as with other unsupervised tasks and models (e.g., part-of-speech induction), we need a mapping from these classes
to step types (and background) in order to evaluate the model's predictions. We follow the evaluation procedure of past work (Sener and Yao, 2018; Sener et al., 2015) by finding the mapping from model states to region labels that maximizes label accuracy, averaged across all videos in the task, using the Hungarian method (Kuhn, 1955). This evaluation condition is only used in the "Unsupervised" section of Table 2 (in the rows marked with optimal accuracy assignment).
Label accuracy The standard metric for action segmentation (Sener and Yao, 2018; Richard et al., 2018) is timestep label accuracy, in datasets with a large amount of background, label accuracy on non-background timesteps. The CrossTask dataset has multiple reference step labels in the groundtruth for around  $1\%$  of timesteps, due to noisy region annotations that overlap slightly. We obtain a single reference label for these timesteps by taking the step that appears first in the canonical step ordering for the task. We then compute accuracy of the model predictions against these reference labels across all timesteps and all videos for a task (in the all label accuracy condition), or by filtering to those timesteps which have a step label (non-background) in the reference (to focus on the model's ability to accurately predict step labels), in the step label accuracy condition.
Step recall This metric (Zhukov et al., 2019) measures a model's ability to pick out instants for each of the possible step types for a task, if they occur in a video. The model of Zhukov et al. (2019) predicted a single frame for each step type; while our extension of their model, ORDEREDDISCRIM, and our HSMM model can predict multiple, when computing this metric we obtain a single frame for each step type to make the numbers comparable to theirs. When a model predicts multiple frames per step type, we obtain a single one by taking the one closest to the middle of the temporal extent of the predicted frames for that step type. We then apply their recall metric: First, count the number of recovered steps, step types from the true labels for the video that were identified by one of the predicted labels (have a predicted label of the same type at one of the true label's frames). These recovered step counts are summed across videos in the evaluation set for a given task, and normalized by the maximum number of possible recovered steps
$^{10}$ https://github.com/tensorflow/models/tree/master/research/audioset/vggish  
 $^{11}$ We also experiment with using features derived from transcribed narration in Appendix G.
(the number of step types in each video, summed across videos) to produce a step recall fraction for the task.
Sequence similarity This measures the similarity of the predicted sequence of regions in a video against the true sequence of regions. As in speech recognition, we are interested in the high-level sequence of steps recognized in a video (and wish to abstract away from noise in the boundaries of the annotated regions). We first compute the negated Levenshtein distance between the true sequence of steps and background  $r_1, \ldots, r_K$  for a video and the and predicted sequence  $\hat{r}_1, \ldots, \hat{r}_K'$ . The negated distance for the sequence pairs for a given video are scaled to be between 0 and 100, where 0 indicates the Levenshtein distance is the maximum possible between two sequences of their respective lengths, and 100 corresponds to the sequences being identical. These similarities are then averaged across all videos in a task.
E Comparing Generative and Discriminative Models
We observe that the generative models tend to obtain higher performance on the action segmentation task, as measured by step label accuracy, than discriminative models of the same or similar class. We attribute this finding to two factors: first, the generative models explicitly parameterize probabilities for the steps, allowing better modeling of the full distribution of step labels. Second, the discriminative models are trained to optimize  $p(l_{t} \mid x_{t})$  for all timesteps  $t$ . We would expect that this would produce better accuracies on metrics aligned with this objective (Klein and Manning, 2002) – and indeed the all timestep accuracy is higher for the discriminative models. However, the discriminative models' high accuracy often comes at the expense of predicting background more frequently, leading to lower performance on step label accuracy.
F Duration Model Ablation
We examine the effect of the (hidden) semi-Markov model's Poisson duration model by comparing to a (hidden) Markov model (HMM in the unsupervised/weakly-supervised settings, or MM in the supervised setting). We use the model as described in Sec. 4 except for fixing all durations to be a single timestep. We then train as described in Sec. 4.1. While this does away with explicit modeling of duration, the transition distribution still
Model	All Label Acc.	Step Label Acc.	Step Recall	Seq. Sim.
Supervised
SMM, gen.	60.5	49.4	28.7	46.6
MM, gen.	60.1	48.6	28.2	46.8
SMM, disc.	66.0	37.3	24.1	50.5
MM, disc.	62.8	32.2	20.1	41.8
Weakly-Supervised
HSMM	31.8	28.8	10.6	31.0
HMM	28.8	30.8	10.3	29.9
HSMM+Ord+Narr	61.2	15.9	17.2	55.0
HMM+Ord+Narr	60.6	17.0	20.0	55.0
Table 4: Comparison between the semi-Markov and hidden semi-Markov models (SMM and HSMM) with the Markov and hidden Markov (MM and HMM) models, which ablate the semi-Markov's duration model.
allows the model to learn expected durations for each region type by implicitly parameterizing a geometric distribution over region length. Results are shown in 4. We observe that results are overall very similar, with the exceptions that removing the duration model decreases performance substantially on all metrics in the discriminative supervised setting, and increases performance on step label accuracy and step recall in the constrained unsupervised setting (HSMM+Ord+Narr and HMM+Ord+Narr). This suggests that the HMM transition distribution is able to model region duration as well as the HSMM's explicit duration model, or that duration overall plays a small role in modeling in most settings relative to the importance of the features.
G Narration Features
The benefit of narration-derived hard constraints on labels (following past work by Zhukov et al. 2019) raises the question of how much narration would help when used to provide features for the models. We obtain narration features for each video using FastText word embeddings (Mikolov et al., 2018) for the video's time-aligned transcribed narration (see Zhukov et al. 2019 for details on this transcription), pooled within a sliding window to allow for imperfect alignment between activities mentioned in the narration and their occurrence in the video. The features for a given timestep  $t$  are produced by a weighted sum of embeddings for all the words in the transcribed narration within a 5-second window of  $t$  (i.e. from  $t - 2$  to  $t + 2$ ), weighted using a Hanning window $^{12}$  (so that words in the center of each window are most heavily weighted for that
12https://docs.scipy.org/doc/numpy/ reference/generated/numpy.hanning.html
(a) Feature vectors colored by their step label in the reference annotations.
All Label Acc.	Step Label Acc.	Step Recall
Supervised
Gaussian mixture	70.4 (+1.0)	43.7 (+3.1)	34.9 (+3.4)
SMM, generative	63.3 (+2.8)	53.2 (+3.8)	32.1 (+3.4)
Weakly-Supervised
HSMM+Ord	53.6 (-1.9)	9.5 (+1.2)	8.5 (+1.2)
HSMM+Narr	68.9 (+3.2)	8.0 (-1.6)	12.6 (+4.1)
HSMM+Narr+Ord	64.3 (+3.1)	17.9 (+2.0)	21.9 (+4.7)
Table 5: Performance of key supervised and weakly-supervised models on the validation data when adding narration vectors as features. Numbers in parentheses give the change from adding narration vectors to the systems from Table 2.
window). We did not tune the window size, or experiment with other weighting functions. The word embeddings are pretrained on Common Crawl, and are not fine-tuned with the rest of the model parameters.
Once these narration features are produced, as above, we treat them in the same way as the other feature types (activity recognition, object classification, and audio) described in Appendix B: reducing their dimensionality with PCA, and concatenating them with the other feature groups to produce the features  $x_{t}$ .
In Table 5, we show performance of key supervised and weakly-supervised models on the validation set, when using these narration features in addition to activity recognition, object detection, and audio features. Narration features improve performance over the corresponding systems from Table 2 (differences are shown in parentheses) in 13 out of 15 cases, typically by  $1 - 4\%$ .
(b) Feature vectors colored by the id of the video they occur in.
H Feature Visualizations
To give a sense for feature similarities both within step types and within a video, we visualize feature vectors for 20 videos randomly chosen from the change a tire task, dimensionality-reduced using t-SNE (Maaten and Hinton, 2008) so that similar feature vectors are close in the visualization.
Figure 4a shows feature vectors colored by step type: we see little consistent clustering of feature vectors by step. On the other hand, we observe a great deal of similarity across step types within a video (see Figure 4b); when we color feature vectors by video, different steps from the same video are close to each other in space. These together suggest that better featurization of videos can improve action segmentation.
I Segmentation Visualizations
In the following pages, we show example segmentations from the various systems. Figure 5 and 6 visualize predicted model segmentations for the unstructured Gaussian mixture and structured semi-Markov model in the supervised setting, in comparison to the ground-truth and the ordered uniform baseline. We see that while both models typically make similar predictions in the same temporal regions of the video, the structured model produces steps that are much less fragmented.
Figure 7 and 8 visualize segmentations in the unsupervised and weakly-supervised settings for the HSMM model and ORDEREDDISCRIM of Zhukov et al. (2019). The unsupervised HSMM has difficulty distinguishing steps from background (see Appendix H), while the model trained with
weak supervision from ordering and narration (HSMM+Ord+Narr) is better able to induce meaningful steps. The ORDEREDDISCRIM model, although it has been modified to allow predicting multiple timesteps per step, collapses to predicting a single label, background, nearly everywhere, which we conjecture is because the model is discriminatively trained: jointly inferring labels that are easy to predict, and the model parameters to predict them.
Figure 5: Supervised segmentations We visualize segmentations from the validation set for a video from the task make kimchi fried rice. We show the ground truth (GT), ordered uniform baseline (Uniform), and predictions from the unstructured Gaussian mixture model (GMM), and structured semi-Markov model (SMM) trained in the supervised setting. Predictions from the unstructured model are more fragmented than predictions from the SMM. The x-axis gives the timestep in the video.
Figure 6: Supervised segmentations We visualize segmentations from the validation set for a video from the task build simple floating shelves. We show the ground truth (GT), ordered uniform baseline (Uniform), and predictions from the unstructured Gaussian mixture model (GMM), and structured semi-Markov model (SMM) trained in the supervised setting. Predictions from the unstructured model are more fragmented than predictions from the SMM. The x-axis gives the timestep in the video.
Figure 7: Unsupervised and weakly-supervised segmentations We visualize segmentations from the validation set for a video from the task make pancakes. We show the ground truth (GT), ordered uniform baseline (Uniform), and predictions from the hidden semi-markov trained without constraints (HSMM) and with constraints from narration and ordering (HSMM+Narr+Ord), and the system of Zhukov et al. The x-axis gives the timestep in the video.
Figure 8: Unsupervised and weakly-supervised segmentations We visualize segmentations from the validation set for a video from the task grill steak. We show the ground truth (GT), ordered uniform baseline (Uniform), and predictions from the hidden semi-markov trained without constraints (HSMM) and with constraints from narration and ordering (HSMM+Narr+Ord), and the system of Zhukov et al. The x-axis gives the timestep in the video.