SlowGuess/ACL_2020_OCR huggingface dataset:
We first define several baselines based on dataset statistics (Sec. 7.1), which we will find to be strong in comparison to past work. We then analyze each
Figure 2: Baseline and model performance on two key metrics: step label accuracy and step recall. Points are colored according to their supervision type, and labeled with their row number from Table 2. We also label particular important models.
aspect of our proposed model on the dataset in a supervised training setting (Sec. 7.2), removing some error sources of unsupervised learning and evaluating whether a given model fits the dataset (Liang and Klein, 2008). Finally, we move to our main setting, the weakly-supervised setting of past work, incrementally adding step ordering and narration constraints (see Sec. 6.1) to evaluate the degree to which each helps (Sec. 7.3).
Results are given in Table 2 for models trained on the CrossTask training set of primary tasks, and evaluated on the held-out validation set. We will describe and analyze each set of results in turn. See Figure 2 for a plot of models' performance on two key metrics, and Appendix I for example predictions.
7.1 Dataset Statistic Baselines
Table 2 (top block) shows baselines that do not use video (or narration) features, but predict steps according to overall statistics of the training data. These demonstrate characteristics of the data, and the importance of using multiple metrics.
Predict background (B1) Since most timesteps are background, a model that predicts background everywhere can obtain high overall label accuracy, showing the importance of also using step label accuracy as a metric for action segmentation.
Sample from the training distribution (B2)
For each timestep in each video, we sample a label from the empirical distribution of step and background label frequencies for the video's task in the training data.
This reduces the number of parameters that need to be learned in the emission distributions, both by reducing the dimensionality and allowing a diagonal covariance matrix. In early experiments we found PCA improved performance.
#	Model	All Label Accuracy	Step Label Accuracy	Step Recall	Sequence Similarity	Predicted Bkg. %	Num. Segments.
Dataset Statistic Baselines (Sec. 7.1)
GT	Ground truth	100.0	100.0	100.0	100.0	71.9	7.7
B1	Predict background	71.9	0.0	0.0	9.0	100.0	0.0
B2	Sample from train distribution	54.6	7.2	8.3	12.8	72.4	69.5
B3	Ordered uniform	55.6	8.1	12.2	55.0	73.0	7.4
Supervised (Sec. 7.2)
Unstructured
S1	Discriminative linear	71.0	36.0	31.6	30.7	73.3	27.1
S2	Discriminative MLP	75.9	30.4	27.7	41.1	82.8	13.0
S3	Gaussian mixture	69.4	40.6	31.5	33.3	68.9	23.9
Structured
S4	ORDEREDDISCRIM	75.2	18.1	45.4	54.4	90.7	7.4
S5	SMM, discriminative	66.0	37.3	24.1	50.5	65.9	8.5
S6	SMM, generative	60.5	49.4	28.7	46.6	52.4	10.6
Un- and Weakly-Supervised (Sec. 7.3)
Fully Unsupervised
U1	HSMM (with opt. acc. assignment)	31.8	28.8	10.6	31.0	31.1	15.4
Ordering Supervision
U2	ACTIONSETS	40.8	14.0	12.1	55.0	49.8	7.4
U3	ORDEREDDISCRIM (without Narr.)	69.5	0.2	2.8	55.0	97.2	7.4
U4	HSMM + Ord	55.5	8.3	7.3	55.0	70.6	7.4
Narration Supervision
U5	HSMM + Narr	65.7	9.6	8.5	35.1	84.6	4.5
Ordering + Narration Supervision
U6	ORDEREDDISCRIM	71.0	1.8	24.5	55.0	97.2	7.4
U7	HSMM + Narr + Ord	61.2	15.9	17.2	55.0	73.7	7.4
Table 2: Model comparison on the CrossTask validation data. We evaluate primarily using all label accuracy and step label accuracy to evaluate action segmentation, and step recall to evaluate step recognition.
Ordered uniform (B3) For each video, we predict step regions in the canonical step order, separated by background regions. The length of each region is set so that all step regions in a video have equal duration, and the percentage of background timesteps is equal to the corpus average. See Uniform in Figure 3a for sample predictions.
Sampling each timestep label independently from the task distribution (row B2), and using a uniform step assignment in the task's canonical ordering with background (B3) both obtain similar step label accuracy, but the ordered uniform baseline improves substantially on the step recall metric, indicating that step ordering is a useful inductive bias for step recognition.
7.2 Full Supervision
Models in the unstructured block of Table 2 are classification models applied independently to all timesteps, allowing us to compare the performance of the feature models used as components in our structured models. We find that a Gaussian mixture model (row S3), which is used as the feature model in the HSMM, obtains comparable step recall and substantially higher step label accuracy
than a discriminative linear classifier (row S1) similar to the one used in Zhukov et al. (2019), which is partially explained by the discriminative classifier overpredicting the background class (comparing Predicted Background % for those two rows). Using a higher capacity discriminative classifier, a neural net with a single hidden layer (MLP), improves performance over the linear model on several metrics (row S2); however, the MLP still overpredicts background, substantially underperforming the Gaussian mixture on the step label accuracy metric.
In the structured block of Table 2, we compare the full models which use step constraints (Zhukov et al., 2019) or learned transition distributions (the SMM) to model task structure. The structured models learn (or in the case of Zhukov et al., enforce) orderings over the steps, which greatly improve their sequence similarity scores when compared to the unstructured models, and decrease step fragmentation (as measured by num. segments). Figure 3a shows predictions for a typical video, demonstrating this decreased fragmentation.[9]
<sup>9</sup>We also perform an ablation study to understand the effect of the duration model. See Appendix F for details.
(a) Step segmentations in the full supervision condition for a video from the make kimchi fried rice task, comparing the ground truth (GT), ordered uniform baseline (Uniform), and predictions from the Gaussian mixture (GMM) and semi-Markov (SMM) models.
(b) Step segmentations in the no- or weak-supervision conditions for a video from the make pancakes task, comparing the ground truth (GT) to predictions from our model without (HSMM) and with constraint supervision (HSMM+Narr+Ord) and from Zhukov et al. (2019) (ORDEREDDISCRIM).
Figure 3: Step segmentation visualizations for two sample videos in supervised (left) and unsupervised (right) conditions. The x-axes show timesteps, in seconds. See Appendix I for more visualizations.
We see two trends in the supervised results:
(1) Generative models obtain substantially higher step label accuracy than discriminative models of the same or similar class. This is likely due to the fact that the generative models directly parameterize the step distribution. (See Appendix E.)
(2) Structured sequence modeling naturally improves performance on sequence-level metrics (sequence similarity and number of segments predicted) over the unstructured models. However, none of the learned structured models improve on the strong ordered uniform baseline (B3) which just predicts the canonical ordering of a task's steps (interspersed with background regions). This will motivate using this canonical ordering as a constraint in unsupervised learning.
Overall, the SMM models obtain strong action segmentation performance (high step label accuracy without fragmenting segments or overpredicting background).
7.3 No or Weak Supervision
Here models are trained without supervision for the labels  $l_{1:T}$ . We compare models trained without any constraints, to those that use constraints from step ordering and narration, in the Un- and Weakly Supervised block of Table 2. Example outputs are shown in Appendix I.
Our generative HSMM model affords training without any constraints (row U1). This model has high step label accuracy (compared to the other unsupervised models) but low all label accuracy, and similar scores for both metrics. This hints, and other metrics confirm, that the model is not adequately distinguishing steps from background: the percentage of predicted background is very low  $(31\%)$  compared to the ground truth  $(72\%,$  row GT).
See HSMM in Figure 3b for predictions for a typical video. These results are attributable to features within a given video (even across step types) being more similar than features of the same step type in different videos (see Appendix H for feature visualizations). The induced latent model states typically capture this inter-video diversity, rather than distinguishing steps across tasks.
We next add in constraints from the canonical step ordering, which our supervised results showed to be a strong inductive bias. Unlike in the fully unsupervised setting, the HSMM model with ordering (HSMM+Ord, row U4) learns to distinguish steps from background when constrained to predict each step region once in a video, with predicted background timesteps  $(70.6\%)$  close to the ground-truth  $(72\%)$ . However, performance of this model is still very low on the task metrics – comparable to or underperforming the ordered uniform baseline with background (row B3) on all metrics.
This constrained step ordering setting also allows us to apply ACTIONSETS (Richard et al., 2018) and ORDEREDDISCRIM (Zhukov et al., 2019). ACTIONSETS obtains high step label accuracy, but substantially underpredicts background, as evidenced by both the all label accuracy and the low predicted background percentage. The tendency of ORDEREDDISCRIM to overpredict background which we saw in the supervised setting (row S4) is even more pronounced in this weakly-supervised setting (row U3), resulting in scores very close to the predict background baseline (B1).
Next, we use narration constraints (U5), which are enforced only during training time, following Zhukov et al. (2019). Narration constraints substantially improve all label accuracy (comparing U1 and U5). However, the model overpredicts
All Label Acc.	Step Label Acc.	Step Recall
ORDEREDDISCRIM	71.3	1.2	17.9
HSMM+Narr+Or	66.0	5.6	14.2
Table 3: Unsupervised and weakly supervised results in the cross-validation setting.
background, likely because it doesn't enforce each step type to occur in a given video. Overpredicting background causes step label accuracy and step recall to decrease.
Finally, we compare the HSMM and ORDERED-DISCRIM models when using both narration constraints (in training) and ordering constraints (in training and testing) in the ordering + narration block. Both models benefit substantially from narration on all metrics when compared to using only ordering supervision, more than doubling their performance on step label accuracy and step recall (comparing U6 and U7 to U3 and U4).
Our weakly-supervised results show that:
(1) Both action segmentation metrics – all label accuracy and step label accuracy – are important to evaluate whether models adequately distinguish meaningful actions from background.
(2) Step constraints derived from the canonical step ordering provide a strong inductive bias for unsupervised step induction. Past work requires these constraints and the HSMM, when trained without them, does poorly, learning to capture diversity across videos rather than to identify steps.
(3) However, ordering supervision alone is not sufficient to allow these models to learn better segmentations than a simple baseline that just uses the ordering to assign labels (ordered uniform); narration is also required.
7.4 Comparison to Past Work
Finally, we compare our full model to the ORDEREDDISCRIM model of Zhukov et al. (2019) in the primary data evaluation setup from that work: averaging results over 20 random splits of the primary data (Table 3). This is a low data setting which uses only 30 videos per task as training data in each split.
Accordingly, both models have lower performance, although the relative ordering is the same: higher step label accuracy for the HSMM, and higher all label accuracy and step recall for ORDEREDDISCRIM. Although in this low-data setting, models overpredict background even more,
this problem is less pronounced for the HSMM:  $97.4\%$  of timesteps for ORDEREDDISCRIM are predicted background (explaining its high all label accuracy), and  $87.1\%$  for HSMM.
We find that unsupervised action segmentation in naturalistic instructional videos is greatly aided by the inductive bias given by typical step orderings within a task, and narrative language describing the actions being done. While some results are more mixed (with the same supervision, different models are better on different metrics), we do observe that across settings and metrics, step ordering and narration increase performance. Our results also illustrate the importance of strong baselines: without weak supervision from step orderings and narrative language, even state-of-the-art unsupervised action segmentation models operating on rich video features underperform feature-agnostic baselines. We hope that future work will continue to evaluate broadly.
While action segmentation in videos from diverse domains remains challenging – videos contain both a large variety of types of depicted actions, and high visual variety in how the actions are portrayed – we find that structured generative models provide a strong benchmark for the task due to their abilities to capture the full diversity of action types (by directly modeling distributions over action occurrences), and to benefit from weak supervision. Future work might explore methods for incorporating richer learned representations both of the diverse visual observations in videos, and the narration that describes them, into such models.
Thanks to Dan Klein, Andrew Zisserman, Lisa Anne Hendricks, Aishwarya Agrawal, Gábor Melis, Angeliki Lazaridou, Anna Rohrbach, Justin Chiu, Susie Young, the DeepMind language team, and the anonymous reviewers for helpful feedback on this work. DF is supported by a Google PhD Fellowship.
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016. YouTube-8M: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. 2016. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Dare A Baldwin, Jodie A Baird, Megan M Saylor, and M Angela Clark. 2001. Infants parse dynamic action. Child Development, 72(3):708-717.
Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. 2014. Weakly supervised action labeling in videos under ordering constraints. In Proceedings of the European Conference on Computer Vision (ECCV).
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the Kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. 2019. D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Li Ding and Chenliang Xu. 2018. Weakly-supervised action segmentation with iterative soft boundary assignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In Proceedings of the Workshop on Structured Prediction for NLP.
Ehsan Elhamifar and Zwe Naing. 2019. Unsupervised procedure learning via joint dynamic summarization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Yazan Abu Farha and Jurgen Gall. 2019. MS-TCN: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for largescale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. 2016. Connectionist temporal modeling for weakly supervised action labeling. In Proceedings of the European Conference on Computer Vision (ECCV).
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The Kinetics human action video dataset. arXiv preprint arXiv:1705.06950.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Dan Klein and Christopher D. Manning. 2002. Conditional structure versus conditional estimation in NLP models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Hilde Kuehne, Ali Arslan, and Thomas Serre. 2014. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hilde Kuehne, Alexander Richard, and Juergen Gall. 2017. Weakly supervised learning of actions from transcripts. In CVU.
Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97.
Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall. 2019. Unsupervised learning of action classes with continuous temporal embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. 2017. Temporal convolutional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Dani Levine, Daphna Buchsbaum, Kathy Hirsh-Pasek, and Roberta M Golinkoff. 2019. Finding events in a continuous world: A developmental account. Developmental Psychobiology, 61(3):376-389.
Percy Liang and Dan Klein. 2008. Analyzing the errors of unsupervised learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research (JMLR), 9(Nov):2579-2605.
Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nicholas Johnston, Andrew Rabinovich, and Kevin Murphy. 2015. What's cookin'? interpreting cooking videos using text, speech and vision. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Bridgette A Martin and Barbara Tversky. 2003. Segmenting ambiguous events. In Proceedings of the Annual Meeting of the Cognitive Science Society.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC).
Kevin Murphy. 2002. Hidden semi-markov models. Unpublished tutorial.
Janne Pylkkonen and Mikko Kurimo. 2004. Duration modeling techniques for continuous speech recognition. In *Eighth International Conference on Spoken Language Processing*.
Alexander Richard, Hilde Kuehne, and Juergen Gall. 2017. Weakly supervised action learning with RNN based fine-to-coarse modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Alexander Richard, Hilde Kuehne, and Juergen Gall. 2018. Action sets: Weakly supervised action segmentation without ordering constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. 2012. A database for fine grained activity detection of cooking activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211-252.
Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.
Fadime Sener and Angela Yao. 2018. Unsupervised learning and segmentation of complex activities from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
O. Sener, A. Zamir, S. Savarese, and A. Saxena. 2015. Unsupervised semantic parsing of video collections. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Tanya Sharon and Karen Wynn. 1998. Individuation of actions from continuous motion. Psychological Science, 9(5):357-362.
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR).
Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. 2016. A multi-stream bidirectional recurrent neural network for fine-grained action detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. 2019. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ercenur Ünal, Yue Ji, and Anna Papafragou. 2019. From event representation to linguistic meaning. Topics in Cognitive Science.
Huijuan Xu, Abir Das, and Kate Saenko. 2017. R-C3D: Region convolutional 3d network for temporal activity detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. 2018. Every moment counts: Dense detailed labeling of actions in complex videos. International Journal of Computer Vision (IJCV), 126(2-4):375-389.
Shoou-I Yu, Lu Jiang, and Alexander Hauptmann. 2014. Instructional videos for unsupervised harvesting and learning of action examples. In Proceedings of the ACM International Conference on Multimedia (MM).
Shun-Zheng Yu. 2010. Hidden semi-markov models. Artificial intelligence, 174(2):215-243.
Jeffrey M Zacks and Khena M Swallow. 2007. Event segmentation. Current Directions in Psychological Science, 16(2):80-84.