SlowGuess/ACL_2020_OCR huggingface dataset:
Daniel Fried‡ Jean-Baptiste Alayrac† Phil Blunsom† Chris Dyer† Stephen Clark† Aida Nematzadeh†
$\dagger$  DeepMind, London, UK
$^{\ddagger}$ Computer Science Division, UC Berkeley
dfried@berkeley.edu
{jalayrac,pblunsom,cdyer,clarkstephen,nematzadeh}@google.com
We apply a generative segmental model of task structure, guided by narration, to action segmentation in video. We focus on unsupervised and weakly-supervised settings where no action labels are known during training. Despite its simplicity, our model performs competitively with previous work on a dataset of naturalistic instructional videos. Our model allows us to vary the sources of supervision used in training, and we find that both task structure and narrative language provide large benefits in segmentation quality.
1 Learning to Segment Actions
Finding boundaries in a continuous stream is a crucial process for human cognition (Martin and Tversky, 2003; Zacks and Swallow, 2007; Levine et al., 2019; Ünal et al., 2019). To understand and remember what happens in the world around us, we need to recognize the action boundaries as they unfold and also distinguish the important actions from the insignificant ones. This process, referred to as temporal action segmentation, is also an important first step in systems that ground natural language in videos (Hendricks et al., 2017). These systems must identify which frames in a video depict actions – which amounts to distinguishing these frames from background ones – and identify which actions (e.g., boiling potatoes) each frame depicts. Despite recent advances (Miech et al., 2019; Sun et al., 2019), unsupervised action segmentation in videos remains a challenge.
The recent availability of large datasets of naturalistic instructional videos provides an opportunity for modeling of action segmentation in a rich task context (Yu et al., 2014; Zhou et al., 2018; Zhukov et al., 2019; Miech et al., 2019; Tang et al., 2019);
in these videos, a person teaches a specific high-level task (e.g., making croquettes) while describing the lower-level steps involved in that task (e.g., boiling potatoes). However, the real-world nature of these datasets introduces many challenges. For example, more than  $70\%$  of the frames in one of the YouTube instructional video datasets, CrossTask (Zhukov et al., 2019), consist of background regions (e.g., the video presenter is thanking their viewers), which do not correspond to any of the steps for the video's task.
These datasets are interesting because they provide (1) narrative language that roughly corresponds to the activities demonstrated in the videos and (2) structured task scripts that define a strong signal of the order in which steps in a task are typically performed. As a result, these datasets provide an opportunity to study the extent to which task structure and language can guide action segmentation. Interestingly, young children can segment actions without any explicit supervision (Baldwin et al., 2001; Sharon and Wynn, 1998), by tapping into similar cues – action regularities and language descriptions (Levine et al., 2019).
While previous work mostly focuses on building action segmentation models that perform well on a few metrics (Richard et al., 2018; Zhukov et al., 2019), we aim to provide insight into how various modeling choices impact action segmentation. How much do unsupervised models improve when given implicit supervision from task structure and language, and which types of supervision help most? Are discriminative or generative models better suited for the task? Does explicit structure modeling improve the quality of segmentation? To answer these questions, we compare two existing models with a generative hidden semi-Markov model, varying the degree of supervision.
On a challenging and naturalistic dataset of instructional videos (Zhukov et al., 2019), we find
Work begun while DF was interning at DeepMind. Code is available at https://github.com/dpfried/action-segmentation.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2569-2588 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
that our model and models from past work both benefit substantially from the weak supervision provided by task structure and narrative language, even on top of rich features from state-of-the-art pretrained action and object classifiers. Our analysis also shows that: (1) Generative models tend to do better than discriminative models of the same or similar model class at learning the full range of step types, which benefits action segmentation; (2) Task structure affords strong, feature-agnostic baselines that are difficult for existing systems to surpass; (3) Reporting multiple metrics is necessary to understand each model's effectiveness for action segmentation; we can devise feature-agnostic baselines that perform well on single metrics despite producing low-quality action segments.
Typical methods (Rohrbach et al., 2012; Singh et al., 2016; Xu et al., 2017; Zhao et al., 2017; Lea et al., 2017; Yeung et al., 2018; Farha and Gall, 2019) for temporal action segmentation consist of assigning action classes to intervals of videos and rely on manually-annotated supervision. Such annotation is difficult to obtain at scale. As a result, recent work has focused on training such models with less supervision: one line of work assumes that only the order of actions happening in the video is given and use this weak supervision to perform action segmentation (Bojanowski et al., 2014; Huang et al., 2016; Kuehne et al., 2017; Richard et al., 2017; Ding and Xu, 2018; Chang et al., 2019). Other approaches weaken this supervision and use only the set of actions that occur in each video (Richard et al., 2018), or are fully unsupervised (Sener and Yao, 2018; Kukleva et al., 2019).
Instructional videos have gained interest over the past few years (Yu et al., 2014; Sener et al., 2015; Malmaud et al., 2015; Alayrac et al., 2016; Zhukov et al., 2019) since they enable weakly-supervised modeling: previous work most similar to ours consists of models that localize actions in narrated videos with minimal supervision (Alayrac et al., 2016; Sener et al., 2015; Elhamifar and Naing, 2019; Zhukov et al., 2019).
We present a generative model of action segmentation that incorporates duration modeling, narration and ordering constraints, and can be trained in all of the above supervision conditions by maximizing the likelihood of the data; while these past works have had these individual components, they
have not yet all been combined.
3 The CrossTask Dataset
We use the recent CrossTask dataset (Zhukov et al., 2019) of instructional videos. To our knowledge, CrossTask is the only available dataset that has tasks from more than one domain, includes background regions, provides step annotations and naturalistic language. Other datasets lack one of these; e.g. they focus on one domain (Kuehne et al., 2014) or do not have natural language (Tang et al., 2019) or step annotations (Miech et al., 2019). An example instance from the dataset is shown in Figure 1, and we describe each aspect below.
Tasks Each video comes from a task, e.g. make a latte, with tasks taken from the titles of selected WikiHow articles, and videos curated from YouTube search results for the task name. We focus on the primary section of the dataset, containing 2,700 videos from 18 different tasks.
Steps and canonical order Each task has a set of steps: lower-level action step types, e.g., steam milk and pour milk, which are typically completed when performing the task. Step names consist of a few words, typically naming an action and an object it is applied to. The dataset also provides a canonical step order for each task: an ordering, like a script (Schank and Abelson, 1977; Chambers and Jurafsky, 2008), in which a task's steps are typically performed. For each task, the set of step types and their canonical order were hand-constructed by the dataset creators based on section headers in the task's WikiHow article.
Annotations Each video in the primary section of the dataset is annotated with labeled temporal segments identifying where steps occur. (In the weak supervision setting, these step segment labels are used only in evaluation, and never in training.) A given step for a task can occur multiple times, or not at all, in any of the task's videos. Steps in a video also need not occur in the task's canonical ordering (although in practice our results show that this ordering is a helpful inductive bias for learning). Most of the frames in videos (72% over the entire corpus) are background – not contained in any step segment.
Narration Videos also have narration text (transcribed by YouTube's automatic speech recognition system) which typically consists of a mix of the
Figure 1: An example video instance from the CrossTask dataset (Sec. 3). The video depicts a task, make pancakes, and is annotated with region segments, which can be either action steps (e.g., pour mixture into pan) or background regions. Videos also are temporally-aligned with transcribed narration. We learn to segment the video into these regions and label them with the action steps (or background), without access to region annotations during training.
task demonstrator describing their actions and talking about unrelated topics. Although narration is temporally aligned with the video, and steps (e.g., pour milk) are sometimes mentioned, these mentions often do not occur at the same time as the step they describe (e.g., "let the milk cool before pouring it"). Zhukov et al. (2019) guide weakly-supervised training using the narration by defining a set of narration constraints for each video, which identify where in the video steps are likely to occur, using similarity between the step names and temporally-aligned narration (see Sec. 6.1).
Our generative model of the video features and labeled task segments is a first-order semi-Markov model. We use a semi-Markov model for the action segmentation task because it explicitly models temporal regions of the video, their duration, their probable ordering, and their features. It can be trained in an unsupervised way, without labeled regions, to maximize the likelihood of the features.
Timesteps Our atomic unit is a one-second region of the video, which we refer to as a timestep. A video with  $T$  timesteps has feature vectors  $x_{1:T}$ . The features  $x_{t}$  at timestep  $t$  are derived from the video, its narration, or both, and in our work (and past work on the dataset) are produced by pre-trained neural models which summarize some non-local information in the region containing each timestep, which we describe in Sec. 6.3.
Regions Our model segments a video with  $T$  timesteps into a sequence of regions, each of which consists of a consecutive number of timesteps (the region's duration). The number of regions  $K$  in a
video and the duration  $d_{k}$  of each region can vary; the only constraint is that the sum of the durations equals the video length:  $\sum_{k=1}^{K} d_{k} = T$ . Each region has a label  $r_{k}$ , which is either one of the task's step labels (e.g., pour milk) or a special label BKG indicating the region is background. In our most general, unconstrained model, a given task step can occur multiple times (or not at all) as a region label in any video for the task, allowing step repetitions, dropping, and reordering.
Structure We define a first-order Markov (bigram) model over these region labels:
$$
P \left(r _ {1: K}\right) = P \left(r _ {1}\right) \prod_ {k = 2} ^ {K} P \left(r _ {k} \mid r _ {k - 1}\right) \tag {1}
$$
with tabular conditional probabilities. While region labels are part of the dataset, they are primarily used for evaluation: we seek models that can be trained in the unsupervised and weakly-supervised conditions where labels are unavailable. This model structure, while simple, affords a dynamic program allowing efficient enumeration over both all possible segmentations of the video into regions and assignments of labels to the regions, allowing unsupervised training (Sec. 4.1).
Duration Our model, following past work (Richard et al., 2018), parameterizes region durations using Poisson distributions, where each label type  $r$  has its own mean duration  $\lambda_r$ :  $d_k \sim \mathrm{Poisson}(\lambda_{r_k})$ . These durations are constrained so that they partition the video: e.g., region  $r_2$  begins at timestep  $d_1$  (after region  $r_1$ ), and the final region  $r_K$  ends at the final timestep  $T$ .
Timestep labels The region labels  $r_{1:K}$  (step, or background) and region durations  $d_{1:K}$  together give a sequence of timestep labels  $l_{1:T}$  for all timesteps, where a timestep's label is equal to the label for the region it is contained in.
<sup>1</sup>Semi-Markov models are also shown to be successful in the similar domain of speech recognition (e.g., Pylkkonen and Kurimo, 2004).
Feature distribution Our model's feature distribution  $p(x_{t}|l_{t})$  is a class-conditioned multivariate Gaussian distribution:  $x_{t} \sim \mathrm{Normal}(\mu_{l_{t}},\Sigma)$ , where  $l_{t}$  is the step label at timestep  $t$ . (We note that the assignment of labels to steps is latent and unobserved during unsupervised and weakly-supervised training.) We use a separate learned mean  $\mu_{l}$  for each label type  $l$ , both steps and background. Labels are atomic and task-specific, e.g., the step type pour milk when it occurs in the task make a latte does not share parameters with the step add milk when it occurs in the task make pancakes. We use a diagonal covariance matrix  $\Sigma$  which is fixed to the empirical covariance of each feature dimension.
In the unsupervised setting, labels  $l$  are unavailable at training (used only in evaluation). We describe training in this setting, as well as two supervised training methods which we use to analyze properties of the dataset and compare model classes.
Unsupervised We train the generative model as a hidden semi-Markov model (HSMM). We optimize the model's parameters to maximize the log marginal likelihood of the features for all video instance features  $x^{(i)}$  in the training set:
$$
\mathcal {M L} = \sum_ {i} ^ {N} \log P (x _ {1: T _ {i}} ^ {(i)}) \tag {2}
$$
Applying the semi-Markov forward algorithm (Murphy, 2002; Yu, 2010) allows us to marginalize over all possible sequences of step labels to compute the log marginal likelihood for each video as a function of the model parameters, which we optimize directly using backpropagation and minibatched gradient descent with the Adam (Kingma and Ba, 2015) optimizer. See Appendix A for optimization details.
Generative supervised Here the labels  $l$  are observed; we train the model as a generative semiMarkov model (SMM) to maximize the log joint likelihood:
$$
\mathcal {J L} = \sum_ {i} ^ {N} \log P \left(l _ {1: T _ {i}} ^ {(i)}, x _ {1: T _ {i}} ^ {(i)}\right) \tag {3}
$$
Richard et al. (2018)	Zhukov et al. (2019)	Ours
step reordering	✓	✓
step repetitions	✓	✓
step duration	✓	✓
language	✓	✓
generative model	✓	✓
Table 1: Characteristics of each model we compare.
We maximize this likelihood over the entire training set using the closed form solution given the dataset's sufficient statistics (per-step feature means, average durations, and step transition frequencies).
Discriminative supervised To train the SMM model discriminatively in the supervised setting, we use gradient descent to maximize the log conditional likelihood:
$$
\mathcal {C L} = \sum_ {i} ^ {N} \log P \left(l _ {1: T} ^ {(i)} \mid x _ {1: T} ^ {(i)}\right) \tag {4}
$$
We identify five modeling choices made in recent work: imposing a fixed ordering on steps (not allowing step reordering); allowing for steps to repeat in a video; modeling the duration of steps; using the language (narrations) associated with the video; and using a discriminative/generative model. We picked the recent models of Zhukov et al. (2019) and Richard et al. (2018) since they have non-overlapping strengths (see Table 1).
ORDEREDDISCRIM This work (Zhukov et al., 2019) uses a discriminative classifier which gives a probability distribution over labels at each timestep:  $p(l_{t} \mid x_{t})$ . Inference finds an assignment of steps to timesteps that maximizes  $\sum_{t} \log p(l_{t} \mid x_{t})$  subject to the constraints that: all steps are predicted exactly once; steps occur in the fixed canonical ordering defined for the task; one background region occurs between each step. Unsupervised training of the model alternates between inferring labels using the dynamic program, and updating the classifier to maximize the probability of these inferred labels.[5]
ACTIONSETS This work (Richard et al., 2018) uses a generative model which has structure similar to ours, but uses dataset statistics (e.g., average video length and number of steps) to learn the
5. To allow the model to predict step regions with duration longer than a single timestep, we modify this classifier to also predict a background class, and incorporate the scores of the background class into the dynamic program.
2We experimented with sharing steps, or step components, across tasks in initial experiments, but found that it was helpful to have task-specific structural probabilities.
<sup>3</sup>We found that using a shared diagonal covariance matrix outperformed using full or unshared covariance matrices.
4This is the same as mini-batched Expectation Maximization using gradient descent on the M-objective (Eisner, 2016).
structure distributions, rather than setting parameters to maximize the likelihood of the data. As in our model, region durations are modeled using a class-conditional Poisson distribution. The feature distribution is modeled using Bayesian inversion of a discriminative classifier (a multi-layer perceptron) with an estimated label prior. The structural parameters of the model (durations and class priors) are estimated using the length of each video, and the number of possible step types. As originally presented, this model depends on knowing which steps occur in a video at training time; for fair comparison, we adapt it to the same supervision conditions of Zhukov et al. (2019) by enforcing the canonical step ordering for the task during both training and evaluation.
6 Experimental Setting
We compare models on the CrossTask dataset across supervision conditions. We primarily evaluate the models on action segmentation (Sec. 1). Past work on the dataset (Zhukov et al., 2019) has focused on a step recognition task, where models identify individual timesteps in videos that correspond to possible steps; for comparison, we also report performance for all models on this task.
6.1 Supervision Conditions
In all settings, the task for a given video is known (and hence the possible steps), but the settings vary in the availability of other sources of supervision: step labels for each timestep in a video, and constraints from language and step ordering. Models are trained on a training set and evaluated on a separate held-out testing set, consisting of different videos (from the same tasks).
Supervised Labels for all timesteps  $l_{1:T}$  are provided for all videos in the training set.
Fully unsupervised No labels for timesteps are available during training. The only supervision is the number of possible step types for each task (and, as in all settings, which task each video is from). In evaluation, the task for a given video (and hence the possible steps, but not their ordering) are known. We follow past work in this setting (Sener et al., 2015; Sener and Yao, 2018) by finding a mapping from model states to region labels that maximizes label accuracy, averaged across all videos in the task. See Appendix C for details.
Weakly supervised No labels for timesteps are available, but two supervision types are used in the form of constraints (Zhukov et al., 2019):
(1) Step ordering constraints: Step regions are constrained to occur in the canonical step ordering (see Sec. 3) for the task, but steps may be separated by background. We constrain the structure prior distributions  $p(r_1)$  and transition distributions  $p(r_{k + 1}|r_k)$  of the HSMM to enforce this ordering. For  $p(r_1)$ , we only allow non-zero probability for the background region, BKG, and for the first step in the task's ordering.  $p(r_k|r_{k - 1})$  constrains each step type to only transition to the next step in the constrained ordering, or to BKG. As step ordering constraints change the parameters of the model, when we use them we enforce them during both training and testing. While this obviates most of the learned structure of the HSMM, the duration model (as well as the feature model) is still learned.
(2) Narration constraints: These give regions in the video where each step type is likely to occur. Zhukov et al. (2019) obtained these using similarities between word vectors for the transcribed narration and the words in the step labels, and a dynamic program to produce constraint regions that maximize these similarities, subject to the step ordering matching the canonical task ordering. See Zhukov et al. for details. We enforce these constraints in the HSMM by penalizing the feature distributions to prevent any step labels that occur outside of one of the allowed constraint regions for that step. Following Zhukov et al., we only use these narration constraints during training. $^7$
6.2 Evaluation
We use three metrics from past work, outlined here and described in more detail in Appendix D. To evaluate action segmentation, we use two varieties of the standard label accuracy metric (Sener and Yao, 2018; Richard et al., 2018): all label accuracy, which is computed on all timesteps, including background and non-background, as well as step label accuracy: accuracy only for timesteps that occur in a non-background region (according to the ground-truth annotations). Since these two accuracy metrics are defined on individual frames,
To enforce ordering when steps are separated by BKG, we annotate BKG labels with the preceding step type (but all BKG labels for a task share feature and duration parameters, and are merged for evaluation).
<sup>7</sup>We also experiment with using features derived from transcribed narration in Appendix G.
they penalize models if they don't capture the full temporal extent of actions in their predicted segmentations. Our third metric is step recall, used by past work on the CrossTask dataset (Zhukov et al., 2019) to measure step recognition (defined in Sec. 6). This metric evaluates the fraction of step types which are correctly identified by a model when it is allowed to predict only one frame per step type, per video. A high step recall indicates a model can accurately identify at least one representative frame of each action type in a video.
We also report three other statistics to analyze the predicted segmentations: (1) Sequence similarity: the similarity of the sequence of region labels predicted in the video to the groundtruth, using inverse Levenshtein distance normalized to be between 0 and 100. See Appendix D for more details. (2) Predicted background percentage: the percentage of timesteps for which the model predicts the background label. Models with a higher percentage than the ground truth background percentage  $(72\%)$  are overpredicting background. (3) Number of segments: the number of step segments predicted in a video. Values higher than the ground truth average (7.7) indicate overly-fragmented steps. Sequence similarity and number of segments are particularly relevant for measuring the effects of structure, as they do not factor over individual timesteps (as do the all label and step label accuracies and step recall).
We average values across the 18 tasks in the evaluation set (following Zhukov et al., 2019).
6.3 Features
For our features  $x_{1:T}$ , we use the same base features as Zhukov et al. (2019), which are produced by convolutional networks pre-trained on separate activity, object, and audio classification datasets. See Appendix B for details. In our generative models, we apply PCA (following Kuehne et al., 2014 and Richard et al., 2018) to project features to 300 dimensions and decorrelate dimensions (see Appendix B for details).