Learning Scripts as Hidden Markov Models	Scripts have been proposed to model the stereotypical event sequences found in narratives. They can be applied to make a variety of inferences including filling gaps in the narratives and resolving ambiguous references. This paper proposes the first formal framework for scripts based on Hidden Markov Models (HMMs). Our framework supports robust inference and learning algorithms, which are lacking in previous clustering models. We develop an algorithm for structure and parameter learning based on Expectation Maximization and evaluate it on a number of natural datasets. The results show that our algorithm is superior to several informed baselines for predicting missing events in partial observation sequences.

The initial state INLINEFORM22 and the final state INLINEFORM23 respectively emit the distinguished observation symbols, “ INLINEFORM24 ” and “ INLINEFORM25 ” in INLINEFORM26 , which are emitted by no other state. The concatenation of observations in successive states consitutes a sample of the distribution represented by the HMM. Because the null observations are removed from the generated observations, the length of the output string may be smaller than the number of state transitions. It could also be larger than the number of distinct state transitions, since we allow observations to be generated on the self transitions. Thus spurious and missing observations model insertions and deletions in the outputs of HMMs without introducing special states as in profile HMMs  . In this paper we address the following problem. Given a set of narrative texts, each of which describes a stereotypical event sequence drawn from a fixed but unknown distribution, learn the structure and parameters of a Left-to-Right HMM model that best captures the distribution of the event sequences. We evaluate the algorithm on natural datasets by how well the learned HMM can predict observations removed from the test sequences.

More recently still, graphical models have been proposed for representing script-like knowledge, but these lack the temporal component that is central to this paper and to the early script work. These models instead focus on learning bags of related events  ,  . While the above approches demonstrate the learnability of script-like knowledge, they do not offer a probabilistic framework to reason robustly under uncertainty taking into account the temporal order of events. In this paper we present the first formal representation of scripts as Hidden Markov Models (HMMs), which support robust inference and effective learning algorithms. The states of the HMM correspond to event types in scripts, such as entering a restaurant or opening a door. Observations correspond to natural language sentences that describe the event instances that occur in the story, e.g., “John went to Starbucks. He came back after ten minutes.” The standard inference algorithms, such as the Forward-Backward algorithm, are able to answer questions about the hidden states given the observed sentences, for example, “What did John do in Starbucks?” There are two complications that need to be dealt with to adapt HMMs to model narrative scripts.

In this paper, we restrict our attention to the so-called “Left-to-Right HMMs” which have acyclic graphical structure with possible self-loops, as they support more efficient inference algorithms than general HMMs and suffice to model most of the natural scripts. We consider the problem of learning the structure and parameters of scripts in the form of HMMs from sequences of natural language sentences. Our solution to script learning is a novel bottom-up method for structure learning, called SEM-HMM, which is inspired by Bayesian Model Merging (BMM)  and Structural Expectation Maximization (SEM)  . It starts with a fully enumerated HMM representation of the event sequences and incrementally merges states and deletes edges to improve the posterior probability of the structure and the parameters given the data. We compare our approach to several informed baselines on many natural datasets and show its superior performance. We believe our work represents the first formalization of scripts that supports probabilistic inference, and paves the way for robust understanding of natural language texts.

Acknowledgments	We would like to thank Nate Chambers, Frank Ferraro, and Ben Van Durme for their helpful comments, criticism, and feedback. Also we would like to thank the SCALE 2013 workshop. This work was supported by the DARPA and AFRL under contract No. FA8750-13-2-0033. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA, the AFRL, or the US government.

We define INLINEFORM3 to be the joint probability that the HMM is in state INLINEFORM4 at time INLINEFORM5 and has made the observations INLINEFORM6 . This is computed by the forward pass of the algorithm using the following recursion. Equations EQREF5 and represent the base case of the recursion, while Equation represents the case for null observations. Note that the observation index INLINEFORM7 of the recursive call is not advanced unlike in the second half of Equation where it is advanced for a normal observation. We exploit the fact that the HMM is Left-to-Right and only consider transitions to INLINEFORM8 from states with indices INLINEFORM9 . The time index INLINEFORM10 is incremented starting 0, and the observation index INLINEFORM11 varies from 0 thru INLINEFORM12 .   DISPLAYFORM0    The backward part of the standard Forward-Backward algorithm starts from the last time step INLINEFORM0 and reasons backwards. Unfortunately in our setting, we do not know INLINEFORM1 —the true number of state transitions—as some of the observations are missing.

The INLINEFORM6 parameters assign weights to each component in the prior. The semantic constraints are learned from the event sequences for use in the model prior. The constraints take the simple form “ INLINEFORM0 never follows INLINEFORM1 .” They are learned by generating all possible such rules using pairwise permutations of event types, and evaluating them on the training data. In particular, the number of times each rule is violated is counted and a INLINEFORM2 -test is performed to determine if the violation rate is lower than a predetermined error rate. Those rules that pass the hypothesis test with a threshold of INLINEFORM3 are included. When evaluating a model, these contraints are considered violated if the model could generate a sequence of observations that violates the constraint. Also, in addition to incrementally computing the transition and observation counts, INLINEFORM0 and INLINEFORM1 , the likelihood, INLINEFORM2 can be incrementally updated with structure changes as well. Note that the likelihood can be expressed as INLINEFORM3 when the state transitions are observed. Since the state transitions are not actually observed, we approximate the above expression by replacing the observed counts with expected counts.

Experiments and Results	 We now present our experimental results on SEM-HMM and SEM-HMM-Approx. The evaluation task is to predict missing events from an observed sequence of events. For comparison, four baselines were also evaluated. The “Frequency” baseline predicts the most frequent event in the training set that is not found in the observed test sequence. The “Conditional” baseline predicts the next event based on what most frequently follows the prior event. A third baseline, referred to as “BMM,” is a version of our algorithm that does not use EM for parameter estimation and instead only incrementally updates the parameters starting from the raw document counts. Further, it learns a standard HMM, that is, with no INLINEFORM0 transitions. This is very similar to the Bayesian Model Merging approach for HMMs  . The fourth baseline is the same as above, but uses our EM algorithm for parameter estimation without INLINEFORM1 transitions. It is referred to as “BMM + EM.” The Open Minds Indoor Common Sense (OMICS) corpus was developed by the Honda Research Institute and is based upon the Open Mind Common Sense project  .

It describes 175 common household tasks with each task having 14 to 122 narratives describing, in short sentences, the necessary steps to complete it. Each narrative consists of temporally ordered, simple sentences from a single author that describe a plan to accomplish a task. Examples from the “Answer the Doorbell” task can be found in Table 2. The OMICS corpus has 9044 individual narratives and its short and relatively consistent language lends itself to relatively easy event extraction. The 84 domains with at least 50 narratives and 3 event types were used for evaluation. For each domain, forty percent of the narratives were withheld for testing, each with one randomly-chosen event omitted. The model was evaluated on the proportion of correctly predicted events given the remaining sequence. On average each domain has 21.7 event types with a standard deviation of 4.6. Further, the average narrative length across domains is 3.8 with standard deviation of 1.7. This implies that only a frcation of the event types are present in any given narrative. There is a high degree of omission of events and many different ways of accomplishing each task.

Structure Learning	 We now describe our structure learning algorithm, SEM-HMM. Our algorithm is inspired by Bayesian Model Merging (BMM)  and Structural EM (SEM)  and adapts them to learning HMMs with missing observations. SEM-HMM performs a greedy hill climbing search through the space of acyclic HMM structures. It iteratively proposes changes to the structure either by merging states or by deleting edges. It evaluates each change and makes the one with the best score. An exact implementation of this method is expensive, because, each time a structure change is considered, the MAP parameters of the structure given the data must be re-estimated. One of the key insights of both SEM and BMM is that this expensive re-estimation can be avoided in factored models by incrementally computing the changes to various expected counts using only local information. While this calculation is only approximate, it is highly efficient. During the structure search, the algorithm considers every possible structure change, i.e., merging of pairs of states and deletion of state-transitions, checks that the change does not create cycles, evaluates it according to the scoring function and selects the best scoring structure.

Hence, the prediction task is reasonably difficult, as evidenced by the simple baselines. Neither the frequency of events nor simple temporal structure is enough to accurately fill in the gaps which indicates that most sophisticated modeling such as SEM-HMM is needed. The average accuracy across the 84 domains for each method is found in Table 1. On average our method significantly out-performed all the baselines, with the average improvement in accuracy across OMICS tasks between SEM-HMM and each baseline being statistically significant at a .01 level across all pairs and on sizes of INLINEFORM0 and INLINEFORM1 using one-sided paired t-tests. For INLINEFORM2 improvement was not statistically greater than zero. We see that the results improve with batch size INLINEFORM3 until INLINEFORM4 for SEM-HMM and BMM+EM, but they decrease with batch size for BMM without EM. Both of the methods which use EM depend on statistics to be robust and hence need a larger INLINEFORM5 value to be accurate. However for BMM, a smaller INLINEFORM6 size means it reconciles a couple of documents with the current model in each iteration which ultimately helps guide the structure search.

Equation computes the joint probability of a state-transition associated with a null observation by first multiplying the state transition probability by the null observation probability given the state transition and the appropriate INLINEFORM0 and INLINEFORM1 values. It then marginalizes out the observation index INLINEFORM2 . Again we need to compute a convolution with respect to INLINEFORM3 to take into account the variation over the total number of state transitions. Equation calculates the same probability for a non-null observation INLINEFORM4 . This equation is similar to Equation with two differences. First, we ensure that the observation is consistent with INLINEFORM5 by multiplying the product with the indicator function INLINEFORM6 which is 1 if INLINEFORM7 and 0 otherwise. Second, we advance the observation index INLINEFORM8 in the INLINEFORM9 function. Since the equations above are applied to each individual observation sequence, INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 all have an implicit index INLINEFORM4 which denotes the observation sequence and has been omitted in the above equations. We will make it explicit below and calculate the expected counts of state visits, state transitions, and state transition observation triples.

Introduction	 Scripts were developed as a means of representing stereotypical event sequences and interactions in narratives. The benefits of scripts for encoding common sense knowledge, filling in gaps in a story, resolving ambiguous references, and answering comprehension questions have been amply demonstrated in the early work in natural language understanding  . The earliest attempts to learn scripts were based on explanation-based learning, which can be characterized as example-guided deduction from first principles  ,  . While this approach is successful in generalizing from a small number of examples, it requires a strong domain theory, which limits its applicability. More recently, some new graph-based algorithms for inducing script-like structures from text have emerged. “Narrative Chains” is a narrative model similar to Scripts  . Each Narrative Chain is a directed graph indicating the most frequent temporal relationship between the events in the chain. Narrative Chains are learned by a novel application of pairwise mutual information and temporal relation learning. Another graph learning approach employs Multiple Sequence Alignment in conjunction with a semantic similarity function to cluster sequences of event descriptions into a directed graph  .

Like  , we found that a high weight on the verb similarity was important to finding meaningful clusters of events. The most frequent verb in each cluster is extracted to name the event type that corresponds to that cluster. The initial configuration of the HMM is a Prefix Tree Acceptor, which is constructed by starting with a single event sequence and then adding sequences by branching the tree at the first place the new sequence differs from it  ,  . By repeating this process, an HMM that fully enumerates the data is constructed.

In the first, “forward” pass from INLINEFORM13 to INLINEFORM14 , it calculates the INLINEFORM15 value of each node INLINEFORM16 that represents the probability that a sequence that passes through INLINEFORM17 also passes through INLINEFORM18 while emitting no observation. In the second, “backward” pass, it computes the INLINEFORM19 value of a node INLINEFORM20 that represents the probability that a sequence that passes through INLINEFORM21 emits no observation and later passes through INLINEFORM22 . The product of INLINEFORM23 and INLINEFORM24 gives the probability that INLINEFORM25 is passed through when going from INLINEFORM26 to INLINEFORM27 and emits no observation. Multiplying it by the expected number of transitions INLINEFORM28 gives the expected number of additional counts which are added to INLINEFORM29 to compensate for the deleted transition INLINEFORM30 .

Hence, we define INLINEFORM2 as the conditional probability of observing INLINEFORM3 in the remaining INLINEFORM4 steps given that the current state is INLINEFORM5 . This allows us to increment INLINEFORM6 starting from 0 as recursion proceeds, rather than decrementing it from INLINEFORM7 .   DISPLAYFORM0    Equation EQREF7 calculates the probability of the observation sequence INLINEFORM0 , which is computed by marginalizing INLINEFORM1 over time INLINEFORM2 and state INLINEFORM3 and setting the second index INLINEFORM4 to the length of the observation sequence INLINEFORM5 . The quantity INLINEFORM6 serves as the normalizing factor for the last three equations. DISPLAYFORM0    Equation , the joint distribution of the state and observation index INLINEFORM0 at time INLINEFORM1 is computed by convolution, i.e., multiplying the INLINEFORM2 and INLINEFORM3 that correspond to the same time step and the same state and marginalizing out the length of the state-sequence INLINEFORM4 . Convolution is necessary, as the length of the state-sequence INLINEFORM5 is a random variable equal to the sum of the corresponding time indices of INLINEFORM6 and INLINEFORM7 .

Parameter Estimation with EM	 In this section we describe our parameter estimation methods. While parameter estimation in this kind of HMM was treated earlier in the literature  ,  , we provide a more principled approach to estimate the state-dependent probability of INLINEFORM0 transitions from data without introducing special insert and delete states  . We assume that the structure of the Left-to-Right HMM is fixed based on the preceding structure learning step, which is described in Section SECREF10 . The main difficulty in HMM parameter estimation is that the states of the HMM are not observed. The Expectation-Maximization (EM) procedure (also called the Baum-Welch algorithm in HMMs) alternates between estimating the hidden states in the event sequences by running the Forward-Backward algorithm (the Expectation step) and finding the maximum likelihood estimates (the Maximization step) of the transition and observation parameters of the HMM  . Unfortunately, because of the INLINEFORM0 -transitions the state transitions of our HMM are not necessarily aligned with the observations. Hence we explicitly maintain two indices, the time index INLINEFORM1 and the observation index INLINEFORM2 .

Structure Scoring	 We now describe how we score the structures produced by our algorithm to select the best structure. We employ a Bayesian scoring function, which is the posterior probability of the model given the data, denoted INLINEFORM0 . The score is decomposed via Bayes Rule (i.e., INLINEFORM1 ), and the denominator is omitted since it is invariant with regards to the model. Since each observation sequence is independent of the others, the data likelihood INLINEFORM0 is calculated using the Forward-Backward algorithm and Equation EQREF7 in Section SECREF4 . Because the initial model fully enumerates the data, any merge can only reduce the data likelihood. Hence, the model prior INLINEFORM1 must be designed to encourage generalization via state merges and edge deletions (described in Section SECREF10 ). We employed a prior with three components: the first two components are syntactic and penalize the number of states INLINEFORM2 and the number of non-zero transitions INLINEFORM3 respectively. The third component penalizes the number of frequently-observed semantic constraint violations INLINEFORM4 . In particular, the prior probabilty of the model INLINEFORM5 .

HMM-Script Learning	 At the top level, the algorithm is input a set of documents INLINEFORM0 , where each document is a sequence of natural language sentences that describes the same stereotypical activity. The output of the algorithm is a Left-to-Right HMM that represents that activity. Our approach has four main components, which are described in the next four subsections: Event Extraction, Parameter Estimation, Structure Learning, and Structure Scoring. The event extraction step clusters the input sentences into event types and replaces the sentences with the corresponding cluster labels. After extraction, the event sequences are iteratively merged with the current HMM in batches of size INLINEFORM0 starting with an empty HMM. Structure Learning then merges pairs of states (nodes) and removes state transitions (edges) by greedy hill climbing guided by the improvement in approximate posterior probability of the HMM. Once the hill climbing converges to a local optimum, the maxmimum likelihood HMM parameters are re-estimated using the EM procedure based on all the data seen so far. Then the next batch of INLINEFORM1 sequences are processed. We will now describe these steps in more detail.

This is repeated until the structure can no longer be improved (see Algorithm SECREF10 ). LearnModel INLINEFORM0 , Data INLINEFORM1 , Changes INLINEFORM2   INLINEFORM0 INLINEFORM1 = AcyclicityFilter INLINEFORM2 INLINEFORM3   INLINEFORM0 INLINEFORM1 INLINEFORM2    The Merge States operator creates a new state from the union of a state pair's transition and observation distributions. It must assign transition and observation distributions to the new merged state. To be exact, we need to redo the parameter estimation for the changed structure. To compute the impact of several proposed changes efficiently, we assume that all probabilistic state transitions and trajectories for the observed sequences remain the same as before except in the changed parts of the structure. We call this “locality of change” assumption, which allows us to add the corresponding expected counts from the states being merged as shown below. DISPLAYFORM0     The second kind of structure change we consider is edge deletion and consists of removing a transition between two states and redistributing its evidence along the other paths between the same states. Again, making the locality of change assumption, we only recompute the parameters of the transition and observation distributions that occur in the paths between the two states.

Event Extraction	 Given a set of sequences of sentences, the event extraction algorithm clusters them into events and arranges them into a tree structured HMM. For this step, we assume that each sentence has a simple structure that consists of a single verb and an object. We make the further simplifying assumption that the sequences of sentences in all documents describe the events in temporal order. Although this assumption is often violated in natural documents, we ignore this problem to focus on script learning. There have been some approaches in previous work that specifically address the problem of inferreing temporal order of events from texts, e.g., see  . Given the above assumptions, following  , we apply a simple agglomerative clustering algorithm that uses a semantic similarity function over sentence pairs INLINEFORM0 given by INLINEFORM1 , where INLINEFORM2 is the verb and INLINEFORM3 is the object in the sentence INLINEFORM4 . Here INLINEFORM5 is the path similarity metric from Wordnet  . It is applied to the first verb (preferring verbs that are not stop words) and to the objects from each pair of sentences. The constants INLINEFORM6 and INLINEFORM7 are tuning parameters that adjust the relative importance of each component.

We re-estimate the parameters due to deleting an edge INLINEFORM0 , by effectively redistributing the expected transitions from INLINEFORM1 to INLINEFORM2 , INLINEFORM3 , among other edges between INLINEFORM4 and INLINEFORM5 based on the parameters of the current model. This is done efficiently using a procedure similar to the Forward-Backward algorithm under the null observation sequence. Algorithm SECREF10 takes the current model INLINEFORM0 , an edge ( INLINEFORM1 ), and the expected count of the number of transitions from INLINEFORM2 to INLINEFORM3 , INLINEFORM4 , as inputs. It updates the counts of the other transitions to compensate for removing the edge between INLINEFORM5 and INLINEFORM6 . It initializes the INLINEFORM7 of INLINEFORM8 and the INLINEFORM9 of INLINEFORM10 with 1 and the rest of the INLINEFORM11 s and INLINEFORM12 s to 0. It makes two passes through the HMM, first in the topological order of the nodes in the graph and the second in the reverse topological order.

After the distribution of the evidence, all the transition and observation probabilities are re-estimated for the nodes and edges affected by the edge deletion DeleteEdgeModel INLINEFORM0 , edge INLINEFORM1 , count INLINEFORM2   INLINEFORM0 INLINEFORM1 INLINEFORM2 to INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6   INLINEFORM0 downto INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 = INLINEFORM5 INLINEFORM6 = INLINEFORM7 INLINEFORM8  Forward-Backward algorithm to delete an edge and re-distribute the expected counts.  In principle, one could continue making incremental structural changes and parameter updates and never run EM again. This is exactly what is done in Bayesian Model Merging (BMM)  . However, a series of structural changes followed by approximate incremental parameter updates could lead to bad local optima. Hence, after merging each batch of INLINEFORM0 sequences into the HMM, we re-run EM for parameter estimation on all sequences seen thus far.

Formally, an HMM is a 4-tuple INLINEFORM1 , where INLINEFORM2 is a set of states, INLINEFORM3 is the probability of transition from INLINEFORM4 to INLINEFORM5 , INLINEFORM6 is a set of possible non-null observations, and INLINEFORM7 is the probability of observing INLINEFORM8 when in state INLINEFORM9 , where INLINEFORM11 , and INLINEFORM12 is the terminal state. An HMM is Left-to-Right if the states of the HMM can be ordered from INLINEFORM13 thru INLINEFORM14 such that INLINEFORM15 is non-zero only if INLINEFORM16 . We assume that our target HMM is Left-to-Right. We index its states according to a topological ordering of the transition graph. An HMM is a generative model of a distribution over sequences of observations. For convenience w.l.o.g. we assume that each time it is “run” to generate a sample, the HMM starts in the same initial state INLINEFORM17 , and goes through a sequence of transitions according to INLINEFORM18 until it reaches the same final state INLINEFORM19 , while emitting an observation in INLINEFORM20 in each state according to INLINEFORM21 .

Problem Setup	 Consider an activity such as answering the doorbell. An example HMM representation of this activity is illustrated in Figure FIGREF1 . Each box represents a state, and the text within is a set of possible event descriptions (i.e., observations). Each event description is also marked with its conditional probability. Each edge represents a transition from one state to another and is annotated with its conditional probability. In this paper, we consider a special class of HMMs with the following properties. First, we allow some observations to be missing. This is a natural phenomenon in text, where not all events are mentioned or extracted. We call these null observations and represent them with a special symbol INLINEFORM0 . Second, we assume that the states of the HMM can be ordered such that all transitions take place only in that order. These are called Left-to-Right HMMs in the literature  ,  . Self-transitions of states are permitted and represent “spurious” observations or events with multi-time step durations. While our work can be generalized to arbitrary HMMs, we find that the Left-to-Right HMMs suffice to model scripts in our corpora.

Conclusions	 In this paper, we have given the first formal treatment of scripts as HMMs with missing observations. We adapted the HMM inference and parameter estimation procedures to scripts and developed a new structure learning algorithm, SEM-HMM, based on the EM procedure. It improves upon BMM by allowing for INLINEFORM0 transitions and by incorporating maximum likelihood parameter estimation via EM. We showed that our algorithm is effective in learning scripts from documents and performs better than other baselines on sequence prediction tasks. Thanks to the assumption of missing observations, the graphical structure of the scripts is usually sparse and intuitive. Future work includes learning from more natural text such as newspaper articles, enriching the representations to include objects and relations, and integrating HMM inference into text understanding.

Further, the locality of change assumption allows us to easily calculate the effect of changed expected counts and parameters on the likelihood by dividing it by the old products and multiplying by the new products. We call this version of our algorithm SEM-HMM-Approx.

The accuracy for “SEM-HMM Approx.” is close to the exact version at each batch level, while only taking half the time on average.

DISPLAYFORM0     Equation EQREF8 counts the total expected number of visits of each state in the data. Also, Equation estimates the expected number of transitions between each state pair. Finally, Equation computes the expected number of observations and state-transitions including null transitions. This concludes the E-step of the EM procedure. The M-step of the EM procedure consists of Maximum Aposteriori (MAP) estimation of the transition and observation distributions is done assuming an uninformative Dirichlet prior. This amounts to adding a pseudocount of 1 to each of the next states and observation symbols. The observation distributions for the initial and final states INLINEFORM0 and INLINEFORM1 are fixed to be the Kronecker delta distributions at their true values. DISPLAYFORM0    The E-step and the M-step are repeated until convergence of the parameter estimates.

First, both the set of states, i.e., event types, and the set of observations are not pre-specified but are to be learned from data. We assume that the set of possible observations and the set of event types to be bounded but unknown. We employ the clustering algorithm proposed in  to reduce the natural language sentences, i.e., event descriptions, to a small set of observations and states based on their Wordnet similarity. The second complication of narrative texts is that many events may be omitted either in the narration or by the event extraction process. More importantly, there is no indication of a time lapse or a gap in the story, so the standard forward-backward algorithm does not apply. To account for this, we allow the states to skip generating observations with some probability. This kind of HMMs, with insertions and gaps, have been considered previously in speech processing  and in computational biology  . We refine these models by allowing state-dependent missingness, without introducing additional “insert states” or “delete states” as in  .

Question: Which datasets do they evaluate on?
Answer: The Open Minds Indoor Common Sense (OMICS) corpus