Evaluations ::: Evaluation of Transition Dynamics	One matter of interest is whether or not the transitions are capturing what they are supposed to capture, appropriate sentiment. Since we used the sentiment tagger Vader for training tags, we again utilize it to evaluate whether using transitions of a certain sentiment actually leads the model to produce outputs with the given sentiment. To perform this evaluation, we give as input to our models (and the S2S baseline) the sentiment tags for a sentence and allow it to generate a sentence conditioned on these sentiment tags. We then tag the generated sentences with Vader and see if the sentiment tags match the originals. We calculate the F1 score across all sentiment tags and report the macro average. In Table TABREF47 we see that having labels is incredibly important for meaningful transitions. There is a large drop in F1 as the amount of labels given to the model is decreased. The SLDS model that is trained with 100% of the labels performs a little better than even S2S, despite not having direct access to the sentiment labels (SLDS only uses the sentiment labels to decide which transition to use while the S2S model uses attention directly on the sentiment labels).

Evaluations ::: Automatic Evaluation of Interpolations	We automatically evaluate on four different types of interpolations (where different combinations of sentences are removed and the model is forced to regenerate them), We evaluate the generations with the ROUGE  and METEOR  metrics using the true sentences as targets. Table TABREF33 shows the automatic evaluation results from interpolations using our proposed models and baselines. The #Sent(s) column indicates which sentence(s) were removed, and then regenerated by the model. We gave the baselines a slight edge over SLDS because they pick the best out of 1000 samples while SLDS is only out of 50. The SLDS models see their largest gain over the baseline models when at least the first sentence is given as an input. The baseline models do better when the first and second sentence need to be imputed. This is likely due to the fact that having access to the earlier sentences allows a better initialization for the Gibbs sampler. Surprisingly, the semi-supervised variants of the SLDS models achieve higher scores. The reasons for this is discussed below in the Perplexity section.

Evaluations ::: Generating the Interpolations	For the SLDS models, the interpolations are generated via the Gibbs sampling algorithm described earlier. In all experiments for the SLDS models we draw 50 samples (including burn in samples) and output the interpolation that maximizes the probability of the given sentence(s). Since the baselines do not have the means for doing interpolations, we simulate `interpolations' for the baselines; we draw 1000 samples using top k (with k=15) truncated sampling (conditioned on the given initial sentences, if available). We then output the sample that maximizes the probability of the clamped sentences around which we are interpolating the others. We allow the S2S access to the gold sentiment tags. To give a lower bound on the performance of the SLDS model, we do not provide it with gold tags. We instead provide the SLDS model with the semi-noisy tags that are output from $q(S_i | X)$.

Evaluations ::: Human Evaluation of Interpolations ::: Human Evaluation Results	Table TABREF41 reports the result of human evaluations of SLDS and baseline generations. We can observe that people preferred narratives generated by SLDS over the ones generated by baseline models (LM and S2S) as they found the former model more coherent, which is an important criteria for narrative generation. 51.3% of the time SLDS generates better narratives than the LM model while LM in turn does it only 35.0% of the times. 13.7% of the generations end up in tie. The mean sentence level coherence score for SLDS is around 12.5% larger than that of the LM, with a slightly lower standard deviation. We see similar results when compared against the S2S model.

Evaluations	As described above, our model is able to perform narrative interpolations via an approximate Gibbs sampling procedure. At the core of our evaluations is thus a fill-in-the-sentences task. We provide 1 or 2 sentences, and require the model to generate the rest of the narrative . We evaluate this via automatic evaluations as well as with crowd-sourced human evaluations. We also report perplexity to evaluate the models' ability to fit the data. Lastly, we look at whether the transitions learned by the SLDS models capture what they are intended to capture: does using the transition matrix associated with a sentiment tag (positive/negative/neutral) lead to a generated sentence with that sentiment?

A Switching Dynamical System for Narrative Generation ::: Narrative Scaffold - Emotional Trajectory	What the switching variables actually represent can be chosen by the user. Straightforward narrative scaffolds include event sequences , keywords , or latent template ids . More complex but potentially more informative scaffolds may be created using concepts such as story grammar non-terminals , , or character action taken throughout a story . In our work, we use the sentiment trajectory of the narrative as the scaffold. That is, each $S_i$ for a sentence indicates the overall coarse sentiment of the sentence (Positive, Negative, or Neutral). Though simple, the overall sentiment trajectory of a narrative is important in defining the high level `shape' of a narrative often shared among different narratives , . Furthermore, sentiment trajectory has been shown to be fairly useful in story understanding tasks , . We discuss in the conclusion future directions for using different types of scaffolds.

A Switching Dynamical System for Narrative Generation ::: Narrative Scaffolds as Switching Variables	A more fitting model would thus be a Switching Linear Dynamical System , , . In an SLDS, we assume there exists a set of $K$ different sets of dynamics, $\lbrace (A_1, \Sigma _1),...(A_K,\Sigma _K)\rbrace $. At time step $i+1$, one of these sets of dynamics is used. The one used depends on the value of a discrete variable at time step $i+1$ called the switching variable, $S_{i+1} \in \lbrace 1,...K\rbrace $: There is a switching variable $S_i$ associated with each time step. The switching variable value itself evolves over time by a prior Markov process, $P(S_{i+1} | S_{i})$. This top level chain of switching variables thus forms our narrative scaffold, indicating what transitions we must go through in the narrative, with the dynamics matrices indicating how they transition.

Training Details ::: Baselines	Language Model (LM): We train a two layer recurrent neural language model with GRU cells of hidden size 512. Sequence-to-Sequence Attention Model (S2S): We train a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512. Sentiments tags for a narrative (1 for each sentence) are given as input to the model and the corresponding sentences are concatenated together as the output with only one <eos> tag at the end. This model is trained with a 0.1 dropout. This model is comparable to the static model of , and other recent works employing a notion of scaffolding into neural generation (albeit adapted for our setting). Linear Dynamical System (LDS): We also train a linear dynamical system as discussed in Section SECREF1 as one of our baselines for fair comparisons. Apart from having just a single transition matrix this model has the same architectural details as SLDS.

Generating Narrative Text in a Switching Dynamical System	Early work on narrative modeling used explicit plans and goals to generate stories, but the language generation itself was restricted and inflexible. Modern methods use language models for more robust generation, but often lack an explicit representation of the scaffolding and dynamics that guide a coherent narrative. This paper introduces a new model that integrates explicit narrative structure with neural language models, formalizing narrative modeling as a Switching Linear Dynamical System (SLDS). A SLDS is a dynamical system in which the latent dynamics of the system (i.e. how the state vector transforms over time) is controlled by top-level discrete switching variables. The switching variables represent narrative structure (e.g., sentiment or discourse states), while the latent state vector encodes information on the current state of the narrative. This probabilistic formulation allows us to control generation, and can be learned in a semi-supervised fashion using both labeled and unlabeled data. Additionally, we derive a Gibbs sampler for our model that can fill in arbitrary parts of the narrative, guided by the switching variables. Our filled-in (English language) narratives outperform several baselines on both automatic and human evaluations.

A Switching Dynamical System for Narrative Generation ::: Narrative Dynamics in a Dynamical System	The specifics of the narrative (characters, setting, etc.), will differ between stories, but as  notes, the way they transition to the next point in the narrative (what we refer to as “narrative dynamics") is often shared. Let's say that, as done often, we represent the `narrative specifics' at time step $i$ with a latent vector $Z_i$. A natural way to explicitly model how this state evolves over time that fits with the above observation is as a Linear Dynamical System: Where $A$ is a matrix, shared across all narratives, and $\Sigma $ is a noise term that takes into consideration idiosyncrasies different narratives will have. The fact that the shared transition matrix $A$ is linear means that narratives will have linearly analogous trajectories through time, despite having different details (comparable to stories with different settings but matching structures such as Ran/King Lear, Ulysses/Odyssey, etc). Of course, the fatal flaw of the model is that it assumes there exists only one transition matrix, and thus only one possible way to transition through a narrative!

Training Details ::: Switching Linear Dynamical System (SLDS)	SLDS has RNN encoder and decoder networks with single layer GRU cells of hidden size 1024. Model uses an embedding size of 300. We train the model using Adam optimizer with the defaults used by PyTorch. We stop training the models when the validation loss does not decrease for 3 consecutive epochs. Training details remain same as above unless otherwise mentioned.

Semi-Supervised SLDS (SLDS-X%): To gauge the usability of semi-supervision, we also train semi-supervised SLDS models with varying amount of labelled sentiment tags unlike the original model which uses 100% tagged data. We refer to these as SLDS-X%, where X is the % labelled data used for training: 1%, 10%, 25%, and 50%.

Interpolations via Gibbs Sampling ::: Gibbs Sampling Interpolation Overview	The variables in the Gibbs sampler are first initialized using some heuristics (see Supplemental Materials for details). After initialization, performing the interpolations with Gibbs sampling follows the below two step process: For each $Z_i$, sample a value $Z^\prime $ from equation $(1)$ and set $Z_i$ to $Z^\prime $. For each $X_i$ in $\mathbf {X}^-$, find a new value for $X_i$ by running greedy decoding using the conditional language model.

By using the d-separation properties of the graph, and substituting the true posterior over $Z_{i}$ with our approximate posterior $q$, we can show the first distribution is approximately proportional to The last line is the product between a Gaussian density over $Z_{i+1}$ and $Z_{i}$, respectively. With some algebraic manipulations, one can show the last line is proportional to a single Gaussian PDF over $Z_i$: To find the second conditional, one can use the d-separation properties of the graph to find that it is proportional to: These two distributions are simply factors of our conditional language model, and both terms can thus be evaluated easily. In theory, one could use this fact to sample the original conditional via Metropolis-Hastings . Unfortunately, we found this approach to be much too slow for practical purposes. We observed that the simple heuristic of deterministically assigning $X_i$ to be the greedy decoded output of the conditional language model $P(X_{i} | X_{:i}, Z_{i})$ works well, as evidenced by the empirical results. We leave it for future work to research different conditional language model parameterizations that allow easy sampling from this conditional

Evaluate the data likelihood and KL term(s) with these samples. Take the gradients of the objective function w.r.t. all parameters, using the reparameterization trick for $q_{Z_i}$  or the Gumbel-Softmax trick for $q_{S_i}$ , and optimize.

Conclusion and Future Work	In this paper, we formulated the problem of narrative generation as a switching dynamical system. We showed how this formulation captures notions important in narrative generation, such as narrative dynamics and scaffolds. We developed an approximate Gibbs sampling algorithm for the model that permits the system to generate interpolations conditioned on arbitrary parts of the narrative, and evaluated these interpolations using both human and automatic evaluations. Though in this work we used sentiment tags for our scaffolds/switching variables, future work may look at utilizing different kinds of information to guide the generation of narratives. Utilizing the main predicate of a sentence as a scaffold would be a logical next step, and may prove more informative then the sentiment trajectory. A scaffold such as this can take on many more possible values then a sentiment tag, and as such, it may prove difficult to assign a set of dynamics to each value. Another avenue for future work would deal with this possible problem. One potential solution could be to associate each switching variable value with a (learned) vector in a probability simplex, and use this vector to combine a small set of “primitive" dynamics matrices in order to get that value's associated set of dynamics.

A Switching Dynamical System for Narrative Generation	In this section, we give a brief overview of Switching Dynamical systems and how they can be used to capture both a scaffold of the narrative as well as the narrative dynamics. We then describe in detail the components of our model and its relation to existing models.

Evaluations ::: Language Modeling Perplexity Score	As our models are essentially language models, we evaluated their per-sentence negative log-likelihood and per-word perplexity scores, which can be viewed as an indirect measure of how well a system works as a generative model of narrative text. For the SLDS and LDS models these scores are approximations, an upper bound (the negative of the ELBO) to the actual values. For the other two models the scores are exact. A good model should assign low perplexity scores to its test set. In Table TABREF44 SLDS achieves the lowest scores, implying that it is able to model the data distribution well. In Table TABREF45 we also calculate the perplexity scores for the semi-supervised SLDS models to assess the effectiveness of semi-supervised training. Surprisingly, the models with less supervision scored better in terms of perplexity. One possibility for this might be the use of the soft Gumbel-Softmax in the semi-supervised models. The soft Gumbel-Softmax variant does not commit to using a single transition matrix at each time step (instead linearly combining them, weighted by the Softmax weights).

(2) The second form, $q(Z_i| Z_{i-1}, S_i, X_{:i}, X_{i}) = q_{Z_i}$ is realized by functions $f_{\mu }(Z_{i-1}, S_i, X_{:i}, X_{i})$ and $f_\sigma (Z_{i-1}, S_i, X_{:i}, X_{i})$ that output the mean and variance, respectively, of a Gaussian over $Z_i$. Borrowing terminology from VAEs, the approximate posterior (the factors given above) act as an `encoder', while the generative model from the previous section can be seen as the `decoder'. This type of training has been previously used in , , , , .

Interpolations via Gibbs Sampling	One of the benefits of probabilistic formulation is the possibility (if an inference procedure can be found) of generating narratives with specific constraints, where the constraints may be specified as clamped variables in the model. In this section, we show how narratives may be generated conditioned on arbitrary bits and pieces of the narrative already filled in, using approximate Gibbs sampling. This allows one to, for example, interpolate a narrative given the first and the last sentence (similar to how earlier story generation systems were able to generate with a given end goal in mind). Some examples of these interpolations generated by our system can be found in Table TABREF37. We give the equations and summarize the algorithm in the next sections.

Learning and Posterior Inference	Due to the conditionals parameterized by neural networks we use amortized variational inference in a manner similar to Variational AutoEncoders , both to learn an approximate posterior $q(S, Z | X)$ and to learn the generative model parameters by maximizing a lower bound on the data likelihood (ELBO). We assume that the approximate posterior factorizes as follows: Like in VAEs, computing these individual factors is done through a parameterized function called the inference or recognition network whose parameters are trained jointly with the generative model. In our case there are two forms for the factors in our posterior: (1) The first form, $q(S_i | \textbf {X}) = q_{S_i}$ is parameterized by a classifier that takes in the set of sentences $\mathbf {X}$ and outputs a categorical distribution over the switching variables.

Training Details ::: Dataset and Preprocessing	We use the ROCStories corpora introduced in . It contains 98,159 short commonsense stories in English as training, and 1,570 stories for validation and test each. Each story in the dataset has five-sentences and captures causal and temporal commonsense relations. We limit our vocabulary size to 16,983 based on a per-word frequency cutoff set to 5. For sentiment tags, we automatically tag the entirety of the corpus with the rule based sentiment tagger, Vader , and bucket the polarity scores of Vader into three tags: neutral, negative, and positive. These tags form the label set of the $S$ variables in our SLDS model. We tokenize the stories with Spacy tokenizer. Each sentences in the input narrative has an <eos> tag except for the S2S model discussed below.

Learning and Posterior Inference ::: Lower bound formula & exact training algorithm	As mentioned previously, we optimize all parameters (including the variational factor functions) by optimizing a lower bound on the data likelihood. The model may be trained either with supervision labels for the switching states (in our case, sentiment labels) or without supervised labels. If one is training without the sentiment labels, then the lower bound on the marginal likelihood (and thus our optimization objective) may be written as follows: The derivation for this objective is identical to that found in , , and simply relies on using properties of iterated expectations. All expectations are estimated with Monte Carlo samples. If training with the sentiment labels $S_1,...,S_N$, then the objective is similar (but without the sampling of the switching states), and is augmented with an additional supervision objective as done in : Final training procedure for a single narrative is: For each sentence (starting from the first), sample the switching state $S_i$ from $q(S_i | \textbf {X})$. For each sentence (starting from the first), sample the latent $Z_i$ from $q(Z_i | S_i, Z_{i-1}, X)$.

Related Work	Story/narrative generation has a rich history in the field of AI. Many early systems were based on structured formalisms for describing common narrative structures , , , many being inspired by the initial work of . There has been a swath of recent work that has looked to add some semblance of a `narrative scaffold' back into generation methods , , , . Many of these methods work as conditional LMs (conditioned directly on the scaffold). This line of work may be combined with our formalization as well, by conditioning the generation on the switching state as well, as done in the model of . Recent work by  has similar goals to ours in permitting more controlability in generation systems, developing a RL-based system that allows users to specify an end goal for a story (by specifying the event class that is desired to appear at the end). Their work differs from ours in that it does not deal with text directly, modeling only the sequences of events in the narrative. It may be possible to utilize this model as the scaffolding component in our model (utilizing their RL policy for the scaffold planner, rather than the simple Markovian distribution used here).

Interpolations via Gibbs Sampling ::: Conditionals for Gibbs Sampling	For our Gibbs sampling algorithm we give the narrative scaffold (switching variables), $S_1,...,S_T \in \mathbf {S}$ and a set of observed sentences, $\mathbf {X^+}$. This may be any set of sentences (the first and last, just the second sentence, etc) as inputs to the system. We wish to find values for the unobserved sentences in set $\mathbf {X^-}$ by sampling from the distribution $P(\mathbf {X^-}, Z_1,...,Z_T | \mathbf {S},\mathbf {X^+})$. We perform this sampling via Gibbs sampling. Two different forms of conditionals need to be derived to do Gibbs sampling. One over some $Z_i$ conditioned on everything else, and one over some $X_i$ conditioned on everything else.

The joint over all variables factorizes as below into the following components ($X_{:i}$ stands for all sentence before $X_i$): ❶ Narrative Scaffold Planner: The factor $P(S_i | S_{i-1})$ is a transition matrix, which we calculate via count based statistics from training. It is fed in as prior knowledge and fixed. ❷ Narrative Dynamics Network: The factor $P(Z_i | Z_{i-1}, S_i)$ is determined like a switching linear dynamical system: which is equivalent to drawing $Z_i$ from a Normal distribution with mean $A_{S_i}Z_{i-1}$ and variance $B_{S_i}B_{S_i}^T$. ❸ Conditional Language model: The factor $P(X_i | Z_i, X_{:i})$ is parameterized by an RNN language model conditioned on the latent $Z_i$.

Evaluations ::: Human Evaluation of Interpolations ::: Annotation Scheme	As automatic evaluation metrics are not sufficient to assess the quality of any creative task such as narrative generation, we measure the quality of the generations through human evaluation of 200 stories on the Amazon Mechanical Turk platform. We provided Turkers with two generated narratives from two different models, each with five sentences. The first and last sentences were fed to each model as input, and the middle three sentences were generated. Each pair of narratives is graded by 3 users each with two tasks: (1) to rank on a scale of 0-3 each of the sentences except the first one on the basis of its coherency with the previous sentence(s) and (2) compare and rank the two narratives based on their overall coherency, ie how well the story connects the starting/ending sentences.

This fact may permit the model greater flexibility in fitting the training data. While this leads to better scores in metrics such as perplexity or BLEU, it does leads to transitions that are worse in capturing the properties they should be capturing, as we shall see in the next section.

A Switching Dynamical System for Narrative Generation ::: The Full Model	The final component of the model is a conditional language model that generates sentence $i$ conditioned on the current $Z_i$, and all previous sentences, $X_{:i}$. Generation continues until an <eos> is reached. This conditional language model may be parameterized as desired, but in this work, we parameterize it as an RNN neural network language model. The graphical model for our SLDS is pictured in Figure FIGREF8. The model consists of three sets of variables: (1) Switching variables $S_1,...,S_N$, (2) Latent state variables $Z_1,...,Z_N$ capturing the details of the narrative at sentence $i$, (3) The sentences themselves $X_1,...X_N$, where each sentence $X_i$ has $n_i$ words, $x^i_1,...x^i_{n_i}$.

Question: What baselines are used?
Answer: a two layer recurrent neural language model with GRU cells of hidden size 512, a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512, a linear dynamical system, semi-supervised SLDS models with varying amount of labelled sentiment tags