Venue: ACL 2020

# Toward Better Storylines with Sentence-Level Language Models

Daphne Ippolito+

daphneui@seas.upenn.edu

David Grangier+

grangier@google.com Douglas Eck+

deck@google.com &Chris Callison-Burch+

ccb@seas.upenn.edu

University of Pennsylvania, \({}_{t}\)Google

###### Abstract

We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embeddings and predicts an embedding for the next sentence, which is more efficient than predicting word embeddings. Notably this allows us to consider a large number of candidates for the next sentence during training. We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks.

## 1 Introduction

Computer generation of stories and other kinds of creative writing is a challenging endeavor. It entangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline. In the recent year, neural language models have made tremendous progress with respect to fluency Bahdanau et al. (2015); Vaswani et al. (2017); Bengio et al. (2003); Devlin et al. (2019), but coherency is still a major challenge See et al. (2019). The generation of coherent stories has recently been addressed with additional conditioning: Fan et al. (2018) suggest conditioning on a story prompt, Clark et al. (2018) propose collaboration between a generative model and a human writer, and Guan et al. (2019) suggest attending to a commonsense graph relevant to the story plot. Conditioning based on a generated story plan Martin et al. (2018); Fan et al. (2019); Yao et al. (2019), a sequence of images Chandu et al. (2019) or character roles Liu et al. (2020) have also been considered.

Our work is orthogonal to these efforts. Rather than considering additional conditioning, we propose a model which takes as input several sentences of context and selects the best next sentence within a large set of fluent candidate sentences. We leverage pre-trained BERT embeddings Devlin et al. (2019) to build this sentence-level language model. Given the embeddings of the previous sentences of the story, our model learns to predict a likely embedding of the next sentence.

This task isolates the modeling of long-range dependencies from the prediction of individual words, which has several advantages. First, since our model only needs to determine how well each candidate sentence would fit as a coherent continuation to the story, it does not spend capacity and time to learn fluency. Second, our model does not manipulate individual words but full sentences, which allows us to consider tens of thousands of candidate sentences at a time. This contrasts with prior work Logeswaran and Lee (2018) where the need to learn token-level representations limited the number of candidate next sentences that could be considered to a few hundred. Third, we can rely on compact model architectures that train quickly because we take advantage of strong semantic representation from a pre-trained bidirectional language model, BERT, as our sentence embeddings. Of course, these benefits also imply that our sentence representation is limited to the information extracted by the pre-trained model. Nevertheless, we show that our model achieves state-of-the-art accuracy among unsupervised approaches on the Story Cloze task: predicting which of two sentences coherently ends a short story.

Our work also opens up the possibility of ranking thousands of candidate sentences from a large literature repository. On the ROC Stories dataset,

we observe that training with a large number of candidates is key for selecting the most coherent ending among a large set of candidates at test time. We also show preliminary results on the efficacy of our method for ranking candidate next sentence on the Toronto Book Corpus Kiros et al. (2015), a much larger book dataset. We envision that our methods for scoring many candidate next sentences by their coherence with the context might be useful to downstream generation tasks where it is possible to generate many fluent continuations of a text, but it remains an unsolved problem how to refine and choose the best of them. To encourage this exploration, we release our code and models1.

Footnote 1: Code for ROC stories experiments can be found at [https://github.com/google-research/google-research/tree/master/better_storylines](https://github.com/google-research/google-research/tree/master/better_storylines).

## 2 Proposed Method

We propose a sentence-level language model: our model estimates \(P(s_{t+1}|s_{1:t})\), the probability distribution for sentence \(s_{t+1}\) given the \(t\) previous sentences, \(s_{1},\ldots s_{t}\). Since it is intractable to marginalize over all possible candidate next sentences, we consider a finite but large set of \(N\) valid, fluent sentences. Without loss of generality, we can consider \(s_{t+1}\in\{1,\dots,N\}\) as an integer index into that set of possible next sentences. This strategy resembles negative sampling in word2vec Mikolov et al. (2013).

Our model represents sentences with precomputed vector embeddings. Specifically, sentences are represented by the mean of the 768-dimensional contextual word embeddings of the second-to-last layer of BERT Devlin et al. (2019). This representation has shown to encode more transferable features compare to other layers Liu et al. (2019). Alternative sentence representations were considered, including embeddings from the universal sentence encoder Cer et al. (2018) and a weighted mean of the BERT embeddings using inverse document frequency weighting Zhang et al. (2019). None of these alternatives improved our results however.

Motivated by simplicity, we consider a classical multi-layer perceptron (MLP) \(f_{\theta}\) which takes as input the context sentence embeddings concatenated into a single vector. At the output layer, we perform a softmax operation. If we represent candidate sentences \(\{1,\dots,N\}\) by the embeddings \(\{e_{i}\}^{N}_{i=1}\), our model estimates the probability that \(i\) is the next sentence by the softmax

\[\log P(s_{t+1}=i|s_{1:t})=e_{i}^{\top}h-\log Z(h)\]

where \(h=f_{\theta}(s_{1:t})\) is the output of the MLP given context \(s_{1:t}\), and \(Z(h)=\sum_{j=1}^{N}\exp e_{j}^{\top}h\) is the partition function. At train time, the candidate set \(\{1,\dots,N\}\) consists of the correct next sentence along with \(N-1\) distractor sentences. The distractors can either be static (the same set used throughout training) or dynamic (picked at random from a larger set for each train batch). In this case, the "vocabulary" of next values to choose from changes with each train step, similar to negative sampling Mikolov et al. (2013). At test time, novel sentences can be embedded with BERT and scored by our model.

Like a classical language model, we optimize for the likelihood of the true next sentence's embedding. However, when training we found that the sentences from the context (\(s_{1},\dots,s_{t}\)) often ended up being given very high scores by our model. Inspired by work in sentence reordering Lapata (2003); Logeswaran and Lee (2018), we incorporated an auxiliary loss, which we refer to as **CSLoss**, that only includes the context sentences \(s_{1:t}\) in the distractor set.

Lastly, we consider a residual variant of the MLP (referred to as **resMLP**) with skip connection between layers, as described in He et al. (2016). The residual model trains faster and sometimes achieves higher accuracy than the non-residual model. Though we experimented with recurrent Sundermeyer et al. (2012) and self-attention Vaswani et al. (2017) models, we did not observe improvements, perhaps because the input to our model is already the high-dimensional output of a large mask language model. We leave deeper architecture exploration, which will be especially critical as context length is extended, to future work.

## 3 Experimental Setup

We first describe our experiments on the ROC Stories dataset of short 5-sentence stories before showing our setup on the larger Toronto Book Corpus.

### ROC Stories

DatasetOur experiments use the ROC Stories dataset, which consists of stories focusing on common sense Mostafazadeh et al. (2016). The training set has 98k stories, with five sentences each.

 

The validation and test sets each contain 1.8k stories consisting of four sentences followed by two alternative endings: one ending is coherent with the context; the other is not. The dataset was introduced for the Story Cloze task, inspired by Taylor (1953), where the goal is to select the coherent ending. While the dataset and task were introduced as a way to probe for coherence and commonsense in models trained only on the unlabeled portion, most research derived from this dataset focuses on a supervised setting, using the validation set as a smaller, labeled training set Chaturvedi et al. (2017); Sun et al. (2019); Cui et al. (2019); Li et al. (2019); Zhou et al. (2019). Our work is faithful to the original task objective. We train solely on the training set, i.e. the model never sees incoherent endings at training time.

ModelWe consider two models, an MLP and a residual MLP. They take as input the previous sentences represented as the concatenation of their embeddings. Alternative context aggregation strategies were considered with recurrent Sundermeyer et al. (2012) and attention Vaswani et al. (2017) architectures, without strong empirical advantages. The models maps its input to a vector which is compared to a set of candidate sentence embeddings via dot product. The embedding of the true next sentence should receive the highest score. For each example, we consider all other fifth sentences in the training set (96k in total) as the candidate set.

The input of our model is 3,072 dimensional, i.e. 4 context sentences represented by 768 dimensional BERT embeddings. After an architecture search, our best MLP has 3 layers of 1,024 units, and our best resMLP has a single residual layer with hidden size of 1,024. Both contain just over 6M trainable parameters. Both apply dropout with a rate of 0.5 after each ReLU, and layer normalization is performed on the concatenated context sentence embedding passed in as input to the network and on the final predicted embedding for the next sentence. For the Story Cloze task, the two architectures achieve similar validation accuracy, but when considering more than two distractors, the resMLP significantly outperforms the standard MLP. The resMLP also converges quicker than the MLP. Training to convergence takes under 2 hours for each model on a Tesla V100.

### Toronto Book Corpus

DatasetROC Stories contains only self-contained five-sentence stories, focusing on everyday life scenarios. They contain no dialog and very little flowery, expository language. Ideally our method would also be successful at scoring potential continuations to more naturally-written stories. To this end, we test out our approach on excerpts from the Toronto Book Corpus Kiros et al. (2015), a dataset of self-published novels. The dataset contains over 7,000 unique books totalling over 45 million sentences. Since these stories are much longer than the ROC Stories ones and many of the sentences are uninformative (nearly 5% of sentences are 3 words or shorter, and 14% are 5 words or shorter), we double the context length to 8 sentences.

ModelIn addition to experimenting with a similar residual MLP architecture to the one used on ROC Stories, we also ran experiments with a Transformer model Vaswani et al. (2017). The residual MLP architecture contains 2 residual layers with hidden size of 1024 (11M params total). The transformer has 4 self-attention layers with hidden size of 768, filter size of 2048 and 8 attention heads (22M params total). While the residual MLP is trained to predict the 9th sentence given the previous 8 sentences, the Transformer is trained to predict each next sentence given the previous sentences in a sequence of length 10 sentences. However, we only evaluate the Transformer on the task of predicting the 9th sentence so that evaluation results are directly comparable to the residual MLP.

\begin{table}
\begin{tabular}{l l||r|r|r|r}  & & Valid 2016 & Test 2016 & Valid 2018 & Test 2018 \\ \hline Our model & MLP & 69.7 & 68.8 & 70.1 & 69.0 \\  & + CSLoss & **73.5** & **73.0** & **73.1** & **72.1** \\ \hline Alternatives & Peng et al. (2017) & – & 62.3 & – & – \\  & Schenk and Chiarcos (2017) & 62.9 & 63.2 & – & – \\ \hline Lang. Models & Schwartz et al. (2017) & – & 67.7 & – & – \\  & GPT-2 Radford et al. (2019) & 54.5 & 55.4 & 53.8 & – \\  & GPT-2 + finetuning & 59.0 & 59.9 & 59.0 & – \\ \end{tabular}
\end{table}
Table 1: Accuracies (%) for the Story Cloze binary classification task. Schwartz et al. (2017) is a semi-supervised technique. GPT-2 refers to predicting the more likely ending according to the 355M parameter model, and GPT-2 finetuning was done on the ROC Stories train set.

 

For each batch during training, 2k distractors are randomly selected from the train set. Like with ROC Stories, we experiment with an auxiliary loss where just sentences from the context were used as distractors. Table 3 reports the results.

## 4 Results

We evaluate on the Story Cloze task, a binary classification task, as well as on the task of ranking a large set of possible next sentences.

### Story Cloze Task

Table 1 shows that our method outperforms unsupervised alternatives. The introduction of the CSLoss which considers only context sentences as candidates improves accuracy compared to only using a loss over all possible fifth sentences.

For comparison, we include the accuracies of the best unsupervised methods in the literature. Schenk and Chiarocs (2017) construct negative examples for their binary classification task by pairing contexts with random fifth sentences selected from the training set. Peng et al. (2017) train a language model to predict a representation of the semantic frame, entities, and sentiment of the fifth sentence given the representations of the previous sentences, then take the more likely fifth sentence. We achieve higher accuracy without relying on a task-specific architecture.

Table 1 also shows that picking the ending that is more likely according to a word-level language model, in our case GPT-2's 355M parameter model, does not yield very high accuracies, even when the language model is finetuned on ROC Stories text Radford et al. (2019). Lastly, we also include the accuracy reported by Schwartz et al. (2017), where a logistic classifier is trained to combine multiple language model scores.

It is worth noting that state-of-the-art on the Story Cloze task is over 90% accuracy Li et al. (2019); Cui et al. (2019) for semi-supervised settings. The methods achieving this level of performance are not comparable to our _unsupervised_ approach as they require training on the _labeled_ validation set. The language model approach from Schwartz et al. (2017) also falls into this category.

### Ranking Many Sentences on ROC Stories

For generation and suggestion scenarios, it is useful to be able to surface the best next sentence out of hundreds or thousands of candidates. In Table 3, we show the performance of our method on the 2018 validation set when all 98,161 fifth sentences in the training set plus all 1,571 correct 5th sentences in the 2018 validation are considered as candidate endings. Top-10 accuracy is highest, at 10.3%, when training a residual MLP without CSLoss.

Interestingly, strong performance on the Story Cloze task does not necessarily translate to strong performance on the large-scale ranking task. The CSLoss improves performance on the Story Cloze task but hurts it for large-scale ranking.

In Figure 1, we show how large-scale ranking performance improves as the size of the train-time distractor set is increased. However, on the Story Cloze task, the number of training distractors has no significant impact on performance. Even when only a single distractor is randomly chosen at each step of training, our method achieves over 70% 2016 test accuracy. It seems that training for the goal of detecting the true next sentence out of a very diverse candidate set is useful at test time only when the set of distractors at test time is similarly large and diverse. The many-distractors training regime might be less useful for the Story Cloze task since the two candidate endings are designed to be quite topically similar to each other.

Some qualitative examples are shown in Table

\begin{table}
\begin{tabular}{l||c c}  & P@10 & MRR \\ \hline MLP & 6.2 & 0.052 \\ +CSLoss & 3.4 & 0.029 \\ \hline ResMLP & **10.3** & **0.087** \\ +CSLos & 6.2 & 0.051 \\ \hline Random & 0.01 & 2e-5 \\ \end{tabular}
\end{table}
Table 2: Precision@10 and mean-reciprocal rank on the 2018 valid set when considering all 5th sentences in the train and valid sets (98k total) as candidate endings.

Figure 1: The impact of the number of negative sentences used during training on the rank of the true ending out of 98k distractors. Results are with the resMLP on the 2018 valid set.

 

4. The failure examples showcase a side-effect of relying on pre-trained sentence embeddings: if common names like "Becky" or "Laura" or sports such as "fishing" and "golf" are close to each other in embedding space, our model will fail to distinguish between them.

### Ranking Many Sentences on Toronto Book Corpus

When evaluating with 100k distractors, about as many as our ROC Stories large-scale ranking task, P@10 is at best 7.1%, compared with 22.7% for ROC Stories. We suspect that this task would benefit from longer contexts and better selection of distractors. In particular, a qualitative evaluation of the data highlighted the presence of a large quantify of short, generic sentences in the high ranking sentences (e.g. "he said." and "Yes."). We see reducing the density of such sentences at training time as a potential for improvement.

In addition, further investigation is necessary into why the Transformer did not work as well as the residual MLP. The use of variable sequence length architectures like the Transformer will become more critical as the input sequence length is increased beyond what an MLP can easily handle.

## 5 Conclusions

This work introduces a sentence-level language model which takes a sequence of sentences as context and predicts a distribution over a finite set of candidate next sentences. It takes advantage of pretrained BERT embeddings to avoid having to learn token-level fluency, allowing the model to focus solely on the coherence of the sentence sequences. Our results on the Story Cloze task highlight the advantage of this strategy over word-level language models. At train time, our model considers much larger amounts of text per update than typical token-level language models. We show that this strategy allows our model to surface appropriate endings to short stories out of a large set of candidates.

As future work, we plan to further evaluate the impact of different sequential architectures, longer contexts, alternative sentence embeddings, and cleverer selection of distractors. Inspired by deliberation networks and automatic post editing methods Xia et al. (2017); Freitag et al. (2019), we ultimately want to apply our model to two-step generation, first selecting a sentence from a large set before refining it to fit the context.

## Acknowledgements

This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

\begin{table} \end{table}
Table 4: Top-scoring sentences (using resMLP without CSLoss) among 98k possible endings when using prompts from the validation set. Two success and two failures cases are shown.

\begin{table} \end{table}
Table 3: Precision@10 On Toronto Book Corpus for retrieving the correct next sentence (given the 8 previous sentences) when considering 10k or 100k distractor sentences, or all of the sentences from the same book as distractors.

 



## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _International Conference on Learning Representation (ICLR)_.
* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. _Journal of Machine Learning Research (JMLR)_, 3(Feb):1137-1155.
* Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. _arXiv preprint arXiv:1803.11175_.
* Chandu et al. (2019) Khyathi Chandu, Eric Nyberg, and Alan W Black. 2019. Storyboarding of recipes: Grounded contextual generation. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Chaturvedi et al. (2017) Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. pages 1603-1614.
* Clark et al. (2018) Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In _23rd International Conference on Intelligent User Interfaces_, pages 329-340. ACM.
* Cui et al. (2019) Yiming Cui, Wanxiang Che, Wei-Nan Zhang, Ting Liu, Shijin Wang, and Guoping Hu. 2019. Discriminative sentence modeling for story ending prediction. _CoRR_, abs/1912.09008.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
* Fan et al. (2019) Angela Fan, Mike Lewis, and Yann N. Dauphin. 2019. Strategies for structuring story generation. In _Association for Computational Linguistics, ACL_, pages 2650-2660. Association for Computational Linguistics.
* Freitag et al. (2019) Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 34-44, Florence, Italy. Association for Computational Linguistics.
* Guan et al. (2019) Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI_, pages 6473-6480.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Advances in neural information processing systems_, pages 3294-3302.
* Volume 1_, ACL '03, pages 545-552, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Li et al. (2019) Zhongyang Li, Xiao Ding, and Ting Liu. 2019. Story ending prediction by transferable BERT. _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_, pages 1800-1806.
* Liu et al. (2020)Danyang Liu, Juntao Li, Meng-Hsuan Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020. A character-centric neural model for automated story generation. In _AAAI Conference on Artificial Intelligence_.
* Liu et al. (2019) Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1073-1094.
* Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In _International Conference on Learning Representations_.
* Martin et al. (2018) Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2018. Event representations for automated story generation with deep neural nets. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of the 26th International Conference on Neural Information Processing Systems_

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 839-849, San Diego, California. Association for Computational Linguistics.
* Peng et al. (2017) Haoruo Peng, Snigdha Chaturvedi, and Dan Roth. 2017. A joint model for semantic sequences: Frames, entities, sentiments. pages 173-183.
* Radford et al. (2019) Alec Radford, Jeffrey Wu Wu, Rewon Child, David Luan Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Schenk and Chiarcos (2017) Niko Schenk and Christian Chiarcos. 2017. Resource-lean modeling of coherence in commonsense stories. In _Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics_, pages 68-73.
* Schwartz et al. (2017) Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. 2017. Story cloze task: Uw nlp system. In _Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics_, pages 52-55.
* See et al. (2019) Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In _Conference on Natural Language Learning CONLL_.
* Sun et al. (2019) Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 2633-2643.
* Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In _Thirteenth annual conference of the international speech communication association_.
* Taylor (1953) Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. _Journalism Bulletin_, 30(4):415-433.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Xia et al. (2017) Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 1784-1794. Curran Associates, Inc.
* Yao et al. (2019) Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In _AAAI Conference on Artificial Intelligence_.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zhou et al. (2019) Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2019. Story ending selection by finding hints from pairwise candidate endings. _IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)_, 27(4):719-729.