SlowGuess/ACL_2020_OCR huggingface dataset:
To further use the information across two consecutive lines, ScriptWriter piles up all the context-narrative matching matrices and all the context-response matching matrices to construct two cubes  $\mathbf{Q}_{cp} = \{\mathbf{M}_{s_i,p}[j,k]\}_{i=1}^n$  and  $\mathbf{Q}_{cr} = \{\mathbf{M}_{s_i,r}[j,k]\}_{i=1}^n$ , where  $n$  is the number of lines in the session. Then ScriptWriter employs 3D convolutions to distill important matching features from the whole cube. We denote these two feature vectors as  $f(c,p)$  and  $f(c,r)$ . For narrative-response matching, ScriptWriter conducts 2D convolutions on  $\mathbf{M}_{p,r}$  to distill matching features between the narrative and the response, denoted as  $f(p,r)$ .
The three types of matching features are concatenated together, and the matching score  $g(c,p,r)$  for ranking response candidates is computed by an MLP with a sigmoid activation function, which is defined as:
$$
f (c, p, r) = [ f (c, p); f (c, r); f (p, r) ], \tag {12}
$$
$$
g (c, p, r) = \operatorname {s i g m o i d} \left(\mathbf {W} ^ {T} f (c, p, r) + b\right), \tag {13}
$$
where  $\mathbf{W}$  and  $b$  are parameters.
ScriptWriter learns  $g(c, p, r)$  by minimizing cross entropy with  $\mathcal{D}$ . The objective function is formulated as:
$$
\begin{array}{l} L (\theta) = - \sum_ {(y, c, p, r) \in \mathcal {D}} [ y \log (g (c, p, r)) \\ \left. + (1 - y) \log (1 - g (c, p, r)) \right]. \tag {14} \\ \end{array}
$$
5.1 Evaluation setup
As presented in Table 1, we randomly split the the GraphMovie collection into training, validation and test set. The split ratio is 18:1:1. We split the sessions into micro-sessions: given a session with  $n$  lines in the context, we will split it into  $n$  micro-sessions with length varying from 1 to  $n$ . These micro-sessions share the same narrative. By doing this, the model is asked to learn to select one line as the response from a set of candidates at any point during the session, and the dataset, in particular for training, can be significantly enlarged.
We conduct two kinds of evaluation as follows:
Turn-level task asks a model to rank a list of candidate responses based on its given context and narrative for a micro-session. The model then selects the best response for the current turn. This setting is similar to the widely studied response selection task (Wu et al., 2017; Zhou et al., 2018b; Zhang et al., 2018). We follow these previous studies and employ recall at position  $k$  in  $n$  candidates  $(\mathrm{R_n}@\mathrm{k})$  and mean reciprocal rank (MRR) (Voorhees, 1999) as evaluation metrics. For example,  $\mathrm{R}_{10}@\mathrm{l}$  means recall at one when we rank ten candidates (one positive sample and nine negative samples). The final results are average numbers over all micro-sessions in the test set.
Session-level task aims to predict all the lines in a session gradually. It starts with the first line of the session as the context and the given narrative and predicts the best next line. The predicted line is then incorporated into the context to predict the
8652
next line. This process continues until the last line of the session is selected. Finally, we calculate precision over the whole original session and report average numbers over all sessions in the test set. Precision is defined as the number of correct selection divided by the number of lines in a session. We consider two measures: 1)  $\mathrm{P}_{\mathrm{strict}}$  which accepts a right response at the right position; 2)  $\mathrm{P}_{\mathrm{weak}}$  which accepts a right response at any position.
As no previous work has been done on narrative-based script generation, no proper baseline exists. Nevertheless, some existing multi-turn conversation models based on context can be adapted to work with a narrative: the context is simply extended with the narrative. Two different extension methods have been tested: the narrative is added into the context together with the previous lines; the narrative is used as a second context. In the latter case, two matching scores are obtained for context-narrative and narrative-response. They are aggregated through an MLP to produce a final score. This second approach turns out to perform better. Therefore, we only report the results with this latter method<sup>5</sup>.
(1) MVLSTM (Wan et al., 2016): it concatenates all previous lines as a context and uses an LSTM to encode the context and the response candidate. A matching score is determined by an MLP based on a map of cosine similarity between them. A matching score for narrative-response is produced similarly.
(2) DL2R (Yan et al., 2016): it encodes the context by an RNN followed by a CNN. The matching score is computed similarly to MVLSTM.
(3) SMN (Wu et al., 2017): it matches each line with response sequentially to produce a matching vector with CNNs. The matching vectors are aggregated with an RNN.
(4) DAM (Zhou et al., 2018b): it represents a context and a response by using self-attention and cross-attention operation on them. It uses CNNs to extract features and uses an MLP to get a score. Different from our model, this model only considers the context-response matching and does not track what in the narrative has already been expressed by the previous lines, i.e., context.
(5) DUA (Zhang et al., 2018): it concatenates the last line with each previous line in the context and response, respectively. Then it performs a self-attention operation to get refined representations, based on which matching features are extracted with CNNs and RNNs.
5.3 Training Details
All models are implemented in Tensorflow $^6$ . Word embeddings are pre-trained by Word2vec (Mikolov et al., 2013) on the training set with 200 dimensions. We test the stack number in  $\{1,2,3\}$  and report our results with three stacks. Due to the limited resources, we cannot conduct experiments with a larger number of stacks, which could be tested in the future. Two 3D convolutional layers have 32 and 16 filters, respectively. They both use [3,3,3] as kernel size, and the max-pooling size is [3,3,3]. Two 2D convolutional layers on narrative-response matching have 32 and 16 filters with [3,3] as kernel size. The max-pooling size is also [3,3]. All parameters are optimized with Adam optimizer (Kingma and Ba, 2015). The learning rate is 0.001 and decreased during training. The initial value for  $\gamma$  is 0.5. The batch size is 64. We use the validation set to select the best models and report their performance on the test set. The maximum number of lines in context is set as ten, and the maximum length of a line, response, and narrative sentence is all set as 50. All sentences are zero-padded to the maximum length. We also padded zeros if the number of lines in a context is less than 10. Otherwise, we kept the latest ten lines. The dataset and the source code of our model are available on GitHub $^7$ .
5.4 Results and Analysis
5.4.1 Evaluation Results
The experimental results are reported in Table 2. The results on both turn-level and session-level evaluations indicate that ScriptWriter dramatically outperforms all baselines, including DAM and DUA, which are two state-of-the-art models on multi-turn response selection. All improvements are statistically significant  $(p$ -value  $\leq 0.01)$ . DAM performs better than other baselines, which confirms the effectiveness of the self and cross attention mechanism used in this model. The DUA model also uses the attention mechanism. It outper
We also tested some basic models such as RNN, LSTM, and BiLSTM (Lowe et al., 2015) in our experiments. However, they cannot achieve comparable results to the selected baselines.
$^{6}$ https://www.tensorflow.org
<sup>7</sup>https://github.com/DaoD/ScriptWriter
Table 2: Evaluation results on two response selection tasks: turn-level and session-level. Our ScriptWriter model is represented as SW.  $\dagger$  and  $\star$  denote significant improvements with SW in t-test with  $p\leq 0.01$  and  $p\leq 0.05$  respectively.
Turn-level	Session-level
Method	R2@1	R10@1	R10@5	MRR	Pstrict	Pweak
MVLSTM	0.651†	0.217†	0.732†	0.395†	0.198†	0.224†
DL2R	0.643†	0.210†	0.638†	0.314†	0.230†	0.243†
SMN	0.641†	0.176†	0.696†	0.392†	0.197†	0.236†
DAM	0.631†	0.240†	0.733†	0.408†	0.226†	0.236†
DUA	0.654†	0.237†	0.736†	0.396†	0.223†	0.251†
SW	0.730	0.365	0.814	0.503	0.373	0.383
SWstatic	0.723	0.351	0.801	0.484†	0.338†	0.366
SW-PR	0.654†	0.246†	0.721†	0.398†	0.223†	0.239†
SW-CP	0.710*	0.326†	0.793†	0.473†	0.329†	0.352†
SW-CR	0.725	0.316†	0.766†	0.466†	0.335†	0.382
forms the other baselines that do not use attention. Both observations confirm the advantage of using attention mechanisms over pure RNN.
Between the two session-level measures, we observe that our model is less affected when moving from  $\mathrm{P_{weak}}$  to  $\mathrm{P_{strict}}$ . This shows that ScriptWriter can better select a response in the right position. We attribute this behavior to the utilization of narrative coverage.
5.4.2 Model Ablation
We conduct an ablation study to investigate the impact of different modules in ScriptWriter. First, we remove the updating mechanism by setting  $\gamma = 0$  (i.e., the representation of the narrative is not updated but static). This model is denoted as ScriptWriterstatic in Table 2. Then we remove narrative-response, context-narrative, and matching-response, respectively. These variants are denoted as ScriptWriter-PR, ScriptWriter-CP, and ScriptWriter-CR.
Model ablation results are shown in the second part of Table 2. We have the following findings: 1) ScriptWriter performs better than ScriptWriterstatic, demonstrating the effectiveness of updating mechanism for the narrative. The optimal value of  $\gamma$  is at around 0.647 after training, which means that only about  $35\%$  of information is kept when a line conveys it. 2) In both turn-level and session-level evaluations, the performance drops the most when we remove narrative-response matching. This indicates that the relevance of the response to the narrative is the most useful information in narrative-
Figure 4: The performance of ScriptWriter (SW) and DUA on the test set with different types of narrative in session-level evaluation.
guided script generation. 3) When we remove context-narrative matching, the performance drops too, indicating that context-narrative matching may provide implicit and complementary information for controlling the alignment of response and narrative. 4) In contrast, when we remove the context-response matching, the performance also drops, however, at a much smaller scale, especially on  $\mathrm{P}_{\mathrm{weak}}$ , than when narrative-response matching is removed. This contrast indicates that narrative is a more useful piece of information than context to determine what should be said next, thus it should be taken into account with an adequate mechanism.
5.4.3 Performance across Narrative Types
As we explained, narratives in our dataset are contributed by netizens, and they vary in style. Some narratives are detailed, while others are general. The question we analyze is how general vs. detailed narratives affect the performance of response selection. We use a simple method to evaluate roughly the degree of detail of a narrative: a narrative that has a high lexical overlap with the lines in the session is considered to be detailed. Narratives are put into six buckets depending on their level of detail, as shown in Figure 4.
We plot the performance of ScriptWriter and DUA in session-level evaluation over different types of narratives. The first type "0" means no word overlap between narrative and dialogue sessions. This is the most challenging case, representing extremely general narratives. It is not surprising to see that both ScriptWriter and DUA performs poorly on this type compared with other types in terms of  $\mathrm{P}_{\text{strict}}$ . The performance tends to become better when the overlap ratio is increased. This
8654
is consistent with our intuition: when a narrative is more detailed and better aligned with the session in wording, it is easier to choose the best responses. This plot also shows that our ScriptWriter can achieve better performance than DUA on all types of narratives, which further demonstrates the effectiveness of using narrative to guide the dialogue.
We also observe that the buckets "[0, 0.2)" and "[0.2, 0.4)" contain the largest proportions of narratives. This indicates that most netizens do not use the original lines to retell a story. The problem we address in this paper is thus non-trivial.
Although story generation has been extensively studied in the literature, no existing work addressed the problem of generating movie scripts following a given storyline or narrative. In this paper, we addressed this problem in the context of generating dialogues in a movie script. We proposed a model that uses the narrative to guide the dialogue generation/retrieval. We keep track of what in the narrative has already been expressed and what is remaining to select the next line through an updating mechanism. The final selection of the next response is based on multiple matching criteria between context, narrative and response. We constructed a new large-scale data collection for narrative-guided script generation from movie scripts. This is the first public dataset available for testing narrative-guided dialogue generation/selection. Experimental results on the dataset showed that our proposed approach based on narrative significantly outperforms the baselines that use a narrative as an additional context, and showed the importance of using the narrative in a proper manner. As a first investigation on the problem, our study has several limitations. For example, we have not considered the order in the narrative description, which could be helpful in generating dialogues in correct order. Other methods to track the dialogue state and the coverage of narrative can also be designed. Further investigations are thus required to fully understand how narratives can be effectively used in dialogue generation.
Ruihua Song and Zhicheng Dou are the corresponding authors. This work was supported by National Natural Science Foundation of China
No. 61872370 and No. 61832017, and Beijing Outstanding Young Scientist Program NO. BJWZYJH012019100020098.
David Bamman, Snigdha Chaturvedi, Elizabeth Clark, Madalina Fiterau, and Mohit Iyyer, editors. 2019. Proceedings of the First Workshop on Narrative Understanding. Association for Computational Linguistics, Minneapolis, Minnesota.
Selmer Bringsjord and David Ferrucci. 1999. Artificial intelligence and literary creativity: Inside the mind of brutus, a storytelling machine. Psychology Press.
Marc Cavazza, Fred Charles, and Steven J. Mead. 2002. Planning characters' behaviour in interactive storytelling. Journal of Visualization and Computer Animation, 13(2):121-131.
Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. 2019. Towards coherent and cohesive long-form text generation. In Proceedings of the First Workshop on Narrative Understanding, pages 1-11, Minneapolis, Minnesota. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650-2660, Florence, Italy. Association for Computational Linguistics.
Pablo Gervás, Belén Díaz-Agudo, Federico Peinado, and Raquel Hervás. 2005. Story plot generation based on CBR. Knowl. Based Syst., 18(4-5):235-242.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5110-5117.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of
8655
Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6473-6480.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Esther Levin, Shrikanth S. Narayanan, Roberto Pieraccini, Konstantin Biatov, Enrico Bocchieri, Giuseppe Di Fabbrizio, Wieland Eckert, Sungbok Lee, A. Pokrovsky, Mazin G. Rahim, P. Ruscitti, and Marilyn A. Walker. 2000. The at&t-darpa communicator mixed-initiative spoken dialog system. In Sixth International Conference on Spoken Language Processing, ICSLP 2000 / INTERSPEECH 2000, Beijing, China, October 16-20, 2000, pages 122-125.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294, Prague, Czech Republic. Association for Computational Linguistics.
James R. Meehan. 1977. Tale-spin, an interactive program that writes stories. In Proceedings of the 5th International Joint Conference on Artificial Intelligence. Cambridge, MA, USA, August 22-25, 1977, pages 91-98.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49, New Orleans, Louisiana. Association for Computational Linguistics.
Rafael Pérez y Pérez and Mike Sharples. 2001. MEX-ICA: A computer model of a cognitive account of creative writing. J. Exp. Theor. Artif. Intell., 13(2):119-139.
William J Rapaport, Erwin M Segal, Stuart C Shapiro, David A Zubin, Gail A Bruder, Judith Felson Duchan, and David M Mark. 1989. Cognitive and computer systems for understanding narrative text.
Mark O. Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. J. Artif. Intell. Res., 39:217-268.
Melissa Roemmele. 2019. Identifying sensible lexical relations in generated stories. In Proceedings of the First Workshop on Narrative Understanding, pages 44-52, Minneapolis, Minnesota. Association for Computational Linguistics.
Stephanie Seneff, Edward Hurley, Raymond Lau, Christine Pao, Philipp Schmid, and Victor Zue. 1998. GALAXY-II: a reference architecture for conversational system development. In The 5th International Conference on Spoken Language Processing, Incorporating The 7th Australian International Speech Science and Technology Conference, Sydney Convention Centre, Sydney, Australia, 30th November - 4th December 1998.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586, Beijing, China. Association for Computational Linguistics.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624-5634, Florence, Italy. Association for Computational Linguistics.
Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. 2017. How to make context more useful? an empirical study on context-aware neural conversational models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231-236, Vancouver, Canada. Association for Computational Linguistics.
G. Tur and R. D. Mori. 2011. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons.
Scott R Turner. 1994. Minstrel: A computer model of creativity and storytelling.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
8656
you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. CoRR, abs/1506.05869.
Ellen M. Voorhees. 1999. The TREC-8 question answering track report. In Proceedings of The Eighth Text Retrieval Conference, TREC 1999, Gaithersburg, Maryland, USA, November 17-19, 1999.
Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016. A deep architecture for semantic matching with multiple positional sentence representations. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2835-2841.
Yeyi Wang, Li Deng, and Alex Acero. 2011. Semantic frame-based spoken language understanding. Spoken language understanding: systems for extracting semantic information from speech, pages 41-91.
Jason D. Williams and Steve J. Young. 2007. Partially observable markov decision processes for spoken dialog systems. Comput. Speech Lang., 21(2):393-422.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794-3804, Florence, Italy. Association for Computational Linguistics.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505, Vancouver, Canada. Association for Computational Linguistics.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 3351-3357.
Chen Xing, Yu Wu, Wei Wu, Yalou Huang, and Ming Zhou. 2018. Hierarchical recurrent attention network for response generation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5610-5617.