SlowGuess/ACL_2020_OCR huggingface dataset:
Yutao Zhu $^{1}$ , Ruihua Song $^{2,*}$ , Zhicheng Dou $^{3,*}$ , Jian-Yun Nie $^{1}$ , Jin Zhou $^{4}$
<sup>1</sup>Université de Montréal, Montréal, Québec, Canada
2Microsoft XiaoIce, Beijing, China
$^{3}$ Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
<sup>4</sup>Beijing Film Academy, Beijing, China
yutao.zhu@umontreal.ca, rsong@microsoft.com
dou@ruc.edu.cn, nie@iro.umontreal.ca, whitezj@vip.sina.com
It is appealing to have a system that generates a story or scripts automatically from a storyline, even though this is still out of our reach. In dialogue systems, it would also be useful to drive dialogues by a dialogue plan. In this paper, we address a key problem involved in these applications - guiding a dialogue by a narrative. The proposed model ScriptWriter selects the best response among the candidates that fit the context as well as the given narrative. It keeps track of what in the narrative has been said and what is to be said. A narrative plays a different role than the context (i.e., previous utterances), which is generally used in current dialogue systems. Due to the unavailability of data for this new application, we construct a new large-scale data collection GraphMovie from a movie website where end-users can upload their narratives freely when watching a movie. Experimental results on the dataset show that our proposed approach based on narratives significantly outperforms the baselines that simply use the narrative as a kind of context.
Narrative is generally understood as a way to tell a story. WordNet defines it as “a message that tells the particulars of an act or occurrence or course of events; presented in writing or drama or cinema or as a radio or television program”<sup>1</sup>. Narrative plays an important role in many natural language processing (NLP) tasks. For example, in storytelling, the storyline is a type of narrative, which helps generate coherent and consistent stories (Fan et al., 2018, 2019). In dialogue generation, narrative can be used to define a global plan for the whole conversation session, so as to avoid generating inconsistent
Narrative	Jenny doesn't like to go home. To accompany Jenny, Gump decides to go home later. Gump is Jenny's best friend.珍妮不喜欢回家。为了陪珍妮，甘决定晚点回家。甘是珍妮最好的朋友。
Initial line	Mama's going to worry about me.妈会担心我的
1stline	Just stay a little longer. 再坐一会！
Yeah, and I'll bet you $ 10,000 he laughs his ass off.我打赌他会笑破肚皮
2ndline	Ok, Jenny, I'll stay. 好，珍妮，我留下来
She lived in an old house.她家的房子破旧
3rdline	He was a very loving man.他是个非常有爱心的人
You are my most special friend.你是我最特别的朋友
Figure 1: An example of part of a script with a narrative extracted from our GraphMovie dataset. The checked lines are from a ground-truth session, while the unchecked responses are other candidates that are relevant but not coherent with the narrative.
and scattered responses (Xing et al., 2018; Tian et al., 2017; Ghazvininejad et al., 2018).
In this work, we investigate the utilization of narratives in a special case of text generation - movie script generation. This special form of conversation generation is chosen due to the unavailability of the data for a more general form of application. Yet it does require the same care to leverage narratives in general conversation, and hence can provide useful insight to a more general form of narrative-guided conversation. The dataset we use to support our study is collected from GraphMovie $^2$ , where an end-user retells the story of a movie by uploading descriptive paragraphs in his/her own words. More details about the dataset will be presented in Section 3.2. An example is shown in Figure 1, where the narrative
1http://wordnetweb.princeton.edu/perl/webwn?s=narrative
$^{2}$ http://www.graphmovies.com/home/2/index.php. Unfortunately, we find this website was closed recently.
8647
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8647-8657
is uploaded to retell several lines of a script in a movie. Our task is to generate/select the following lines by leveraging the narrative.
Our problem is closely related to dialogue generation that takes into account the context (Wu et al., 2017; Zhang et al., 2018; Zhou et al., 2018b). However, a narrative plays a different and more specific role than a general context. In particular, a narrative may cover the whole story (a part of a script), thus a good conversation should also cover all the aspects mentioned in a narrative, which is not required with a general context. In this paper, we propose a new model called ScriptWriter to address the problem of script generation/selection with the help of a narrative. ScriptWriter keeps track of what in the narrative has been said and what is remaining to select the next line by an updating mechanism. The matching between updated narrative, context, and response are then computed respectively and finally aggregated as a matching score. As it is difficult to evaluate the quality of script generation, we frame our work in a more restricted case - selecting the right response among a set of candidates. This form of more limited conversation generation - retrieval-based conversation - has been widely used in the previous studies (Wu et al., 2017; Zhou et al., 2018b), and it provides an easier way to evaluate the impact of narratives.
We conduct experiments on a dataset we collected and made publicly available (see Section 5). The experiments will show that using a narrative to guide the generation/selection of script is a much more appropriate approach than using it as part of the general context.
Our work has three main contributions:
(1) To our best knowledge, this is the first investigation on movie script generation with a narrative. This task could be further extended to a more general text generation scenario when suitable data are available.
(2) We construct the first large-scale data collection GraphMovie to support research on narrative-guided movie script generation, which is made publicly accessible.
(3) We propose a new model in which a narrative plays a specific role in guiding script generation. This will be shown to be more appropriate than a general context-based approach.
2.1 Narrative Understanding
It has been more than thirty years since researchers proposed "narrative comprehension" as an important ability of artificial intelligence (Rapaport et al., 1989). The ultimate goal is the development of a computational theory to model how humans understand narrative texts. Early explorations used symbolic methods to represent the narrative (Turner, 1994; Bringsjord and Ferrucci, 1999) or rule-based approaches to generate the narrative (Riedl and Young, 2010). Recently, deep neural networks have been used to tackle the problem (Bamman et al., 2019), and related problems such as generating coherent and cohesive text (Cho et al., 2019) and identifying relations in generated stories (Roemmle, 2019) have also been addressed. However, these studies only focused on how to understand a narrative itself (e.g., how to extract information from a narrative). They did not investigate how to utilize the narrative in an application task such as dialogue generation.
2.2 Dialogue Systems
Existing methods of open-domain dialogue can be categorized into two groups: retrieval-based and generation-based. Recent work on response generation is mainly based on sequence-to-sequence structure with attention mechanism (Shang et al., 2015; Vinyals and Le, 2015), with multiple extensions (Li et al., 2016; Xing et al., 2017; Zhou et al., 2018a, 2020; Zhu et al., 2020). Retrieval-based methods try to find the most reasonable response from a large repository of conversational data, instead of generating a new one (Wu et al., 2017; Zhou et al., 2018b; Zhang et al., 2018). In general, the utterances in the previous turns are taken together as the context for selecting the next response. Retrieval-based methods are widely used in real conversation products due to their more fluent and diverse responses and better efficiency. In this paper, we focus on extending retrieval-based methods by using a narrative as a plan for a session. This is a new problem that has not been studied before.
Contrary to open-domain chatbots, task-oriented systems are designed to accomplish tasks in a specific domain (Seneff et al., 1998; Levin et al., 2000; Wang et al., 2011; Tur and Mori, 2011). In these systems, a dialogue state tracking component is designed for tracking what has happened in a dia
8648
Table 1: Statistics of GraphMovie corpus.
Training	Validation	Test
# Sessions	14,498	805	806
# Micro-sessions	136,524	37,480	38,320
# Candidates	2	10	10
Min. #turns	2	2	2
Max. #turns	34	27	17
Avg. #turns	4.71	4.66	4.75
Avg. #words in Narr.	25.04	24.86	24.18
logue (Williams and Young, 2007; Henderson et al., 2014; Xu and Rudnicky, 2000). This inspires us to track the remaining information in the narrative that has not been expressed by previous lines of conversation. However, existing methods cannot be applied to our task directly as they are usually predefined for specific tasks, and the state tracking is often framed as a classification problem.
2.3 Story Generation
Existing studies have also tried to generate a story. Early work relied on symbolic planning (Meehan, 1977; Cavazza et al., 2002) and case-based reasoning (y Pérez and Sharples, 2001; Gervás et al., 2005), while more recent work uses deep learning methods. Some of them focused on story ending generation (Peng et al., 2018; Guan et al., 2019), where the story context is given, and the model is asked to select a coherent and consistent story ending. This is similar to the dialogue generation problem mentioned above. Besides, attempts have been made to generate a whole story from scratch (Fan et al., 2018, 2019). Compared with the former task, this latter is more challenging since the story framework and storyline should all be controlled by the model.
Some recent studies also tried to guide the generation of dialogues (Wu et al., 2019; Tang et al., 2019) or stories (Yao et al., 2019) with keywords - the next response is asked to include the keywords. This is a step towards guided response generation and bears some similarities with our study. However, a narrative is more general than keywords, and it provides a description of the dialogue session rather than imposing keywords to the next response.
3 Problem Formulation and Dataset
Suppose that we have a dataset  $\mathcal{D}$ , in which a sample is represented as  $(y, c, p, r)$ , where  $c =$
$\{s_1,\dots ,s_n\}$  represents a context formed by the preceding sentences/lines  $\{s_i\}_{i = 1}^n$  .  $p$  is a predefined narrative that governs the whole script session, and  $r$  is a next line candidate (we refer to it as a response);  $y\in \{0,1\}$  is a binary label, indicating whether  $r$  is a proper response for the given  $c$  and  $p$  Intuitively, a proper response should be relevant to the context, and be coherent and aligned with the narrative. Our goal is to learn a model  $g(c,p,r)$  with  $\mathcal{D}$  to determine how suitable a response  $r$  is to the given context  $c$  and narrative  $p$
3.2 Data Collection and Construction
Data is a critical issue in research on story/dialogue generation. Unfortunately, no dataset has been created for narrative-guided story/dialogue generation. To fill the gap, we constructed a test collection from GraphMovie, where an editor or a user can retell the story of a movie by uploading descriptive paragraphs in his/her own words to describe screenshots selected from the movie. A movie on this website has, on average, 367 descriptions. A description paragraph often contains one to three sentences to summarize a fragment of a movie. It can be at different levels - from retelling the same conversations to a high-level description. We consider these descriptions as narratives for a sequence of dialogues, which we call a session in this paper. Each dialogue in a session is called a line of script (or simply a line).
To construct the dataset, we use the top 100 movies in  $\mathrm{IMDB}^3$  as an initial list. For each movie, we collect its description paragraphs from GraphMovie. Then we hire annotators to watch the movie and annotate the start time and end time of the dialogues corresponding to each description paragraph through an annotation tool specifically developed for this purpose. According to the start and end time, the sequence of lines is extracted from the subtitle file and aligned with a corresponding description paragraph.
As viewers of a movie can upload descriptions freely, not all description paragraphs correspond to a narrative and are suitable for our task. For example, some uploaded paragraphs express one's subjective opinions about the movie, the actors, or simply copy the script. Therefore, we manually review the data and remove such non-narrative data. We also remove sessions that have less than two lines. Finally, we obtain 16,109 script sessions,
8649
each of which contains a description paragraph (narrative) and corresponding lines of the script. As shown in Table 1, on average, a narrative has about 25 words, and a session has 4.7 lines. The maximum number of lines in a session is 34.
Our task is to select one response from a set of candidates at any point during the session. By moving the prediction point through the session, we obtain a set of micro-sessions, each of which has a sequence of previous lines as context at that point of time, the same narrative as the session, and the next line to predict. The candidates to be selected contain one ground-truth line - the one that is genuinely the next line, together with one (in the training set) or nine (in the validation/test set) other candidates retrieved with the previous lines by Solr<sup>4</sup>. The above preparation of the dataset follows the practice in the literature (Wu et al., 2017) for retrieval-based dialogue.
4 Proposed Method: ScriptWriter
A good response is required to be coherent with the previous lines, i.e., context, and be consistent with the given narrative. For example, "Just stay a little longer" can respond "Mama's going to worry about me" and it has no conflict with the narrative in Figure 1. Furthermore, as our target is to generate all lines in the session successively, it is also required that the following lines should convey the information that the former lines have not conveyed. Otherwise, only a part of the narrative is covered, and we will miss some other aspects specified in the narrative.
We propose an attention-based model called ScriptWriter to solve the problem. ScriptWriter follows a representation-matching-aggregation framework. First, the narrative, the context, and the response candidate are represented in multiple granularities by multi-level attentive blocks. Second, we propose an updating mechanism to keep track of what in a narrative has been expressed and explicitly lower their weights in the updated narrative so that more emphasis can be put on the remaining parts. Third, matching features are extracted between different elements: between context and response to capture whether it is a proper reply; between narrative and response to capture whether it is consistent with the narrative; and between context and narrative to implicitly track what in the
narrative has been expressed in the previous lines. Finally, the above matching features are concatenated together and a final matching score is produced by convolutional neural networks (CNNs) and a multi-layer perceptron (MLP).
4.2 Representation
To better handle the gap in words between two word sequences, we propose to use an attentive block, which is similar to that used in Transformer (Vaswani et al., 2017). The input of an attentive block consists of three sequences, namely query  $(\mathbf{Q})$ , key  $(\mathbf{K})$ , and value  $(\mathbf{V})$ . The output is a new representation of the query and is denoted as AttentiveBlock  $(\mathbf{Q}, \mathbf{K}, \mathbf{V})$  in the remaining parts. This structure is used to represent a response, lines in the context, and a narrative.
More specifically, given a narrative  $p = (w_1^p,\dots ,w_{n_p}^p)$ , a line  $s_i = (w_1^{s_i},\dots ,w_{n_{s_i}}^{s_i})$  and a response candidate  $r = (w_1^r,\dots ,w_{n_r}^r)$ , ScriptWriter first uses a pre-trained embedding table to map each word  $w$  to a  $d_e$ -dimension embedding  $\mathbf{e}$ , i.e.,  $w\Rightarrow \mathbf{e}$ . Thus the narrative  $p$ , the line  $s_i$  and the response candidate  $r$  are represented by matrices  $\mathbf{P}^0 = (\mathbf{e}_1^p,\dots ,\mathbf{e}_{n_p}^p)$ ,  $\mathbf{S}_i^0 = (\mathbf{e}_1^{s_i},\dots ,\mathbf{e}_{n_{s_i}}^{s_i})$  and  $\mathbf{R}^0 = (\mathbf{e}_1^r,\dots ,\mathbf{e}_{n_r}^r)$ .
Then ScriptWriter takes  $\mathbf{P}^0$ ,  $\{\mathbf{S}_i^0\}_{i=1}^n$  and  $\mathbf{R}^0$  as inputs and uses stacked attentive blocks to construct multi-level self-attention representations. The output of the  $(l-1)^{th}$  level of attentive block is input into the  $l^{th}$  level. The representations of  $p$ ,  $s_i$ , and  $r$  at the  $l^{th}$  level are defined as follows:
$$
\mathbf {P} ^ {l} = \text {A t t e n t i v e B l o c k} \left(\mathbf {P} ^ {l - 1}, \mathbf {P} ^ {l - 1}, \mathbf {P} ^ {l - 1}\right), \tag {1}
$$
$$
\mathbf {S} _ {i} ^ {l} = \text {A t t e n t i v e B l o c k} \left(\mathbf {S} _ {i} ^ {l - 1}, \mathbf {S} _ {i} ^ {l - 1}, \mathbf {S} _ {i} ^ {l - 1}\right), \tag {2}
$$
$$
\mathbf {R} ^ {l} = \text {A t t e n t i v e B l o c k} \left(\mathbf {R} ^ {l - 1}, \mathbf {R} ^ {l - 1}, \mathbf {R} ^ {l - 1}\right), \tag {3}
$$
where  $l$  ranges from 1 to  $L$ .
Inspired by a previous study (Zhou et al., 2018b), we apply another group of attentive blocks, which is referred to as cross-attention, to capture semantic dependency between  $p$ ,  $s_i$  and  $r$ . Considering  $p$  and  $s_i$  at first, their cross-attention representations are defined by:
$$
\overline {{\mathbf {P}}} _ {s _ {i}} ^ {l} = \text {A t t e n t i v e B l o c k} \left(\mathbf {P} ^ {l - 1}, \mathbf {S} _ {i} ^ {l - 1}, \mathbf {S} _ {i} ^ {l - 1}\right), \tag {4}
$$
$$
\overline {{\mathbf {S}}} _ {i, p} ^ {l} = \text {A t t e n t i v e B l o c k} \left(\mathbf {S} _ {i} ^ {l - 1}, \mathbf {P} ^ {l - 1}, \mathbf {P} ^ {l - 1}\right). \tag {5}
$$
Here, the words in the narrative can attend to all words in the line, and vice versa. In this way, some inter-dependent segment pairs, such as "stay" in the
4https://lucene.apache.org/solr/
Figure 2: Updating mechanism in ScriptWriter. The representation of the narrative is updated by lines in the context one by one. The information that has been expressed is decayed. Thus the updated narrative focuses more on the remaining information.
line and "go home later" in the narrative, become close to each other in the representations. Similarly, we compute cross-attention representations between  $p$  and  $r$  and between  $r$  and  $s_i$  at different levels, which are denoted as  $\overline{\mathbf{P}}_r^l$ ,  $\overline{\mathbf{R}}_p^l$ ,  $\overline{\mathbf{S}}_{i,r}^l$  and  $\overline{\mathbf{R}}_{s_i}^l$ . These representations further provide matching information across different elements in the next step.
4.3 Updating Mechanism
We design an updating mechanism to keep track of the coverage of the narrative by the lines so that the selection of the response will focus on the uncovered parts. The mechanism is illustrated in Figure 2. We update a narrative gradually by all lines in the context one by one. For the  $i^{th}$  line  $s_i$ , we conduct a matching between  $\mathbf{S}_i$  and  $\mathbf{P}$  by their cosine similarity at all levels  $(l)$  of attentive blocks:
$$
\mathbf {T} _ {s i, p} ^ {l} [ j ] [ k ] = \cos (\mathbf {S} _ {i} ^ {l} [ j ], \mathbf {P} ^ {l} [ k ]), \tag {6}
$$
where  $j$  and  $k$  stand for the  $j^{th}$  word in  $s_i$  and  $k^{th}$  word in  $p$  respectively. To summarize how much information in  $p$  has been expressed by  $s_i$ , we compute a vector  $\mathbf{D}_i$  by conducting summations along vertical axis on each level in the matching
Figure 3: The context-narrative matching. All lines and the narrative are represented by attentive blocks and the matching between them results in a matching cube  $\mathbf{Q}_{cp}$ . Matching features are aggregated and distilled by a CNN.
map  $\mathbf{T}_{s_i,p}$ . The summation on the  $l^{th}$  level is:
$$
\mathbf {D} _ {i} ^ {l} = \left[ d _ {i, 1} ^ {l}, d _ {i, 2} ^ {l}, \dots , d _ {i, n _ {p}} ^ {l} \right], \tag {7}
$$
$$
d _ {i, k} ^ {l} = \gamma \sum_ {j = 1} ^ {n _ {s _ {i}}} \mathbf {T} _ {s _ {i}, p} ^ {l} [ j ] [ k ], \tag {8}
$$
where  $n_p, n_{s_i}$  denotes the number of words in  $p$  and  $s_i$ ;  $\gamma \in [0,1]$  is a parameter to learn and works as a gate to control the decaying degree of the mentioned information. Finally, we update the narrative's representation as follows for the  $i^{th}$  line  $s_i$  in the context:
$$
\mathbf {P} _ {i + 1} ^ {l} = \left(1 - \mathbf {D} _ {i} ^ {l}\right) \mathbf {P} _ {i} ^ {l}. \tag {9}
$$
The initial representation  $\mathbf{P}_0^l$  is equal to  $\mathbf{P}^l$  defined in Equation (1). If there are  $n$  lines in the context, this update is executed  $n$  times, and  $(1 - \mathbf{D}^l)$  will produce a continuous decaying effect.
4.4 Matching
The matching between the narrative  $p$  and the line  $s_i$  is conducted based on both their self-attention and cross-attention representations, as shown in Figure 3.
First, ScriptWriter computes the dot product on these two representations separately as follows:
$$
\mathbf {m} _ {s _ {i}, p, l} ^ {\text {s e l f}} [ j, k ] = \mathbf {S} _ {i} ^ {l} [ j ] ^ {T} \cdot \mathbf {P} ^ {l} [ k ], \tag {10}
$$
$$
\mathbf {m} _ {s _ {i}, p, l} ^ {\text {c r o s s}} [ j, k ] = \overline {{\mathbf {S}}} _ {i, p} ^ {l} [ j ] ^ {T} \cdot \overline {{\mathbf {P}}} _ {s _ {i}} ^ {l} [ k ], \tag {11}
$$
8651
where  $l$  ranges from 0 to L. Each element is the dot product of the  $j^{th}$  word representation in  $\mathbf{S}_i^l$  or  $\overline{\mathbf{S}}_{i,p}^{l}$  and the  $k^{th}$  word representation in  $\mathbf{P}^l$  or  $\overline{\mathbf{P}}_{s_i}^l$ . Then the matching maps in different layers are concatenated together as follows:
$$
\mathbf {m} _ {s _ {i}, p} ^ {\text {s e l f}} [ j, k ] = \left[ \mathbf {m} _ {s _ {i}, p, 0} ^ {\text {s e l f}} [ j, k ]; \dots ; \mathbf {m} _ {s _ {i}, p, L} ^ {\text {s e l f}} [ j, k ] \right],
$$
$$
\mathbf {m} _ {s _ {i}, p} ^ {c r o s s} [ j, k ] = \left[ \mathbf {m} _ {s _ {i}, p, 0} ^ {c r o s s} [ j, k ]; \dots ; \mathbf {m} _ {s _ {i}, p, L} ^ {c r o s s} [ j, k ] \right],
$$
where  $[\cdot ]$  is concatenation operation. Finally, the matching features computed by the self-attention representation and the cross-attention representation are fused as follows:
$$
\mathbf {M} _ {s _ {i}, p} \left[ j, k \right] = \left[ \mathbf {m} _ {s _ {i}, p} ^ {s e l f} \left[ j, k \right]; \mathbf {m} _ {s _ {i}, p} ^ {c r o s s} \left[ j, k \right] \right].
$$
The matching matrices  $\mathbf{M}_{p,r}$  and  $\mathbf{M}_{s_i,r}$  for narrative-response and context-response are constructed in a similar way. For the sake of brevity, we omit the formulas. After concatenation, each cell in  $\mathbf{M}_{s_i,p}$ ,  $\mathbf{M}_{p,r}$  or  $\mathbf{M}_{s_i,r}$  has  $2(L + 1)$  channels and contains matching information at different levels.
The matching between narrative, context, and response serves for different purposes. Context-response matching  $(\mathbf{M}_{s_i,r})$  serves to select a response suitable for the context. Context-narrative matching  $(\mathbf{M}_{s_i,p})$  helps the model "remember" how much information has been expressed and implicitly influences the selection of the next responses. Narrative-response matching  $(\mathbf{M}_{p,r})$  helps the model to select a more consistent response with the narrative. As the narrative keeps being updated along with the lines in context, ScriptWriter tends to dynamically choose the response that matches what remains unexpressed in the narrative.
4.5 Aggregation