SlowGuess/NAACL_2021_OCR huggingface dataset:
Bo-Hsiang Tseng $^{1*}$ , Shruti Bhargava $^{2}$ , Jiarui Lu $^{2}$   
Joel Ruben Antony Moniz $^{2}$ , Dhivya Piraviperumal $^{2}$ , Lin Li $^{2}$ , Hong Yu $^{2}$ $^{1}$ Engineering Department, University of Cambridge, UK  
 $^{2}$ Apple  
bht26@cam.ac.uk  
{shruti_bhargava, jiarui LU, joelrubenantony_moniz, dhivyaprp, lli9, hong_yu}@apple.com
Anaphora and ellipses are two common phenomena in dialogues. Without resolving referring expressions and information omission, dialogue systems may fail to generate consistent and coherent responses. Traditionally, anaphora is resolved by coreference resolution and ellipses by query rewrite. In this work, we propose a novel joint learning framework of modeling coreference resolution and query rewriting for complex, multi-turn dialogue understanding. Given an ongoing dialogue between a user and a dialogue assistant, for the user query, our joint learning model first predicts coreference links between the query and the dialogue context, and then generates a self-contained rewritten user query. To evaluate our model, we annotate a dialogue based coreference resolution dataset, MuDoCo, with rewritten queries. Results show that the performance of query rewrite can be substantially boosted  $(+2.3\%)$  F1) with the aid of coreference modeling. Furthermore, our joint model outperforms the state-of-the-art coreference resolution model  $(+2\%)$  F1) on this dataset.
In recent years, dialogue systems have attracted growing interest, and been applied to various scenarios, ranging from chatbots to task-oriented dialogues to question answering. Despite rapid progress in dialogue systems, several difficulties remain in the understanding of complex, multi-turn dialogues. Two major problems are anaphora resolution (Clark and Manning, 2016a,b) and ellipsis (Kumar and Joshi, 2016) in follow-up turns. Take the dialogue in Figure 1 as an example: ellipsis happens in user turn 2 where the user is asking for the capital of "Costa Rica" without explicitly mentioning the country again; coreference happens in user turn 3 where "the capital" refers to
(Turn 1)User: Who is the president of Costa Rica?System: Carlos Alvarado Quesada is the president of Costa Rica.
(Turn 2)User: And what is the capital?[Rewrite: And what is the capital of Costa Rica?]System: San Jose is the capital of Costa Rica.
(Turn 3)User: What is the population of the capital?[Rewrite: What is the population of San Jose?]System (i): The population of San Jose is 1.03 million. (California)System (ii): The population of San Jose is 342 thousands. (Costa Rica)
Figure 1: An example of a question-answering dialogue where coreference and ellipsis happen in user query, and the corresponding query rewrite annotation. References to the same entity are highlighted in the same color, and can be resolved by coreference resolution modeling. The two system responses in Turn 3 indicate two possible interpretations of the city San Jose by the system.
"San Jose". Without resolving the anaphoric reference and the ellipsis, dialogue systems may fail to generate coherent responses.
Query rewrite (Quan et al., 2019) is an approach that converts a context-dependent user query into a self-contained utterance so that it can be understood and executed independent of previous dialogue context. This technique can solve many cases where coreference or ellipsis happens. For instance, "the capital" in user turn 3 is changed to "San Jose" in the rewrite. Furthermore, the ellipsis of the country name "Costa Rica" in user turn 2 can be revealed through rewriting. The rewritten utterance improves multi-turn dialogue understanding (Yang et al., 2019) by reducing dependency on the previous turns.
Although query rewrite implicitly resolves coreference resolution, there is information not contained in a rewrite. First, it does not provide a distinct coreference link between mentions across dialogue turns as in the classic coreference resolution task. This is particularly disadvantageous
*Work done while the first author was an intern at Apple.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3390-3406 June 6-11, 2021. ©2021 Association for Computational Linguistics
when there is entity ambiguity in the rewritten sentence. For example, in Figure 1, since "San Jose" in Rewrite turn 3 can be either San Jose in Costa Rica or San Jose in California, it is likely that the system ends up with an incorrect response by generating System (i) instead of System (ii) due to the wrong interpretation of San Jose. Second, mention detection, an essential step in coreference resolution (Peng et al., 2015), is not involved in query rewrite. By knowing which span in an utterance is a mention, downstream systems like named entity recognition and intent understanding can perform better (Bikel et al., 2009). Third, if coreference links to dialogue context are available, downstream systems can skip entity linking, which is time-consuming and may introduce noise.
To resolve the above issues, we propose a novel joint learning framework that incorporates the benefits of reference resolution into the query rewrite task. To the best of our knowledge, there does not exist, at the time of writing, an English conversational dataset that couples annotations of both query rewrite and coreference resolution (as links or clusters). This motivates us to collect annotations for query rewrite on a recent dialogue dataset - MuDoCo (Martin et al., 2020), which already has coreference links between user query and dialogue context. Compared to existing query rewrite datasets (Quan et al., 2019; Anantha et al., 2020), rewriting in MuDoCo is much more challenging since it involves reasoning over multiple turns and spans multiple domains.
We design a joint learning model adopting the GPT-2 (Radford et al., 2019) architecture that learns both query rewrite and coreference resolution. Given an ongoing dialogue, our model first predicts the coreference links, if any, between the latest user query and the dialogue context. Then it generates the rewritten query by drawing upon the coreference results. Our experiments show that query rewrite performance can be substantially boosted with the aid of coreference training. In addition, our model outperforms strong baselines for the two individual tasks. Since both the tasks fundamentally solve reference resolution, the joint training facilitates knowledge sharing.
- We present a novel joint learning framework of modeling coreference resolution and query rewrite for multi-turn dialogues.
- Our annotations of query rewrite augment the
MuDoCo dataset with query rewrite labels. To the best of our knowledge, our augmented MuDoCo is the first English dialogue dataset with both coreference resolution and query rewrite annotations.
- We propose a novel GPT-2 based model to tackle the two target tasks, and show that joint training with coreference resolution helps in improving the quality of the query rewrites.
The augmented dataset with our annotations along with the modeling source code are available at https://github.com/apple/ml-cread.
Query Rewrite The most relevant line of research is the adoption of query rewrite in dialogues to tackle anaphora or ellipses. Many prior works employ an LSTM-based seq-to-seq model, which takes the dialogue context and user query as input, and generates the rewritten query. Quan et al. (2019) use the pointer-generator model (See et al., 2017) to rewrite the user query on restaurant-domain task-oriented dialogues. By comparison, query rewrite in MuDoCo dataset is more challenging as it covers 6 domains and the rewriting patterns are more complex and diverse than in the CamRest676 dataset (Wen et al., 2017). Rastogi et al. (2019) introduce an auxiliary objective of copying entity tokens from the delexicalized utterances to augment the learning of pointer network. In Su et al. (2019), two separate attention distributions are learned for the dialogue context and the user query respectively with a control gate. This modified copy mechanism shows improvements over the standard pointer-generator on both LSTM-based models and transformer-based models (Vaswani et al., 2017). Note that in the dataset used in their work, the dialogue context has only 2 utterances; MuDoCo, in contrast, has up to 8 utterances, making it much more challenging for query rewrite.
Coreference Resolution Research on document-based coreference resolution has a long history (a detailed survey can be found in Ng (2010)). Various approaches have been proposed, ranging from learning mention-pair classifiers (Ng and Cardie, 2002; Bengtson and Roth, 2008), latent structured-based models (Fernandes et al., 2012; Björkelund and Kuhn, 2014; Martschat and Strube, 2015) to the more recent neural pipeline based systems
that rely on syntactic parsers (Raghunathan et al., 2010) and clustering algorithms (Clark and Manning, 2016a,b). The first neural end-to-end coreference resolution model was proposed in Lee et al. (2017) and achieved better results without external resources. An improved version was proposed in Lee et al. (2018), which considers higher-order structures by iteratively refining span representations. Recently, powerful pre-trained models have been used to extract representations for these end-to-end models using BERT (Joshi et al., 2019) or SpanBERT (Joshi et al., 2020). Wu et al. (2020) approach the problem in a question answering framework. For each detected mention candidate, the sentence it resides in serves as the query and is used to predict the referent in the passage. Different from these works, we focus on coreference resolution in dialogues with the following main distinctions: 1) the speaker information in dialogues is clear; 2) less descriptive content may cause the pronoun mention to be more ambiguous; and 3) coreference resolution is conducted only between the latest user query and the previous dialogue context — unlike in document-based coreference resolution where a model can look ahead for the resolution, future turns are not available to a dialogue agent. We encourage the reader to refer to Martin et al. (2020) for more details.
Joint Learning In contrast to prior works that focus solely on either query rewrite or coreference resolution, we present a novel joint learning approach to tackle both the tasks using one single model. We hope that this work serves as a first step towards this new, challenging and practical problem for dialogue understanding.
3 Dataset and Task
The MuDoCo dataset contains 7.5k task-oriented multi-turn dialogues across 6 domains. A dialogue has an average of 2.6 turns and a maximum of 5 (a turn includes a user query and a system response). Figure 2 shows an example. For each partial dialogue, the coreference links, if existing, are annotated between the latest user query and its dialogue context. For example, when we consider the partial dialogue preceding up to user turn 2, there is a coreference link between the anaphora "this" in user turn 2 and the antecedent "song" in user turn 1. When an anaphora has multiple antecedents in the context, e.g., "song" in user turn 1 and "Yellow Sub
(Turn 1)
User: Play the Beatles song about submarines.
System: Yellow Submarine by the Beatles playing on YouTube Music.
(Turn 2)
User: What album is this from?
[Rewrite: What album is Yellow Submarine by the Beatles from?]
System: Yellow Submarine is from the album Yellow Submarine.
(Turn 3)
User: Can I hear the next song on the album?
[Rewrite: Can I hear the next song on the Yellow Submarine album?]
System: Sure, Only A Northern Song on the album now playing.
Figure 2: An example from the MuDoCo dataset in the music domain with our query rewrite annotation. Word spans in the same color belong to the same mention cluster.
marine” in system turn 1, only one of them is annotated as its referent in the coreference link.
On top of the existing coreference labels, we annotate the rewrite for each utterance. The goal is to rewrite the query into a self-contained query independent of the dialogue context. 30 annotators are recruited for the data collection. Each of them is shown a partial dialogue, and is asked (1) to decide if the query needs to be rewritten due to coreference or ellipsis; and (2) to provide the rewritten query, when rewriting is required. We notice that there can be various ways of rewriting an utterance. For example, some annotators might include every detail of the rewritten entity, while others might choose a precise term; some might paraphrase the rewritten utterance, while others keep the same expression. To ensure data consistency and high annotation quality, we designed a comprehensive guideline for the annotators to follow and undertook a two-stage collection process: 1) we organized two training sessions with annotators. In each session, 50 representative examples were selected and assigned to each annotator. An author inspected these training results individually and provided feedback to the annotators. 2)  $5\%$  of the grading results were manually evaluated by an author for quality assurance. Detailed annotation guidelines can be found in the Appendix.
The joint learning task requires the machine to predict both coreference links and the rewritten query for the latest user query given an ongoing dialogue. The outputs of the two individual tasks complement each other and provide more comprehensive information for dialogue understanding. For instance, the "Yellow Submarine" in Figure 2 can be either a song name or an album name. Explicit coreference resolution helps to disambiguate
Figure 3: The proposed model for joint learning of coreference resolution and query rewrite, designed using the GPT-2 architecture. Given a dialogue context and a user query, the model first detects the mentions in the query (Step 1); resolves the corresponding reference spans (Step 2); predicts whether the query needs a rewrite or not (Step 3); and, if the model decides to rewrite, generates the rewritten query (Step 4). In this example dialogue, there is a coreference link existing between the mention "this" and its referent "Yellow Submarine".
between various possibilities by linking entities to previously resolved ones. More importantly, the supervision of coreference resolution can be beneficial to rewriting the anaphora to its antecedent.
Our proposed model for jointly learning coreference resolution and query rewrite is designed based on the GPT-2 architecture, presented in Figure 3. The input to the model is the concatenation of the dialogue context and the latest user query, where special tokens are used to separate utterances and indicate speaker information. Passing through the standard decoder layers, the hidden state  $h_t^l \in \mathbb{R}^d$  and attention score  $a_t^{l,j} \in \mathbb{R}^T$  at each position of the input sequence are calculated, where  $l$ ,  $j$  and  $t$  denote the index of the decoder layer, that of the attention head, and the input token position respectively;  $d$  and  $T$  denote the embedding size and the length of the input sequence respectively. Inspired by the end-to-end coreference resolution model (Lee et al., 2017), our model first predicts mentions in the user query and grounds them to their corresponding referent in the dialogue context using attention heads. The model then generates the rewritten query conditioned on the resolved coreference links. The prediction process has four main steps, described in detail below:
Step 1: Mention Detection First, the model detects any possible referring expressions in the user query. Here we use the term mention to include all those expressions that require reference resolution (e.g., pronouns or partial entity names). We formulate mention detection as a sequence labeling problem: each token in a query is labelled as one of three classes  $\{S,E,N\}$ , referring to Start of mention, End of mention and None respectively. This sequence tagger in the mention detector, parameterized by a feed-forward network, takes the hidden states of the query from the last decoder layer as input, and predicts the sequence of class labels. Then the mention spans in the query can be determined by a pair of mention start  $S$  and end  $E$  tags. For instance, in Figure 3 the label of position "this" is class  $S$  and that of "from" is class  $E$ , while the rest of the positions in the query are labelled as class  $N$ . We use  $m_S$  and  $m_E$  to respectively denote the start and end position index of a predicted mention  $m$ .
Step 2: Reference Resolution For each detected mention  $m$ , the model resolves it to the antecedent (or referent) in the dialogue context by predicting the span boundaries: the position index of the referent start  $r_S$  and end  $r_E$ . Essentially, the distributions of the boundaries  $(r_S$  and  $r_E$ ) are learned by supervising multiple attention heads associated with the target mention  $m$ . In other words, the at
tention distribution  $a_{m_S}$  (the attention score of each position associated with the mention start  $m_S$ ) is supervised to focus on the referent start  $r_S$ . Similarly, attention scores  $a_{m_E}$  associated with the mention end  $m_E$  are used to learn the boundary of referent end  $r_E$ . Concretely:
$$
q _ {r _ {S}} = \frac {1}{L ^ {\prime} J ^ {\prime}} \sum_ {l} ^ {L ^ {\prime}} \sum_ {j} ^ {J ^ {\prime}} a _ {m _ {S}} ^ {l, j},
$$
$$
q _ {r _ {E}} = \frac {1}{L ^ {\prime} J ^ {\prime}} \sum_ {l} ^ {L ^ {\prime}} \sum_ {j} ^ {J ^ {\prime}} a _ {m _ {E}} ^ {l, j},
$$
where  $q_{r_S}$  and  $q_{r_E}$  are the probability distributions that a given token represents the referent start  $r_S$  and end  $r_E$  respectively;  $L'$  and  $J'$  are the specified number of the involved decoder layers and attention heads. We then take the argmax of these boundary distributions to resolve the referent  $r$ . Our design of reference resolution effectively leverages the powerful attention mechanism in GPT-2 without adding any extra components for reference resolution.
Step 3: Binary Rewriting Classification The model completes the coreference resolution in steps 1 and 2, after which it starts producing the rewritten query. Unlike existing query rewrite systems that directly generate the rewrite given the input, our model first predicts whether the incoming query requires to be rewritten using a binary classifier. As shown in Figure 3, the classifier, a two-layer feed-forward network followed by a softmax layer, takes as input the hidden state of the first decoding step and predicts a vector with two entries representing the rewrite and no-rewrite classes. Only when the binary prediction is true, i.e., the classifier predicts the class indicating that a rewrite is required, does the model enter Step 4 to generate the rewritten query; otherwise, the input query will be directly copied as the output. We show that a well-learned binary classifier with  $93\%$  accuracy functions as a filter that helps the model not only minimize the risk of incorrectly rewriting already self-contained queries, but also allows the rest of the generation process to solely focus on how to rewrite incomplete queries during training.
Step 4: Query Rewrite Generation In this final step, the model runs the generation step based on its binary decision of whether or not to rewrite. Unlike the standard language modeling setup in GPT-2, where the output sequence is generated
directly from the last hidden states, we design the Coref2QR attention layer that allows information gained during coreference resolution to effectively assist in the query rewrite generation.
First, all relevant hidden states of mentions and referents predicted in Steps 1 and 2 are assembled to form a memory pool  $M$ . Note that it is possible for an example to have more than one coreference link. At each time step  $t'$  during the rewrite generation, the Coref2QR attention layer, operating as the standard multi-head attention mechanism, takes  $h_{t'}$  as query to attend over the coreference related states  $M$  by treating them as keys and values. The resulting attention head  $c_{t'}$  is summed with  $h_{t'}$  to obtain the feature  $f_{t'}$  before the final output token classifier. This design improves information flow between the two tasks, enabling the model to directly utilize information regarding previously resolved coreferents during rewrite generation.
The Coref2QR attention can be applied to any arbitrary decoder layer to facilitate the deeper interaction between rewrite and coreference resolution in the model. Formally, at each decoder layer  $l$ , the memory pool  $M^l$  stores coreference related states produced at layer  $l$ . At the generation step  $t'$ , the Coref2QR layer takes  $h_{t'}^l$  as query to attend over  $M^l$  to obtain  $c_{t'}^l$ . The final feature  $f_{t'}$  before the output token classifier is then obtained by  $f_{t'} = h_{t'}^L + \frac{1}{L} \sum_l c_{t'}^l$ . For simplicity, in Figure 3 we only illustrate the Coref2QR attention for the last decoder layer. Our results and analysis show that this Coref2QR attention design benefits the quality of query rewrite, especially in rewriting an anaphora into its antecedent.
Optimization During training, an input sequence with length  $T$  is formed by the concatenation of the dialogue context, the user query and the target query rewrite. Four objectives, corresponding to each step in the model, are used for training. For mention detection, the objective is the cross-entropy between the predicted sequence of mention class  $p^M$  and its ground-truth sequence  $y^M$ :
$$
L ^ {M} = \sum_ {t = q _ {S}} ^ {q _ {E}} - \log \left(\left(\mathbf {y} _ {t} ^ {M}\right) ^ {\top} \mathbf {p} _ {t} ^ {M}\right), \tag {2}
$$
where  $q_{S}$  and  $q_{E}$  denote the start and end index of the query respectively.  $\top$  is the transpose operation.
For each coreference link  $n$ , the loss is calculated using the cross-entropy between the predicted
distributions of the antecedent boundaries  $q^n$  and the corresponding ground-truth  $y^{R_n}$ . The final loss for reference resolution is the sum of losses from the existing coreference links:
$$
L ^ {R} = \sum_ {n = 1} ^ {N} - \log \left(\left(\mathbf {y} _ {r _ {S}} ^ {R _ {n}}\right) ^ {\top} \mathbf {q} _ {r _ {S}} ^ {n}\right) - \log \left(\left(\mathbf {y} _ {r _ {E}} ^ {R _ {n}}\right) ^ {\top} \mathbf {q} _ {r _ {E}} ^ {n}\right), \tag {3}
$$
where  $N$  is the number of coreference links in an example.  $\mathbf{q}_{r_S}^n$  and  $\mathbf{q}_{r_E}^n$  represent the predicted distributions of reference start  $r_S$  and reference end  $r_E$  respectively. When an example does not contain a coreference link,  $L^R$  would be 0.
For query rewrite, the binary classification loss is the two-class cross entropy between the prediction  $p^B$  and the binary rewriting label  $y^{B}$ :
$$
L ^ {B} = - \log \left(\left(\mathbf {y} ^ {B}\right) ^ {\top} \mathbf {p} ^ {B}\right) \tag {4}
$$
For generation, as in the standard language modeling task, we use cross-entropy between the predicted sequence  $p^Q$  and its ground-truth sequence  $y^{Q}$ :
$$
L ^ {Q} = \sum_ {t ^ {\prime} = q _ {E} + 1} ^ {T} - \log \left(\left(\mathbf {y} _ {t ^ {\prime}} ^ {Q}\right) ^ {\top} \mathbf {p} _ {t ^ {\prime}} ^ {Q}\right) \tag {5}
$$
where  $t'$  is the time step in the word sequence of query rewrite. Note that  $L^Q$  is 0 for examples that do not need rewrite. The final loss is the sum of all these losses:
$$
L = L ^ {M} + L ^ {R} + L ^ {B} + L ^ {Q}. \tag {6}
$$