Venue: NAACL 2022

# Mining Clues from Incomplete Utterance:

A Query-enhanced Network for Incomplete Utterance Rewriting

Shuzheng Si\({}^{1,2}\); Shuang Zeng\({}^{1,2}\); Baobao Chang\({}^{1}\)

\({}^{1}\)Key Laboratory of Computational Linguistics, Peking University, MOE, China

\({}^{2}\)School of Software and Microelectronics, Peking University, China

sishuzheng@stu.pku.edu.cn {zengs,chbb}@pku.edu.cn

Equal contribution.Corresponding author.

###### Abstract

Incomplete utterance rewriting has recently raised wide attention. However, previous works do not consider the semantic structural information between incomplete utterance and rewritten utterance or model the semantic structure implicitly and insufficiently. To address this problem, we propose a **QUE**ry-**En**hanced **N**etwork (**QUEEN**). Firstly, our proposed query template explicitly brings guided semantic structural knowledge between the incomplete utterance and the rewritten utterance making model perceive where to refer back to or recover omitted tokens. Then, we adopt a fast and effective edit operation scoring network to model the relation between two tokens. Benefiting from extra information and the well-designed network, QUEEN achieves state-of-the-art performance on several public datasets.

## 1 Introduction

Multi-turn dialogue modeling, a classic research topic in the field of human-machine interaction, serves as an important application area for pragmatics Leech (2003) and Turing Test. The major challenge in this task is that interlocutors tend to use incomplete utterances for brevity, such as referring back to (i.e., coreference) or omitting (i.e., ellipsis) entities or concepts that appear in dialogue history. As shown in Table 1, the incomplete utterance \(u_{3}\) refers to "Smith" ('*') from \(u_{1}\) and \(u_{2}\) using a pronoun "He" ('*') and omits "the type of cuisine" ('*') from \(u_{2}\). This may cause referential ambiguity and semantic incompleteness problems if we only read this single utterance \(u_{3}\), which is a common case of downstream applications like retrieval-based dialogue systems Boussaha et al. (2019). Moreover, previous studies Su et al. (2019); Pan et al. (2019) also find that coreference and ellipsis exist in more than 70% of the utterances, especially in pro-drop languages like Chinese. These phenomena make it imperative to effectively model dialogue in incomplete utterance scenarios.

To cope with this problem, previous works Kumar and Joshi (2016); Elgohary et al. (2019); Su et al. (2019) propose the **I**ncomplete **U**tterance **R**ewriting (**IUR**) task. It aims to rewrite an incomplete utterance into a semantically equivalent but self-contained utterance by mining semantic clues from the dialogue history. Then the generated utterance can be understood without reading dialogue history. For example, in Table 1, after recovering the referred and omitted information from \(u_{3}\) into \(u_{3}^{\prime}\), we could better understand this utterance comprehensively than before.

Early works use coreference resolution methods Clark and Manning (2016) to identify the entity that a pronoun refers to. However, they ignore the more common cases of ellipsis. So the text generation-based methods Su et al. (2019); Pan et al. (2019) are introduced to generate the rewritten sequence from the incomplete sequence by jointly considering coreference and ellipsis problems. Though effective, generation models neglect a key trait of the IUR task, where the main semantic structure of a rewritten utterance is usually similar to the original incomplete utterance. So the

\begin{table}
\begin{tabular}{c c} Turns & Utterances with Translation \\ \hline \multirow{2}{*}{\(u_{1}\)} & Smith needs to find an expensive restaurant nearby. \\  & **[**]

inherent structure-unawareness and uncontrollable feature of generation-based models impede their performances. For semantic structure-aware methods, Liu et al. (2020) utilize an edit operation matrix (e.g., substitution, insertion operations) to convert an incomplete utterance into a complete one. They formulate this task as a semantic segmentation problem with a CNN-based model Ronneberger et al. (2015) on the matrix to capture the semantic structural relations between words implicitly. Xu et al. (2020) attempt to add additional semantic information to language models Devlin et al. (2019) by annotating semantic role information but it is time-consuming and costly. Huang et al. (2021) propose a semi-autoregressive generator using a tagger to model the some considerable overlapping regions between the incomplete utterance and rewritten utterance, yet only implicitly learn the difference between them. Although these methods maintain some similarities between the incomplete utterance and the rewritten utterance (i.e., the overlap between them), it is difficult for these methods to explicitly model the semantic structure, especially the difference between the two utterances, ignoring the information in the incomplete utterance, such as which tokens are more likely to be replaced and which positions are more likely to require the insertion of new tokens. Therefore, there are still limitations of existing methods for IUR task, especially in jointly considering coreference and ellipsis cases and better utilizing semantic structural information.

This paper proposes a simple yet effective **QUE**ry-**E**nhanced **N**etwork (**QUEEN**) to solve the IUR task. QUEEN jointly considers coreference and ellipsis problems that frequently happen in multi-turn utterances. Specifically, we propose a straightforward query template featuring two linguistic properties and concatenate this query with utterances as input text. This query explicitly brings semantic structural guided information shared between the incomplete and the rewritten utterances, i.e., making model perceive where to refer back to or recover omitted tokens. We regard the rewritten utterance as the output from a series of edit operations on the incomplete utterance by constructing a token-pair edit operation matrix, which attempts to model the the overlap between the incomplete utterance between the rewritten utterance. Different from Liu et al. (2020), we adopt a well-designed edit operation scoring network on the matrix to perform incomplete utterance rewriting, which is faster and more effective. QUEEN brings semantic structural information from linguistics into the model more explicitly and avoids unnecessary overheads of labeled data from other tasks. Experiments on several IUR benchmarks show that QUEEN outperforms previous state-of-the-art methods. Extensive ablation studies also confirm that the proposed query template makes key contributions to the improvements of QUEEN.

## 2 Methodology

OverviewOur QUEEN mainly consists of two modules: query template construction module (Sec. 2.1) and edit operation scoring network module (Sec. 2.2). From two linguistic perspectives, the former module aims to generate a query template for each incomplete utterance, i.e., coreference-ellipsis-oriented query template, to cope with coreference and ellipsis problems. This query template explicitly hints the model where to refer back and recover omitted tokens. The latter module tries to capture the semantic structural relations between tokens by constructing an edit operation matrix. As shown in Figure 1, our goal is to learn a model to generate correct edit operations on this matrix and compute edit operation scores between token pairs so as to convert the incomplete utterance into the complete one.

### Query Template Construction Module

By observing incomplete and rewritten utterance pairs in existing datasets, we find that pronouns and referential noun phrases in the incomplete utterance often need to be substituted by text spans in dialogue history. And ellipsis often occurs in some specific positions of incomplete utterance, conforming to a certain syntactic structure. In this module, we expect to encode these linguistic prior knowledge into the input of QUEEN. The query template is constructed as follows:

Coreference-oriented Query TemplateIn order to make QUEEN perceive the positions of coreference that need to be substituted by text spans from dialogue history, we use a special token [COREF] to replace pronouns and referential noun phrases in the incomplete utterance so as to get our coreference-oriented query template. For example, the coreference-oriented query template of the incomplete utterance "No, he does not care" ("\({}^{*}\!\!\mathcal{K}\), "\({}_{\!\!\!\mathcal{K}}\!\times\!\!\mathcal{L}\)") is "No, [COREF] does not 

care" ("[COREF] ). To get the target complete utterance, this query explicitly tells the model we should replace the "He" ("[\({}^{+}\)]") with text spans (such as 'Smith'("[\({}^{-}\)]") ) from dialogue history, rather than replacing other words. Here, we find all pronouns that required to be replaced using a predefined pronoun collection.

#### 3.4.2 Ellipsis-oriented Query Template

To make QUEEN perceive the positions of ellipsis that need to be inserted by text spans from dialogue history, we define a special token [ELLIP] and put it in a linguistically right place of the incomplete utterance. Since a self-contained utterance usually contains a complete S-V-O (Subject-Verb-Object) structure, if an incomplete utterance lack any of these key elements, we could assume there is a case of ellipsis in its corresponding text position. So we perform dependency parsing on the incomplete utterance to get the structure of the incomplete utterance. For example, the parsing result of the incomplete utterance "No, he does not care" ("[\({}^{*}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \! 

\(R\) is a transformation matrix from RoPE to inject position information and \(s^{\alpha}_{ij}\) is the score for \(\alpha\)-th edit operation from \(i\)-th token in dialogue history to \(j\)-th token in incomplete utterance.

During decoding for \(\alpha\)-th operation, edit operation label \(\mathcal{Y}^{\alpha}_{ij}\) satisfies:

\[\mathcal{Y}^{\alpha}_{ij}=\left\{\begin{array}{ll}1&s^{\alpha}_{ ij}>=\theta\\ 0&s^{\alpha}_{ij}<\theta\end{array}\right. \tag{4}\]

where \(\theta\) is a hyperparameter. Once \(\mathcal{Y}^{\alpha}_{ij}\) equals to 1, the edit operation \(\alpha\) should be performed between token \(i\) and token \(j\).

Since the label distribution of edit operation is very unbalanced (most elements are zeros), we employ Circle Loss Sun et al. (2020) as our objective function to mitigate this problem :

\[log(1+\sum_{(i,j)\in\Omega_{pos}}e^{-s^{\alpha}_{i,j}})+log(1+\sum_{(i,j)\in \Omega_{neg}}e^{s^{\alpha}_{i,j}}) \tag{5}\]

## 3 Experiments

### Experimental Setup

**Datasets** We evaluate our model on four IUR benchmarks from different domains and languages: REWRITE Chinese, Su et al. (2019), Restoration-20OK Chinese, Pan et al. (2019), TASK English, Quan et al. (2019), CANARD English, Elgohary et al. (2019). More statistical details for datasets are shown in the Appendix.

**Evaluation** We use BLEU Papineni et al. (2002), ROUGE Lin (2004) and the exact match (EM score) as our evaluation metrics.

**Baseline Models** We compare our model with a large number of baselines and state-of-the-art models. (i) **Baselines and Generation models** include sequence to sequence model (L-Gen) Bahdanau et al. (2014), the hybrid pointer generator network (L-Ptr-Gen) See et al. (2017a), the basic transformer model (T-Gen) Vaswani et al. (2017) and the transformer-based pointer generator (T-Ptr-Gen) See et al. (2017a), Syntactic Kumar and Joshi (2016b), PAC Pan et al. (2019), L-Ptr-\(\lambda\) and T-Ptr-\(\lambda\)Su et al. (2019), GECOR Quan et al. (2019). Above methods need to generate rewritten utterances from scratch, neglecting the semantic structure between a rewritten utterance and the original incomplete utterance. (ii) **Structure-aware models** include CSRL Xu et al. (2020), RUN Liu et al. (2020), SARG Huang et al. (2021).

 

find the correct span, due to our model making full use of the prior information of semantic structure from our coreference-ellipsis-oriented query template. On the Chinese datasets Table 2 and Table 3, QUEEN outperforms previous methods. Since Chinese is a pro-drop language where coreference and ellipsis often happen, the improvement confirms that QUEEN is superior in finding the correct ellipsis and referring back positions. The results on data sets of different domains and languages also show that our model is robust and effective.

**Ablation Study** To verify the effectiveness of the query in our model, we present an ablation study in Table 8 in the Appendix. It is clear that query is important to improve performance on all evaluation metrics. Meanwhile, only using coreference-oriented or ellipsis-oriented template still improves the performance, as it can also bring semantic structure information.

**Inference Speed** Meanwhile, we compare inference speed between our Edit Operation Scoring Network and the current fastest model RUN in A.5. As we remove the heavy model U-net of RUN and apply the simple network, our Edit Operation Scoring Network is faster than previous SOTA RUN.

**Case Study** We also do a case study in the Appendix. Our model avoids the uncontrolled situations that the generation-based model is prone to, and our model can more easily capture the correct semantic span.

## 4 Conclusion

We propose a simple yet effective query-enhanced network for IUR task. Our well-designed query template explicitly brings semantic structural information to improve the ability to predict correct edit operations between incomplete utterance and complete one. Experiments have shown that our model with this well-designed query achieves promising results than previous methods.

## 5 Acknowledgements

This paper is supported by the National Science Foundation of China under Grant No.61876004, 61936012, the National Key R&D Program of China under Grand No.2020AAA0106700.

## 6 Ethics Consideration

We collect our data from public datasets that permit academic use. The open-source tools we use are freely accessible online without copyright conflicts.

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_.
* El Amel Boussaha et al. (2019) Basma El Amel Boussaha, Nicolas Hernandez, Christine Jacquin, and Emmanuel Morin. 2019. Deep retrieval-based dialogue systems: A short review. _CoRR_, abs/1907.12878.
* Che et al. (2020) Wanxiang Che, Yunlong Feng, Libo Qin, and Ting Liu. 2020. N-ltp: A open-source neural chinese language technology platform with pretrained models. _arXiv preprint arXiv:2009.11616_.
* Clark and Manning (2016) Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 643-653, Berlin, Germany. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Elgohary et al. (2019) Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5918-5924, Hong Kong, China. Association for Computational Linguistics.
* Hao et al. (2021) Jie Hao, Linfeng Song, Liwei Wang, Kun Xu, Zhaopeng Tu, and Dong Yu. 2021. RAST: Domain-robust dialogue rewriting as sequence tagging. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4913-4924, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Huang et al. (2021) Mengzuo Huang, Feng Li, Wuhe Zou, and Weidong Zhang. 2021. SARG: A novel semi autoregressive generator for multi-turn incomplete utterance restoration. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 

2021, Virtual Event, February 2-9, 2021_, pages 13055-13063. AAAI Press.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kumar and Joshi (2016a) Vineet Kumar and Sachindra Joshi. 2016a. Non-sentential question resolution using sequence to sequence learning. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 2022-2031, Osaka, Japan. The COLING 2016 Organizing Committee.
* Kumar and Joshi (2016b) Vineet Kumar and Sachindra Joshi. 2016b. Non-sentential question resolution using sequence to sequence learning. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 2022-2031.
* Leech (2003) Geoffrey Leech. 2003. Pragmatics and dialogue. In _The Oxford handbook of computational linguistics_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu et al. (2020) Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020. Incomplete utterance rewriting as semantic segmentation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
* Pan et al. (2019) Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn incomplete utterance restoration. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1824-1833, Hong Kong, China. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Quan et al. (2019) Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. Gecor: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4539-4549.
* Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In _International Conference on Medical image computing and computer-assisted intervention_, pages 234-241. Springer.
* August 4, Volume 1: Long Papers_, pages 1073-1083. Association for Computational Linguistics.
* See et al. (2017b) Abigail See, Peter J Liu, and Christopher D Manning. 2017b. Get to the point: Summarization with pointer-generator networks. _arXiv preprint arXiv:1704.04368_.
* Su et al. (2019) Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance ReWriter. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 22-31, Florence, Italy. Association for Computational Linguistics.
* Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. _arXiv preprint arXiv:2104.09864_.
* Sun et al. (2020) Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6398-6407.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Xu et al. (2020) Kun Xu, Haochen Tan, Linfeng Song, Han Wu, Haisong Zhang, Linqi Song, and Dong Yu. 2020. Semantic Role Labeling Guided Multi-turn Dialogue RewWriter. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6632-6639, Online. Association for Computational Linguistics.
* Zhang et al. (2021) Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_, pages 3999-4006. ijcai.org 

Experimental Details

### Constructing Supervision

The expected supervision for our model is the edit operation matrix, but existing datasets only contain rewritten utterances. So we adopt Longest Common Subsequence (LCS) and 'Distant Supervision' Liu et al. (2020) to get correct supervision, which contains edit operations _Substitute_ and _PreInsert_.

### Details in Query Construction

Coreference-oriented Query During the training, we use the ground truth of pronouns and referential noun phrases to construct the coreferenc-oriented query. During the inference, we use the constructed pronoun collection to construct the coreference-oriented query, which contains pronouns and referential noun phrases from training data and common pronouns (except "?k", "me" etc. ).

Ellipsis-oriented Query ConstructionIf the parsing result of the incomplete utterance is an S-V (Subject-Verb) structure and lacks subject element, we insert an [ELLIP] at the end of the incomplete utterance as the query. When there is not the S-V structure after parsing, we insert an [ELLIP] at the beginning of the incomplete utterance as the query. In other cases, we insert [ELLIP] at both the beginning and end of the incomplete utterance as the query. We use spaCy1 for English and LTPChe et al. (2020) for Chinese to get the result of parsing.

Footnote 1: [https://spacy.io/](https://spacy.io/)

Extra findingsDuring the experiment, we find two interesting points: (i) As [COREF] and [ELLIP] are sparse respectively, we use a unified token [UNK] to replace [COREF] and [ELLIP] in the query to relieve the sparsity. (ii) In most cases, if there is the referring back in the utterance, there is generally no ellipsis in the utterance. Redundant [ELLIP] tokens can't bring correct guided information in this case. Therefore, once we construct Coreference-oriented Query Template successfully, we will not try to construct the Ellipsis-oriented Query Template. Our experimental results are improved by the above two tricks.

### Datasets details

Some statistics are shown in Table 6. REWRITE and Restoration-200K are constructed from Chinese Open-Domain Dialogue. TASK is from English Task-oriented Dialogue. CANARD is constructed from English Context Question Answering. We fellow the same data split as their original paper.

### Details in Scoring Network

Hyper-parametersWe implement our model on top of a BERT-base model Devlin et al. (2019). We initialize QUEEN with bert-base-uncased for English and bert-base-chinese for Chinese. We use Adam Kingma and Ba (2015) with learning rate 1e-5. The batch size is set to 16 for REWRITE and TASK, 12 for Restoration-200K, 4 for on CAMARD. Meanwhile, \(\theta\) in Equation 4 is set to 0.1 for REWRITE and TASK, 0.05 for Restoration-200K and CANARD.

Circle LossWe tune Circle Loss the same as Zhang et al. (2021) and Su2. We refer readers to their paper for more details.

Footnote 2: A helpful Chinese blog: [https://kexue.fm/archives/7359](https://kexue.fm/archives/7359)

### Inference Speed

To compare the inference speed between the current fastest model RUN and our Edit Operation Scoring Network, we conduct experiments using the code released 3. Both models are implemented in PyTorch on a single NVIDIA V100. The batch size is set to 16. Meanwhile, In order to fairly compare the speed of the two networks, we performed Distant Supervision and Query Construction before comparing. The results are shown in Table 7.

\begin{table}
\begin{tabular}{l c c c c} \hline  & Restoration-200K & REWRITE & TASK & CANARD \\ \hline Language & Chinese & Chinese & English & English \\ His Avglen & 25.8 & 17.7 & 52.6 & 85.4 \\ Inc Avglen & 8.6 & 6.5 & 9.4 & 7.5 \\ Rew Agylen & 12.4 & 10.5 & 11.3 & 11.6 \\ \hline \end{tabular}
\end{table}
Table 6: Statistics of different datasets. ’Avglen’ for average length, ’His’ for historical utterance, ‘Inc’ for incomplete utterance, and ’Rev’ for rewritten utterance.

\begin{table}
\begin{tabular}{c c c} \hline Model & Inference Speed & Speedup \\ \hline RUN+BERT & 1.69it/s & 1.00\(\times\) \\
**QUEEN** & **2.13it/s** & **1.26\(\times\)** \\ \hline \end{tabular}
\end{table}
Table 7: The inference speed comparison between RUN and QUEEN on REWRITE dataset.

 



### Case Study

Table 9 gives 3 examples that indicate the representative situations as Hao et al. (2021). The first example illustrates the cases when RUN inserts unexpected characters into the wrong places. T-Ptr-Gen just copies the incomplete utterance. Due to our generated query, the position that needs to be inserted has been explicitly promoted by the query. The second example shows a common situation for generation-based models. T-Ptr-Gen messes up by repeating stupidly. However, this situation doesn't happen to our model, as it is not a generation-based model. The last example refers to a long and complex entity. For these cases, it is easier for our model to get the correct span. This is because our model learns the span boundaries from the edit operation matrix. Compared to the generation-based model, we don't generate sentences from scratch and this reduces the difficulty. Meanwhile, our model is not based on CNN as RUN, which suffers from the limitation of receptive-field to find a longer span.

### Limitation

One limitation of current edit-based IUR models, is that only tokens that have appeared in the history dialogue can be selected. Therefore, these models, including ours, cannot generate novel words, e.g., conjunctions and prepositions, to cater to other metrics, like fluency. However, this can be alleviated by incorporating an additional word dictionary as See et al. (2017b) and Liu et al. (2020) deals with the out-of-vocabulary (OOV) words to improve fluency. For fairness, we keep the same words during the experiment as RUN to mitigat it. We will consider this question as a promising direction for future works.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline
**Variant** & EM & \(B_{1}\) & \(B_{2}\) & \(B_{3}\) & \(B_{4}\) & \(R_{1}\) & \(R_{2}\) & \(R_{L}\) \\ \hline
**QUEEN** & **70.1** & **94.7** & **92.1** & **89.5** & **86.9** & **96.0** & **90.9** & **94.6** \\ \hline -w/o query & 67.4 & 92.9 & 90.5 & 88.1 & 85.7 & 95.2 & 90.1 & 94.0 \\ -w/o CQT & 67.9 & 93.5 & 91.0 & 88.5 & 86.0 & 95.5 & 90.4 & 94.2 \\ -w/o EQT & 67.4 & 93.5 & 91.1 & 88.5 & 85.9 & 95.4 & 90.3 & 94.3 \\ \hline \end{tabular}
\end{table}
Table 8: The ablation results on the development set of REWRITE. “w/o query” means that we do not append a designed query before encoding that semantic information into our model. “w/o CQT” means that we only perform Ellipsis-oriented Query Template. “w/o EQT” means that we only perform Coreference-oriented Query Template and use the incomplete utterance as query if match fails.





\begin{table}
\begin{tabular}{c c} Example # 1 \\ \hline Historical Utterance 1 & \\  & I have a lot of complaints \\ \hline Historical Utterance 2 & \\  & Keep it yourself if there’s any \\ \hline Incomplete Utterance & \\  & Don’t want to keep it myself \\ \hline Gold & \\  & Don’ t want to keep the complaints myself \\ \hline T-Ptr-Gen & \\  & Don’t want to keep it myself \\ \hline RUN & \\  & Complaints don’t want to keep the complaints myself \\ \hline QUEEN & \\  & Don’t want to keep the complaints myself \\ \end{tabular} \begin{tabular}{c c} Example # 2 \\ \hline Historical Utterance 1 & \\  & Please help me on IESTS \\ \hline Historical Utterance 2 & \\  & What is tested first for IESTS \\ \hline Incomplete Utterance & \\  & It's oral test \\ \hline Gold & \\  & It’s oral test for IESTS \\ \hline T-Ptr-Gen & \\  & It’s oral test oral test \\ \hline RUN & \\  & It’s oral test for IESTS \\ \hline QUEEN & \\  & It’s oral test for IESTS \\ \end{tabular} 
\begin{tabular}{c c} Example # 3 \\ \hline Historical Utterance 1 & \\  & Can you help me find a free ride from Xi’an to Shangluo \\ \hline Historical Utterance 2 & \\  & Where is it \\ \hline Incomplete Utterance & \\  & Can you find any \\ \hline Gold & \\  & Can you find any free ride from Xi’an to Shangluo \\ \hline T-Ptr-Gen & \\  & Can you find any free ride to Shangluo \\ \hline RUN & \\  & Can you find any free ride to Shangluo \\ \hline QUEEN & \\  & Can you find any free ride from Xi’an to Shangluo \\ \end{tabular}
\end{table}
Table 9: Case Study