SlowGuess/NAACL_2022_OCR huggingface dataset:
Shuzheng Si $^{1,2*}$ , Shuang Zeng $^{1,2*}$ , Baobao Chang $^{1\dagger}$
sishuzheng@stu.pku.edu.cn {zengs, chbb} @pu.edu.cn
Incomplete utterance rewriting has recently raised wide attention. However, previous works do not consider the semantic structural information between incomplete utterance and rewritten utterance or model the semantic structure implicitly and insufficiently. To address this problem, we propose a QUEry-Enhanced Network (QUEEN). Firstly, our proposed query template explicitly brings guided semantic structural knowledge between the incomplete utterance and the rewritten utterance making model perceive where to refer back to or recover omitted tokens. Then, we adopt a fast and effective edit operation scoring network to model the relation between two tokens. Benefiting from extra information and the well-designed network, QUEEN achieves state-of-the-art performance on several public datasets.
Multi-turn dialogue modeling, a classic research topic in the field of human-machine interaction, serves as an important application area for pragmatics (Leech, 2003) and Turing Test. The major challenge in this task is that interlocutors tend to use incomplete utterances for brevity, such as referring back to (i.e., coreference) or omitting (i.e., ellipsis) entities or concepts that appear in dialogue history. As shown in Table 1, the incomplete utterance  $u_{3}$  refers to "Smith" ("史密斯") from  $u_{1}$  and  $u_{2}$  using a pronoun "He" ("他") and omits "the type of cuisine" ("菜肴的类型") from  $u_{2}$ . This may cause referential ambiguity and semantic incompleteness problems if we only read this single utterance  $u_{3}$ , which is a common case of downstream applications like retrieval-based dialogue systems (Boussaha et al., 2019). Moreover, previous studies (Su et al., 2019; Pan et al., 2019)
Turns	Utterances with Translation
u1	Smith needs to find an expensive restaurant nearby.
史密斯需要在附近找一家昂贵的餐馆。
u2	Does Smith care the type of cuisine?
史密斯关心菜肴的类型吗?
u3	No, he does not care.
不，他不关心。
u3'	No, Smith does not care about the type of cuisine.
不，史密斯不关心菜肴的类型。
Table 1: An example in multi-turn dialogue including dialogue utterance history  $u_{1}$  and  $u_{2}$ , incomplete utterance  $u_{3}$  and rewritten utterance  $u_{3}'$ .
also find that coreference and ellipsis exist in more than  $70\%$  of the utterances, especially in pro-drop languages like Chinese. These phenomena make it imperative to effectively model dialogue in incomplete utterance scenarios.
To cope with this problem, previous works (Kumar and Joshi, 2016a; Elgohary et al., 2019; Su et al., 2019) propose the Incomplete Utterance Rewriting (IUR) task. It aims to rewrite an incomplete utterance into a semantically equivalent but self-contained utterance by mining semantic clues from the dialogue history. Then the generated utterance can be understood without reading dialogue history. For example, in Table 1, after recovering the referred and omitted information from  $u_{3}$  into  $u_{3}^{\prime}$ , we could better understand this utterance comprehensively than before.
Early works use coreference resolution methods (Clark and Manning, 2016) to identify the entity that a pronoun refers to. However, they ignore the more common cases of ellipsis. So the text generation-based methods (Su et al., 2019; Pan et al., 2019) are introduced to generate the rewritten sequence from the incomplete sequence by jointly considering coreference and ellipsis problems. Though effective, generation models neglect a key trait of the IUR task, where the main semantic structure of a rewritten utterance is usually similar to the original incomplete utterance. So the
Human Language Technologies, pages 4839 - 4847
inherent structure-unawareness and uncontrollable feature of generation-based models impede their performances. For semantic structure-aware methods, Liu et al. (2020) utilize an edit operation matrix (e.g., substitution, insertion operations) to convert an incomplete utterance into a complete one. They formulate this task as a semantic segmentation problem with a CNN-based model (Ronneberger et al., 2015) on the matrix to capture the semantic structural relations between words implicitly. Xu et al. (2020) attempt to add additional semantic information to language models (Devlin et al., 2019) by annotating semantic role information but it is time-consuming and costly. Huang et al. (2021) propose a semi-autoregressive generator using a tagger to model the some considerable overlapping regions between the incomplete utterance and rewritten utterance, yet only implicitly learn the difference between them. Although these methods maintain some similarities between the incomplete utterance and the rewritten utterance (i.e., the overlap between them), it is difficult for these methods to explicitly model the semantic structure, especially the difference between the two utterances, ignoring the information in the incomplete utterance, such as which tokens are more likely to be replaced and which positions are more likely to require the insertion of new tokens. Therefore, there are still limitations of existing methods for IUR task, especially in jointly considering coreference and ellipsis cases and better utilizing semantic structural information.
This paper proposes a simple yet effective QUery-Enhanced Network (QUEEN) to solve the IUR task. QUEEN jointly considers coreference and ellipsis problems that frequently happen in multi-turn utterances. Specifically, we propose a straightforward query template featuring two linguistic properties and concatenate this query with utterances as input text. This query explicitly brings semantic structural guided information shared between the incomplete and the rewritten utterances, i.e., making model perceive where to refer back to or recover omitted tokens. We regard the rewritten utterance as the output from a series of edit operations on the incomplete utterance by constructing a token-pair edit operation matrix, which attempts to model the overlap between the incomplete utterance between the rewritten utterance. Different from Liu et al. (2020), we adopt a well-designed edit operation scoring network on
the matrix to perform incomplete utterance rewriting, which is faster and more effective. QUEEN brings semantic structural information from linguistics into the model more explicitly and avoids unnecessary overheads of labeled data from other tasks. Experiments on several IUR benchmarks show that QUEEN outperforms previous state-of-the-art methods. Extensive ablation studies also confirm that the proposed query template makes key contributions to the improvements of QUEEN.
Overview Our QUEEN mainly consists of two modules: query template construction module (Sec. 2.1) and edit operation scoring network module (Sec. 2.2). From two linguistic perspectives, the former module aims to generate a query template for each incomplete utterance, i.e., coreference-ellipsis-oriented query template, to cope with coreference and ellipsis problems. This query template explicitly hints the model where to refer back and recover omitted tokens. The latter module tries to capture the semantic structural relations between tokens by constructing an edit operation matrix. As shown in Figure 1, our goal is to learn a model to generate correct edit operations on this matrix and compute edit operation scores between token pairs so as to convert the incomplete utterance into the complete one.
2.1 Query Template Construction Module
By observing incomplete and rewritten utterance pairs in existing datasets, we find that pronouns and referential noun phrases in the incomplete utterance often need to be substituted by text spans in dialogue history. And ellipsis often occurs in some specific positions of incomplete utterance, conforming to a certain syntactic structure. In this module, we expect to encode these linguistic prior knowledge into the input of QUEEN. The query template is constructed as follows:
Coreference-oriented Query Template In order to make QUEEN perceive the positions of coreference that need to be substituted by text spans from dialogue history, we use a special token [COREF] to replace pronouns and referential noun phrases in the incomplete utterance so as to get our coreference-oriented query template. For example, the coreference-oriented query template of the incomplete utterance “No, he does not care” (“不，他不关心”) is “No, [COREF] does not
Figure 1: General architecture of QUEEN.
care”（“不，[COREF] 不关心”）. To get the target complete utterance, this query explicitly tells the model we should replace the “He”（“他”）with text spans (such as 'Smith'（“史密斯”）) from dialogue history, rather than replacing other words. Here, we find all pronouns that required to be replaced using a predefined pronoun collection.
Ellipsis-oriented Query Template To make QUEEN perceive the positions of ellipsis that need to be inserted by text spans from dialogue history, we define a special token [ELLIP] and put it in a linguistically right place of the incomplete utterance. Since a self-contained utterance usually contains a complete S-V-O (Subject-Verb-Object) structure, if an incomplete utterance lack any of these key elements, we could assume there is a case of ellipsis in its corresponding text position. So we perform dependency parsing on the incomplete utterance to get the structure of the incomplete utterance. For example, the parsing result of the incomplete utterance "No, he does not care" ("不,他不关心") is an S-V structure and lack object element, thus we put [ELLIP] at the end of the sentence to get the ellipsis-oriented query template as "No, he does not care [ELLIP]" ("不,他不关心 [ELLIP])". Other circumstances are detailed in the Appendix.
Then we fuse these two query templates into the final coreference-ellipsis-oriented query template. For incomplete utterance “No, he does not care” (“不,他不关心”), we get “No, [COREF] does not care [ELLIP]’（“不,[COREF] 不关心 [ELLIP]”）as our final query template. Under supervised setting, the models will perceive the positions to refer back and recover omitted
tokens for this utterance. For a multi-turn dialogue  $d = (u_{1},\ldots ,u_{N - 1},u_{N})$  containing  $N$  utterances where  $u_{1}\sim u_{N - 1}$  are dialogue history and the last utterance  $u_{N}$  needs to be rewritten, we could get the dialogue history text  $s = (w_1^1,\dots,w_i^n,\dots,w_{L_N}^N)$  where  $w_{i}^{n}$  is the  $i$ -th token in the  $n$ -th utterance and  $L_{n}$  is the length of  $n$ -th utterance. We then concatenate our coreference-ellipsis-oriented query template with the dialogue history text to get our final input text  $s' = (w_1^q,\dots,w_k^q,\dots,w_M^q,w_1^1,\dots,w_i^n,\dots,w_{L_N}^N)$  where  $w_{k}^{q}$  is the  $k$ -th token of the query template and  $M$  is the length of query template.
2.2 Edit Operation Scoring Network Module
Since pre-trained language models have been proven to be strongly effective on several natural language processing tasks, we employ BERT (Devlin et al., 2019) to encode our input text to get the contextualized hidden representation  $H = (h_1^q,\dots,h_k^q,\dots,h_M^q,h_1^1,\dots,h_i^n,\dots,h_{LN}^N)$ . Our model attempts to predict whether there is an edit operation between each token pair. To this end, we define an operation scoring function as follows. Since the order of utterance is also important for dialogue, we further use RoPE (Su et al., 2021) to provide relative position information:
$$
q _ {i} ^ {\alpha} = W ^ {\alpha} h _ {i} + b ^ {\alpha} \tag {1}
$$
$$
k _ {j} ^ {\alpha} = W ^ {\alpha} h _ {j} + b ^ {\alpha} \tag {2}
$$
$$
s _ {i j} ^ {\alpha} = \left(R _ {i} q _ {i} ^ {\alpha}\right) ^ {T} \left(R _ {j} k _ {j} ^ {\alpha}\right) \tag {3}
$$
where  $\alpha$  is edit operation type including Substitution and Pre-Insertion. For different operations, we use different trainable parameters  $W^{\alpha}$  and  $b^{\alpha}$ .
Model	EM	B2	B4	R2	RL
T-Gen†	35.4	72.7	62.5	74.5	82.9
L-Ptr-λ†	42.3	82.9	73.8	81.1	84.1
L-Gen†	47.3	81.2	73.6	80.9	86.3
L-Ptr-Gen†	50.5	82.9	75.4	83.8	87.8
T-Ptr-Gen†	53.1	84.4	77.6	85.0	89.1
T-Ptr-λ†	52.6	85.6	78.1	85.0	89.0
T-Ptr-λ + BERT†	57.5	86.5	79.9	86.9	90.5
CSRL	60.5	86.8	77.8	85.9	90.5
RUN†	66.4	91.4	86.2	90.4	93.5
QUEEN	70.1	92.1	86.9	90.9	94.6
Table 2: Experimental results on REWRITE. †: Results from Liu et al. (2020).
Model	EM	B1	B2	R1	R2
Syntactic†	-	84.1	81.2	89.3	80.6
L-Gen†	-	84.9	81.7	88.8	80.3
L-Ptr-Gen†	-	84.7	81.7	89.0	80.9
BERT‡	-	85.2	82.5	89.5	80.9
PAC†	-	89.9	86.3	91.6	82.8
CSRL‡	-	85.8	82.9	89.6	83.1
SARG	-	92.2	89.6	92.1	86.0
RUN†	49.3	92.3	89.6	92.4	85.1
QUEEN	53.5	92.4	89.8	92.5	86.3
$R$  is a transformation matrix from RoPE to inject position information and  $s_{ij}^{\alpha}$  is the score for  $\alpha$ -th edit operation from  $i$ -th token in dialogue history to  $j$ -th token in incomplete utterance.
During decoding for  $\alpha$ -th operation, edit operation label  $\mathcal{V}_{ij}^{\alpha}$  satisfies:
$$
\mathcal {Y} _ {i j} ^ {\alpha} = \left\{ \begin{array}{l l} 1 & s _ {i j} ^ {\alpha} > = \theta \\ 0 & s _ {i j} ^ {\alpha} <   \theta \end{array} \right. \tag {4}
$$
where  $\theta$  is a hyperparameter. Once  $\mathcal{Y}_{ij}^{\alpha}$  equals to 1, the edit operation  $\alpha$  should be performed between token  $i$  and token  $j$ .
Since the label distribution of edit operation is very unbalanced (most elements are zeros), we employ Circle Loss (Sun et al., 2020) as our objective function to mitigate this problem:
$$
\log \left(1 + \sum_ {(i, j) \in \Omega_ {p o s}} e ^ {- s _ {i, j} ^ {\alpha}}\right) + \log \left(1 + \sum_ {(i, j) \in \Omega_ {n e g}} e ^ {s _ {i, j} ^ {\alpha}}\right) (5)
$$
Datasets We evaluate our model on four IUR benchmarks from different domains and lan
Table 3: Experimental results on Restoration-200K. Additionally, we reproduce from the released code to get EM of RUN. †: Results from Liu et al. (2020). ‡: Results from Xu et al. (2020).
Model	EM	B4
Ellipsis Recovery†	50.4	74.1
GECOR 1†	68.5	83.9
GECOR 2†	66.2	83.0
RUN†	69.2	85.6
QUEEN	71.6	86.3
Table 4: Experimental results on TASK. †: Results and evaluation metrics from Liu et al. (2020).
Model	B1	B2	B4	R1	R2	RL
Copy†	52.4	46.7	37.8	72.7	54.9	68.5
Pronoun Sub†	60.4	55.3	47.4	73.1	63.7	73.9
L-Ptr-Gen†	67.2	60.3	50.2	78.9	62.9	74.9
RUN†	70.5	61.2	49.1	79.1	61.2	74.7
QUEEN	72.4	65.2	54.4	82.5	68.1	81.8
Table 5: Experimental results on CANARD. †: Results and evaluation metrics from Liu et al. (2020).
guages: REWRITE (Chinese, Su et al., 2019), Restoration-200K (Chinese, Pan et al., 2019), TASK (English, Quan et al., 2019), CANARD (English, Elgohary et al., 2019). More statistical details for datasets are shown in the Appendix.
Evaluation We use BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and the exact match (EM score) as our evaluation metrics.
Baseline Models We compare our model with a large number of baselines and state-of-the-art models. (i) Baselines and Generation models include sequence to sequence model (L-Gen) (Bahdanau et al., 2014), the hybrid pointer generator network (L-Ptr-Gen) (See et al., 2017a), the basic transformer model (T-Gen) (Vaswani et al., 2017) and the transformer-based pointer generator (T-Ptr-Gen) (See et al., 2017a), Syntactic (Kumar and Joshi, 2016b), PAC (Pan et al., 2019), L-Ptr-λ and T-Ptr-λ (Su et al., 2019), GECOR (Quan et al., 2019). Above methods need to generate rewritten utterances from scratch, neglecting the semantic structure between a rewritten utterance and the original incomplete utterance. (ii) Structure-aware models include CSRL (Xu et al., 2020), RUN (Liu et al., 2020), SARG (Huang et al., 2021).
Main Results We report the experiment results in Table 2, Table 3, Table4 and Table 5. On all datasets with different languages and evaluation metrics, our approach outperforms all previous state-of-the-art methods. The improvement in EM shows that our model has a stronger ability to
find the correct span, due to our model making full use of the prior information of semantic structure from our coreference-ellipsis-oriented query template. On the Chinese datasets Table 2 and Table 3, QUEEN outperforms previous methods. Since Chinese is a pro-drop language where coreference and ellipsis often happen, the improvement confirms that QUEEN is superior in finding the correct ellipsis and referring back positions. The results on data sets of different domains and languages also show that our model is robust and effective.
Ablation Study To verify the effectiveness of the query in our model, we present an ablation study in Table 8 in the Appendix. It is clear that query is important to improve performance on all evaluation metrics. Meanwhile, only using coreference-oriented or ellipsis-oriented template still improves the performance, as it can also bring semantic structure information.
Inference Speed Meanwhile, we compare inference speed between our Edit Operation Scoring Network and the current fastest model RUN in A.5. As we remove the heavy model U-net of RUN and apply the simple network, our Edit Operation Scoring Network is faster than previous SOTA RUN.
Case Study We also do a case study in the Appendix. Our model avoids the uncontrolled situations that the generation-based model is prone to, and our model can more easily capture the correct semantic span.
We propose a simple yet effective query-enhanced network for IUR task. Our well-designed query template explicitly brings semantic structural information to improve the ability to predict correct edit operations between incomplete utterance and complete one. Experiments have shown that our model with this well-designed query achieves promising results than previous methods.
This paper is supported by the National Science Foundation of China under Grant No.61876004, 61936012, the National Key R&D Program of China under Grand No.2020AAA0106700.
6 Ethics Consideration
We collect our data from public datasets that permit academic use. The open-source tools we use
are freely accessible online without copyright conflicts.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Basma El Amel Boussaha, Nicolas Hernandez, Christine Jacquin, and Emmanuel Morin. 2019. Deep retrieval-based dialogue systems: A short review. CoRR, abs/1907.12878.
Wanxiang Che, Yunlong Feng, Libo Qin, and Ting Liu. 2020. N-ltp: A open-source neural chinese language technology platform with pretrained models. arXiv preprint arXiv:2009.11616.
Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5918-5924, Hong Kong, China. Association for Computational Linguistics.
Jie Hao, Linfeng Song, Liwei Wang, Kun Xu, Zhaopeng Tu, and Dong Yu. 2021. RAST: Domain robust dialogue rewriting as sequence tagging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4913-4924, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mengzuo Huang, Feng Li, Wuhe Zou, and Weidong Zhang. 2021. SARG: A novel semi autoregressive generator for multi-turn incomplete utterance restoration. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI
2021, Virtual Event, February 2-9, 2021, pages 13055-13063. AAAI Press.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Vineet Kumar and Sachindra Joshi. 2016a. Non-sentential question resolution using sequence to sequence learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2022-2031, Osaka, Japan. The COLING 2016 Organizing Committee.
Vineet Kumar and Sachindra Joshi. 2016b. Non- sentential question resolution using sequence to sequence learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2022-2031.
Geoffrey Leech. 2003. Pragmatics and dialogue. In *The Oxford handbook of computational linguistics*.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020. Incomplete utterance rewriting as semantic segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn incomplete utterance restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1824-1833, Hong Kong, China. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. Gecor: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4539-4549.