SlowGuess/NAACL_2022_OCR huggingface dataset:
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017a. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1073-1083. Association for Computational Linguistics.
Abigail See, Peter J Liu, and Christopher D Manning. 2017b. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance ReWriter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 22-31, Florence, Italy. Association for Computational Linguistics.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398-6407.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Kun Xu, Haochen Tan, Linfeng Song, Han Wu, Haisong Zhang, Linqi Song, and Dong Yu. 2020. Semantic Role Labeling Guided Multi-turn Dialogue ReWriter. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6632-6639, Online. Association for Computational Linguistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3999-4006. ijcai.org.
A.1 Constructing Supervision
The expected supervision for our model is the edit operation matrix, but existing datasets only contain rewritten utterances. So we adopt Longest Common Subsequence (LCS) and 'Distant Supervision' (Liu et al., 2020) to get correct supervision, which contains edit operations Substitute and PreInsert.
A.2 Details in Query Construction
Coreference-oriented Query During the training, we use the ground truth of pronouns and referential noun phrases to construct the coreference-oriented query. During the inference, we use the constructed pronoun collection to construct the coreference-oriented query, which contains pronouns and referential noun phrases from training data and common pronouns (except "我", "me" etc.).
Ellipsis-oriented Query Construction If the parsing result of the incomplete utterance is an S-V (Subject-Verb) structure and lacks subject element, we insert an [ELLIP] at the end of the incomplete utterance as the query. When there is not the S-V structure after parsing, we insert an [ELLIP] at the beginning of the incomplete utterance as the query. In other cases, we insert [ELLIP] at both the beginning and end of the incomplete utterance as the query. We use spaCy<sup>1</sup> for English and LTP(Che et al., 2020) for Chinese to get the result of parsing.
Extra findings During the experiment, we find two interesting points: (i) As [COREF] and [ELLIP] are sparse respectively, we use a unified token [UNK] to replace [COREF] and [ELLIP] in the query to relieve the sparsity. (ii) In most cases, if there is the referring back in the utterance, there is generally no ellipsis in the utterance. Redundant [ELLIP] tokens can't bring correct guided information in this case. Therefore, once we construct Coreference-oriented Query Template successfully, we will not try to construct the Ellipsis-oriented Query Template. Our experimental results are improved by the above two tricks.
A.3 Datasets details
Some statistics are shown in Table 6. REWRITE and Restoration-200K are constructed from Chi
Restoration-200K REWRITE	TASK	CANARD
Language	Chinese	Chinese	English
His Avglen	25.8	17.7	52.6
Inc Avglen	8.6	6.5	9.4
Rew Avglen	12.4	10.5	11.3
Table 6: Statistics of different datasets. 'Avgren' for average length, 'His' for historical utterance, 'Inc' for incomplete utterance, and 'Rew' for rewritten utterance.
Model	Inference Speed	Speedup
RUN+BERT	1.69it/s	1.00×
QUEEN	2.13it/s	1.26×
Table 7: The inference speed comparison between RUN and QUEEN on REWRITE dataset.
nese Open-Domain Dialogue. TASK is from English Task-oriented Dialogue. CANARD is constructed from English Context Question Answering. We fellow the same data split as their original paper.
A.4 Details in Scoring Network
Hyper-parameters We implement our model on top of a BERT-base model (Devlin et al., 2019). We initialize QUEEN with bert-base-uncased for English and bert-base-chinese for Chinese. We use Adam (Kingma and Ba, 2015) with learning rate 1e-5. The batch size is set to 16 for REWRITE and TASK, 12 for Restoration-200K, 4 for on CANARD. Meanwhile,  $\theta$  in Equation 4 is set to 0.1 for REWRITE and TASK, 0.05 for Restoration-200K and CANARD.
Circle Loss We tune Circle Loss the same as Zhang et al. (2021) and  $\mathrm{Su}^2$ . We refer readers to their paper for more details.
A.5 Inference Speed
To compare the inference speed between the current fastest model RUN and our Edit Operation Scoring Network, we conduct experiments using the code released<sup>3</sup>. Both models are implemented in PyTorch on a single NVIDIA V100. The batch size is set to 16. Meanwhile, In order to fairly compare the speed of the two networks, we performed Distant Supervision and Query Construction before comparing. The results are shown in Table 7.
2A helpful Chinese blog: https://kexue.fm/archives/7359
<sup>3</sup>https://github.com/microsoft/ContextualSP/tree/master/incomplete_utterance_rewriting
1https://spacy.io/
Variant	EM	B1	B2	B3	B4	R1	R2	RL
QUEEN	70.1	94.7	92.1	89.5	86.9	96.0	90.9	94.6
-w/o query	67.4	92.9	90.5	88.1	85.7	95.2	90.1	94.0
-w/o CQT	67.9	93.5	91.0	88.5	86.0	95.5	90.4	94.2
-w/o EQT	67.4	93.5	91.1	88.5	85.9	95.4	90.3	94.3
Table 8: The ablation results on the development set of REWRITE. "w/o query" means that we do not append a designed query before encoding that semantic information into our model. "w/o CQT" means that we only perform Ellipsis-oriented Query Template. "w/o EQT" means that we only perform Coreference-oriented Query Template and use the incomplete utterance as query if match fails.
A.6 Case Study
Table 9 gives 3 examples that indicate the representative situations as Hao et al. (2021). The first example illustrates the cases when RUN inserts unexpected characters into the wrong places. T-Ptr-Gen just copies the incomplete utterance. Due to our generated query, the position that needs to be inserted has been explicitly promoted by the query. The second example shows a common situation for generation-based models. T-Ptr-Gen messes up by repeating stupidly. However, this situation doesn't happen to our model, as it is not a generation-based model. The last example refers to a long and complex entity. For these cases, it is easier for our model to get the correct span. This is because our model learns the span boundaries from the edit operation matrix. Compared to the generation-based model, we don't generate sentences from scratch and this reduces the difficulty. Meanwhile, our model is not based on CNN as RUN, which suffers from the limitation of receptive-field to find a longer span.
One limitation of current edit-based IUR models, is that only tokens that have appeared in the history dialogue can be selected. Therefore, these models, including ours, cannot generate novel words, e.g., conjunctions and prepositions, to cater to other metrics, like fluency. However, this can be alleviated by incorporating an additional word dictionary as See et al. (2017b) and Liu et al. (2020) deals with the out-of-vocabulary (OOV) words to improve fluency. For fairness, we keep the same words during the experiment as RUN to mitigate it. We will consider this question as a promising direction for future works.
Example # 1
Historical Utterance 1	我意见很大
I have a lot of complaints
Historical Utterance 2	有意见保留
Keep it yourself if there's any
Incomplete Utterance	不想保留
Don't want to keep it myself
Gold	不想保留意见
Don't want to keep the complaints myself
T-Ptr-Gen	不想保留
Don't want to keep it myself
RUN	意见不想意保留
Complaints don't want to keep the complaints myself
QUEEN	不想保留意见
Don't want to keep the complaints myself
Example # 2
Historical Utterance 1	你帮我考雅思
Please help me on IELTS
Historical Utterance 2	雅思第一项考什么
What is tested first for IELTS
Incomplete Utterance	考口语啊
It's oral test
Gold	雅思第一项考口语啊
It's oral test for IELTS
T-Ptr-Gen	考口语考口语啊
It's oral test oral test
RUN	雅思第一项考口语啊
It's oral test for IELTS
QUEEN	雅思第一项考口语啊
It's oral test for IELTS
Example # 3
Historical Utterance 1	帮我找一下西安到商洛的顺风车
Can you help me find a free ride from Xi’an to Shangluo
Historical Utterance 2	哪的
Where is it
Incomplete Utterance	能不能找到
Can you find any
Gold	能不能找到西安到商洛的顺风车
Can you find any free ride from Xi’an to Shangluo
T-Ptr-Gen	能不能找到商洛的顺风车
Can you find any free ride to Shangluo
RUN	能不能找到商洛的顺风车
Can you find any free ride to Shangluo
QUEEN	能不能找到西安到商洛的顺风车
Can you find any free ride from Xi’an to Shangluo
Table 9: Case Study