SlowGuess/NAACL_2022_OCR huggingface dataset:
Shumpei Inoue $^{1}$ , Tsungwei Liu $^{1}$ , Nguyen Hong Son $^{1}$ , Minh-Tien Nguyen $^{1,2,*}$ $^{1}$ Cinnamon AI, 10th floor, Geleximco building, 36 Hoang Cau, Dong Da, Hanoi, Vietnam. {sinoue, tsungwei.liu, levi, ryan.nguyen}@cinnamon.is  
 $^{2}$ Hung Yen University of Technology and Education, Hung Yen, Vietnam. tiennm@utehy.edu.vn
This paper introduces a model for incomplete utterance restoration (IUR) called JET (Joint learning token Extraction and Text generation). Different from prior studies that only work on extraction or abstraction datasets, we design a simple but effective model, working for both scenarios of IUR. Our design simulates the nature of IUR, where omitted tokens from the context contribute to restoration. From this, we construct a Picker that identifies the omitted tokens. To support the selector, we design two label creation methods (soft and hard labels), which can work in cases of no annotation data for the omitted tokens. The restoration is done by using a Generator with the help of the Picker on joint learning. Promising results on four benchmark datasets in extraction and abstraction scenarios show that our model is better than the pretrained T5 and non-generative language model methods in both rich and limited training data settings.<sup>1</sup>
Understanding conversational interactions through NLP has become important with increasing connectivity and range of capabilities. The applications using natural conversations cover a wide range of solutions including dialogue systems, information extraction, and summarization. For example, Adiwardana et al. (2020); Su et al. (2020) aimed to build the dialogue system where an intelligent virtual agent answers human conversations and makes suggestions in an open/closed domain. Bak and Oh (2018); Karan et al. (2021) attempted to detect decision-related utterances from multi-party meeting recordings, while Tarnpradab et al. (2017) applied extractive summarization for online forum discussions. These features allow users to
to quickly catch up with the current situation, decisions and next-action without having to follow a lengthy or comprehensive dialogue. However, utterances, the components of a conversation, are generally not self-contained and are difficult to understand by their own. This comes from the nature of multi-turn dialogue where each utterance contains co-references, rephrases, and ellipses (Figure 1). Su et al. 2019 also showed that co-references and ellipses occur in over  $70\%$  of utterances in conversations. This is a ubiquitous problem in conversational AI, making the challenge for building practical systems with conversations.
Incomplete Utterance Restoration (IUR) (Pan et al., 2019) is one solution to restore semantically underspecified utterances (i.e., incomplete utterances) in conversations. Figure 1 shows an example of IUR, in which the model rewrites the incomplete utterance to the reference. IUR is a challenging task due to two reasons. Firstly, the gold utterance (the reference) overlaps a lot of tokens with the pre-restored, incomplete utterance, while it overlaps only a few tokens with utterances in the context. We observed that for CANARD (Elgohary et al., 2019),  $85\%$  of tokens in incomplete utterances were directly cited for rewriting, while only  $17\%$  of tokens in context was cited for rewriting. Secondly, it is important to detect omitted tokens in incomplete utterances and to include them in the restoration process. In actual cases of IUR, no matter how fluent and grammatically correct the machine's generation is, it is useless as long as important tokens are left out.
Recent studies used several methods for IUR. It includes the extraction of omitted tokens for restoration (PAC) (Pan et al., 2019), two-stage learning (Song et al., 2020), seq2seq fine-tuning (Bao et al., 2021), semantic segmentation (RUNBERT) (Liu et al., 2020), or the tagger to detect which tokens in incomplete utterances should be kept, deleted or changed for restoration (SARG)
\*Corresponding Author.   
1The code is available at https://github.com/ umpei19/JET
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3149 - 3158 July 10-15, 2022 ©2022 Association for Computational Linguistics
Context	UI: Orson Welles
U2: Goodwill ambassador
U3: What is a Goodwill ambassador?
U4: The OCIAA sponsored cultural tours to Latin America and appointed goodwill ambassadors
U5: What did he do?
U6: In late November 1941, Welles was appointed as a goodwill ambassador to Latin America by Nelson Rockefeller,
U7: Was he good at it?
U8: Whitney wrote Welles, "Personally believe you would make great contribution to hemisphere solidarity with this project."
U9: What do you find interesting about the article?
U10: In addition to working on his ill-fated film project, It's All True, Welles was responsible for radio programs, lectures,
interviews and informal talks as part of his OCIAA-sponsored cultural mission
Incomplete Utterance	Did he have any other responsibilities?
Gold Utterance	In addition to radio programs, lectures, interviews and informal talks, did Orson Welles have any other responsibilities?
Tokens Defined
by Hard Labeling	["addition", "radio", "programs", "lectures", "interviews", "informal", "talks", "Orson", "Welles"]
Prediction	SARG	Did orson welles have any other responsibilities besides the all cultural?
Seq2Seq-tuning	Did orson welles have any other responsibilities other than being a goodwill ambassador to latin america?
T5	Did orson welles have any other responsibilities besides radio programs, lectures, interviews and informal talks as part
of his ociaa-sponsored cultural mission?
JET	Did orson welles have any other responsibilities besides radio programs, lectures, interviews and informal talks?
Figure 1: The sample data from CANARD. IUR models rewrite the incomplete utterance to be as similar as possible to the reference. The blue tokens are omitted tokens (excluding stop words) in the incomplete utterance. The red tokens are defined by our hard labeling approach as an important token.
(Huang et al., 2021). However, we argue that these methods can only work on neither extractive nor abstractive IUR datasets. For example, SARG and seq2seq achieve promising results on Restoration 200k (Pan et al., 2019) where omitted tokens can be directly extracted from the context (extraction). But they are not the best on CANARD (Elgohary et al., 2019), which requires more abstraction for restoration. In Figure 1, $^{2}$  we can observe that the output of SARG and seq2seq are worse than that of our JET. Text editing strategy by SARG is limited in its ability to generate abstractive rewriting while seq2seq has the problem in picking omitted tokens. As the result, the generality of these methods is still an open question.
We introduce a simple but effective model to deal with the generality of IUR methods named JET (Joint learning token Extraction and Text generation). The model is designed to work widely from extractive to abstractive scenarios. To do that, we first address the problem of identifying omitted tokens from the dialogue context by introducing a selector. The selector uses a new matching method for dealing with various forms of tokens (Figure 1) in the extraction style. We next consider the abstraction aspect of restoration by offering a generator. The generator utilizes the power of the pre-trained T5 model to rewrite incomplete utterances. The selector and generator share the T5's encoder and are jointly trained in a unified model for IUR. This paper makes three main contributions:
- We propose JET, a simple but effective model based on T5 for utterance restoration in multi-turn conversations. Our model jointly optimizes two tasks: picking important tokens (the selector) and generating re-written utterances (the generator). To our best knowledge, we are the first to utilize T5 for the IUR task.
- We design a method for identifying important tokens for training the selector. The method facilitates IUR models in actual cases, in which there are no (a few) existing gold labels.
- We demonstrate the validity of the model by comparing it to strong baselines from multiple perspectives such as limited data setting (Section 5.2), human evaluation (Section 5.4) and output observation (Section 5.5).
Sentence rewriting IUR can be considered to be similar to the sentence rewriting task (Xu and Veeramachaneni, 2021; Lin et al., 2021; Chen and Bansal, 2018; Cao et al., 2018). Recent studies have been addressed the IUR task with various sophisticated methods. For example, Pan et al. 2019 introduced a pick-then-combine model for IUR. The model picks up omitted tokens which are combined with incomplete utterances for restoration. Liu et al. 2020 proposed a semantic segmentation method that segments tokens in an edit matrix then applied an edit operation to generate utterances.
2The performance of RUN-BERT is limited on CANARD.
Figure 2: The proposed model (JET) for utterance restoration. The left is the selector and the right is the generator. The model jointly optimizes two tasks for doing restoration. The input format for our model is described in 3.2.1
Huang et al. 2021 presented a complicated model which uses a tagger for detecting kept, deleted, or changed tokens for restoration. We share the idea of using a tagger with Huang et al. 2021 for IUR. However, we design a more simple but effective model which includes a selector (picking omitted tokens) and a generator for the restoration of incomplete utterances.
Text generation IUR can be formulated as text generation by using the seq2seq model (Pan et al., 2019; Huang et al., 2021). For the generation, several well-known pre-trained models have been applied (Lewis et al., 2020; Brown et al., 2020; Raffel et al., 2020) with promising results. We employ the T5 model (Raffel et al., 2020) as the main component to rewrite utterances. To address the problem of missing important tokens in model's rewriting, we enhance T5 by introducing a Picker and two labeling methods (Section 3.2).
3 The Utterance Restoration Model
This work focuses on the incomplete utterance restoration of conversations. Let  $H = \{h_1, h_2, \dots, h_m\}$  be the history of the dialogue (context),  $U = \{u_1, u_2, \dots, u_n\}$  is the incomplete utterance that needs to be re-written. The task is to learn a mapping function  $f(H, U|\Theta) = R$ , where  $R = \{r_1, r_2, \dots, r_k\}$  is the re-written version of  $U$ . The learning of  $\Theta$  is composed by only using utterance generation (the generator) or the combination of two tasks: important token identification (the
picker) and utterance generation (the generator).
3.2 The Proposed Model
Our model is shown in Figure 2. The Picker receives the context to identify omitted tokens. The Generator receives incomplete utterances for restoration. The model jointly learns to optimize the two tasks. Our model distinguishes in three significant differences compared to PAC (Pan et al., 2019) and SARG (Huang et al., 2021). First, our model bases on a sing pre-trained model for both selector and generator while other models (i.e. PAC and SARG) use different architectures for the two steps. This makes two advantages for our model. (i) Our design can be easily adapted to create a new unified model for different tasks by using a single generative LM (Paolini et al., 2021). (ii) Our model can work well in several scenarios: extraction vs. abstraction (data characteristics) and full vs. limited training data (Section 5). Second, we design a joint training process to implicitly take into account the suggestion from the selector to the generator instead of using a two-step model as PAC which explicitly copes extracted tokens from the Pick for generation. Our joint training model can reduce the error accumulation compared to the two-step framework. Finally, we design a heuristic approach to build important tokens, which enable the model to work on a wider range of datasets and scenarios.
3.2.1 Input representation
As shown in Figure 2, we introduced three kinds of special tokens into the input text;  $[X1]$ ,  $[X2]$  and
$\langle \backslash s\rangle$  . [X1] and [X2] are our newly defined special tokens and  $<  \backslash s >$  is the EOS token in the T5's vocabulary. We inserted [X1] at the end of each utterance in the context, [X2] after the incomplete utterance and  $<  \backslash s>$  at the end of whole input. [X1] and [X2] convey two pieces of useful information to the model; the signal indicating the switch of speakers and the cue to distinguish whether the utterance is from context or incomplete utterance.
The embedding of each token in the entire input sequence  $S = \{w_{1}, w_{2}, \dots, w_{l}\}$  was obtained as  $x_{i} = WE(w_{i}) + PE(w_{i})$ . Here,  $WE$  is word embedding initialized from a pretrained model by using a wordpiece vocabulary.  $PE$  is relative position embedding representing the position of each token in the sequence. These embeddings were fed into the  $L$  stacked Encoder of T5;  $E^{l} = \text{EncoderBlock}(E^{l-1})$  where  $E^{0} = \{x_{1}, \dots, x_{l}\}$ .  $E^{L}$  is the contextual representation of the whole input used by Picker and Generator in next sections.
3.2.2 The Picker
It is possible to directly use T5 (Raffel et al., 2019) for IUR. However, we empower T5 with a Picker to implicitly take into account information from important tokens. The idea of selecting important tokens was derived from Pan et al. (2019), in which the authors suggested the use of important tokens contributes the performance of utterance restoration. We extend this idea by designing an end-to-end model which includes important token identification and generation, instead of using the two-step framework as Pan et al. (2019).
Given the context and the incomplete utterance, the Picker identifies tokens that are included in context utterances but omitted in the incomplete utterance. We call these tokens as important tokens. However, no important tokens are originally provided except for Restoration 200k in four datasets (please refer to Table 1). Besides, the form of important tokens could change after restoration such as from plural to singular or nouns to verbs (Figure 1). To overcome this issue, we introduce a label creation method that automatically identifies important tokens from the context for restoration.
Important token identification Since building a set of important tokens is time-consuming and important tokens are usually not defined in practical cases, we introduce a heuristic strategy to automatically construct important tokens. In the following processing, stop words in the context,
incomplete utterances, and gold references are removed in advance, assuming that stop words are the out of scope of important tokens. In addition, we applied lemmatization and stemming, the process of converting tokens to their base or root form, to alleviate the spelling variants.
First, we extracted tokens, called "clue tokens", that exist in gold but not in incomplete utterances. If some tokens in context are semantically similar to some of the clue tokens, we can naturally presume that these tokens in the context are cited as important tokens for the rewriting. Therefore, we performed scoring by the distance  $d_{ij}$  between the word representations of  $i$ -th token in context  $h_i$  and  $j$ -th clue tokens  $c_j$ ;  $d_{ij} = \text{cosine\_sim}(h_i, c_j)$  where cosine_sim() is the score of Cosine similarity. We used word representations of  $h_i$  and  $c_j$  from fastText (Bojanowski et al., 2017) trained on Wikipedia as a simple setting of our model.
According to the distance  $d_{ij}$ , we introduce two types of labels for the Picker, soft $_i$  as soft labels and hard $_i$  as hard labels.
$$
s o f t _ {i} = \max _ {j} d _ {i j}
$$
$$
h a r d _ {i} = \left\{ \begin{array}{l l} 1 & \max  _ {j} d _ {i j} = 1 \\ 0 & o t h e r w i s e \end{array} \right.
$$
Here, the max operation was applied based on the assumption that at most one clue token corresponds to a token in the context.
Intuitively, the soft label method takes into consideration the cases that could not be handled by lemmatization and stemming, such as paraphrasing by synonyms, and reflects them as the importance score in the range of 0 to 1. On the other hand, the hard label is either 0 or 1 where an important token is defined only when there is an exact matching between the context tokens and the clue tokens in the form after lemmatization and stemming. We provide the two methods to facilitate important token identification.
Important token selection The Picker takes encoded embeddings  $E^{L} = \{E_{1}^{L},\dots,E_{l}^{L}\}$  and predicts the scores of the soft label or hard label corresponding to each input token.
$$
p (y _ {i} | E _ {i} ^ {L}) = s o f t m a x (\mathrm {F N N} (E _ {i} ^ {L}))
$$
where FNN() is the vanilla feedforward neural network, which stands for projecting encoded embedding to the soft label or hard label space. Then
cross-entropy was adopted as the loss function.
$$
\mathcal {L} _ {\text {p i c k e r}} = - \sum_ {i = 1} ^ {l} q _ {i} \log p (y _ {i} | E _ {i} ^ {L})
$$
where  $q_{i}$  is the selector's label for the  $i$ -th input token. To optimize loss function  $\mathcal{L}_{\text{picker}}$  is equal to minimize the KL Divergence if the label is a soft label. In the hard labeling case, we assign three types of tags for tokens by following the BIO tag format as a sequence tagging problem.
3.2.3 The Generator
We explore the restoration task by using Text-to-Text Transfer Transformer (T5) (Raffel et al., 2019). This is because T5 provides promising results for the text generation task. We initialized transformer modules from T5-base, which uses 12 layers, and fine-tuned it for our IUR task.
For restoration, encoder's representation  $E^L$  was fed into a  $L$  stacked decoder with cross attention.  $D_i^l = \text{DecoderBlock}(D_i^{l-1}, E^L)$  where  $D_i^0 = R_{<i}$ , with  $R_{<i} = \{ <s >, r_1, \ldots, r_{i-1} \}$  and  $
$  is the SOS token. The probability  $p$  of a token  $t$  at the time step  $i$  was obtained by feeding the decoder's output  $D^L$  into the softmax layer.
$$
p (t \mid R _ {<   i}, H, U) = \operatorname {s o f t m a x} \left(\operatorname {l i n e a r} \left(D _ {i} ^ {L}\right)\right) \cdot v (t)
$$
Here,  $v(t)$  is a one-hot vector of a token  $t$  with the dimension of vocabulary size. The objective is to minimize the negative likelihood of conditional probability between the predicted outputs from the model and the gold sequence  $R = \{r_1, r_2, \dots, r_k\}$ .
$$
\mathcal {L} _ {\text {g e n e r a t o r}} = - \sum_ {i = 1} ^ {k} \log p (r _ {i} \mid R _ {<   i}, H, U)
$$
3.2.4 Joint learning
JET aims to optimize the Picker and the Generator jointly as a setting of Multi-Task Learning. Different from PAC (Pan et al., 2019) that directly copies extracted tokens to generation, JET can implicitly utilize knowledge from the Picker, in which the learned patterns of the Picker to identify important tokens can be leveraged by the Generator. It can reduce error accumulation in the two-step framework as PAC. The final loss of the proposed model is defined as follows.
$$
\mathcal {L} = \alpha \mathcal {L} _ {\text {p i c k e r}} + \mathcal {L} _ {\text {g e n e r a t o r}}
$$
where the hyperparameter  $\alpha$  balances the influence of the task-specific weight. Our simple setting enables us to implement minimal experiments to evaluate how much important token extraction makes the contribution to generation.
4 Settings and Evaluation Metrics
Data We conducted all experiments on four well-known datasets of utterance rewriting in Table 1.
Table 1: Four conversational datasets. ext is extraction and abs is abstraction. CN: Chinese; EN: English.
Data	train	dev	test	type	lang
Restoration 200k	194k	5k	5k	ext	CN
REWRITE	18k	0	2k	ext	CN
TASK	2.2k	0	2k	ext	EN
CANARD	32k	4k	6k	abs	EN
Restoration 200k (Pan et al., 2019) and REWRITE (Su et al., 2019) include Chinese conversations. TASK (Quan et al., 2019) and CANARD (Elghary et al., 2019) are in English, in which CANARD includes English questions from QuAC (Choi et al., 2018). The datasets range from extraction to abstraction challenging UIR models.
Settings For running JET, we used AdamW with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and a weight decay of 0.01 with a batch size of 12 and learning rate of  $5e^{-5}$ . We used 3 FFN layers (dimension as 768, 256, 64) with ReLu as the activation function. The final dimension is 1 for soft labeling and 3 for hard labeling. We set  $\alpha = 1$  for the loss function. We applied beam search with the beam size of 8. For selector's label creation, we used stop words from NLTK for English and from stopwords $^{3}$  for Chinese. For lemmatization and stemming, NLTK's WordNetLemmatizer and PorterStemmer were adopted for English, while lemmatization and stemming were skipped for Chinese. The pre-trained model was T5-base (English $^{4}$  and Chinese $^{5}$ ). In the full training data setting (Section 5.1), the epoch size of 6 was used for Restoration200k and CANARD and 20 for REWRITE and TASK. In the limited training data setting (Section 5.2), the epoch size of 20 was used for all four datasets (Table 1). All models were trained on a single Tesla P100 GPU.
3https://pypi.org/project/stopwordsiso/
4https://huggingface.co/t5-base
<sup>5</sup>https://huggingface.co/lemon234071/t5-base-Chinese
Table 2: The comparison of JET and T5. Bold numbers show statistically significant improvements with  $p \leq 0.05$ . Underline is comparable (applied to Tables 3 and 4). The results come from the hard label method.
Data	Method	ROUGE-1	ROUGE-2	BLEU-1	BLEU-2	f1	f2	f3
Restoration	T5-base	92.7	86.1	91.4	88.9	61.3	51.2	44.8
200k	JET	93.1	86.9	92.0	89.6	63.0	53.3	47.1
REWRITE	T5-base	95.5	90.3	92.8	90.5	89.0	82.1	77.2
JET	95.8	90.6	93.5	91.2	89.8	82.7	77.5
TASK	T5-base	95.8	91.7	93.9	92.6	76.2	71.2	68.1
JET	96.1	91.8	94.3	93.0	76.3	72.1	69.6
CANARD	T5-base	83.9	70.2	77.8	70.8	56.2	44.6	39.3
JET	84.3	71.1	78.8	72.0	57.3	45.9	40.7
Table 3: The comparison of JET and strong baselines; For Restoration 200k, results were derived from Liu et al. (2020) and Huang et al. (2021). For CANARD, we reproduced strong baselines which output promising results on Restoration 200k. The results of RUN-BERT on CANARD were derived from the code of Liu et al. (2020). The results of JET come from hard labels.
Data	Method	ROUGE-1	ROUGE-2	BLEU-1	BLEU-2	f1	f2	f3
Restoration 200k	Syntactic	89.3	80.6	84.1	81.2	47.9	38.8	32.5
CopyNet	89.0	80.9	84.7	81.7	50.3	41.1	34.9
T-Ptr	90.1	83.0	90.3	87.4	51.0	40.4	33.3
PAC	91.6	82.8	89.9	86.3	63.7	49.7	40.4
s2s-ft	91.4	85.0	90.8	88.3	56.8	46.4	39.8
RUN	91.0	82.8	91.1	88.0	60.3	47.7	39.3
RUN-BERT	RUN-BERT	92.4	85.1	92.3	89.6	68.6	56.0	47.7
SARG	92.1	86.0	92.2	89.6	62.4	52.5	46.3
JET	93.1	86.9	92.0	89.6	63.0	53.3	47.1
CANARD	s2s-ft	83.1	69.0	78.6	71.2	55.1	43.2	37.9
RUN-BERT	80.6	62.7	70.2	61.2	44.2	30.5	24.9
SARG	80.3	63.7	70.5	62.7	42.7	30.5	25.9
JET	84.3	71.1	78.8	72.0	57.3	45.9	40.7
Evaluation metrics We followed prior work (Pan et al., 2019; Elgohary et al., 2019; Liu et al., 2020; Huang et al., 2021) to use three different metrics for evaluation, including ROUGE-scores, BLUE-scores, and f-scores.
5.1 Full Training Data Setting
We provide two scenarios of comparison with full training data: comparison with T5 and comparison with non-generative LM models.
Comparison with T5 We first compare our model against a strong pre-trained T5 model used for the generator as the first scenario. This scenario ensures fair comparison among strong pre-trained models for text generation and also shows the contribution of the Picker. Results in Table 2 show that JET is consistently better than T5 across all metrics on all four datasets. This is because the