SlowGuess/NAACL_2022_OCR huggingface dataset:
Yue Fang  $^{1}$ , Hainan Zhang  $^{2}$ , Hongshen Chen  $^{2}$ , Zhuoye Ding  $^{2}$ , Bo Long  $^{2}$ , Yanyan Lan  $^{2}$  and Yanquan Zhou
<sup>1</sup>Beijing University of Posts and Telecommunications, Beijing, China  
<sup>2</sup>JD.com, Beijing, China
{fangyue, zhouyanquan}@bupt.edu.com, zhanghainan1990@163.com, ac@chenhongshen.com, lanyanyan@ict.ac.cn,{dingzhuoye, bo.long}@jd.com
Due to the dialogue characteristics of unstructured contexts and multi-parties with first-person perspective, many successful text summarization works have failed when dealing with dialogue summarization. In dialogue summarization task, the input dialogue is usually spoken style with ellipsis and co-references but the output summaries are more formal and complete. Therefore, the dialogue summarization model should be able to complete the ellipsis content and co-reference information and then produce a suitable summary accordingly. However, the current state-of-the-art models pay more attention on the topic or structure of summary, rather than the consistency of dialogue summary with its input dialogue context, which may suffer from the personal and logical inconsistency problem. In this paper, we propose a new model, named ReWriteSum, to tackle this problem. Firstly, an utterance rewriter is conducted to complete the ellipsis content of dialogue content and then obtain the rewriting utterances. Then, the co-reference data augmentation mechanism is utilized to replace the referential person name with its specific name to enhance the personal information. Finally, the rewriting utterances and the co-reference replacement data are used in the standard BART model. Experimental results on both SAMSum and DialSum datasets show that our ReWriteSum significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on multi-speakers also shows that ReWriteSum can obtain relatively higher improvement with more speakers, validating the correctness and property of ReWriteSum.
Despite many existing text summarization works on single-speaker written documents, such as news and encyclopedia articles (Rush et al., 2015;
Example one
Ann: Hi, is the laptop still available?
Josh: Yes it is.
Ann: I can pay 200 dollars.
Josh: The price is 250 and it's non-negotiable.
Ann: Do you have a bag for it? Some other accessories?
Josh: I have a bag and a small usb mouse.
Ann: Sounds good, I'll take it, where can I pick it up?	Ground-Truth:
Ann wants to buy Josh's laptop for $200. Josh doesn't want to negotiate the price. Ann will take it for $250 with accessories.
BART Prediction:
Ann will pay 200 dollars for the laptop and the price is non-negotiable. Ann will pick a bag from Josh.
Example two
Mike: Dude, Wendy has grown prettier.
Dave: I know right?
Mike: Yeah, since she came from Houston, she looks like an angel.
Dave: I'll have to hit on her soon.
Mike: Haha, stay off, I hear Jerry is her lover.
Dave: Since when?
Mike: Haha, I don't know, but you can push your luck.
Dave: Haha, I will.	Ground-Truth:
Mike and Dave notice Wendy got prettier. Dave wants to hit on her, but she's with Jerry. he'll try anyway.
BART Prediction:
Wendy has grown prettier since she came from Houston.
Mike will have to hit on her soon. Jerry is Wendy's lover.
Table 1: Two personal and logical inconsistent examples from the state-of-the-art model in dialogue summarization. Green words in ground-truth indicate the dialogue facts. Red words in BART show the inconsistent content, results from ellipsis and co-reference.
Gehrmann et al., 2018), dialogue summarization has gain increasing attention (Zhang et al., 2021). One reason is that it has various promising applications in real world, such as customer services and doctor-patient interaction. More importantly, the dialogue summarization process is more difficult since there are more interactive participants with first-person perspective, and unstructured context to consider (Chen and Yang, 2021), which poses great challenges for researchers in this area.
For this task, it is clear that there is a big gap between the input spoken dialogue and the output formal summaries. That is, in dialogue, users tend to use many incomplete utterances, which al
*Work done during internship at JD.com.
Human Language Technologies, pages 3859 - 3869
ways omit or refer back to entities appeared in the history, called ellipsis and co-reference. But the summary is usually formal and written, which contains rich and complete salient information. Here we give two examples, as shown in Table 1. In the first example, the incomplete utterance "I will take it" omits "laptop" which can be seen in the first sentence, while the ground-truth summary contains the complete information "Ann will take it for $250 with accessories". We can see that the generated summary by BART confuses the accessories "bag" with the subject "laptop" and then generate a logic inconsistent summary "pick a bag". And in the second example, many people's names are in the contexts, which are more difficult for the summarization model to distinguish the co-reference relationship, i.e., "I'll have to hit on her" refers to "Dave" via "T". As a result, BART confuses "Mike" with "Dave", and then generates a personal inconsistent summary "Mike will have to hit on her". What's more, such factual inconsistencies have also been observed in previous studies (Cao et al., 2018; Krysciński et al., 2019, 2020). Therefore, it is critical to complete the omission and co-reference information in dialogue utterances for dialogue summarization task.
However, the current models pay more attention on introducing intrinsic information, such as dialogue acts (Goo and Chen, 2018), key point sequence (Liu et al., 2019a) and co-reference information (Liu et al., 2021b). They demonstrate that the introduction of intrinsic information and human annotation is effective in improving the quality of summary generation. However, dialogue acts and key point sequence require a lot of human effort, so they can not be widely used in applications. The coreference chain is integrated by GNN, which only pays attention to the referencing information of entities but not supplement and restore the referred and omitted pronouns in the dialogue utterances, resulting in the misunderstanding of omitted contents. More importantly, they all ignore the consistency between the dialogue summary and its source dialogue, which may lead to the personal and logical inconsistency problem caused by multi-speakers.
In this paper, we propose a new model, namely ReWriteSum, to tackle this problem. The core idea is to use the utterance rewriting mechanism to complete the omitted content and utilize the data augmentation strategy to enhance the co-reference information. Specifically, we first use the utter
ance rewriter to complete the ellipsis content in dialogue contexts, and then obtain the rewritten utterances dataset. Then, we use the co-reference data augmentation mechanism to replace the referential person name with its specific name with a certain probability to enhance the personal information. Finally, we use both the rewritten utterances and the co-reference replacement data as input, and utilize the state-of-the-art model BART to generate the corresponding summary.
In our experiments, we use two public datasets to evaluate our proposed models, i.e. SAMSum and DialSum. The results show that ReWriteSum has the ability to produce more consistent and suitable summary than traditional summarization models. Besides, we conduct an analysis on multi-speakers, and the results show that the ReWriteSum obtains relatively higher improvement with more speakers, which indicates that the incomplete utterance rewriting and co-reference data augmentation mechanism by our model are reasonable.
2.1 Document Summarization
The aim of automatic document summarization is to convert a well-structured document into short text containing salient information. It has received widespread attention in recent literature, especially abstractive document summarization. For example, Rush et al. (2015) introduce an attention-based sequence-to-sequence model for abstractive document summarization. To solve out-of-vocabulary and content repeat issues, See et al. (2017) propose a pointer-generator network with copy and coverage mechanism. Chen and Bansal (2018) leverage reinforcement learning to extract salient sentences in document and then generate summary. Recent studies have focused on the pre-trained models. Liu and Lapata (2019) take use of pre-trained language model BERT (Kenton and Toutanova, 2019) in extractive summarization and abstractive summarization. Lewis et al. (2020) propose BART which combined bi-directional encoder from BERT and auto-regressive decoder from GPT (Radford et al., 2018) to obtain the results of language generation.
2.2 Dialogue Summarization
Compared with document summarization, dialogue summarization aims at generating condensed text from the dialogue contexts among multiple speakers. For instance, Shang et al. (2018) propose an
unsupervised multi-sentence compression method to generate meeting summaries. Zhao et al. (2019) employ a hierarchical encoder and a reinforced decoder based on sequence-to-sequence model to generate meeting summaries.
Some studies have focused on employing conversational analysis for dialogue summarization. Goo and Chen (2018) use sentence-gated mechanism to apply dialogue act in the generation process. Liu et al. (2019a) design a key point sequence as auxiliary information to describe the logic of the abstract. Liu et al. (2019c) and Li et al. (2019) introduce topic information for dialogue summarization. However, their methods need a large amount of human annotation. To avoid this issue, Chen and Yang (2020) use diverse conversational structures like topic segments and conversational stages to design a multi-view summarizer. Recent works often introduce intrinsic information to better model the dialogue process. Liu et al. (2021b) use the graph neural network to employ co-reference information to generate summaries. Feng et al. (2020) introduce the dialogue discourse information, and design Meeting Graph to describe them. Lei et al. (2021) introduce speaker information to improve the generation performance in the context with multi-speakers.
2.3 Incomplete Utterance Rewriting
Incomplete utterance rewriting has received extensive research attention. In question answering, Kumar and Joshi (2016) propose non-sentential utterance resolution based on sequence-to-sequence model for utterance rewriting. To resolve incomplete follow-up questions, retrieval-based sequence-to-sequence model (Kumar and Joshi, 2017) and copy-based sequence-to-sequence model (Elgohary et al., 2019; Quan et al., 2019) are proposed, which can generate complete questions. Liu et al. (2019b) take use of question structures to rewrite utterance in conversational semantic parsing. Pan et al. (2019) leverage BERT to select words, and use these words to generate rewritten utterance. Su et al. (2019) distinguish the weights of context utterances for utterance rewriting. Liu et al. (2020) employ edit-based text generation and semantic similarity measurement for utterance rewriting.
In this section, we will describe our ReWriteSum model in detail, with architecture shown in Figure 1.
Figure 1: The model architecture of ReWriteSum. the left shows the co-reference data augmentation module and the right shows the incomplete utterance rewriting module.
ReWriteSum consists of an incomplete utterance rewriter, a co-reference data augmentation and a transformer-based BART summarization model.
For incomplete utterance rewriting, we establish a word-level edit matrix (Liu et al., 2020), whose element determines three editing operations: substitute, insert and none. To obtain the edit matrix, we conduct three neural networks, as shown in Figure 2: a BiLSTM-based context layer to obtain the word representation, an encoder layer to model the local information, and a segmentation layer to model the global information.
For co-reference data augmentation, we replace the co-reference word with its specific name entity through a co-reference resolution model (Joshi et al., 2020) and then augment the dataset with these replacement data. Specifically, we firstly use the co-reference resolution model to obtain the co-reference chain of the entire dialogue, then replace the pronoun in the co-reference chain with the specific name entity based on a certain probability. Finally, we utilize these replacement data to augment the personal information for dialogue summarization task.
Given the dialogue content set  $D = \{u_{1},\ldots ,u_{|D|}\} \in \mathbb{D}$ , each utterance in  $D$  is represented as  $u_{j} = \{x_{1}^{(j)},\dots,x_{L}^{(j)}\}$ , where  $x_{k}^{(j)}$  represents the  $k^{th}$  word in utterance  $u_{j}$ . The corresponding summary of  $D$  is represented as
Figure 2: Architecture of incomplete utterance rewriter.
$S = \{y_{1},\ldots ,y_{L_{S}}\}$  , where  $y_{j}$  represents the  $j^{th}$  word in summary  $S$
We adopt a neural model for abstractive dialogue summarization. In detail, given the dialogue  $D$  as input, we firstly utilize the utterance rewriting system and the co-reference resolution system to generate the new complete rewriting dialogue  $D'$ . And then, we use the rewriting dialogue  $D'$  as input, instead of dialogue  $D$ , to generate the dialogue summary.
3.2 Incomplete Utterance Rewriting
Given the whole dialogue  $D = \{u_{1},\ldots ,u_{|D|}\}$ , we define the context as  $C = \{u_{1},\dots,u_{t - 1}\}$  and the incomplete utterance as  $u_{t}(t\leq |D|)$ . Incomplete utterance rewriting aims at rewriting  $u_{t}$  to  $u_{t}^{*}$  through the context  $C$ . After rewriting,  $u_{t}^{*}$  should not only have the same meaning as  $u_{t}$ , but also can be understood separately. Specifically, we concatenate all the contextual utterances  $C$  into a K-length word sequence  $\mathbf{c} = (c_1,\dots c_K)$ . At the same time, the incomplete utterance is represented as  $u_{t} = \{x_{1},\dots x_{L}\}$ , where  $L$  is the length of  $u_{t}$ . And then, the rewritten utterance  $u_{t}^{*}$  can be obtained by editing the incomplete dialogue  $u_{t}$  using the words in  $\mathbf{c}$ .
In order to determine the editing operation, we define a word-level edit matrix  $\mathbf{M}$  (Liu et al., 2020), where each element  $m_{kl}$  represents the editing type between  $c_k$  and  $x_l$ . There are three editing types: substitute, insert and None. The substitute operation means replacing the word  $x_l$  with the context word  $c_k$ . The insert operation means inserting a word  $c_k$  before or after a certain token  $x_l$ . And None means no operation. Following Liu et al. (2020), we establish a word-level edit matrix through three neural layers: a context layer, an en
coding layer and a subsequent segmentation layer, as shown in Figure 2, and then generate rewritten utterance based on this word-level edit matrix.
3.2.1 Context Layer
Given the contextual word sequence  $\mathbf{c}$  and the incomplete utterance  $u_{t}$ , we firstly concatenate the  $\mathbf{c}$  and  $u_{t}$  as input, and employ Glove (Pennington et al., 2014) to initialize the word embedding. And then, we use BiLSTM (Schuster and Paliwal, 1997) with both the left-to-right and right-to-left text representations to obtain the contextual information:
$$
B i L S T M (\mathbf {c}; u _ {t}) = \left(\mathbf {g} _ {1, \dots , K}; \mathbf {h} _ {1, \dots , L}\right),
$$
where  $\mathbf{g}_k$  is the hidden state of contextual word  $c_k$  in  $\mathbf{c}$  and  $\mathbf{h}_l$  is the hidden state of the word  $x_l$  in  $u_t$ .
3.2.2 Encoding Layer
After obtaining the context-aware hidden states  $\mathbf{g}$  and  $\mathbf{h}$ , we use three similarity functions to calculate the word-level relevance between context and incomplete utterance. Specifically, for each word  $c_{k}$  and  $x_{l}$ , a D-dimensional vector  $\mathbf{F}(x_l,c_k)$  is set to indicate the relevance:
$$
\mathbf {F} \left(x _ {l}, c _ {k}\right) = \left[ \mathbf {h} _ {l} \odot \mathbf {g} _ {k}; \cos \left(\mathbf {h} _ {l}, \mathbf {g} _ {k}\right); \mathbf {h} _ {l} \mathbf {W} _ {B i} \mathbf {g} _ {k} \right], \tag {1}
$$
where  $\odot$  is the element multiplication operation to obtain the element-wise similarity,  $\cos (.,.)$  is the cosine similarity, and  $\mathbf{W}_{Bi}$  is a learned parameter in learned bi-linear similarity. Finally, we obtain the feature map matrix  $\mathbf{F} \in \mathbb{R}^{L \times K \times D}$ .
Similarity function is used to describe word-to-word relevance from various aspects, which is a necessary condition for the edit type. However, the encoder layer can only obtain local information, which is not enough for incomplete utterance rewriting. Therefore, we conduct a segmentation layer to introduce the global information.
3.2.3 Segmentation Layer
Given the feature map matrix  $\mathbf{F} \in \mathbb{R}^{L \times K \times D}$  in Equation 1, we use the segmentation layer to calculate the word-level edit matrix  $\mathbf{M} \in \mathbb{R}^{L \times K}$ . The segmentation layer is inspired by UNet (Ronneberger et al., 2015), consisting of five convolutional neural network(CNN) with skip-connection mechanism, which is used to extract the global con
textual editing information, as shown in Figure 2:
$$
\mathbf {F} ^ {\prime} = \operatorname {C N N} (\mathbf {F}),
$$
$$
\mathbf {F} ^ {\prime \prime} = \operatorname {C N N} \left(\operatorname {P o o l} \left(\mathbf {F} ^ {\prime}\right)\right),
$$
$$
\mathbf {F} ^ {\prime \prime \prime} = \operatorname {D e C o n v} (\operatorname {C N N} (\operatorname {P o o l} (\mathbf {F} ^ {\prime \prime}))),
$$
$$
\mathbf {F} ^ {\prime \prime \prime \prime} = \operatorname {D e C o n v} \left(\operatorname {C N N} \left(\mathbf {F} ^ {\prime \prime \prime}, \mathbf {F} ^ {\prime \prime}\right)\right),
$$
$$
\mathbf {M} = \operatorname {F e e d F o r w a r d} \left(\operatorname {C N N} \left(\mathbf {F} ^ {\prime \prime \prime \prime}, \mathbf {F} ^ {\prime}\right)\right),
$$
where  $\mathbf{CNN}(.)$  is the two layers of convolutional modules,  $\mathbf{Pool}(.)$  is the MaxPooling operation,  $\mathbf{DeConv}(.)$  is the deconvolution neural network, and  $\mathbf{FeedForward}(.)$  is the feedforward layer.
Given the word-level edit matrix  $\mathbf{M}$ , for each word  $c_{k}$  in contextual utterances and  $x_{l}$  in incomplete utterance, the element  $M_{kl}$  determines one of three editing operations: substitute, insert and none. Specifically, when  $M_{kl}$  is close to 0, the corresponding operation is none. When close to 1, the operation is substitution, 2 is inserting before and 3 is inserting after. After that, we can rewrite every utterance  $u_{t}$  in  $D$  as  $u_{t}^{*}$  based on  $M$ . Finally, we use all the rewritten utterances  $u^{*}$  to replace  $D$  as  $D^{\prime}$ , and obtain the rewriting dataset  $\mathbb{D}_{rew} = \{(D_1', S_1), (D_2', S_2), \ldots, (D_N', S_N)\}$ .
3.3 Co-reference Data Augmentation
Taking into account that there are a large number of names and referential relations in the dialogue process, we propose to use the data augmentation mechanism to enhance the personal information for dialogue summarization task.
Given a dialogue content  $D$ , we utilize a co-reference resolution system (Joshi et al., 2020) to obtain its corresponding co-referential chain set  $E = \{e_1, e_2, \ldots, e_{|E|}\}$ , where  $e_i = \{x_{i1}, x_{i2}, \ldots, x_{i|e_i|\}\}$  is represented as the  $i^{th}$  co-referential chain in dialogue  $D$  and  $x_{ij}$  denotes the  $j^{th}$  word in co-referential chain  $e_i$ . Take the example two in Table 1 as an example, the  $E = \{\{Mike_0, \ldots, I_{64}\}, \{Wendy_4, \ldots, her_{52}\}, \{Dave_9, \ldots, I_{79}\}, \{Jerry_{50}\}\}$ , where the word  $idx$  is the  $idx^{th}$  word in  $\mathbf{c}$ .
Then, we refer to all the pronouns in the whole dialogue  $D$  and replace it with its corresponding person name  $x_{name\_i}$  based on a certain probability: when the length of pronouns  $|e_i^{(pron)}| > = 5$ , if the output probability of co-reference system  $P(x_{ij}) > = 0.5$ , then replace  $x_{ij}$  with  $x_{name\_i}$ , otherwise, no replacement; when  $0 < |e_i^{(pron)}| < 5$ , if  $P(x_{ij}) > = 0.8$ , then replace; when  $|e_i^{(pron)}| = 0$ , remove this example.
Finally, after the person's name replacement, we obtain an additional dialogue dataset  $\mathbb{D}_{\text{aug}} = \{(D_1'', S_1), \ldots, (D_G'', S_G)\}$ , where  $G$  is the number of dialogue-summary pairs after removing.
3.4 Summary Generation
Given  $\mathbb{D}_{rew}$  and  $\mathbb{D}_{aug}$ , we combine them to obtain our rewriting dataset  $\mathbb{D}_{rws}$ . To generate the summary, we utilize the state-of-the-art model BART (Lewis et al., 2020) to encode the dialogue content  $D$  and decode the summary  $S$  step by step.
We use maximum likelihood estimation to train our model. Given a pair of dialogue  $D$  and summary  $S = \{y_{1},\dots ,y_{L_{S}}\}$  from  $\mathbb{D}_{rws}$ , we minimize the negative log-likelihood of the target sequence:
$$
\begin{array}{l} \mathcal {L} = \\ - \sum_ {\mathbb {D} _ {r w s}} \sum_ {t = 1} ^ {| L _ {S} |} \log \mathrm {P} \left(y _ {t} \mid y _ {1} \dots y _ {t - 1}, D; \theta_ {B A R T \_ l a r g e}\right). \\ \end{array}
$$
In this section, we conduct experiments on two English dialogue summarization datasets SAMSum (Gliwa et al., 2019) and DialSum (Chen et al., 2021) to evaluate our proposed method.
We first introduce some empirical settings, i.e., datasets, baselines, and evaluation measures.
We use two public dialogue summarization datasets. SAMSum contains everyday English message-like dialogues and annotated summary. We randomly split the SAMSum data to training, validation, and testing sets, which contains 14,732, 818 and 819 pairs, respectively. DialSum<sup>1</sup> contains English speaking practice dialogue and annotated summary, which has been cleaned and pre-processed by publisher, including deleting non-English characters, correcting spelling errors and grammatical errors. We randomly split the DialSum data to training, validation, and testing sets, which contains 12,460, 500 and 500 pairs, respectively.
4.1.2 Baselines and Parameters Setting
Seven baseline models are used for comparison on SAMSum, and four baseline models on DialSum. Lead3 (See et al., 2017) model extracts
$^{1}$ https://github.com/cylnlp/DialogSum
the first three leading sentences in the article as the summary. LONGEST (Gliwa et al., 2019) model selects the top N longest sentences as the summary. PTGen (See et al., 2017) model introduces copy and coverage mechanisms into the basic sequence-to-sequence model. FastAbs-RL (Chen and Bansal, 2018) model firstly selects salient sentences and then generates abstractive summaries through reinforcement learning. DynamicConv + GPT-2/News (Wu et al., 2018) model replaces the attention mechanism with a lightweight dynamic in transformer. BART (Lewis et al., 2020) is a pre-trained model, which uses the noise function to destroy text, and then reconstructs the original text, including two versions, BART(base) and BART(large). Multiview BART (Chen and Yang, 2020) extracts different views of dialogue features, and then uses a multi-view decoder to combine these features to generate summaries.
Our model uses a pre-trained model BART(large) $^2$  for initialization. In detail, BART (large) has 12 layers of encoder-decoder Transformer structure. Each layer has 16 attention heads. The hidden size and feed forward filter size are 1024 and 4096, respectively. It contains a total of 400M trainable parameters. The dropout rates for all layers are set to 0.1. The optimizer uses Adam (Kingma and Ba, 2015) with 200 warmup. The learning rates of SAMSum and DialSum are both 3e-5, and the maximum tokens for a certain batch are 800 and 1000, respectively. We run our models on a Tesla V100 GPU card with Pytorch.
4.1.3 Evaluation Measures