SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiangru Tang† Arjun Nair† Borui Wang† Bingyao Wang† Jai Desai† Aaron Wade† Haoran Li¶ Asli Celikyilmaz¶ Yashar Mehdad¶ Dragomir Radev†† Yale University ¶ Meta AI {xiangru.tang, arjun.nair, borui.wang, dragomir.radev}@yale.edu {aimeeli, aslic, mehdad}@fb.com
Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluation. In this work, we first devised a typology of factual errors to better understand the types of hallucinations generated by current models and conducted human evaluation on popular dialog summarization dataset. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called CONFiT. To tackle top factual errors from our annotation, we introduce additional contrastive loss with carefully designed hard negative samples and self-supervised dialogue-specific loss to capture the key information between speakers. We show that our model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarization. On both datasets, we achieve significant improvements over state-of-the-art baselines using both automatic metrics, ROUGE and BARTScore, and human evaluation.
Text summarization is used to generate a concise and accurate summary of a long text while focusing on the sections that convey the most useful information (Gurevych and Strube, 2004). In recent years, the resurgence of dialogue summarization has attracted significant research attentions (McCowan et al., 2005; Gliwa et al., 2019; Koay et al., 2020; Zhang et al., 2021; Zhong et al., 2021; Zhu et al., 2021; Chen et al., 2021a; Li et al., 2021; Chen et al., 2021c; Fabbri et al., 2021; Chen et al., 2021d). The goal of dialogue summarization is to condense the conversational input into brief sentences version but cover salient information (McCowan et al.,
2005; Yuan and Yu, 2020). Significant progress has been made recently on abstractive dialogue summarization with various pre-trained models. However, such pre-trained models are susceptible to generating hallucinate content that is not supported by the source documents (Cao et al., 2018; Maynez et al., 2020; Kryscinski et al., 2020). To tackle the issue of factual inconsistency in dialogue summarization, recent works correctly encode the names of speakers (Zhu et al., 2020), explicitly incorporate coreference information (Liu et al., 2021b), and order the personal named entities (Liu and Chen, 2021). But it is still challenging to improve the quality of summaries generated by different models and decrease the hallucination at the same time.
To better understand the types of hallucinations generated by the pre-trained models, we devised a linguistically motivated taxonomy of factual errors for dialogue summarization, instead of simply classifying the summary as faithful or not. Based on our typology, we defined an annotation protocol for factuality evaluation of dialogue summarization. We then conducted a human evaluation of several pre-trained abstractive summarizers, including BART (Lewis et al., 2020), Pegasus (Zhang et al., 2020), and T5 (Raffel et al., 2020), aiming at identifying the proportion of different types of factual errors and studying the weaknesses of the pre-trained models. Our typology and annotation helps us gain deeper insights into the causes of factual inconsistency. Unlike news summarization (Pagnoni et al., 2021), we found that the challenges posed by dialogue summarization are more related to dialogue flow modeling, informal interactions between speakers, and complex coreference resolution. Figure 1 shows a dialogue-summary pair with three specific errors.
In order to tackle the top factual errors produced by existing models, we propose to replace the most commonly used fine-tuning with a linguistically-informed contrastive fine-tuning approach. For
Human Language Technologies, pages 5657 - 5668
(a) Coreference Error
(b) Modality & Tense Error
(c) Missing Information
Figure 1: Sample summary of a SAMSum dialogue (Gliwa et al., 2019). The summary is generated by BART (Lewis et al., 2020). Errors are highlighted.
example, the reason for producing wrong reference errors is that models cannot understand the role in the dialogue, which goes beyond the events. Our goal is to drive the model to pay attention to the grounds of specific errors during the finetuning, and learn how to reduce the generation of such errors. To be more specific, CONFiT learns to distinguish whether there are factual errors in the summaries and capture the key information in the dialogue content, such as numbers and person names. Experiments on SAMSum (Gliwa et al., 2019) and AMI (McCowan et al., 2005) show the generalizability of CONFiT when it is applied to different pre-trained models and datasets. Furthermore, we employ both automatic evaluation and human evaluation on faithfulness and show that CONFiIT significantly reduces all different factual errors and generates summaries that are more factually consistent. Moreover, we analytically find that optimizing the contrastive fine-tuning is quite beneficial for improving the robustness of models, which brings further benefits.
- We introduce the first typology of factual errors for dialogue summarization and use it to conduct comprehensive annotation and focused analysis.
- Targeting different categories of factual errors in the annotations, we reduce occurrence of such errors generated by various pre-trained models with a novel linguistically-informed contrastive fine-tuning CONFiT approach.
- We validate our method on a widely used dia
logue summarization corpus, SAMSum, and extend it to a meeting summarization corpus AMI. Evaluations of output summaries on automatic metrics like ROUGE, BARTScore as well as human evaluations show that CONFiT outperforms baseline pre-trained models.
2 New Taxonomy of Factuality Errors for Abstractive Dialogue Summarization
In order to gain deeper insights into the types of factuality errors introduced by different abstractive dialogue summarization systems, we proposed a new taxonomy of factuality errors for abstractive dialogue summarization based on our empirical experiments and annotations of the performance of a set of representative baseline summarization models on the SAMSum dataset, which is a widely-used large-scale dialogue summarization dataset of chat message dialogues in English (see Section 4.1). Specifically, we generate summaries of SAMSum dialogues using state-of-the-art abstractive dialogue summarization models, including models fine-tuned based on T5 (Raffel et al., 2020), Pegasus (Zhang et al., 2020), BART (Lewis et al., 2020), D-HGN (Xiachong et al., 2021), and S-BART (Chen and Yang, 2021b). We then manually annotate all different types of errors in these generated summaries that are inconsistent with the source dialogue, compute detailed statistics of all these factuality errors, and then classify them into different categories. Based on our annotation and analysis, we propose a new taxonomy of errors with the majority focusing on factuality error, which includes the following 8 error types:
Category 1 - Missing Information: The content of the generated summary is incomplete compared to the reference.
[Reference Summary] Williams invites Ms. Blair for a coffee. They will go to her favourite coffee place near the square in a side alley at 2 p.m.
[Model-Generated Summary] Ms. Blair is going to a coffee place near the square in a side alley.
Category 2 - Redundant Information: There is redundant content in the generated summary compared to the reference.
[Reference Summary] Paula helped Charlotte with correct pronunciation of "Natal Lily."
[Model-Generated Summary] Charlotte asks Paula how to pronounce the name of the plant "Natal Lily." Paula confirms that the stress on the second syllable is 2nd.
Category 3 - Circumstantial Error: Circumstantial information (e.g., date, time, location) about the predicate doesn't match the reference.
[Reference Summary] The USA was founded in 1776.
[Model-Generated Summary] The USA was founded in 1767.
Category 4 - Wrong Reference Error: A pronoun is with an incorrect or nonexistent antecedent, or a personal named entity in the generated summary is in the place of a different personal entity in the reference.
[Reference Summary] Mohit asked Darlene about the test.
[Model-Generated Summary] Darlene asked Mohit about the test.
Category 5 - Negation Error: This encompasses factual errors resulting from missing or erroneous negation in the generated summary compared to the reference.
[Reference Summary] Justin likes books.
[Model-Generated Summary] Justin does not like books.
Category 6 - Object Error: This covers factual errors resulting from incorrect direct or indirect objects (for non-personal entities only; errors of this nature involving personal entities are designated as Wrong Reference Errors).
[Reference Summary] Tara raised her glass.
[Model-Generated Summary] Tara raised her spoon.
Category 7 - Tense Error: This encompasses factual errors resulting from discrepancies in grammatical tense between the generated summary and the reference.
[Reference Summary] The children will go to the library.
[Model-Generated Summary] The children went to the library.
Category 8 - Modality Error: This includes factual errors resulting from modal discrepancies, such getting words like "may", "should", "could" wrong, between the generated summary and the reference.
[Reference Summary] School may be cancelled today.
[Model-Generated Summary] School is cancelled today.
2.1 Annotation and Analysis
Using our proposed taxonomy of factuality errors, we compute the proportion of each type of factuality errors across different summarization models. We then investigate the model generation behavior that is indicative of errors, which guides the design of our proposed model.
We performed a human evaluation of four model outputs from 19 SAMSum dialogues in order to identify the limitations of abstractive summarization models in dialogue summarization tasks. The four models used in this human evaluation are two BART models with different random seeds
Figure 2: Percentage of error types in each model during preliminary human evaluation of 19 SAMSum dialogues.
(ROUGE-L 48 and 49) (Lewis et al., 2020), D-HGN (ROUGE-L 40) (Xiachong et al., 2021), and S-BART (ROUGE-L 48 (Chen and Yang, 2021b)). BART and S-BART are pre-trained models (PLM), and D-HGN is trained from scratch. Since we are focusing on the dialogue domain, most of the factual errors in the model summaries are related to coreference, anaphora, and other dialogue-specific characteristics. In fact, approximately  $45\%$  of all errors fall into the categories of Missing Information and Wrong Reference. The distribution of these errors throughout these pre-existing models informs the limitations of each model. Our proposed CONFiT model targets the top errors generated by the current state-of-the-art models to reduce factual inconsistency.
3 CONFIT Model
Standard fine-tuning parameterizes the probability  $p_{\alpha}$  of the generator on a task-specific labeled dataset by maximizing cross-entropy loss.
$$
\mathcal {L} = - \sum \log P \left(\tilde {t} _ {l} \mid t _ {<   l}, \mathbf {D}\right) \tag {1}
$$
However, the cross-entropy loss has several shortcomings that can lead to factual inconsistency in dialogue summarization due to its sub-optimal generalization and instability. We propose a more efficient fine-tuning method CONFiT for factual consistency driven by the intuition that good generalization requires capturing the similarity in one class and contrasting them in other classes. In CONFi T, we introduce two additional losses: contrastive loss and self-supervised loss. We use two weights, actually which is coefficients, to adjust the ratio of  $L_{con}$  and  $L_{self}$  in the total loss of CONFi T.
The final training objective  $\mathcal{J}(\theta)$  of the proposed framework is as follows:
$$
\mathcal {J} (\theta) = \mathcal {L} + \alpha \mathcal {L} _ {\text {c o n}} + \beta \mathcal {L} _ {\text {s e l f}} \tag {2}
$$
Our linguistically-informed typology and annotation help us gain deeper insights into the causes of different factual errors. To help our models generate more faithful summaries, the proposed CONFiT learns to concentrate on the essential elements of dialogue and capture the dynamic role information as illustrated in Figure 3.
Figure 3: A demonstration of our model.
3.1 Contrastive Loss
In order to reduce the occurrence of factual errors, we propose a contrastive loss that uses the following negative sample generation techniques to target each error type in our proposed taxonomy:
- Swap the nouns in the reference summary with each other randomly. This aims to reduce wrong reference and object errors by providing negative samples.
- Swap the verbs in the reference summary with each other randomly. This aims to the model reduce circumstance (and, to a lesser extent, tense and modality) errors.
- Mask numbers and years in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce circumstance errors.
- Randomly delete  $30\%$  of the sentences in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce missing information errors.
- Mask-and-fill coreferent entities with BART in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce wrong reference errors.
Equation 3 demonstrates our contrastive loss function. During the fine-tuning, we have the positive samples, which is the reference summaries and another set of incorrect summaries, which is the negative samples. The contrastive objectives are learning representations that are invariant to different views of positive pairs; while maximizing the distance between negative pairs (Gunel et al., 2020). Our goal is to maximize the likelihoods of the positive samples and minimize the likelihoods of the negative samples as well. We use the following contrastive learning objective
$$
\mathcal {L} _ {c o n} = - \sum_ {y _ {j} \neq y _ {i}} \log \frac {\exp \left(\cos \left(\boldsymbol {c} _ {i} , \boldsymbol {c} _ {j}\right)\right)}{\sum_ {y _ {k} \neq y _ {i}} \exp \left(\cos \left(\boldsymbol {c} _ {i} , \boldsymbol {c} _ {k}\right)\right)} \tag {3}
$$
where  $y_{i}$  and  $y_{j}$  are positive summary pairs generated by back translation technology and  $y_{k}$  is from negative set of examples and  $\pmb{c}_i,\pmb{c}_j,\pmb{c}_k$  are their BART encoder representations.
3.2 Self-supervised Loss
One unique challenge in abstractive dialogue summarization is the use of first-person pronouns (such as "I" or "we") in speaker utterances, which the model has to correctly identify as being a reference to the speaker. This can lead to wrong reference errors in the summary, as the model cannot understand which participant is speaking and thus cannot accurately resolve first-person references. To address this problem, we design a self-supervised loss that aims to determine whether two tokens belong to the same speaker. Based on these findings, we design a self-supervised loss to enable CONFIT to capture the dynamic roles in the dialogue.
After the BART encoder, the input dialogue is encoded into hidden vectors  $C$ . Here, we first randomly select  $k$  pairs of two tokens  $t_m$  and  $t_n$  from the input dialogue, with labels  $s_m$  and  $s_n$  denoting which speaker they are coming from. We also do the same for utterances. Given the concatenation of the encoder representation of dialogue,  $t_m$  and  $t_n$ , we use the following loss function to classify whether the two tokens or two utterances are from the same speaker.
$$
\mathcal {L} _ {s e l f} = - \sum_ {m = 1} ^ {k} \sum_ {n = 1} ^ {k} \log P (s _ {m} = s _ {n} | t _ {m}, t _ {n}, C) \quad (4)
$$
This supplementary loss function helps CONFiT keep track of speaker information, thus improving the faithfulness of its summaries for dialogues that contain several first-person references.
We evaluate our new model on the popular SAMSum dialogue summarization dataset. Then, we extend our model to meeting summarization with the AMI Meeting Corpus. SAMSum (Gliwa et al., 2019) is a recently proposed large-scale dialogue summarization dataset consisting of 16,369 chat message dialogues in English written by linguists, and each message dialogue is annotated with a multi-sentence summary written by language experts.  $75\%$  of the dialogues in the SAMSum dataset (Gliwa et al., 2019) are between two interlocutors, and the other  $25\%$  are among three or more interlocutors. The AMI Corpus is another well-known dialogue summarization dataset consisting of 137 multiparty meeting transcripts extracted from 100 hours of meeting recordings. Each meeting transcript in the dataset is also annotated with a generic abstractive summary. We use these two representative dialogue summarization datasets to empirically test our new model's abstractive summarization performance in the settings of both short conversation-style dialogues and long meeting-style dialogues. See Table 2 for detailed statistics of the two datasets.
In our experiment using SAMSum, we trained BART for 3 epochs with a learning rate of  $1e - 05$ , Pegasus for 20 epochs with a learning rate of  $1e - 04$ , and T5 for 20 epochs with a learning rate of  $1e - 05$ . In our experiment using AMI, we trained BART for 6,000 steps with a learning rate of  $1e - 05$ , Pegasus for 24,000 steps with a learning rate of  $1e - 05$ , and T5 for 20,000 steps with a learning rate of  $1e - 05$ .
To evaluate our model, we use three metrics:
ROUGE (Lin, 2004): ROUGE measures Ngram overlap between the reference and the automatically generated summaries.
BARTScore (Yuan et al., 2021): Because ROUGE scores only measure token overlap, other automated metrics (Rebuffel et al., 2021; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al.,
Model	AMI	SAMSum
R-1	R-2	R-L	R-1	R-2	R-L
Extractive and Abstractive Models
TextRank (Mihalcea and Tarau, 2004)	35.19*	6.13*	15.70*	29.27*	8.02*	28.78*
Fast Abs RL (Chen and Bansal, 2018)	38.76	15.13	35.18	40.96	17.18	39.05
PGN (See et al., 2017)	48.34*	16.02*	23.49*	40.08*	15.28*	36.63*
\( \text{PGN}(\mathcal{D}_{\text{ALL}}) \) (Feng et al., 2021b)	50.91*	17.75*	24.59*	-	-	-
Pre-trained Models
T5 (Raffel et al., 2020)	42.16	13.94	39.39	48.41	24.79	44.61
Pegasus (Zhang et al., 2020)	46.02	15.85	43.73	48.04	22.94	43.40
BART (Lewis et al., 2020)	47.92	16.00	45.36	51.74	26.46	48.72
Multi-view BART (Chen and Yang, 2020)	-	-	-	49.52	26.52	48.29
Ours
T5-ConFiT	47.18	13.19	43.55	52.13	27.12	47.62
Pegasus-ConFiT	48.47	17.61	45.75	52.65	28.21	48.15
BART-ConFiT	50.31	17.29	47.98	53.89	28.85	49.29
Table 1: Dialogue summarization ROUGE evaluation on the AMI (McCowan et al., 2005) and SAMSum (Gliwa et al., 2019) datasets. We adopt some results reported from the literature (Feng et al., 2021a) and implement the pre-trained models for a fair comparison. All results marked with an asterisk (*) are from Feng et al. (2021b).
Dialogue	Speakers	Turns	Length
SAMSum
Train	14732	2.40	11.17	23.44
Validation	818	2.39	10.83	23.42
Test	819	2.36	11.25	23.12
AMI
All	137	4	289	322
Table 2: Details about SAMSum and AMI.
2021) have been proposed to evaluate faithfulness more precisely. BARTScore is a transformer-based measure that scores a dialogue and the corresponding automatically generated summary and has been shown to be strongly correlated with human evaluations of faithfulness (Yuan et al., 2021).
Human Evaluation: Finally, we conduct human evaluations on 100 SAMSum (Gliwa et al., 2019) and 20 AMI (McCowan et al., 2005) dialogues. Tang et al. (2021) found that Likert scales are a more consistent measure of factuality for abstractive dialogue summarization than Best-Worst Scaling. We have human evaluators directly rate the summaries on a scale from 1 to 10 corresponding to their faithfulness. In addition, using the error taxonomy proposed in Section 2, we have them mark whether each error type appeared in the given summary. We do this in a blinded fashion, so that the annotators do not see the corresponding model of the summary. Additionally, in order to prevent model information from leaking to the annotators, we randomly shuffle outputs within each dialogue before assigning them to annotators.
Table 1 shows the ROUGE scores of our models, the baseline models they were fine-tuned from, and a number of other abstractive summarization models on the SAMSum and AMI datasets. Tables 5 and 6 show the average human faithfulness and BART scores respectively for each model's outputs on 100 SAMSum and 20 AMI dialogues.
We observe that for all three pretrained models CONFIT significantly beat baselines on ROUGE-1, ROUGE-L, and human faithfulness score for both datasets. For BARTScore, we note that, while performance increases on SAMSum for all models, it decreases on AMI. However, given the fact that human evaluators rated the outputs of all three CONFIT models as more faithful than those of their corresponding baselines on both datasets, the decreases in BARTScore on AMI can likely be attributed to the imperfection of automated metrics at capturing faithfulness in text.
Tables 3 and 4 show the percentage of summaries that were labeled with each error type in our taxonomy of factual errors (discussed in Section 2.) for both the baseline and CONFiT models on the SAMSum and AMI datasets respectively.
We observe that on SAMSum, our fine-tuning method greatly reduces missing information, redundant information, wrong reference, and circumstance errors for all models. The largest reduction is on the "wrong reference" error type (20%, 7%, and 33% for BART, Pegasus, and T5 respectively),
Error Type	BART	BART-ConFiT	Pegasus	Pegasus-ConFiT	T5	T5-ConFiT
Missing Information	55%	44%	56%	50%	63%	48%
Redundant Information	12%	7%	7%	4%	7%	4%
Wrong Reference	37%	17%	25%	18%	46%	13%
Circumstance	14%	8%	16%	10%	8%	9%
Negation	4%	1%	7%	2%	1%	1%
Object	10%	6%	4%	7%	2%	7%
Tense	2%	1%	3%	1%	2%	2%
Modality	6%	1%	3%	5%	5%	8%
Table 3: Percentage of autogenerated summaries containing each error type, according to our human evaluation of model outputs from 100 SAMSum dialogues. Note that a single summary can contain multiple error types, so they do not add up to  $100\%$ .
Error Type	BART	BART-ConFiT	Pegasus	Pegasus-ConFiT	T5	T5-ConFiT
Missing Information	90%	85%	80%	70%	80%	85%
Redundant Information	10%	15%	60%	25%	0%	25%
Wrong Reference	35%	30%	35%	30%	50%	50%
Circumstance	35%	35%	30%	30%	40%	35%
Negation	20%	15%	5%	15%	25%	0%
Object	45%	40%	45%	25%	55%	55%
Tense	10%	10%	0%	5%	10%	10%
Modality	10%	15%	5%	5%	20%	10%
Table 4: Percentage of autogenerated summaries containing each error type, according to our human evaluation of model outputs from 20 AMI dialogues. Note that a single summary can contain multiple error types, so they do not add up to  $100\%$ .
Faithfulness Score	SAMSum	AMI
BART	5.540	4.850
BART-ConFiT	7.250	5.600
Pegasus	6.260	5.250
Pegasus-ConFiT	6.770	5.895
T5	5.422	4.150
T5-ConFiT	6.920	4.950
Table 5: Average faithfulness score (on a scale of 1-10) given to each model by human evaluators on 100 SAMSum and 20 AMI dialogues. Highest scores for each dataset have been bolded.
BARTScore	SAMSum	AMI
BART	-1.613	-3.644
BART-ConFiT	-1.468	-3.669
Pegasus	-1.615	-2.967
Pegasus-ConFiT	-1.608	-3.369
T5	-1.993	-3.406
T5-ConFiT	-1.677	-3.798
Table 6: Average BARTScore for each model on 100 SAMSum and 20 AMI dialogues. Highest scores for each dataset have been bolded.
likely owing to the self-supervised loss function introduced in Section 3.2 that was designed to help the model more effectively capture speaker information. For AMI, however, our fine-tuning method is not as consistent at reducing the frequency of
each error type across models. It is possible that this is due to sample size (20 AMI dialogues vs. 100 SAMSum dialogues).
5.2 Case Study