Venue: NAACL 2022

# ConFiT: Toward Faithful Dialogue Summarization with

Linguistically-Informed Contrastive Fine-tuning

Xiangru Tang\({}^{\dagger}\) Arjun Nair\({}^{\dagger}\) Borui Wang\({}^{\dagger}\) Bingyao Wang\({}^{\dagger}\) Jai Desati\({}^{\dagger}\) Aaron Wade\({}^{\dagger}\) Haoran Li\({}^{\ddagger}\) Asli Celikyilmaz\({}^{\ddagger}\) Yashar Mehdad\({}^{\ddagger}\) Dragomir Radev\({}^{\dagger}\)

\({}^{\dagger}\) Yale University \({}^{\dagger}\) Meta AI

{xiangru.tang, arjun.nair, borui.wang, dragomir.radev}@yle.edu

{aimeeli, aslic, mehdad}@fb.com

###### Abstract

Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluation. In this work, we first devised a typology of factual errors to better understand the types of hallucinations generated by current models and conducted human evaluation on popular dialog summarization dataset. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called ConFiT. To tackle top factual errors from our annotation, we introduce additional contrastive loss with carefully designed hard negative samples and self-supervised dialogue-specific loss to capture the key information between speakers. We show that our model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarization. On both datasets, we achieve significant improvements over state-of-the-art baselines using both automatic metrics, ROUGE and BARTScore, and human evaluation.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Text summarization is used to generate a concise and accurate summary of a long text while focusing on the sections that convey the most useful information Gurevych and Strube (2004). In recent years, the resurgence of dialogue summarization has attracted significant research attentions McCowan et al. (2005); Gliwa et al. (2019); Koay et al. (2020); Zhang et al. (2021); Zhong et al. (2021); Zhu et al. (2021); Chen et al. (2021a); Li et al. (2021); Chen et al. (2021c); Fabbri et al. (2021); Chen et al. (2021d). The goal of dialogue summarization is to condense the conversational input into brief sentences version but cover salient information McCowan et al. (2005); Yuan and Yu (2020). Significant progress has been made recently on abstractive dialogue summarization with various pre-trained models. However, such pre-trained models are susceptible to generating hallucinate content that is not supported by the source documents Cao et al. (2018); Maynez et al. (2020); Kryscinski et al. (2020). To tackle the issue of factual inconsistency in dialogue summarization, recent works correctly encode the names of speakers Zhu et al. (2020), explicitly incorporate coreference information Liu et al. (2021), and order the personal named entities Liu and Chen (2021). But it is still challenging to improve the quality of summaries generated by different models and decrease the hallucination at the same time.

To better understand the types of hallucinations generated by the pre-trained models, we devised a linguistically motivated taxonomy of factual errors for dialogue summarization, instead of simply classifying the summary as faithful or not. Based on our typology, we defined an annotation protocol for factuality evaluation of dialogue summarization. We then conducted a human evaluation of several pre-trained abstractive summarizers, including BART Lewis et al. (2020), Pegasus Zhang et al. (2020), and T5 Raffel et al. (2020), aiming at identifying the proportion of different types of factual errors and studying the weaknesses of the pre-trained models. Our typology and annotation helps us gain deeper insights into the causes of factual inconsistency. Unlike news summarization Pagnoni et al. (2021), we found that the challenges posed by dialogue summarization are more related to dialogue flow modeling, informal interactions between speakers, and complex coreference resolution. Figure 1 shows a dialogue-summary pair with three specific errors.

In order to tackle the top factual errors produced by existing models, we propose to replace the most commonly used fine-tuning with a linguistically-informed contrastive fine-tuning approach. For

example, the reason for producing wrong reference errors is that models cannot understand the role in the dialogue, which goes beyond the events. Our goal is to drive the model to pay attention to the grounds of specific errors during the fine-tuning, and learn how to reduce the generation of such errors. To be more specific, ConFiT learns to distinguish whether there are factual errors in the summaries and capture the key information in the dialogue content, such as numbers and person names. Experiments on SAMSum Gliwa et al. (2019) and AMI McCowan et al. (2005) show the generalizability of ConFiT when it is applied to different pre-trained models and datasets. Furthermore, we employ both automatic evaluation and human evaluation on faithfulness and show that ConFiT significantly reduces all different factual errors and generates summaries that are more factually consistent. Moreover, we analytically find that optimizing the contrastive fine-tuning is quite beneficial for improving the robustness of models, which brings further benefits.

Our contributions are as follows:

* We introduce the first typology of factual errors for dialogue summarization and use it to conduct comprehensive annotation and focused analysis.
* Targeting different categories of factual errors in the annotations, we reduce occurrence of such errors generated by various pre-trained models with a novel linguistically-informed contrastive fine-tuning ConFiT approach.
* We validate our method on a widely used dialogue summarization corpus, SAMSum, and extend it to a meeting summarization corpus AMI. Evaluations of output summaries on automatic metrics like ROUGE, BARTScore as well as human evaluations show that ConFiT outperforms baseline pre-trained models.

## 2 New Taxonomy of Factuality Errors for Abstractive Dialogue Summarization

In order to gain deeper insights into the types of factuality errors introduced by different abstractive dialogue summarization systems, we proposed a new taxonomy of factuality errors for abstractive dialogue summarization based on our empirical experiments and annotations of the performance of a set of representative baseline summarization models on the SAMSum dataset, which is a widely-used large-scale dialogue summarization dataset of chat message dialogues in English (see Section 4.1). Specifically, we generate summaries of SAMSum dialogues using state-of-the-art abstractive dialogue summarization models, including models fine-tuned based on T5 Raffel et al. (2020), Pegasus Zhang et al. (2020), BART Lewis et al. (2020), D-HGN Xiaichong et al. (2021), and S-BART Chen and Yang (2021). We then manually annotate all different types of errors in these generated summaries that are inconsistent with the source dialogue, compute detailed statistics of all these factuality errors, and then classify them into different categories. Based on our annotation and analysis, we propose a new taxonomy of errors with the majority focusing on factuality error, which includes the following 8 error types:

Figure 1: Sample summary of a SAMSum dialogue Gliwa et al. (2019). The summary is generated by BART Lewis et al. (2020). Errors are highlighted.

 

- Missing Information:** The content of the generated summary is incomplete compared to the reference.

**Example:**

[Reference Summary]_Williams invites Ms. Blair for a coffee. They will go to her favourite coffee place near the square in a side alley at 2 p.m._

[Model-Generated Summary]_Ms. Blair is going to a coffee place near the square in a side alley._

* Redundant Information:** There is redundant content in the generated summary compared to the reference.

**Example:**

[Reference Summary]_Paula helped Charlotte with correct pronunciation of "Natal Lily."_

[Model-Generated Summary]_Charlotte asks Paula how to pronounce the name of the plant "Natal Lily." Paula confirms that the stress on the second syllable is 2nd._

* Circumstantial Error:** Circumstantial information (e.g., date, time, location) about the predicate doesn't match the reference.

**Example:**

[Reference Summary]_The USA was founded in 1776._

[Model-Generated Summary]_The USA was founded in 1767._

* Wrong Reference Error:** A pronoun is with an incorrect or nonexistent antecedent, or a personal named entity in the generated summary is in the place of a different personal entity in the reference.

**Example:**

[Reference Summary]_Mohit asked Darlene about the test._

[Model-Generated Summary]_Darlene asked Mohit about the test._

* Negation Error:** This encompasses factual errors resulting from missing or erroneous negation in the generated summary compared to the reference.

**Example:**

[Reference Summary]_Justin likes books._

[Model-Generated Summary]_Justin does not like books._

* Object Error:** This covers factual errors resulting from incorrect direct or indirect objects (for non-personal entities only; errors of this nature involving personal entities are designated as Wrong Reference Errors).

**Example:**

[Reference Summary]_Tara raised her glass._

[Model-Generated Summary]_Tara raised her spoon._

* Tense Error:** This encompasses factual errors resulting from discrepancies in grammatical tense between the generated summary and the reference.

**Example:**

[Reference Summary]_The children will go to the library._

[Model-Generated Summary]_The children went to the library._

* Modality Error:** This includes factual errors resulting from modal discrepancies, such getting words like "may", "should", "could" wrong, between the generated summary and the reference.

**Example:**

[Reference Summary]_School may be cancelled today._

[Model-Generated Summary]_School is cancelled today._

### Annotation and Analysis

Using our proposed taxonomy of factuality errors, we compute the proportion of each type of factuality errors across different summarization models. We then investigate the model generation behavior that is indicative of errors, which guides the design of our proposed model.

We performed a human evaluation of four model outputs from 19 SAMSum dialogues in order to identify the limitations of abstractive summarization models in dialogue summarization tasks. The four models used in this human evaluation are two BART models with different random seeds 

(ROUGE-L 48 and 49) Lewis et al. (2020), D-HGN (ROUGE-L 40) Xiaichong et al. (2021), and S-BART (ROUGE-L 48 Chen and Yang (2021)). BART and S-BART are pre-trained models (PLM), and D-HGN is trained from scratch. Since we are focusing on the dialogue domain, most of the factual errors in the model summaries are related to coreference, anaphora, and other dialogue-specific characteristics. In fact, approximately 45% of all errors fall into the categories of Missing Information and Wrong Reference. The distribution of these errors throughout these pre-existing models informs the limitations of each model. Our proposed ConFiT model targets the top errors generated by the current state-of-the-art models to reduce factual inconsistency.

## 3ConFiT Model

Standard fine-tuning parameterizes the probability \(p_{\alpha}\) of the generator on a task-specific labeled dataset by maximizing cross-entropy loss.

\[\mathcal{L}=-\sum\log P(\tilde{t}_{l}|t_{<l},\mathbf{D}) \tag{1}\]

However, the cross-entropy loss has several shortcomings that can lead to actual inconsistency in dialogue summarization due to its sub-optimal generalization and instability. We propose a more efficient fine-tuning method ConFiT for factual consistency driven by the intuition that good generalization requires capturing the similarity in one class and contrasting them in other classes. In ConFiT, we introduce two additional losses: contrastive loss and self-supervised loss. We use two weights, actually which is coefficients, to adjust the ratio of \(L_{con}\) and \(L_{self}\) in the total loss of ConFit. The final training objective \(\mathcal{J}(\theta)\) of the proposed framework is as follows:

\[\mathcal{J}(\theta)=\mathcal{L}+\alpha\mathcal{L}_{\text{con}}+\beta\mathcal{ L}_{\text{self}} \tag{2}\]

Our linguistically-informed typology and annotation help us gain deeper insights into the causes of different factual errors. To help our models generate more faithful summaries, the proposed ConFiT learns to concentrate on the essential elements of dialogue and capture the dynamic role information as illustrated in Figure 3.

### Contrastive Loss

In order to reduce the occurrence of factual errors, we propose a contrastive loss that uses the following negative sample generation techniques to target each error type in our proposed taxonomy:

* Swap the nouns in the reference summary with each other randomly. This aims to reduce wrong reference and object errors by providing negative samples.
* Swap the verbs in the reference summary with each other randomly. This aims to the model reduce circumstance (and, to a lesser extent, tense and modality) errors.
* Mask numbers and years in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce circumstance errors.
* Randomly delete 30% of the sentences in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce missing information errors.

Figure 3: A demonstration of our model.

Figure 2: Percentage of error types in each model during preliminary human evaluation of 19 SAMSum dialogues.

 

* Mask-and-fill coreferent entities with BART in the dialogue and then pass it into the model to generate a negative sample summary. This aims to reduce wrong reference errors.

Equation 3 demonstrates our contrastive loss function. During the fine-tuning, we have the positive samples, which is the reference summaries and another set of incorrect summaries, which is the negative samples. The contrastive objectives are learning representations that are invariant to different views of positive pairs; while maximizing the distance between negative pairs Gunel et al. (2020). Our goal is to maximize the likelihoods of the positive samples and minimize the likelihoods of the negative samples as well. We use the following contrastive learning objective

\[{\cal L}_{con}=-\sum_{y_{j}\neq y_{i}}\log\frac{\exp(\cos(\mathbf{c}_{i},\mathbf{c}_{ j}))}{\sum\limits_{y_{k}\neq y_{i}}\exp(\cos(\mathbf{c}_{i},\mathbf{c}_{k}))} \tag{3}\]

where \(y_{i}\) and \(y_{j}\) are positive summary pairs generated by back translation technology and \(y_{k}\) is from negative set of examples and \(\mathbf{c}_{i}\) ,\(\mathbf{c}_{j}\), \(\mathbf{c}_{k}\) are their BART encoder representations.

### Self-supervised Loss

One unique challenge in abstractive dialogue summarization is the use of first-person pronouns (such as "I" or "we") in speaker utterances, which the model has to correctly identify as being a reference to the speaker. This can lead to wrong reference errors in the summary, as the model cannot understand which participant is speaking and thus cannot accurately resolve first-person references. To address this problem, we design a self-supervised loss that aims to determine whether two tokens belong to the same speaker. Based on these findings, we design a self-supervised loss to enable ConFiT to capture the dynamic roles in the dialogue.

After the BART encoder, the input dialogue is encoded into hidden vectors \(C\). Here, we first randomly select \(k\) pairs of two tokens \(t_{m}\) and \(t_{n}\) from the input dialogue, with labels \(s_{m}\) and \(s_{n}\) denoting which speaker they are coming from. We also do the same for utterances. Given the concatenation of the encoder representation of dialogue, \(t_{m}\) and \(t_{n}\), we use the following loss function to classify whether the two tokens or two utterances are from the same speaker.

\[{\cal L}_{self}=-\sum_{m=1}^{k}\sum_{n=1}^{k}\log P(s_{m}=s_{n}|t_{m},t_{n},C) \tag{4}\]

This supplementary loss function helps ConFiT keep track of speaker information, thus improving the faithfulness of its summaries for dialogues that contain several first-person references.

## 4 Experiments

### Dataset

We evaluate our new model on the popular SAMSum dialogue summarization dataset. Then, we extend our model to meeting summarization with the AMI Meeting Corpus. SAMSum Gliwa et al. (2019) is a recently proposed large-scale dialogue summarization dataset consisting of 16,369 chat message dialogues in English written by linguists, and each message dialogue is annotated with a multi-sentence summary written by language experts. 75% of the dialogues in the SAMSum dataset Gliwa et al. (2019) are between two interlocutors, and the other 25% are among three or more interlocutors. The AMI Corpus is another well-known dialogue summarization dataset consisting of 137 multiparty meeting transcripts extracted from 100 hours of meeting recordings. Each meeting transcript in the dataset is also annotated with a generic abstractive summary. We use these two representative dialogue summarization datasets to empirically test our new model's abstractive summarization performance in the settings of both short conversation-style dialogues and long meeting-style dialogues. See Table 2 for detailed statistics of the two datasets.

### Experiment Settings

In our experiment using SAMSum, we trained BART for 3 epochs with a learning rate of \(1e-05\), Pegasus for 20 epochs with a learning rate of \(1e-04\), and T5 for 20 epochs with a learning rate of \(1e-05\). In our experiment using AMI, we trained BART for 6,000 steps with a learning rate of \(1e-05\), Pegasus for 24,000 steps with a learning rate of \(1e-05\), and T5 for 20,000 steps with a learning rate of \(1e-05\).

### Evaluation Metrics

To evaluate our model, we use three metrics:

**ROUGE**Lin (2004): ROUGE measures N-gram overlap between the reference and the automatically generated summaries.

**BARTScore**Yuan et al. (2021): Because ROUGE scores only measure token overlap, other automated metrics Rebuffel et al. (2021); Kryscsinski et al. (2020); Wang et al. (2020); Scialom et al. ( 

2021) have been proposed to evaluate faithfulness more precisely. BARTScore is a transformer-based measure that scores a dialogue and the corresponding automatically generated summary and has been shown to be strongly correlated with human evaluations of faithfulness Yuan et al. (2021).

**Human Evaluation**: Finally, we conduct human evaluations on 100 SAMSum Gliwa et al. (2019) and 20 AMI McCowan et al. (2005) dialogues Tang et al. (2021) found that Likert scales are a more consistent measure of factuality for abstractive dialogue summarization than Best-Worst Scaling. We have human evaluators directly rate the summaries on a scale from 1 to 10 corresponding to their faithfulness. In addition, using the error taxonomy proposed in Section 2, we have them mark whether each error type appeared in the given summary. We do this in a blinded fashion, so that the annotators do not see the corresponding model of the summary. Additionally, in order to prevent model information from leaking to the annotators, we randomly shuffle outputs within each dialogue before assigning them to annotators.

## 5 Results

Table 1 shows the ROUGE scores of our models, the baseline models they were fine-tuned from, and a number of other abstractive summarization models on the SAMSum and AMI datasets. Tables 5 and 6 show the average human faithfulness and BART scores respectively for each model's outputs on 100 SAMSum and 20 AMI dialogues.

We observe that for all three pretrained models ConFiT significantly beat baselines on ROUGE-1, ROUGE-L, and human faithfulness score for both datasets. For BARTScore, we note that, while performance increases on SAMSum for all models, it decreases on AMI. However, given the fact that human evaluators rated the outputs of all three ConFiT models as more faithful than those of their corresponding baselines on both datasets, the decreases in BARTScore on AMI can likely be attributed to the imperfection of automated metrics at capturing faithfulness in text.

### Error Analysis

Tables 3 and 4 show the percentage of summaries that were labeled with each error type in our taxonomy of factual errors (discussed in Section 2.) for both the baseline and ConFiT models on the SAMSum and AMI datasets respectively.

We observe that on SAMSum, our fine-tuning method greatly reduces missing information, redundant information, wrong reference, and circumstance errors for all models. The largest reduction is on the "wrong reference" error type (20%, 7%, and 33% for BART, Pegasus, and T5 respectively),

\begin{table}
\begin{tabular}{l||c c c||c c c} \hline \hline  & \multicolumn{2}{|c||}{**AMI**} & \multicolumn{2}{|c}{**SAMSum**} \\

- \\ \hline \hline \multicolumn{8}{c}{_Pre-trained Models_} \\ \hline T5 Raffel et al. (2020) & 42.16 & 13.94 & 39.39 & 48.41 & 24.79 & 44.61 \\ Pegasus Zhang et al. (2020) & 46.02 & 15.85 & 43.73 & 48.04 & 22.94 & 43.40 \\ BART Lewis et al. (2020) & 47.92 & 16.00 & 45.36 & 51.74 & 26.46 & 48.72 \\ Multi-view BART Chen and Yang (2020) & – & – &
- & 49.52 & 26.52 & 48.29 \\ \hline \hline \multicolumn{8}{c}{_Ours_} \\ \hline T5-ConFiT & 47.18 & 13.19 & 43.55 & 52.13 & 27.12 & 47.62 \\ Pegasus-ConFiT & 48.47 & 17.61 & 45.75 & 52.65 & 28.21 & 48.15 \\ BART-ConFiT & 50.31 & 17.29 & 47.98 & 53.89 & 28.85 & 49.29 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dialogue summarization ROUGE evaluation on the AMI McCowan et al. (2005) and SAMSum Gliwa et al. (2019) datasets. We adopt some results reported from the literature Feng et al. (2021a) and implement the pre-trained models for a fair comparison. All results marked with an asterisk (*) are from Feng et al. (2021b).

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline  & _Diadogue_ & _Speakers_ & _Turns_ & _Length_ \\ \hline \multicolumn{5}{c}{_SAMSum_} \\ \hline Train Validation & 14732 & 2.40 & 11.17 & 23.44 \\   Test & 818 & 2.39 & 10.83 & 23.42 \\ \hline \multicolumn{5}{c}{_AMI_} \\ \hline All & 137 & 4 & 289 & 322 \\ \hline \end{tabular}
\end{table}
Table 2: Details about SAMSum and AMI.

 

likely owing to the self-supervised loss function introduced in Section 3.2 that was designed to help the model more effectively capture speaker information. For AMI, however, our fine-tuning method is not as consistent at reducing the frequency of each error type across models. It is possible that this is due to sample size (20 AMI dialogues vs. 100 SAMSum dialogues).

### Case Study

Figure 4 shows the results of human annotation on the model outputs of a selected SAMSum dialogue. Note that all of the autogenerated summaries, both baseline and ConFiT, were marked as having missing information errors by the annotator, likely due to the omission of Ernest's relief upon hearing that the car that was crashed into did not belong to Mike. As a result, none of the models achieved a perfect factuality score on this dialogue; however, the scores for each ConFiT model were higher than those of their corresponding baselines.

It can be observed that while baseline BART outputs a summary with a circumstance error, mistakenly asserting that Mike parked his car on Ernest's street, the BART+ConFiT fixes this error, correctly asserting that Mike took his car to the garage today; as a result, the human annotator gave this summary a higher score than the predicted summary from baseline BART. Baseline T5 outputs a summary with two coreference errors; specifically, it contains a missing subject in the first sentence and incorrectly implies that the car that got crashed into belonged to Mike in the second sentence. The T5+ConFiT is able to fix both of these errors,

\begin{table}
\begin{tabular}{l l l} \hline \hline
**BARTScore** & **SAMSum** & **AMI** \\ \hline \hline BART & -1.613 & **-3.644** \\ BART-ConFiT & **-1.468** & -3.669 \\ \hline Pegasus & -1.615 & **-2.967** \\ Pegasus-ConFiT & **-1.608** & -3.369 \\ \hline T5 & -1.993 & **-3.406** \\ T5-ConFiT & **-1.677** & -3.798 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Average BARTScore for each model on 100 SAWSum and 20 AMI dialogues. Highest scores for each dataset have been bolded.

\begin{table}
\begin{tabular}{l r r r r r} \hline
**Error Type** & **BART** & **BART-ConFiT** & **Pegasus** & **Pegasus-ConFiT** & **T5** & **T5-ConFiT** \\ \hline Missing Information & 90\% & 85\% & 80\% & 70\% & 80\% & 85\% \\ Redundant Information & 10\% & 15\% & 60\% & 25\% & 0\% & 25\% \\ Wrong Reference & 35\% & 30\% & 35\% & 30\% & 50\% & 50\% \\ Circumstance & 35\% & 35\% & 30\% & 30\% & 40\% & 35\% \\ Negation & 20\% & 15\% & 5\% & 15\% & 25\% & 0\% \\ Object & 45\% & 40\% & 45\% & 25\% & 55\% & 55\% \\ Tense & 10\% & 10\% & 0\% & 5\% & 10\% & 10\% \\ Modality & 10\% & 15\% & 5\% & 5\% & 20\% & 10\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Percentage of autogenerated summaries containing each error type, according to our human evaluation of model outputs from 20 AMI dialogues. Note that a single summary can contain multiple error types, so they do not add up to 100%.

\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Fairthfulness Score** & **SAMSUm** & **AMI** \\ \hline \hline BART & 5.540 & 4.850 \\ BART-ConFiT & **7.250** & **5.600** \\ \hline Pegasus & 6.260 & 5.250 \\ Pegasus-ConFiT & **6.770** & **5.895** \\ \hline T5 & 5.422 & 4.150 \\ T5-ConFiT & **6.920** & **4.950** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average faithfulness score (on a scale of 1-10) given to each model by human evaluators on 100 SAWSum and 20 AMI dialogues. Highest scores for each dataset have been bolded.

 

adding _"Mike"_ to the beginning of the first sentence and changing _"his red Honda"_ to _"a red Honda just like Mike's"_ in the second sentence. Similarly, the output of baseline Pegasus contains a coreference error in the first sentence, implying that Mike owns the car that was crashed into while the output of Pegasus+ConFiT does not.

## 6 Related Work

Multi-party dialogues are especially challenging to summarize using automated models, given that they often contain pauses, false starts, recontrinrations, hesitations, and speaker interruptions Sacks et al. (1978); Feng et al. (2021); Chen and Yang (2021). Previous work in the field has addressed these challenges by incorporating semantic features, including keywords Zhu et al. (2020), domain terminologies Koay et al. (2020), topics Zhao et al. (2020); Liu et al. (2021), entailment knowledge Li et al. (2018), and background knowledge Feng et al. (2021). Other works exploit personal named entities Liu and Chen (2021) and coreference information Liu et al. (2021) to learning to distinguish complex coreferent relationships expressed through personal pronouns (including the first person "I") in the conversation Lei et al. (2021). Researchers have also explored conversational structure Zhao et al. (2021), utterance flow modelling Chen et al. (2021), syntactic structure Lee et al. (2021), granularity control Wu et al. (2021), but they have not yet converged to a simple and practical solution.

Our proposed taxonomy of factual errors and annotations help us gain deeper insights into the causes of factual inconsistency in abstractive dialogue summarization outputs.

## 7 Conclusion

We presented ConFit, a novel method to improve the faithfulness of abstractive dialogue summarization models via contrastive and self-supervised fine-tuning. By adapting the objective function during fine-tuning to incorporate a contrastive loss that learns to distinguish positives from examples with factual errors, and a self-supervised dialogue-specific loss that captures important dialogue information flow between multiple interlocutors, ConFit can significantly improve the faithfulness of the abstractive summaries generated by transformer-based sequence-to-sequence language models, and reduce multiple categories of factuality errors in the abstractive summaries by large margins. In our experiment on SAMSum and AMI, we demonstrated that ConFit achieves better empirical performance compared to the baseline models fine-tuned with the traditional cross-entropy loss, based on both automatic evaluation metrics and human evaluation. Our work provides new insights into improving the faithfulness of abstractive summarization systems using carefully designed novel objective functions for fine-tuning that captures important structures and features of the text to summarize.

Figure 4: Model outputs for selected SAMSum dialogue, along with the corresponding reference summary, human factuality scores, and errors.

 

Ethics Statement

Human EvaluationWe recruited seven volunteer participants for our error annotation, requesting speakers of English. The internal annotators are Xiangru Tang, Arjun Nair, Borui Wang, Jai Desai, Aaron Wade, Anushka Nijhawan, and Dragomir Radev. These annotators are participating voluntarily. Our participants are free to opt out of the study at any point in time. We have written four scripts for use in the annotation process: (1) the first script generates an annotation spreadsheet and a key spreadsheet from the model outputs. The annotation spreadsheet does not contain the model names; however, it contains an id that can be used to recover the model name from the key spreadsheet. For ease of annotation, summaries from the same dialogue are grouped together; however, they are randomly shuffled within each dialogue so that the annotators cannot guess from the ordering as to which model is which. (2) The second script splits an annotation spreadsheet into multiple spreadsheets so that the work can be distributed amongst annotators. (3) The third one merges these spreadsheets back together after the annotation process is finished. (4) The last script recovers the model names from the key spreadsheet and inserts them into the annotation spreadsheet. Each evaluator is asked to examine whether there is an error and the full context (dialogue, generated summaries, and reference) and give a score on a scale of 1 to 10 for each of the criteria. We only consider faithfulness, instead of general quality. E.g. 1: very poor, 3: poor, 5: neutral; 7: good; 10: very good. We asked each internal annotator to evaluate 300 samples.

Other Ethical Issues(1) We did not use any personally identifiable information in the experiments. (2) The goal of the project, improving the faithfulness of automatically generated summaries, is to make the output of the summarization system more reliable and minimize confusion for the readers of the summaries. (3) We used existing summarization datasets that do not contain any sensitive information and are unlikely to cause any harm to the annotators.

## References

* Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* Chen and Yang (2020) Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4106-4118, Online. Association for Computational Linguistics.
* Chen and Yang (2021a) Jiaao Chen and Diyi Yang. 2021a. Simple conversational data augmentation for semi-supervised abstractive dialogue summarization. In _Proceedings of the Association for Computational Linguistics: EMNLP 2020_, Online. Association for Computational Linguistics.
* Chen and Yang (2021b) Jiaao Chen and Diyi Yang. 2021b. Structure-aware abstractive conversation summarization via discourse and action graphs. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1380-1391, Online. Association for Computational Linguistics.
* Chen et al. (2021a) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021a. Summscreen: A dataset for abstractive screenplay summarization.
* Chen et al. (2021b) Wang Chen, Piji Li, Hou Pang Chan, and Irwin King. 2021b. Dialogue summarization with supporting utterance flow modelling and fact regularization. _Knowledge-Based Systems_, 229:107328.
* Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
* Chen et al. (2021c) Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021c. DialogSum: A real-life scenario dialogue summarization dataset. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 5062-5074, Online. Association for Computational Linguistics.
* Chen et al. (2021d) Yulong Chen, Yang Liu, and Yue Zhang. 2021d. Dialogsum challenge: Summarizing real-life scenario dialogues. In _Proceedings of the 14th International Conference on Natural Language Generation_, pages 308-313.
* Fabbri et al. (2021) Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. ConvoSumm: Conversation summarization benchmark and improved abstractive summarization with argument mining. In _Proceedings of 

the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, Online. Association for Computational Linguistics.
* Feng et al. (2021a) Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021a. A survey on dialogue summarization: Recent advances and new frontiers. _arXiv preprint arXiv:2107.03175_.
* Feng et al. (2021b) Xiaachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021b. Language model as an annotator: Exploring dialogpt for dialogue summarization. _arXiv preprint arXiv:2105.12544_.
* Feng et al. (2021c) Xiaichong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021c. Language model as an annotator: Exploring DialoGPT for dialogue summarization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1479-1491, Online. Association for Computational Linguistics.
* Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
* Gunel et al. (2020) Beliz Gunel, Jingfei Du, Alexis Conneau, and Vesy Stoyanov. 2020. Supervised contrastive learning for pretrained language model fine-tuning. _arXiv preprint arXiv:2011.01403_.
* Gurevych and Strube (2004) Iryna Gurevych and Michael Strube. 2004. Semantic similarity applied to spoken dialogue summarization. In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_, pages 764-770, Geneva, Switzerland. COLING.
* Koay et al. (2020) Jia Jin Koay, Alexander Roustai, Xiaojin Dai, Dillon Burns, Alec Kerrigan, and Fei Liu. 2020. How domain terminology affects meeting summarization performance. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 5689-5695, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346, Online. Association for Computational Linguistics.
* Lee et al. (2021) Seolhwa Lee, Kisu Yang, Chanjun Park, Joao Sedoc, and Heuisceok Lim. 2021. Who says like a style of vitamin: Towards syntax-aware dialoguesummarization using multi-task learning. _arXiv preprint arXiv:2109.14199_.
* Lei et al. (2021) Yuejie Lei, Yuanmeng Yan, Zhiyuan Zeng, Keqing He, Ximing Zhang, and Weiran Xu. 2021. Hierarchical speaker-aware sequence-to-sequence model for dialogue summarization. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7823-7827. IEEE.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2021) Daniel Li, Thomas Chen, Albert Tung, and Lydia Chilton. 2021. Hierarchical summarization for long-form spoken dialog.
* Li et al. (2018) Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1430-1441, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2021a) Jumpen Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021a. Topic-aware contrastive learning for abstractive dialogue summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, Online. Association for Computational Linguistics.
* Liu and Chen (2021) Zhengyuan Liu and Nancy F. Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In _Proceedings of the Association for Computational Linguistics: EMNLP 2020_, Online. Association for Computational Linguistics.
* Liu et al. (2021b) Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021b. Coreference-aware dialogue summarization. In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 509-519, Singapore and Online. Association for Computational Linguistics.
* Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919, Online. Association for Computational Linguistics.
* McCowan et al. (2018) Iain McCowan, Jean Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas 

Hain, J Kadlec, Vasilis Karaiskos, et al. 2005. The ami meeting corpus. In _Proceedings of the 5th international conference on methods and techniques in behavioral research_, volume 88, page 100. Citeseer.
* Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In _Proc. of EMNLP_, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
* Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4812-4829, Online. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Rebuffel et al. (2021) Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-questeval: A referenceless metric for data to text semantic evaluation. _arXiv preprint arXiv:2104.07555_.
* Sacks et al. (1978) Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In _Studies in the organization of conversational interaction_, pages 7-55. Elsevier.
* Scalom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacobo Staiano, and Alex Wang. 2021. Questeval: Summarization asks for fact-based evaluation.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proc. of ACL_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Tang et al. (2021) Xiangru Tang, Alexander R. Fabbri, Ziming Mao, Griffin Adams, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. Investigating crowdsourcing protocols for evaluating the factual consistency of summaries.
* Wang et al. (2020)Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5008-5020, Online. Association for Computational Linguistics.
* Wu et al. (2021) Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021. Controllable abstractive dialogue summarization with sketch supervision. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 5108-5122, Online. Association for Computational Linguistics.
* Xiaachong et al. (2021) Feng Xiaachong, Feng Xiao Cheng, and Qin Bing. 2021. Incorporating commonsense knowledge into abstractive dialogue summarization via heterogeneous graph networks. In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_, pages 964-975, Huhhot, China. Chinese Information Processing Society of China.
* Yuan and Yu (2020) Lin Yuan and Zhou Yu. 2020. Abstractive dialog summarization with semantic scaffolds.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. _arXiv preprint arXiv:2106.11520_.
* Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In _International Conference on Machine Learning_, pages 11328-11339. PMLR.
* Zhang et al. (2021) Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, and Mohit Bansal. 2021. EmailSum: Abstractive email thread summarization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6895-6909, Online. Association for Computational Linguistics.
* Zhao et al. (2020)Lulu Zhao, Weiran Xu, and Jun Guo. 2020. Improving abstractive dialogue summarization with graph structures and topic words. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 437-449, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Zhao et al. (2021)Lulu Zhao, Keyuan Yang, Weiran Xu, Sheng Gao, and Jun Guo. 2021. Improving abstractive dialogue summarization with conversational structure and factual knowledge.
* Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Muethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5905-5921, Online. Association for Computational Linguistics.
* Zhu et al. (2021) Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. Mediasum: A large-scale media interview dataset for dialogue summarization. _arXiv preprint arXiv:2103.06410_ 

Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 194-203, Online. Association for Computational Linguistics.