Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
Year: 2023

Abstract:
A series of datasets and models have been proposed for summaries generated for wellformatted documents such as news articles. Dialogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT. We define finegrained factual error detection as a sentencelevel multi-label classification problem, and we evaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation.1

heading: 1 Introduction
text: Factual inconsistency in abstractive summarization — a phenomenon where model-generated summaries contain facts that are inconsistent with the source document — is a widely known problem and has been studied extensively in the document summarization community. An example is shown in Figure 1, where the source document is a dialogue — the type of documents that this paper focuses on.
Existing work covers topics on factual inconsistency including error typology and factuality annotations of state-of-the-art neural summarization models (Maynez et al., 2020; Huang et al., 2020; Pagnoni et al., 2021; Goyal and Durrett, 2021; Fabbri et al., 2021; Gao and Wan, 2022; Tang et al., 2022a), automatic factual error detectors (Wang et al., 2020; Goyal and Durrett, 2020; Kryscinski
1The dataset and code are available at https://github. com/731935354/Dia-Sum-Fact
et al., 2020; Durmus et al., 2020; Zeng et al., 2021; Scialom et al., 2021), methods to correct factual errors in summaries (Cao et al., 2020; Dong et al., 2020; Chen et al., 2021a) and methods to produce factually more consistent summaries (Zhao et al., 2020; Cao and Wang, 2021; Tang et al., 2022b; Zhu et al., 2021; Aralikatte et al., 2021; Chen et al., 2021b; Balachandran et al., 2022). Almost all of these works focus on news summarization based on two datasets: CNN/DAILYMAIL (Hermann et al., 2015; Nallapati et al., 2016) and XSUM (Narayan et al., 2018).
Dialogue summarization (cf Figure 1), which aims to produce a condensed version of a dialogue while maintaining its salient information, is equally important due to its application to summarizing meeting transcripts (Li et al., 2019; Zhu et al., 2020; Zhong et al., 2022), daily conversations (Chen and Yang, 2020; Liu and Chen, 2021; Feng et al., 2021), customer service dialogues (Liu et al., 2019; Zou et al., 2021) and medical dialogues (Joshi et al., 2020; Krishna et al., 2021). However, factual consistency in dialogue summarization is under explored as there are currently no benchmark datasets that contain fine-grained error categories. This paper aims to fill in this gap.
To investigate factual consistency in dialogue
6825
summarization, we release DIASUMFACT with fine-grained sentence-level annotations regarding factual consistency for 475 model summaries (1,340 sentences) from six neural dialogue summarization models on two popular datasets: SAMSUM (Gliwa et al., 2019) and QMSUM (Zhong et al., 2021). We adopt a two-dimensional typology that considers the semantic roles and verifiability of error spans separately.
We formulate factual error detection as a sentence-level multi-label classification task and use DIASUMFACT to evaluate two state-of-the-art factual error detection models designed for document summarization. As there are no existing error detection model for fine-grained error categories, we adapt the two binary classification models to fit to our task. Empirical results show that they don’t work well on the task, indicating its difficulty and the domain gap between document summarization and dialogue summarization.
We then propose two models: BERTMULTI and ENDERANKER. BERTMULTI is a multi-class classification model trained on synthetic data, which is created by corrupting sentences from reference summaries (Kryscinski et al., 2020). ENDERANKER is a simple unsupervised model that can leverage any pretrained encoder-decoder model to detect factual errors. Given a model-generated summary sentence containing a span of interest for error detection, ENDERANKER computes log likelihood scores for the sentence and its variants containing replacement spans fetched from the source dialogue. The scores are computed as BARTSCORE (Yuan et al., 2021), which will be explained in 4.2. We compare the scores of the sentences to determine if the span of interest and hence the summary sentence contains a factual error. We run experiments with T5 (Raffel et al., 2020), BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020), fine-tuned either on news summarization or dialogue summarization, as the encoder-decoder for ENDERANKER. The results show that BERTMULTI and ENDERANKER performs on par with the adapted state-of-the-art models in terms of macro-averaged F1.
Motivated by the strong complementarity between models, we further present two ensemble models combining the four models above. The results, while exceeding those of the individual models, are still far from indicating a practical model for factual error detection over dialogue summaries.
This calls for further studies, for which our dataset and results form a solid foundation.
To summarise, this paper makes the following contributions:
• We annotate and present DIASUMFACT, the first dataset with fine-grained sentence-level factual errors for dialogue summarization, providing rich annotation including error classes, erroneous spans and explanation.
• We investigate the effectiveness of adapting state-of-the-art factual error detection models for document summarization on modelgenerated dialogue summaries, demonstrating the difficulty of the task.
• We propose BERTMULTI, a weaklysupervised multi-class classifier and ENDERANKER, an unsupervised factual error detector that requires no human labeled data for training and can leverage existing pre-trained encoder-decoder models. Both models perform on par with adapted SOTA factual error detection models for document summarization.
• Our experiments and analyses reveal the strengths and weaknesses of different factual error detection models, and point out future directions to improve them.

heading: 2 Related Work
text: Error typology and datasets. There are a few existing datasets on factual errors. Some of them use binary (factually consistent or inconsistent) labels (Kryscinski et al., 2020; Wang et al., 2020) and 5-point Likert Scale labels (Fabbri et al., 2021; Gao and Wan, 2022), which require lower efforts to annotate, but they do not provide information on how and where factual errors were made. To support fine-grained analysis, multi-class and multidimensional typologies are designed. Pagnoni et al. (2021) propose a linguistically motivated annotation framework that covers semantic frame errors, discourse errors and content verifiability errors. Goyal and Durrett (2021) use a 2-dimensional typology, where content verifiability and semantic error types are considered separately. Cao et al. (2022) focus on hallucinations and consider both factual and non-factual hallucination. Tang et al. (2022a) unify different error types from previous
works into a hierarchical taxonomy. These datasets mostly focus on news summaries.
DialSummEval (Gao and Wan, 2022) is another popular dataset that contains annotation on factual consistency of model-generated dialogue summaries. The core difference of our work is that we consider fine-grained error categories and the text span (i.e., starting and ending position) of an error. Thus it provides a more elaborate, diagnostic assessment as to what and where goes wrong when a summary is not factually consistent. In comparison, DialSummEval only considers coarsegrained assessment of factuality using 5-point Likert Scale (Joshi et al., 2015), without specifying the actual error type (e.g., entity error). Factual error detection models. Most popular factual error detectors are based on either textualentailment or question-answering (QA).
Textual-entailment-based models are generally binary classifiers that take as input the source document and a model-generated summary. For example, Kryscinski et al. (2020) train binary factual error classifiers using synthetic data. Zeng et al. (2021) use a gradient-based adversarial method to improve model accuracy. Goyal and Durrett (2020) leverage dependency-level entailment achieving better performance and interpretability.
QA-based models first generate questions from a model-generated summary (or source dialogue), and then answer those questions based on its source dialogue (or a model-generated summary). The factual consistency is decided by the similarity between the ground truth answer and the predicted answer. For example, Wang et al. (2020); Durmus et al. (2020) use a precision-oriented method that generates questions from model-generated summaries and answer them using the source document. Scialom et al. (2019) instead generate questions from a source document and answer them using the summary, making it a recall-oriented method. Scialom et al. (2021) combine recall and precisionoriented techniques into a single framework. Fabbri et al. (2022) refine the model component design and obtain a QA-based method that outperforms textual-entailment-based methods.
Our unsupervised method ENDERANKER compares a span (e.g., a person name) in a modelgenerated sentence with candidates (e.g., other people’s names in the dialogue) and decide the factual consistency of the span based on its rank among candidates. It achieves comparable macro F1 with
adapted SOTA factual error detectors for document summarization but requires no labelled resources.

heading: 3 The DIASUMFACT Dataset
text: This section presents our DIASUMFACT dataset and procedures to construct the dataset.

heading: 3.1 Data Source
text: To cover dialogues from different domains, we selected two popular datasets SAMSUM (Gliwa et al., 2019) and QMSUM (Zhong et al., 2021). SAMSUM contains daily conversations and gold summaries. QMSUM comes with queries and answers based on meeting transcripts. The answers to each query can be seen as a summary to an aspect of the meeting transcript.
For both SAMSUM and QMSUM, we randomly sampled 60 dialogues and their summaries in its test split.2 For QMSUM, we only chose queries whose gold utterances contain no more than 700 tokens according to Bert tokenizer. 3 We manually filtered out dialogues with sensitive contents (e.g., dirty words and potential bias on gender or race). More statistics on the dataset can be found in Appendix Table 5 and Table 6.

heading: 3.2 Summary Generation Models
text: We generally choose models with publicly accessible pretrained model checkpoints or generated outputs instead of training models ourselves.
On SAMSUM, we use five models: BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020), S-BART (Chen and Yang, 2021), CONDIGSUM (Liu et al., 2021) and GPT3 (Brown et al., 2020). For S-BART and CONDIGSUM, we obtain model outputs from the original papers. For BART and PEGASUS, we generate output by running their pre-trained models.4 For GPT-3, we fine-tune curie over SAMSum dataset and generate summaries using the official API.5
On QMSUM, we use three models: PEGASUS, BART and DialogLM (Zhong et al., 2022). Since we only focus on specific queries (i.e., queries that
2For QMSUM we also have the queries, in addition to the dialogues and summaries.
350% of the queries on aspects of meeting transcripts satisfy this constraint.
4We use linydub/bart-large-samsum for BART and transformersbook/pegasus-samsum for PEGASUS. Both are from https://huggingface.co/models.
5We fine-tuned it on May 27th, 2022 following https: //beta.openai.com/docs/guides/fine-tuning.
Dialogue Lucas: Where r u? I’m waiting at the airport. Vanessa: There was a foul-up with the flight. I’m trying to get another ticket. Lucas: OMG. How come? Vanessa: No bloody idea. All of the flights are booked cos students are returning from holidays. Lucas: I’ve called the airport and they said there’s a flight to New York at 9:45 p. m. Vanessa: Great, I’ll book it now.
Error Description Example Summary In/Ex
EntE The core arguments or their attributes in a semantic frame are wrong, such as the subjects and objects. Vanessa is waiting at the airport. In
PredE The predicate, which is usually a verb, of a semantic frame is wrong. Lucas has emailed the airport and got some information about the flight to New York. Ex
CirE The non-core arguments, such as location modifiers, temporal modifiers are wrong. Lucas is waiting at the train station. Ex
CorefE A pronoun or a reference (e.g., this picture) has a wrong antecedent or has no antecedents. Vanessa is trying to get another ticket for themselves. N/A
LinkE The relationship, e.g., a causal relationship, between statements is wrong. Vanessa will book the flight to New York at 9:45 pm because students are returning from holidays. N/A
only ask about an aspect of a meeting, instead of summarizing the whole meeting), which is a subset of the original dataset, we fine-tuned them using specific queries only. The fine-tuned models achieve ROUGE scores that are better or comparable to state-of-the-art models on the complete dataset.6

heading: 3.3 Typology of Factual Errors
text: Motivated by Goyal and Durrett (2021); Pagnoni et al. (2021), we adopt a 2-dimensional typology that treats semantic role and content verifiability of error spans separately.
On the semantic role dimension, we consider six error classes Entity Error (EntE), Predicate Error (PredE), Circumstance Error (CirE), Coreference Error (CorefE), Link Error (LinkE) and Others, with definitions and examples shown in Table 1. EntE, PredE, CirE are semantic frame errors, and CorefE, LinkE are discourse errors. When a sentence in the summary does not contain any factual error, we label it as No Error.
For content verifiability, we consider Intrinsic
6The ROUGE scores of the fine-tuned models are shown in Appendix A.2. We also tried 2-shot GPT-3 but found that it didn’t work well in preliminary experiments and for that reason didn’t include GPT-3.
Error (i.e., the error span consists of tokens from the source dialogue) and Extrinsic Error (i.e., the error span consists of tokens not mentioned in the source dialogue), a.k.a. hallucinations. This dimension is only defined for EntE, PredE and CirE.

heading: 3.4 Annotation Procedure
text: We recruited 12 workers for the annotation task, including nine PhD students majored in natural language processing and three Master’s students majoring in linguistics and information technology. All annotators are fluent English speakers. We take an in-house annotation approach because a trial on Amazon Mechanical Turk did not yield meaningful results, even though high-quality crowd-sourced workers were sourced through strict constraints. The 12 annotators form six pairs randomly where each pair annotates 10 dialogues from each dataset.
The annotation is done in three stages: pilot study, full annotation and annotation adjudication.
An annotation task involves analysing a dialogue and the summaries generated by all corresponding models. During the pilot study, annotators are required to go through the definition and examples for each error class to learn the labelling typology. Then, they will work on two pilot tasks, which are the same for all workers. For each task, a source di-
alogue and a model-generated summary are shown at the same time, and the annotator needs to label any factual errors in each individual sentence in the summary. When all sentences in the summary are done, another summary generated by a different model will be shown. Models are anonymized and their generations are shown in random order.
During the full annotation stage, we assign each annotator 10 tasks from each dataset, which are different from the tasks in pilot study. The annotations are only done for the semantic role dimension.
In the adjudication stage, the two annotators of a pair along with an annotation quality controller (one of the authors of this paper) go through the annotations to resolve any disagreements, and detailed notes were taken for reaching the final decisions (which is released as part of the dataset as it can be useful for future analysis). Annotation mistakes are also corrected in this process. In the end, a total of 1340 sentences (99.7%) with agreed annotations were obtained, while the rest of the sentences were discarded because no agreement can be made.
Note that the annotations on the content verifiability dimension are manually created by the annotation quality controller based on the detailed meeting notes of the last stage. It is a product of a post-annotation process because the original annotators did not explicitly label the error type as extrinsic or intrinsic. Instead, the annotators mark an Extrinsic Error for all error spans that are not mentioned in the source dialogue. The annotation quality controller takes this information and further split them into EntE, PredE and CirE based on the semantic role of an error span, and assign Intrinsic Error to all original EntE, PredE and CirE, thus obtaining a 2-dimensional annotation.

heading: 3.5 Inter-annotator Agreement
text: We use Cohen’s Kappa (McHugh, 2012) to evaluate the inter-annotator agreement. The scores in each group before adjudication are as follows. We first evaluate the agreement for binary label by merging all error types into a single negative class. The scores are 0.39, 0.44, 0.57, 0.59, 0.43, 0.51. For multi-class label, the scores are 0.34, 0.33, 0.44, 0.31, 0.31, 0.25. After adjudication we have full agreement for all instances (as explained in Section 3.4).

heading: 3.6 Results on the Summarization Models
text: We summarize the performance results of the summarization models as derived from the annotations in this subsection. Figure 2 and Figure 3 show the factual error class distribution of the summarization models evaluated on SAMSUM and QMSUM.
Overall, 33.3% and 41.9% sentences in modelgenerated summaries contain one or more factual errors in SAMSUM and QMSUM, respectively. The average number of errors for a factually inconsistent sentence is 1.14. This indicates a broad existence of factual errors in the model-generated summaries, thus emphasizing the importance to resolve factual errors in dialogue summarization.
Semantic frame errors (i.e., EntE, PredE and CirE) are more frequent than discourse errors (i.e., CorefE and LinkE) overall, while their distributions are not the same on both datasets. SAMSAM has a higher portion of factually inconsistent sentences caused by semantic frame errors (76.9%) than QMSUM has (58.9%), while QMSUM has a higher portion of discourse errors (24.0%) than SAMSAM (11.3%). We observe two main reasons for this
discrepancy. First, the sentences in QMSUM are longer and exhibit more complex discourse structures, especially causal relations, which can be challenging for models to summarize. Second, models fine-tuned on QMSUM tend to copy large chunks of the input dialogue. Many pronouns are directly copied from the source dialogue without proper context, causing Coreference Errors (CorefE).
Among the different summarization models, BART and PEGASUS have been evaluated on both datasets where BART generates summaries with fewer factual errors consistently. On SAMSUM, 24.0% of the sentences generated by BART contain factual errors, which is the fewest, while the highest portion is reported by GPT-3, i.e., 58.7%. CONDIGSUM and S-BART are variants of BART that achieve better ROUGE scores than BART using contrastive learning and dialogue structure information, respectively. Our results reveal that both models produced more sentences with factual errors than BART did, indicating that improvement in ROUGE may not help with the factual consistency of summaries. This result emphasizes the importance of more benchmark datasets for dialogue summarization model evaluation. On QMSUM, BART is still the best, while DIALOGLM produced the highest proportion of sentences with factual errors.
On the content verifiability dimension, models on QMSUM produce more extrinsic errors than on SAMSum. A potential reason is that reference summaries in QMSUM contain more tokens outside the source dialogue. For SAMSUM, all models are mainly dominated by intrinsic errors, while GPT-3 produces more extrinsic errors than intrinsic ones.

heading: 4 Detecting Factual Errors
text: In this section, we automate factual error detection in model-generated summaries. We first adapt two state-of-the-art factual error detection models from document summarization. We then propose a weakly supervised multi-class classifier and a simple yet effective unsupervised model that can utilize any pretrained encoder-decoder model to identify factual errors. Finally, we present ensemble-based models combining all techniques above.
Problem statement. We formulate factual error detection as a sentence-level multi-label classification task, i.e., given an input dialogue and a sentence from a model-generated summary, we classify whether the sentence contains any (seman-
tic role) factual errors as outlined in Section 3.3.

heading: 4.1 Adapted State-of-the-Art Models
text: DAE (Goyal and Durrett, 2020) is based on dependency-level entailment, which predicts whether a dependency arc in a model-generated sentence is entailed by the input document (e.g., a dialogue in our problem). To adapt it to our problem, we design rules to map from dependency arc types to our factual error classes, as shown in Table 2. Given a summary sentence, we use the trained DAE provided by the authors to predict dependency arcs in the sentence. The union of all factual error classes corresponding to the types of the predicted erroneous dependency arcs will be used as our factual error predictions. Note that not all factual error classes have corresponding dependency arc types and hence not all error classes can be detected by this model.
QAFactEval (Fabbri et al., 2022) is a QA-based factual error detector. Given a question generation model (QG) and a question answering model (QA), which are trained on existing datasets for the question answering task, it works as follows: (1) Question-worthy spans (s), which are noun phrases and named entities, are extracted from a modelgenerated summary. (2) For each s, a question is generated by QG based on s and the summary. (3) The QA model predicts an answer a based on the question and the source document. (4) The similarity between s and a is measured by some metric. (5) The factual consistency of the summary is made based on the similarity scores for all s in it.
We use the learned metric LERC (QuIP) mentioned in the paper and report a factual error if the similarity score between s and a is smaller than a threshold Tqa (a hyper-parameter). Questionworthy spans of different semantic roles correspond
to our semantic role-based factual error classes, as outlined in Algorithm 1 in Appendix. We obtain the semantic role of a question-worthy span by a pretrained structured prediction model in AllenNLP 2.9.3.7
WEAKLY-SUPERVISED-CLASSIFIER is a multi-class classifier that we construct. It takes as input a source dialogue and a generated summary sentence to predict factual error classes in the sentence, motivated by Kryscinski et al. (2020). We create synthetic training data by corrupting sentences in reference summaries as follows.
For Entity Error, Circumstance Error and Coreference Error, we replace named entities or pronouns with those randomly picked from the same category. For Predicate Error, we replace verbs with other randomly chosen verbs. We match the form (e.g., tense) of the selected verbs to the original one. Negative replacements for all above classes are extracted from either the source dialogue or the whole dataset. For Link Error, we replace a discourse marker corresponding to causal relation (e.g., because) with another one indicating a reversed causal relation (e.g., so). More details on our synthetic data generation are in Appendix A.3.1.
We use cross entropy loss to train the classifier, which is based on BERT (Devlin et al., 2019) with a linear layer on top of [CLS] representation for classification. We concatenate the source dialogue and a sentence, delimited by [SEP], as input.
7We use structured-prediction-srl-bert and choose the semantic role of the shortest span containing s.

heading: 4.2 ENDERANKER
text: Here, we present our proposed unsupervised model, ENDERANKER. Given a generated summary sentence, it first identifies a set of spans of interest (SOI) which may correspond to factual errors. For each SOI, ENDERANKER replaces it with different candidate spans and calculates a score for each span including the SOI. The factuality of the SOI is then decided based on its score among the scores of all candidate spans. Figure 4 summarizes the workflow of ENDERANKER. Below we detail core steps of ENDERANKER: (1) SOI identification, (2) candidate span generation, (3) span scoring and (4) ranking-based factual error detection.
Span of interest identification. An SOI is a snippet in a sentence for factual error classification. We consider noun phrases, named entities and verbs as SOIs, which are obtained using spaCy 3.1.4.8 We obtain the semantic roles of the SOIs like for QAFACTEVAL, which will be used to decide the error class of an SOI later.
Candidate span generation. For each SOI, we create a set of candidate spans that can potentially replace it in the model generated summary sentence. For a named entity SOI, the candidate spans are entities of the same named entity class (e.g., PERSON) of the SOI extracted from the input dialogue. For the PERSON class, in particular, we include all speaker names on top of all other PERSON named entities extracted. For a verb SOI, we extract all verbs from the input dialogue according
8https://spacy.io/
to the Part-of-Speech tags and match the form (e.g., tense) with the SOI. For a noun phrase SOI, all noun phrases from the input dialogue are considered as candidate spans. All candidate spans are extracted using spaCy 3.1.4.
Span scoring. Let D be an input dialogue and S be a generated summary sentence with n tokens {w1, w2, · · · , wn−1, wn}, which includes a candidate span or an SOI, denoted by c. We adopt a encoder-decoder model M to calculate a sentence score for S conditioned on D as follows, which is used as the score of span c, denoted by scorec. M can be any pre-trained encoder-decoder model, such as a summarization model.
scorec = 1
n
n∑
i=1
log p(wi|w<i, D) (1)
Intuitively, the score is the average log likelihood of each token wi in S, conditioning on the previous tokens in S (i.e., w<i) and D. Here, w0 is the starting token of the decoder.
Ranking-based factual error detection. Given a set of candidate spans C = {c1, c2, · · · , c|C|} of an SOI, we form |C| sentences by replacing the SOI with each of the candidate spans. We calculate span scores for the SOI and the candidate spans, and rank the spans by their scores in descending order. If the SOI has a rank larger than a threshold T (a hyper-parameter), we report it as erroneous
and determine its error class based on its semantic role, as summarized in Algorithm 1 (cf. Appendix). The same process is repeated for all SOIs in S. The union of all error classes detected for the SOIs is the final factual error classes predicted for S.

heading: 4.3 Ensemble Modeling
text: We further build two simple ensemble models based on the four models above: Most Frequent Voting (FREQVOTING) and Logistic regression (LOGISTIC). FREQVOTING takes all predicted error classes from the four models above and uses the class(es) with the largest frequency as the final prediction. For LOGISTIC, we train a logistic regression model for each factual error class that takes the binary outputs from the four models above as features. We use the union of all factual error classes predicted by the different logistic regression models as the final prediction.

heading: 4.4 Experiments
text: To evaluate the models described in the last section, we perform 5-fold cross validation (Stone, 1978) using DIASUMFACT.9 Implementation details and parameter settings are discussed in Appendix A.3. We record the F1 scores (mean and standard deviation) of the models on each error class in Table 3.
9As it gives more reliable results considering the size of our dataset, compared to a usual train/test split.
Results: All models can detect EntE significantly and consistently better than the other classes. Different models show advantage on different error classes, while no model can outperform all the others on all error classes.
QAFACTEVAL performs the best on EntE (0.45) and CirE (0.23) but poorly on the other error classes. The reason is that only named entities and noun phrases are treated as question-worthy spans. Future work may consider question-worthy spans of different types, such as verbs and discourse markers, to cover more error classes.
DAE performs well on EntE and Others, while it suffers on CirE, PredE and CorefE. The main reason is that not all error classes are covered in the rules mapping from dependency arc to error class. Since a dependency arc is related to two words, rule designing is not easy. Future work may leverage learned models to predict error class automatically.
BERTMULTI shows the best results on CorefE (0.29) but poor performance on CirE, PredE and Others, despite its high performance on synthetic validation dataset (0.98 accuracy). It indicates the difference between synthetic and real factual errors.
Our proposed model ENDERANKER using different pretrained encoder-decoder models generally exhibits strong results on EntE, PredE and CorefE, while more improvements need to be done on CirE and Others. Among all variants of ENDERANKER, PEGASUS-CNN performs on par with QAFACTEVAL in terms of macro-averaged F1 score, while it does not require question generation and question answering models.
The two ensemble models improve on the micro and macro-averaged F1, indicating complementarity among the models. For most error classes, the ensemble models usually have the best or second best performance.
Overall, none of the models yielded a particularly high F1 score for any error class. It shows that fine-grained factual error detection in dialogue summaries is a challenging problem which calls for further studies, for which our results and dataset will serve as a solid foundation.

heading: 5 Conclusions
text: We created a fine-grained multi-faceted dataset named DIASUMFACT on factual consistency of dialogue summarization. DIASUMFACT offers insights into how and where current neural summarization models fail when they produce factually
inconsistent details in dialogue summaries. It can also serve as a testbed for automating factual error detection. Our proposed error detection method, ENDERANKER, is shown to perform on par with state-of-the-art models even though it requires no labelled training data. That said, we ultimately found that even ensembling several error detection methods do not produce results that are good enough for practical use, indicating opportunities for future research in this area.

heading: 6 Limitations
text: ENDERANKER is only tested on DIASUMFACT. Further tests on more datasets are required to establish its general applicability.

heading: 7 Ethics Statement
text: This study is conducted under the guidance of the ACL code of Ethics. We manually filtered out potential offensive content and removed all information related to the identification of annotators. The annotators are all fairly paid based on the Australian minimum wage. The annotation protocol is approved under Human Ethics LNR Application with reference number 2022-24233-30104-3.

heading: Acknowledgements
text: This research was undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200. We want to thank Gisela Vallejo, Han Sun, Miao Li, Rui Xing, Wei Gao, Yanchuan Chang, Yulia Otmakhova, Zheng Wei Lim, Zhexi Li, Zhuohan Xie for their help in the annotation.

heading: B Data Annotation
text: B.1 Error Typology
For CorefE, if a reference comes without antecedents in the input dialogue, we ignore the error in the summary.
B.2 Annotation Tool
We modify a web application developed originally for FRANK (Pagnoni et al., 2021)15 to fit to our task. Specifically, we replace the example article and model summaries with an example dialogue and manually composed summaries to help explain different error types. We also add an input field for error span annotation in the main page. Screenshots are shown in Figures 6, 7, 8 and 9.
For in-house annotation, we deploy the web application on Firebase16 and provide with annotators URLs to the tasks directly.
15https://github.com/artidoro/ frank-annotation-platform
16https://firebase.google.com/
B.3 Annotation Procedure
The initial annotation by all annotators follows the typology proposed by Pagnoni et al. (2021), which includes two additional classes: Out-of-Article Error (i.e., Extrinsic Error in our paper) and Grammar Error. We merge Grammar Error to Others, and treat Extrinsic Error as a separate dimension, as outlined in 3.3.
B.4 Payments to Annotators
All our annotators are volunteers. We pay 100 AUD to each annotator. The annotation task begins after they agree to the amount of payment.
B.5 Demographic Characteristics of Annotators
1 annotator come from Colombia, 1 annotator comes from Russia, 1 annotator come from Malaysia, 9 annotators come from China. There are 6 female and 6 male annotators.
B.6 Consent from Annotators We show the consent form in the annotation web application. Annotation can only begin after consent form is received from annotators.

heading: C Case Study
text: As shown in Figure 5, our ENDERANKER successfully identifies an error of the span “The team” because its rank is larger than the threshold T = 3. Since the semantic role of the span is arg0, the model predicts Entity Error according to Algorithm 1. On the right-side example, ENDERANKER fails to report the error of “muchroom picking”, although the factual consistent span “horse racing” is ranked at the top among candidates. The reason is that T is too large. For future work, we may design error identification methods using SOIspecific thresholds rather than a universal threshold for all SOIs.

heading: D Potential Risks
text: The factual error detection models we propose, which are BERTMULTI and ENDERANKER, do not produce satisfactory performance to be used for real applications. We do not advise people to use them directly in real applications as factual error detectors for dialogue summarization without further improvements.
E Intended Use of Existing Artifacts
The SAMSum (Gliwa et al., 2019) dataset is shared on terms of the AttributionNonCommercialNoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. We provide additional information (i.e., model-generated summaries and human annotations) without modifying the original data (i.e., dialogues and reference summaries).
Identifying Wrong Facts in Summaries of Dialogues
Toggle Instructions
Directions
Main Task Part 3: Find Mistakes
Using the article as ground-truth, find mistakes in the highlighted sentence in the summary, and categorize the mistakes. To help you, we underline the sentences in the article that are likely to contain relevant information. However, this is for guidance only: you should not restrict yourself to the highlighted sentences when answering the questions.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section Limitations between conclusion and References

heading: 3 A2. Did you discuss any potential risks of your work?
text: Appendix D

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Our created artifacts are provided in an anonymous github repository. The artifacts we use are mentioned in Section 3.1; Section 4.1; Appendix A.2 and A.3.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.1; Section 4.1; Appendix A.2 and A.3.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: The license is included in the anonymous github repositories.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix E.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The end of Section 3.1 mentions how we filter out offensive contents. The Ethics Statments section mentions that we protect annotators privacy.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The documentation of our data and code are provided in the anonymous github repositories, mentioned in the footnote of abstract.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Introduction, Table 5, Table 4, Appendix A.3.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.4, Appendix A

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.2, A.3

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.2, A.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Figure 2, Figure 3, Table 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.2, Appendix A.3.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Figure 6, 7, 8 and 9 in Appendix.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix B.4, Ethics Statement.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix B.6.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics Statement

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix B.5.

heading: Future Work
text: For future work, we may design error identification methods using SOIspecific thresholds rather than a universal threshold for all SOIs