SlowGuess/NAACL_2022_OCR huggingface dataset:
The model weights are initialized with ELECTRA (electra-base discriminator, 110M parameters, Clark et al., 2020), the structural adapters are pretrained using the release 3.0 of the AMR corpus containing 55,635 gold annotated AMR graphs, and the text adapters are pretrained using synthetic generated data. The adapters' hidden dimension is 32, which corresponds to about  $1.4\%$  of the parameters of the original ELECTRA encoders. The number of considered document graphs  $(k)$  is  $5$ . We report the test results when the balanced accuracy (BACC) on dev set is optimal. Following previous work (Kryscinski et al., 2020; Goyal and Durrett, 2021), we evaluate our models using BACC and Micro F1 scores.
We compare FACTGRAPH with different methods for factuality evaluation: two QA-based methods, namely QAGS (Wang et al., 2020) and QUALS (Nan et al., 2021), and FACTCC (Kryscinski et al., 2020). We fine-tune FACTCC using the training set, that is, it is trained on both synthetic data and FACTCOLLECT. We call this approach FACTCC+.
Table 2 presents the results. QA-based approaches perform comparatively worse than FACTCC on CNN/DM, while QAGS has a general better performance than QUALs. FACTCC has a strong performance on CNN/DM, as it was trained
on synthetic data derived from this dataset. However, the FACTCC's performance does not transfer to XSum. FACTCC+ has a large increase in performance, especially on XSum, demonstrating the importance of human-annotated data for training improved factuality models.
FACTGRAPH outperforms FACTCC+ by 2.4 BACC points in both subsets and by 10.3 BACC in XSum, even though FACTCC+ was pretrained on millions of synthetic examples. This indicates that considering semantic representations is beneficial for factuality evaluation and FACTGRAPH can be trained on a small number of annotated examples. Pretraining structural adapters improves the performance on CNN/DM and XSum. Finally, FACTGRAPH's performance further improves when both structural and text adapters are pretrained, improving over FACTCC+ by 3.7 BACC points. $^{4}$
5.1 Correlation with Human Judgments
We also evaluate the model performance using correlations with human judgments of factuality (Pagnoni et al., 2021). In this experiment, FACTCC+ and FACTGRAPH are trained with the FACTCOLLECT data without the Pagnoni et al. (2021)'s subset, which is used as dev and test sets, according to its split. For both models, following Pagnoni et al. (2021), we obtain a binary factuality label for each sentence and take the average of these labels as the final summary score. We use the official script to calculate the correlations.
AMR and Factuality. We investigate whether SMATCH (Cai and Knight, 2013), a metric that measures the degree of overlap between two AMRs, correlates with factuality judgments. We calculate the SMATCH score between all the summary sentence graphs and  $k$  document sentence graphs, with  $k \in \{1, 3, 5\}$ . We obtain one score per summary
$^4$ FACTGRAPH is significantly better than FACTCC+ with  $p < 0.05$  on both BACC and F1 scores.
<sup>5</sup>https://github.com/artidoro/frank
$^{3}$ Hyperparameter details and pretraining procedures are described in Appendix B.
All data	CNN/DM	XSum
Pearson, Spearman	ρ	p-val	r	p-val	ρ	p-val	r	p-val	ρ	p-val	r	p-val
BLEU	.10	.00	.05	.02	.06	.06	.07	.02	.16	.00	.15	.00
METEOR	.13	.00	.10	.00	.11	.00	.11	.00	.16	.00	.08	.01
ROUGE-L	.13	.00	.09	.00	.09	.00	.10	.00	.17	.00	.09	.01
BERTSCORE	.16	.00	.11	.00	.13	.00	.12	.00	.19	.00	.10	.00
SMATCH-AMR1	.07	.00	-.01	.62	.07	.02	.03	.26	.09	.01	.07	.05
SMATCH-AMR3	.11	.00	.10	.00	.15	.00	.14	.00	.06	.10	.04	.21
SMATCH-AMR5	.13	.00	.13	.00	.17	.00	.16	.00	.05	.17	.04	.28
SMATCH-AMRref	.08	.00	.03	.20	.05	.12	.03	.35	.13	.00	.08	.02
QAGS	.22	.00	.23	.00	.34	.00	.27	.00	.07	.05	.06	.09
QUALS	.22	.00	.19	.00	.31	.00	.27	.00	.14	.00	.07	.03
DAE	.17	.00	.20	.00	.27	.00	.22	.00	.03	.38	.33	.00
FACTCC	.20	.00	.29	.00	.36	.00	.30	.00	.06	.07	.19	.00
FACTCC+	.32	.00	.38	.00	.40	.00	.28	.00	.24	.00	.16	.00
FACTGRAPH	.35	.00	.42	.00	.45	.00	.34	.00	.30	.00	.49	.00
Table 3: Partial Pearson and Spearman correlation coefficients and p-values between human judgements and methods scores for the test split of Pagnoni et al. (2021).
sentence by maxing over its scores with the sentence graphs, then averaging over the summary sentence scores to obtain the summary-level score. We also calculate the SMATCH between the generated summary and the reference summary graphs. As shown in Table 3, SMATCH approaches have a small but consistent correlation, slightly improving over n-gram based metrics (e.g., METEOR and ROUGE-L) in CNN/DM, suggesting that AMR, which has a higher level of abstraction than plain text, may be a semantic representation alternative to content verification.
QA-based approaches have higher correlation on the CNN/DM dataset than XSum where their correlation is relatively reduced, and DAE shows higher Spearman correlation than FACTCC on XSum. FACTCC+ and FACTGRAPH, which are trained on data from FACTCOLLECT, have a overall higher performance than models trained on synthetic data, such as FACTCC, again demonstrating the importance of the human-annotation signal when training factuality evaluation approaches. Finally, FACTGRAPH has the highest correlations in both datasets, with a large improvement in XSum, suggesting that representing facts as semantic graphs is effective for more abstractive summaries.
Types of Errors. Figure 3 shows the influence of the different types of factuality errors (Pagnoni et al., 2021) for each approach. Semantic Frame Errors are errors in a frame, core, and non-core frame elements.<sup>6</sup> Discourse Errors extend beyond a single semantic frame introducing erroneous links between discourse segments. Content Verifiability
Figure 3: Variation in partial Pearson correlation when omitting error types. Higher variation indicates greater influence of an error type in the overall correlation.
Errors capture cases when it is not possible to verify the summary against the source document due to the difficulty in aligning it to the source. Note that whereas BERTSCORE strongly correlates with content verifiability errors as it is a token-level similarity metric, the other methods improve in Semantic Frame Errors. FACTGRAPH has the highest performance suggesting that graph-based MRs are able to capture different semantic errors well. In particular, FACTGRAPH improves in capturing content verifiability errors by  $48.2\%$ , suggesting that representing facts using AMR is helpful.
5.2 Edge-level Factuality Classification
We assess factuality beyond sentence-level with FACTGRAPH-E (§3.5). We train and evaluate the model against the sentence-level factuality data from Maynez et al. (2020). In this dataset, human annotations for sentence and span levels are
7Refer to Pagnoni et al. (2021) for a detailed description of the error categories and the correlation computations.
${}^{6}$  A semantic frame is a representation of an event, relation, or state (Baker et al., 1998).
Sentence-level models	BACC
Sent-Factuality (Goyal and Durrett, 2021)	65.6
FACTGRAPH	74.9
Edge-level models	BACC
DAE (Goyal and Durrett, 2021)	78.7
FACTGRAPH-E	81.1
available. We derive the edge labels required for FACTGRAPH-E training as follows: For each edge in the summary graph, if one of the nodes connected to this edge is aligned with a word that belongs to a span labeled as non-factual, the edge is annotated as non-factual. $^{8}$  Summary-level labels are obtained from edge-level predictions: if any edge in the summary graph is classified as non-factual, the summary is labeled as non-factual. We use the same splits from Goyal and Durrett (2021). $^{9}$  We compare FACTGRAPH-E with DAE and additionally with a sentence-level baseline (Goyal and Durrett, 2021) and FACTGRAPH.
Table 4 shows that the edge-level factuality classification gives better performance than sentence-level classification, and FACTGRAPH performs better in both sentence and edge classification levels. FACTGRAPH-E outperforms DAE, demonstrating that training on subsentence-level factuality annotations enables it to accurately predict edge-level factuality and output summary-level factuality.
Finally, while the semantic representations contribute to overall performance, extracting those representations adds some overhead in preprocessing time (and slightly more in inference time), as shown in Appendix C.
5.3 Model Ablations
In Table 5, we report an ablation study on the impact of distinct FACTGRAPH's components. First, note that only encoding the textual information leads to better performance than just encoding graphs. This is expected since pretrained encoders are known for good performance in NLP textual tasks due to their transfer learning capabilities and the full document text encodes more information than the selected  $k$  document graphs. Moreover, AMR representations abstract aspects such as verb tenses, making the graphs agnostic regarding more
Table 4: Sentence-level BACC in human-annotated XSum generated summaries (Maynez et al., 2020).
Model	BACC	F1
Only graphs	77.7	78.0
Only text	88.4	88.6
FACTGRAPH	91.2	91.3
Table 5: Ablation study for different components of the model in the FACTCOLLECT's dev set.
fine-grained information. However, this is compensated in FACTGRAPH, which captures coarse-grained details from the text modality. Future work can consider incorporating such information into the graph representation in order to improve the factuality assessment.
Ultimately, FACTGRAPH, which uses both document and summary graphs, gives the overall best performance, demonstrating that semantic graph representations complement the text representation and are beneficial for factuality evaluation.
Number of Document Graphs Table 6 shows the influence of the number of considered document graphs measured on FACTCOLLECT's dev set performance. Note that generally more document graphs leads to better performance with a peak in 5. This suggests that using all graph sentences from the source document is not required for better performance. Moreover, the results indicate that our strategy of selecting document graphs using the contextual representations of the document sentences which are compared to the summary performs well in practice.
We additionally present the performance of FACTGRAPH with other semantic representations in Appendix D.
5.4 Comparison to Full Fine-tuning
FACTGRAPH only trains adapter weights that are placed into each layer of both text and graph encoders. We compare FACTGRAPH with a model with similar architecture, with both text and graph encoders, but without (structural) adapter layers. We then fine-tune all the model parameters. Table 7 shows that FACTGRAPH performs better even though it trains only  $1.4\%$  of the parameters of the fully fine-tuned model, suggesting that the structural adapters help to adapt the graph encoder to semantic graph representations.
FACTGRAPH-E computes factuality scores for each edge of the AMR summary graph and those predictions are aggregated to generate a sentence
<sup>8</sup>We use the JAMR aligner (Flanigan et al., 2014) to obtain node-to-word alignments.
<sup>9</sup>We sample 100 datapoints from the training set as dev set to execute hyperparameter search.
Article: Margaret Fleming, 36, was last seen at her home in Inverkip by her two carers at about 17:40 on Friday 28 October. She is described as about 5ft 5in tall, [...] . Police had said they were trying to build a picture of Ms Fleming's life, part of which she kept "quite private". When last seen, she was wearing a green tartan fleece[...] . She also had a satchel-type handbag. A police spokesman said: "There is a specialist search team combing the area around where the missing person was last seen, this includes in the garden of her last known address." [...] The detective said that Ms Fleming was a student at James Watt College in Greenock between 1996 and 1997. He said he was keen to speak to anyone who remembered her from then, and who might have been in touch with her over the years.
Figure 4: An example of a document, its generated summary and factuality predictions for word pairs, based on the dependency graph (DAE) versus AMR graph (FACTGRAPH-E).  $+ / -$  means the predicted label for that edge.
# Graphs	BACC	F1
1	90.1	90.3
3	90.9	91.0
5 (final)	91.2	91.3
7	89.8	90.0
level label (§5.2). Alternatively, it is possible to identify specific inconsistencies in the generated summary based on the AMR graph structure. This factuality information at subsentence-level can provide deeper insights on the kinds of factual inconsistencies made by different summarization models (Maynez et al., 2020) and can supply text generation approaches with localized signals for training (Cao et al., 2020; Zhu et al., 2021).
Figure 4 shows a document, its generated summary, and factuality edge predictions by DAE and FACTGRAPH-E.10 First, note that since DAE uses dependency arcs and FACTGRAPH-E is based on AMR, the sets of edges in both approaches, that is, the relations between nodes and hence words, are different. Second, both methods are able to detect the hallucination six years, which was never mentioned in the source document. However, DAE does not consider that police appealed for help in tracing is factual whereas FACTGRAPH-E captures it. This piece of information is related to a span in the document with a very different but semantically related form (highlighted in bold in Figure 4). This poses challenges to DAE, since it classifies semantic relations independently and only considers the
Table 6: Effect in the FACTCOLLECT's dev set of the number of considered AMR graphs from the document.
BACC	F1	Parameters
Fully fine-tuned	90.3	90.3	100.0%
FACTGRAPH	91.2	91.3	1.4%
Table 7: Comparison between FACTGRAPH and fully fine-tuning in the dev set of FACTCOLLECT.
text surface. On the other hand, FACTGRAPH-E matches the summary against the document not only at text surface level but semantic level.
We presented FACTGRAPH, a graph-based approach to explicitly encode facts using meaning representations to identify factual errors in generated text. We provided an extensive evaluation of our approach and showed that it significantly improves results on different factuality benchmarks for summarization, indicating that structured semantic representations are beneficial to factuality evaluation. Future work includes (i) exploring approaches to develop document-level semantic graphs (Naseem et al., 2021), (ii) an explainable graph-based component to highlight hallucinations and (iii) to combine different meaning representations in order to capture distinct semantic aspects.
We thank the anonymous reviewers for their valuable feedback. We also thank Shiyue Zhang, Kevin Small, and Yang Liu for their feedback on this work. Leonardo F. R. Ribeiro has been supported by the German Research Foundation (DFG) as part of the Research Training Group "Adaptive Preparation of Information form Heterogeneous Sources" (AIPHES, GRK 1994/1).
10 Appendix E presents the complete AMR and dependency summary graphs.
Impact Statement
In this paper, we study the problem of detecting factual inconsistencies in summaries generated from input documents. The proposed models better consider the text internal meaning structure and could benefit general generation applications by evaluating their output regarding factual consistency, which could ensure that these systems are more trustworthy. This work is built using semantic representations extracted using AMR parsers. In this way, the quality of the parser used to generate the semantic representations can significantly impact the results of our models. In our work, we mitigate this risk by employing a state-of-the-art AMR parser (Bevilacqua et al., 2021).
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL '98/COLING '98, page 86-90, USA. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, IJCAI'07, page 2670-2676, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273-283, Melbourne, Australia. Association for Computational Linguistics.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12564-12573.
Mihaela Bornea, Ramon Fernandez Astudillo, Tahira Naseem, Nandana Mihindukulasooriya, Ibrahim Abdelaziz, Pavan Kapanipathi, Radu Florian, and Salim
Roukos. 2021. Learning to transpile AMR into SPARQL. arXiv e-prints.
Deng Cai and Wai Lam. 2020. AMR parsing via graph- sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301, Online. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251-6258, Online. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Moye Chen, Wei Li, Jiachen Liu, Xinyan Xiao, Hua Wu, and Haifeng Wang. 2021. SgSum:transforming multi-document summarization into sub-graph selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4063-4074, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Marco Damonte, Rahul Goel, and Tagyoung Chung. 2019. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 16-23, Minneapolis, Minnesota. Association for Computational Linguistics.
Shibhansh Dohare, Vivek Gupta, and Harish Karnick. 2018. Unsupervised semantic abstractive summarization. In Proceedings of ACL 2018, Student Research Workshop, pages 74-83, Melbourne, Australia. Association for Computational Linguistics.
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2021. Analyzing the abstractiveness-factuality tradeoff with nonlinear abstractiveness constraints. arXiv e-prints.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4186-4196, Hong Kong, China. Association for Computational Linguistics.
Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Baltimore, Maryland. Association for Computational Linguistics.
Qiankun Fu, Linfeng Song, Wenyu Du, and Yue Zhang. 2021. End-to-end AMR coreference resolution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4204-4214, Online. Association for Computational Linguistics.
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD '19, page 166-175, New York, NY, USA. Association for Computing Machinery.
Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3592-3603, Online. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,