Venue: NAACL 2022

# FactGraph: Evaluating Factuality in Summarization

with Semantic Graph Representations

Leonardo F. R. Ribeiro\({}^{\dagger}\)1, Mengwen Liu\({}^{\ddagger}\), Iryna Gurevych\({}^{\dagger}\), Markus Dreyer\({}^{\ddagger}\), Mohit Bansal\({}^{\ddagger,\lx@sectionsign}\)

Footnote 1: Work done as an intern at Amazon Alexa AI.

###### Abstract

Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not _factually consistent_ with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FactGraph, a method that decomposes the document and the summary into structured _meaning representations_ (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FactGraph encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph connectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%. Furthermore, FactGraph improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.2

Footnote 2: Our code is publicly available at [https://github.com/amazon-research/fact-graph](https://github.com/amazon-research/fact-graph)

+UKP Lab, Technical University of Darmstadt

++Amazon Alexa AI, $UNC Chapel Hill

ribeiro@aiphes.tu-darmstadt.de, {mengwliu, mdredreyer, mobansal}@amazon.com

gurevych@ukp.informatik.tu-darmstadt.de, mbansal@cs.unc.edu

## 1 Introduction

Recent summarization approaches based on pretrained language models (LM) have established a new level of performance Zhang et al. (2020); Lewis et al. (2020), generating summaries that are grammatically fluent and capable of combining salient parts of the source document. However, current models suffer from a severe limitation, generating summaries that are _not factually consistent_, that is, the content of the summary does not meet the facts of the source document, an issue also known as _hallucination_. Previous studies Cao et al. (2018); Falke et al. (2019); Maynez et al. (2020); Dreyer et al. (2021) report rates of hallucinations in generated summaries ranging from 30% to over 70%. In the face of such a challenge, recent works employ promising ideas such as question answering (QA) Durmus et al. (2020); Nan et al. (2021) and weakly supervised approaches Kryscinski et al. (2020) to assess factuality. Another line of work explores dependency arc entailment to improve the localization of subsentence-level errors within generated summaries Goyal and Durrett (2020).

However, these methods have a reduced correlation with human judgments and may not capture well semantic errors Pagnoni et al. (2021). One reason for the poor performance is the lack of good quality factuality training data. Second, it is challenging to properly encode core semantic content from the document and summary Lee et al. (2021) and reason over salient pieces of information in

Figure 1: Example of (a) a document, (b) a summary, and (c) the corresponding document and (d) summary graph-based meaning representations. The summary graph does not contain the “consider” node, indicating a factual error (red dashed edge).



order to assess the summary factuality. Third, previous work (DAE, Goyal and Durrett, 2021) treats semantic relations as isolated units, not simultaneously considering the entire semantic structure of _both_ document and summary texts.

To mitigate the above issues, we explore _meaning representations_ (MR) as a form of content representation for factuality evaluation. We present FactGraph, a novel graph-enhanced approach that incorporates core information from the document and the summary into the factuality model using graph-based MRs, which are more suitable for factuality evaluation: As shown in Figure 1, graph-based MRs capture semantic relations between entities, abstracting away from syntactic structure and producing a canonical representation of meaning.

Different from previous methods (Kryscinski et al., 2020; Goyal and Durrett, 2021), FactGraph is a dual approach which encodes both text and graph modalities, better integrating linguistic knowledge and structured semantic knowledge. As shown in Figure 2, it is composed of parameter-efficient text and graph encoders which share the same pretrained model and differ by their adapter weights (Houlsby et al., 2019). The texts from the document and summary are encoded using the adapter-based text encoder whereas the entire semantic structures that represent document and summary facts are used as input to the graph encoder augmented structure-aware adapters Ribeiro et al. (2021b). The representations of the two modalities thus are combined to generate the factuality score.

In particular, we explore _Abstract Meaning Representation_ (AMR) Banarescu et al. (2013), a semantic formalism that has received much research interest Song et al. (2018); Guo et al. (2019); Ribeiro et al. (2019, 2021a); Opitz et al. (2020, 2021); Fu et al. (2021) and has been shown to benefit downstream tasks such as spoken language understanding Damonte et al. (2019), machine translation Song et al. (2019), commonsense reasoning Lim et al. (2020), and question answering Kapanipathi et al. (2021); Bornea et al. (2021).

Intuitively, AMR provides important benefits: First, it encodes core concepts as it strives for a more logical and less syntactic representation, which has been shown to benefit text summarization Hardy and Vlachos (2018); Dohare et al. (2018); Lee et al. (2021). Furthermore, AMR captures semantics at a high level of abstraction explicitly modeling relations in the text and reducing the negative influence of diverse text surface variances with the same meaning. Lastly, recent studies Dryer et al. (2021); Ladhak et al. (2021) demonstrate that there is a trade-off between factuality and abstractiveness. Structured semantic representations are potentially beneficial for reducing data sparsity and localizing generation errors in abstractive scenarios. Figure 1 shows examples of (c) document and (d) summary AMRs, where the summary AMR is missing a crucial modifying node present in the document AMR, which indicates a factual error in the summary.

We consolidate a factuality dataset with human annotations derived from previous works Wang et al. (2020); Kryscinski et al. (2020); Maynez et al. (2020); Pagnoni et al. (2021). This dataset is constructed from the widely-used CNN/DM Hermann et al. (2015) and XSum Nallapati et al. (2016) benchmarks. Extensive experimental results demonstrate that FactGraph achieves substantial improvements over previous approaches, improving factuality performance by up to \(15\%\) and correlation with human judgments by up to \(10\%\), capturing more content verifiability errors and better classifying factuality in semantic relations.

## 2 Related Work

Evaluating Factuality.Recently, there has been a surge of new methods for factuality evaluation in text generation, especially for summarization. Falke et al. (2019) propose to rerank summary hypotheses generated via beam search based on entailment scores to the source document. Kryscinski et al. (2020) introduce FactCC, a model-based approach trained on artificially generated data, to measure if the summary can be entailed by the source document in order to assess the summary factuality. QA-based methods Wang et al. (2020); Durmus et al. (2020); Honovich et al. (2021); Nan et al. (2021) generate questions from the document and summary, and compare the corresponding answers in order to assess factuality. Xie et al. (2021) formulate causal relationships among the document, summary, and language prior to evaluate the factuality via counterfactual estimation.

Categorizing Factual Errors.A thread of analysis work has focused on identifying different categories of factual errors in summarization. Maynez et al. (2020) show that semantic inference-based automatic measures are better representations of summarization quality, whereas Pagnoni et al. (2021) 

propose a linguistically grounded typology of factual errors and develop a fine-grained benchmark for factuality evaluation, moving to a fine-grained measure, instead of using a binary evaluation. Fabbri et al. (2021) introduce different resources for summarization evaluation which include a toolkit for evaluating summarization models.

Factuality versus AbstractivenessRecent works Dreyer et al. (2021); Ladhak et al. (2021) investigate the trade-off between factuality and abstractiveness of summaries and observe that factuality tends to drop with increased abstractiveness. Semantic graphs are uniquely suitable to detect factual errors in abstractive summaries as they abstract away from the lexical surface forms of documents and summaries, enabling direct comparisons of the underlying semantic concepts and relations of a document-summary pair.

Graph-based Representations for Summarization.A growing body of work focuses on using graph-based representations for improving summarization. Whereas different approaches encode graphs into neural models for multi-document summarization Fan et al. (2019); Li et al. (2020); Pasunuru et al. (2021); Wu et al. (2021); Chen et al. (2021), AMR structures have been shown to benefit both document representation and summary generation Liu et al. (2015); Liao et al. (2018); Hardy and Vlachos (2018); Dohare et al. (2018) and have the potential of improving controllability in summarization Lee et al. (2021). The above works are related to FactGraph as they use semantic graphs for content representation, but also different because they utilize graphs for the downstream summarization task, whereas FactGraph employs them for factuality evaluation.

Semantic Representations for Factuality Evaluation.More closely related to our work, Goodrich et al. (2019) extract tuples from the document and summary and measure the factual consistency by overlapping metrics. Xu et al. (2020) weight facts present in the source document according to the facts from the gold summary using contextual embeddings, and verify whether a generated summary is able to capture the same facts as the target. Recently, dependency arc entailment DAE, Goyal and Durrett (2020) is used to measure subsentence-level factuality by classifying pairs of words defined by dependency arcs which often describe semantic relations. However, FactGraph is considerably different from those approaches, since it explicitly encodes the entire graph semantic structure into the model. Moreover, while DAE considers semantic edge relations of the summary only, FactGraph encodes the semantic structures of both the input document and summary leading to better factuality performance at both sentence and subsentence levels.

## 3 FactGraph Model

We introduce FactGraph, a method that employs semantic graph representations for factuality evaluation in text summarization, describing its intuition (SS3.3) and defining it formally (SS3.4).

### Problem Statement

Given a source document \(D\) and a _sentence-level_ summary \(S\), we aim to check whether \(S\) is _factual_ with respect to \(D\). For each sentence \(d\in D\) we extract a semantic graph \({\mathcal{G}}_{d}\). Similarly, for the summary sentence \(S\) we extract its semantic graph \({\mathcal{G}}_{s}\). We use texts and graphs from both document and summary for factuality evaluation. Sentence-level summary predictions can be aggregated to generate a factuality score for a multi-sentence summary.

### Extracting Semantic Graphs

We select AMR as our MR, but FactGraph can be used with other graph-based semantic representations, such as OpenIE Banko et al. (2007). AMR is a linguistically-grounded semantic formalism that represents the meaning of a sentence as a rooted graph, where nodes are _concepts_ and edges are _semantic relations_. AMR abstracts away from surface text, aiming to produce a more language-neutral representation of meaning. We use a state-of-the-art AMR parser Bevilacqua et al. (2021) to extract an AMR graph \({\mathcal{G}}_{a}=({\mathcal{V}}_{a},{\mathcal{E}}_{a},{\mathcal{R}}_{a})\) with a node set \({\mathcal{V}}_{a}\) and labeled edges \((u,r,v)\in{\mathcal{E}}_{a}\), where \(u,v\in{\mathcal{V}}_{a}\) and \(r\in{\mathcal{R}}_{a}\) is a relation type. Each \({\mathcal{G}}_{a}\) aims to explicitly represent the core concepts in each sentence. Figure 1 shows an example of a (b) sentence and its (d) corresponding AMR graph.2

Footnote 2: Appendix A presents other examples of AMRs extracted from sentences of documents and generated summaries.

Graph Representation.We convert each \({\mathcal{G}}_{a}\) into a bipartite graph \({\mathcal{G}}_{b}=({\mathcal{V}}_{b},{\mathcal{E}}_{b})\), replacing each labeled edge \((u,r,v)\in{\mathcal{E}}_{a}\) with two unlabeled edges \(\{(u,r),(r,v)\}\in{\mathcal{E}}_{b}\). Similar to Beck et al. (2018), this procedure transforms the graph into its unlabeled version. Pretrained models typically use 

a vocabulary with subword tokens, which makes it complicated to properly represent a graph using subword tokens as nodes. Inspired by Ribeiro et al. (2020, 2018), we transform each \({\cal G}_{b}\) into a new token graph \({\cal G}=({\cal V},{\cal E})\), where each token of a node \(v_{b}\in{\cal V}_{b}\) becomes a node \(v\in{\cal V}\). We convert each edge \((u_{b},v_{b})\in{\cal E}_{b}\) into a set of edges and connect every token of \(u_{b}\) to every token of \(v_{b}\).

### Intuition of Semantic Representation

In order to represent facts to better assess the summary factuality, we draw inspiration from traditional approaches to summarization that condense the source document to a set of "semantic units" Liu et al. (2015); Liao et al. (2018). Intuitively, the semantic graphs from the source document represent the core factual information, explicitly modeling relations in the text, whereas the semantic summary graph captures the essential content information in a summary Lee et al. (2021). The document graphs can be compared with the summary graph, measuring the degree of semantic overlap to assess factuality Cai and Knight (2013).

Recently, sets of fact triples from summaries were used to estimate factual accuracy Goodrich et al. (2019). That approach is related to FactGraph as it uses graph-based MRs, but also different because it compares the reference and the generated summary, whereas we compare the generated summary with the input document. Moreover, differently from Goodrich et al. (2019), FactGraph explicitly encodes the semantic structures using a graph encoder and employs AMR as a semantic representation. Finally, in contrast to DAE Goyal and Durrett (2021), which focuses only on extracting summary graph representations, FactGraph uses semantic graphs for both document and summary.

### Model

Figure 2 illustrates FactGraph, which is composed of text and graph encoders. The text encoder, denoted by \(E_{t}\), uses a pretrained encoder \(E\), augmented with adapter modules which receives the summary \(S\) and document \(D\) and outputs a contextual text representation. Conversely, the graph encoder, denoted by \(E_{g}\), uses the same \(E\), but is augmented with structure-aware adapters. \(E_{g}\) receives the summary and multiple document semantic graphs corresponding to its sentences, and outputs graph-aware contextual representations that are used to generate the final graph representation. During training, only adapter weights are trained, whereas the weights from \(E\) are kept frozen. Finally, both graph and text representations are concatenated and fed to a final classifier, which predicts whether the summary is factual or not.

Text Encoder.We employ an adapter module before and after the feed-forward sub-layer of each layer of the encoder. We modify the adapter architecture from Houlsby et al. (2019). We compute the adapter representation for each token \(i\) at each layer \(l\), given the token representation \(\mathbf{h}^{l}_{i}\), as follows:

\[\hat{\mathbf{z}}^{l}_{i}=\mathbf{W}^{l}_{o}(\sigma(\mathbf{W}^{l}_{p~{}\text{LN}}(\mathbf{h}^ {l}_{i})))+\mathbf{h}^{l}_{i}\,, \tag{1}\]

where \(\sigma\) is the activation function and \(\text{{LN}}(\cdot)\) denotes layer normalization. \(\mathbf{W}^{l}_{o}\in\mathbb{R}^{d\times m}\) and \(\mathbf{W}^{l}_{p}\in\mathbb{R}^{m\times d}\) are adapter parameters. The representation of the [CLS] token is used as the final textual representation, denoted by \(\mathbf{t}\).

Graph Encoder.In order to re-purpose the pre-trained encoder to structured inputs, we employ a structural adapter Ribeiro et al. (2021). In particular, for each node \(v\in{\cal V}\), given the hidden

Figure 2: Overview of FactGraph. A sentence-level summary and document graphs are encoded by the graph encoder with structure-aware adapters. Text and graph encoders use the same pretrained model and only the adapters parameters are trained.

 

representation \({\mathbf{h}}_{v}^{l}\), the encoder layer \(l\) computes:

\[\begin{split}&{\mathbf{g}}_{v}^{l}=\mathsf{GraphConv}_{l}(\texttt{LN}({\mathbf{ h}}_{v}^{l}),\{\texttt{LN}({\mathbf{h}}_{u}^{l})\colon u\in\mathcal{N}(v)\})\\ &{\mathbf{z}}_{v}^{l}={\mathbf{W}}_{c}^{l}\sigma({\mathbf{g}}_{v}^{l})+{\mathbf{h }}_{v}^{l}\,,\\ \end{split} \tag{2}\]

where \(\mathcal{N}(v)\) is the neighborhood of the node \(v\) in \(\mathcal{G}\) and \({\bf W}_{e}^{l}\in\mathbb{R}^{d\times m}\) is an adapter parameter. \(\mathsf{GraphConv}_{l}(\cdot)\) is the graph convolution that computes the representation of \(v\) based on its _neighbors_ in the graph. We employ a Relational Graph Convolutional Network (Schlichtkrull et al., 2018) as graph convolution, which considers differences in the incoming and outgoing relations. Since AMRs are directed graphs, encoding edge directions is beneficial for downstream performance (Ribeiro et al., 2019). The structural adapter is placed before, whereas the normal adapter is kept after the feed-forward sub-layer of each encoder layer.

We calculate the final representation of each graph from the pooling denoted as \({\mathbf{z}}^{\text{G}}=\{{\mathbf{z}}_{v}^{(L)}~{}|~{}v\in\mathcal{V}\}\), where \({\mathbf{z}}_{v}^{(L)}\) is the final representation of \(v\). Thus, we use a multi-head self-attention (Vaswani et al., 2017) layer to estimate to what extent each sentence graph contributes to the document semantic representation based on the summary graph. This mechanism allows encoding a global document representation based on the summary graph. In particular, each attention head computes:

\[\begin{split}\alpha_{i}&=\mathtt{Attn}({\mathbf{ z}}_{s}^{\text{G}},{\mathbf{z}}_{i}^{\text{G}}),\\ {\mathbf{g}}&=\sum\nolimits_{i=1}^{k}\alpha_{i}\,{\mathbf{W }}_{r}\,{\mathbf{z}}_{i}^{\text{G}},\end{split} \tag{3}\]

where \({\mathbf{z}}_{s}^{\text{G}}\) is the final representation of \(\mathcal{G}_{s}\), \(k\) is the number of considered sentence graphs from the input document and \({\bf W}_{r}\in\mathbb{R}^{d\times d}\) is a parameter.

The final representation is derived from the text and graph representations, \({\mathbf{q}}=[{\mathbf{t}};{\mathbf{g}}]\), and fed into a classification layer that outputs a probability distribution over the labels \(y\) = {Factual, Non-Factual}.

### Edge-level Factuality Model

Inspired by Goyal and Durrett (2021), we evaluate the factuality at the edge level. In this setup, we use the same text and graph encoders; however, we encode the semantic graphs differently. In particular, we concatenate \(\mathcal{G}_{s}\) with each \(\mathcal{G}_{d}\in D\) and feed the concatenation to the graph encoder. The representation of a node \(v\in\mathcal{G}_{s}\) is calculated as:

\[\begin{split}&{\mathbf{r}}_{tv}=\sum\nolimits_{w\in A(v)}E_{t}(D;S)_{ w}\\ &{\mathbf{r}}_{g_{v}}=\sum\nolimits_{d=1}^{k}E_{g}(\mathcal{G}_{s}; \mathcal{G}_{d})_{v}\\ &{\mathbf{r}}_{v}=[{\mathbf{r}}_{tv};{\mathbf{r}}_{g_{v}}]\end{split} \tag{4}\]

where \(A(v)\) is the set of all summary words aligned with \(v\), and \({\mathbf{r}}_{tv}\) and \({\mathbf{r}}_{g_{v}}\) are the word and node representations, respectively. Edge representations are derived for each AMR edge \((u,v)\in\mathcal{E}:{\mathbf{r}}_{e}=[{\mathbf{r}}_{u};{\mathbf{r}}_{v}]\). The edge representation \({\mathbf{r}}_{e}\) is fed into a classification layer that outputs a probability distribution over the output labels (\(y_{e}\) = {Factual, Non-Factual}). We assign the label _non-factual_ for an edge in \(\mathcal{G}_{s}\) if one of the nodes in this edge is aligned with a word that belongs to a span annotated as _non-factual_. Otherwise, the edge is assigned the label _factual_. We call this variant FactGraph-E.

## 4 Experimental Setup

### Data

One of the main challenges in developing models for factuality evaluation is the lack of training data. Existing synthetic data generation approaches are not well-suited to factuality evaluation of current summarization models and human-annotated data can improve factuality models (Goyal and Durrett, 2021). In order to have a more effective training signal, we gather human annotations from different sources and consolidate a factuality dataset that can be used to train FactGraph and other models.

The source collections of the dataset are presented in Table 1. The dataset covers two parts, namely CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016). CNN/DM contains news articles from two providers, CNN and DailyMail; while XSum contains BBC articles. CNN/DM has considerably lower levels of abstraction, and the summary exhibits high overlap with the article; a typical CNN/DM summary consists of several bullet points. In XSum, the first sentence is removed from an article and used as a summary, making it highly abstractive. After we remove duplicated annotations, the total number of datapoints is 9,567, which we divide into train (8,667), dev (300) and test (600) sets. We call this dataset FactCollect.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**Source** & **\# datapoints** & **Domain** \\ \hline Wang et al. (2020) & 953 & CNN/DM, XSum \\ Krysciński et al. (2020) & 1,434 & CNN/DM \\ Maynez et al. (2020) & 2,500 & XSum \\ Pagnoni et al. (2021) & 4,942 & CNN/DM, XSum \\ \hline Total & 9,829 & CNN/DM, XSum \\ \hline \hline \end{tabular}
\end{table}
Table 1: Consolidated human annotations.

 



### Method Details

We limit the number of considered document graphs due to efficiency reasons. In particular, we compute the pairwise cosine similarity between the embeddings of each sentence \(d\in D\) and the summary sentence \(S\), generated by Sentence Transformers Reimers and Gurevych (2019). We thus select \(k\) sentences from the source document with the highest scores to be used to generate the document semantic graphs.

The model weights are initialized with electra (electro-base discriminator, 110M parameters, Clark et al. (2020), the structural adapters are pretrained using the release 3.0 of the AMR corpus containing 55,635 gold annotated AMR graphs, and the text adapters are pretrained using synthetic generated data. The adapters' hidden dimension is 32, which corresponds to about 1.4% of the parameters of the original Electra encoders. The number of considered document graphs (\(k\)) is 5.3 We report the test results when the balanced accuracy (BACC) on dev set is optimal. Following previous work Kryscinski et al. (2020); Goyal and Durrett (2021), we evaluate our models using BACC and Micro F1 scores.

Footnote 3: Hyperparameter details and pretraining procedures are described in Appendix B.

## 5 Results and Analysis

We compare FactGraph with different methods for factuality evaluation: two QA-based methods, namely QAGS Wang et al. (2020) and QUALS Nan et al. (2021), and FactCC Kryscinski et al. (2020). We fine-tune FactCC using the training set, that is, it is trained on both synthetic data and FactCollect. We call this approach FactCC+.

Table 2 presents the results. QA-based approaches perform comparatively worse than FactCC on CNN/DM, while QAGS has a general better performance than QUALS. FactCC has a strong performance on CNN/DM, as it was trained on synthetic data derived from this dataset. However, the FactCC's performance does not transfer to XSum. FactCC+ has a large increase in performance, especially on XSum, demonstrating the importance of human-annotated data for training improved factuality models.

FactGraph outperforms FactCC+ by 2.4 BACC points in both subsets and by 10.3 BACC in XSum, even though FactCC+ was pretrained on millions of synthetic examples. This indicates that considering semantic representations is beneficial for factuality evaluation and FactGraph can be trained on a small number of annotated examples. Pretraining structural adapters improves the performance on CNN/DM and XSum. Finally, FactGraph's performance further improves when both structural and text adapters are pretrained, improving over FactCC+ by 3.7 BACC points.4

Footnote 4: FactGraph is significantly better than FactCC+ with \(p{<}0.05\) on both BCC and F1 scores.

### Correlation with Human Judgments

We also evaluate the model performance using correlations with human judgments of factuality Pagnoni et al. (2021). In this experiment, FactCC+ and FactGraph are trained with the FactCollect data without the Pagnoni et al. (2021)'s subset, which is used as dev and test sets, according to its split. For both models, following Pagnoni et al. (2021), we obtain a binary factuality label for each sentence and take the average of these labels as the final summary score. We use the official script to calculate the correlations.5

Footnote 5: [https://github.com/artidoro/frank](https://github.com/artidoro/frank)

AMR and Factuality.We investigate whether Smatch Cai and Knight (2013), a metric that measures the degree of overlap between two AMRs, correlates with factuality judgments. We calculate the Smatch score between all the summary sentence graphs and \(k\) document sentence graphs, with \(k\in\{1,3,5\}\). We obt

sentence by making over its scores with the sentence graphs, then averaging over the summary sentence scores to obtain the summary-level score. We also calculate the Smatch between the generated summary and the reference summary graphs. As shown in Table 3, Smatch approaches have a small but consistent correlation, slightly improving over n-gram based metrics (e.g., METEOR and Rouge-L) in CNN/DM, suggesting that AMR, which has a higher level of abstraction than plain text, may be a semantic representation alternative to content verification.

QA-based approaches have higher correlation on the CNN/DM dataset than XSum where their correlation is relatively reduced, and DAE shows higher Spearman correlation than FactCC on XSum. FactCC+ and FactGraph, which are trained on data from FactCollect, have a overall higher performance than models trained on synthetic data, such as FactCC, again demonstrating the importance of the human-annotation signal when training factuality evaluation approaches. Finally, FactGraph has the highest correlations in both datasets, with a large improvement in XSum, suggesting that representing facts as semantic graphs is effective for more abstractive summaries.

Types of Errors.Figure 3 shows the influence of the different types of factuality errors [14] for each approach. _Semantic Frame Errors_ are errors in a frame, core, and non-core frame elements.6_Discourse Errors_ extend beyond a single semantic frame introducing erroneous links between discourse segments. _Content Verifiability Errors_ capture cases when it is not possible to verify the summary against the source document due to the difficulty in aligning it to the source.7 Note that whereas BERTScore strongly correlates with content verifiability errors as it is a token-level similarity metric, the other methods improve in _Semantic Frame Errors_. FactGraph has the highest performance suggesting that graph-based MRs are able to capture different semantic errors well. In particular, FactGraph improves in capturing content verifiability errors by \(48.2\%\), suggesting that representing facts using AMR is helpful.

Footnote 7: Refer to Pagnoni et al. (2021) for a detailed description of the error categories and the correlation computations.

### Edge-level Factuality Classification

We assess factuality beyond sentence-level with FactGraph-E (SS3.5). We train and evaluate the model against the sentence-level factuality data from Maynez et al. (2020). In this dataset, human annotations for sentence and span levels are

Figure 3: Variation in partial Pearson correlation when omitting error types. Higher variation indicates greater influence of an error type in the overall correlation.

 

available. We derive the edge labels required for FactGraph-E training as follows: For each edge in the summary graph, if one of the nodes connected to this edge is aligned with a word that belongs to a span labeled as non-factual, the edge is annotated as non-factual.8 Summary-level labels are obtained from edge-level predictions: if any edge in the summary graph is classified as non-factual, the summary is labeled as non-factual. We use the same splits from Goyal and Durrett (2021).9 We compare FactGraph-E with DAE and additionally with a sentence-level baseline Goyal and Durrett (2021) and FactGraph.

Footnote 7: [https://github.com/facegraph-e/](https://github.com/facegraph-e/)

Table 4 shows that the edge-level factuality classification gives better performance than sentence-level classification, and FactGraph performs better in both sentence and edge classification levels. FactGraph-E outperforms DAE, demonstrating that training on subsentence-level factuality annotations enables it to accurately predict edge-level factuality and output summary-level factuality.

Finally, while the semantic representations contribute to overall performance, extracting those representations adds some overhead in preprocessing time (and slightly more in inference time), as shown in Appendix C.

### Model Ablations

In Table 5, we report an ablation study on the impact of distinct FactGraph's components. First, note that only encoding the textual information leads to better performance than just encoding graphs. This is expected since pretrained encoders are known for good performance in NLP textual tasks due to their transfer learning capabilities and the full document text encodes more information than the selected \(k\) document graphs. Moreover, AMR representations abstract aspects such as verb tenses, making the graphs agnostic regarding more fine-grained information. However, this is compensated in FactGraph, which captures coarse-grained details from the text modality. Future work can consider incorporating such information into the graph representation in order to improve the factuality assessment.

Ultimately, FactGraph, which uses both document and summary graphs, gives the overall best performance, demonstrating that semantic graph representations complement the text representation and are beneficial for factuality evaluation.

Number of Document GraphsTable 6 shows the influence of the number of considered document graphs measured on FactCollect's dev set performance. Note that generally more document graphs leads to better performance with a peak in 5. This suggests that using all graph sentences from the source document is not required for better performance. Moreover, the results indicate that our strategy of selecting document graphs using the contextual representations of the document sentences which are compared to the summary performs well in practice.

We additionally present the performance of FactGraph with other semantic representations in Appendix D.

### Comparison to Full Fine-tuning

FactGraph only trains adapter weights that are placed into each layer of both text and graph encoders. We compare FactGraph with a model with similar architecture, with both text and graph encoders, but without (structural) adapter layers. We then fine-tune all the model parameters. Table 7 shows that FactGraph performs better even though it trains only 1.4% of the parameters of the fully fine-tuned model, suggesting that the structural adapters help to adapt the graph encoder to semantic graph representations.

### Case Study

FactGraph-E computes factuality scores for each edge of the AMR summary graph and those predictions are aggregated to generate a sentence

\begin{table}
\begin{tabular}{l l} \hline \hline
**Sentence-level models** & **BACC** \\ Sent-Factuality (Goyal and Durrett, 2021) & 65.6 \\ FactGraph & 74.9 \\ \hline
**Edge-level models** & **BACC** \\ DAE (Goyal and Durrett, 2021) & 78.7 \\ FactGraph-E & **81.1** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Sentence-level BACC in human-annotated XSum generated summaries Maynez et al. (2020).

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **BACC** & **F1** \\ \hline Only graphs & 77.7 & 78.0 \\ Only text & 88.4 & 88.6 \\ FactGraph & 91.2 & 91.3 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study for different components of the model in the FactCollect’s dev set.

 

level label (SS5.2). Alternatively, it is possible to identify specific inconsistencies in the generated summary based on the AMR graph structure. This factuality information at subsence-level can provide deeper insights on the kinds of factual inconsistencies made by different summarization models Maynez et al. (2020) and can supply text generation approaches with localized signals for training Cao et al. (2020); Zhu et al. (2021).

Figure 4 shows a document, its generated summary, and factuality edge predictions by DAE and FactGraph-E.10 First, note that since DAE uses dependency arcs and FactGraph-E is based on AMR, the sets of edges in both approaches, that is, the relations between nodes and hence words, are different. Second, both methods are able to detect the hallucination _six years_, which was never mentioned in the source document. However, DAE does not consider that _police appealed for help in tracing_ is factual whereas FactGraph-E captures it. This piece of information is related to a span in the document with a very different but semantically related form (highlighted in bold in Figure 4). This poses challenges to DAE, since it classifies semantic relations independently and only considers the text surface. On the other hand, FactGraph-E matches the summary against the document not only at text surface level but semantic level.

Footnote 10: Appendix E presents the complete AMR and dependency summary graphs.

## 6 Conclusion

We presented FactGraph, a graph-based approach to explicitly encode facts using meaning representations to identify factual errors in generated text. We provided an extensive evaluation of our approach and showed that it significantly improves results on different factuality benchmarks for summarization, indicating that structured semantic representations are beneficial to factuality evaluation. Future work includes (i) exploring approaches to develop document-level semantic graphs Naseem et al. (2021), (ii) an explainable graph-based component to highlight hallucinations and (iii) to combine different meaning representations in order to capture distinct semantic aspects.

## Acknowledgments

We thank the anonymous reviewers for their valuable feedback. We also thank Shiyue Zhang, Kevin Small, and Yang Liu for their feedback on this work. Leonardo F. R. Ribeiro has been supported by the German Research Foundation (DFG) as part of the Research Training Group "Adaptive Preparation of Information form Heterogeneous Sources" (AIPHES, GRK 1994/1).

\begin{table}
\begin{tabular}{l c c} \hline \hline
**\# Graphs** & **BACC** & **F1** \\ \hline
1 & 90.1 & 90.3 \\
3 & 90.9 & 91.0 \\
5 (**final**) & 91.2 & 91.3 \\
7 & 89.8 & 90.0 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Effect in the FactCollect’s dev set of the number of considered AMR graphs from the document.

Figure 4: An example of a document, its generated summary and factuality predictions for word pairs, based on the dependency graph (DAE) versus AMR graph (FactGraph-E). \(+/-\) means the predicted label for that edge.

 



### Impact Statement

In this paper, we study the problem of detecting factual inconsistencies in summaries generated from input documents. The proposed models better consider the text internal meaning structure and could benefit general generation applications by evaluating their output regarding factual consistency, which could ensure that these systems are more trustworthy. This work is built using semantic representations extracted using AMR parsers. In this way, the quality of the parser used to generate the semantic representations can significantly impact the results of our models. In our work, we mitigate this risk by employing a state-of-the-art AMR parser (Bevilacqua et al., 2021).

## References

* D. Cai and W. Lam (2020)AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1290-1301. External Links: Link Cited by: SS4.1.
* S. Cai and K. Knight (2013)Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria, pp. 748-752. External Links: Link Cited by: SS4.1.
* M. Cao, Y. Dong, J. Wu, and J. C. Kit Cheung (2020)Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 6251-6258. External Links: Link Cited by: SS4.1.
* Z. Cao, F. Wei, W. Li, and S. Li (2018)Faithful to the original: fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence32 (1). External Links: Link Cited by: SS4.1.
* M. Chen, W. Li, J. Liu, X. Xiao, H. Wu, and H. Wang (2021)SGSum: transforming multi-document summarization into sub-graph selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 4063-4074. External Links: Link Cited by: SS4.1.
* K. Clark, M. Luong, Q. V. Le, and C. D. Manning (2020)Electra: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, Minneapolis, Minnesota, pp. 16-23. External Links: ISBN 978-3-319-1450-3, Document, Link Cited by: SS4.1.
* M. Damonte, R. Goel, and T. Chung (2019)Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), Minneapolis, Minnesota, pp. 16-23. External Links: Link Cited by: SS4.1.
* S. Dohare, V. Gupta, and H. Karnick (2018)Unsupervised semantic abstractive summarization. In Proceedings of ACL 2018, Student Research Workshop, Melbourne, Australia, pp. 74-83. External Links: Link Cited by: SS4.1.
* M. Dreyer, M. Liu, F. Nan, S. Atluri, and S. Ravi (2021)Analyzing the abstractiveness-factuality tradeoff with nonlinear abstractiveness constraints. arXiv e-prints. External Links: Link Cited by: SS4.1.
* E. Durmus, H. He, and M. Diab (2020)FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5055-5070, Online. Association for Computational Linguistics.
* Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Krycsinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. _Transactions of the Association for Computational Linguistics_, 9:391-409.
* Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
* Fan et al. (2019) Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4186-4196, Hong Kong, China. Association for Computational Linguistics.
* Flanigan et al. (2014) Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the Abstract Meaning Representation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1426-1436, Baltimore, Maryland. Association for Computational Linguistics.
* Fu et al. (2021) Qiankun Fu, Linfeng Song, Wenyu Du, and Yue Zhang. 2021. End-to-end AMR coreference resolution. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4204-4214, Online. Association for Computational Linguistics.
* Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining_, KDD '19, page 166-175, New York, NY, USA. Association for Computing Machinery.
* Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3592-3603, Online. Association for Computational Linguistics.
* Guyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1449-1462, Online. Association for Computational Linguistics.
* Guo et al. (2019) Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. _Transactions of the Association for Computational Linguistics_, 7:297-312.
* Hardy and Vlachos (2018) Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using Abstract Meaning Representation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 768-773, Brussels, Belgium. Association for Computational Linguistics.
* Volume 1_, NIPS'15, page 1693-1701, Cambridge, MA, USA. MIT Press.
* Honovich et al. (2021) Or Honorich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q\({}^{2}\): Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. _CoRR_, abs/2104.08202.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzewski, Bruna Morrone, Quentin De Laroussille, Andrea Gesmundo, Monta Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Kapanipathi et al. (2021) Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ramon Fernandez Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfio Gliozzo, Sairm Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunya Li, Francois Luis, Meliuvnwo Makondo, Namata Mihandukulasooriya, Taira Naseen, Sumit Neelam, Lucian Popa, Revanth Gangi Reddy, Ryan Riegel, Gaetano Rossiello, Udit Sharma, G P Shrivatsa Bhargav, and Mo Yu. 2021. Leveraging Abstract Meaning Representation for knowledge base question answering. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3884-3894, Online. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Knight et al. (2018) Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Ulf Hermjakob Kira Griffitt, Daniel Marcu, Martha Palmer, Tim O'Gorman, 

and Nathan Schneider. 2020. Abstract meaning representation (AMR) annotation release 3.0.
* Kryscinski et al. [2020] Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346, Online. Association for Computational Linguistics.
* Ladhak et al. [2021] Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2021. Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization. _arXiv e-prints_.
* Lee et al. [2021] Fei-Tzin Lee, Chris Kedzie, Nakul Verma, and Kathleen McKeown. 2021. An analysis of document graph construction methods for AMR summarization. _arXiv e-prints_.
* Lewis et al. [2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880.
* Li et al. [2020] Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6232-6243, Online. Association for Computational Linguistics.
* Liao et al. [2018] Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1178-1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Lim et al. [2020] Jungwoo Lim, Dongsuk Oh, Yoonna Jang, Kisu Yang, and Heuisceok Lim. 2020. I know what you asked: Graph path learning using AMR for commonsense reasoning. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 2459-2471, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Liu et al. [2015] Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1077-1086, Denver, Colorado. Association for Computational Linguistics.
* Manning et al. [2014] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
* Maynez et al. [2020] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919, Online. Association for Computational Linguistics.
* Nallapati et al. [2016] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
* Nan et al. [2021] Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6881-6894, Online. Association for Computational Linguistics.
* Naseem et al. [2021] Tahira Naseem, Austin Blodgett, Sadhana Kumarabel, Tim O'Gorman, Young-Suk Lee, Jeffrey Flanigan, Ramon Fernandez Astudillo, Radu Florian, Salim Roukos, and Nathan Schneider. 2021. DocAMR: Multi-sentence AMR representation and evaluation. _arXiv e-prints_.
* Opitz et al. [2021] Jurii Opitz, Angel Daza, and Anette Frank. 2021. Weiskeiler-leman in the bamboo: Novel AMR graph metrics and a benchmark for AMR graph similarity. _Transactions of the Association for Computational Linguistics_, 9:1425-1441.
* Opitz et al. [2020] Jurii Opitz, Letitia Parcalabscu, and Anette Frank. 2020. AMR similarity metrics from principles. _Transactions of the Association for Computational Linguistics_, 8:522-538.
* Pagnoni et al. [2021] Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4812-4829, Online. Association for Computational Linguistics.
* Pasunuru et al. [2021] Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021. Efficiently summarizing text and graph encodings of multi-document clusters. In _Proceedings of the 2021 Conference of 

the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4768-4779, Online. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Ribeiro et al. (2019) Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3183-3194, Hong Kong, China. Association for Computational Linguistics.
* Ribeiro et al. (2021a) Leonardo F. R. Ribeiro, Jonas Pfeiffer, Yue Zhang, and Iryna Gurevych. 2021a. Smelting gold and silver for improved multilingual AMR-to-Text generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 742-750, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Ribeiro et al. (2020) Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. 2020. Modeling global and local node contexts for text generation from knowledge graphs. _Transactions of the Association for Computational Linguistics_, 8:589-604.
* Ribeiro et al. (2021b) Leonardo F. R. Ribeiro, Yue Zhang, and Iryna Gurevych. 2021b. Structural adapters in pretrained language models for AMR-to-text generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Punta Cana, November 7-11, 2021_.
* Schlichtkrull et al. (2018) Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In _ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings_, pages 593-607.
* Song et al. (2019) Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. _Transactions of the Association for Computational Linguistics_, 7:19-31.
* Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1616-1626, Melbourne, Australia. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5008-5020, Online. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtiowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv e-prints_.
* Wu et al. (2021) Wenhao Wu, Wei Li, Xinyan Xiao, Jiachen Liu, Ziqiang Cao, Sujian Li, Hua Wu, and Haifeng Wang. 2021. BASS: Boosting abstractive summarization with unified semantic graph. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6052-6067, Online. Association for Computational Linguistics.
* Xie et al. (2021) Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 100-110, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Xu et al. (2020) Xinnuo Xu, Ondrej Dusek, Jingyi Li, Verena Rieser, and Ioannis Konstas. 2020. Fact-based content weighting for evaluating abstractive summarisation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5071-5081, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 11328-11339. PMLR.
* Zhu et al. (2021) Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 718-733, Online. Association for Computational Linguistics.

 



## Appendes

In this supplementary material, we detail experiments' settings, additional model evaluations and additional information about semantic graph representations.

## Appendix A Additional Examples

Figure 6 shows examples of AMR representations generated from summaries and salient sentences from the respective source document.

## Appendix B Details of Models and Hyperparameters

The experiments were executed using the version \(3.3.1\) of the _transformers_ library released by Hugging Face Wolf et al. (2019). In Table 8, we report the hyperparameters used to train FactGraph. We use the Adam optimizer Kingma and Ba (2015) and employ a linearly decreasing learning rate schedule without warm-up. Mean pooling is used to calculate the final representation of each graph.

Structural Adapters' Pretraining.The structural adapters are pretrained using AMR graphs from the release 3.0 (LDC202OT02) of the AMR annotation corpus Knight et al. (2020).11 Similarly to the masked language modeling objective, we execute self-supervised node-level prediction, where we randomly mask and classify AMR nodes. The goal of this pretraining phase is to capture domain specific AMR knowledge by learning the regularities of the node/edge attributes distributed over graph structure.

Footnote 11: [https://catalog.ldc.upenn.edu/LDC2020T02](https://catalog.ldc.upenn.edu/LDC2020T02)

Text Adapters' Pretraining.The text adapters are pretrained using synthetically created data, which is generated by applying a series of rule-based transformations to the sentences of source documents Kryscinski et al. (2020). The pretraining task is to classify each summary sentence as factual or non-factual. The goal of this pretraining phase is to learn suitable text representations to better identify whether summary sentences remain factually consistent to the input document after the transformation.

## Appendix C Speed Comparison

FactGraph encodes the structured semantic representations that encode facts from the document and summary. Despite their effectiveness, extracting semantic graphs, such as AMR, is computationally expensive because current models employ Transformer-based encoder-decoder architectures based on Transformers and pretrained language models.

In this experiment, we compare the time execution of FactGraph-E and DAE in a sample of 1000 datapoints extracted from the XSum test set. In order to extract the semantic graphs, we investigate two AMR parsers, **Parser1**: a dual graph-sequence parser that iteratively refines an incrementally constructed graph Cai and Lam (2020) , and **Parser2**: a linearized graph model that employs BART Bevilacqua et al. (2021). The execution of the AMR parsers is parallelized using four Tesla V100 GPUs. We use Parser2 for the experiments in this paper since it is the current state of the art in AMR parsing, although it is slower in preprocessing than Parser1.

As shown in Table 9, DAE's preprocessing is much faster compared to this phase in FactGraph-E, since DAE employs a fast en

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Graph Type** & **BACC** & **FI1** \\ \hline Only text & 88.4 & 88.6 \\ FactGraph-Dependency & 90.2 & 90.3 \\ FactGraph-OpenIE & 90.5 & 90.7 \\ FactGraph-AMR & 91.2 & 91.3 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Effect of different graph representations in the factuality model (on the dev set of FactCollect).

\begin{table}
\begin{tabular}{l r} \hline \multicolumn{1}{r}{**Computing Infrastructure**} & 32GB NVIDIA U100 GPU \\
**Optimizer** & \multicolumn{1}{r}{Adam} \\
**Optimizer Params** & \multirow{2}{*}{\(\beta=(0.9,0.999),\epsilon=10^{-8}\)} \\
**learning rate** & & Le-4 \\
**Learning Rate Decay** & Linear \\
**Weight Decay** & & 0 \\
**Warmup Steps** & & 0 \\
**Maximum Gradient Norm** & & 1 \\
**batch size** & & 4 \\
**epoch** & & 10 \\
**Adapter dimension** & & 32 \\
**\# document graphs (\(k\))** & & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameter settings for our methods.

\begin{table}
\begin{tabular}{l r r} \hline \hline  & **Preprocessing** & **Inference** \\ \hline DAE & 135.8 & 62.6 \\ FactGraph-E - **Parser1** & 427.9 & 79.0 \\ FactGraph-E - **Parser2** & 1332.2 & 75.4 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Speed Comparison. Execution time is measured in seconds.



hanced dependency model from the Stanford CoreNLP tool Manning et al. (2014). This model builds a parse by performing a linear-time scan over the words of a sentence. Finally, note that FactGraph is slower than DAE in inference because it employs adapters and encodes both graphs and texts from the document and summary, whereas the DAE model encodes only the texts.

## Appendix D Comparing Semantic Representations for Factuality Evaluation

OpenIE graph-based structures were used in order to improve factuality in abstractive summarization Cao et al. (2018), whereas dependency arcs were shown to be beneficial for evaluating factuality Goyal and Durrett (2020). We thus investigate different graph-based meaning representations using FactGraph. AMR is a more logical representation that models relations between core concepts, and has a rough alignment between nodes and spans in the text. Conversely, dependencies capture more fine-grained relations between words, and all words are mapped into nodes in the dependency graph. OpenIE constructs a graph with node descriptions similar to the original text and uses open-domain relations, leading to relations that are hard to compare.

As shown in Table 10, whereas OpenIE performs slightly better than dependency graphs, AMR gives the best results according to the two metrics, highlighting the potential use of AMRs in representing salient pieces of information. Different from our work, Lee et al. (2021) and Naseem et al. (2021) propose a graph construction approach which generates a single document-level graph created using the individual sentences' AMR graphs by merging identical concepts - this is orthogonal to our sentence-level AMR representation and can be incorporated in future work.

## Appendix E Semantic Representations

In Figure 5 we show AMR and dependency representations for the summary sentence _"police have appealed for help in tracing a woman who has been missing for six years."_. In SS5.5 those semantic representations are used to predict subsentence-level factuality using edge-level information. In particular, FactGraph-E employs AMR (Figure 4(a)) whereas DAE uses dependencies (Figure 4(b)).

Figure 5: (a) AMR and (b) dependency representations for the summary _“police have appealed for help in tracing a woman who has been missing for six years.”_



Example 1 Document:[..] Workers believe the local man, who was working at Fruehauf on Houghton Road, fell from a cherry picker at about 13:45 GMT on Thursday.[..]

Figure 6: Examples of graph-based meaning representations parsed from sentences of documents and generated summaries.