SlowGuess/NAACL_2022_OCR huggingface dataset:
FACTGRAPH: Evaluating Factuality in Summarization with Semantic Graph Representations
Leonardo F. R. Ribeiro\*, Mengwen Liu‡, Iryna Gurevych†, Markus Dreyer‡, Mohit Bansal‡,§
†UKP Lab, Technical University of Darmstadt ‡Amazon Alexa AI, §UNC Chapel Hill
ribeiro@aiphes.tu-darmstadt.de,{mengwliu, mddreyer, mobansal}@amazon.com
gurevych@ukp.informatik.tu-darmstadt.de,mbansal@cs.unc.edu
Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FACTGRAPH, a method that decomposes the document and the summary into structured meaning representations (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FACTGRAPH encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph connectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FACTGRAPH outperforms previous approaches by up to  $15\%$ . Furthermore, FACTGRAPH improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.<sup>1</sup>
Recent summarization approaches based on pretrained language models (LM) have established a new level of performance (Zhang et al., 2020; Lewis et al., 2020), generating summaries that are grammatically fluent and capable of combining salient parts of the source document. However, current models suffer from a severe limitation, generating summaries that are not factually consistent, that is, the content of the summary does not meet the
(a) [...] Mr Mueller was given the role of special counsel by the justice department to lead its investigation into alleged Russian interference in last year's US election [...] The NYT has reported that Mr Trump has considered firing Mr Mueller [...].
(b) US President Donald Trump has said he will fire special counsel Robert Mueller, who is investigating alleged Russian interference in the US election.
Figure 1: Example of (a) a document, (b) a summary, and (c) the corresponding document and (d) summary graph-based meaning representations. The summary graph does not contain the "consider" node, indicating a factual error (red dashed edge).
facts of the source document, an issue also known as hallucination. Previous studies (Cao et al., 2018; Falke et al., 2019; Maynez et al., 2020; Dreyer et al., 2021) report rates of hallucinations in generated summaries ranging from  $30\%$  to over  $70\%$ . In the face of such a challenge, recent works employ promising ideas such as question answering (QA) (Durmus et al., 2020; Nan et al., 2021) and weakly supervised approaches (Kryscinski et al., 2020) to assess factuality. Another line of work explores dependency arc entailment to improve the localization of subsentence-level errors within generated summaries (Goyal and Durrett, 2020).
However, these methods have a reduced correlation with human judgments and may not capture well semantic errors (Pagnoni et al., 2021). One reason for the poor performance is the lack of good quality factuality training data. Second, it is challenging to properly encode core semantic content from the document and summary (Lee et al., 2021) and reason over salient pieces of information in
* Work done as an intern at Amazon Alexa AI.  
<sup>1</sup>Our code is publicly available at https://github.com/amazon-research/fact-graph
Human Language Technologies, pages 3238 - 3253
order to assess the summary factuality. Third, previous work (DAE, Goyal and Durrett, 2021) treats semantic relations as isolated units, not simultaneously considering the entire semantic structure of both document and summary texts.
To mitigate the above issues, we explore meaning representations (MR) as a form of content representation for factuality evaluation. We present FACTGRAPH, a novel graph-enhanced approach that incorporates core information from the document and the summary into the factuality model using graph-based MRs, which are more suitable for factuality evaluation: As shown in Figure 1, graph-based MRs capture semantic relations between entities, abstracting away from syntactic structure and producing a canonical representation of meaning.
Different from previous methods (Kryscinski et al., 2020; Goyal and Durrett, 2021), FACT-GRAPH is a dual approach which encodes both text and graph modalities, better integrating linguistic knowledge and structured semantic knowledge. As shown in Figure 2, it is composed of parameter-efficient text and graph encoders which share the same pretrained model and differ by their adapter weights (Houlsby et al., 2019). The texts from the document and summary are encoded using the adapter-based text encoder whereas the entire semantic structures that represent document and summary facts are used as input to the graph encoder augmented structure-aware adapters (Ribeiro et al., 2021b). The representations of the two modalities thus are combined to generate the factuality score.
In particular, we explore Abstract Meaning Representation (AMR) (Banarescu et al., 2013), a semantic formalism that has received much research interest (Song et al., 2018; Guo et al., 2019; Ribeiro et al., 2019, 2021a; Opitz et al., 2020, 2021; Fu et al., 2021) and has been shown to benefit downstream tasks such as spoken language understanding (Damonte et al., 2019), machine translation (Song et al., 2019), commonsense reasoning (Lim et al., 2020), and question answering (Kapanipathi et al., 2021; Bornea et al., 2021).
Intuitively, AMR provides important benefits: First, it encodes core concepts as it strives for a more logical and less syntactic representation, which has been shown to benefit text summarization (Hardy and Vlachos, 2018; Dohare et al., 2018; Lee et al., 2021). Furthermore, AMR captures semantics at a high level of abstraction explicitly modeling relations in the text and reducing
the negative influence of diverse text surface variances with the same meaning. Lastly, recent studies (Dreyer et al., 2021; Ladhak et al., 2021) demonstrate that there is a trade-off between factuality and abstractiveness. Structured semantic representations are potentially beneficial for reducing data sparsity and localizing generation errors in abstractive scenarios. Figure 1 shows examples of (c) document and (d) summary AMRs, where the summary AMR is missing a crucial modifying node present in the document AMR, which indicates a factual error in the summary.
We consolidate a factuality dataset with human annotations derived from previous works (Wang et al., 2020; Kryscinski et al., 2020; Maynez et al., 2020; Pagnoni et al., 2021). This dataset is constructed from the widely-used CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016) benchmarks. Extensive experimental results demonstrate that FACTGRAPH achieves substantial improvements over previous approaches, improving factuality performance by up to  $15\%$  and correlation with human judgments by up to  $10\%$ , capturing more content verifiability errors and better classifying factuality in semantic relations.
Evaluating Factuality. Recently, there has been a surge of new methods for factuality evaluation in text generation, especially for summarization. Falke et al. (2019) propose to rerank summary hypotheses generated via beam search based on entailment scores to the source document. Kryscinski et al. (2020) introduce FACTCC, a model-based approach trained on artificially generated data, to measure if the summary can be entailed by the source document in order to assess the summary factuality. QA-based methods (Wang et al., 2020; Durmus et al., 2020; Honovich et al., 2021; Nan et al., 2021) generate questions from the document and summary, and compare the corresponding answers in order to assess factuality. Xie et al. (2021) formulate causal relationships among the document, summary, and language prior to evaluate the factuality via counterfactual estimation.
Categorizing Factual Errors. A thread of analysis work has focused on identifying different categories of factual errors in summarization. Maynez et al. (2020) show that semantic inference-based automatic measures are better representations of summarization quality, whereas Pagnoni et al. (2021)
propose a linguistically grounded typology of factual errors and develop a fine-grained benchmark for factuality evaluation, moving to a fine-grained measure, instead of using a binary evaluation. Fabbri et al. (2021) introduce different resources for summarization evaluation which include a toolkit for evaluating summarization models.
Factuality versus Abstractiveness. Recent works (Dreyer et al., 2021; Ladhak et al., 2021) investigate the trade-off between factuality and abstractiveness of summaries and observe that factuality tends to drop with increased abstractiveness. Semantic graphs are uniquely suitable to detect factual errors in abstractive summaries as they abstract away from the lexical surface forms of documents and summaries, enabling direct comparisons of the underlying semantic concepts and relations of a document-summary pair.
Graph-based Representations for Summarization. A growing body of work focuses on using graph-based representations for improving summarization. Whereas different approaches encode graphs into neural models for multi-document summarization (Fan et al., 2019; Li et al., 2020; Pasunuru et al., 2021; Wu et al., 2021; Chen et al., 2021), AMR structures have been shown to benefit both document representation and summary generation (Liu et al., 2015; Liao et al., 2018; Hardy and Vlachos, 2018; Dohare et al., 2018) and have the potential of improving controllability in summarization (Lee et al., 2021). The above works are related to FACTGRAPH as they use semantic graphs for content representation, but also different because they utilize graphs for the downstream summarization task, whereas FACTGRAPH employs them for factuality evaluation.
Semantic Representations for Factuality Evaluation. More closely related to our work, Goodrich et al. (2019) extract tuples from the document and summary and measure the factual consistency by overlapping metrics. Xu et al. (2020) weight facts present in the source document according to the facts from the gold summary using contextual embeddings, and verify whether a generated summary is able to capture the same facts as the target. Recently, dependency arc entailment (DAE, Goyal and Durrett, 2020) is used to measure subsentence-level factuality by classifying pairs of words defined by dependency arcs which often describe semantic relations. However, FACTGRAPH is con
siderably different from those approaches, since it explicitly encodes the entire graph semantic structure into the model. Moreover, while DAE considers semantic edge relations of the summary only, FACTGRAPH encodes the semantic structures of both the input document and summary leading to better factuality performance at both sentence and subsentence levels.
3 FACTGRAPH Model
We introduce FACTGRAPH, a method that employs semantic graph representations for factuality evaluation in text summarization, describing its intuition (§3.3) and defining it formally (§3.4).
Given a source document  $D$  and a sentence-level summary  $S$ , we aim to check whether  $S$  is factual with respect to  $D$ . For each sentence  $d \in D$  we extract a semantic graph  $\mathcal{G}_d$ . Similarly, for the summary sentence  $S$  we extract its semantic graph  $\mathcal{G}_s$ . We use texts and graphs from both document and summary for factuality evaluation. Sentence-level summary predictions can be aggregated to generate a factuality score for a multi-sentence summary.
3.2 Extracting Semantic Graphs
We select AMR as our MR, but FACTGRAPH can be used with other graph-based semantic representations, such as OpenIE (Banko et al., 2007). AMR is a linguistically-grounded semantic formalism that represents the meaning of a sentence as a rooted graph, where nodes are concepts and edges are semantic relations. AMR abstracts away from surface text, aiming to produce a more language-neutral representation of meaning. We use a state-of-the-art AMR parser (Bevilacqua et al., 2021) to extract an AMR graph  $\mathcal{G}_a = (\mathcal{V}_a,\mathcal{E}_a,\mathcal{R}_a)$  with a node set  $\mathcal{V}_a$  and labeled edges  $(u,r,v)\in \mathcal{E}_a$ , where  $u,v\in \mathcal{V}_a$  and  $r\in \mathcal{R}_a$  is a relation type. Each  $\mathcal{G}_a$  aims to explicitly represent the core concepts in each sentence. Figure 1 shows an example of a (b) sentence and its (d) corresponding AMR graph.
Graph Representation. We convert each  $\mathcal{G}_a$  into a bipartite graph  $\mathcal{G}_b = (\mathcal{V}_b, \mathcal{E}_b)$ , replacing each labeled edge  $(u, r, v) \in \mathcal{E}_a$  with two unlabeled edges  $\{(u, r), (r, v)\} \in \mathcal{E}_b$ . Similar to Beck et al. (2018), this procedure transforms the graph into its unlabeled version. Pretrained models typically use
2Appendix A presents other examples of AMRs extracted from sentences of documents and generated summaries.
Figure 2: Overview of FACTGRAPH. A sentence-level summary and document graphs are encoded by the graph encoder with structure-aware adapters. Text and graph encoders use the same pretrained model and only the adapters parameters are trained.
a vocabulary with subword tokens, which makes it complicated to properly represent a graph using subword tokens as nodes. Inspired by Ribeiro et al. (2020, 2021b), we transform each  $\mathcal{G}_b$  into a new token graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where each token of a node  $v_{b}\in \mathcal{V}_{b}$  becomes a node  $v\in \mathcal{V}$ . We convert each edge  $(u_b,v_b)\in \mathcal{E}_b$  into a set of edges and connect every token of  $u_{b}$  to every token of  $v_{b}$ .
3.3 Intuition of Semantic Representation
In order to represent facts to better assess the summary factuality, we draw inspiration from traditional approaches to summarization that condense the source document to a set of "semantic units" (Liu et al., 2015; Liao et al., 2018). Intuitively, the semantic graphs from the source document represent the core factual information, explicitly modeling relations in the text, whereas the semantic summary graph captures the essential content information in a summary (Lee et al., 2021). The document graphs can be compared with the summary graph, measuring the degree of semantic overlap to assess factuality (Cai and Knight, 2013).
Recently, sets of fact triples from summaries were used to estimate factual accuracy (Goodrich et al., 2019). That approach is related to FACTGRAPH as it uses graph-based MRs, but also different because it compares the reference and the generated summary, whereas we compare the generated summary with the input document. Moreover, differently from Goodrich et al. (2019), FACTGRAPH explicitly encodes the semantic structures using a graph encoder and employs AMR as a semantic representation. Finally, in contrast to DAE (Goyal and Durrett, 2021), which focuses only on extracting summary graph representations, FACTGRAPH uses semantic graphs for both document and summary.
3.4 Model
Figure 2 illustrates FACTGRAPH, which is composed of text and graph encoders. The text encoder, denoted by  $E_{t}$ , uses a pretrained encoder  $E$ , augmented with adapter modules which receives the summary  $S$  and document  $D$  and outputs a contextual text representation. Conversely, the graph encoder, denoted by  $E_{g}$ , uses the same  $E$ , but is augmented with structure-aware adapters.  $E_{g}$  receives the summary and multiple document semantic graphs corresponding to its sentences, and outputs graph-aware contextual representations that are used to generate the final graph representation. During training, only adapter weights are trained, whereas the weights from  $E$  are kept frozen. Finally, both graph and text representations are concatenated and fed to a final classifier, which predicts whether the summary is factual or not.
Text Encoder. We employ an adapter module before and after the feed-forward sub-layer of each layer of the encoder. We modify the adapter architecture from Houlsby et al. (2019). We compute the adapter representation for each token  $i$  at each layer  $l$ , given the token representation  $h_i^l$ , as follows:
$$
\hat {\boldsymbol {z}} _ {i} ^ {l} = \boldsymbol {W} _ {o} ^ {l} \left(\sigma \left(\boldsymbol {W} _ {p} ^ {l} \mathsf {L N} \left(\boldsymbol {h} _ {i} ^ {l}\right)\right)\right) + \boldsymbol {h} _ {i} ^ {l}, \tag {1}
$$
where  $\sigma$  is the activation function and  $\mathsf{LN}(\cdot)$  denotes layer normalization.  $W_{o}^{l}\in \mathbb{R}^{d\times m}$  and  $W_{p}^{l}\in \mathbb{R}^{m\times d}$  are adapter parameters. The representation of the [CLS] token is used as the final textual representation, denoted by  $\pmb{t}$ .
Graph Encoder. In order to re-purpose the pretrained encoder to structured inputs, we employ a structural adapter (Ribeiro et al., 2021b). In particular, for each node  $v \in \mathcal{V}$ , given the hidden
representation  $h_{v}^{l}$ , the encoder layer  $l$  computes:
$$
\boldsymbol {g} _ {v} ^ {l} = \operatorname {G r a p h C o n v} _ {l} \left(\mathbb {L N} \left(\boldsymbol {h} _ {v} ^ {l}\right), \left\{\mathbb {L N} \left(\boldsymbol {h} _ {u} ^ {l}\right): u \in \mathcal {N} (v) \right\}\right)
$$
$$
\boldsymbol {z} _ {v} ^ {l} = \boldsymbol {W} _ {e} ^ {l} \sigma \left(\boldsymbol {g} _ {v} ^ {l}\right) + \boldsymbol {h} _ {v} ^ {l}, \tag {2}
$$
where  $\mathcal{N}(v)$  is the neighborhood of the node  $v$  in  $\mathcal{G}$  and  $\mathbf{W}_e^l\in \mathbb{R}^{d\times m}$  is an adapter parameter. GraphConv  $_l(\cdot)$  is the graph convolution that computes the representation of  $v$  based on its neighbors in the graph. We employ a Relational Graph Convolutional Network (Schlichtkrull et al., 2018) as graph convolution, which considers differences in the incoming and outgoing relations. Since AMRs are directed graphs, encoding edge directions is beneficial for downstream performance (Ribeiro et al., 2019). The structural adapter is placed before, whereas the normal adapter is kept after the feed-forward sub-layer of each encoder layer.
We calculate the final representation of each graph from the pooling denoted as  $z^{\mathrm{G}} = \{\mathbf{z}_v^{(L)}\mid v\in \mathcal{V}\}$ , where  $\mathbf{z}_v^{(L)}$  is the final representation of  $v$ . Thus, we use a multi-head self-attention (Vaswani et al., 2017) layer to estimate to what extent each sentence graph contributes to the document semantic representation based on the summary graph. This mechanism allows encoding a global document representation based on the summary graph. In particular, each attention head computes:
$$
\alpha_ {i} = \mathrm {A t t n} (\pmb {z} _ {s} ^ {\mathrm {G}}, \pmb {z} _ {i} ^ {\mathrm {G}}),
$$
$$
\boldsymbol {g} = \sum_ {i = 1} ^ {k} \alpha_ {i} \boldsymbol {W} _ {r} \boldsymbol {z} _ {i} ^ {\mathrm {G}}, \tag {3}
$$
where  $\mathbf{z}_s^{\mathrm{G}}$  is the final representation of  $\mathcal{G}_s$ ,  $k$  is the number of considered sentence graphs from the input document and  $\mathbf{W}_r \in \mathbb{R}^{d \times d}$  is a parameter.
The final representation is derived from the text and graph representations,  $\mathbf{q} = [t; \mathbf{g}]$ , and fed into a classification layer that outputs a probability distribution over the labels  $y = \{\text{Factual}, \text{Non-Factual}\}$ .
3.5 Edge-level Factuality Model
Inspired by Goyal and Durrett (2021), we evaluate the factuality at the edge level. In this setup, we use the same text and graph encoders; however, we encode the semantic graphs differently. In particular, we concatenate  $\mathcal{G}_s$  with each  $\mathcal{G}_d\in D$  and feed the concatenation to the graph encoder. The representation of a node  $v\in \mathcal{G}_s$  is calculated as:
$$
\boldsymbol {r} _ {t _ {v}} = \sum_ {w \in A (v)} E _ {t} (D; S) _ {w}
$$
$$
\boldsymbol {r} _ {g _ {v}} = \sum_ {d = 1} ^ {k} E _ {g} \left(\mathcal {G} _ {s}; \mathcal {G} _ {d}\right) _ {v} \tag {4}
$$
$$
\boldsymbol {r} _ {v} = \left[ \boldsymbol {r} _ {t v}; \boldsymbol {r} _ {g v} \right]
$$
Source	# datapoints	Domain
Wang et al. (2020)	953	CNN/DM, XSum
Kryscinski et al. (2020)	1,434	CNN/DM
Maynez et al. (2020)	2,500	XSum
Pagnoni et al. (2021)	4,942	CNN/DM, XSum
Total	9,829	CNN/DM, XSum
Table 1: Consolidated human annotations.
where  $A(v)$  is the set of all summary words aligned with  $v$ , and  $\boldsymbol{r}_{t_v}$  and  $\boldsymbol{r}_{g_v}$  are the word and node representations, respectively. Edge representations are derived for each AMR edge  $(u,v)\in \mathcal{E}:\boldsymbol{r}_e = [r_u;r_v]$ . The edge representation  $\boldsymbol{r}_e$  is fed into a classification layer that outputs a probability distribution over the output labels ( $y_{e} = \{\mathrm{Factual},\mathrm{Non - Factual}\}$ ). We assign the label non-factual for an edge in  $\mathcal{G}_s$  if one of the nodes in this edge is aligned with a word that belongs to a span annotated as non-factual. Otherwise, the edge is assigned the label factual. We call this variant FACTGRAPH-E.
One of the main challenges in developing models for factuality evaluation is the lack of training data. Existing synthetic data generation approaches are not well-suited to factuality evaluation of current summarization models and human-annotated data can improve factuality models (Goyal and Durrett, 2021). In order to have a more effective training signal, we gather human annotations from different sources and consolidate a factuality dataset that can be used to train FACTGRAPH and other models.
The source collections of the dataset are presented in Table 1. The dataset covers two parts, namely CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016). CNN/DM contains news articles from two providers, CNN and DailyMail; while XSum contains BBC articles. CNN/DM has considerably lower levels of abstraction, and the summary exhibits high overlap with the article; a typical CNN/DM summary consists of several bullet points. In XSum, the first sentence is removed from an article and used as a summary, making it highly abstractive. After we remove duplicated annotations, the total number of datapoints is 9,567, which we divide into train (8,667), dev (300) and test (600) sets. We call this dataset FACT-COLLECT.
Model	All data	CNN/DM	XSum
BACC	F1	BACC	F1	BACC	F1
QAGS (Wang et al., 2020)	79.8	79.7	64.2	76.2	59.3	85.2
QUALS (Nan et al., 2021)	78.3	78.5	60.8	76.2	57.5	82.2
FACTCC (Kryscinski et al., 2020)	76.0	76.3	69.0	77.8	55.9	73.9
FACTCC+	83.9 (0.4)	84.2 (0.4)	68.0 (1.0)	83.7 (0.5)	58.3 (2.2)	84.9 (1.0)
FACTGRAPH	86.3 (1.3)	86.7 (1.1)	73.0 (2.3)	86.8 (0.8)	68.6 (2.3)	86.6 (2.0)
FACTGRAPH (pretrained structural adapters)	86.4 (0.6)	86.8 (0.5)	74.1 (1.0)	87.4 (0.3)	70.4 (1.9)	85.9 (1.4)
FACTGRAPH (pretrained structural and text adapters)	87.6 (0.7)	87.8 (0.7)	76.0 (2.8)	87.5 (0.4)	69.9 (2.3)	88.4 (1.2)
Table 2: BACC and F1 scores for factuality models in the test set of FACTCOLLECT. Mean (±s.d.) over 5 seeds.
4.2 Method Details
We limit the number of considered document graphs due to efficiency reasons. In particular, we compute the pairwise cosine similarity between the embeddings of each sentence  $d \in D$  and the summary sentence  $S$ , generated by Sentence Transformers (Reimers and Gurevych, 2019). We thus select  $k$  sentences from the source document with the highest scores to be used to generate the document semantic graphs.