SlowGuess/ACL_2020_OCR huggingface dataset:
Xinnuo Xu†, Ondrej Dušek‡, Jingyi Li†, Verena Rieser† and Ioannis Konstas†
†The Interaction Lab, MACS, Heriot-Watt University, Edinburgh, UK
$^{\ddagger}$ Charles University, Faculty of Mathematics and Physics, Prague, Czechia
xx6, jl125,v.t.rieser,i.konstas@hw.ac.uk
odusek@ufal.mff.cuni.cz
Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are biased towards specific words in the human reference. We introduce a new evaluation metric which abstracts away from the word-level and instead is based on fact-level content weighting, i.e. relating the facts of the document to the facts of the summary. We follow the assumption that a good summary will reflect all relevant facts, i.e. the ones present in the ground truth (human-generated reference summary). We confirm this hypothesis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight-based metric of Hardy et al. (2019).
Text summarisation compresses long textual documents into short summaries while retaining the most important information from the source. In contrast to extractive summarisation, which directly copies the most relevant fragments, abstractive summarization retains the most important facts and expresses them via paraphrasing, aggregating and even inferring new facts. Recent advances in neural decoders led to a number of single-document summarisation systems that exhibit some level of abstraction in their outputs, usually in the simplest form of paraphrasing (See et al. (2017); Narayan et al. (2018); Liu and Lapata (2019), inter alia).
Evaluating abstractive summarisation remains an open challenge (Schluter, 2017; Krysciński et al., 2019): First, decoders are amenable to pathogeniessuch as hallucination and/or omission of important information, which are hard to capture using existing evaluation metrics (Cao et al., 2018; Rohrbach et al., 2018; Dušek et al., 2020). Second, most datasets used for abstractive summarisation only contain a single reference summary,
e.g. (Narayan et al., 2018; Volske et al., 2017), which most existing automatic metrics evaluate against, e.g. ROUGE using exact n-gram overlap (Lin, 2004), and thus tend to downvote paraphrases.
We propose a new evaluation metric based on content weighting, where we abstract away from the particular surface form of the target summary, but represent it as facts using Semantic Role Labelling (SRL). In this way, we aim to better capture the semantic correctness of a summary, i.e. be more sensitive to hallucinations and omissions.<sup>1</sup>
In particular, we weight the facts present in the source document according to the facts selected by a human-written summary. This alignment is conducted using contextual, rather than token-level, embeddings, e.g., BERT (Devlin et al., 2019). For evaluation, we measure whether an automatically generated summary is able to capture the same facts as the target. We also show that the computed weights correlate well with human perception. Our code is available at https://github.com/XinnuoXu/CorrFA_for_Summarizaion.
The problem of reference bias has been addressed in several ways. First, metrics based on token-level or wider context embedding similarities which aim to better capture paraphrases but remain largely word-oriented, e.g. (Sun and Nenkova, 2019; Zhang et al., 2019; Zhao et al., 2019; Clark et al., 2019). Goodrich et al. (2019) come close to our approach by using entity and relation extraction, but their approach is limited to texts that lend themselves to be represented by RDF triples.
An alternative is manual evaluation against the source document. This entails selecting content either using domain experts, e.g., the PYRAMID method (Nenkova and Passonneau, 2004), factoids
<sup>1</sup>Note that we do not make any claims about fluency, which we assume is less of a problem for neural text generation.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5071-5081 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: List of SRL propositions and corresponding tree MR with two facts for the sentence "The queen has tweeted her thanks to people who sent her 90th birthday messages on social media".
(Teufel and van Halteren, 2004), or via crowdsourcing (Shapira et al., 2019; Hardy et al., 2019). However, evaluation based on a small human-labelled test set is noisy, time consuming, and costly. Xenouleas et al. (2019) propose a referenceless metric, which only checks properties of the summary, not its relation to the original document. Sun and Nenkova (2019) compare average token and sentence ELMo embeddings against the document and claim good (system-level) correlations.
Another option to avoid reference bias is question-based evaluation, either elicited manually (Clarke and Lapata, 2010; Narayan et al., 2018) or automatically (Scialom et al., 2019). However, it requires reference summaries as base for generating questions, thus only checking the summary contents indirectly.
3 Content Weighting
3.1 Fact Representation
We represent facts in a sentence by adapting SRL (Palmer et al., 2005), which roughly captures "who did what to whom" in terms of predicates and their arguments. Given a list of parsed propositions for a sentence, each predicate-argument structure is considered as one separate fact, where the predicate stands for the event and its arguments are mapped to actors, recipients, time, place, etc (see Fig. 1). Following a simple observation that arguments can function as separate predicates themselves, we construct a hierarchical tree structure for the whole sentence. We create the tree meaning representation
tion (MR) from the list of facts by choosing the fact with the largest coverage as the root and recursively build sub-trees by replacing arguments with their corresponding sub-facts (ARG2 in FACT1 is replaced by FACT2 in Fig. 1).<sup>3</sup>
3.2 Automatic Content Weighting
We compute argument and fact weights by measuring the similarity of facts/arguments in the original document and the target summary based on their BERT word embeddings (for content words only) and their distance in the tree MR. We denote tokens of a document  $D$  and its summary  $S$  as  $\mathbf{t}^D = \{t_1^D, t_2^D, \dots, t_n^D\}$  and  $\mathbf{t}^S = \{t_1^S, t_2^S, \dots, t_m^S\}$ . To get their corresponding contextual embeddings  $e_k^D$  and  $e_k^S$ , we concatenate the two texts, feed them into a pre-trained BERT model (Devlin et al., 2019) and take the contextualized embedding output from its last Transformer layer.
Argument-based weighting: We first represent the summary and the document as two sequences of leaf arguments  $\left\{A_1^D, A_2^D, \dots, A_N^D\right\}$  and  $\left\{A_1^S, A_2^S, \dots, A_M^S\right\}$  respectively, and weight the  $i$ -th leaf argument in the document as:
$$
w _ {i} ^ {a} = \underset {j = 1 \dots M} {\operatorname {a v g}} \cos \operatorname {d i s t} \left(E _ {i} ^ {D}, E _ {j} ^ {S}\right) \tag {1}
$$
i.e. the average embedding cosine distance to all arguments in the summary. Argument embeddings  $E_{i}^{D}$  and  $E_{j}^{S}$  are average embeddings of content-word tokens belonging to the arguments:
$$
E _ {i} ^ {*} = \underset {k \in A _ {i} ^ {*}, k \notin \text {s t o p s}} {\operatorname {a v g}} e _ {k} ^ {*} \tag {2}
$$
$\ast \in \{D,S\}$  , "stops" denotes a list of stopwords.
Fact-based weighting: We can represent the summary and the document as two sequences of facts  $\{F_1^D,F_2^D,\dots F_{N'}^D\}$  and  $\{F_1^S,F_2^S,\dots F_{M'}^S\}$ , and weight the  $i$ -th fact in the document by its average distance to facts in the summary:
$$
w _ {i} ^ {f} = \operatorname {a v g} _ {j \in 1 \dots M ^ {\prime}} d _ {i j} ^ {f} \tag {3}
$$
<sup>3</sup>We avoid using sentence-level MRs such as AMR (Banarescu et al., 2013), since current state-of-the-art performance of parsers is far behind compared to the simpler SRL task.
4By concatenating, the information in each text can be embedded in each other through self-attention. This is useful since the summary sometimes contains additional and/or common-sense knowledge not captured in the document.
For example, in Fig. 1, ARG0, V, ARG1 in FACT1, and all the arguments in FACT2 are leaf arguments in the sentence, whereas ARG2 in FACT1 is not.
For example, in Fig. 1, "her" and "thanks" are two tokens directly attached to the argument ARG1 of FACT1. Thus, the embedding for ARG1 of FACT1 is the average embedding of these two tokens.
2We use the SRL implementation of He et al. (2018) found in https://allennlp.org with 86.49 test F1 on the Ontonotes 5.0 dataset.
The fact-level distance  $d_{ij}^{f}$  is defined on top of argument weighting:
$$
\overline {{d}} _ {i j} ^ {f} = \underset {A _ {l} ^ {D} \in F _ {i} ^ {D}, A _ {k} ^ {S} \in F _ {j} ^ {S}} {\operatorname {a v g}} \beta_ {i l} \beta_ {j k} \left\lfloor \operatorname {c o s d i s t} \left(E _ {l} ^ {D}, E _ {k} ^ {S}\right) \right\rfloor_ {> \gamma} \tag {4}
$$
It is computed as the average cosine distance over embeddings of all leaf arguments in the subtrees of fact  $F_{i}^{D}$  in the document and fact  $F_{j}^{S}$  in the summary, which is (1) filtered by a threshold  $\gamma$  to discard argument pairs with weak semantic relation<sup>7</sup> and (2) weighted by MR tree distances of arguments to facts:  $\beta_{il} = \frac{1}{\sqrt{\mathrm{tree}(F_i,A_l)}}$ .<sup>8</sup>
4 Content-weighting-based Metrics
We now use these weights to introduce two metrics: Corr-F (fact-level) and Corr-A (argument-level). Let  $\mathbf{w}_{gold}^{f}$  and  $\mathbf{w}_{cand}^{f}$  denote the fact-level content weights calculated using the procedure from Section 3 based on human-reference and system-generated summaries, respectively. Similarly,  $\mathbf{w}_{gold}^{a}$ , and  $\mathbf{w}_{cand}^{a}$  denote the argument-level weights. Corr-F is then the Pearson Correlation Coefficient (PCC) between  $\mathbf{w}_{gold}^{f}$  and  $\mathbf{w}_{cand}^{f}$ . Corr-A is PCC between  $\mathbf{w}_{gold}^{a}$  and  $\mathbf{w}_{cand}^{a}$ . In other words, Corr-F and Corr-A indicate whether the generated summary focuses on the informative main points in the document (i.e. the same points as the reference summary), on two different levels of granularity.
5 Metrics Evaluation
We validate our Corr-F and Corr-A metrics by collecting human judgements. In the following, we (1) collect content highlights from human judges using the Amazon Mechanical Turk platform<sup>9</sup> and calculate manual content weighting based on them, (2) calculate correlations of the manual content weights with our automatic content weights, (3) compare our metrics against existing reference-based ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019), as well as the referenceless manual HROUGE score (Hardy et al., 2019).<sup>10</sup>
We use the extreme summarisation dataset (XSum; Narayan et al., 2018), which consists of
In this work, we set the threshold to 0.6.
$^{8}$ E.g., in Fig. 1, treedist(FACT1, "ARG1: her thanks") = 1, treedist(FACT1, "ARG0: people") = 2, treedist(FACT2, "ARG0: people") = 1.
<sup>9</sup>Using the interface from https://github.com/ sheffieldnlp/highres.
<sup>10</sup>Note that Corr-F/A are calculated with content weighting with respect to the reference. Therefore, strictly speaking, Corr-F/A are different to all existing metrics but still share some properties with them. We show the correlation between Corr-F/A and existing metrics in terms of relative system ranking, rather than a head-to-head metrics comparison.
BBC articles and accompanying single-sentence summaries, i.e. sub-headlines of the original articles, professionally written by the authors of the articles. Due to the abstractive nature of the summaries, factoid content selection on phrase level is required beyond sentence-level extraction or token-level matching, making this dataset a popular test bed for abstractive summarisation.
We use the outputs of three recent abstractive summarization systems as evaluation targets for our metrics: (i) the Pointer-Generator model (PTGEN; See et al., 2017); (ii) the Topic-aware Convolutional Sequence-to-Sequence model (TCONVS2S; Narayan et al., 2018) and (iii) the abstractive summarization model using pretrained BERT encoders (BERTSUMABS; Liu and Lapata, 2019).
5.1 Manual Annotation Collection
Manual Content Highlighting: By extending the framework of Hardy et al. (2019), we collect manual content highlights on fact and argument levels, where we present human judges with the source document and the gold summary, with one fact/argument typeset in bold. The judges are required to select phrases or sentences in the document that support the bolded fact/argument (see Figure 4-9 in Appendix B). In both cases, judges are allowed to select parts of the text with any granularity. We limit the number of allowed continuous chunks and the maximum number of words to encourage highlights of fact/argument level.[12] We employ 3 judges per document in both cases. We use the same 50 articles and gold summaries sampled from the XSum test set as Hardy et al. (2019).
Manual Content Weighting Calculation:
Argument Level: Given a document  $D$  and a summary  $S$ , we define the weight of each token  $t_k^D$  with respect to a summary argument  $A_j^S$  as:
$$
w _ {k j} ^ {t} = \frac {\operatorname {N u m H} \left(t _ {k} ^ {D} , A _ {j} ^ {S}\right)}{\operatorname {N u m A} \left(A _ {j} ^ {S}\right)} \tag {5}
$$
$\mathrm{NumH}(t_k^D,A_j^S)$  denotes the number of times token  $t_k$  was selected and  $\mathrm{NumA}(A_j^S)$  is the total number of annotators who were shown  $A_{j}^{S}$  bolded. We use token weights to compute manual argument-level weights  $\mathbf{w}_{man}^{a}$  (parallel to Eq. 1):
$$
w _ {m a n, i} ^ {a} = \underset {j = 1 \dots M} {\operatorname {a v g}} \underset {t _ {k} ^ {D} \in A _ {i} ^ {D}} {\operatorname {a v g}} w _ {k j} ^ {t} \tag {6}
$$
<sup>11</sup>For the first two, we use candidate summaries provided by the authors. For the third, we generated summaries by training a model with code and data offered by the authors.
12We allow 4 chunks of max. 50 words total for fact-level and 5 chunks of max. 20 words for argument-level annotation.
5073
Granularity	PCC-W	PCC-S
Argument-level	0.3326	0.4762
Fact-level	0.3129	0.7291
Fact Level: By adapting Eq. 5, we calculate a weight  $w_{ki}^{t}$  for each token in document  $D$  w.r.t. bolded fact  $F_{i}^{S}$  in the summary  $S$ . The weight  $w_{ij}^{f}$  between fact  $F_{i}^{D}$  in the document and  $F_{j}^{S}$  in its summary is calculated using Eq. 6. We use Eq. 3 to get the manual fact content weighting  $\mathbf{w}_{man}^{f}$ .
5.2 Agreement with Manual Weighting
Correlation: We evaluate how automatic content weighting  $\mathbf{w}_{gold}^{a}$  and  $\mathbf{w}_{gold}^{f}$  correlates with manual content weighting  $\mathbf{w}_{man}^{a}$  and  $\mathbf{w}_{man}^{f}$ . Using the Pearson Correlation Coefficient directly over the content weights (PCC-W), we evaluate the correlation between content weights assigned by human judges and automatically calculated weights - PCC  $(\mathbf{w}_{gold}^{*}, \mathbf{w}_{man}^{*})$ . As a more extreme form of weighting, we compute the correlation between content "selected" (i.e. ignoring computed weights) by human judges and the automatic mechanism (PCC-S); we set the value to 1 if the weight is over 0, meaning the fact/argument is selected.
While content-weighting correlations are just moderate, content-selection correlations are strong, especially the fact-based (Table 1). In other words, the automatic method attends to facts human judges consider important, but weighs them differently.
System-level Agreement: We check system-level agreement on Corr-F and Corr-A metrics when using automatic vs. manual content weighting (Table 2): We compute fact/argument-level content weights  $\mathbf{w}_{\text {cand }}^{*}$  for each system (cf. Section 4). We then calculate Corr-F and Corr-A of  $\mathbf{w}_{\text {cand }}^{*}$  against both  $\mathbf{w}_{\text {man }}^{*}$  (manual weighting) and  $\mathbf{w}_{\text {gold }}^{*}$  (automatic weighting) on the 50 articles with human annotation introduced in Section 5.1.
The Corr-F metric shows the same system-level ordering for both manual and automatic content weighting. Furthermore, both manual and automatic content weighting agree that TCONVS2S and PTGEN achieve similar performance but are strongly outperformed by BERTSUMABS.
5.3 Comparison to existing metrics
Corr-F/A vs. referenceless metrics: HROUGE score (Hardy et al., 2019) is a content-weighting-based referenceless evaluation metric. Unlike our
Table 1: Correlation of automatic content weighting and selection with human highlights.
Model	Corr-F	Corr-A
Manual content weighting - w*cand vs. w*man
TCONVS2S	0.2274	0.2464
PTGEN	0.2180	0.2433
BERTSUMABS	0.2508	0.2662
Automatic content weighting - w*cand vs. w*gold
TCONVS2S	0.6203	0.6280
PTGEN	0.5822	0.5727
BERTSUMABS	0.6714	0.6533
Table 2: System-level scores for manual and automatic content weighting on 50 human-annotated documents.
Model	Unigram	Bigram
Pre	Rec	Pre	Rec
TCONVS2S	7.64	5.37	3.16	2.08
PTGEN	7.62	6.42	3.25	2.61
BERTSUMABS	8.24	6.25	3.29	2.41
Table 3: HROUGE on 50 human-annotated documents. approach, it operates on token level and is entirely based on manual annotation. The evaluation results in Table 3 show that Corr-F/A's ranking is identical to HROUGE's unigram and bigram precision, with Corr-F also assigning similar proportions. $^{13}$
Corr-F/A vs. reference-based metrics: ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019) are both reference-based metrics, which compute a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches as used in ROUGE, BERTScore computes token similarity using contextual embeddings. Comparing to ROUGE and BERTScore on the full XSum test set (see Table 4) shows full agreement on system ordering for both metrics.
6.1 Error Analysis
We now provide examples demonstrating the strength and weaknesses of Corr-F/A by analysing system outputs where BERTScore and Corr-F/A demonstrate different ordering.
Strengths: (1) Corr-F/A are more sensitive to content-level hallucination than BERTScore. Summaries with facts/arguments never mentioned in the original document get much lower Corr-F/A scores than summaries with content that appears in the document verbatim or as a paraphrase. Example 1 in Table 5 shows Corr-F/A penalizing the incorrect fact “to become the next president” generated by BERTSUMABS, while giving higher scores to TCONVS2S which paraphrased “abdicate” with
<sup>13</sup>We computed HROUGE for BERTSUMABS using https://github.com/sheffield.nlp/highres.
Model	CorrF/A	CorrF/A(L)	ROUGE	BERTScore
Corr-F	Corr-A	Corr-F	Corr-A	R1	R2	RL	P	R	F1
TCONVS2S	0.616	0.636	0.700	0.650	31.89	11.54	25.75	0.613	0.573	0.591
PTGEN	0.596	0.623	0.664	0.620	29.70	9.21	23.24	0.577	0.566	0.570
BERTSUMABS	0.655	0.683	0.715	0.670	38.53	16.09	30.80	0.628	0.616	0.621
Table 4: Summarisation models evaluated using Corr-F/A on full test set, with ROUGE and BERTScore scores. Note that Corr-F/A(L) is Corr-F/A calculated using a lower-performing SRL tool (He et al., 2017, see Section 6.2).
#	Source	Summary	Corr-F	Corr-A	BS-F1
Ground truth	Japan's emperor Akihito has expressed his desire to abdicate in the next few years, public broadcaster NHK reports.
1	BERTSUMABS	Japan's emperor Akihito is considering whether to become the next president of the country, reports say.	0.68	0.68	0.67
TCONVS2S	Japan's emperor Akihito has announced that he will step down in the Japanese capital, Tokyo.	0.81	0.71	0.67
Ground truth	Dick Advocaat has resigned as Sunderland boss, with the team yet to win in the Premier League this season.
2	BERTSUMABS	Sunderland manager Dick Advocaat has left the club by mutual consent after only eight games in charge.	0.60	0.66	0.65
PTGEN	Sunderland have appointed former boss Dick Advocaat as manager at the end of the season to sign a new deal.	0.26	0.34	0.65
Ground truth	A Chinese space capsule carrying three crew members has returned to Earth following a 13-day mission.
3	BERTSUMABS	China has successfully landed its first ever space flight, in a move hailed as a “historic moment”.	0.56	0.67	0.53
TCONVS2S	China has successfully launched the first ever robotic mission to date for the first time in its history.	0.85	0.68	0.51
Ground truth	A council plans to employ its own staff to help young people with mental health problems.
4	BERTSUMABS	A new academy to train people with mental health problems is to be set up in West Berkshire.	0.82	0.68	0.64
TCONVS2S	A new academy for children with mental health problems is being launched in West Berkshire.	0.73	0.56	0.67
Table 5: Examples of system outputs where Corr-F/A and BERTScore-F1 disagree on system ordering.
"step down". (2) Corr-F/A better identify paraphrases, especially those containing extra content mentioned in the document but not in the ground-truth summary. Example 2 in Table 5 shows that Corr-F/A do not penalize BERTSUMABS for generating the argument "after only eight games in charge", which is mentioned in the document.
Weaknesses: (1) Corr-F is weaker in identifying token-level hallucination,[14] as in Example 3 in Table 5. Corr-F gives a higher score to TCONVS2S output with one hallucinated token "robotic". However, Corr-A's more fine-grained approach works slightly better in this case. (2) Corr-F/A tend to under-score summaries containing content mentioned in the ground truth but only touched briefly in the document. In Example 4 in Table 5, Corr-F/A score the output of TCONVS2S lower, even though it correctly captures "an academy for children with mental health", which is mentioned only once in the document.
In sum, Corr-F/A is less dependent on the reference summary by also considering the source document, and thus has less of a reference bias than BERTScore. In addition, Corr-F/A helps to identify ungrounded facts, i.e. content-level hallucinations, which is important for identifying misinformation in automated news reporting.
6.2 Robustness of Corr-F/A
As noted in Section 3.1, Corr-F/A is based on publicly available SRL tools. To demonstrate the robustness of our metrics, we evaluate the same sys
tem outputs with Corr-F/A calculated using a lower-performing SRL tool (He et al., 2017). The results are shown as Corr-F/A(L) in Table 4 and show full agreement with Corr-F/A in terms of system ordering. However, the better performing original SRL system widens the margin between systems.
We present an automatic evaluation framework for abstractive summarisation, which is low-cost and robust, as it does not rely on expert annotators nor is susceptible to crowdsourcing noise. Using fact representations, we are able to capture semantically similar, but at the same time distant in surface form, content in the summary that aligns with arbitrarily far-apart parts of the input document, casting our metric to be directly interpretable. Our metric is more sensitive to perturbations of the facts in the target summary, which resemble common hallucination phenomena of neural decoders (see Figure 2-3 in Appendix A for examples). In the future, we intend to investigate different meaning representation formalisms, such as AMR (Banarescu et al., 2013) and Dynamic Syntax (Kempson et al., 2001) and extend to other datasets (e.g. multiple-reference summarization) and tasks (e.g. response generation in dialogue).
This research received funding from the EPSRC project MaDrIgAL (EP/N017536/1) and Charles University project PRIMUS/19/SCI/10. We would like to acknowledge the AWS Cloud Credits for Research programme.
$^{14}$ Token-level hallucination means an incorrect token within an otherwise correct fact structure. Content-level hallucination happens when whole facts or arguments are hallucinated.
$^{15}\mathrm{F}1$  on the Ontonotes 5.0 dataset is  $81.6\%$ $(\delta = -4.89)$