SlowGuess/ACL_2020_OCR huggingface dataset:
Pointer Generator Model with Coverage (PGC) (See et al., 2017) uses the copy mechanism (Vinyals et al., 2015) to allow copying words from the source. The adapted coverage mechanism (Tu et al., 2016) is incorporated to alleviate repetition by keeping track of source words that have been summarized. This copy mechanism is widely adopted by subsequent models.
Fast Abstractive Summarization with Reinforce (FASTRL) (Chen and Bansal, 2018) first uses an extractor agent to select salient sentences from the document, then condenses the extracted sentences using the Pointer-Generator summarizer.
Bottom-up Summarization Model (BOTTOMUP) (Gehrmann et al., 2018) first selects words from the source document that are likely to appear in the summary, then generates using the Pointer-Generator model, where the copying mechanism is constrained to the previously selected words. It improves upon PGC by explicitly learning the selector to avoid copying long text spans.
Topic-aware Convolutional Sequence-to-Sequence model (TCONVS2s) (Narayan et al., 2018) is a convolutional neural network-based model conditioned on the topics of the article. It is shown to be effective in capturing long-range dependencies in the documents.
BERT-based model (BERTSUM) (Liu and Lapata, 2019) is a two-stage fine-tuning approach
where the BERT-based encoder is first fine-tuned on the extractive summarization task and then on the abstractive summarization task with the decoder (denoted as BERTSUMEXTABS in the original paper).
C Details of Human Annotations
C.1 Grammaticality Annotation Guidelines
For grammaticality annotation, we present only the output sentence to the workers. We collect annotations from 5 workers for both of the tasks. For this task, given the output sentence, we provide workers the following guidelines:
1. First select whether the given sentence is "Nonsensical" or "Makes sense".
2. If the given text is not a complete sentence, mark it as "Nonsensical".
3. If you can understand the meaning of the sentence, despite grammaticality errors, and you are able to make sense of it, select "Makes sense".
4. If you did not select "Nonsensical", evaluate whether the sentence is "Grammatical" or "Has Minor Grammaticality Issues".
C.2 Faithfulness Annotation Guidelines
We present workers both the source and the output sentence and provide the following guidelines:
1. Read the sentence and the source fully.
2. If the information conveyed by the sentence is not expressed in the source, select "unfaithful".
3. Avoid using general knowledge, and check if the sentence is consistent with the source.
4. If you select "unfaithful", for the second part, select whether the information expressed by the sentence is not contained in the source or conflicting with the source.
D Additional Analysis
D.1 Examples for nonsensical sentences
- Sandals, £34, office.co.uk, luluguinness.com. (generated by PGC for CNN/DM)
Source	Output Sentence	Category
...Although her due date has not officially been confirmed, the duchess of Cambridge told wellwishers at a charity event last month: I am due mid-April, to the end of April...	The duchess of Cambridge told wellwishers at a charity event last month: “The duke's intention is to be at the commemorations”.	IC
...Carragher spoke to a local TV starton during his time in Girona. Carragher posted a picture on his Instagram account of the opening ceremony...	Carragher posted a picture on his son play in the famous youth tournament.	IC
A body was found by a member of the public on private land near Leighton, about 10 miles (16.09km) away from the centre of Shrewsbury, on Monday. Mr Bebbington's family has been informed, West Mercia Police confirmed.	The death of a man whose body was found in a river in Cumbria has been identified as murder.	H
The incident happened near Dr. Gray's hospital shortly after 10:00. The man was taken to the hospital with what police said were serious but not life-threatening injuries. The a96 was closed in the area for several hours, but it has since reopened.	A man has been taken to hospital after he was hit by a lorry in Dumfries.	H
Table 11: Examples of meaningful but unfaithful sentences. Category corresponds to the category of unfaithfulness error for the output sentence. IC: Incorrect Concatenation, H: Hallucination.
Reference	Output Sentence
... University of Nebraska researcher has re-vealed why stress is bad for you. Limited pe-riods of stress are good, as they release corti-sol...	University of Nebraska researcher has revealed why stress is bad for you, stimulating your body to produce an important hormone called cortisol.
...Indian air force and Nepalese army medical team launch rescue mission to bring injured people to hospitals in Kathmandu. Forshani Tamang's family carried her for four hours to reach help after she was wounded when their home was destroyed...	Indian air crew and Nepalese army medical team were killed in Nepal's Sindhupalchok quake.
Table 12: Examples of unfaithful sentence with high content overlap (computed by ROUGE-L) with the reference.
- He says easter triduum is a progression, although the word itself – triduum. (generated by FASTRL for CNN/DM)
- Chelsea beat Chelsea  $5 - 3$  in the Premier League on Saturday. (generated by FASTRL for CNN/DM)
- 12 years a slave actress Lupita Woodley and oily vegetables. (generated by BOTTOMUP for CNN/DM)
- A judge in Japan has ordered a judge to order a woman who has absconded from Japan to Japan. (generated by PGC for XSum)
Stoke City moved up to third in the Premier League with victory over Stoke City at Stoke. (generated by TCONV for XSum)
- Johnny Depp's management group is su ing his management group over his "lavish lifestyle". (generated by BERTSUM for XSum)
D.2 Examples for meaningful but unfaithful sentences
Table 11 includes examples that are annotated as meaningful but unfaithful. First three examples are picked from the models trained on CNN/DM, and last three are from the models trained on XSum. We observe that majority of sentences with faithfulness errors for CNN/DM dataset are generated by incorrect concatenation (IC). The models fuse two sentences from the source and generate a new sentence that is not consistent with the context of the source. Within this category, however, the models make a wide-range of mistakes such as copying the wrong entity, date, and quote.
For XSum, the faithfulness mistakes are mostly hallucinations. Models tend to hallucinate information (e.g. entities, events, date) that is not present in the source.
D.3 Examples for sentences with high content overlap with reference that are unfaithful
Although current summarization models are evaluated with respect to the content overlap between the reference and the output, these metrics do not necessarily provide any guarantees for the faithfulness of the output. Table 12 includes examples with similar content overlap scores as the faithful
examples but are unfaithful. We can see that although the output sentences include similar words and refer to similar topics, they include hallucinations and inaccurate information.
D.4 Limitations of the datasets
Since CNN/DM and XSum datasets are automatically crawled, we find that there is noise in the data. For example, source documents can include phrases such as "click here for the latest news". We further observe that reference can carry information that is not in the source document since some of these one sentence highlights are written using additional world knowledge. Table 13 shows an example where the reference is unfaithful since it includes information that is not in the source (i.e. the fact that Ms. Wood's first name is Leanne and she is Plaid Cymru leader.).
Source	Reference
Ms Wood blamed the Conservatives in particular for claiming the SNP posed a threat to the future of the UK. She claimed "progressive" parties like hers were offering a "collaborative" alternative to "combative" politics. "This election presents an opportunity for harmonious coexistence between our nations," she said. Ms Wood's comments followed Conservative claims that Labour dependence on support from the SNP to form a government after the election on 7 May would threaten the break-up of the UK. Campaigning in south Wales on Monday, she said: "The parties advocating progressive, inclusive non-partisan cooperation in this election are not those who claim to cherish the political union above all others, but the national parties of Wales and Scotland. Along with the Greens in England, our parties have provided people across these islands with a collaborative alternative to the traditional combative Westminster politics". Ms Wood added that she had received "hundreds" of supportive messages from people in England following the televised debates.	Plaid Cymru leader Leanne Wood has accused rival parties of "dangerous and divisive rhetoric" in a "desperate" attempt to win votes.
Table 13: Example where reference includes information that is not in the source.