SlowGuess/ACL_2020_OCR huggingface dataset:
Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159-165.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231-2242. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3244-3254, Hong Kong, China. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019. An entity-driven framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3278-3289, Hong Kong, China. Association for Computational Linguistics.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616-1626, Melbourne, Australia. Association for Computational Linguistics.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171-1181, Vancouver, Canada. Association for Computational Linguistics.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
A.1 Experiment Details
Statistics of Knowledge Graphs. We show the statistics of knowledge graphs on two datasets in Table 4. On each dataset, we construct a large graph with abundant relations for each article. Note that on CNN/DM we have more arguments but fewer predicates in a document than those on NYT. This indicates CNN/DM has fewer coreferred entities.
Dataset	Doc # word	DOCGRAPH	SEGGRAPH
# Arg.	# Pre.	# Arg.	# Pre.	# Para.
NYT	795.9	131.6	87.3	6.40	3.74	23.5
CNN/DM	789.9	138.1	85.2	6.30	3.57	24.2
Table 4: Statistics of NYT and CNN/DM datasets. # Arg.: number of arguments in each document or paragraph. # Pre.: number of predicates in each document or paragraph. # Para.: number of paragraphs in each document. Two datasets have comparable graph size.
Training Details. We utilize Adam (Kingma and Ba, 2015) with a gradient clipping of 2.0 and a batch size of 32 for all models. During ML training, a learning rate of 0.001 is used; during RL stage, it is reduced to 0.0001 (Paulus et al., 2018).
We use the base version of BERT model (Devlin et al., 2019) to select candidate answers and we fine-tune the base version of RoBERTa model (Liu et al., 2019) to build our QA model. We take pretrained models from Wolf et al. (2019).
A.2 Human Evaluation Guideline
In our human evaluation, each human annotator is presented with 100 news articles. The annotators are asked to evaluate four summaries (in random order) for each article on two aspects (informativeness and fluency) on a scale of 1 to 5 (1 being very poor and 5 being very good). Furthermore, for unfaithfulness, we define three types of unfaithful errors and ask annotators to label whether summaries contain any type of error. Instructions in Table 5 are given to human judges.
Here are descriptions of the aspects:
- Informativeness: Whether the summary provides enough and necessary content coverage from the input article.
- Fluency: Whether the summary is free of obvious grammatically incorrect sentences (e.g., fragments, missing components) that make the text difficult to read.
- Faithfulness: Whether the summary accords with the facts expressed in the source.
Article: With a Little Extra Cash.
What to do with a bonus? The right thing, of course, is to pay off debts or save it for a time when there are not any bonuses. But in Albany, any financial windfall invites hordes of legislators hungrily seeking ways to spend it. This has already started to happen, with lawmakers eyeballing a projected budgetary surplus of just under $1 billion – not all that grand when you consider that the total state budget is in the neighborhood of $120 billion, but a healthy number nonetheless.
But one essential part of the equation is different this year: a new governor guarding the state finances. Nobody knows quite yet how Gov. Eliot Spitzer will manage a Legislature that wants to add a lot of its favorite things to his budget before they return it for his approval. One suggestion: Mr. Spitzer should keep his fist as tightly closed as possible, especially on his new school aid formula and his Medicaid adjustments. (....)
Informativeness:
1 Not relevant to the article
e.g., "editorial on gov eliot spitzer's plan to spend it. of new governor guarding state finances . and to spitzer should keep his fist as tightly closed as possible , especially on new school aid formula and his medicaid adjustments ."
3 Relevant, but misses the main point of the article
e.g., "editorial on new gov eliot spitzer's new governor guarding state finances . says spitzer should keep his new school aid formula and his medicaid adjustments"
5 Successfully captures the main point of the article
e.g., "Editorial says New York Gov Eliot Spitzer, faced with projected $ 0 billion budget surplus, should be tight-fisted and cautious about overspending"
Fluency:
1 Summary is full of garbage fragments and is hard to understand
e.g., “of new governor guarding state finances. and to spitzer should keep his fist as tightly closed as possible, to”
2 Summary contains fragments, missing components but has some fluent segments
e.g., "editorial on gov eliot spitzer's plan to spend it. of new governor guarding state finances . and to spitzer should keep his fist as tightly closed as possible , especially on new school aid formula and his medicaid adjustments."
3 Summary contains some grammar errors but is in general fluent
e.g., "editorial on any financial windfall invites hordes of legislators hungrily seeking ways to spend it . how gov eliot spitzer will manage legislature that wants to add lot of its favorite to his budget before they return it for his approval."
4 Summary has relatively minor grammatical errors
e.g., "article on in any financial windfall invites hordes of legislators hungrily seeking ways to spend it"
5 Fluent summary
e.g., "editorial says new new jersey gov eliot spitzer guarding state finances . says spitzer should keep his new school aid formula and his medicaid adjustments"
Faithfulness:
We define three types of unfaithful errors. Each type is labeled as "0" or "1" independently. "0" means summary does not make this type of error and "1" suggests this type of error occurs. Three types of errors are :
i Hallucination error: Fabricated content that does not occur in the original article e.g., "correction of dec 0 about new york column on state budget"
ii Out-of-Context error: Fact occurs in the article, but fails without correct context e.g., "Editorial says one essential part of the equation is different this year: a no governor guarding the tate finances."
iii Deletion or Substitution error: Summary contains incorrectly edited, missing elements; or summary incorrectly concatenates elements from different sentences.
e.g., "editorial says new new jersey gov eliot spitzer guarding state finances, keeping his new school aid formula adjustments."
Table 5: Sample summaries with explanations on human evaluation aspect scales, and the definition of three types of unfaithful errors.