SlowGuess/ACL_2020_OCR huggingface dataset:
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the Original: Fact Aware Neural Abstractive Summarization. In AAAI, New Orleans, LA, USA. ArXiv: 1711.04434.
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019. Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. In ACL, Florence, Italy.
James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36(3):411-441.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG Challenge. Computer Speech & Language, 59:123-156.
Ben Goodrich, Vinay Rao, Mohammad Saleh, and Peter J. Liu. 2019. Assessing The Factual Accuracy of Generated Text. In KDD, Anchorage, AK, USA. ArXiv: 1905.13322.
Hardy, Shashi Narayan, and Andreas Vlachos. 2019. HighRES: Highlight-based reference-less evaluation of summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3381-3392, Florence, Italy.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364-369, Melbourne, Australia.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473-483, Vancouver, Canada. Association for Computational Linguistics.
Ruth M Kempson, Wilfried Meyer-Viol, and Dov M Gabbay. 2001. Dynamic syntax: The flow of language understanding. Blackwell Oxford.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3728-3738, Hong Kong, China.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallucination in Image Captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, Brussels, Belgium.
Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41-45, Valencia, Spain. Association for Computational Linguistics.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers
Unite! Unsupervised Metrics for Reinforced Summarization Models. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) and 9th International Joint Conference on Natural Language Processing (IJCNLP), Hong Kong. ArXiv: 1909.01610.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada.
Ori Shapira, David Gabay, Yang Gao, Hadar Rosen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation. In NAACL, Minneapolis, MN, USA. ArXiv: 1904.05929.
Simeng Sun and Ani Nenkova. 2019. The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) and 9th International Joint Conference on Natural Language Processing (IJCNLP), Hong Kong.
Simone Teufel and Hans van Halteren. 2004. Evaluating information content by factoid analysis: Human annotation and stability. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 419-426, Barcelona, Spain. Association for Computational Linguistics.
Michael Volske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark. Association for Computational Linguistics.
Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. SUMQE: a BERT-based Summary Quality Estimation Model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6007-6013, Hong Kong, China.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) and 9th International Joint Conference on Natural Language Processing (IJCNLP), Hong Kong. ArXiv: 1909.02622.
A Fact-level Content Weighting Examples
Fig. 2 and 3 show examples for documents weighted using Corr-F/Corr-A with respect to different summaries.
In Fig. 2, the left column shows one document weighted by the reference summary and two system-generated summaries from BERTSUM-ABS and TCONVS2S respectively (summaries are shown in the right column). As we can see, there are 4 relatively important facts in the document weighted by the reference summary. BERTSUM-ABS and TCONVS2S capture 3 and 2 out of 4, respectively. Other than the important facts highlighted by the reference summary, TCONVS2S also assigns high weights to other facts; that leads to the hallucinated generation and lower Corr-F Corr-A scores. On the other hand, BERTSUM-ABS's summary weighs facts in the document in a similar way to the reference summary, which lead to a strongly related summary and high Corr-F and Corr-A scores.
In Fig. 3, there are 5 relatively important facts in the document weighted by the reference summary. BERTSUMABS and TCONVS2S capture 4 and 3 out of 5, respectively. Both systems miss the fact "Pope Francis, who has taken a more liberal stance on homosexuality". However, the weight of this fact given by BERTSUMABS's output is higher than with TCONVS2S's. The Corr-F and Corr-A are lower for TCONVS2S due to misweighting of informative facts in the document.
B Annotation Interface
We provide the following illustrations of the human annotation interface:
- Annotation interface for manual content weighting examples, including the instructions, for fact-level (Fig. 4 and 5) and argument-level (Fig. 6 and 7) annotation,
Examples of human annotation results for fact (Fig. 9) and argument (Fig. 8) level.
Please refer to the individual figure captions for detailed descriptions.
turia pitt - renowned burns to be of her body and was told she would never compete again after the2011 ultra - marathon in western australia after 200 operations , she completed the 226 km @ @ 140 mile@ @ hawaii event on sunday she has been hailed on social media as an " amazing role model " find outhow to get into triathlon in our special guide ms pitt completed a 3.8 km swim , 180 km bike ride and42 km run in a time of 14:37:30 the mining engineer , author and motivational speaker completed theevent using custom - made gear and brake levers on her bike to accommodate the severe injuries to herhands she also wore special race clothing to deal with the extreme heat and humidity because of myburns , i ca n't regulate my own body temperature so i 've had to make some adjustments , or usstdandard tri gear in different ways , " she revealed last month i ' m going to need things like coolingsleeves and white suits so i do n't overheat after her encounter with the bushfire ms pitt spent 864days in hospital and underwent many operations to treat her injuries she made her comeback inthe ironmanaustralia triathlon in may . Competing in ironman has ultimately showed me that i literally cando anything i put my mind to , " she said on her blog ms pitt was widely praised on social media as aninspiration such a great example of what can be done through hard work belief in one ' s self and gooda support system , " one person wrote on her facebook page you continue to amaze and inspire , " saidanother
turia pitt received burns in 83 at of her body and was told she would never compete again after the 2011 ultra - marathon in western australia after 200 operations , she completed the 226 km @ @ 140 mile@@ hawaii event on sunday she has been hailed on social media as an " amazing role model " find outhow to get into triathlon in our special guide ms pitt completed a 3.8 km swim , 180 km bike ride and42 km run in a time of 14:37:30 the mining engineer , author and motivational speaker completed theevent using custom - made gear and brake levers on her bike to accommodate the severe injuries to herhands she also wore special race clothing to deal with the extreme heat and humidity because of myburns , i ca n't regulate my own body temperature so i 've had to make some adjustments , or usstandard tri gear in different ways , " she revealed last month i ' m going to need things like coolingsleeves and white suits so i do n't overheat after her encounter with the bushfire ms pitt spent 864days in hospital and underwent many operations to treat her injuries she made her comeback inthe ironman australia triathlon in may . competing in ironman has ultimately showed me that i literally cando anything i put my mind to , " she said on her blog ms pitt was widely praised on social media as aninspiration such a great example of what can be done through hard work belief in one ' s self and gooda support system , " one person wrote on her facebook page you continue to amaze and inspire , " saidanother
turia pitt received burns to  $65 \%$ of her body and was told she would never compete again after the2011 ultra - marathon in western australia after 200 operations , she completed the  $226 \mathrm{~km @} 140$  mile@ @ hawaii event on sunday she has been hailed on social media as an " amazing role model " find outhow to get into triathlon in our special guide ms pitt completed a  $3.8 \mathrm{~km}$  swim , 180 km bike ride and42 km run in a time of 14:37:30 the mining engineer , author and motivational speaker completed theevent using custom - made gear and brake levers on her bike to accommodate the severe injuries to herhands she also wore special race clothing to deal with the extreme heat and humidity because of myburns , i ca n't regulate my own body temperature so i 've had to make some adjustments , or usstandard tri gear in different ways , " she revealed last month i 'm going to need things like coolingsleeves and white suits so i do n't overheat after her encounter with the bushfire ms pitt spent 864days in hospital and underwent many operations to treat her injuries she made her comeback in theironman australia triathlon in may . Competing in Ironman has ultimately showed me that i literally cando anything i put my mind to , " she said on her blog ms pitt was widely praised on social media as aninspiration such a great example of what can be done through hard work belief in one ' s self and gooda support system , " one person wrote on her facebook page you continue to amaze and inspire , " saidanother
An Australian runner who suffered like threatening burns when she was trapped by a bushfire during a race has completed the hawaii ironman, seen as the world's toughest triathlon
BertSumAbs:
An Australian runner who suffered severe burns in a bushfire in hawaii has completed an ironman triathlon
Corr-F:0.96 Corr-A:0.88
TConvS2S:
An australian runner become the first person to win a race for the first time in almost 30 years
Corr-F:0.67 Corr-A:0.73
Figure 2: A document (left) weighted with respect to a reference summary and two system outputs (right), with Corr-F/Corr-A scores. The colour represents the sum of argument- and fact-level weights for each token (Eqs. 3 and 4). The darker the colour, the more important the fact is.
the french government proposed senior diplomat laurent stefanini for the post in January but the vatican is yet to respond to approve the choice the vatican usually responds within six weeks to Approve such a new ambassador the nomination of mr stefanini was seen as a litmus test for pope francis , who has taken a more liberal stance on homosexuality a french government spokesman said there had been negotiations with the vatican over the appointment france has chosen its ambassador to the vatican . this choice was stefanini and that remains the french proposal , " said spokesman stephane le foll observers say most vatican appointments are confirmed within six weeks and that this long silence should be read as a rejection the vatican traditionally makes no statement if it intends to decline a nomination mr stefanini served in the holy see as a deputy ambassador in the french embassy from 2001 to 2005 he was described by the country ' s foreign ministry as " one of our best diplomats " france legalised same - sex marriage in 2013 , despite opposition from the catholic church pope francis is regarded as more tolerant of homosexuality than previous popes . " who am i to judge ? " , he said in 2013
the french government proposed senior diplomat laurent stefanini for the post in January but the vatican is yet to respond to approve the choice the vatican usually responds within six weeks to approve such a new ambassador the nomination of mr stefanini was seen as a litmus test for pope francis, who has taken a more liberal stance on homosexuality a french government spokesman said there had been negotiations with the vatican over the appointment france has chosen its ambassador to the vatican . this choice was stefanini and that remains the french proposal , " said spokesman stephane le foll observers say most catholic appointments are confirmed within six weeks and that this long silence should be read as a rejection the vatican traditionally makes no statement if it intends to decline a nomination mr stefanini served in the holy see as a deputy ambassador in the french embassy from 2001 to 2005 he was described by the country 's foreign ministry as " one of our best diplomats " france legalised same - sex marriage in 2013 , despite opposition from the catholic church pope francis is regarded as more tolerant of homosexuality than previous popes . " who am i to judge ? ", he said in 2013
the french government proposed senior diplomat laurent stefanini for the post in january but the vatican is yet to respond to approve the choice the vatican usually responds within six weeks to approve such a new ambassador the nomination of mr stefanini was seen as a litmus test for pope francis, who has taken a more liberal stance on homosexuality a french government spokesman said there had been negotiations with the vatican over the appointment france has chosen its ambassador to the vatican. this choice was stefanini and that remains the french proposal , " said spokesman stephane le foll observers say most vatican appointments are confirmed within six weeks and that this long silence should be read as a rejection the vatican traditionally makes no statement if it intends to decline a nomination mr stefanini served in the holy see as a deputy ambassador in the french embassy from 2001 to 2005 he was described by the country 's foreign ministry as " one of our best diplomats " france legalised same - sex marriage in 2013 , despite opposition from the catholic church pope francis is regarded as more tolerant of homosexuality than previous popes . " who am i to judge ? ", he said in 2013
France has said it will not back down over its nomination of an openly gay ambassador to the Vatican.
France has said it is considering whether to appoint a French ambassador to the Vatican as a replacement for the right-wing politician.
The Vatican has announced the appointment of a new ambassador to the Vatican.
Corr-F: 0.69 Corr-A: 0.39
Figure 3: Another document (left) weighted with respect to a reference summary and two system outputs (right), with Corr-F/Corr-A scores (see Fig. 2 for details).
Main Points Highlight
This task requires three steps:
Step 1: Highlighting Informative Phrases in a Document that Support the Bolded Part in a Given Summary the Best
In this step, you will be given two components a summary with one bolded main point and a document.
1) read the summary and identify the bolded main point.
2) read the document
3) select sentences/phrases that support the bolded main point the best
An example of highlights:
researchers have identified a gene that may put people at greater risk of strokes and heart attacks
(The main point you should identify is: a gene that may put people at greater risk of strokes and heart attacks)
Figure 4: The instruction for fact-level human highlight annotation.
Figure 5: The human annotation interface for fact level. Human judges are required to highlight content in the document that is supporting the fact printed in bold "The Queen has tweeted her thanks" (FACT1 of the summary in Figure 1 in the paper).
You can highlight 4 sentences/phrases at most. The maximum combined length of all highlights is 50 words.
Please don't refresh the page.
Your task is to highlight informative phrases in the document that support the bolded part in the given summary the best.
The maximum combined length of all highlighted phrases is 50 words.
To highlight, use your mouse to select phrases from the document, and click on the pen icon.
To delete a group of highlights, right click on it and confirm.
Summary: the queen has tweeted her thanks to people who sent her 90th birthday messages on social media
"I am most grateful for the many digital messages of goodwill I have received and would like to thank you all for your kindness," she wrote.
The monarch , whose milestone birthday was marked withnumerous events , signed off the rare message " Elizabeth R " .
The Queen sent her first ever tweet in 2014 when she opened a new exhibition at the Science Museum in London.
Britain's longest-serving monarch celebrated her 90th birthday on 21 April, and a host of events were held over three months, from April to June.
The Queen has two birthdays - her real birthday on 21 April, and her official birthday held on a Saturday in June - a tradition going back 250 years. It was introduced to try to ensure better weather for the monarch 's official celebrations.
Words left
50 words.
Chunks left
4 chunks.
Highlighted Phrases:
Noun Phrases Highlight
In this step, you will be given two components a summary with one bolded noun phrase and a document.
1) read the summary and identify the bolded noun phrase
2) read the document
3) select Noun Phrases that support the bolded noun phrase the best
Examples of highlights:
(The noun phrase you should identify is: a gene)
...Writing in PLOS ONE they say the gene fault may encourage the formation of blood clots - the ultimate
cause of most heart attacks and strokes …
Around one in 10 people in the Caucasian population carries this variation of the gene , named PIA2 …
They found individuals with PIA2 were more likely to have a stroke
the scientists show PIA2 is also linked to an increased risk of heart attacks in people under 45 ...
You can highlight 5 noun phrases at most. The maximum combined length of all highlights is 20 words. Please note that for some cases, the bolded noun phrase is paraphrased in the document. You should also highlight the paraphrased phrases.
Figure 6: The instruction for argument-level human highlight annotation.
The maximum combined length of all highlighted phrases is 20 words.
the queen has tweeted her thanks to people who sent her 90th birthday messages on social media
The monarch , whose milestone birthday was marked with numerous events , signedoff the rare message " Elizabeth R " .
The Queen has two birthdays - her real birthday on 21 April, and her official birthday held on a Saturday in June - a tradition going back 250 years. It was introduced to try to ensure better weather for the monarch's official celebrations.
Her official birthday this year was 11 June and the annual Trooping the Colour was held on Horse Guards Parade, followed by an RAF flypast which the Royal Family watched from the balcony of Buckingham Palace.
The following day the Queen hosted the Patron's Lunch, a street party for some 10,000 people along The Mall which recognised her patronage of more than 600 organisations in the UK and around the Commonwealth.
Queen Elizabeth II at 90 Find out more about Queen Elizabeth II on BBCiWonder
Click to submit
20 words.
5 chunks.
Figure 7: The human annotation interface for argument level. Human judges are required to highlight content in the document that is supporting the phrase printed in bold "on social media" (argument ARGM-LOC of FACT2 of the summary in Figure 1 in the paper).
" i am most grateful for the many digital messages of goodwill i have received and would like to thank you all for your kindness , " shewrote . the monarch ' s official celebrations . her official birthday this year was 11 june andthe annual troopoping the colour was held on horse guards parade , followed by an raf flypast which the royal family watched from thebalcony of buckingham palace . the following day the queen hosted the patron ' s lunch , a street party for some 10,000 people along themall which recognised her patronage of more than 600 organisations in the uk and around the commonwealth . queen elizabeth ii at 90 findout more about queen elizabeth ii on bbc iwonder
Figure 8: Human highlight annotation for the argument ARG1 of FACT1 "her thanks" of the summary in Figure 1 in the paper.