SlowGuess/ACL_2020_OCR huggingface dataset:
10https://github.com/google/
11 ROUGE scores were computed by SumEval. We used MeCab (Kudo et al., 2004) for Japanese tokenization.
<sup>9</sup>https://github.com/pytorch/fairseq
Figure 3: The distribution of the support scores on the English Gigaword dataset.
Table 5 shows the main results. The baseline model with full training data obtained 35.80 ROUGE-1 score on the English Gigaword dataset and 48.08 ROUGE-1 score on JAMUL. The entailment filter lowered ROUGE scores on both of the datasets probably because of the smaller number of training instances, but the self-training strategy improved ROUGE scores on the Gigaword dataset, outperforming the baseline model.
In contrast, the self-training strategy could not show an improvement for ROUGE scores on JAMUL. Although it is difficult to find the exact cause of this result, we suspect that the filtering step reduced the training instances too much (0.8M instances) for the self-training method to be effective. Another possibility is that the writing style of articles of non-entailment pairs in JNC/JAMUL is so distant that the self-training method generated headlines that are too different from reference ones.
The column "Sup" presents the support score computed by the recall-oriented ROUGE-1 between source documents and generated headlines (explained in Section 2.2). The table indicates that the filtering and self-training strategies obtain higher support scores than the baseline. Figures 3 and 4 depict histograms of the support scores for the baseline and filtering+pseudo settings on Gigaword and JAMUL, respectively. We could confirm that the filtering+pseudo strategy increased the number of headlines with high support scores.
The column "Entail" shows the entailment ratio measured by the entailment classifier. Again, the filtering+pseudo strategy obtained the highest entailment ratio on both the Gigaword dataset and JAMUL. Although this result may be interpreted
Figure 4: The distribution of the support scores on JAMUL.
as natural because we selected training instances based on the same entailment classifier, it is interesting to see that we can control the entailment ratio without changing the model.
In order to examine whether the filtering strategy can deliver noticeable improvements for human readers, we asked a human subject to judge the truthfulness of the headlines generated by the baseline setting and filtering+pseudo strategy. Presented with both a source document and a headline generated by the model, the human subject judged whether the headline was truthful, untruthful, or incomprehensible. We conduct this evaluation for 109 instances randomly sampled from the test sets of Gigaword and JAMUL.
The "Truthful" column in Table 5 reports the ratio of truthful headlines. Consistently with the entailment ratio, we could confirm that the filtering+pseudo strategy generated truthful headlines more than the baseline setting on both of the datasets. During the human evaluation, one instance in both full and filtered+pseudo settings from the Gigaword dataset judged as incomprehensible.
To sum up the results, improving the truthfulness of the supervision data does help improving the truthfulness of generated headlines. We could confirm the improvements from the support scores, entailment ratio, and human judgments. However, the ROUGE scores between system and reference headlines did not indicate a clear difference.
The ROUGE metric was proposed to measure the relevance of a summary when extractive summarization was the central approach (in the early 2000s). Obviously, the truthfulness of summaries
Dataset	Training data (amount)	R-1	R-2	R-L	Sup	Entail	Truthful
Gigaword	Full	(3.8 M)	35.80	17.63	33.69	75.38	85.78%	77.06%
Filtered	(2.7 M)	35.24	17.29	33.14	77.61	91.50%	—
Filtered+pseudo	(3.8 M)	35.85	17.94	33.72	79.91	93.56%	85.32%
JAMUL	Full	(1.7 M)	48.08	22.21	40.02	89.10	90.29%	89.91%
Filtered	(0.8 M)	46.08	20.81	38.07	90.14	95.67%	—
Filtered+pseudo	(1.7 M)	45.62	20.55	38.10	90.65	96.26%	92.66%
Table 5: Results on the test set. We used F1 full-length ROUGE score: R-1 (ROUGE-1), R-2 (ROUGE-2), and R-L (ROUGE-L). "Sup" denotes support score. "Entail" presents the percentage of outputs to which the entailment classifier predicts the entailment relation (built in Section 4.1). "Truthful" show the percentage of outputs to which a human subject judged as truthful headlines.
is out of the scope of ROUGE. The experimental results in this paper suggest that we should consider both relevance and truthfulness when evaluating the quality of abstractive summarization.
Rush et al. (2015) first applied the neural sequence-to-sequence (seq2seq) architecture (Sutskever et al., 2014; Bahdanau et al., 2015) to abstractive summarization. They obtained a dataset for abstractive summarization from the English Gigaword (Graff and Cieri, 2003; Naples et al., 2012). After this work, a large number of studies followed the task setting (Takase et al., 2016; Zhou et al., 2017; Cao et al., 2018a; Song et al., 2019; Wang et al., 2019).
Some researchers pointed out that abstractive summarization models based on seq2seq sometimes generate summaries with inaccurate facts. Cao et al. (2018b) reported that  $30\%$  of the summaries generated by a seq2seq model include different facts from source articles. In addition, Krysciński et al. (2019) reported that ROUGE scores have only a weak correlation with human judgments in abstractive summarization and that the current evaluation protocol is inappropriate for factual consistency.
Several studies approach the problem of inconsistency between input and output by improving the model architecture or learning method. Cao et al. (2018b) applied an information extraction tool to extract tuples of subject, predicate, and object from source documents and utilized them as an additional input to the model. Pasunuru and Bansal (2018) incorporated an entailment classifier as a reward in reinforcement learning. Guo et al. (2018) presented a multi-task learning method between summarization and entailment generation where hypotheses entailed by a given document
(as a premise) are generated. Li et al. (2018) introduced an entailment-aware encoder-decoder model to ensure the correctness of the summary. Kiyono et al. (2018) reduced incorrect generations by modeling token-wise correspondences between input and output. Falke et al. (2019) proposed a re-ranking method of beam search based on factual correctness from a classifier of textual entailment.
As another direction, Kryscinski et al. (2019) evaluated the factual consistency of a source document and the generated summary with a weakly-supervised model.
A few studies raised concerns about the data set and task setting. Tan et al. (2017) argued that lead sentences do not provide an adequate source for the headline generation task. The researchers reported that making use of multiple summaries as well as the lead sentence of an articles improved the performance of headline generation on the New York Times corpus. In contrast, our paper is the first to analyze the truthfulness of existing datasets and generated headlines, provide a remedy to the supervision data, and demonstrate the importance of truthfulness in headline generation.
In this paper, we showed that the current headline generation model yields unexpected words. We conjectured that one of the reasons lies in the defect in the task setting and data set, where generating a headline from the source document is impossible because of the insufficiency of the source information. We presented an approach for removing from the supervision data headlines that are not entailed by their source documents. Experimental results demonstrated that the headline generation model trained on filtered supervision data showed no clear difference in ROUGE scores but remarkable im
provements in automatic and manual evaluations of the truthfulness of the generated headlines. We also presented the importance of evaluating truthfulness in abstractive summarization.
In the future, we explore a more sophisticated method to improve the relevance and truthfulness of generated headlines, for example, removing only deviated spans in untruthful headlines rather than removing untruthful headlines entirely from the supervision data. Other directions include an extensive evaluation of relevance and truthfulness of abstractive summarization and an establishment of an automatic evaluation metric for truthfulness.
Moreover, it will be also interesting to see whether the same issue occurs in other related tasks such as data-to-text generation. We believe that the concern raised in this paper is beneficial to other tasks.
The research results have been achieved by "Research and Development of Deep Learning Technology for Advanced Multilingual Speech Translation", the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018a. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 152-161.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018b. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), pages 4784-4791.
James Clarke and Mirella Lapata. 2008. Global inference for sentence compression: an integer linear programming approach. Journal of Artificial Intelligence Research, 31(1):399-429.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS).
Günes Erkan and Dragomir R. Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22(1):457-479.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2214-2220.
Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Mark Kantrowitz. 2000. Multi-document summarization by sentence extraction. In Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization, pages 40-48.
David Graff and Christopher Cieri. 2003. English Gigaword (LDC2003T05). https://catalog.ldc.upenn.edu/LDC2003T05, Linguistic Data Consortium, Philadelphia.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 687-697.
Yuta Hitomi, Yuya Taguchi, Hideaki Tamori, Ko Kikuta, Jiro Nishitoba, Naoaki Okazaki, Kentaro Inui, and Manabu Okumura. 2019. A large-scale multi-length headline corpus for improving length-constrained headline generation model evaluation. In Proceedings of the 12th International Conference on Natural Language Generation (INLG), pages 436-441.
Yohei Kikuta. 2019. BERT pretrained model trained on Japanese Wikipedia articles. https://github.com/yoheikikuta/bert-japanese.
Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui, and Masaaki Nagata. 2018. Reducing odd generation from neural headline generation. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation (PACLIC).
Kevin Knight and Daniel Marcu. 2000. Statistics-based summarization - step one: Sentence compression. In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth
Conference on Innovative Applications of Artificial Intelligence (IAAI), pages 703-710.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. ArXiv, abs/1910.12840.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pages 66-71.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 230-237.
Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics (Coling), pages 1430-1441.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pages 150-157.
Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT), pages 510-520.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Inderjeet Mani. 2001. Automatic Summarization. John Benjamins Publishing Company.
Rada Mihalcea. 2004. Graph-based ranking algorithms for sentence extraction, applied to text summarization. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 170-173.
Kazuma Murao, Shintaro Takemae, Hayato Kobayashi, Taichi Yatsuka, Masaki Noguchi, Hitoshi Nishikawa, and Takenobu Tokunaga. 2019. Neural headline generation with self-training. In Proceedings of Computation+Journalism Symposium 2019.
Courtney Naples, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 95-100.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multireward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 646-653.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 379-389.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning (ICML), pages 5926-5936.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), pages 3104-3112.
Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hiroa, and Masaaki Nagata. 2016. Neural headline generation on abstract meaning representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1054-1059.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. From neural sentence summarization to headline generation: A coarse-to-fine approach. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), pages 4109-4115.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), pages 5998-6008.
Kai Wang, Xiaojun Quan, and Rui Wang. 2019. BiSET: Bi-directional selective encoding with template for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2153-2162.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1112-1122.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Fangfang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiveness of neural document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 785-790.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1095-1104.
- All facts of the headline are covered by those of the article.
- If the headline includes an expression that do not appear in the article, but if the fact mentioned by the expression can be derived from the article, judge the pair as "Entail".
Non-entail
- The statement of the headline conflicts with the article.
- The headline mentions facts that cannot be confirmed by the article.
- Impossible to judge because the article or headline is unreadable. If the headline is not grammatically complete but correct as the headline style, please try to judge either entail or non-entail.
- Other problems such as garbled characters.
Figure 5: Guideline for entailment labeling
A Guideline for entailment labeling
Figure 5 presents a guideline for the entailment labeling task in Section 3. Given a pair of an article and headline, a crowd worker is expected to judge whether the article entails the headline, and label the pair with either of the labels shown in this figure.
Figure 6 shows some examples of the generated headlines from the models described in Section 5. In the first example, the baseline model added "in Kashmir" in the headline, but this is incorrect. The correct location is in Southern Egypt, which was mentioned in the reference headline. The filtered+pseudo model generates a safe headline. The second headline generated by the baseline includes the verb 'begin' although the report was written two years ago. The baseline model added "dollar lower against yen" in the headline. There is a correlation indeed that dollar is lower against yen when Tokyo stocks rise, but we cannot confirm the fact
Suspected Muslim militants shot and killed five men who had formed a civilian patrol group to counter the radicals in their village, police officials said Monday.
Full (baseline):
Suspected Militants Kill Five in Kashmir
Filtered+pseudo:
Suspected Muslim Militants Kill Five
Divers searched the Mississippi River for bodies still trapped beneath the twisted debris of a collapsed freeway bridge Thursday, as finger - pointing began over a federal report two years ago that found the bridge was ‘‘ structurally deficient.
FEDERAL REPORT BEGINS IN MISSISSIPPI
Divers Search Mississippi River for Bodies
Tokyo stocks rose Tuesday as investors snapped up domestic demand - related issues due to receding jitters among investors over last week ’ s plunge .
Tokyo stocks rise, dollar lower against yen
Tokyo stocks end higher
Figure 6: Examples of the improved headlines.
from the source document.