Improving Neural Abstractive Summarization via Reinforcement Learning with BERTScore

ive summarization aims to paraphrase long text with a short summary. While it is a common practice to train encoder-decoder models in an end-to-end supervised learning fashion to maximize the log-likelihood objective, Paulus et al. (2018) introduce a method which utilizes reinforcement learning to directly maximize the non-differentiable ROUGE score, enhancing the model’s performance. ROUGE score (Lin, 2004) is the most widely-used evaluation metric for summarization, which simply evaluates the summarization quality by computing n-gram overlap between generated summaries and reference summaries. While ROUGE score is simple and easy to compute, there is a gap between the benchmark and the actual summarization performance. As ROUGE score only measures token hard-match, in some cases the ROUGE score will penalize two sentences conveying exactly the same semantic information, but highly reward sentences with completely different semantics yet in similar surface forms. BERTScore (Zhang et al., 2019) is a recently proposed evaluation metric. Similar to ROUGE score, it computes a similarity score for each token in the generated summaries with each token in the reference summaries. However, by computing token similarity using contextualized word embeddings provided by BERT (Devlin et al., 2019), BERTScore successfully incorporates semantic information behind sentences, thus can provide better evaluations for cases where ROUGE score fails to account for meaning-preserving lexical and semantic diversity. As BERTScore is demonstrated to have better correlations with human judgments for natural language generation (Table 1), can we use reinforcement learning with BERTScore to improve neural abstractive summarization? In this work, we answer this question by comparing different training schemas, such as supervised learning, reinforcement learning with ROUGE score (RL-ROUGE) and reinforcement learning with BERTScore (RL-BERTScore). We choose the strongly-performed pointer-generator networks with coverage loss (See et al., 2017) as our baseline model, and use CNN/Daily Mail corpus (Hermann et al., 2015) for training and evaluations. Our experiments demonstrate that RL-BERTScore improves both ROUGE score and BERTScore, whereas RL-ROUGE only improves ROUGE score. We manually compare 50 summaries generated by RL-BERTScore and RL-ROUGE with regard to their fluency, redundancy, and overall quality. We find RL-BERTScore achieves much better quality than RL-ROUGE for human evaluation. Our analysis and case studies further explain the success of BERTScore as a reward function. We hope our work could cast light on the design of better RL reward for natural language generation. Metric Spearman ρ Pearson ρ ROUGE-L 14.54 14.51 BERTScore 21.26 22.75 Table 1: Correlations of ROUGE-L and BERTScore with human evaluations. Scores are computed on the humanannotated summarization dataset (Chaganty et al., 2018). ∗Code available at https://drive.google. com/open? id=1VJBbjKSh3Hd4AgDT1HMqDrj1HoVomU3e CS229: Machine Learning, Fall 2019, Stanford University, CA. (LateX template borrowed from NIPS 2017.)