Enhancing a Text Summarization System with ELMo
Authors: Claudio Mastronardo, Fabio Tamburini (Università di Bologna, 2019)
Topic: Language Models

Abstract
Text summarization has gained a considerable amount of research interest due to deep learning based techniques. We leverage recent results in transfer learning for Natural Language Processing (NLP) using pre-trained deep contextualized word embeddings in a sequence-to-sequence architecture based on pointer-generator networks. We evaluate our approach on the two largest summarization datasets: CNN/Daily Mail and the recent Newsroom dataset. We show how using pre-trained contextualized embeddings on Newsroom improves significantly the state-of-the-art ROUGE-1 measure and obtains comparable scores on the other ROUGE values.

Introduction
The amount of human generated data is outstanding: every day we generate about 2 quintillion bytes of unstructured data and this number is expected to grow. With such a huge amount of information, swiftly accessing and comprehending large piece of textual data is becoming more and more difficult. Automatic text summarization constitutes a powerful tool which can provide a useful solution to this problem. In recent years, automatic text summarization systems have gained a considerable amount of research interest due to deep learning powered NLP impressive results (AUTHOR; AUTHOR; AUTHOR; AUTHOR; AUTHOR; AUTHOR). Neural network (NN) based approaches have always been considered data hungry techniques because they often require a large amount of training data, but, in the latest years, several works have made a huge contribution in this direction (AUTHOR; AUTHOR; AUTHOR). Text summarization systems can be divided into two main categories: Extractive and Abstractive AUTHOR. The first generate summaries by purely copying the most representative chunks from the source text (AUTHOR; AUTHOR), while in the second summarization algorithms make up summaries by using novel phrases and words in order to rephrase and compress the information in the source text AUTHOR. Some works shed light on using both approaches through hybrid neural architectures attempting to gather the best characteristics of each world (AUTHOR; AUTHOR). NLP has seen a tremendous amount of attention after several deep learning based important results (AUTHOR; AUTHOR). Most of them relied on the concept of distributed representation of words, defining them as real-valued vectors learned from data (AUTHOR; AUTHOR). Recent results were able to generate richer word embeddings by exploiting their linguistic context in order to model word polysemy AUTHOR. In this paper, we build upon the work of AUTHOR on the Pointer-Generator Network for text summarization by integrating it with recent advances in transfer learning for NLP with deep contextualized word embeddings, namely an ELMo model AUTHOR. We show that, using pre-trained deep contextualized word embeddings, integrating them with pointer-generator networks and learning the ELMo parameters for combining the various model layers together with the text summarization model, we can improve substantially some of the ROUGE evaluation metrics. Our experiments were based on two datasets commonly used to evaluate this task: CNN/Daily Mail AUTHOR and Newsroom AUTHOR.

Conclusion
In this work we leveraged recent results in transfer learning for NLP with deep contextualized word embeddings in conjunction with pointer-generator NN for automatic abstractive text summarization. We noticed a considerable increase of model’s performance in terms of the ROUGE score, achieving state-of-the-art on the Newsroom dataset for the ROUGE-1 metric. This is a dataset designed for testing abstractive systems while the other dataset (CNN/Daily Mail) contains summaries formed by sentences extracted from the original texts and it is more suitable for testing extractive systems. Then, it is reasonable that we got improvements only when using the Newsroom dataset. Intrinsic, corpus-based metrics based on string overlap, string distance, or content overlap, such as BLEU and ROUGE, suffer from the need to have a reference output provided by the gold standard corpus in order to evaluate the system outputs. That seems very problematic (e.g. see AUTHOR) because the reference summary is only one of the possible summaries that humans can produce. By looking at the supplementary material regarding some examples of our system output, one can immediately recognize that, even if very different from the reference one, the summaries produced by the proposed system are in most cases acceptable. The definition of proper metrics capturing in the right way the correctness of system outputs remains, in our opinion, a critical open issue. As discussed also in the recent review by AUTHOR about Machine Translation (MT) metrics, “When reference translations are used [...] MT outputs that are very similar to the reference translation are boosted and not similar MT outputs are penalised even if they are good”, the so-called “reference bias”. The same metrics are currently used also in text summarization leading to similar problems.