Suitable Doesn’t Mean Attractive. Human-Based Evaluation of Automatically Generated Headlines
Authors: Michele Cafagna, Lorenzo De Mattei, Davide Bacciu, Malvina Nissim (Università di Pisa, CNR-ILC, University of Groningen, 2019)
Topic: Language Models

Abstract
We train three different models to generate newspaper headlines from a portion of the corresponding article. The articles are obtained from two mainstream Italian newspapers. In order to assess the models’ performance, we set up a human-based evaluation where 30 different native speakers expressed their judgment over a variety of aspects. The outcome shows that: (i) pointer networks perform better than standard sequence-to-sequence models, creating mostly correct and appropriate titles; (ii) the suitability of a headline to its article for pointer networks is on par with or better than the gold headline; (iii) gold headlines are still by far more inviting than generated headlines to read the whole article, highlighting the contrast between human creativity and content appropriateness.

Introduction
Progress in language generation has made it really hard to tell if a text is written by a human or is machine-generated. The recently developed GPT-2 transformer-based language model (Radford et al., 2019), when prompted with an arbitrary input, is able to generate synthetic texts which are impressively human-like. But what makes generated text good text? We investigate this question in the context of automatically generated news headlines. Headlines could be seen as very short summaries, so that one could use evaluation methods typical of summarisation (Gatt and Krahmer, 2018), but they are in fact a very special kind of summaries. In addition to being suitable in terms of content, newspaper titles must also be inviting towards reading the whole article. A model that, given an article, learns how to generate its title must then be able to cover both the summarisation as well as the luring aspect. We collect articles from Italian newspapers online, and generate their headlines automatically. In contrast to the feature-rich approach of Colmenares et al. (2015), which requires substantial linguistic preprocessing for feature extraction, we rely on recent developments in language modelling, and train three different sequence-to-sequence models that learn to generate a headline given (a portion of) its article. We compare these generated headlines to one another and to the gold headline through a series of human-based evaluations which take several aspects into account, ranging from grammatical correctness to attractiveness towards reading the full article. The factors we measure are in line with the requirements for human-based evaluation mentioned by Gatt and Krahmer (2018), and are useful since it is known that standard metrics based on lexical overlap are not accurate indicators for the goodness of generated text (Liu et al., 2016).

Conclusion
The quality of three different sequence-to-sequence models that generate headlines starting from an article was comparatively assessed through human judgement, which we contextually used to evaluate the original headlines as well. The best system is a pointer network model, with correctness judgements on par with the gold headlines. Evaluating the generated output on different levels, especially attractiveness, which typically characterises news headlines, uncovered an interesting aspect: gold headlines appear to be the most attractive to read the whole article, but are not considered the most suitable, on the contrary, they are judged as the most unsuitable of all. Therefore, when automatically generating headlines, just relying on content might never lead us to titles that are human-like and attractive enough for people to read the article. This should be considered in any future work on news headline generation. At the evaluation stage, it would also be beneficial to involve professional journalists. A first contact with one of the newspapers at the early stages of our evaluation experiments did not yet yield any concrete collaboration, but expert judgement on the quality of the generated headlines is something we would like to include in the future. One aspect that we have not explicitly considered in our experiments is that the headlines come from different newspapers (positioned at opposite ends of the political spectrum), and can carry newspaper-specific characteristics. Robust headline generation should consider this, too.