SlowGuess/NAACL_2021_OCR huggingface dataset:
Lucas Bandarkar
Marti A. Hearst*
Recent progress in Natural Language Understanding (NLU) has seen the latest models outperform human performance on many standard tasks. These impressive results have led the community to introspect on dataset limitations, and iterate on more nuanced challenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corresponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. On HLGD, human annotators achieve high performance of around 0.9 F-1, while current state-of-the-art Transformer models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Generator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high-performing models with consistency tests, and find that models are not consistent in their predictions, revealing modeling limits of current architectures.
Headlines are a key component in everyday news consumption. As the first piece of text the user interacts with when learning about a story, the headline can play many roles, including: summarize the main points of the story, promote a particular detail, and convince the reader to choose one source over another (Bonyadi and Samuel, 2013).
News aggregators amass content from many disparate news sources and have become popular, in part because they offer news readers access to diverse sources (Chowdhury and Landoni, 2006). Flaxman et al. (2016) find that news aggregators help news readers access content they are unfamiliar with, and potentially on opposite sides of the
Date	Headline	Group
03/27	Russian-U.S. crew makes belated arrival at space station	A
03/28	Russian spacecraft brings 3-man crew to ISS after 2-day delay	A
03/29	Space 'makes the heart grow rounder'	B
04/01	Astronauts' hearts become spherical during prolonged trips in space, study finds	B
Figure 1: Snippets of timeline in the HeadLine Grouping dataset (HLGD). The headlines are part of the Space timeline, one of 10 timelines in HLGD. Headlines labeled A are part of a group; those labeled B are part of another group.
political spectrum. At the heart of a news aggregator is the ability to group relevant content together, to support a reader in finding varying views and angles on the news.
Natural Language Understanding (NLU) has seen rapid progress in recent years. The creation of multi-task benchmarks such as the General Language Understanding Evaluation collection (GLUE), paired with fast-paced progress in Transformer-based architectures has led to models outperforming human baseline performance on many tasks, such as paraphrase identification (Dolan et al., 2004), semantic similarity (Cer et al., 2017), and extractive question-answering (QA) (Rajpurkar et al., 2018).
This success has led to the questioning of the composition of benchmarks, and the subsequent creation of ever-more challenging datasets, for example by increasing the diversity of texts in textual entailment datasets (Williams et al., 2018), or introducing unanswerable questions in QA datasets (Rajpurkar et al., 2018).
1.1 HeadLine Grouping Definition
In this paper, we propose the novel task of Headline Grouping. Although news articles may discuss several topics, because of length constraints, headlines predominantly describe a single event.
\*Author emails: {phillab, lucasban-darkar,hearst} @berkeley.edu
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3186-3198 June 6-11, 2021. ©2021 Association for Computational Linguistics
Therefore, for the task of headline grouping, we define two headlines to be in the same group if they describe the same event: an action that occurred at a specific time and place. We do not require headlines to contain fully identical information to be placed into the same group. For example, one headline might report an exact number of deaths, while another might report a rounded number, or omit the number altogether. Figure 1 shows an example from our dataset. The first two headlines are in group A, and the third and fourth are part of group B. The headlines are divided into groups A versus B because they describe different events in the timeline (astronauts arriving at the space station vs. a study about hearts in space). The two headlines in B show the lexical and syntactic diversity of groups in this dataset – they appear in the same group because they describe the same underlying event. Appendix D gives a longer excerpt.
We build a large dataset for the task of HeadLine Grouping, crowd-sourcing the annotation of large timelines of news headlines in English. We cast the task as a binary classification: given a pair of headlines, determine whether they are part of a headline group (1) or whether they relate to distinct events (0).
Our main contribution, described in Section 3, is the design of the HeadLine Grouping task (HLG), and the creation of the HeadLine Grouping Dataset (HLGD) that is focused on detecting when headlines refer to the same underlying event. We show that the human annotations in our dataset have strong inter-annotator agreement (average 0.81), and a human annotator can achieve high performance on our corpus (around 0.9 F-1), while current state-of-the-art Transformer-based model performance stands around 0.75 F-1.
A second contribution is a novel unsupervised approach for the task of HeadLine Grouping relying on modifying a headline generator model. The model achieves the best performance on HLGD amongst unsupervised methods. Section 4 presents the performance of this algorithm compared to several baselines, including supervised and unsupervised methods.
Our final contribution, presented in Section 5, is an analysis of the consistency of the best performing model on HLGD. We specifically analyze whether the model follows commutative and transi
tive behavior expected to be trivially true in HeadLine Grouping. $^{1}$
Paraphrase Identification, Textual Entailment and Semantic Similarity are three common NLP tasks that resemble HeadLine Grouping. In Paraphrase Identification (PI) (Ozmutlu, 2016; Xu et al., 2014), the objective is to determine whether two sentences are semantically equivalent. We show in Table 2 that only one third of positive headline pairs in HLGD qualify as paraphrases. We further show in Section 4 that a trained model on MRPC (Dolan et al., 2004), a PI dataset of news text, performs poorly on HLGD. Textual entailment (Bentivogli et al., 2009), or Natural Language Inference (NLI) (Williams et al., 2018), determines whether a premise text implies a hypothesis. Apart from the non-symmetricality of the entailment relationship, we believe entailment is not well-suited to the domain of headlines because of the strict nature of the relationship. A large portion of headlines in a group differ in level of detail, and under an entailment task, would need to be labeled as neutral or contradicting. Finally, semantic similarity assigns a strength of similarity between two candidate sentences, for example in the Semantic Textual Similarity Benchmark (STS-B) (Cer et al., 2017), similarity is ranked from 1 to 5. This flexibility seems like a good fit; however, the lexical and syntactic diversity of headlines about the same underlying content do not correspond well to a similarity range.
Topic Detection and Tracking (TDT) (Allan, 2002) was a DARPA-sponsored initiative to investigate methods to group news articles by topics (referred to as timelines in this paper). We view TDT as a precursor to the task of HeadLine Grouping: in TDT, the focus is on detecting and tracking a timeline of related events, while in HeadLine Grouping, the timeline is given, and the focus is on subdividing it into finer groups. We considered using the TDT datasets and annotating them for our purposes. However, the TDT developers acknowledge (Graff et al., 2006) several important errors in the way the TDT datasets were acquired (e.g., some publication dates were not properly attributed) that could have an impact on the quality
1The code, model checkpoints and dataset are available at: https://github.com/tingofurro/headline_grouping
of the final dataset.
News Headlines in NLP. Headlines are popular as a challenging source for generation tasks such as summarization (Rush et al., 2015), style transfer (Jin et al., 2020), and style-preserving translation (Joshi et al., 2013). Headlines have been leveraged to detect political bias (Gangula et al., 2019), click-bait and fake news phenomena (Bourgonje et al., 2017). Finally, sentiment analysis of headlines has received attention (Bostan et al., 2020), with some work showing headline sentiment can be a useful signal in finance (Moore and Rayson, 2017).
Grouping Headlines has been explored in prior work. Wubben et al. (2009) propose a TF-IDF based clustering algorithm, but do not evaluate its agreement with human annotations. Pronoza et al. (2015) build a corpus of Russian headlines pairs, but limit pairs in the dataset by filtering out headlines that are distant syntactically. We find that headline groups often contain syntactically distant headlines (see Figure 3). Bouamor et al. (2012) and Shinyama et al. (2002) present a simple strategy, relying on the assumption that all articles on a topic published on the same day form a group. As will be shown below, this assumption is not always correct (see Figure 2).
Several of the most-used news aggregators, such as Yahoo News  $^{2}$ , Google News  $^{3}$ , and Bloomberg's LSTM (Bambrick et al., 2020) present headlines in groups. As these systems do not have published algorithms, we cannot comment on their methods; nonetheless we hope that the release of the HLG dataset offers a common evaluation test-bed to benchmark systems.
3 HeadLine Grouping Dataset
We now present the HeadLine Grouping Dataset. We describe the dataset of news articles we collected for annotation, our annotation procedure, an analysis of the resulting dataset, and the challenges we propose to the community.
3.1 Dataset Source
We collect a set of 10 news timelines from an existing open-source news collection in English (Laban and Hearst, 2017). A timeline is a collection of news articles about an evolving topic, consisting of a series of events. The timelines we use to build HLGD consist of time-stamped English news arti
Story Name	Size	Groups	+ pairs	IAA
Tunisia Protests	111	46	219	0.758
Ireland Abortion Vote	180	81	406	0.727
Ivory Coast Army Mutiny	128	45	329	0.781
International Space Station	257	107	499	0.831
US Bird Flu Outbreak	79	36	91	0.924
Human Cloning	119	55	259	0.830
Facebook Privacy Scandal	194	105	274	0.753
Equifax Breach	159	81	261	0.855
Brazil Dam Disaster	273	132	634	0.818
Wikileaks Trials	180	101	550	0.859
Total / Average	1679	789	3522	0.814
Table 1: Names and statistics of the ten news timelines in HLGD. Size is the number of headlines in the timeline, Groups the number of distinct headline groups, + pairs the number of pairs of headlines in all groups, and IAA the inter-annotator agreement. Timelines are separated into training (1-6), development (7-8), and test (9-10).
cles originating from 34 international news sources. The timelines range in size from 80 to 274 news articles, and span 18 days to 10 years.
We choose to use timelines as the source for the dataset for two reasons. First, news timelines center around a theme, and as successive events occur, many pairs of headlines will be semantically close, yielding challenging samples for the dataset. Second, this task requires annotating headlines by pairs. If there are  $n$  headlines, there could be on the order of  $n^2$  headline pairs to annotate. By having annotators assign group labels to a chronologically organized timeline, the annotation procedure requires only one label per headline, or  $n$  labels total.
We attempted to diversify topics and geographical locations represented in the 10 selected timelines. Topics and statistics are shown in Table 1.
3.2 Annotation Procedure
To reduce the effects of varying judgement inherent to the task, annotations were obtained from five independent judges and merged using the procedure described in the following subsection. Annotators worked on an entire timeline at a time, using the following procedure:
- The timeline was presented in a spreadsheet, in chronological order, with a single headline per row, and the corresponding publication date (year, month, day),
- Annotators went over the timeline one headline at a time in chronological order,
- If the headline being annotated did not match a previously created group, the annotator as
$^{2}$ https://news.yahoo.com
<sup>3</sup>https://news.google.com
signed it a new group number,
- Otherwise, the annotator could assign the headline to a previous group number, grouping it with previously added headlines.
We note that the annotation relied on annotators' ability to discern an event described by a news headline. However, a headline is not always written in an event-centric manner – for example when the headline is vague (e.g., A way forward in gene editing in the Human Cloning timeline), or overly brief (e.g., Waste not, want not in the International Space Station timeline). Annotators were instructed to create a separate group for such cases, isolating non-event-centric headlines.
Roughly one fifth of the annotations were produced by authors of the paper, and the remaining annotations were obtained by recruiting 8 crowd-workers on the Upwork platform. The crowd-workers were all native English speakers with experience in either proof-reading or data-entry, and were remunerated at $14/hour.
Annotators were first trained by reading a previously annotated timeline, and given the opportunity to clarify the task before starting to annotate. Exact instructions given to the annotators are transcribed in Appendix A.
3.3 Merging Annotations
In order to merge the five annotations, we follow a standard procedure to produce a single grouping that represents an aggregate of annotations.
We create a graph  $G$ , with each headline in a timeline represented by a node  $n_i$ . An edge  $(n_i, n_j)$  is added to  $G$  if a majority of the annotators (three or more of the five) put the two headlines in the same group. We apply a community detection algorithm, the Louvain method (Blondel et al., 2008), to  $G$  to obtain a grouping of the headlines that we call the global groups.
3.4 Inter-annotator Agreement
We compare the groups of each annotator to the global groups for each timeline, and measure agreement between annotator groups and a leave-one-out version of the global groups using the standard Adjusted Mutual Information (Vinh et al., 2010). The average inter-annotator agreement is 0.814, confirming that consensus amongst annotators is high. Inter-annotator agreement is reported for each timeline in Table 1.
Figure 2: Histogram of difference in publication dates of positive and negative pairs in final dataset. Most positive headline pairs are published on the same day or within one day of each other. We down-sample negative pairs to only keep headlines published within 4 days of each other.
Section 4 provides individual annotator performance on HLGD, which obtain the highest performance of about 0.9 F-1, further confirming that the task is well defined for human annotators.
3.5 Creating the Final Dataset
We transform the global groups into a binary classification task by generating pairs of headlines in the timelines: labeling the pair with a 1 if it belongs to the same group, and 0 otherwise.
With this procedure, we obtain 3,522 distinct positive headline pairs, and 154,156 negative pairs. This class imbalance is expected: two headlines picked at random in a timeline are unlikely to be in the same group. In order to reduce class imbalance, we down-sample negative pairs in the dataset.
Figure 2 shows the distribution of differences in publication dates for pairs of headlines in the final dataset. Publication date is indeed a strong signal to determine whether headlines are in the same group, as most positive pairs are published on the same day or one day apart. However, we show in Section 4 that using time as a sole indicator is not enough to perform well on the dataset.
In Figure 2, it can also be observed that  $98\%$  of positive headline pairs are published within 4 days of each other. Therefore, we only retain negative pairs that are within a 4 day window, filtering out simpler negative pairs from the final dataset. This final dataset has a class imbalance of roughly 1 positive pair to 5 negative pairs, for a total of 20,056 of labeled headlines pairs. This is similar in size to other NLU datasets, such as MRPC (5,801
<sup>4</sup>https://www.upwork.com
Reasoning	Description	Example Headline Pair	Percentage
Difference in Detail	A headline conveys additional details, such as a name, or a cause	NASA delays work on Moon rocket during virus pandemic	37%
Nasa's Moon plans take a hit
Exact Para-phrase	Both headlines convey the same information	Equifax takes web page offline after reports of new cyber attack	30%
Equifax takes down web page after reports of new hack
Difference in Focus	Headlines focus on a different aspect of the event group	Astronauts to Get Thanksgiving Feast in Space	26%
A Brief History of Thanksgiving Turkey in Space
Pun, Play-on-word, etc.	A headline has a unique stylistic element to attract readers	New privacy law forces some U.S. media offline in Europe	7%
US websites blacked out in Europe on 'Happy GDPR Day'
Negative Examples
Reasoning	Description	Example Headline Pair	Percentage
Independent events	Headlines describe two distinct events involving common actors	Brazil dam disaster leaves 34 dead, hundreds missing	44%
Alert raised over imminent risk to another Brazil mining dam
Related Sub-events	Describing two events that are related, e.g., follow each other	Irish abortion referendum voting opens	36%
Ireland set to end abortion ban as exit polls signal landslide vote
Headline too broad	One of the headlines is too broad to assign to a particular event	Obama commutes Chelsea Manning sentence	16%
Who is Chelsea Manning? - video profile
Borderline / Noise	Could be positive or negative, based on interpretation	4%
Table 2: Results of a manual typology of a subset of HLGD. An analysis of 200 positive, and 200 negative same-day headline pairs reveals there are several reasons why headlines get grouped or not in our dataset.
Figure 3: Levenshtein Similarity Distribution for positive pairs in MRPC, SST-B and HLGD. Pairs of sentences in HLGD can be positive examples (i.e., in the same group) while being less syntactically similar than in other NLU datasets such as STS-B or MRPC.
samples), or STS-B (8,628 samples).
Figure 3 shows the distribution of Levenshtein Ratio (Levenshtein, 1966) defined as:
$$
\operatorname {R a t i o} \left(S _ {1}, S _ {2}\right) = 1 - \frac {\text {L e v e n s h t e i n} \left(S _ {1} , S _ {2}\right)}{\max \left(\left| S _ {1} \right| , \left| S _ {2} \right|\right)} \tag {1}
$$
for positive pairs  $(S_{1}, S_{2})$  in MRPC and STS-B, two common NLU datasets, as well as HLGD, computed at the character level. The average similarity in HLGD (0.51) is lower than in the two others (0.72 and 0.74, respectively). Furthermore, a classifier using solely the Levenshtein Ratio obtains an F-1 score of 0.81 on MRPC, but only 0.485 on HLGD. This suggests lexical distance alone does
not contain a strong signal for good performance on HLGD.
To gain insight into the linguistic phenomena that occur within and outside headline groups, the first author manually inspected 200 positive and 200 negative headline pairs in HLGD. Positive pairs were selected from randomly sampled large groups, and negative samples from same-day negative pairs, because headlines that appear on the same day but are not in the same group cannot be distinguished using time information and are likely to overlap semantically the most. In Table 2, we list the phenomena we observed, give an example for each, and show the frequency in our sample. Within a group, headlines can be exact paraphrases, differ in detail level, differ in the element of focus, or involve stylistic elements such as puns. Negative headline pairs analyzed were either about independent events, related sub-events or involved a headline that was not specific enough. Additionally, around  $4\%$  of the negative samples analyzed were judged as borderline, interpretable as either positive or negative, showing that some ambiguity in the task is unavoidable. We believe this diversity in phenomena are ingredients that make HeadLine Grouping challenging and interesting for NLU research.
3.7 Challenges
To allow for diversity in approaches to HeadLine Grouping, we propose to sub-divide HLGD into several challenges, limiting in each the data used to solve the classification task:
- Challenge 1: Headline-only. Access to the headline pairs only; similar to Paraphrase Identification and Textual Similarity tasks.
- Challenge 2: Headline + Time. Access to the headline pairs and their publication dates.
- Challenge 3: Headline + Time + Other. Access to the headline pairs, publication dates, and other information such as full content, author(s), and news source (a URL to the original article provides this access).
We believe these different challenges provide flexibility to probe a diversity of methods on the HLGD task. Challenge 1 fits the standard text-pair classification of NLU, similar to paraphrase identification, textual similarity and NLI, while additional meta-data available in Challenge 3 might be more compatible with the goals of the information retrieval community.
4 Results on the Challenges
In Table 3, we report the performance of a human annotator and a baseline, as well as unsupervised and supervised methods on HLGD. We chose Electra (Clark et al., 2020) for experiments based on a bi-directional Transformer (Vaswani et al., 2017), as initial experiments with other BERT (Devlin et al., 2019) variants performed similarly. Implementation details, model sizes and hyperparameters are listed in Appendix B.
4.1 Human Performance and Baseline
Human Performance reports the F-1 score of human annotators performing the task. Human performance is estimated by obtaining a sixth set of annotations for each timeline in the development and testing set, beyond the five used for dataset creation. These annotations were completed after several hours of practice on the training set timelines.
Human performance is distinct from the inter-annotator agreement (IAA) analysis presented in §3.4. IAA was performed on the five annotations used to create the dataset. We note that human
Figure 4: Schematic of the Headline Generator Swap model. We adapt a headline generator - a finetuned GPT-2 - to the task of HeadLine Grouping. The score of a pair of headlines is based on whether the generative model believes a swap of headlines is likely.
performance can theoretically achieve a perfect F-1 score of 1.0 if the sixth annotator grouped the headlines identically to the global group.
Time only reports the performance of a logistic regression baseline based on the difference in days of publication between the two headlines. Data plotted in Figure 2 shows that a majority of positive pairs are published within two days of each other.
4.2 Unsupervised Models
Electra MRPC Zero-shot stands for an Electra model trained on the Microsoft Paraphrase Corpus (MRPC), achieving an F-1 of 0.92 on its development set. The objective is to evaluate whether a competitive paraphrase identification system achieves high performance on HLGD. The threshold to predict a label of one is tuned on the training portion of HLGD. This model only accesses headlines, and falls under Challenge 1.
Electra MRPC Zero-shot + Time corresponds to the previous model, adding publication time into the model in the following way:
$$
P ^ {\prime} (Y = 1 | X) = P (Y = 1 | X) \cdot e ^ {- \lambda \Delta T} \tag {2}
$$