SlowGuess/NAACL_2021_OCR huggingface dataset:
where  $X$  represents the pair of headline inputs,  $P(Y = 1|X)$  represents the model's confidence of the headline pair being in the same group, and  $\Delta T$  the difference in days of publication of the headlines.  $\lambda$  is tuned on the training set. Because this method leverages headline and time information, it falls under Challenge 2.
Headline Generator Swap is a novel approach we propose for zero-shot headline grouping, summarized in Figure 4.
Transformer-based Language Models like GPT-2 (Radford et al., 2019) model the probability of
Method	Challenge	HLGD Dev F-1	HLGD Test F-1
Human Performance and Baseline
Human Performance	3	0.884	0.900
Time-only	2	0.654	0.585
Unsupervised / Zero-shot Models
Electra MRPC Zero-Shot	1	0.562	0.626
Electra MRPC Zero-Shot + Time	2	0.666	0.688
Headline Generator Swap	3	0.671	0.651
Headline Gen. Swap + Time	3	0.727	0.722
Supervised Models
Electra Finetune on HLGD	1	0.728	0.796
Electra Finetune on HLGD content	3	0.652	0.723
Electra Finetune on HLGD + Time	2	0.753	0.828
Table 3: F-1 performance of several methods on the development and test portions of the HeadLine Grouping Dataset. Methods are separated into baselines: (1) the performance of a human annotator, and performance using only publication date, (2) unsupervised or zero-shot methods that do not leverage the training set for predictions, and (3) supervised methods. Each method falls under a challenge setting (1, 2 or 3) based on data used.
a word sequence. As the first step in Headline Generator Swap, we use the GPT-2 model to create a headline generator to estimate the likelihood of a (headline, content) pair:  $P_{LM}(H|C)$ .
In more detail, we finetune a GPT-2 model to read through the first 512 words of a news article and generate its headline. The headline generator is trained with teacher-forcing supervision, and a large corpus of 6 million (content, headline) pairs (Laban and Hearst, 2017), not overlapping HLGD.
The second step in Headline Generator swap is to use this probability to produce a symmetric score for two articles  $A_{1} = (H_{1},C_{1})$  and  $A_{2} = (H_{2},C_{2})$ :
$$
S \left(A _ {1}, A _ {2}\right) = P _ {L M} \left(H _ {2} \mid C _ {1}\right) + P _ {L M} \left(H _ {1} \mid C _ {2}\right) \tag {3}
$$
This score evaluates the likelihood of a swap of headlines between articles  $A_{1}$  and  $A_{2}$ , according to the GPT-2 language model. We argue that if the model believes a swap is likely, the headlines must be part of the same group. The threshold above which  $S(A_{1},A_{2})$  predicts a 1 is determined using the training portion of the data. Because this model uses the headline and content of the article, it falls under Challenge 3.
Headline Gen. Swap + Time corresponds to the Headline Generator Swap model, adding publication date information similarly to the Electra MRPC Zero-shot + Time model:
$$
S ^ {\prime} \left(A _ {1}, A _ {2}\right) = S \left(A _ {1}, A _ {2}\right) \cdot e ^ {- \lambda \Delta T} \tag {4}
$$
This model uses the headline, publication data and content of the article, and falls under Challenge 3.
Unsupervised models were allowed to pick a single hyper-parameter based on training set performance: to learn the threshold in score differentiating between class 1 and class 0. Strictly speaking, because we tune this single parameter, the methods could be seen as supervised. However, we label them as unsupervised because model parameters were not modified.
4.3 Supervised Methods
Electra Finetune stands for an Electra model finetuned on the training set of HLGD, inputting the two headlines, divided by a separator token. Headline order is chosen randomly at each epoch. Because we train a model for several epochs (see Appendix B), a model is likely to see pairs in both orders. This model only uses headlines of articles for prediction, and falls under Challenge 1.
Electra Finetune on content represents a similar model to that described above, with the difference that the model makes predictions based on the first 255 words of the contents of the two news articles, instead of the headline. This evaluates the informativeness of contents in determining headline groups. This experiment requires the contents and falls under Challenge 3.
Electra Finetune + Time corresponds to an Electra model with time information. The model's output goes through a 768x1 feed-forward layer, and is concatenated with the day difference of publication, which is run through a 2x2 feed-forward, and a softmax layer. This model uses headline and time information, and falls under Challenge 2.
4.4 Discussion of Results
Human performance can be high, close to 0.9 F-1 both on development and test timelines.
Using time alone gives a lower-bound baseline on HLGD, achieving an F-1 of 0.585 on the test set, and confirming that publication date of an article is not enough to perform competitively on HLGD.
Regarding Unsupervised and Zero-shot approaches, the Headliner Generator Swap outperforms Electra MRPC Zero-shot. With additional time information (+ time), the generator-based model is able to get close to strong supervised models. The model benefits from pre-training on a large corpus of (content, headline) pairs, having learned a good representation for headlines.
Unsurprisingly, best performance on HLGD is achieved by a supervised approach, Electra Finetune HLGD + Time, which uses both headline and time information. With an F-1 performance on the development set of 0.753, the model is still 0.13 F-1 points below human performance (0.07 F-1 difference on the test set).
When finetuning the Electra model with contents instead of headlines, performance drops by 0.07 F-1 points. This is particularly surprising as it could be expected that content contains strictly more information than the headline. We interpret this performance of the content-based model as evidence that the contents are more broad and do not solely focus on the distinguishing fact that is necessary to perform the grouping.
Finally, publication date yields a performance gain of 0.025 to 0.1 F-1 points over models without time information. This confirms that even though time information alone does not achieve high performance, it can be used to enhance models effectively. Because human annotators read timelines chronologically and had access to publication date while annotating, we do not have an upper-bound of human performance without using time.
5 Analysis of Model Consistency
Checking whether deep learning models are consistent across predictions has recently become a subject of interest, for example with QA systems with text (Ribeiro et al., 2019) and image (Shah et al., 2019) inputs. We analyze model consistency by probing the Electra Finetune + Time model, which achieves highest performance in terms of F-1 score. We propose a commutative test and transitive test, both illustrated in Figure 5.
Figure 5: Simplified representation of commutativity (left) and transitivity (right). We verify whether predictions from our best-performing model are consistent with regards to these two properties.
In order to evaluate consistency across training runs, we trained six versions of the Electra Finetune + Time model with the same hyper-parameters. Because each training run processes through the data in a different order, the models are distinct from each other. With regard to performance, the models perform very similarly, achieving within 0.01 F-1 of each other on the development and test sets.
5.1 Commutative Test
The HeadLine Grouping task requires two sentences to be compared, both playing a symmetric role.
Most model architectures process the headline pair as a single sequence, and an arbitrary ordering of the pair is chosen for processing. We study whether this arbitrary choice has an impact on the model's prediction. Specifically, we make predictions for all pairs of headlines in the development portion of HLGD, running each pair in both  $(A,B)$  and  $(B,A)$  order.
On average across the 6 model checkpoints, swapping the order of headlines is enough to make the model change its prediction (put higher probability on 0 in one case and 1 in the other) on  $6.3\% (\pm 0.5)$  of the pairs.
Furthermore, in other cases when the prediction does not change, the probability of the predicted class fluctuates by  $0.061(\pm 0.005)$  on average, showing the impact sentence order has on all predictions.
The relatively small standard deviations across training runs indicates that this phenomenon is inherent to the training procedure and not only existent in a subset of models.
A remedy is to build a symmetric classifier:
$$
P _ {S} (Y \mid A, B) = \frac {P (Y \mid A , B) + P (Y \mid B , A)}{2} \tag {5}
$$
where  $P_{S}$  follows the symmetric rule by design, by predicting for both  $(H_{1}, H_{2})$  and  $(H_{2}, H_{1})$  and averaging. When applying this patch to models presented in Section 4, we observe an average gain in F-1 performance of 0.01. Even though encouraging, this gain is a post hoc fix, and enforcing symmetry during training might yield further gains.
5.2 Transitive Test
Transitivity involves triplets of headlines A, B and C. The assumption is that if A and B are part of the same group, and A and C are part of the same group, then B and C must be in the same group as well. The procedure followed during annotation — assigning group IDs to headlines — implies that the transitivity is preserved, as all headline pairs within the same group are positive pairs.
To test a model's consistency with regards to the transitive rule, we use the Electra Finetune + Time model to produce a prediction for all pairs of headlines in the development portion of HLGD.
For each triplet (A,B,C) of headlines in the timeline, the model produces three predictions for the (A,B), (A,C), and (B,C) pairs. We focus our attention on triplets where the model has predicted at least 2 positive pairs: if the third pair is predicted to be positive, transitivity is conserved (111 triangle), but if it is predicted to be negative, the triplet breaks the transitivity rule (110 triangle).
On average across the six model checkpoints, we find that of the 60,660 triplets for which the model predicted at least 2 positives pairs, 44,627 triplets had a negative third prediction, and 16,033 had a positive one. In short, the model is consistent only  $26.4\% (\pm 1.4)$  of the time on these triplets.
Improving model consistency with regards to transitivity is challenging, as it would involve presenting the model with triples in some way. Imposing this constraint could yield performance improvements on the task.
We note however that transitivity is a strong assumption, as it is possible for groups of headlines to have stronger and weaker subgroups. It is possible that human annotations would not always follow transitivity if tasked to do so. For this reason, we do not expect models to be  $100\%$  consistent, but there is room for improvement.
In this work we present the new task of HeadLine Grouping (HLG) a new challenging NLU task, with
an accompanying dataset (HLGD). Even though state-of-the-art NLU models have achieved close to human performance on many NLU tasks, we show that there is a considerable gap between best model performance (0.75 F-1) and human performance (about 0.9 F-1) on HLGD. We therefore propose this dataset as a challenge for future NLU benchmarks. We propose to repurpose a Headline Generator for the task of headline grouping, based on prompting it for the likelihood of a headline swap, and achieve within 3 F-1 of the best supervised model, paving the way for other unsupervised methods to repurpose generators for NLU. Analysis of models on HLGD reveals that they are not consistent in trivial ways, suggesting further improvements needed to NLU models.
We would like to thank the Upwork crowd-workers for their assistance in creating HLGD, as well as Katie Stasaski, Dongyeop Kang and the ACL reviewers for their helpful comments. This work was supported by a Bloomberg Data Science grant. We also gratefully acknowledge support received from an Amazon Web Services Machine Learning Research Award and an NVIDIA Corporation GPU grant.
James Allan. 2002. Introduction to topic detection and tracking. In *Topic detection and tracking*, pages 1-16. Springer.
Joshua Bambrick, Minjie Xu, Andy Almonte, Igor Malioutov, Guim Perarnau, Vittorio Selo, and Iat Chong Chan. 2020. Nstm: Real-time query-driven news overview composition at Bloomberg. In ACL (System Demonstrations), pages 350-361.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The sixth pascal recognizing textual entailment challenge. In Text Analysis Conference. NIST.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008.
Alireza Bonyadi and Moses Samuel. 2013. Headlines in newspaper editorials: A contrastive study. Sage Open, 3(2):2158244013494863.
Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020. GoodNewsEveryone: A corpus of
news headlines annotated with emotions, semantic roles, and reader perception. In LREC, pages 1554-1566.
Houda Bouamor, Aurélien Max, and Anne Vilnat. 2012. Une étude en 3d de la paraphrase: types de corpus, langues et techniques (a study of paraphrase along 3 dimensions: Corpus types, languages and techniques)[in french]. In JEP-TALN-RECITAL 2012, volume 2: TALN, pages 267-280.
Peter Bourgonje, J. M. Schneider, and Georg Rehm. 2017. From clickbait to fake news detection: An approach based on detecting the stance of headlines to articles. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, pages 84-89.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics.
Sudatta Chowdhury and Monica Landoni. 2006. News aggregator services: user expectations and experience. Online Information Review, 30(2):100-115.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING, pages 350-356.
Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, and online news consumption. *Public opinion quarterly*, 80(S1):298-320.
Rama Rohit Reddy Gangula, Suma Reddy Duggenpudi, and R. Mamidi. 2019. Detecting political bias in news articles using headline attention. In ACL 2019.
David Graff, Junbo Kong, Kazuaki Maeda, and Stephanie Strassel. 2006. Tdt5 multilingual text. Technical Report LDC2006T18, Linguistic Data Consortium, Philadelphia, PA.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. In ACL, pages 5082-5093.
Aditya Joshi, Kashyap Popat, Shubham Gautam, and P. Bhattacharyya. 2013. Making headlines in Hindi: Automatic English to Hindi news headline translation. In IJCNLP.
Philippe Laban and Marti A Hearst. 2017. newsLens: building and visualizing long-ranging news stories. In Proceedings of the Events and Stories in the News Workshop (ACL 2017), pages 1-9.
Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet physics. Doklady, 10:707-710.
Andrew Moore and Paul Rayson. 2017. Lancaster at SemEval-2017 task 5: Evaluation metrics matter: predicting sentiment from financial news headlines. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 581-585, Vancouver, Canada. Association for Computational Linguistics.
Asli Eyecioglu Ozmutlu. 2016. Paraphrase identification using knowledge-lean techniques. Ph.D. thesis, University of Sussex.
Ekaterina Pronoza, Elena Yagunova, and Anton Pronoza. 2015. Construction of a Russian paraphrase corpus: unsupervised paraphrase extraction. In *Russian Summer School in Information Retrieval*, pages 146-157. Springer.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In ACL (Short Papers), pages 784-789.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In ACL, pages 6174-6184.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6649-6658.
Yusuke Shinyama, Satoshi Sekine, and Kiyoshi Sudo. 2002. Automatic paraphrase acquisition from news articles. In Proceedings of the second international conference on Human Language Technology Research, pages 313-318. Morgan Kaufmann Publishers Inc.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2010. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research, 11:2837-2854.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In *NAACL*, pages 1112–1122.
Sander Wubben, Antal Van Den Bosch, Emiel Krahmer, and Erwin Marsi. 2009. Clustering and matching headlines for automatic paraphrase acquisition. In Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009), pages 122-125.
W. Xu, Alan Ritter, Chris Callison-Burch, W. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from twitter. Transactions of the Association for Computational Linguistics, 2:435-448.
A Annotator Instructions
The wording of the prompt given to the eight crowd annotators we recruited was the following:
Your task will be to annotate News Headline timelines, and decide which are referring to the same event.
You are given a list of news headlines in chronological order, with a headline on each line of a Spreadsheet. For each headline, the task is to assign it a number: either a new number if the headline represents a new event that hasn't appeared yet, or the number of the existing headline it is a "repetition" of.
For each headline, you are also given a date of publication, which you can use to determine whether two headlines should be in the same event as well: two headlines several months apart must be about different events, even if they are very close lexically (protests in June and September are different events).
In some cases, headlines in the timelines were too vague or did not describe an event specifically, and annotators were encouraged to put such headlines in a group of their own.
B Model Size and Hyper-parameters
In order to ease reproducibility, we report relevant hyper-parameters of models whose results are present in Section 4. We used implementation of Transformer models from the HuggingFace Transformer library<sup>5</sup>. For Electra models, we initialized using the electra-base-discriminator. For GPT-2 based models, we initialized with the gpt2 model (which corresponds to a base model as well). Additional model-specific details:
- Electra MRPC Zero-shot: The model produces a probability for label 1:  $P(Y = 1|X)$ . If this probability is above a threshold  $T$ , the model predicts a 1, and below it predicts a 0.  $T = 0.23$  for this model.
- Electra MRPC Zero-shot + Time: We use a time constant of  $\lambda = 0.15$ , and  $T = 0.14$  for this model.
- Headline Generator Swap: the threshold for this model is  $T = 0.0012$ . This might seem small, but it is the conditional probability of a headline according to the  $GPT - 2$  model, and corresponds to a log-probability of  $-6.75$ .
- Headline Gen. Swap + Time: we use a time constant  $\lambda = 0.07$ , and a threshold of  $T = 0.00056$ , which corresponds to a log-probability of  $-7.49$ .
- Supervised models: All supervised models are trained for 3 epochs, with a batch size of 32, and the Adam Optimizer with a learning rate of  $LR = 10^{-5}$  with an exponential decay and a linear-warmup over the first 1000 steps. All weights of the model are finetuned.
C HLGD Format and Removal process
The dataset is a JSON file that can be processed using standard JSON parsing libraries. Each entry in the JSON object follows the schema:
The dataset will include scripts for processing the url for accessing the full content of the articles and other article data and for an option for content owners to request removal.
D Excerpt of Timeline
Table A1: Excerpt of the timeline about the International Space Station in HLGD. Group is the global group, aggregating labels from the five annotators. The full timeline contains 257 headlines and 107 distinct groups.