SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 5: Type-wise percentage overlap of unique mentions for years 2014 to 2018, with respect to the year 2019. For all types, there is a general increase in overlap as we move temporally closer to 2019.
Mention Overlap Figure 4 presents the overlap of unique entity mentions with respect to the test data (2019). There is a clear increase in surfaceform overlap as we get temporally closer to the target data, and is potentially an important factor for the F1 score improvement we see in our empirical analysis.
Type-wise Mention Overlap Figure 5 shows the surface-form overlap of entity mentions over types of years 2014 to 2018, with respect to the data from 2019. The figure adds further evidence of temporal data drift at the mention level. For all three entity types (LOC, PER, ORG) in our data set, smaller temporal distance leads to a greater percentage of overlap. Interestingly, the PER overlap
Train	GloVe	Flair
PER	ORG	LOC	PER	ORG	LOC
2014	74.45	41.63	52.78	79.66	53.78	56.90
2015	73.39	45.97	52.14	81.91	52.23	58.77
2016	78.42	49.12	57.60	81.58	58.19	60.85
2017	74.63	51.23	52.97	81.82	60.10	58.41
2018	79.40	56.29	59.25	83.47	61.83	64.90
Table 3: Type-wise F1 when testing on data from 2019. The models are trained on each year individually. The training data sets are of the same size (2,000 tweets).
is much lower than other types. This is expected, as the people discussed on social media rapidly change with developments in current events (Derczynski et al., 2017). We see that the 2017 data set has a lower overlap for LOC than both 2016 and 2018, which could explain the off-trend performance of the 2017 model in our empirical results (Figure 1).
Type-wise Model Performance Table 3 shows the NER performance by entity type, to gain more insight into which types are affected by data drift.
First, we notice that the improved performance of Flair embeddings seen in previous analyses is caused by better performance across all types. Overall, the PER type obtains the best performance for both models, with an F1 of around 20 points higher than the other two types. This is despite the fact that the PER type has the lowest overall overlap between training and test, which indicates that the model is adequately learning the contexts that PER entities appear in. ORG and LOC show similar absolute performance in both setups.
Next, we study the temporal differences in performance by type. When using GloVe embeddings, the smallest gap between training on different data splits is for PER (4.95 F1), while ORG suffers from substantial drift in performance, resulting in a 14.66 F1 drop on ORG performance. When using Flair embeddings, the most notable difference in performance when training across different years is still for the ORG type (up to 8.05 F1). However, the gap has proportionally tightened the most as compared to when using GloVe embeddings. These observations correspond with the analysis from Figure 5, where we see the largest increase in overlap between mentions from the training data and the test data over the five years ( $\tilde{8}\%$  increase for ORG, compared to  $3 - 4\%$  increase for LOC and PER).
7611
Train	Unseen Recall
GloVe	Flair
2014	53.72	56.62
2015	54.22	56.64
2016	58.90	58.98
2017	59.61	57.88
2018	64.36	60.00
Table 4: Recall for named entity mentions in the test data (2019) unseen in the training data, for models trained on each year individually.
We also observe that the slight drop in performance of the model using GloVe embeddings trained on the 2017 data is caused primarily by a decline in performance on the LOC type which holds across both models.
Mentions Unseen in the Training Data In addition to the increase in surface-form overlap across years, we investigate whether mentions unseen in the training data are impacted by the temporal distance between the training and test data. Table 4 shows the recall for these mentions using both the GloVe and Flair embeddings. Notably, for GloVe, the performance steadily improves as the temporal distance decreases, with an almost 5 point improvement in recall when moving from 2017 to 2018. Although less pronounced, there is a similar trend with the Flair embeddings. This indicates that surface-form overlap is not the only factor determining temporal data drift. The model is potentially able to learn more relevant context from the training data of temporally close years, perhaps due to changes in language use over time.
6 Modeling Temporal Information
Supported by the analysis that temporal drift in the training data can impact the performance of NER systems, in this section, we experiment with techniques to account for temporal information while training the NER model. We look at leveraging temporality in two broad ways: a) by altering the architecture of the base model; b) by modifying how the training data set is constructed. These methods are intended to be an initial exploration of using temporal information, with a focus on techniques that do not require significant modification to the base model. We present these in the hope
that they will inspire future research on models robust to temporal drift. The specific methods are discussed below, followed by experimental results.
Sequential Temporal Training Our analysis from Section 5 showed that using more data is beneficial, irrespective of temporal distance from the target, but individually, the closest data is most useful. Based on this analysis, we attempt to train our model by ordering our training data year-wise such that the model is trained on the temporally closest data last. Specifically, we start with training on the year temporally furthest away from the target data and repeatedly tune the model on the chronological sequence of years (i.e., first train on 2014 data, then 2015 data, and so on up to 2018).
Temporal Fine-tuning The analysis showed that training on the model temporally closest to the target data set obtains the best overall performance. Based on this observation, we decide to train the base model on the entire data set of tweets from the years 2014-2018. Then, we fine-tune the trained model on the data from the year temporally closest to the target (2018). The fine-tuning process is simply retraining the model on the 2018 data with the same hyperparameter settings.
Instance Weighting Previous work in domain adaptation shows that giving higher weights to training instances similar to the target domain can improve performance (Wang et al., 2017). Similarly, we decide to assign a higher weight to tweets temporally closer to the test data (i.e., the 2018 tweets are up-weighted). In our experiments, we up-weight the tweets by a factor of 2.
We note that the above methods do not require any change to the model, making integration of these methods for existing systems very practical.
Year Prediction as an Auxiliary Task Finally, we aim to guide the model to learn temporal features in training. Inspired by related work in domain adaptation (Chen et al., 2018), we enhance the architecture with a multi-task learning component that models an auxiliary task. While training the model for NER, this component uses the LSTM hidden states to predict the year that the tweet was created in. Since the input embeddings and the LSTM are shared between the NER task and the year prediction task, the intuition is that the parameters learned will retain a notion of temporality that
7612
GloVe	Flair
Base Model	70.80	74.72
Sequential Temporal Training	68.47	74.42
Temporal Fine-tuning	71.93	74.95
Instance Weighting	70.59	75.54
Year Prediction	71.01	74.70
Table 5: Performance of proposed methods of using temporal information in NER modeling when compared to the base model. Results are F1 scores averaged over five runs with different random seeds. Bold indicates the best F1 score.
can influence the NER prediction. The training objective is the sum of the NER loss and the auxiliary task loss.
Table 5 presents the experimental results. The base model combines the training data (2014-2018) without using any temporal information, the current standard setup for most NLP systems.
The results show that we can overall obtain a better performance over the base model by using simple techniques to incorporate temporal information. The margin of improvement is overall lower when using Flair embeddings than with GloVe (+0.82 compared to +1.13). This potentially indicates that semantic drift can be captured partially through contextual embeddings.
Fine-tuning the model on the temporally closest data (i.e., 2018) leads to the best F1 scores when using GloVe embeddings, reaching a 1.13 increase in F1. For the Flair embeddings, we observe that up-weighting the training instances from the year 2018 leads to the best result, a 0.82 improvement in F1 over the base model.
We highlight that these straightforward methods that improve over the base model do not involve any architecture changes, other than a change in how the data is fed to the model. It thus has the potential to both be readily applicable to existing NER implementations as well as generalize to other NLP tasks.
Finally, we find that using an auxiliary task for predicting the year improves the performance slightly when using GloVe embeddings, but has the oposite effect when using Flair embeddings. This is likely because the GloVe embeddings are finetuned during the model training and are therefore
influenced by the auxiliary loss, while the contextual Flair embeddings are not.
This paper studies and models text data drift in the information extraction task of named entity recognition. We introduce a new data set of 12,000 English tweets stratified by time, which allows us to study the effects of drift and evaluate named entity recognition models in a realistic scenario of performing inference on temporally unseen data. By analyzing the data, we quantify the temporal drift in named entity type and mention usage and identify that, as expected, the data distribution is more similar when drawn from closer time intervals. We then use current state-of-the-art approaches for named entity recognition and demonstrated that, through modeling of temporal information, performance can be improved when testing on future data. We expect our data, results, and error analysis to inform the design of similar experimental setups for other NLP tasks beyond NER, such as part-of-speech tagging or relation extraction.
We would like to thank Leslie Barrett, Liang-Kang Huang, Prabhanjan Kambadur, Mayank Kulkarni, Amanda Stent, Umut Topkara, Jing Wang, Chuck-Hou Yee and the other members of the Bloomberg AI group. They provided invaluable feedback on the experiments and the paper. We also thank the anonymous reviewers for their valuable suggestions. Shruti Rijhwani is supported by a Bloomberg Data Science Ph.D. Fellowship.
Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and Ani Nenkova. 2020. Interpretability analysis for named entity recognition to understand system predictions and how they can improve. *ArXiv*, abs/2004.04564.
Prabal Agarwal, Jannik Strötgen, Luciano del Corro, Johannes Hoffart, and Gerhard Weikum. 2018. diAned: Time-aware named entity disambiguation for diachronic corpora. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 686-693, Melbourne, Australia. Association for Computational Linguistics.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019.
7613
FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54-59, Minneapolis, Minnesota. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. Computer Speech & Language, 44:61-83.
David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, ICML, pages 113-120.
Amparo E Cano, Daniel Preotiuc-Pietro, Danica Radovanovic, Katrin Weller, and Aba-Sah Dadzie. 2016. # microposts2016: 6th workshop on making sense of microposts: Big things come in small packages. In Proceedings of the 25th International Conference Companion on World Wide Web, pages 1041-1042.
Amparo E Cano, Giuseppe Rizzo, Andrea Varga, Matthew Rowe, Milan Stankovic, and Aba-Sah Dadzie. 2014. Making sense of microposts: (#microposts2014) named entity extraction & linking challenge. In Proceedings of the 23rd International Conference Companion on World Wide Web, volume 1141, pages 54-60.
Nathanael Chambers. 2012. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 98-106, Jeju Island, Korea. Association for Computational Linguistics.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. ArXiv, abs/1312.3005.
Cen Chen, Yinfei Yang, Jun Zhou, Xiaolong Li, and Forrest Sheng Bao. 2018. Cross-domain review: helpfulness prediction based on convolutional neural networks with auxiliary domain discriminators. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 602-607, New Orleans, Louisiana. Association for Computational Linguistics.
Laura Chiticariu, Rajasekar Krishnamurthy, Yunyao Li, Frederick Reiss, and Shivakumar Vaithyanathan. 2010. Domain adaptation of rule-based annotators for named-entity recognition tasks. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1002-1012, Cambridge, MA. Association for Computational Linguistics.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708-716, Prague, Czech Republic. Association for Computational Linguistics.
Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 423–429, Barcelona, Spain.
Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In Proceedings of the 22nd International Conference on World Wide Web, WWW, pages 307-318.
Leon Derczynski, Isabelle Augenstein, and Kalina Bontcheva. 2015a. USFD: Twitter NER with drift compensation and linked data. In Proceedings of the Workshop on Noisy User-generated Text, pages 48-53, Beijing, China. Association for Computational Linguistics.
Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1169-1179, Osaka, Japan. The COLING 2016 Organizing Committee.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015b. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32-49.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association
7614
for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mark Dredze, Tim Oates, and Christine Piatko. 2010. We're not in kansas anymore: Detecting domain changes in streams. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 585-595, Cambridge, MA. Association for Computational Linguistics.
Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. *PloS one*, 9(11):e113114.
Tim Finin, William Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 80-88, Los Angeles. Association for Computational Linguistics.
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Hanna Wallach, Fernando Diaz, and Jacob Eisenstein. 2016. The social dynamics of language change in online networks. In International Conference on Social Informatics, SocInfo, pages 41-57.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Computational Linguistics.
Yu He, Jianxin Li, Yangqiu Song, Mutian He, and Hao Peng. 2018. Time-evolving text classification with deep neural networks. In *IJCAI*, pages 2241-2247.
Xiaolei Huang and Michael J. Paul. 2018. Examining temporality in document classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694-699, Melbourne, Australia. Association for Computational Linguistics.
Xiaolei Huang and Michael J. Paul. 2019. Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4113-4123, Florence, Italy. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. ArXiv, abs/1508.01991.
Nattiya Kanhabua and Kjetil Nørvåg. 2008. Improving temporal language models for determining time of non-timestamped documents. In International Conference on Theory and Practice of Digital Libraries, pages 358-370.
Jayant Krishnamurthy and Tom M. Mitchell. 2015. Learning a compositional semantics for Freebase with an open predicate vocabulary. Transactions of the Association for Computational Linguistics, 3:257-270.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, WWW, pages 625-635.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384-1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Bill Yuchen Lin and Wei Lu. 2018. Neural adaptation layers for cross-domain named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2012-2022, Brussels, Belgium. Association for Computational Linguistics.
Xiaohua Liu, Shaodian Zhang, Furu Wei, and Ming Zhou. 2011. Recognizing named entities in tweets. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 359-367, Portland, Oregon, USA. Association for Computational Linguistics.
Mounica Maddela, Wei Xu, and Daniel Preoticiuc-Pietro. 2019. Multi-task pairwise neural ranking for hash-tag segmentation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2538-2549, Florence, Italy. Association for Computational Linguistics.
James McCorriston, David Jurgens, and Derek Ruths. 2015. Organizations are users too: Characterizing and detecting the presence of organizations on twitter. In Ninth International AAAI Conference on Web and Social Media, ICwsm.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes, 30(1):3-26.
Vlad Niculae, Marcos Zampieri, Liviu Dinu, and Alina Maria Ciobanu. 2014. Temporal text ranking and automatic dating of texts. In Proceedings of the 14th Conference of the European Chapter of the
7615
Association for Computational Linguistics, volume 2: Short Papers, pages 17-21, Gothenburg, Sweden. Association for Computational Linguistics.
Brendan O'Connor, Michel Krieger, and David Ahn. 2010. Tweetmotif: Exploratory search and topic summarization for Twitter. In Fourth International AAAI Conference on Weblogs and Social Media, ICwsm.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Preoticiuc-Pietro and Trevor Cohn. 2013. A temporal model of text periodicities using Gaussian processes. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 977-988, Seattle, Washington, USA. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348, Copenhagen, Denmark. Association for Computational Linguistics.
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '11, pages 1524-1534, Stroudsburg, PA, USA. Association for Computational Linguistics.
Giuseppe Rizzo, Amparo Elizabeth Cano Basave, Bianca Pereira, Andrea Varga, Matthew Rowe, Milan Stankovic, and A Dadzie. 2015. Making sense of microposts (microposts2015) named entity recognition and linking (neel) challenge. In Proceedings of the 24th International Conference Companion on World Wide Web, pages 44-53.