SlowGuess/ACL_2020_OCR huggingface dataset:
Shruti Rijhwani*
srijhwan@cs.cmu.edu
Daniel Pretoic-Pietro
Natural language processing models often have to make predictions on text data that evolves over time as a result of changes in language use or the information described in the text. However, evaluation results on existing data sets are seldom reported by taking the timestamp of the document into account. We analyze and propose methods that make better use of temporally-diverse training data, with a focus on the task of named entity recognition. To support these experiments, we introduce a novel data set of English tweets annotated with named entities.<sup>1</sup> We empirically demonstrate the effect of temporal drift on performance, and how the temporal information of documents can be used to obtain better models compared to those that disregard temporal information. Our analysis gives insights into why this information is useful, in the hope of informing potential avenues of improvement for named entity recognition as well as other NLP tasks under similar experimental setups.
Natural language processing models are now deployed on a large scale in many applications and used to drive automatic analyses or for making predictions. The usual setup is that these models are trained and evaluated on the data available at model building time, but are used to make inferences on data coming in at a future time, making models susceptible to data drift. The data distribution of the test set used to measure the model's performance after training may be different from the distribution of data from future time periods (Huang and Paul, 2018). This temporal drift in data often results in lower performance during inference. Drift is especially prevalent in information extraction tasks,
such as named entity recognition (NER), where the context and the target entities differ across time as a result of changes in language use or the events being discussed (Derczynski et al., 2016).
Despite its intuitive value, there has been little research on using the temporal information contained in text documents to inform modeling of a task (Huang and Paul, 2018; He et al., 2018), and no past research on modeling sequence labeling tasks in particular. Since sequence labeling models are currently trained and evaluated by randomly splitting the available data, performance is measured in an artificially temporal drift-free scenario that is not realistic or similar to how models are used in practice (Dredze et al., 2010). When splitting the available training data temporally and testing on the data from the most recent time period, we formulate the following hypotheses:
a) models trained on data from a closer time to the test set obtain better results, assuming the same model and data size are used;
b) models trained on the combined data from all time periods outperform models trained on subsets of the data, as more data usually leads to better models. In these cases, the commonly used setup of pooling all the data for training while disregarding temporal information may lead to sub-optimal performance.
In this paper, we study the temporal aspects of text data, focusing on the information extraction task of named entity recognition in the Twitter domain. We make the following contributions:
a) a new data set for Twitter Named Entity Recognition consisting of 12,000 English tweets evenly distributed across six years;
b) experimental results that demonstrate the performance drift of models trained on data from
*Work done during an internship at Bloomberg  
<sup>1</sup>Our data set is available at https://github.com/shrutirij/temporal-twitter-corpus.
7605
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605-7617 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
different time periods and tested on data from a future interval;
c) extensive analysis of the data that highlights temporal drift in the context of named entities and illustrates future modeling opportunities;
d) simple extensions to state-of-the-art NER models that leverage temporal information associated with the training data, which results in an improvement in F1 score over standard pooling methods.
Language change is a popular topic of research in linguistics (Stephen, 1962). In natural language processing, using data from online platforms such as Twitter or discussion fora, language change and adoption have been studied at the community level (Danescu-Niculescu-Mizil et al., 2013; Eisenstein et al., 2014; Goel et al., 2016; Stewart and Eisenstein, 2018) and at the individual level (Zhang et al., 2019). In some cases, the senses of the same word are known to shift over time (Wijaya and Yeniterzi, 2011), and modeling such changes in word semantics has been explored using diachronic word embeddings (Kulkarni et al., 2015; Hamilton et al., 2016; Kutuzov et al., 2018).
Temporal information has been used to create topic models of better quality, usually by adding smoothing properties (Blei and Lafferty, 2006; Wang et al., 2008). For text classification, the temporal periodicity of Twitter hashtags was modeled in Preojuc-Pietro and Cohn (2013) and used as a prior for text classification models for predicting hashtags on future data, which resulted in performance improvements.
Most similar to our experimental setup, Huang and Paul (2018) study the impact of temporal data splits in text classification, finding that performance worsens on data from future periods, and use standard domain adaptation techniques to incorporate time information and improve results. He et al. (2018) introduce a method for training neural networks on data from multiple time intervals while enforcing temporal smoothness between representations. Temporal information has also been used to improve named entity disambiguation on a data set of historical documents (Agarwal et al., 2018). Finally, Huang and Paul (2019) present a model that uses diachronic word embeddings combined with a method inspired by domain adaptation to improve document classification.
A related, but distinct, task built on the assumption of language change with time is automatic prediction of the date on which a document is written (Kanhabua and Nørvåg, 2008; Chambers, 2012; Niculae et al., 2014).
Named entity recognition (NER) is the task of identifying entities such as organizations, persons, and locations in natural language text. NER is a well-studied NLP task over the past 20 years (Nadeau and Sekine, 2007; Yadav and Bethard, 2018) and is a key information extraction task as its used in various downstream applications such as named entity linking (Cucerzan, 2007), relation extraction (Culotta and Sorensen, 2004) and question answering (Krishnamurthy and Mitchell, 2015). On social media text, such as tweets, the performance lags far behind that of standard news corpora (Derczynski et al., 2015b), with data drift as one of the suggested causes (Derczynski et al., 2015a). Agarwal et al. (2020) show that NER models decay substantially on entity mentions from a different distribution than those seen in training.
NER systems struggle to generalize over diverse genres with limited training data (Augenstein et al., 2017). Domain adaptation for NER (Chiticariu et al., 2010; Lin and Lu, 2018; Wang et al., 2020) is related to our task of improving performance over temporal drift, as the data from a future time period can be considered as a target domain with an unknown distribution. However, the relationship between domains is implied from temporal similarity, and temporal information is very fine-grained in contrast to the standard single source to single target domain adaptation setup.
3 Temporal Twitter Data Set
In this paper, we focus on the task of named entity recognition on English tweets as a case study for our hypotheses and analysis regarding model drift with time. Twitter data represents an ideal testbed for our analysis as it contains readily accessible timestamp information for each tweet. Further, users on social media post about current events, which are likely to include entities that change over time. Social media also reflects changes in language use more timely than other sources of data (e.g., newswire), resulting in the potentially rapid evolution of the contexts and ways in which named entities are discussed in natural language. This drift in Twitter data has previously been demonstrated qualitatively in the context of named entity
7606
Year	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019
Broad Twitter Corpus	5	127	2,414	275	6,022	-	-	-	-	-
Current Data Set	-	-	-	-	2,000	2,000	2,000	2,000	2,000	2,000
Table 1: Number of tweets from each year in the BTC data set and in the data set introduced in this paper.
recognition (Derczynski et al., 2015a).
Previous research has introduced data sets of tweets annotated with named entities, including the data sets from Finin et al. (2010), Ritter et al. (2011), Liu et al. (2011), the WNUT-17 Corpus (Derczynski et al., 2017), the Microposts NEEL Challenge Corpora (Rowe et al., 2013; Cano et al., 2014; Rizzo et al., 2015; Cano et al., 2016) and the Broad Twitter Corpus (Derczynski et al., 2016). However, these data sets usually consist of tweets collected within a limited time period, making them unsuitable for our proposed work. Of note is the Broad Twitter Corpus (Derczynski et al., 2016), which contains tweets collected over several years, from 2009 to 2014. However, the majority of tweets are from either 2012 or 2014, with fewer than 300 tweets from the other years (details in Table 1). Further, combining existing data sets is challenging, because of the different entity tagging schemes, annotation guidelines and sampling strategies used.
Therefore, we create a new collection of tweets annotated with named entities that attempts to alleviate the lack of temporal diversity in existing Twitter data sets as well as provide us with a suitable experimental setup to study our research questions about temporal entity drift and NER model performance. In this section, we present the details of our data set, including the collection and annotation methodology, as well as an analysis of the named entity mentions in the corpus. The data set can be downloaded at https://github.com/shrutirij/temporal-twitter-corpus.
The primary goal of creating a new data set is ensuring wide-enough temporal diversity for our work as well as future directions that can leverage timestamp information. We use the public Twitter Search  $\mathrm{API}^2$  to sample tweets spanning six years: 2014, 2015, 2016, 2017, 2018 and 2019.
We aim to ensure that the data set is represent-
tative of multiple English-speaking locales and a variety of topics, as well as making it comparable to existing data sets. Thus, we follow the same sampling strategy for corpus diversity used by the creators of the Broad Twitter Corpus (Derczynski et al., 2016). Specifically, we collect tweets across six English-speaking regions (the United States, the United Kingdom, New Zealand, Ireland, Canada, and Australia), and focus on two contrasting sets of Twitter handles: a) the twitterati, i.e., individuals from array of domains including musicians, journalists and celebrities; b) Twitter accounts for mainstream news organizations, covering both larger networks like CNN and ABC, as well as local news outlets. The Twitter handles correspond to users from the segments F and G of the Broad Twitter Corpus (Derczynski et al., 2016).
Overall, to maintain uniformity across time, we annotated 2,000 tweets for each year from 2014 to 2019 by randomly subsampling tweets from each year. This resulted in a temporally varied and balanced corpus of 12,000 tweets. Table 1 illustrates the temporal data distribution of our data set, as compared to the Broad Twitter Corpus.
In annotating our data with entities, we use a tagset consisting of three entity classes – Organizations (ORG), Persons (PER), and Locations (LOC). This scheme is consistent with some existing data sets for the task (Finin et al., 2010; Derczynski et al., 2016), overlapping with the majority of other general NER datasets in the social media domain (Liu et al., 2011; Rowe et al., 2013) and beyond (Tjong Kim Sang and De Meulder, 2003a).
We use the annotation guidelines used in standard NER data sets (Tjong Kim Sang and De Meulder, 2003a) supplemented with examples that are specific to Twitter data.
Further, we observe in other data sets that usernames are some of the most frequent tokens classified as entities (Ritter et al., 2011; Derczynski et al., 2016). For our experiments, we consider all usernames as non-entities, as otherwise, identifying
2https://developer.twitter.com/en/docs/tweets/search/overview
these using character features would be trivial, and typing entities would be similar to the task of Twitter handle classification (McCorriston et al., 2015; Wood-Doughty et al., 2018), which is outside the scope of the current paper.
We preprocess the data set by normalizing URLs, usernames, and Twitter-specific tokens (e.g., RT). We leave hashtags intact as these are often used as words in the context of the tweet, and can be or contain named entities. We use Twokenizer (O'Connor et al., 2010), a Twitter-specific tokenizer to split the tweets into tokens. To limit the impact of imperfect tokenization on the performance of the NER models - especially in the case of hashtags containing multiple tokens (Maddela et al., 2019) - we expanded sub-token annotations to their closest matching token. If multiple sub-token entity annotations match the same token, then we select the label of the first sub-entity in order of appearance.
The data was annotated by multiple annotators that have experience with named entity recognition annotation tasks. Specifically, we used 15 annotators in total, with two annotations per tweet. The inter-annotator agreement is  $78.34\%$  on full tweets (same entity types and spans). If the annotators disagree on a tweet in their tagging, we adjudicate in favor of the annotator that had the highest confidence on the task, as judged through measuring their agreement with our annotations on a set of test questions ( $10\%$  of the total).
In our experiments, we use temporal splits of the data from 2014-2018 for training, and the most recent data (i.e., the tweets from 2019) to evaluate our models, to simulate a "future time period" setup. Thus, we wanted to ensure that the model performance is evaluated on data that has as few annotation errors as possible. Hence, each tweet was checked by either of the authors of the paper, both with significant experience in linguistic annotations, and corrected if needed to ensure additional consistency. This process had the effect of reducing the measurement error of the model performance but ultimately did not affect the conclusions of the experimental results. The type-wise distribution of named entities in for each year in our data set, after annotator adjudication and correction, is shown in Table 2.
4 Base Model Architecture
This section describes the base model architecture we use to perform named entity recognition ex
Year	PER	ORG	LOC	Total
2014	371	454	350	1,175
2015	363	479	393	1,235
2016	435	501	320	1,256
2017	432	516	314	1,262
2018	468	597	395	1,460
2019	725	881	475	2,081
Table 2: Year-wise number of named entities of each type in the data set introduced in this paper.
periments throughout the paper. We use the same underlying architecture to provide a controlled experimental setup and isolate temporal modeling aspects from other model-related factors.
4.1 Neural Architecture
We use the neural architecture based on a stacked BiLSTM-CRF model introduced in Huang et al. (2015), which is the core model architecture for several state-of-the-art NER results over the past years (Lample et al., 2016; Peters et al., 2018; Akbik et al., 2018). For each sentence, the token representations are fed into two different LSTM layers, each processing the sentence in different directions (one forward and one backward). The output of these two layers are concatenated and passed through a feed-forward layer that produces a distribution over the output tag space. Finally, a Conditional Random Field is applied to the class predictions with the role of jointly assigning predictions to the entire sequence. This also has the function of ensuring that the output tag sequence takes into account the constraints of the IOB2 entity tagging scheme (e.g. I-LOC cannot follow B-ORG) (Tjong Kim Sang and De Meulder, 2003b).
A key component in the base architecture is how the tokens are represented as inputs. Initial research (Lample et al., 2016) on LSTM-CRF models use static pre-trained word embeddings, such as GloVe (Pennington et al., 2014), to initialize the inputs, which are subsequently fine-tuned on the NER training data. More recently, contextual word embeddings, which represent each token differently based on its context, were shown to obtain an improvement of 2-3 F1 points on the English news CoNLL data set (Peters et al., 2018; Akbik
et al., 2018; Devlin et al., 2019). In this paper, we conduct experiments with both the static GloVe embeddings (Pennington et al., 2014) and the state-of-the-art contextual Flair embeddings (Akbik et al., 2018) to test the robustness of our findings to different input representations. All embeddings were trained outside of the time range of our data: the GloVe embeddings were trained on Twitter data before 2014, while the Flair embeddings were trained on the 1-billion word corpus (Chelba et al., 2013) which contains data up to 2012. Exploiting embeddings trained on data more recent than the NER corpus is an avenue of future work.
In addition to token embeddings, we use character embeddings to model subword information that may be indicative of named entities and better represent out-of-vocabulary tokens. We use a character-level BiLSTM with randomly initialized character embeddings to produce the character-based word representations (Lample et al., 2016). These are concatenated to the token embeddings described above and then used as input to the token-level BiLSTM.
4.3 Data Split
We split the data temporally for our experiments. We use the data authored in 2019 as the test data, as this is the most recent data available and best replicates the scenario of making predictions on text from future time periods. We use a random sample of 500 tweets (25%) from the 2019 data as the validation set.
For training, we use data authored between 2014 to 2018 in various temporal splits, depending on the specific experimental setting.
4.4 Implementation and Hyperparameters
We use the PyTorch framework (Paszke et al., 2017) for the implementation of the models. For the model using the GLoVe embeddings, we use the same hyperparameter settings as the original creators of the base models (Lample et al., 2016; Akbik et al., 2018) and ensure the correctness of our implementation by replicating their results on the CoNLL-2003 English NER data set (Tjong Kim Sang and De Meulder, 2003a). Specifically, the character embeddings are of size 32, the character-level LSTM hidden size is 64, and the word-level LSTM has a hidden size of 256. We also use a dropout of 0.5 on the input word embeddings and replace singleton words in the training set with an out-of-vocabulary symbol with a proba
Figure 1: Evaluating the effect of temporal distance: the model is trained on each year individually. F1 score on 2019 data averaged over five random seeds is shown.
bility of 0.5 to improve robustness to unseen words. We use the flairNLP library (Akbik et al., 2019) for the contextual Flair embedding experiments, using the same hyperparameters as the state-of-the-art result in Akbik et al. (2018). For each experimental setting, we use the training checkpoint with the best performance on the validation set (i.e., early stopping).
Following the recommendation from Reimers and Gurevych (2017), who study the variance of LSTM-CRF models with different random seeds, we report all experimental results as the mean of five runs. The main metric we use for evaluation is span-level named entity F1 score, reported using the official CoNLL evaluation script (Tjong Kim Sang and De Meulder, 2003a).
5 Data Drift
To determine the utility of temporal information, we first attempt to evaluate whether temporal drift in the data affects the performance of NER models. To this end, we conduct experiments to answer the following research questions:
1) What is the effect of the temporal distance between the training and target data sets on NER performance?
2) How do the size and temporal distribution of the training data affect NER performance?
5.1 Effect of Temporal Distance
We empirically study the effect of temporal distance between the training and test data sets by training the base model on each year, from 2014-2018, individually. Based on the design of our data set, each model has access to the same number of
7609
(a) Pretrained GloVe embeddings.
(b) Pretrained Flair embeddings.
Figure 2: Cumulatively training the model on random subsamples of tweets from all the years (2014-2018) compared with temporally adding tweets to the training data, starting from the year 2014 and cumulatively adding data from subsequent years.
training instances (2,000 tweets), to remove the impact of this factor in our results.
The results are shown in Figure 1. We observe that the temporal distance between the training and test sets seems to affect NER performance. The F1 score increases as we move temporally closer to the target data, for both the GloVe and Flair embeddings, apart from a slight decrease when moving from 2016 to 2017 when using GloVe embeddings. When using the contextual Flair embeddings, the performance numbers are overall higher, which is consistent with past research (Akbik et al., 2018), as contextual embeddings are more expressive.
5.2 Effect of Data Set Size and Distribution
We now study how the number of instances in the training data and their temporal distribution impact the performance of the model. We first train models on cumulative random samples from the combined training data set (all tweets from 2014-2018), adding 2,000 tweets at each step. Then, we train models starting with the 2,000 tweets from
Figure 3: Type distribution across years in our data set.
2014 and incrementally add tweets from subsequent years from 2015 up to 2018.
The NER F1 scores are shown in Figures 2a and 2b, with both "Random" and "Temporal" cumulative compositions of the training data set.
Looking at the "Random" sampling strategy, we see that the performance steadily increases as we add more tweets to the training set - as we would expect for most supervised machine learning models. We see that the "Temporal" model with only the 2014 data (2,000 tweets) has a lower performance than randomly selecting 2,000 tweets across all years. This is indicative of the data drift across time, as training on a random sample of tweets from all the years is more informative and leads to a better NER model than using just the 2014 data.
Moreover, as we add tweets temporally closer to the target into the training data set, the "Temporal" strategy converges with the "Random" strategy. This observation strengthens the hypothesis that temporal information can potentially play an important role while selecting training data and designing model architectures.
To understand why the temporal distribution of the training data impacts the performance of an NER model, we analyze the distribution of entity mentions in our data set to uncover the extent to which data drift occurs at the lexical level.
Type Distribution Figure 3 shows the distribution of entity types across years in our data set. The distribution looks approximately even, with minor differences in the fraction of location (LOC) entities. Since similar types of entities occur in the data set year-wise, this likely does not cause the change in performance across time indicated in the previous sections.
7610
Figure 4: Percentage overlap of unique surface forms of named entity mentions for years 2014 to 2018, with respect to the year 2019. The gradual increase in overlap is an indication of temporal drift.