Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?
Year: 2023

Abstract:
The CoNLL-2003 English named entity recognition (NER) dataset has been widely used to train and evaluate NER models for almost 20 years. However, it is unclear how well models that are trained on this 20-year-old data and developed over a period of decades using the same test set will perform when applied on modern data. In this paper, we evaluate the generalization of over 20 different models trained on CoNLL-2003, and show that NER models have very different generalization. Surprisingly, we find no evidence of performance degradation in pre-trained Transformers, such as RoBERTa and T5, even when fine-tuned using decades-old data. We investigate why some models generalize well to new data while others do not, and attempt to disentangle the effects of temporal drift and overfitting due to test reuse. Our analysis suggests that most deterioration is due to temporal mismatch between the pre-training corpora and the downstream test sets. We found that four factors are important for good generalization: model architecture, number of parameters, time period of the pre-training corpus, in addition to the amount of fine-tuning data. We suggest current evaluation methods have, in some sense, underestimated progress on NER over the past 20 years, as NER models have not only improved on the original CoNLL-2003 test set, but improved even more on modern data. Our datasets can be found at https:// github.com/ShuhengL/acl2023_conllpp.

heading: 1 Introduction
text: The progress of natural language processing (NLP) is typically measured using performance metrics like accuracy or F1 score on public test sets. For instance, the top line in Figure 1 shows the steady improvement of selected models on the CoNLL2003 English named entity recognition (NER) test set (Tjong Kim Sang and De Meulder, 2003) over the course of 15 years (2005-2020) as measured by published F1 scores.
However, these scores are all calculated using the same publicly available test set, which raises several questions. One concern is how much of this progress is actually due to adaptive overfitting, i.e. over-estimating performance by reusing the same test set, as opposed to genuine improvement (Recht et al., 2019; Roelofs et al., 2019; Gorman and Bedrick, 2019). In addition, there is also the issue of temporal drift as training data ages, which can negatively impact performance on modern data (Rijhwani and Preotiuc-Pietro, 2020; Agarwal and Nenkova, 2022; Luu et al., 2022).
Performance degradation is a significant concern in applications that use NER, such as text deidentification (Morris et al., 2022), relation extraction (Zhong and Chen, 2021), linking entities to a knowledge base (De Cao et al., 2022), etc. However, continuously annotating, training, and evaluating new models on new data is not always possible. NER models that are trained on decadesold data and evaluated on heavily-used public development and test sets may struggle to perform well on modern data, which highlights the need to consider these factors when assessing performance.
8254
To understand how well NER works when models have been developed over 20 years using the same dataset, we created a new test set called CoNLL++. We closely modeled CoNLL++ after the CoNLL-2003 test set, using news articles from 2020 instead of 1996, as in the original dataset. We carefully controlled for other variables, making results on the two datasets as comparable as possible, with the exception of the time frame. An example of an annotated sentence from CoNLL++ is shown below:
AMBASSADOR O TO O THE O UNITED I-ORG NATIONS I-ORG : O LINDA I-PER THOMAS-GREENFIELD I-PER
Using CoNLL++, we conduct an empirical study of more than 20 NER models that were trained on the original CoNLL-2003 training split. Our analysis shows that different models can have very different generalization when moving to modern data. Simply comparing the performance of models on the CoNLL-2003 test set does not tell the whole story of progress on NER over the past 20 years.
Similar to the findings of Recht et al. (2019) on the ImageNet dataset (Deng et al., 2009), we do not observe evidence of widespread overfitting on CoNLL-2003. On average, each point of F1 improvement on the CoNLL-2003 test set translates to a larger improvement on CoNLL++ (see Figure 2), suggesting overall improvements on the original dataset between 2003-2020 are mostly not due to overfitting. Rather, most performance deterioration appears to be caused by temporal misalignment (Luu et al., 2022).
Suprisingly, for some models (e.g. RoBERTa and T5), we find no evidence of performance degradation at all, despite the fact they are fine-tuned on a 20-year-old public dataset. We conduct an extensive analysis, which suggests that model size,
architecture, amount of fine-tuning data, and pretraining corpus are all important factors for generalization in NER.

heading: 2 Annotating a New Test Set to Measure Generalization
text: Data Collection: The CoNLL-2003 shared task collected English data from the Reuters Corpus, including Reuters news articles published between Aug. 1996 and Aug. 1997. The test set was collected from December 1996 according to Tjong Kim Sang and De Meulder (2003). We find that almost all articles were published between Dec. 5th and 7th, 1996, except one article published on Nov. 29th and another on Dec. 16th. Our dataset follows this distribution to collect Reuters news articles published between December 5th and 7th, 2020, collected from the Common Crawl Foundation.2 We tokenize the data with the same tokenizer used for the CoNLL-2003 shared task, and randomly select articles to match the total number of tokens in the original test set. Annotation: We manually labeled this new dataset, which we refer to as CoNLL++, using the BRAT annotation interface (Stenetorp et al., 2012). Articles were distributed between two authors, where one author annotated 96.1% of the articles and the other annotated 50.0%. The first author’s annotation is used as the gold standard.3 During the annotation process, articles from the CoNLL-2003 test set were interleaved with new articles from
1We notice that our dataset contains fewer entities than CoNLL-2003. This is mainly because there are a number of tabular data, with information such as results of sports events (e.g. 1. Jesper Ronnback ( Sweden ) 25.76 points), in CoNLL2003. Such data greatly contribute to the number of entities. These tabular data also cause the average sentence length of CoNLL-2003 to be smaller than that of CoNLL++. By removing these data, we found that the average sentence length increased to 18.50, much more comparable to CoNLL++. Model perfomances reported in Figure 2 were also not affected by the removal of these tabular data. We include further analysis and explanation in the Appendix (§ A).
2http://commoncrawl.org/ 3Articles only annotated by the second author were re-
viewed and then used as the gold standard.
2020, in order to measure how closely the annotators follow the style of the original dataset. Inter-Rater Agreement: We find that the CoNLL++ annotations closely follow the style of the original dataset. When considering labels in the CoNLL-2003 test set as gold, our manual reannotation achieves a 95.46 F1 score.4 The second author’s annotation, when considering the first author’s as gold, receives a 96.23 F1 score on overlapping articles. The token-level Cohen’s Kappa between the two authors is 97.42, which can be considered almost perfect agreement (Artstein and Poesio, 2008). Table 1 summarizes the statistics of
4For reference, the current state of the art for automated NER taggers is 94.60 (Wang et al., 2021).
the two datasets.

heading: 3 Experimental Setup
text: We select models with a variety of architectures and pre-training corpora and fine-tune these models to study how different factors affect generalization. None of the models used any pre-training data that temporally overlap with CoNLL++, eliminating the possibility of articles in CoNLL++ appearing in any pre-training corpus. A list of all models and their implementation details can be found in Table 2.
Scripts for fine-tuning Flair and ELMo are
adapted from Reiss et al. (2020). 5 Other recurrent neural network (RNN) models are trained using various GitHub repositories (see footnotes 6, 7 and 8). We fine-tune the BERT and RoBERTa models with the HuggingFace transformers library (Wolf et al., 2020), except LUKE with AllenNLP (Gardner et al., 2018). T5 is fine-tuned to conditionally generate NER tags around entities (e.g. <per> Jane Doe </per>).
A hyperparameter search is conducted for each model. We follow the recommended search space for a model if available in its publication. {8, 16, 32} and {1e-5, 2e-5, 3e-5, 5e-5} are used for most searches for batch sizes and learning rates respectively. Appendix B provides more details on the hyperparameter search.
We train models on the CoNLL-2003 training set for 10 epochs, and use the dev set to select the best epoch and other hyperparameters for evaluation. Each model is evaluated five times with different random seeds on the CoNLL-2003 test set and on CoNLL++ to obtain the average F1.
In Table 2, we report the percentage change of F1, calculated as:
∆F1 = F1CoNLL++ − F1CoNLL-2003
F1CoNLL-2003 × 100
5Implementation from Reiss et al. (2021) 6Implementation from Jie (2020) 7Implementation from Kanakarajan (2019) 8Implementation from Reimers and Gurevych (2017)
where F1CoNLL++ and F1CoNLL-2003 are the F1 scores on the CoNLL++ and CoNLL-2003 test sets respectively. The results are visualized in Figure 2. Raw F1 scores are shown in Table 5 in the Appendix (§ C.1).

heading: 4 What Ingredients are Needed for Good Generalization?
text: As we can see in Figure 2 and Table 2, different models have very different generalization. Some models (e.g. RoBERTa-based models and T53B), have no performance drop on CoNLL++, whereas other models’ performances decrease significantly.
In the following sub-sections, we evaluate the impact of a number of factors on generalization. In §5, we attempt to disentangle to what extent the observed performance drops on CoNLL++ are caused by temporal deterioration, or adaptive overfitting.

heading: 4.1 Model Size
text: It has been shown that the size of pre-trained models affects their performance (Kaplan et al., 2020; Raffel et al., 2020). This inspired us to investigate the effect of model size on generalization. We compare the performance of BERT, RoBERTa, ALBERT and T5 models with different sizes on CoNLL++ and CoNLL-2003. The results are visualized in Figure 3. Details are available in Table 6 in the Appendix (§ C.2).
We observe, from Table 6, that larger models perform better on both test sets, but more impor-
tantly, as illustrated in Figure 3, performance degradation on CoNLL++ diminishes or even disappears as the model size grows. The only exception is the RoBERTa-based models, whose base-sized model already achieves comparable performance on CoNLL++. Figure 3 suggests that larger model sizes not only increase performance on a static test set, but also help models generalize better to new data.
It is also informative to look at the individual trend within each model family. Whereas T5 models exhibit a linear relationship between the log number of parameters and ∆F1, the improvement of ∆F1 for ALBERT models diminishes as the size grows larger. Additionally, models of similar sizes do not necessarily exhibit similar generalization. For example, BERTBase & RoBERTaBase (∼100M), ALBERTXXLarge & T5Base (∼220M) and BERTLarge & RoBERTaLarge (∼300M) all have similar sizes, but the performance changes within each pair are very different.
Both RoBERTaLarge and T53B achieve a performance increase of ∼0.6%, but the number of parameters of T53B is ∼10 times of that of RoBERTaLarge. This shows that the generalizability of model is also affected by factors other than the size of the model, but with the same architectures, larger models tend to generalize better.

heading: 4.2 Model Architecture
text: Based on the results from Table 2, we also observe that model architecture has a significant impact on generalizability. Most BERT, RoBERTa and T5 models have a small performance drop (less than 4% F1) on CoNLL++. The performances of RoBERTaLarge, news-RoBERTaBase, LUKE and LongformerBase improved slightly on CoNLL++. The fact that most Transformer-based models achieve higher rankings in CoNLL++ also confirms that pre-trained Transformers generalize better to new data.
BiLSTM models with Flair and ELMo embeddings, despite performing exceptionally on CoNLL-2003, show larger performance drops on CoNLL++ (5-6% F1), and the performance of BiLSTM+GloVe models drops even more significantly (greater than 6% F1). Such results show a clear trend that Transformer-based models generalize better to new data.

heading: 4.3 Number of Fine-Tuning Examples
text: The generalizability of a model may also be affected by the size of the fine-tuning dataset. We conduct experiments varying the number of CoNLL-2003 training examples used for finetuning from 10% to 100%. The fine-tuning is done with RoBERTaBase and Flair embeddings using the same experimental setup as in Section 3. We plot the percentage change in F1 against the percentage of training examples in Fig 4.
Both RoBERTaBase and Flair embeddings show improved generalization as we use more training
examples. However, this improvement is more pronounced for Flair than RoBERTaBase. Even with 10% of the training data, RoBERTaBase already shows a positive change in F1 scores, and increasing the amount of training data to 100% only improves the change by an absolute value of 1%. In contrast, increasing the amount of training data from 10% to just 20% can already improve ∆F1 by 2% for Flair.
The empirical evidence supports our hypothesis that having more training examples can improve the generalizability of the model, but such effect may vary across different models. RoBERTa-based models generalize well to new data even when only a small amount of fine-tuning data is available, whereas Flair benefits much more from having more fine-tuning data.

heading: 5 What Causes the Performance Drop Observed for Some Models?
text: Models in Table 2 show different levels of performance drop, or sometimes performance gain, on CoNLL++ compared to the CoNLL-2003 test set, and it is not entirely clear what causes this difference. We hypothesize two potential causes, namely adaptive overfitting (§ 5.1) and temporal drift (§ 5.2). In this section, we investigate each of these potential causes.

heading: 5.1 Adaptive Overfitting
text: We first investigate if the performance drop is caused by adaptive overfitting of models developed over the past 20 years. Roelofs et al. (2019) defined adaptive overfitting as the overfitting caused by reusing the same test set (test reuse). Recht et al. (2019) studied this phenomenon in the context of ImageNet by measuring to what extent can the improvement on the old test set translate to improvement on the new test set (diminishing return). We analyze both effects to conclude the presence of adaptive overfitting.

heading: 5.1.1 Diminishing Return
text: Following Recht et al. (2019), we measure the diminishing return on the CoNLL++ test set. Diminishing return measures if the improvement on CoNLL-2003 test set, gained by the continuous effort of developing NER taggers over 20 years, translates to smaller (diminishing) improvement on CoNLL++.
We fit a line to the data points in Figure 2, and then calculate its slope. A slope greater than 1
indicates that every unit of improvement on the CoNLL-2003 test set by the development of models translates to more than one unit of improvement on CoNLL++, i.e. there is no diminishing return. We measure the slope to be 2.729 > 1, indicating that we have not found any diminishing return on CoNLL++, and therefore no adaptive overfitting caused by the model development over the past two decades.

heading: 5.1.2 Test Reuse
text: If the models are overfitting to the CoNLL-2003 test set due to test reuse, we should see not only a performance degradation on CoNLL++, but also a performance degradation on a test set taken from the same distribution as the CoNLL-2003 test set.
To obtain a new test set taken from the same distribution as the CoNLL-2003 test set, we resampled a new train/dev/test split from the CoNLL-2003 dataset, which we call CoNLL-2003’. Each split contains the same number of articles as its corresponding split in the CoNLL-2003 dataset. The “new” test set is thus certain to come from the same distribution as the original CoNLL-2003 test. We train and evaluate models on CoNLL-2003’ with the same experimental setup as in Section 3, and report the results in Table 3.
We only observe SciBERT and XLMRoBERTaLarge models performing slightly worse on the CoNLL-2003’ test set, while all other models appear to perform better. Most models suffering
from performance degradation on the CoNLL++ also perform better on the CoNLL-2003’ test set. This provides evidence that individual models are not overfitting to the CoNLL-2003 test set.
Based on our results above, the performance degradation on the CoNLL++ is likely not caused by overfitting on CoNLL-2003. Rather, it is more likely caused by temporal drift, which we discuss in the next section.

heading: 5.2 Temporal Drift
text: Temporal drift refers to the performance degradation of a model on the downstream task caused by the temporal difference between the train and test data. Prior work has shown that the performance on NER is affected by temporal drift. For example, Rijhwani and Preotiuc-Pietro (2020) showed that the performance of GloVe and Flair embeddings on NER degrades when the test data is more temporally distant from the train data of the downstream task. Agarwal and Nenkova (2022) also reported the same observation on GloVe embeddings.
In this section, we use the same term “temporal drift” but refer to the deterioration of generalization of models caused by the temporal difference between the pre-training corpus of their word embeddings and the test data of the downstream task. We hypothesize that generalization is largely affected by such temporal drift. We conduct experiments on Flair and ELMo, as well as on RoBERTa.

heading: 5.2.1 Temporal Drift in Flair and ELMo
text: We first investigate if bringing the pre-training corpora of Flair and ELMo closer to the test set can improve their generalizability. We notice that both embeddings were trained on 1B Benchmark (Chelba et al., 2014). This corpus was collected from WMT11 (Callison-Burch et al., 2011) English monolingual data, which is largely comprised of news data between 2007-2011. We hypothesize that pre-training these embeddings on a more recent corpus, e.g. REALNEWS corpus (Zellers et al., 2019) which contains news articles from 2016-2019, will improve their generalizability.
To control the experiment, we randomly sample 1 billion tokens of data from REALNEWS. We train Flair embeddings following the same procedure detailed in Akbik et al. (2018) to obtain both the forward and backward embeddings. Our embeddings achieve character level perplexity on the test set of 2.45 for the forward embeddings and 2.46 for the backward embeddings, comparable to 2.42
reported in Akbik et al. (2018). Similarly, we train ELMo embeddings following Peters et al. (2018), which achieves a perplexity of 40.07 on the test set, comparable to 39.7 reported. We use the same training scripts and hyperparameters as our experiments in Section 4.2 for Flair, Pooled Flair and ELMo. The newly trained models are dubbed as FlairRN, Pooled FlairRN and ELMoRN.
It is clearly shown in Table 4 that having the training corpus for Flair and ELMo embeddings temporally closer to the CoNLL++ test set improves generalization. Notably, the performance gap for ELMo is reduced to -1.43%, better than that of BERTLarge (-2.01%). The improvements in generalization are attributed to the performance drops on the CoNLL-2003 test set and improvements on CoNLL++.
This provides evidence that the generalizability of the LSTM-based contextualized word embeddings is affected by temporal drift. However, even temporally closer data, these models still suffer from performance drops. This suggests that other ingredients, such as model architecture (§ 4.2), are still needed for a good generalization.

heading: 5.2.2 Temporal Drift in RoBERTa
text: Because pre-training a transformer model from scratch is expensive, we continue pre-training from the RoBERTaBase checkpoint, leveraging the findings from Gururangan et al. (2020) that models learn to adapt to the distribution of the new corpora with continued pre-training.
We use the WMT20 English dataset (Barrault et al., 2019), consisting of English news data from 2007 to 2021. To avoid temporal overlap, we only use data from 2007 to 2019 as the pre-training corpora. The data are divided by year, and we preprocess the data such that the number of tokens per
year is the same. We train the RoBERTaBase model for 3 epochs with the masked language modeling (MLM) objective. Checkpoints from each year are then fine-tuned on the CoNLL-2003 dataset, with the same experimental setup described in Section 3. We evaluate the models on the CoNLL-2003 test set and CoNLL++, and plot the results in Figure 5. Detailed performances are reported in Table 9 in the Appendix (§ C.4).
The results show a clear trend of performance degradation when the pre-training corpora is temporally distant from the test set. When the pre-training corpus is more recent, it becomes temporally closer to CoNLL++, leading to better CoNLL++ performance, and hence better generalization. In Figure 5, the ∆F1 shows an upward trend with a correlation coefficients of 0.55, indicating a moderate positive correlation between generalization and the year of the pre-training corpora. This suggests that generalization is affected by the effect of temporal drift. This explains the better generalizability of models such as LUKELarge and T53B, pre-trained on temporally closer data to the CoNLL++ test set, showing that temporal drift is the main driving factor for the different levels of generalization.

heading: 6 Related Work
text: How well pre-trained LMs adapt to data from future time periods has undergone extensive study. Temporal degradation has been found to be a challenge for many tasks, including language modeling (Lazaridou et al., 2021), NER (Augenstein
et al., 2017; Agarwal and Nenkova, 2022; Rijhwani and Preotiuc-Pietro, 2020; Ushio et al., 2022), QA (Dhingra et al., 2022), entity linking (Zaporojets et al., 2022), and others (Luu et al., 2022; Amba Hombaiah et al., 2021). All of this work has found that the performance of LMs degrades as the temporal distance between the training data and the test data increases, sometimes called “temporal misalignment” (Luu et al., 2022). In contrast to the prior work, we study performance deterioration on a dataset that has been heavily used to develop NER models over a period of 20 years, and conduct extensive experiments that aim to disentangle the effects of aging training sets from those due to heavy test reuse.
Most closely related to our work is Agarwal and Nenkova (2022), who analyzed a recently created Twitter NER dataset (Rijhwani and Preotiuc-Pietro, 2020) over the period 2014-2019, and found no performance deterioration when using RoBERTabased representations. We build on this line of work by carefully measuring performance deterioration of models trained on the CoNLL-2003 dataset when evaluated on modern data. We analyze which factors are necessary for an NER model trained on a 20-year-old dataset to generalize well to modern data. Furthermore, the large 20-year gap helps us focus on not only temporal deterioration, but also if the extensive test reuse leads to adaptive overfitting. We present evidence in support of the hypothesis that most performance degradation is due to temporal drift and not adaptive overfitting.
Prior work has attempted to mitigate temporal degradation, mostly through continuously updating LMs with new data (Jang et al., 2022; Jin et al., 2022; Loureiro et al., 2022). Luu et al. (2022) explored this idea but found that temporal adaptation is not as effective as fine-tuning on the data from whose time period the dataset is drawn. In addition, catastrophic forgetting (Robins, 1995) can also be a problem when updating the LMs. Jin et al. (2022) found that applying knowledge distillation (Hinton et al., 2015) based approaches to continual learning can mitigate catastrophic forgetting, while improving the temporal generalization of LMs. Dhingra et al. (2022) proposed to train the LMs with an additional temporal objective by conditioning on the year of data, and found that this effectively mitigated catastrophic forgetting. Jang et al. (2022) created a lifelong benchmark for continuous training and evaluating LMs.

heading: 7 Conclusion and Future Directons
text: In this paper, we evaluate the generalization of NER models using CoNLL++, a CoNLL-style annotated NER test dataset with data from 2020. We conduct experiments on more than 20 models and find that models exhibit different generalizability.
Surprisingly, we find that generalizability is not affected by adaptive overfitting, but rather by temporal drift. To achieve better generalization, we need the combination of four factors: a modern transformer-based architecture, a large number of parameters, a large amount of fine-tuning data and a temporally closer pre-training corpus to the test set. We find that our progress on developping NER taggers is largely successful, showing not only good performance on individual test set but also good generalization on new data. This allows CoNLL2003 taggers to still work in 2023.
Future research can focus on ways to mitigate temporal drift. Investigation on attributes of pretraining or fine-tuning corpora that causes temporal drift, such as change of entities mentioned, different usage of language, etc., can also shed light on the more specific impacts from temporal drift, thereby inspiring new and better ways to mitigate it.
We hope that our work provides insights on factors affecting generalization and how to mitigate the negative impact, and calls for more research on this everlasting problem of generalization in the NLP community.

heading: 8 Limitations
text: Our analysis on temporal drift (§ 5.2) was limited by the fact that the developer of many models in our study did not release the exact time period of the pre-training corpora used. Additionally, models such as BERT and RoBERTa were pre-trained on corpora that could be potentially be temporally close to the CoNLL++ test set.
In the section on test reuse (§ 5.1.2), due to a limited compute budget, we were only able to conduct this experiment on a single new train/dev/test split, so it is possible that the new split happens to be easier than the mean of the distribution. However, our experiments still provide additional evidence models are not overfitting the original CoNLL-2003 test set.
It is worth noting that when using older models trained on the CoNLL-2003 dataset, one additional reason for the performance degradation, especially
in real-world deployment, is that the data used to evaluate the models can be out-of-domain. In our experiments, we attemped to control the domain of the test data on which the models were evaluated to assess other factors for performance degradation. However, we acknowledge that in reality, model performance can be affected by factors such as the emerging text types (e.g. Twitter did not exist when CoNLL-2003 NER task was created), which leads to changes in domain, and therefore affects the generalizability of the models.
We acknowledge that having CoNLL++ will not resolve the problem of generalization to modern data. As new data keep emerging, there will always be the question of how well NER models generalize to that new data. We hope that our paper will encourage researchers in the NLP community to continuously annotate new test set to study this problem, so that we ensure the robustness and generalizability of models.

heading: Acknowledgements
text: We would like to thank Fan Bai, Yang Chen, Chao Jiang, Junmo Kang and Mounica Maddela for providing feedback on earlier drafts of this paper, We would also like to appreciate the comments from the anonymous reviewers on how to improve the paper. This material is based upon work supported by the NSF (IIS-2052498) and IARPA via the BETTER program (2019-19051600004). The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense, IARPA or the U.S. Government.

heading: A Tabular Data in CoNLL-2003
text: We found a significant amount of documents in the CoNLL-2003 test set that list the outcomes of various sports events, which contributes to the larger proportion of named entities in Table 1. These documents appear as though they may have been intended for display on news tickers.9 We present an example below.
SOCCER SHOWCASE-BETTING ON REAL MADRID V BARCELONA . MADRID 1996-12-06 William Hill betting on Saturday ’s Spanish first division match between Real Madrid and Barcelona : To win : 6-5 Real Madrid ; 7-4 Barcelona Draw : 9-4 Correct score : Real Madrid to win Barcelona to win 1-0 13-2 1-0 15-2 2-0 9-1 2-0 12-1 2-1 8-1 2-1 10-1 3-0 20-1 3-0 28-1 3-1 16-1 3-1 22-1 3-2 25-1 3-2 25-1 4-0 50-1 4-0 100-1 4-1 40-1 4-1 80-1 4-2 50-1 4-2 80-1 Draw : 0-0 8-1 1-1 11-2 2-2 14-1 3-3 50-1 Double result : half-time full-time 5-2 Real Madrid Real Madrid 14-1 Real Madrid Draw 28-1 Real Madrid Barcelona 5-1 Draw Real Madrid 4-1 Draw Draw 11-2 Draw Barcelona 25-1 Barcelona Real Madrid 14-1 Barcelona Draw 4-1 Barcelona Barcelona First goalscorer of match : Real Madrid Barcelona 9-2 Davor Suker 9-2 Ronaldo
9https://en.wikipedia.org/wiki/News_ticker
5-1 Pedrag Mijatovic 7-1 Luis Figo 7-1 Raul Gonzalez 7-1 Juan Pizzi 12-1 Fernando Redondo 9-1 Giovanni 14-1 Victor Sanchez 12-1 Guillermo Amor 16-1 Jose Amavisca 14-1 Roger Garcia 16-1 Manolo Sanchis 14-1 Gheorghe Popescu 16-1 Roberto Carlos 16-1 JosepGuardiola 20-1 Fernando Hierro 20-1 Ivan de laPena 20-1 Luis Milla 25-1 Luis Enrique 33-1 Fernando Sanz 25-1 AbelardoFernandez 40-1 Carlos Secretario 28-1 Sergi Barjuan 40-1 Rafael Alkorta 33-1 Albert Ferrer 40-1 Chendo Porlan 33-1 Miguel Nadal 40-1 Laurent Blanc
Being one of the 231 articles (0.43%) in the CoNLL-2003 test set, this article contains 59 (1.04%) named entities, including 23 ORG (1.38%), 34 PER (2.10%), 1 LOC and 1 MISC. We counted that there are in total 71 (30.7%) such files which contribute to 872 ORG (52.5%), 889 PER (55.0%), 657 LOC (39.4%) and 159 MISC (22.6%).
Additionally, as each line is considered to be a sentence in CoNLL-2003 dataset (separated by an empty line in the original format), and as items by spaces are considered to be tokens, this also demonstrates why the average token per sentence is much lower in CoNLL-2003 than in CoNLL++. The tabular data contains much shorter sentences in plethora, which significantly lowers the average token per sentence.

heading: B Hyperparameter Search
text: In this section, we include the details on how we conducted the hyperparameter search for the transformer-based models. We trained most models with different sets of hyperparameters for 10 epochs and save the checkpoints that achieved the highest dev F1 score. For each model, we compare performance on the dev set of checkpoints trained with different hyperparameters and select the set of hyperparameters with the best performance.
We tuned the learning rate and batch sizes for all models. If the instructions on how to tune the hyperparameters for a model are stated in its publication, we followed the instructions as closely as possible. Otherwise, we would tune the model using a default set of hyperparameters, where learning_rate = {1e-5, 2e-5, 3e-5, 5e-5} and batch_size = {8, 16, 32}. Here we only list models for which we did not use the default set of hyperparameters.
• ALBERT:
– learning_rate = {1e-5, 2e-5, 3e-5, 5e-5} – batch_size = {16, 32, 48, 128}
• GigaBERT:
– learning_rate = {1e-5, 2e-5, 5e-5, 1e-4} – batch_size = {4, 8, 16, 32}
• Longformer:
– learning_rate = {1e-5, 2e-5, 3e-5, 5e-5} – batch_size = {16, 32} – total_num_epoch = 15
• news_roberta_base:
– learning_rate = {1e-5, 2e-5, 3e-5} – batch_size = {16, 32}
• XLM-RoBERTa:
– learning_rate = {1e-5, 2e-5, 3e-5, 5e-5} – batch_size = {16, 32}
• T5:
– learning_rate = {2e-5, 3e-5, 5e-5, 1e-4} – batch_size = {4, 8}

heading: C Detailed Results
text: In this section, we include all the performance statistics.
C.1 CoNLL-2003 vs CoNLL++ Table 5 shows the performance statistics of all models on the CoNLL++ and CoNLL-2003 test sets.
C.2 Model Size Table 6 includes the results from Section 4.1, showing the performance statistics of BERT-based, ALBERT-based, RoBERTa-based, and T5-based models with various sizes on the CoNLL-2003 and CoNLL++ test set. One side note is that our results also confirms the previous findings that the performance on a downstream task has a positive correlation with the model size.
C.3 Number of Fine-Tuning Examples Table 7 and Table 8 show the results from Section 4.3 of the RoBERTa-based and Flair-based models on the two test sets respectively when varying the number of examples fine-tuned on.
C.4 Temporal Drift Table 9 show the results from Section 5.2. The “Year" column indicates the time period from which the data used for continued pre-training on RoBERTaBase was used.
C.5 Test Reuse Table 10 shows the results from Section 5.1.2.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract at the beginning of the paper, and introduction in Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Table 2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our task is on named entity recognition, for which having actual names in the dataset is more desirable
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1
C 3 Did you run computational experiments? Section 4 and 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.1 does discuss the number of parameters in some of the models used. However, number of parameters in other models, total computational budget and computing infrastructure are largely irrelevant to our paper
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Experimental setup in Section 3, and discussion on hyperparameter in Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3, 4 and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Table 2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? The annotations were completed by two authors of the paper. No additional annotators were hired.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future research can focus on ways to mitigate temporal drift. Investigation on attributes of pretraining or fine-tuning corpora that causes temporal drift, such as change of entities mentioned, different usage of language, etc. , can also shed light on the more specific impacts from temporal drift, thereby inspiring new and better ways to mitigate it. We hope that our work provides insights on factors affecting generalization and how to mitigate the negative impact, and calls for more research on this everlasting problem of generalization in the NLP community