Introduction
Named entity recognition and classification (NERC, short NER), the task of recognising and assigning a class to mentions of proper names (named entities, NEs) in text, has attracted many years of research BIBREF0 , BIBREF1 , analyses BIBREF2 , starting from the first MUC challenge in 1995 BIBREF3 . Recognising entities is key to many applications, including text summarisation BIBREF4 , search BIBREF5 , the semantic web BIBREF6 , topic modelling BIBREF7 , and machine translation BIBREF8 , BIBREF9 .
As NER is being applied to increasingly diverse and challenging text genres BIBREF10 , BIBREF11 , BIBREF12 , this has lead to a noisier, sparser feature space, which in turn requires regularisation BIBREF13 and the avoidance of overfitting. This has been the case even for large corpora all of the same genre and with the same entity classification scheme, such as ACE BIBREF14 . Recall, in particular, has been a persistent problem, as named entities often seem to have unusual surface forms, e.g. unusual character sequences for the given language (e.g. Szeged in an English-language document) or words that individually are typically not NEs, unless they are combined together (e.g. the White House).
Indeed, the move from ACE and MUC to broader kinds of corpora has presented existing NER systems and resources with a great deal of difficulty BIBREF15 , which some researchers have tried to address through domain adaptation, specifically with entity recognition in mind BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 . However, more recent performance comparisons of NER methods over different corpora showed that older tools tend to simply fail to adapt, even when given a fair amount of in-domain data and resources BIBREF21 , BIBREF11 . Simultaneously, the value of NER in non-newswire data BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 has rocketed: for example, social media now provides us with a sample of all human discourse, unmolested by editors, publishing guidelines and the like, and all in digital format – leading to, for example, whole new fields of research opening in computational social science BIBREF26 , BIBREF27 , BIBREF28 .
The prevailing assumption has been that this lower NER performance is due to domain differences arising from using newswire (NW) as training data, as well as from the irregular, noisy nature of new media (e.g. BIBREF21 ). Existing studies BIBREF11 further suggest that named entity diversity, discrepancy between named entities in the training set and the test set (entity drift over time in particular), and diverse context, are the likely reasons behind the significantly lower NER performance on social media corpora, as compared to newswire.
No prior studies, however, have investigated these hypotheses quantitatively. For example, it is not yet established whether this performance drop is really due to a higher proportion of unseen NEs in the social media, or is it instead due to NEs being situated in different kinds of linguistic context.
Accordingly, the contributions of this paper lie in investigating the following open research questions:
In particular, the paper carries out a comparative analyses of the performance of several different approaches to statistical NER over multiple text genres, with varying NE and lexical diversity. In line with prior analyses of NER performance BIBREF2 , BIBREF11 , we carry out corpus analysis and introduce briefly the NER methods used for experimentation. Unlike prior efforts, however, our main objectives are to uncover the impact of NE diversity and context diversity on performance (measured primarily by F1 score), and also to study the relationship between OOV NEs and features and F1. See Section "Experiments" for details.
To ensure representativeness and comprehensiveness, our experimental findings are based on key benchmark NER corpora spanning multiple genres, time periods, and corpus annotation methodologies and guidelines. As detailed in Section "Datasets" , the corpora studied are OntoNotes BIBREF29 , ACE BIBREF30 , MUC 7 BIBREF31 , the Ritter NER corpus BIBREF21 , the MSM 2013 corpus BIBREF32 , and the UMBC Twitter corpus BIBREF33 . To eliminate potential bias from the choice of statistical NER approach, experiments are carried out with three differently-principled NER approaches, namely Stanford NER BIBREF34 , SENNA BIBREF35 and CRFSuite BIBREF36 (see Section "NER Models and Features" for details).
Datasets
Since the goal of this study is to compare NER performance on corpora from diverse domains and genres, seven benchmark NER corpora are included, spanning newswire, broadcast conversation, Web content, and social media (see Table 1 for details). These datasets were chosen such that they have been annotated with the same or very similar entity classes, in particular, names of people, locations, and organisations. Thus corpora including only domain-specific entities (e.g. biomedical corpora) were excluded. The choice of corpora was also motivated by their chronological age; we wanted to ensure a good temporal spread, in order to study possible effects of entity drift over time.
A note is required about terminology. This paper refers to text genre and also text domain. These are two dimensions by which a document or corpus can be described. Genre here accounts the general characteristics of the text, measurable with things like register, tone, reading ease, sentence length, vocabulary and so on. Domain describes the dominant subject matter of text, which might give specialised vocabulary or specific, unusal word senses. For example, “broadcast news" is a genre, describing the manner of use of language, whereas “financial text" or “popular culture" are domains, describing the topic. One notable exception to this terminology is social media, which tends to be a blend of myriad domains and genres, with huge variation in both these dimensions BIBREF38 , BIBREF39 ; for simplicity, we also refer to this as a genre here.
In chronological order, the first corpus included here is MUC 7, which is the last of the MUC challenges BIBREF31 . This is an important corpus, since the Message Understanding Conference (MUC) was the first one to introduce the NER task in 1995 BIBREF3 , with focus on recognising persons, locations and organisations in newswire text.
A subsequent evaluation campaign was the CoNLL 2003 NER shared task BIBREF40 , which created gold standard data for newswire in Spanish, Dutch, English and German. The corpus of this evaluation effort is now one of the most popular gold standards for NER, with new NER approaches and methods often reporting performance on that.
Later evaluation campaigns began addressing NER for genres other than newswire, specifically ACE BIBREF30 and OntoNotes BIBREF29 . Both of those contain subcorpora in several genres, namely newswire, broadcast news, broadcast conversation, weblogs, and conversational telephone speech. ACE, in addition, contains a subcorpus with usenet newsgroups. Like CoNLL 2003, the OntoNotes corpus is also a popular benchmark dataset for NER. The languages covered are English, Arabic and Chinese. A further difference between the ACE and OntoNotes corpora on one hand, and CoNLL and MUC on the other, is that they contain annotations not only for NER, but also for other tasks such as coreference resolution, relation and event extraction and word sense disambiguation. In this paper, however, we restrict ourselves purely to the English NER annotations, for consistency across datasets. The ACE corpus contains HEAD as well as EXTENT annotations for NE spans. For our experiments we use the EXTENT tags.
With the emergence of social media, studying NER performance on this genre gained momentum. So far, there have been no big evaluation efforts, such as ACE and OntoNotes, resulting in substantial amounts of gold standard data. Instead, benchmark corpora were created as part of smaller challenges or individual projects. The first such corpus is the UMBC corpus for Twitter NER BIBREF33 , where researchers used crowdsourcing to obtain annotations for persons, locations and organisations. A further Twitter NER corpus was created by BIBREF21 , which, in contrast to other corpora, contains more fine-grained classes defined by the Freebase schema BIBREF41 . Next, the Making Sense of Microposts initiative BIBREF32 (MSM) provides single annotated data for named entity recognition on Twitter for persons, locations, organisations and miscellaneous. MSM initiatives from 2014 onwards in addition feature a named entity linking task, but since we only focus on NER here, we use the 2013 corpus.
These corpora are diverse not only in terms of genres and time periods covered, but also in terms of NE classes and their definitions. In particular, the ACE and OntoNotes corpora try to model entity metonymy by introducing facilities and geo-political entities (GPEs). Since the rest of the benchmark datasets do not make this distinction, metonymous entities are mapped to a more common entity class (see below).
In order to ensure consistency across corpora, only Person (PER), Location (LOC) and Organisation (ORG) are used in our experiments, and other NE classes are mapped to O (no NE). For the Ritter corpus, the 10 entity classes are collapsed to three as in BIBREF21 . For the ACE and OntoNotes corpora, the following mapping is used: PERSON $\rightarrow $ PER; LOCATION, FACILITY, GPE $\rightarrow $ LOC; ORGANIZATION $\rightarrow $ ORG; all other classes $\rightarrow $ O.
Tokens are annotated with BIO sequence tags, indicating that they are the beginning (B) or inside (I) of NE mentions, or outside of NE mentions (O). For the Ritter and ACE 2005 corpora, separate training and test corpora are not publicly available, so we randomly sample 1/3 for testing and use the rest for training. The resulting training and testing data sizes measured in number of NEs are listed in Table 2 . Separate models are then trained on the training parts of each corpus and evaluated on the development (if available) and test parts of the same corpus. If development parts are available, as they are for CoNLL (CoNLL Test A) and MUC (MUC 7 Dev), they are not merged with the training corpora for testing, as it was permitted to do in the context of those evaluation challenges.
[t]
P, R and F1 of NERC with different models evaluated on different testing corpora, trained on corpora normalised by size
Table 1 shows which genres the different corpora belong to, the number of NEs and the proportions of NE classes per corpus. Sizes of NER corpora have increased over time, from MUC to OntoNotes.
Further, the class distribution varies between corpora: while the CoNLL corpus is very balanced and contains about equal numbers of PER, LOC and ORG NEs, other corpora are not. The least balanced corpus is the MSM 2013 Test corpus, which contains 98 LOC NEs, but 1110 PER NEs. This makes it difficult to compare NER performance here, since performance partly depends on training data size. Since comparing NER performance as such is not the goal of this paper, we will illustrate the impact of training data size by using learning curves in the next section; illustrate NERC performance on trained corpora normalised by size in Table UID9 ; and then only use the original training data size for subsequent experiments.
In order to compare corpus diversity across genres, we measure NE and token/type diversity (following e.g. BIBREF2 ). Note that types are the unique tokens, so the ratio can be understood as ratio of total tokens to unique ones. Table 4 shows the ratios between the number of NEs and the number of unique NEs per corpus, while Table 5 reports the token/type ratios. The lower those ratios are, the more diverse a corpus is. While token/type ratios also include tokens which are NEs, they are a good measure of broader linguistic diversity.
Aside from these metrics, there are other factors which contribute to corpus diversity, including how big a corpus is and how well sampled it is, e.g. if a corpus is only about one story, it should not be surprising to see a high token/type ratio. Therefore, by experimenting on multiple corpora, from different genres and created through different methodologies, we aim to encompass these other aspects of corpus diversity.
Since the original NE and token/type ratios do not account for corpus size, Tables 5 and 4 present also the normalised ratios. For those, a number of tokens equivalent to those in the corpus, e.g. 7037 for UMBC (Table 5 ) or, respectively, a number of NEs equivalent to those in the corpus (506 for UMBC) are selected (Table 4 ).
An easy choice of sampling method would be to sample tokens and NEs randomly. However, this would not reflect the composition of corpora appropriately. Corpora consist of several documents, tweets or blog entries, which are likely to repeat the words or NEs since they are about one story. The difference between bigger and smaller corpora is then that bigger corpora consist of more of those documents, tweets, blog entries, interviews, etc. Therefore, when we downsample, we take the first $n$ tokens for the token/type ratios or the first $n$ NEs for the NEs/Unique NEs ratios.
Looking at the normalised diversity metrics, the lowest NE/Unique NE ratios $<= 1.5$ (in bold, Table 4 ) are observed on the Twitter and CoNLL Test corpora. Seeing this for Twitter is not surprising since one would expect noise in social media text (e.g. spelling variations or mistakes) to also have an impact on how often the same NEs are seen. Observing this in the latter, though, is less intuitive and suggests that the CoNLL corpora are well balanced in terms of stories. Low NE/Unique ratios ( $<= 1.7$ ) can also be observed for ACE WL, ACE UN and OntoNotes TC. Similar to social media text, content from weblogs, usenet dicussions and telephone conversations also contains a larger amount of noise compared to the traditionally-studied newswire genre, so this is not a surprising result. Corpora bearing high NE/Unique NE ratios ( $> 2.5$ ) are ACE CTS, OntoNotes MZ and OntoNotes BN. These results are also not surprising. The telephone conversations in ACE CTS are all about the same story, and newswire and broadcast news tend to contain longer stories (reducing variety in any fixed-size set) and are more regular due to editing.
The token/type ratios reflect similar trends (Table 5 ). Low token/type ratios $<= 2.8$ (in bold) are observed for the Twitter corpora (Ritter and UMBC), as well as for the CoNLL Test corpus. Token/type ratios are also low ( $<= 3.2$ ) for CoNLL Train and ACE WL. Interestingly, ACE UN and MSM Train and Test do not have low token/type ratios although they have low NE/Unique ratios. That is, many diverse persons, locations and organisations are mentioned in those corpora, but similar context vocabulary is used. Token/type ratios are high ( $>= 4.4$ ) for MUC7 Dev, ACE BC, ACE CTS, ACE UN and OntoNotes TC. Telephone conversations (TC) having high token/type ratios can be attributed to the high amount filler words (e.g. “uh”, “you know”). NE corpora are generally expected to have regular language use – for ACE, at least, in this instance.
Furthermore, it is worth pointing out that, especially for the larger corpora (e.g. OntoNotes NW), size normalisation makes a big difference. The normalised NE/Unique NE ratios drop by almost a half compared to the un-normalised ratios, and normalised Token/Type ratios drop by up to 85%. This strengthens our argument for size normalisation and also poses the question of low NERC performance for diverse genres being mostly due to the lack of large training corpora. This is examined in Section "RQ2: NER performance in Different Genres" .
Lastly, Table 6 reports tag density (percentage of tokens tagged as part of a NE), which is another useful metric of corpus diversity that can be interpreted as the information density of a corpus. What can be observed here is that the NW corpora have the highest tag density and generally tend to have higher tag density than corpora of other genres; that is, newswire bears a lot of entities. Corpora with especially low tag density $<= 0.06$ (in bold) are the TC corpora, Ritter, OntoNotes WB, ACE UN, ACE BN and ACE BC. As already mentioned, conversational corpora, to which ACE BC also belong, tend to have many filler words, thus it is not surprising that they have a low tag density. There are only minor differences between the tag density and the normalised tag density, since corpus size as such does not impact tag density.
NER Models and Features
To avoid system-specific bias in our experiments, three widely-used supervised statistical approaches to NER are included: Stanford NER, SENNA, and CRFSuite. These systems each have contrasting notable attributes.
Stanford NER BIBREF34 is the most popular of the three, deployed widely in both research and commerce. The system has been developed in terms of both generalising the underlying technology and also specific additions for certain languages. The majority of openly-available additions to Stanford NER, in terms of models, gazetteers, prefix/suffix handling and so on, have been created for newswire-style text. Named entity recognition and classification is modelled as a sequence labelling task with first-order conditional random fields (CRFs) BIBREF43 .
SENNA BIBREF35 is a more recent system for named entity extraction and other NLP tasks. Using word representations and deep learning with deep convolutional neural networks, the general principle for SENNA is to avoid task-specific engineering while also doing well on multiple benchmarks. The approach taken to fit these desiderata is to use representations induced from large unlabelled datasets, including LM2 (introduced in the paper itself) and Brown clusters BIBREF44 , BIBREF45 . The outcome is a flexible system that is readily adaptable, given training data. Although the system is more flexible in general, it relies on learning language models from unlabelled data, which might take a long time to gather and retrain. For the setup in BIBREF35 language models are trained for seven weeks on the English Wikipedia, Reuters RCV1 BIBREF46 and parts of the Wall Street Journal, and results are reported over the CoNLL 2003 NER dataset. Reuters RCV1 is chosen as unlabelled data because the English CoNLL 2003 corpus is created from the Reuters RCV1 corpus. For this paper, we use the original language models distributed with SENNA and evaluate SENNA with the DeepNL framework BIBREF47 . As such, it is to some degree also biased towards the CoNLL 2003 benchmark data.
Finally, we use the classical NER approach from CRFsuite BIBREF36 , which also uses first-order CRFs. This frames NER as a structured sequence prediction task, using features derived directly from the training text. Unlike the other systems, no external knowledge (e.g. gazetteers and unsupervised representations) are used. This provides a strong basic supervised system, and – unlike Stanford NER and SENNA – has not been tuned for any particular domain, giving potential to reveal more challenging domains without any intrinsic bias.
We use the feature extractors natively distributed with the NER frameworks. For Stanford NER we use the feature set “chris2009” without distributional similarity, which has been tuned for the CoNLL 2003 data. This feature was tuned to handle OOV words through word shape, i.e. capitalisation of constituent characters. The goal is to reduce feature sparsity – the basic problem behind OOV named entities – by reducing the complexity of word shapes for long words, while retaining word shape resolution for shorter words. In addition, word clusters, neighbouring n-grams, label sequences and quasi-Newton minima search are included. SENNA uses word embedding features and gazetteer features; for the training configuration see https://github.com/attardi/deepnl#benchmarks. Finally, for CRFSuite, we use the provided feature extractor without POS or chunking features, which leaves unigram and bigram word features of the mention and in a window of 2 to the left and the right of the mention, character shape, prefixes and suffixes of tokens.
These systems are compared against a simple surface form memorisation tagger. The memorisation baseline picks the most frequent NE label for each token sequence as observed in the training corpus. There are two kinds of ambiguity: one is overlapping sequences, e.g. if both “New York City” and “New York” are memorised as a location. In that case the longest-matching sequence is labelled with the corresponding NE class. The second, class ambiguity, occurs when the same textual label refers to different NE classes, e.g. “Google” could either refer to the name of a company, in which case it would be labelled as ORG, or to the company's search engine, which would be labelled as O (no NE).
RQ1: NER performance with Different Approaches
[t]
P, R and F1 of NERC with different models trained on original corpora
[t]
F1 per NE type with different models trained on original corpora
Our first research question is how NERC performance differs for corpora between approaches. In order to answer this, Precision (P), Recall (R) and F1 metrics are reported on size-normalised corpora (Table UID9 ) and original corpora (Tables "RQ1: NER performance with Different Approaches" and "RQ1: NER performance with Different Approaches" ). The reason for size normalisation is to make results comparable across corpora. For size normalisation, the training corpora are downsampled to include the same number of NEs as the smallest corpus, UMBC. For that, sentences are selected from the beginning of the train part of the corpora so that they include the same number of NEs as UMBC. Other ways of downsampling the corpora would be to select the first $n$ sentences or the first $n$ tokens, where $n$ is the number of sentences in the smallest corpus. The reason that the number of NEs, which represent the number of positive training examples, is chosen for downsampling the corpora is that the number of positive training examples have a much bigger impact on learning than the number of negative training examples. For instance, BIBREF48 , among others, study topic classification performance for small corpora and sample from the Reuters corpus. They find that adding more negative training data gives little to no improvement, whereas adding positive examples drastically improves performance.
Table UID9 shows results with size normalised precision (P), recall (R), and F1-Score (F1). The five lowest P, R and F1 values per method (CRFSuite, Stanford NER, SENNA) are in bold to highlight underperformers. Results for all corpora are summed with macro average.
Comparing the different methods, the highest F1 results are achieved with SENNA, followed by Stanford NER and CRFSuite. SENNA has a balanced P and R, which can be explained by the use of word embeddings as features, which help with the unseen word problem. For Stanford NER as well as CRFSuite, which do not make use of embeddings, recall is about half of precision. These findings are in line with other work reporting the usefulness of word embeddings and deep learning for a variety of NLP tasks and domains BIBREF49 , BIBREF50 , BIBREF51 . With respect to individual corpora, the ones where SENNA outperforms other methods by a large margin ( $>=$ 13 points in F1) are CoNLL Test A, ACE CTS and OntoNotes TC. The first success can be attributed to being from the same the domain SENNA was originally tuned for. The second is more unexpected and could be due to those corpora containing a disproportional amount of PER and LOC NEs (which are easier to tag correctly) compared to ORG NEs, as can be seen in Table "RQ1: NER performance with Different Approaches" , where F1 of NERC methods is reported on the original training data.
Our analysis of CRFSuite here is that it is less tuned for NW corpora and might therefore have a more balanced performance across genres does not hold. Results with CRFSuite for every corpus are worse than the results for that corpus with Stanford NER, which is also CRF-based.
To summarise, our findings are:
[noitemsep]
F1 is highest with SENNA, followed by Stanford NER and CRFSuite
SENNA outperforms other methods by a large margin (e.g. $>=$ 13 points in F1) for CoNLL Test A, ACE CTS and OntoNotes TC
Our hypothesis that CRFSuite is less tuned for NW corpora and will therefore have a more balanced performance across genres does not hold, as results for CRFSuite for every corpus are worse than with Stanford NER
RQ2: NER performance in Different Genres
Our second research question is whether existing NER approaches generalise well over corpora in different genres. To do this we study again Precision (P), Recall (R) and F1 metrics on size-normalised corpora (Table UID9 ), on original corpora (Tables "RQ1: NER performance with Different Approaches" and "RQ1: NER performance with Different Approaches" ), and we further test performance per genre in a separate table (Table 3 ).
F1 scores over size-normalised corpora vary widely (Table UID9 ). For example, the SENNA scores range from 9.35% F1 (ACE UN) to 71.48% (CoNLL Test A). Lowest results are consistently observed for the ACE subcorpora, UMBC, and OntoNotes BC and WB. The ACE corpora are large and so may be more prone to non-uniformities emerging during downsampling; they also have special rules for some kinds of organisation which can skew results (as described in Section UID9 ). The highest results are on the CoNLL Test A corpus, OntoNotes BN and MUC 7 Dev. This moderately supports our hypothesis that NER systems perform better on NW than on other genres, probably due to extra fitting from many researchers using them as benchmarks for tuning their approaches. Looking at the Twitter (TWI) corpora present the most challenge due to increased diversity, the trends are unstable. Although results for UMBC are among the lowest, results for MSM 2013 and Ritter are in the same range or even higher than those on NW datasets. This begs the question whether low results for Twitter corpora reported previously were due to the lack of sufficient in-genre training data.
Comparing results on normalised to non-normalised data, Twitter results are lower than those for most OntoNotes corpora and CoNLL test corpora, mostly due to low recall. Other difficult corpora having low performance are ACE UN and WEB corpora. We further explicitly examine results on size normalised corpora grouped by corpus type, shown in Table 3 . It becomes clear that, on average, newswire corpora and OntoNotes MZ are the easiest corpora and ACE UN, WEB and TWI are harder. This confirms our hypothesis that social media and Web corpora are challenging for NERC.
The CoNLL results, on the other hand, are the highest across all corpora irrespective of the NERC method. What is very interesting to see is that they are much higher than the results on the biggest training corpus, OntoNotes NW. For instance, SENNA has an F1 of 78.04 on OntoNotes, compared to an F1 of 92.39 and 86.44 for CoNLL Test A and Test B respectively. So even though OntoNotes NW is more than twice the size of CoNLL in terms of NEs (see Table 4 ), NERC performance is much higher on CoNLL. NERC performance with respect to training corpus size is represented in Figure 1 . The latter figure confirms that although there is some correlation between corpus size and F1, the variance between results on comparably sized corpora is big. This strengthens our argument that there is a need for experimental studies, such as those reported below, to find out what, apart from corpus size, impacts NERC performance.
Another set of results presented in Table "RQ1: NER performance with Different Approaches" are those of the simple NERC memorisation baseline. It can be observed that corpora with a low F1 for NERC methods, such as UMBC and ACE UN, also have a low memorisation performance. Memorisation is discussed in more depth in Section "RQ5: Out-Of-Domain NER Performance and Memorisation" .
When NERC results are compared to the corpus diversity statistics, i.e. NE/Unique NE ratios (Table 4 ), token/type ratios (Table 5 ), and tag density (Table 6 ), the strongest predictor for F1 is tag density, as can be evidenced by the R correlation values between the ratios and F1 scores with the Stanford NER system, shown in the respective tables.
There is a positive correlation between high F1 and high tag density (R of 0.57 and R of 0.62 with normalised tag density), a weak positive correlation for NE/unique ratios (R of 0.20 and R of 0.15 for normalised ratio), whereas for token/type ratios, no such clear correlation can be observed (R of 0.25 and R of -0.07 for normalised ratio).
However, tag density is also not an absolute predictor for NERC performance. While NW corpora have both high NERC performance and high tag density, this high density is not necessarily an indicator of high performance. For example, systems might not find high tag density corpora of other genres necessarily so easy.
One factor that can explain the difference in genre performance between e.g. newswire and social media is entity drift – the change in observed entity terms over time. In this case, it is evident from the differing surface forms and contexts for a given entity class. For example, the concept of “location" that NER systems try to learn might be frequently represented in English newswire from 1991 with terms like Iraq or Kuwait, but more with Atlanta, Bosnia and Kabul in the same language and genre from 1996. Informally, drift on Twitter is often characterised as both high-frequency and high-magnitude; that is, the changes are both rapid and correspond to a large amount of surface form occurrences (e.g. BIBREF12 , BIBREF52 ).
We examined the impact of drift in newswire and Twitter corpora, taking datasets based in different timeframes. The goal is to gauge how much diversity is due to new entities appearing over time. To do this, we used just the surface lexicalisations of entities as the entity representation. The overlap of surface forms was measured across different corpora of the same genre and language. We used an additional corpus based on recent data – that from the W-NUT 2015 challenge BIBREF25 . This is measured in terms of occurrences, rather than distinct surface forms, so that the magnitude of the drift is shown instead of having skew in results from the the noisy long tail. Results are given in Table 7 for newswire and Table 8 for Twitter corpora.
It is evident that the within-class commonalities in surface forms are much higher in newswire than in Twitter. That is to say, observations of entity texts in one newswire corpus are more helpful in labelling other newswire corpora, than if the same technique is used to label other twitter corpora.
This indicates that drift is lower in newswire than in tweets. Certainly, the proportion of entity mentions in most recent corpora (the rightmost-columns) are consistently low compared to entity forms available in earlier data. These reflect the raised OOV and drift rates found in previous work BIBREF12 , BIBREF53 . Another explanation is that there is higher noise in variation, and that the drift is not longitudinal, but rather general. This is partially addressed by RQ3, which we will address next, in Section "RQ3: Impact of NE Diversity" .
To summarise, our findings are:
[noitemsep]
Overall, F1 scores vary widely across corpora.
Trends can be marked in some genres. On average, newswire corpora and OntoNotes MZ are the easiest corpora and ACE UN, WEB and TWI are the hardest corpora for NER methods to reach good performance on.
Normalising corpora by size results in more noisy data such as TWI and WEB data achieving similar results to NW corpora.
Increasing the amount of available in-domain training data will likely result in improved NERC performance.
There is a strong positive correlation between high F1 and high tag density, a weak positive correlation for NE/unique ratios and no clear correlation between token/type ratios and F1
Temporal NE drift is lower in newswire than in tweets
The next section will take a closer look at the impact of seen and unseen NEs on NER performance.
RQ3: Impact of NE Diversity
Unseen NEs are those with surface forms present only in the test, but not training data, whereas seen NEs are those also encountered in the training data. As discussed previously, the ratio between those two measures is an indicator of corpus NE diversity.
Table 9 shows how the number of unseen NEs per test corpus relates to the total number of NEs per corpus. The proportion of unseen forms varies widely by corpus, ranging from 0.351 (ACE NW) to 0.931 (UMBC). As expected there is a correlation between corpus size and percentage of unseen NEs, i.e. smaller corpora such as MUC and UMBC tend to contain a larger proportion of unseen NEs than bigger corpora such as ACE NW. In addition, similar to the token/type ratios listed in Table 5 , we observe that TWI and WEB corpora have a higher proportion of unseen entities.
As can be seen from Table "RQ1: NER performance with Different Approaches" , corpora with a low percentage of unseen NEs (e.g. CoNLL Test A and OntoNotes NW) tend to have high NERC performance, whereas corpora with high percentage of unseen NEs (e.g. UMBC) tend to have low NERC performance. This suggests that systems struggle to recognise and classify unseen NEs correctly.
To check this seen/unseen performance split, next we examine NERC performance for unseen and seen NEs separately; results are given in Table 10 . The “All" column group represents an averaged performance result. What becomes clear from the macro averages is that F1 on unseen NEs is significantly lower than F1 on seen NEs for all three NERC approaches. This is mostly due to recall on unseen NEs being lower than that on seen NEs, and suggests some memorisation and poor generalisation in existing systems. In particular, Stanford NER and CRFSuite have almost 50% lower recall on unseen NEs compared to seen NEs. One outlier is ACE UN, for which the average seen F1 is 1.01 and the average unseen F1 is 1.52, though both are miniscule and the different negligible.
Of the three approaches, SENNA exhibits the narrowest F1 difference between seen and unseen NEs. In fact it performs below Stanford NER for seen NEs on many corpora. This may be because SENNA has but a few features, based on word embeddings, which reduces feature sparsity; intuitively, the simplicity of the representation is likely to help with unseen NEs, at the cost of slightly reduced performance on seen NEs through slower fitting. Although SENNA appears to be better at generalising than Stanford NER and our CRFSuite baseline, the difference between its performance on seen NEs and unseen NEs is still noticeable. This is 21.77 for SENNA (macro average), whereas it is 29.41 for CRFSuite and 35.68 for Stanford NER.
The fact that performance over unseen entities is significantly lower than on seen NEs partly explains what we observed in the previous section; i.e., that corpora with a high proportion of unseen entities, such as the ACE WL corpus, are harder to label than corpora of a similar size from other genres, such as the ACE BC corpus (e.g. systems reach F1 of $\sim $ 30 compared to $\sim $ 50; Table "RQ1: NER performance with Different Approaches" ).
However, even though performance on seen NEs is higher than on unseen, there is also a difference between seen NEs in corpora of different sizes and genres. For instance, performance on seen NEs in ACE WL is 70.86 (averaged over the three different approaches), whereas performance on seen NEs in the less-diverse ACE BC corpus is higher at 76.42; the less diverse data is, on average, easier to tag. Interestingly, average F1 on seen NEs in the Twitter corpora (MSM and Ritter) is around 80, whereas average F1 on the ACE corpora, which are of similar size, is lower, at around 70.
To summarise, our findings are:
[noitemsep]
F1 on unseen NEs is significantly lower than F1 on seen NEs for all three NERC approaches, which is mostly due to recall on unseen NEs being lower than that on seen NEs.
Performance on seen NEs is significantly and consistently higher than that of unseen NEs in different corpora, with the lower scores mostly attributable to lower recall.
However, there are still significant differences at labelling seen NEs in different corpora, which means that if NEs are seen or unseen does not account for all of the difference of F1 between corpora of different genres.
RQ4: Unseen Features, unseen NEs and NER performance
Having examined the impact of seen/unseen NEs on NERC performance in RQ3, and touched upon surface form drift in RQ2, we now turn our attention towards establishing the impact of seen features, i.e. features appearing in the test set that are observed also in the training set. While feature sparsity can help to explain low F1, it is not a good predictor of performance across methods: sparse features can be good if mixed with high-frequency ones. For instance, Stanford NER often outperforms CRFSuite (see Table "RQ1: NER performance with Different Approaches" ) despite having a lower proportion of seen features (i.e. those that occur both in test data and during training). Also, some approaches such as SENNA use a small number of features and base their features almost entirely on the NEs and not on their context.
Subsequently, we want to measure F1 for unseens and seen NEs, as in Section "RQ3: Impact of NE Diversity" , but also examine how the proportion of seen features impacts on the result. We define seen features as those observed in the test data and also the training data. In turn, unseen features are those observed in the test data but not in the training data. That is, they have not been previously encountered by the system at the time of labeling. Unseen features are different from unseen words in that they are the difference in representation, not surface form. For example, the entity “Xoxarle" may be an unseen entity not found in training data This entity could reasonably have “shape:Xxxxxxx" and “last-letter:e" as part of its feature representation. If the training data contains entities “Kenneth" and “Simone", each of this will have generated these two features respectively. Thus, these example features will not be unseen features in this case, despite coming from an unseen entity. Conversely, continuing this example, if the training data contains no feature “first-letter:X" – which applies to the unseen entity in question – then this will be an unseen feature.
We therefore measure the proportion of unseen features per unseen and seen proportion of different corpora. An analysis of this with Stanford NER is shown in Figure 2 . Each data point represents a corpus. The blue squares are data points for seen NEs and the red circles are data points for unseen NEs. The figure shows a negative correlation between F1 and percentage of unseen features, i.e. the lower the percentage of unseen features, the higher the F1. Seen and unseen performance and features separate into two groups, with only two outlier points. The figure shows that novel, previously unseen NEs have more unseen features and that systems score a lower F1 on them. This suggests that despite the presence of feature extractors for tackling unseen NEs, the features generated often do not overlap with those from seen NEs. However, one would expect individual features to give different generalisation power for other sets of entities, and for systems use these features in different ways. That is, machine learning approaches to the NER task do not seem to learn clear-cut decision boundaries based on a small set of features. This is reflected in the softness of the correlation.
Finally, the proportion of seen features is higher for seen NEs. The two outlier points are ACE UN (low F1 for seen NEs despite low percentage of unseen features) and UMBC (high F1 for seen NEs despite high percentage of unseen features). An error analysis shows that the ACE UN corpus suffers from the problem that the seen NEs are ambiguous, meaning even if they have been seen in the training corpus, a majority of the time they have been observed with a different NE label. For the UMBC corpus, the opposite is true: seen NEs are unambiguous. This kind of metonymy is a known and challenging issue in NER, and the results on these corpora highlight the impact is still has on modern systems.
For all approaches the proportion of observed features for seen NEs is bigger than the proportion of observed features for unseen NEs, as it should be. However, within the seen and unseen testing instances, there is no clear trend indicating whether having more observed features overall increases F1 performance. One trend that is observable is that the smaller the token/type ratio is (Table 5 ), the bigger the variance between the smallest and biggest $n$ for each corpus, or, in other words, the smaller the token/type ratio is, the more diverse the features.
To summarise, our findings are:
[noitemsep]
Seen NEs have more unseen features and systems score a lower F1 on them.
Outliers are due to low/high ambiguity of seen NEs.
The proportion of observed features for seen NEs is bigger than the proportion of observed features for unseen NEs
Within the seen and unseen testing instances, there is no clear trend indicating whether having more observed features overall increases F1 performance.
The smaller the token/type ratio is, the more diverse the features.
RQ5: Out-Of-Domain NER Performance and Memorisation
This section explores baseline out-of-domain NERC performance without domain adaptation; what percentage of NEs are seen if there is a difference between the the training and the testing domains; and how the difference in performance on unseen and seen NEs compares to in-domain performance.
As demonstrated by the above experiments, and in line with related work, NERC performance varies across domains while also being influenced by the size of the available in-domain training data. Prior work on transfer learning and domain adaptation (e.g. BIBREF16 ) has aimed at increasing performance in domains where only small amounts of training data are available. This is achieved by adding out-of domain data from domains where larger amounts of training data exist. For domain adaptation to be successful, however, the seed domain needs to be similar to the target domain, i.e. if there is no or very little overlap in terms of contexts of the training and testing instances, the model does not learn any additional helpful weights. As a confounding factor, Twitter and other social media generally consist of many (thousands-millions) of micro-domains, with each author BIBREF54 community BIBREF55 and even conversation BIBREF56 having its own style, which makes it hard to adapt to it as a single, monolithic genre; accordingly, adding out-of-domain NER data gives bad results in this situation BIBREF21 . And even if recognised perfectly, entities that occur just once cause problems beyond NER, e.g. in co-reference BIBREF57 .
In particular, BIBREF58 has reported improving F1 by around 6% through adaptation from the CoNLL to the ACE dataset. However, transfer learning becomes more difficult if the target domain is very noisy or, as mentioned already, too different from the seed domain. For example, BIBREF59 unsuccessfully tried to adapt the CoNLL 2003 corpus to a Twitter corpus spanning several topics. They found that hand-annotating a Twitter corpus consisting of 24,000 tokens performs better on new Twitter data than their transfer learning efforts with the CoNLL 2003 corpus.
The seed domain for the experiments here is newswire, where we use the classifier trained on the biggest NW corpus investigated in this study, i.e. OntoNotes NW. That classifier is then applied to all other corpora. The rationale is to test how suitable such a big corpus would be for improving Twitter NER, for which only small training corpora are available.
Results for out-of-domain performance are reported in Table 11 . The highest F1 performance is on the OntoNotes BC corpus, with similar results to the in-domain task. This is unsurprising as it belongs to a similar domain as the training corpus (broadcast conversation) the data was collected in the same time period, and it was annotated using the same guidelines. In contrast, out-of-domain results are much lower than in-domain results for the CoNLL corpora, even though they belong to the same genre as OntoNotes NW. Memorisation recall performance on CoNLL TestA and TestB with OntoNotes NW test suggest that this is partly due to the relatively low overlap in NEs between the two datasets. This could be attributed to the CoNLL corpus having been collected in a different time period to the OntoNotes corpus, when other entities were popular in the news; an example of drift BIBREF37 . Conversely, Stanford NER does better on these corpora than it does on other news data, e.g. ACE NW. This indicates that Stanford NER is capable of some degree of generalisation and can detect novel entity surface forms; however, recall is still lower than precision here, achieving roughly the same scores across these three (from 44.11 to 44.96), showing difficulty in picking up novel entities in novel settings.
In addition, there are differences in annotation guidelines between the two datasets. If the CoNLL annotation guidelines were more inclusive than the Ontonotes ones, then even a memorisation evaluation over the same dataset would yield this result. This is, in fact, the case: OntoNotes divides entities into more classes, not all of which can be readily mapped to PER/LOC/ORG. For example, OntoNotes includes PRODUCT, EVENT, and WORK OF ART classes, which are not represented in the CoNLL data. It also includes the NORP class, which blends nationalities, religious and political groups. This has some overlap with ORG, but also includes terms such as “muslims" and “Danes", which are too broad for the ACE-related definition of ORGANIZATION. Full details can be found in the OntoNotes 5.0 release notes and the (brief) CoNLL 2003 annotation categories. Notice how the CoNLL guidelines are much more terse, being generally non-prose, but also manage to cram in fairly comprehensive lists of sub-kinds of entities in each case. This is likely to make the CoNLL classes include a diverse range of entities, with the many suggestions acting as generative material for the annotator, and therefore providing a broader range of annotations from which to generalise from – i.e., slightly easier to tag.
The lowest F1 of 0 is “achieved" on ACE BN. An examination of that corpus reveals the NEs contained in that corpus are all lower case, whereas those in OntoNotes NW have initial capital letters.
Results on unseen NEs for the out-of-domain setting are in Table 12 . The last section's observation of NERC performance being lower for unseen NEs also generally holds true in this out-of-domain setting. The macro average over F1 for the in-domain setting is 76.74% for seen NEs vs. 53.76 for unseen NEs, whereas for the out-of-domain setting the F1 is 56.10% for seen NEs and 47.73% for unseen NEs.
Corpora with a particularly big F1 difference between seen and unseen NEs ( $<=$ 20% averaged over all NERC methods) are ACE NW, ACE BC, ACE UN, OntoNotes BN and OntoNotes MZ. For some corpora (CoNLL Test A and B, MSM and Ritter), out-of-domain F1 (macro average over all methods) of unseen NEs is better than for seen NEs. We suspect that this is due to the out-of-domain evaluation setting encouraging better generalisation, as well as the regularity in entity context observed in the fairly limited CoNLL news data – for example, this corpus contains a large proportion of cricket score reports and many cricketer names, occurring in linguistically similar contexts. Others have also noted that the CoNLL datasets are low-diversity compared to OntoNotes, in the context of named entity recognition BIBREF60 . In each of the exceptions except MSM, the difference is relatively small. We note that the MSM test corpus is one of the smallest datasets used in the evaluation, also based on a noisier genre than most others, and so regard this discrepancy as an outlier.
Corpora for which out-of-domain F1 is better than in-domain F1 for at least one of the NERC methods are: MUC7 Test, ACE WL, ACE UN, OntoNotes WB, OntoNotes TC and UMBC. Most of those corpora are small, with combined training and testing bearing fewer than 1,000 NEs (MUC7 Test, ACE UN, UMBC). In such cases, it appears beneficial to have a larger amount of training data, even if it is from a different domain and/or time period. The remaining 3 corpora contain weblogs (ACE WL, ACE WB) and online Usenet discussions (ACE UN). Those three are diverse corpora, as can be observed by the relatively low NEs/Unique NEs ratios (Table 4 ). However, NE/Unique NEs ratios are not an absolute predictor for better out-of-domain than in-domain performance: there are corpora with lower NEs/Unique NEs ratios than ACE WB which have better in-domain than out-of-domain performance. As for the other Twitter corpora, MSM 2013 and Ritter, performance is very low, especially for the memorisation system. This reflects that, as well as surface form variation, the context or other information represented by features shifts significantly more in Twitter than across different samples of newswire, and that the generalisations that can be drawn from newswire by modern NER systems are not sufficient to give any useful performance in this natural, unconstrained kind of text.
In fact, it is interesting to see that the memorisation baseline is so effective with many genres, including broadcast news, weblog and newswire. This indicates that there is low variation in the topics discussed by these sources – only a few named entities are mentioned by each. When named entities are seen as micro-topics, each indicating a grounded and small topic of interest, this reflects the nature of news having low topic variation, focusing on a few specific issues – e.g., location referred to tend to be big; persons tend to be politically or financially significant; and organisations rich or governmental BIBREF61 . In contrast, social media users also discuss local locations like restaurants, organisations such as music band and sports clubs, and are content to discuss people that are not necessarily mentioned in Wikipedia. The low overlap and memorisation scores on tweets, when taking entity lexica based on newswire, are therefore symptomatic of the lack of variation in newswire text, which has a limited authorship demographic BIBREF62 and often has to comply to editorial guidelines.
The other genre that was particularly difficult for the systems was ACE Usenet. This is a form of user-generated content, not intended for publication but rather discussion among communities. In this sense, it is social media, and so it is not surprising that system performance on ACE UN resembles performance on social media more than other genres.
Crucially, the computationally-cheap memorisation method actually acts as a reasonable predictor of the performance of other methods. This suggests that high entity diversity predicts difficulty for current NER systems. As we know that social media tends to have high entity diversity – certainly higher than other genres examined – this offers an explanation for why NER systems perform so poorly when taken outside the relatively conservative newswire domain. Indeed, if memorisation offers a consistent prediction of performance, then it is reasonable to say that memorisation and memorisation-like behaviour accounts for a large proportion of NER system performance.
To conclude regarding memorisation and out-of-domain performance, there are multiple issues to consider: is the corpus a sub-corpus of the same corpus as the training corpus, does it belong to the same genre, is it collected in the same time period, and was it created with similar annotation guidelines. Yet it is very difficult to explain high/low out-of-domain performance compared to in-domain performance with those factors.
A consistent trend is that, if out-of-domain memorisation is better in-domain memorisation, out-of-domain NERC performance with supervised learning is better than in-domain NERC performance with supervised learning too. This reinforces discussions in previous sections: an overlap in NEs is a good predictor for NERC performance. This is useful when a suitable training corpus has to be identified for a new domain. It can be time-consuming to engineer features or study and compare machine learning methods for different domains, while memorisation performance can be checked quickly.
Indeed, memorisation consistently predicts NER performance. The prediction applies both within and across domains. This has implications for the focus of future work in NER: the ability to generalise well enough to recognise unseen entities is a significant and still-open problem.
To summarise, our findings are:
[noitemsep]
What time period an out of domain corpus is collected in plays an important role in NER performance.
The context or other information represented by features shifts significantly more in Twitter than across different samples of newswire.
The generalisations that can be drawn from newswire by modern NER systems are not sufficient to give any useful performance in this varied kind of text.
Memorisation consistently predicts NER performance, both inside and outside genres or domains.
Conclusion
This paper investigated the ability of modern NER systems to generalise effectively over a variety of genres. Firstly, by analysing different corpora, we demonstrated that datasets differ widely in many regards: in terms of size; balance of entity classes; proportion of NEs; and how often NEs and tokens are repeated. The most balanced corpus in terms of NE classes is the CoNLL corpus, which, incidentally, is also the most widely used NERC corpus, both for method tuning of off-the-shelf NERC systems (e.g. Stanford NER, SENNA), as well as for comparative evaluation. Corpora, traditionally viewed as noisy, i.e. the Twitter and Web corpora, were found to have a low repetition of NEs and tokens. More surprisingly, however, so does the CoNLL corpus, which indicates that it is well balanced in terms of stories. Newswire corpora have a large proportion of NEs as percentage of all tokens, which indicates high information density. Web, Twitter and telephone conversation corpora, on the other hand, have low information density.
Our second set of findings relates to the NERC approaches studied. Overall, SENNA achieves consistently the highest performance across most corpora, and thus has the best approach to generalising from training to testing data. This can mostly be attributed to SENNA's use of word embeddings, trained with deep convolutional neural nets. The default parameters of SENNA achieve a balanced precision and recall, while for Stanford NER and CRFSuite, precision is almost twice as high as recall.
Our experiments also confirmed the correlation between NERC performance and training corpus size, although size alone is not an absolute predictor. In particular, the biggest NE-annotated corpus amongst those studied is OntoNotes NW – almost twice the size of CoNLL in terms of number of NEs. Nevertheless, the average F1 for CoNLL is the highest of all corpora and, in particular, SENNA has 11 points higher F1 on CoNLL than on OntoNotes NW.
Studying NERC on size-normalised corpora, it becomes clear that there is also a big difference in performance on corpora from the same genre. When normalising training data by size, diverse corpora, such as Web and social media, still yield lower F1 than newswire corpora. This indicates that annotating more training examples for diverse genres would likely lead to a dramatic increase in F1.
What is found to be a good predictor of F1 is a memorisation baseline, which picks the most frequent NE label for each token sequence in the test corpus as observed in the training corpus. This supported our hypothesis that entity diversity plays an important role, being negatively correlated with F1. Studying proportions of unseen entity surface forms, experiments showed corpora with a large proportion of unseen NEs tend to yield lower F1, due to much lower performance on unseen than seen NEs (about 17 points lower averaged over all NERC methods and corpora). This finally explains why the performance is highest for the benchmark CoNLL newswire corpus – it contains the lowest proportion of unseen NEs. It also explains the difference in performance between NERC on other corpora. Out of all the possible indicators for high NER F1 studied, this is found to be the most reliable one. This directly supports our hypothesis that generalising for unseen named entities is both difficult and important.
Also studied is the proportion of unseen features per unseen and seen NE portions of different corpora. However, this is found to not be very helpful. The proportion of seen features is higher for seen NEs, as it should be. However, within the seen and unseen NE splits, there is no clear trend indicating if having more seen features helps.
We also showed that hand-annotating more training examples is a straight-forward and reliable way of improving NERC performance. However, this is costly, which is why it can be useful to study if using different, larger corpora for training might be helpful. Indeed, substituting in-domain training corpora with other training corpora for the same genre created at the same time improves performance, and studying how such corpora can be combined with transfer learning or domain adaptation strategies might improve performance even further. However, for most corpora, there is a significant drop in performance for out-of-domain training. What is again found to be reliable is to check the memorisation baseline: if results for the out-of-domain memorisation baseline are higher than for in-domain memorisation, than using the out-of-domain corpus for training is likely to be helpful.
Across a broad range of corpora and genres, characterised in different ways, we have examined how named entities are embedded and presented. While there is great variation in the range and class of entities found, it is consistent that the more varied texts are harder to do named entity recognition in. This connection with variation occurs to such an extent that, in fact, performance when memorising lexical forms stably predicts system accuracy. The result of this is that systems are not sufficiently effective at generalising beyond the entity surface forms and contexts found in training data. To close this gap and advance NER systems, and cope with the modern reality of streamed NER, as opposed to the prior generation of batch-learning based systems with static evaluation sets being used as research benchmarks, future work needs to address named entity generalisation and out-of-vocabulary lexical forms.
Acknowledgement
This work was partially supported by the UK EPSRC Grant No. EP/K017896/1 uComp and by the European Union under Grant Agreements No. 611233 PHEME. The authors wish to thank the CS&L reviewers for their helpful and constructive feedback.

Question: What web and user-generated NER datasets are used for the analysis?
Answer: MUC, CoNLL, ACE, OntoNotes, MSM, Ritter, UMBC