Year: 2019

Abstract:
With the proliferation of models for natural language processing (NLP) tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 do not tell us why or how a particular method is better and how dataset biases influence the choices of model design. In this paper, we present a general methodology for interpretable evaluation of NLP systems and choose the task of named entity recognition (NER) as a case study, which is a core task of identifying people, places, or organizations in text. The proposed evaluation method enables us to interpret the model biases, dataset biases, and how the differences in the datasets affect the design of the models, identifying the strengths and weaknesses of current approaches. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive the progress in this area.

heading: 1 INTRODUCTION
text: The development of deep neural networks has greatly sped the evolution of NLP systems. However, these advances have also come with a plethora of design decisions: should we choose a CNN-based (Kalchbrenner et al., 2014; Kim, 2014), RNN-based (Sutskever et al., 2014; Bahdanau et al., 2014) or Transformer-based (Vaswani et al., 2017; Dai et al., 2018) architecture? What variety of pre-training method should we use (Le & Mikolov, 2014; Peters et al., 2018; Devlin et al., 2018; Akbik et al., 2018)? The proliferation of model variants pose a great challenge for current evaluation methodology, which are usually opaque and simply give a single holistic score (Papineni et al., 2002; Banerjee & Lavie, 2005; Popović & Ney, 2011).
To alleviate this problem, researchers have made efforts, mainly focusing in two directions. First, some works (Farrús Cabeceran et al., 2010; Popović & Ney, 2011; Lommel et al., 2014) have attempted to shift the granularity of evaluation from holistic to fine-grained by conducting error analysis. Despite its effectiveness, the process of error analysis usually requires manual examination and depends on some pre-existing assumptions, suffering from confirmation bias, and risking ignoring new types of errors (Neubig et al., 2019). Additionally, this evaluation method based on error analysis is usually applied to only a single dataset (Karpathy et al., 2015; Kummerfeld & Klein, 2013; Kummerfeld et al., 2012), lacking discussion of fine-grained analysis in a multi-dataset setting. As a result, many important questions remain unclear: how to characterize the factors that influence the tasks for different datasets? how do the different choices of datasets influence the models’ performance?
Another way to improve the evaluation strategy is common in our routine experimental design. That is to evaluate our models on multiple datasets with a holistic metric (Peters et al., 2018; Devlin et al., 2018). Currently, researchers are making efforts along this direction by setting up general evaluation benchmarks, such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), which involve a diverse set of datasets. Although it enables us to gain a more comprehensive assessment of the models, the influence of different datasets on models is simply reflected by a holistic metric, which is not interpretable, and consequently, we are not clear about how different datasets influence the choices of model architectures.
In this paper, we argue that a complete evaluation method should not only reflect the individual performance of the model on one dataset or multiple datasets but also be able to interpret the model biases, dataset biases, and their correlation (how the difference in the datasets affects the design
of the models). We draw on the complementary strengths of the fine-grained evaluation and multidataset evaluation, driving fine-grained analysis to the multi-dataset setting. To this end, we devise a generalized evaluation methodology and choose the NER task as a case study. More specifically, we introduce the notion of attribute, which can be defined flexibly as the evaluation task needs. Here, we utilize the attribute to describe the property of each test entity for the NER task (i.e., entity length). Then, the test set will be divided into a set of buckets by different attributes of test entities. This makes it possible to evaluate recognition accuracy of different varieties of entities, achieving much more fine-grained analysis than standard corpus-level measures.
Additionally, the proposed attribute-aided evaluation methodology encourages us to introduce multiple attributes to find more potential factors which affect the NER models on different datasets. We further propose three analytical approaches as shown in Fig. 1: attribute-wise, model-wise, and bucketwise analyses that have the following characteristics accordingly: Attribute-wise (Sec. 3.3.2) analysis could instruct us to find which factors matter for the NER tasks and figure out the commonality of factors across different NER datasets; Model-wise (Sec. 3.3.1) analysis aims to investigate how different attributes influence the performance of models with different architectures and pre-trained knowledge; Bucket-wise (Sec. 3.3.3) analysis diagnoses the strengths and weaknesses of existing models and helps us understand how different choices of datasets influence the model performance.
Our contributions can be precisely summarized as:
1) We draw on the complementary strengths of the fine-grained evaluation and multi-dataset evaluation, proposing a generalized evaluation methodology to interpret model biases, dataset biases, and their correlation. 2) We choose the NER task as a test case, the observations based on the extensive experiments (twelve models, ten attributes and six datasets) suggest directions for improvement and can drive the progress of this area. 3) Although some attributes defined in this paper are task-dependent, we claim our methodology is general since: a) for sequence labeling tasks (i.e., Partof-Speech, text chunking, and extractive summarization), this evaluation method could be transferred without much modification. b) for other types of NLP tasks, we could re-define the attributes and our proposed bucketization strategies, as well as three analytic approaches, are task-agnostic. For example, in machine translation, the attribution could be sentence length (Luong et al., 2015), word itself (n-gram) in the reference file (Kumar & Tsvetkov, 2018); In question answering, the attribution could be answer length/type/position (Seo et al., 2016), document length (Joshi et al., 2017), and query length/type (Chen et al., 2016b). Once we have determined related attributes, we could make similar analyses based on our proposed measures.

heading: 2 PRELIMINARIES
text: We first summarize and compare past evaluation methodologies and then describe the NER task and its current evaluation strategy.

heading: 2.1 PROPERTIES OF EVALUATION METHODOLOGIES
text: Evaluation is gaining increasing interest in NLP, especially on text generation tasks as exemplified by machine translation (Fishel et al., 2012; Irvine et al., 2013; Daems et al., 2014). Generally, different evaluation methods can be characterized by the following three main properties:
Interpretability: Evaluation method could give interpretable results to help us understand where
the weaknesses and strengths are. For example, “error analysis”(Kummerfeld et al., 2012;
Kummerfeld & Klein, 2013; Karpathy et al., 2015) is an interpretable evaluation method since we could figure out detailed limitations of the evaluated systems.
Confirmation bias: It represents a tendency to make tests consistent with the beforehand hypothesis. For example, Mudrakarta et al. (2018) assume that deep learning model are sensitive to question words in question answering tasks and verify it by carefully-designed adversarial examples, and Chen et al. (2016a) pre-defined six error to classify error cases.
Supplementary exam: It is an additional exam (require extra test sets ) for more comprehensive observations. i.e. multi-dataset evaluation (Devlin et al., 2018) on GLUE. Recently, there is a trend going from traditional evaluation to diagnostic test, in which a supplementary test set is required. For example, Jia & Liang (2017) propose an adversarial test for reading comprehension task and Naik et al. (2018) present a stress test method to diagnose natural language inference systems.

heading: 2.2 NER TASK AND CURRENT EVALUATION STRATEGY
text: Task Description Named entity recognition (NER) is usually formulated as a sequence labeling problem (Huang et al., 2015; Ma & Hovy, 2016). Formally, let X = {x1, x2, . . . , xT } be an input sequence and Y = {y1, y2, . . . , yT } be the output tags. The goal of this task is to estimate the conditional probability: P (Y |X) = P (yt|X, y1, · · · , yt−1)
Evaluation Strategy for NER Existing NER systems are commonly evaluated by corpus-level metrics (F1-score) (Sang & De Meulder, 2003) and a small amount of work will conduct some manual error analysis (Ichihara et al., 2015; Derczynski et al., 2015). With the increasing improvement in network architectures and pre-trained knowledge, the NER systems are quickly reaching a performance plateau (Akbik et al., 2018; Akbik et al.). Therefore, fine-grained evaluation is required to identify the specific issues of existing NER systems. On the other hand, with the emergence of more and more NER datasets (Sang & De Meulder, 2003; Collobert et al., 2011; Weischedel et al., 2013), the time is ripe for us to bridge the gap between the insufficient understanding of the nature of datasets itself and model designs. In this paper, we choose the NER task as a test case, interpreting model biases, dataset biases, as well as their correlation under a general framework. This work also takes a step towards interpretable architecture searching.

heading: 3 ATTRIBUTE-AIDED EVALUATION METHODOLOGY
text: Our proposed evaluation methodology involves three key elements: attribute definition (Sec. 3.1), bucketization (Sec. 3.2) and the analytical approach (Sec. 3.3). Specifically, we first introduce the notion of attribute and define it in NER task as some property of the test entity, bywhich the test set will be divided into different sub-sets and the overall performance could be broken down into interpretable categories. Below, we will detail the three key elements.

heading: 3.1 DEFINITION OF ENTITY ATTRIBUTES
text: Entity Attributes refer to the properties that can be used to characterize a given entity. Generally, different types of entity attributes provide different observation angles of system’s performances.
Next, we will introduce entity attributes we explored in this paper in terms of token level, span level and sentence level. We take them into consideration since they are general features and could be transferred to other tasks.
Token-level 1) Token itself: it denotes the words of an entity. 2) Morphology: morphological features plays an essential role in many NLP tasks. Here, we define five cases for each entity token: including token, is upper case, lower case, digit, beginning with a capital letter, and others (such as punctuation).
Span-level 1) Entity itself: each entity is regarded as a unique identifier. 2) Entity length: the number of tokens in an entity. 3) Entity tag: the NER tag of an entity.
Sentence-level 1) Sentence length; 2) Entity density: it is the number of entity tokens divided by sentence length. 3) OOV density: it is the number of oov word divided by sentence length.

heading: 3.2 BUCKETIZATION STRATEGIES
text: Bucketization is an operation that breaks down the holistic performance into different interpretable categories based on the attribute values. This can be achieved by dividing the set of test entities into different subsets of test entities (for span- and sentence-level attributes) or test tokens (for token-level attributes). Without loss of generality, we describe the entity-based bucketization strategies while it can be easily applied to token-based. To legibly describe the studied problem, we follow the commonly used notations throughout the paper. We refer to E, P , K as the sets of entities, entity attributes and attributes values, respectively.
The bucketization process can be formulated in a general form: Ete1 , · · · , Etem = Bucket(Ete|Etr, p), where Etr, Ete represent the sets of training and test entities, respectively, p denotes a type of the attribute. The basic idea is that the test entity set is divided into m buckets based on the attribute p and corresponding training set. where Etr, Ete represent the sets of training and test entities, respectively, p denotes a type of the attribute.
Specifically, each subset of test entities can be obtained as follows: Etei = {e|Atr(e, p) ∈ K̂i,∀e ∈ Ete}, where Atr(e, p) is to query out the value of attribute p for entity e. K̂i is a set of attribute values, with which entities should be put into the i-th bucket: K̂i = {k|f(k) = i,∀k ∈ K}. Above equation shows the key part for bucketization: how do we build the relationship between the value of entity attributes and the bucket number?. That is, we need to determine a criterion f , which guides us to put test entities according to its attribute values into suitable buckets. Here we explore three types of strategies for bucketization.
Strategy-I: Range division of attribute values (R-Bucket) An intuitive strategy is to bucketize the test entities based on their attribute values directly. Specifically, we could divide the range of attribute values into m discrete parts. For example, the sentence length with range [1, · · · , 6] could be divided into [1, 2], [3, 4], [5, 6] three buckets. Tab. 2 shows which attributes are matched to this bucketization strategy.
Strategy-II: Familiarity of attribute values (F-Bucket) The aim of the evaluation is to quantify the generalization errors of the system on the unseen samples. Therefore, by taking into account the degree to which the testing entities (or their attributes) have been seen in the training set, we can better figure out the impact of this attribute on model performance.
To achieve this, here we introduce a notion of familiarity to quantify the degree to which the attribute of a test entity has been seen in the training set.
Fk(p) = |{e|Atr(e, p) = k, ∀e ∈ Etr}|
|{e ∈ Etr}| (1)
Here, Fk(p) ∈ [0, 1] denotes the degree to which the test entity attribute p with value k have been seen in the training set. Then we can define the following criterion to achieve the bucketization: FK̂i(p) ∈ ( i m , i+1 m ]. The basic idea behind the criterion is that test entities could be put into the i-th bucket when the familiarity of their attribute values meets the above condition. Tab. 2 shows which attributes we used for F-Bucket.
Strategy-III: Multi-attribute Familiarity (MF-Bucket) The benefit of our general methodology for interpretable evaluation is that we can easily define new valuable measures based on old measures already defined. Here, we can adapt our F-Bucket strategies to multi-attribute version and modify the
Eq.1 as follows:
Fk(p1, p2) = |{e|Atr(e, p1) = k1 ∧Atr(e, p2) = k2,∀e ∈ Etr}|
|{Atr(e, p1) = k1,∀e ∈ Etr}| (2)
For example, when we instantiate the two entity attributes p1, p2 as entity itself and tag respectively, the familiarity Fk(p1, p2) is a measure with intriguing explanation: for each test entity with tag k, this measure quantifies its category ambiguity: the probability that this entity is labeled as k in the training set.
To give a more intuitive understanding of the two processes: attribute definition and bucketization, Fig. 2 shows a concrete example, in which a set of attributes are defined for the target entity New York, with corresponding values.

heading: 3.3 ANALYTIC APPROACHES
text: To better characterize the relation among attributes, models and datasets, we propose three analytical approaches: attribute-wise, model-wise, and bucket-wise, which can be used to interpret the model biases, dataset biases, and their correlation.
Formally, we refer to M = m1, · · · ,m|M | as a set of models and P = p1, · · · , p|P | as a set of attributes. As describe above, the test set E could be split into different buckets E = Ej1, · · · , E j |E| based on a attribute pj . We introduce the notion of performance table T ∈ R|M |×|P |×|E|, in which Tijk represents the performance of i-th model on the k-th sub-test set (bucket) generated by j-th attribute. Next, we will show how these approaches are defined based on T .

heading: 3.3.1 MODEL-WISE
text: The model-wise analysis aims to investigate how different attributes influence the performance of models with different architectures and pre-trained knowledge. For example, “does the lengths of entities influence the performances of CNN-LSTM-CRF-based NER system?”
Here we adopt two types of statistical variables Sρi,j and S σ i,j to characterize how the j-th attribute influences the i-th model.
Sρi,j = Spearman(T [i, j :], Rj) (3) Sσi,j = Std(T [i, j :]) (4)
where Spearman is a function to calculate Spearman’s rank correlation coefficient (Mukaka, 2012) and Rj is the rank values of buckets based on j-th attribute. Std(·) is the function to compute standard deviation.
Intuitively, Sρi,j reflects the degree to which the i-th model positively (or negatively) correlates with j-th attribute while Sσi,j indicates the degree to which this attribute influences the model.

heading: 3.3.2 ATTRIBUTE-WISE
text: The attribute-wise analysis aims to quantify the degree to which each attribute influences the NER task. To achieve this, we introduce four measures: task-independent variable ζj and task-dependent variables ρ1j ρ 2 j and σj based on Eq.3 and Eq.4:
1) ζj = 1|N | ∑|N | i Atr(ei, j), where N is the number test entities and Atr(ei, j) represents the value of attribute j for entity ei.
For example, when j represents the attribute of sentence length, ζj is the average sentence length of the whole dataset.
2) ρ1j = 1 |M | ∑|M | i |S ρ i,j |, ρ2j = 1|M | ∑|M | i S ρ i,j , σj = 1 |M | ∑|M | i S σ i,j , where |M | is the number of evaluated models. Compared with ρ1j , ρ 2 j can reflect whether the correlation is positive or negative.
Intuitively, a higher absolute value of ρ1j or ρ 2 j suggests that attribute j is a crucial factor, greatly influencing the performance of NER systems.

heading: 3.3.3 BUCKET-WISE
text: The bucket-wise analysis diagnoses the strengths and weaknesses of existing models. Moreover, based on attribute-wise analysis, we could understand how different choices of datasets influence the models’ performance.
To this end, we introduce the following measures:
βj = { maxk(T [a, j, k]− T [b, j, k]) (T [a, j, k] > T [b, j, k], for ∀k) mink(T [a, j, k]− T [b, j, k]) otherwise
(5)
Usually where a, b represent two different models and usually model a has a higher performance (by dataset-level metric).
Intuitively, a negative value of βj suggests that a worse-ranked model (b) outperform the best-ranked model (a) in some aspect (attribute j); By contrast, a positive value shows the largest margin on the attribute j.

heading: 4 EXPERIMENTAL SETTINGS

heading: 4.1 MODELS AND ATTRIBUTES
text: Model Settings To evaluate the importance of different components of the NER systems, we varied our models mainly in terms of three aspects: different choices of character- (ELMo (Peters et al., 2018), Flair (Akbik et al., 2018; Akbik et al.)), subword- (BERT (Peters et al., 2018; Devlin et al., 2018)), word- (GloVe(Pennington et al., 2014)), and sentence-level encoders (LSTM (Hochreiter & Schmidhuber, 1997), CNN (Kalchbrenner et al., 2014)) and decoders (MLP or CRF (Lample et al., 2016; Collobert et al., 2011)). Detailed setting are shown in Tab.3. Totally, we study 12 NER models based on deep neural networks and one traditional method utilizing CRF Lafferty et al. (2001). The hyper-parameter settings of our evaluated models are shown in appendix section.
Attributes In our evaluation methodology, entity attributes can be defined flexibly and attribute values can be continuous or discrete. In this paper, although we investigate 10 types of attributes (or their combinations) as listed in Tab.2, others can be easily introduced.

heading: 4.2 NER DATASETS FOR EVALUATION
text: We conduct experiments on three benchmark datasets: the CoNLL2003 NER dataset, the WNUT16 dataset, and Ontonotes 5.0 dataset. The CoNLL2003 NER dataset (Sang & De Meulder, 2003) is based on Reuters data (Collobert et al., 2011). WNUT16 dataset is provided by the second shared task at WNUT-2016. The Ontonotes 5.0 dataset (Weischedel et al., 2013) is collected from newsgroups, broadcast news (BN), broadcast conversation (BC) and weblogs (WB) and magazine genre (MZ).

heading: 5 ANALYSIS

heading: 5.1 HOLISTIC ANALYSIS
text: Before giving a fine-grained analysis, we present the results of different models on different datasets in the way that traditional multi-dataset evaluation does. As shown in Fig. 3, we observe that there is no one-size-fits-all model, and the models with the best results on different datasets are different. Naturally, the following questions are raised: 1) What factors of the datasets can distinguish themselves and influence the NER task? 2) How do these factors influence the choices of models? 3) Does a worse-ranked model outperform the best-ranked model in some aspect and how the datasets influence the choices of models? The following analyses will be conducted around these questions.
5.2 ATTRIBUTE-WISE ANALYSIS
Attribute-wise measures enable us to characterize the dataset biases quantitatively. Here we utilize a radar chart in Fig. 4 to strikingly display the commonality and speciality between different datasets based on three measures ζ, ρ1, σ defined in Sec. 3.3.2. And we also illustrate measure ρ2 in Fig. 3. Detailed observations are listed as follows:
Category ambiguity and entity length have more consistent influence on NER performance. The common parts of the radar chart Fig. 4 (b-c) illustrate that no matter which datasets are, the performance of NER task is highly correlated with these attributes:
MF-tt (token-tag with MF-buck), MF-et (entity-tag with MF-buck), R-eLen (entity length with R-bucket). Fig. 3 shows the same result. This suggests that the prediction difficulty of named entity is commonly influenced by category ambiguity (MF-tt, MF-et), entity length (R-eLen).
Occurence and sentence length matters but are minor factors. The outliers in radar chart show the peculiarities of different datasets. Intutively, on attributes: R-sLen, F-token, R-oov, the extent to which different datasets are affected varies greatly. Typically, as observed from Fig. 4-(a), a sorted sequence could be obtained according to the attribute F-tok: BN > MZ > WNUT > CoNLL > BC > WB. The reason why the Spearman correlations ρ1 of BC and WB are smaller is that performance on test entities with higher-frequency tokens are lower than entities with lower-frequency tokens. This suggests that F-tok is not a decisive attribute and higher-frequency token can not guarantee a better performance since other crucial factors such as category ambiguity also matter.
Entity density is a swing factor and CoNLL dataset is an outlier. As shown in Fig. 3, the measure ρ2 enables us to know the correlation is positive or negative. We observe that most datasets except CoNLL 2003 have negative Spearman values ρ2 on the attribute of R-eDen, which suggests that a sentence with more entities is relatively harder to process. We can explain the unusual behavior on CoNLL 2003 from its intrinsic value ζ of eDen, the largest one as shown in Fig. 4(a). The dataset of CoNLL contains a lot of short sentences, such as “Chicago 8,674 484,018” and “ SOFIA 1996-12-06”. That’s why it distinguishes itself from other datasets.
MWE
The intrinsic differences in datasets can help us to understand how different datasets influence the different choices of models, which will be explained later (Sec. 5.4).

heading: 5.3 MODEL-WISE ANALYSIS
text: Based on two model-wise measures: Sρi,j and S σ i,j defined in Eq.3 and Eq.4, we investigate how different attributes influence the performance of the models with different architectures and pretrained embeddings. Fig. 4 illustrates the case on CoNLL (other datasets are included in appendix), and we have observed that:

heading: 1) Char-unaware models are more sensitive to the degree of category ambiguity and occurrence
text: of entities. We observe that “CnonWrandlstmCrf ” is negatively related to MF-et, MF-tt, F-ent and F-tok with high values of ρ and σ, suggesting the importance of the character-level encoder, which plays a major role in generalization to rare entities and entities with multiple categories. More importantly, this observation still holds on other datasets. (See appendix)

heading: 2) Sentence length is a swing factor, whose contribution depends on what types of pre-trained
text: embeddings are used. There is a strong negative correlation between models with contextindependent embeddings and the attribute R-sLen (sentence length). The relationship is reversed when contextualized embeddings are used. The reason we believe is that long sentences could provide sufficient context information for contextualized models. It is noticeable, however that flair-related models “CflairWnonelstmCrf ” behave differently compared with other contextualized models, which
we will explain later in Sec 5.4 (Flair performs worse than ELMo when dealing with long sentences due to its structural bias).
3) Character encoders favor the sentences with high entity-density. Only using character-level CNN is easy to overfit the feature of capital letters. As a result, more non-entities are mis-predicted as entities. Based on the understanding of the previous analysis (Sec. 5.2) of the entity density’s influence on CoNLL, we could better explain why “CcnnWnonelstmCrf ” achieves the highest value of ρ and σ in “R-eDen (entity density) attribute in Tab.4.

heading: 5.4 BUCKET-WISE ANALYSIS
text: In this section, we choose several typical models (others are shown in the appendix) as analyzed cases, aiming to seek answers to the following questions: 1) What are the strengths and weakness of different architectural designs? In other words, does a worse-ranked model outperform the best-ranked model in some aspect? 2) How do the different choices of datasets influence model performance? Tab. 5 illustrates the bucket-wise measure β, which is computed based on any pair of models M1 and M2. Next, we list some of our observations. Others are illustrated in the appendix.
CNN v.s. LSTM The sentence encoder of CNN is better at dealing with short sentences, which holds in all datasets we evaluated in this paper. Strikingly, CNN outperforms LSTM by a large margin (dataset-level F1) on MZ dataset while is significantly worse than LSTM on WB (refer to the appendix.). We attempt to explain these discrepancies based on above attribute-wise metric ζ in Fig. 4(a): sentence length and entity density are two major factors for the choices of CNN and LSTM. CNN is better than LSTM when the dataset with higher value of ζsLen and ζeDen. By contrast, when a dataset with lower ζeDen, LSTM is a priority (The ζeDen of WB is the lowest).
CRF v.s. MLP The benefits of using CRF on short sentences are very stable, and improvement can be seen in all datasets. Similarly, based on attribute-wise metric ζ in Fig. 4, we find category ambiguity (MF-et, MF-tt) is a major factor for the choices of CRF and MLP: if a dataset with higher ζMF−et, in which longer entities can benefit more from CRF-based models. In comparison, introducing CRF will lead to more errors on long entities once the dataset (i.e. BN, MZ) has a lower ζMF−et.
CcnnWrand v.s. CcnnWnone The question has been little studied whether we need an extra wordlevel method (i.e., word-level look-up table) to get word representations when we have already used CNN to obtain word representations. Here, we show that CcnnWrand is not always better than CcnnWnone and entity length matters for the choice. Specifically, CcnnWnone could achieve better performance on on the WNUT and MZ datasets. With the help of Fig. 4-(a), we find the two datasets share a property of much higher value of ζR−ele (entity length). Additionally, another commonality between the two datasets can be observed from Tab. 5: the gain of CcnnWnone mainly comes from the entities with the longer length.
ELMo v.s GloVe ELMo consistently outperforms GloVe on all datasets using the holistic F1 score, but it is worse than GloVe when modeling short sentences. Another interesting finding is that for those sentences containing more OOV tokens, GloVe could achieve better performance on all datasets. These phenomena indicate the complementarity between ELMo and GloVe, and our further combination of these two embeddings (CelmWglovelstmCrf in Tab. 3) indeed works better.
Flair v.s. ELMo While the current state-of-the-art NER model Flair has achieved the best performance in terms of dataset-level F1 score, a worse-ranked model (ELMo) could outperform it in some aspects. Typically, Flair performs worse when dealing with long sentences. The reason can be attributed to the its structure design, which adopts a LSTM-based encoder for character language modeling, suffering from long-term dependency problem (especially for character-level language model). A promising improvement is to use the Transformer-based architecture for character language model.

heading: 6 CONCLUSION
text: To bridge the gap between the insufficient understanding of the nature of datasets and model designs, this paper proposes a generalized evaluation methodology to interpret model biases, dataset biases,
CoNLL-2003 WNUT-2016 OntoNote-MZ OntoNote-BC
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
Overall F1 M1: 90.48; M2: 90.14 M1: 40.61; M2: 36.21 M1: 85.39; M2: 88.10 M1:76.04; M2: 76.74
M L S M L S L L LO C O th er
−5 0
5
10
M S S M L L M S SP O O th er
0
20
M M S S M S M L Q U A Lo we
r−20
−10
0
L S L M M L M L N O R P Lo
we r−10
−5 0
5
M1: CcnnWglovelstmCrf
M2: CcnnWglovecnnCrf
Overall F1 M1: 90.48; M2: 88.05 M1: 40.61; M2: 32.84 M1: 85.39; M2: 81.09 M1: 76.04; M2: 70.00 M S M M S M M S
O RG D ig
it
−10
0
10
S S L L L M S S FA C O th er
−10 0
10
20
L M M L S S L L T IM E Lo we
r
−10 0
10 20
M S L M M L S S PE RC U pp er
−10 0
10 M1: CcnnWglovelstmCrf
M2: CcnnWglovelstmMlp
Overall F1 M1: 83.80; M2: 77.01 M1: 22.57; M2: 22.73 M1: 78.85; M2: 79.05 M1: 71.57; M2: 65.01
M S L M S S M S M IS C D ig it
−10 0
10 20
M M L S M S S L T V S
O th
er
−10
−5 0
5
L M S L M M S L PE RC O th er
−10
0
M L S S S S M M LO C Lo
we r−20
0
20
M1: CcnnWrandlstmCrf
M2: CcnnWnonelstmCrf
Overall F1 M1: 91.64; M2: 90.48 M1: 44.56; M2: 40.61 M1: 86.32; M2: 85.39 M1: 77.10; M2: 76.04
M L S L S L L L O RG U pp er
−5 0
5
10
L L S M L M M L C O M Lo we
r
−10
0
10
S L L M M S M S LO C Lo
we r−20
−10 0
10 20
S L S M L M L S M O N D ig it
−60 −40 −20
0 M1: CelmoWnonelstmCrf
M2: CcnnWglovelstmCrf
Overall F1 M1: 93.03; M2: 92.22 M1: 45.96; M2: 45.33 M1: 85.56; M2: 85.70 M1: 77.23; M2: 78.71
S L L S S M M L M IS C D ig it
0
10
S S L L L L M M T V S Lo we
r−10
−5 0
5
S S M M M M S S FA C U pp er
−20
−10
0
L L S S S S M L LA W O th er
−20
−10 0

heading: M1: CflairWglovelstmCrf
text: M2: CelmWglovelstmCrf
Table 5: Illustration of the bucket-wise measure β. Each histogram is obtained based on subtracting the performance of Model1 (M1) from Model2 (M2) on a bucket. For ease of presentation, we roughly classify some attribute values into three categories: small(S), middle(M) and large(L). For example, the first column of the top left histogram represents M2 outperforms M1 when the attribute R-eDen takes the small (S) values.
and their correlation, drawing on the complementary strengths of the fine-grained evaluation and multi-dataset evaluation. We choose the NER task as a test case, the observations based on the extensive experiments suggest directions for improvement and can drive the progress of this area.

heading: A HYPER-PARAMETERS

heading: B MODEL-WISE ANALYSIS ON THE OTHER DATASETS
text: C BUCKET-WISE ANALYSIS
CoNLL-2003 WNUT-2016 OntoNote-MZ OntoNote-BC
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
R -e
D en
R -o ov R -s L en M Fet Fen t M Ftt Fto k R -e L en Fta g R -m or
Overall F1 M1: 92.22; M2: 91.64 M1: 45.33; M2: 44.56 M1: 85.70; M2: 86.32 M1: 78.71; M2: 77.10
M M L L L S L L M IS C D ig it
−10 −5
0 5
S M L M M L L L SP O Lo
we r−10
−5 0
5
L L S S L S L L LA N O th er
−40
−20 0
L L M L M M L M LA W U pp er
−10
−5 0
5
M1: CelmWglovelstmCrf
M2: CelmWnonelstmCrf
Overall F1 M1: 92.22; M2: 91.11 M1: 45.33; M2: 42.50 M1: 85.70; M2: 86.90 M1: 78.71; M2: 81.03
L S L S L M L L O RG D ig it
−10 −5
0 5
S M L M L S M M SP O T itl e
−20
−10 0
L M S S S S S L M O N Lo we
r−20
−10
0
S S M L S M S M FA C D ig it
−10
0M1: CelmWglovelstmCrf
M2: CbertWnonelstmMlp
Table 12: A supplement bucket-wise analysis results to Tab. 5. Each histogram is obtained based on subtracting the performance of Model1 (M1) from Model2 (M2) on a bucket. For ease of presentation, we roughly classify some attribute values into three categories: small(S), middle(M) and large(L). For example, the first column of the top left histogram represents M2 outperforms M1 when the attribute R-eDen takes the small (S) values.

Review summaries:
The manuscript proposes an evaluation methodology to obtain deeper insights regarding the strength and weaknesses of different methods on different datasets. The method considers a set of methods addressing the task of Named Entity Recognition (NER) as case study. In addition, it proposes a set of attribute-based criteria, i.e. bucketization strategies, under which the dataset can be divided and analyzed in order to highlight different properties of the evaluated methods.