Towards Interpretable Evaluations: A Case Study of Named Entity Recognition
Authors: Jinlan Fu
Pengfei Liu
Xuanjing Huang
Venue:  (2020)
Keywords: interpretable evaluation
dataset biases
model biases
NER

Abstract:     With the proliferation of models for natural language processing (NLP) tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 do not tell us \emph{why} or \emph{how} a particular method is better and how dataset biases influence the choices of model design.
    In this paper, we present a general methodology for {\emph{interpretable}} evaluation of NLP systems and choose the task of named entity recognition (NER) as a case study, which is a core task of identifying people, places, or organizations in text. The proposed evaluation method enables us to interpret the \textit{model biases}, \textit{dataset biases}, and how the \emph{differences in the datasets} affect the design of the models, identifying the strengths and weaknesses of current approaches. By making our {analysis} tool available, we make it easy for future researchers to run similar analyses and drive the progress in this area.

Under review as a conference paper at ICLR 2020
TOWARDS INTERPRETABLE EVALUATIONS: STUDY OF NAMED ENTITY RECOGNITION
A CASE
Anonymous authors Paper under double-blind review
ABSTRACT
With the proliferation of models for natural language processing (NLP) tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metries such accuracy, BLEU, or FI do not tell us why or how a particular method is better and how dataset biases influence the choices of model design. In this paper, we present general methodology for interpretable evaluation of NLP systems and choose the task of named entity recognition (NER) as a case study, which is a core task of identifying people, places, organizations in text. The proposed evaluation method enables us to interpret the model biases, dataset biases, and how the differences in the datasets affect the design of the models, identifying the strengths and weaknesses of current approaches. By making our analysis tool available, we make it casy for future researchers to run similar analyses and drive the progress in this area.
INTRODUCTION
The development of deep neural networks has greatly sped the evolution of NLP systems. However, these advances have also come with a plethora of design decisions: should we choose a CNN-based (Kalchbrenner al., 2014; Kim, 2014), RNN-based (Sutskever et al., 2014; Bahdanau et al., 2014) or Transformer-bat based (Vaswani et al., 2017; Dai et al., 2018) architecture? What variety of pre-training method should we use (Le Mikolov, 2014; Peters et al., 2018; Devlin et al., 2018; Akbik et al., 2018)? The proliferation of model variants pose great challenge for current evaluation methodology, which are usually opaque and simply give single holistic score (Papineni et al., 2002; Banerjee & Lavie, 2005; Popović & Ney, 2011).
To alleviate this problem, researchers have made efforts, mainly focusing in two directions. First, some works (Farrús Cabeceran et al., 2010; Popović & Ney, 201 1; Lommel et al., 2014) have attempted to shift the granularity of evaluation from holistic to fine-grate lineigranedd by conducting error analysis. Despite its effectiveness, the process of error analysis usually requires manual examination and depends on some pre-existing assumptions, suffering from confirmation bias, and risking ignoring new types of errors (Neubig al., 2019). Additionally, this evaluation method based on error analysis is usually applied to only a single dataset (Karpathy et al., 2015; Kummerfeld & Klein, 2013; Kummerfeld al., 2012), lacking discussion of fine-grained analysis in multi-dataset setting. As result, many important questions remain unclear: how to characterize the factors that influence the tasks for different datasets? how do the different choices of datasets influence the models' performance?
Another way to improve the evaluation strategy is common in our routine experimental design. That evaluate our models on multiple datasets with holistic metric (Peters et al., 2018; Devlin et al., 2018). Currently, researchers are making efforts along this direction by setting up general evaluation benchmarks, such as GLUE (Wang et al., 2018) and SuperGI.UF (Wang et al., 2019), which involve diverse set of datasets. Although it enables us to gain more comprehensive assessment of the models, the influence of different datasets on models is simply reflected by a holistic metric, which is not interpretable, and consequently, are not clear about how different datasets influence the choices of model architectures.
In this paper, we argue that a complete evaluation method should not only reflect the individual performance of the model on one dataset or multiple datasets but also be able to interpret the model biases, dataset biases, and their correlation (how the difference in the datasets affects the design

---

Under review as a conference paper at ICLR 2020
of the models). We draw on the complementary strengths of the fine-grained evaluation and multi- dataset evaluation, driving fine-grained analysis to the multi-dataset setting. To this end, we devise a generalized evaluation methodology and choose the NER task as case study. More specifically, introduce the notion of attribute, which can be defined flexibly as the evaluation task needs. Here, we utilize the attribute to descrihe the property of each test entity for the NER task (i.e., entity length). Then, the test set will be divided into set of buckets by different attributes of test entities. This makes possible to evaluate recognition accuracy of different varieties of entities, achieving much more fine- teneigriiedd analysis than standard corpus-level measures.
Additionally, the proposed attribute- aided evaluation method- ology encourages us to introduce multiple attributes to find more potential factors which affect the NER models on differ- ent datasets. We further propose three analytical approaches as shown in Fig. I: attribute- model-wise, and bucket- wise analyses that have the following characteristics accord- ingly: Attribute-wise (Sec. 3.3.2) analysis could instruct us to find which factors matter for the NER tasks and figure out the commonality of factors across different NER datasets; Model-wise (Sec. 3.3.1) analysis aims investigate how dif- ferent attributes influence the performance of models with dif- ferent architectures and pre-trained knowledge; Bucket-wise (Sec. 3.3.3) analysis diagnoses the strengths and weaknesses of existing models and helps us understand how different choices of datasets influence the model performance.
Atcributes
Tob: Idasal spjon
Fig 4
Tab. 5 Burdortoniae
Models
Tab,
Datasets
Figure 1: Relation chart among at- tributes, models, and datasets. The attribute-aimed method could bridge the gap between the model biases and dataset biases.
Our contributions can be precisely summarized as:
1) We draw on the complementary strengths of the fine- fine-grained evaluation and multi- mattti-dttsstt evaluation, proposing a generalized evaluation methodology interpret model biases, dataset biases, and their correlation. 2) We choose the NER task as test case, the observations based on the extensive experiments (twelve models, ten attributes and six datasets) suggest directions for improvement and can drive the progress of this area. 3) Although some attributes defined in this paper lask-dependent, we claim our methodology is general since: a) for sequence labeling tasks (i.e., Part- of of-Speech, text chunking, and extractive summarization), this evaluation method could be transferred without much modification. b) for other types of NLP tasks, we could re-define the attributes and our proposed bucketization strategies, well three analytic approaches, are task-agnostic. For example, in machine translation, the attribution could be sentence length (Luong et al., 2015), word itself (n-gram) in the reference file (Kumar & Tsvetkov, 2018); In question answering, the attribution could be answer length/typepposition (Sco et al., 2016), document length (Joshi et al., 2017), and query length/type (Chen et al., 2016b). Once we have determined related attributes, we could make similar analyses based on our proposed measures.
2 PRELIMINARIES
We first summarize and compare past evaluation methodologies and then describe the NER task and its current evaluation strategy.
2.1 PROPERTIES OF EVALUATION METHIODOLOGIES
Evaluation is gaining increasing interest in NLP. especially on text generation tasks as exempli- fied by machine translation (Fishel et al., 2012: Irvine et al., 2013; Daems et al., 2014). Gener- ally. different evaluation methods can be charac- terized by the following three main properties:
Methodology
Interp-
Conform.
Supple-exam
Holistic metric
X
Multi datnset
Errur analysis
Diugnostic
Interpretable metric
Interpretability: Evaluation method could give interpretable results to help us understand where the weaknesses and strengths For example, "error analysía i ""Kummerfelld et al., 2012;
Table 1: Evaluation methodologies characterized by diffeerent properties.
2

---

Under review as a conference paper at ICLR 2020
Kummerfeld & Klein, 2013; Karpathy al., 2015) is interpretable evaluation method since could figure out detailed limitations of the evaluated systems.
Confirmation bias: It represents tendency to make tests consistent with the beforehand hypothesis. For example, Mudrakarta et al. (2018) assume that deep learning model are sensitive to question words in question answering tasks and verify it by carefully-desinnee adversarial examples, and Chen et al. 6a) pre-defined six error to classify error cases.
Supplementary exam: It is additional exam (require extra test sets for more comprehensive observations. i.e. multi-dataset evaluation (Devlin et al., 2018) on GLUE. Recently, there is a trend going from traditional evaluation diagnostic lest, in which supplementary test set is required. For example. Jia & Liang (2017) propose an adversarial test for reading comprehension task and Naik al. (2018) present a stress test method to diagnose natural language inference systems.
2.2 NER TASK AND CURRENT EVALUATION STRATEGY
Task Description Named entity recognition (NER) is usually formulated as sequence labeling problem (Huang et al., 2015; Ma & Hovy, 2016). Formally, let X { 22,..., Fr) be input sequence and y = {yı. .2.............................. yr be the output tags. The goal of this task is to estimate the conditional probability: P(Y[X)= P(y1XX,,, ",,,""
Evaluation Strategy for NER Existing NER systems are commonly evaluated by corpus-level metrics (FI-score) (Sang & De Meulder, 2003) and a small amount of work will conduct some manual error analysis (Ichihara et al., 2015; Derczynski et al., 2015). With the increasing improve- ment in network architectures and pre-trained knowledge, the NER systems are quickly reaching a performance plateau (Akbik et al., 2018; Akbik et al.). Therefore, fine-grained evaluation is required to identify the specific issues of existing NER systems. On the other hand, with the emergence of more and more NER datasets (Sang & De Meulder, 2003; Collobert et al., 2011; Weischedel et al., 2013), the time is ripe for us to bridge the gap between the insufficient understanding of the nature of datasets itself and model designs. In this paper, we choose the NER task as test case, interpreting model biases, dataset biases, as well as their correlation under general framework. This work also takes step towards interpretable architecture searching.
ATTRIBUTE-AIDER EVALUATION METHODOLOGY
Our proposed evaluation methodology involves three key elements: altribute desinition (Sec. 3. 3.1), bucketization (Sec. 3.2) and the analytical approach (Sec. 3.3). Specifically, first introduce the notion of attribute and define it in NER task as some property of the test entity, bywhich the test set will be divided into different sub-sets and the overall performance could be broken down into interpretable categories. Below, we will detail the three key elements.
3.1 DEFINITION OF ENTITY ATTRIBUTES
Entity Attributes refer to the propertics that can be used to characterize a given entity. Generally, different types of entity attributes provide different observation angles of system's performances.
Next, we will introduce entity attributes we explored in this paper in terms of token level, span level and sentence level. We take them into consideration since they general features and could be transferred to other tasks.
Token-level 1) Token itself: it denotes the words of entity. 2) Morpholoyy: morphological features plays an essential role in many NLP tasks. Here, we define five cases for each entity token: including token, is upper case, lower case, digit, beginning with capital letter, and others (such as punctuation).
Span-level 1) Entity itself: each entity is regarded as a unique identifier. 2) Entity length: the number of tokens in an entity. 3) Entity tag: the NER tag of an entity.
Sentence-level 1) Sentence length; 2) Entity density: it is the number of entity tokens divided by sentence length. 3) O0V density: is the number of oov word divided by sentence length.
3

---

Under review as a conference paper at ICLR 2020
3.2 BUCKETIZATION STRATEGIES
Bucketization is operation that breaks down the holistic performance into different interpretable categories based on the attribute values. This can be achieved by dividing the set of test entities into different subsets of test entities (for span- and sentence-level attributes) test tokens (for token-level attributes). Without loss of generality, we describe the entity-based bucketization strategies while can be easily applied to based. To legibly describe the studied problem, we follow the commonly used notations throughout the paper. We refer to E, P, K as the sets of entities, entity attributes and attributes values, respectively.
The bucketization process can be formulated in general form: E E = Bucket (E E p)- where F F represent the sets of training and test entities, respectively, p denotes a type of the attribute. The basic idea is that the test entity set is divided into buckets based the attribute p and corresponding training set. where E E represent the sets training and test entities, respectively, p denotes type of the attribute.
Specifically, each subset of test entities can be obtained follows: E {elAtr(e,p)) p) E
Bucketization Strategies
Attributes
R-Bucket F-Bucket MF-Buacke
Ki, Ve E Ett, where Atr is to query out the value of attribute for entity e. K, is set of attribute values, with which enti- ties should be put into the i-th bucket: K2 = {k] (A) = i, vk K}. Above equation shows the key part for bucketization: how do we build the relationship between the value of entity at- tributes and the bucket number?. That is, we need to determine a criterion f. which guides us to put test entities according to its attribute values into suitable buckets. Here we explore three types of strategies for bucketization.
Token itself (F-aok)
Token Morphology (R-mor)
Entity itself (F-ent) Entity len, (R-eLen) Eniity tag (F-tag)
Span
Sent length (R-s (R-sLen) Eniity dens. (R eDen) OOV dens. (R-oov)
Seat
Entity (MF-et) Token tag (MF-it)
Table 2: their corresponding bucketization strategies.
Entity attributes we used in this paper and
Strategy-I: Range division of attribute val-
ues (R-Bucket) intuitive strategy to bucketize the test entities based on their attribute values directly. Specifically, we could divide the range of attribute values into discrete parts. For exam- ple, the sentence length with range [1.... 6] could be divided into [1, 2). [3. 4]. [5, 6] three buckets. Tab. 2 shows which attributes are matched to this bucketization strategy.
Strategy-II: Familiarity of attribute values (F-Bucket) The aim of the evaluation is to quantify the generalization errors of the system on the unseen samples. Therefore, by taking into account the degree to which the testing entities (or their attributes) have been seen in the training set, we can better figure out the impact this attribute on model performance.
To achieve this, here we introduce notion of familiarity to quantify the degree to which the attribute of a test entity has been seen in the training set.
F& (p) He|Atr(e, = k,
V, Ve ETT} HE E E+rl
(I)
Here, (x( E [0, 1] denotes the degree to which the test entity attribute with value have been seen in the training set. Then we can define the following criterion achieve the bucketization: FR, (p) E (L, m The basic idea behind the criterion is that test entities could be put into the i-th bucket when the familiarity of their attribute values meets the above condition. Tab. 2 shows which attributes we used for F- F-Bukket.
Strategy-III III: Multi-attribute Familiarity (MF-Bucket) The benefit of our general methodology for interpretable evaluation is that we can easily define new valuable measures based on old measures already defined. Here, can adapt our F-Bucket strategies to multi-atrribte version and modify the
4

---

Under review as a conference paper at ICLR 2020
Eq.1 follows:
Fe P2) = HelAtr(e, p1) by Atr(e. k2, Ve E E
[Atr(e, = k1, Ve E HnTH
For example, when we instantiate the two entity attributes pt, as entity itself and tag respectively. the familiarity (p1, p2) is a measure with intriguing explanation: for each test entity with tag k. this measure quantifies category ambiguity: the probability that this entity is labeled as k in the training set.
MF-
New York
To give more intuitive understanding the two processes: attribute definition and bucketization, Fig. 2 shows concrete example, in which set of attributes are defined for the target entity New York, with corresponding values.
Rel
The attribute defini- tion of the entity New York in the sentence: "No new fixtures reported from New York."
Figure 2:
3.3 ANALYTIC APPROACIIES
To better characterize the relation among attributes, models and datasets, propose three analytical approaches: attribute-wiss model-wise, and bucket-wise, which can be used to interpret the model biases, dataset biases, and their correlation.
Formally, we refer to = M1,".. ... M] as a set of models and of attributes. As describe above, the test E could be split into different buckets E ...... ... Eigy based on attribute Pý- We introduce the notion of performance table T E R | which Tiyk represents the performance of i-th model on the k-th sub-test set (bucket) generated by j-th attribute. Next, we will show how these approaches are defined based on T.
P P1," PP
ii
3.3.1 MODEL-WISE
The model-wise analysis aims to investigate how different attributes influence the performance of models with different architectures and pre-trained knowledge. For example, "does the lengths of entities influence the performances LSTM- CRF- NER system?"
Here adopt two types of statistical variables Si,, and S to characterize how the j-th attribute influences the i-th model.
S Spearman(T[(sj :), R,)
(3)
S = Std(T(i,)
where Spearman is function to calculate Spearman's rank correlation coefficient (Mukaka, 2012) and Rg is the rank values of buckets based on j-th attribute. Std(.) is the function to compute standard deviation.
Intuitively, S reflects the degree which the i-th model positively (or negatively) correlates with j-th attribute while SY indicates the degree to which this attribute influences the model.
3.3.2 ATTRIBUTE-WII WISE
The attribute-wise analysis aims to quantify the degree to which each attribute influences the NER task. To achieve this, introduce four measures: task-independent variable &, and task-dependent variables p) /3 and oj based on Eq.3 and Eq.4:
G |N M! Atr(es,)), where N is the number test entities and Atr(e,, represents the value of attribute j for entity ec.
For example, when represents the attribute sentence length, Cy is the average sentence length of the whole dataset.
5

---

Under review as a conference paper at ICLR 2020
p)
M Σ, M1 | N/ E. M. o. y= [M
E!Mis S. where |M is the number of
M evaluated models. Compared with p). 0z can reflect whether the correlation is positive or negative.
Intuitively, higher absolute value of p} or p3 suggests that attribute j is crucial factor, greatly influencing the performance of NER systems.
3.3.3 BUCKET-WISE
The bucket-wise analysis diagnoses the strengths and weaknesses of existing models. Moreover, based on -irib analysis, we could understand how different choices of datasets influence the models' performance.
To this end, we introduce the following measures:
max (Tla, k.) (T,,, Al, for Vk)
(5)
ming (T (.. -
otherwise
Usually where a, represent two different models and usually model a has a higher performance (by dataset-level metric).
Intuitively, a negative value of B, suggests that a worse-ranked model (b) outperform the best-ranked model (a) in some aspect (attribute j); By contrast, positive value shows the largest margin on the attribute j.
EXPERIMENTAL SETTINGS
4.1 MODELS AND ATTRIBUTES
Model Settings To evaluate the importance of different components of the NER systems, varied our models mainly in terms three aspects: different choices of character- (EIMo (Peters et al., 2018), alai (Akbik et al., 2018; Akbik al.)), subword- (BERT (Peters al., 2018; Devlin et al., 2018)), word- (G1 (Ve et al., 201 4)), and sentence-level encoders (LSTM (Hochreiter & Schmidhuber, 1997), CNN (Kalchbrenner al., 2014)) and decoders (MLP CRE (Lample al., 2016; Collobert et al., 2011)). Detailed setting are shown in Tab.3. Totally, we study 12 NER models based on deep neural networks and one traditional method utilizing CRF Lafferty et (2001). The syper-parameter settings of our evaluated models are shown in appendix section.
Character
Word
Seutence Decoder
Overall
Models
nome
cm
elmo
Hair
nome
Istun
CHF
CoNLL WNUT
BC
MZ
WB
Chon off
78.13
17.24
80.36
66.17
73.89
49. 80
WnomelstmCCf
65.01
Coma of
83,80
22.57
83.59
71.57
78.85
52.14
Conn WylovelsmmCrr Orf Comn Wiff Come W WgloveesmeMiip
90.48
40.61
86.78
76.04
85,39
60.17
90.
36.21
86.42
76.74
88.10
49.10
88.05
32.84
84.07
70.00
81.09
56.61
Wnometsmarrf Celm WolfoesumCrff
FT
92.22
45.33
89.35
78.71
85.70
63.26
Cher WnonelstmMip
91.11
42.50
64
$1.03
86.90
66.35
Cliii WnonefemmCr
89.98
41.49
87.98
46
84.
56.71
Clair WelovelstmCCf
93.03
45.96
87.92
77.23
85.56
63.38
Table 3: Neural NER systems with different architectures and pre-trained knowledge studied in this paper. Overall
shows the performances of corresponding systems on different datasets. For the model name, "c" refers to "Character" and "W" refers to "Word". Intuitively, the models are named based on their constituents. For example, Cnon WrandlstmCrff a model without character feature. Its word embedding is randomly initialized, and sentence encoder, as well as the decoder, are LSTM and CRE, respectively.

---

Under review as a conference paper at ICLR 2020
Attributes In evaluation methodology, entity attributes be defined flexibly and attribute values can be continuous or discrete. In this paper, although we investigate 10 types of attributes (or their combinations) listed in Tab.2, others be easily introduced.
4.2 NER DATASETS FOR EVALUATION
We conduct experiments on three benchmark datasets: the CONI.L2003 NER dataset, the WNUTI6 dataset, and Ontonotes 5.0 dataset. The CoNLL2003 NER dataset (Sang & De Meulder, 2003) is based on Reuters data (Collobert et al., 2011). WNUTI6 dataset is provided by the second shared task WNUT-2006. The Ontonotes 5.0 dataset (Weischedel et al., 2013) is collected from newsgroups, broadcast news (BN), broadcast conversation (BC) and weblogs (WB) and magazine genre (MZ).
ANALYSIS
5.1 HOLISTIC ANALYSIS
Before giving fine-graine analysis, we present the results of different models different datasets in the way that traditional multi-dataser evaluation does. As shown Fig. 3, we observe that there is no one-size-fits-al model, and the models with the best results on different datasets are different. Naturally, the following questions are raised: 1) What factors of the datasets can distinguish themselves and influence the NER task? 2) How do these factors influence the choices of models? 3) Does worse-ranked model outperform the best-ranked model in some aspect and how the datasets influence the choices of models? The following analyses will be conducted around these questions.
5.2 ATTRIBUTE- ANALYSIS
Attribute-wise measures enable us to characterize the dataset biases quantitatively. Here we utilize radar chart in Fig. 4 to strikingly display the commonality and speciality between different datasets based on three measures p', defined in Sec. 3.3.2. And we also illustrate measure p² in Fig. 3. Detailed observations are listed as follows:
6.5
Dohts and wXtImIG
OaneNicte-Bi
403
OvtoNite
Category ambiguity and entity length have more consistent influence on NER performance. The common parts the radar chart Fig. 4 (b-c) illustrate that matter which datasets are, the performance of NER task is highly correlated with these attributes: MF-tt (token-tag with MF-buck), MF-et (entity-tag with MF-buck), R-eLen (entity length with R-bucket). Fig. 3 shows the same result. This suggests that the prediction difficulty of named entity commonly influenced by category ambiguity (MF-tt, MF-et), entity length (R-eTen).
TheTbe Rower Red Miieet Milet
Forn
Yook Roslen
Atrnhiates
Figure 3: Illustration of task-dependent mea- sure p².
Occurence and sentence length matters but are minor factors. The outliers in radar chart show the peculiarities of different datasets. Intutively, on attributes: R- RLs,,n, F Fokken, R-oov, the extent to which different datasets are affected varies greatly. Typically, as observed from Fig. 4-(a), a sorted sequence could be obtained according the attribute F-tok: BN WB. The reason why the Spearman correlations p' of BC and WB are smaller is that performance on test entities with higher-freqencyy tokens are lower than entities with twerrffreuency tokens. This suggests that F-tok is not decisive attribute and performance since other crucial factors such as category ambiguity also matter.
WNUT CoNLL > BC >
token can not guarantee a better
Entity density is a swing factor and CoNLL dataset is an outlier. As shown in Fig. 3, the measure p² enables us to know the correlation is positive or negative. We observe that most datasets except CoNLL 2003 have negative Spearman values p² the attribute of R-eDen, which suggests that sentence with more entities is relatively harder to process. We can explain the unusual behavior on CoNLL 2003 from its intrinsic value of eDen, the largest as shown in Fig, 4(a). The dataset of CoNLL contains a lot of short sentences, such as "Chicago 8, 674 48 4, 018 018" and SOF IA 1 1996-12-06" That's why it distinguishes itself from other datasets.
7

---

Under review as a conference paper at ICLR 2020
R-sLan
ResLen
ReDDon
Mi st
R- R-aDen
RoDDen
Affa
M
ont
cLon
eLon
F
oloo
F-tok
F-tuk
F-tok
(a)
(b) P
(c)
Figure 4: Illustration of dataset biases characterized by iask-independent measure ç, task-dependent measures p² and a.
The intrinsic differences in datasets can help us to understand how different datasets influence the different choices of models, which will be explained later (Sec. 5.4).
Spearman
Standard Deviation
R-eDen
R-00v
R-sLen
MF-et
MF-1t
F-ent
F-tok
R-eLen
R-eDen
R-0ov
R-sLen
MF-et
MF-1t
F-ent
F-tok
R-eLen
F-tg
R-mor
Model
F1
CRF++
80.74
60
72
-71
80
80
100
96
-50
7.4
7.7
3.4
12
9.3
9.0
4.8
7.0
6.3
19
Cnon WrandlsttCCf
78.13
67
53
-86
100
100
89
89
-50
7.3
7.0
4.5
15
17
12
9.7
7.8
4.8
12
Conn WnonelstmCrrf
77.01
67
70
-64
60
80
100
82
-50
8.3
9.2
6.0
11
7.1
6.5
2.5
6.3
6.9
16
Conn WranddsttCCrrff
83.80 67
68
-89
100
90
89
61
-50
6.1
5.8
2.6
10
9.8
6.9
3.4
7.3
22
Conn WglovelstmCrrf
90.48
60
40
-75
80
90
71
14
-50
2.9
3.5
1.6
6.7
7.1
3.3
0.6
5.8
5.2
15
Conn WglovecnnCrr
90.14
48
-82
90
77.1
-100
3.4
4.2
1.7
7.1
7.1
3.2
0.9
15
Conn WglovelstmMiIp
88.05
57
42
-71
80
90
96
39
-50
3.2
4.9
2.0
8.2
9.4
3.9
0.9
5.3
6.7
12
Celm WnonelstmCrff
91.64
30
50
90
-50
2.7
2.9
5.5
4.4
2.7
Celm WglovelstmCrff
92.22
45
27
11
80
90
-32
-100
2.2
3.5
0.9
5.1
4.3
2.5
1.0
3.7
5.8
15
Chert WnonelstmMip
91.11
38
5.0
29
80
90
46
14
-50
2.9
3.5
1.6
5.9
4.5
3.2
1.3
2.5
5.6
10
Cita WnonelstmCrff
89.98
19
47
0.0
60
90
100
46
-50
3.1
2.9
1.4
6.2
4.9
3.3
1.2
4.3
6.2
15
Meair W WglovelstmCrf
93.03
26
22
-14
80
90
79
29
-100
2.1
3.2
0.9
4.9
4.0
2.3
0.6
3.5
4.8
11
Table 4: Model-wise measures $1 and S on CoNNL-2003. Pink attributes are used to characterize category ambiguity of entities while blue attributes can measure the degree to which test entities have been seen in training set.
5.3 MODEL-WISE ANALYSIS
Based on two model-wise measures: Sh, and S defined in Eq.3 and Eq.4, we investigate how different attributes influence the performance of the models with different architectures and pre- trained embeddings. Fig. 4 illustrates the case on CoNLL (other datasets are included in appendix), and we have observed that:
1) Char-unaware models are more sensitive to the degree of category ambiguity and occurrence of entities. We observe that "Cnon WrandlstmCrf" negatively related to MF-et, MF-tt, F-ent and F-to' with high values of and suggesting the importance of the character-level encoder, which plays a major role in generalization to rare entities and entities with multiple categories. More importantly, this observation still holds on other datasets. (See appendix)
2) Sentence length a swing factor, whose contribution depends on what types pre-trained embeddings are used. There is strong negative correlation between models with context- independent embeddings and the attribute R-sLen (sentence length). The relationship reversed when contextualized embeddings used. The reason believe is that long sentences could provide sufficient context information for contextualized models. It is noticeable. however that flair-related models "Cflair Wnonelstoonl"""""""" behave differently compared with other contextualized models, which
8

---

Under review as a conference paper at ICLR 2020
will explain later in Sec 5.4 (Flair performs worse than ELMo when dealing with long sentences due to its structural bias).
3) Character encoders favor the sentences with high entity-density. Only using character-level CNN is easy to overfit the feature of capital letters. As a result, more non-entities are mis- dis-predicted entities. Based the understanding of the previous analysis (Sec. 5.2) of the entity density's influence on CoNLL, we could better explain why "Cenn WnonelstmCrf" achieves the highest value of p and in "R-eDen (entity density) attribute in Tab.4.
5.4 BUCKET- WISE ANALYSIS
In this section, choose several typical models (others shown in the appendix) analyzed cases, aiming to seek answers to the following questions: 1) What are the strengths and weakness of different architectural designs? In other words, does worse-ranked model outperform the best-ranked model in some aspect? 2) How do the different choices of datasets influence model performance? Tab. illustrates the bucket-wise measure 3, which is computed based any pair of models M1 and M2. Next, we list some of our observations. Others are illustrated in the appendix.
CNN v.s. LSTM The sentence encoder of CNN is better dealing with short sentences, which holds in all datasets we evaluated in this paper. Strikingly, CNN outperforms LSTM by a large margin (dataset-level F1) on M2 dataset while significantly worse than LSTM on WB (refer to the appendix.). We attempt to explain these discrepancies based on above attribute- -iie metric Ç in Fig. 4(a): sentence length and entity dens ty are two major factors for the choices of CNN and LSTM. CNN is better than LSTM when the dataset with higher value of ÇaLen and CeDen By contrast, when dataset with lower D.Den LSTM is priority (The CeDen WB the lowest).
CRF v.s. MLP The benefits of using CRF on short sentences are very stable, and improvement can be seen in all datasets. Similarly, based on attribute-wise metric C in Fig. 4, we find category ambiguity (MF-et, MF-tt) is major factor for the choices of CRF and MLP: if a dataset with higher SMF-et- in which longer entities can benefit more from CRF-based models. In comparison, introducing CRF will lead to more errors long entities once the dataset (i.e. BN, MZ) has lower FMF-et-
word- level method (i.e., word-level look-up table) to get word representations when we have already used CNN obtain word representations. Here, we show that CennWrand is not always better than CennWnone and entity length matters for the choice. Specifically, CennWnone could achieve better performance on on the WNUT and datasets. With the help of Fig. 4-(a), we find the two datasets share a property of much higher value of ÇR-ete (entity length). Additionally, another commonality between the two datasets can be observed from Tab. 5: the gain of CennWnone mainly comes from the entities with the longer length.
Cenn Wrand Cenn The question has been little studied whether we need
ELMO v.s GloVe ELMo consistently outperforms GloVe on all datasets using the holistic F1 score, but it is worse than GloVe when modeling short sentences. Another interesting finding is that for those sentences containing more OOV tokens, Glo Ve could achieve better performance on all datasets. These phenomena indicate the complementarity between ELMo and GloVe, and our further combination of these two embeddings (CelmWgloveLsttmCC in Tab. 3) indeed works better.
Flair v.s. ELMo While the current state-of-the-ar art NER model Flair has achieved the best perfor- mance in terms of datasct-level Fl score, a worse-ranked model (ELMo) could outperform it in some aspects. Typically, Flair performs worse when dealing with long sentences. The reason be attributed to the its structure design, which adopts a LSTM-based encoder for character language modeling, suffering from long-term dependency problem (especially for character-level language model). A promising improvement is to use the Transformer-based architecture for character language model.
CONCLUSION
To bridge the gap between the insufficient understanding of the nature datasets and model designs, this paper proposes generalized evaluation methodology interpret model biases, dataset biases,

---

Under review as a conference paper at ICLR 2020
200L
WNUT-2016
OntoNote MZ
OnioNote BC
R-DDon 20000
suee Mi-at
F-gnt
MF-4
F-ant
MF
Ponk
RedLen
MF-4
40.61;22
M2:
MI:16.04
10
MI: CrunWggoeeimCCff
Cf
OV
QUA
Overall P1
MI: 90.48, M2 $8.05
MI: 40.61; MZ. 32.84
MI: 85.39, MZ. 81,09
MI: 76.02; MZ: 70.00
of
10
M2: Cown CloeWalooeooonilly
10
Overall
MI: 83.50: 77.01
22.57,
Ml: MD: 79.05
71.57: 65.01
20
MI: Cf
0
M2: ConWWonddemCyf
-10
ADSO Dight
Othnr
PHRC
Overall F1
Ml: 91 64: M2: 90.48
MI: 44.56; M2. 4061
M1: 86 32; M2: 85.39
M1: 77.10: M2: 76.04
MI: Cf
M2: ComWalerelanCd
-10
ORE Uppee
Overall F1
M1: 93.03; M2:
45.96, 45.33
M1: MZ
77.25, 78.71
MI: CliirWiooeeonCCf
M2: CelnWaloeeemCrf
-1
LoWN
Upper
LAWE OfLAr
Table 5: Illustration of the bucket-wise measure 3. Each histogram is obtained based on subtracting the performance of Modell (M1) from Model2 (M2) bucket. For ease of presentation, roughly classify some attribute values into three categories: small(S), middle(M) and large(I.). For example, the first column of the top left histogram represents M2 outperforms MI when the attribute R-eDen takes the small (S) values.
and their correlation, drawing on the complementary strengths of the fine-grained evaluation and multi-dataset evaluation. We choose the NER task as a test case, the observations based on the extensive experiments suggest. directions for improvement and can drive the progress of this area.
REFERENCES
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. Pooled contextualized embeddings for named entity recognition.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. Proceedings of the 27th International Conference on Computational Linguistics, pp. 1649, 2018.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ArXiv e-prints, September 2014.
10

---

Under review as a conference paper at ICLR 2020
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2358-228 2367, Berlin, Germany, August 20.6a. Association for Computational Linguistics. doi: 10. 10.18653/1//111-1222. URL https: /www , a web. org/ anthol ogy/ /P1 61223..
Danqi Chen, Jason Bolton, and Christopher D Manning. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv: 02855, 2016b.
Ronan Collohert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12 (Au2):2493-2537, 2011.
Joke Daems, Lieve Macken, and Sonia Vandepitte. On the origin of emors: A fine-grained analysis of mt and pe errors and their relationship. In IRECC, pp. 62-66, 2014.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Language modeling with longer-term dependency. 2018.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp. Genevieve Gorrell, Raphaël Troney, Johann Petrak, and Kalina Bontcheva. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51 (2):322-44 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Mireia Farrús Cabeceran, Marta Ruiz Costa-Jussà, José Bernardo Mariño Acebal, and José Adrián Rodríguez. Fonollosa. Linguistic-based evaluation criteria to identify statistical machine translation errors. In 14th Annual Conference of the European Association for Machine Translation, pp.
167-173, 2010.
Mark Fishel, Ondřej Bojar, and Maja Popovic. Terra: a collection of translation error-annotated corpora. 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging. arXiv preprint arXiv:1508. 01991, 2015.
Masaaki Ichihara, Kanako Komiya, Tomoya Iwakura, and Maiko Yamazaki. Error analysis of named entity recognition in bcewj. Recall, 61:2641, 2015.
Ann Irvine, John Morgan, Marine Carpuat, Hal Daumé III, and Dragos Munteanu. Measuring machine translation errors in new domains. Transactions of the Association for Computational Linguistics, 1:429-440, 2013.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv: 170700777 2017.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 170.003551 2017.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In Proceedings of ACL, 2014.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint I 2015.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on EMNLP, pp. 1746-1751, 2014.
11

---

Under review as a conference paper at ICLR 2020
Sachin Kumar and Yulia Tsvetkov. Von mises-fisher loss for training sequence sequence models with continuous outputs. arXiv preprint arXiv:1812.04616, 2018.
Jonathan K. Kummerfeld and Dan Klein. Error-driven analysis of challenges in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, PP. 265-277, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: / /wwwwwww..ommmo aclweb........................ , org/ ant wwtthhoooogheo /a1ogy//33111011
Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. Parser showdown the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1048-1059, Jeju Island, Korea, July 2012. Association for 1096.
Computational Linguistics. URL https: /www. ac web. . www.g//nhhoogyy//ooo..o
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields; Probabilistic models for segmenting and labeling sequence data. 2001.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pp. 260-270, 2016.
Quoe Le and Tomas Mikolov. Distributed representations of sentences and documents. In Interna- tional Conference on Machine Learning, pp. 1188-1196, 2014.
Arle Lommel, Aljoscha Burchardt, Maja Popovic, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. Using a new analytic measure for the annotation and analysis of mt errors on real data. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation, pp. 165-172 2014.
Luong, Hicu Pham, and Christopher D Manning. Effective approaches to attention-base neural machine translation. arXiv preprint 1580:5550.04025. 2015.
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional tem-cnns-crf. In Proceedings of the 54th Annual Meeting of ACL., volume 1, pp. 1064 1074, 2016.
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere, Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1896-1906, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/w1/P18- 1176. URL https: // //www. ac web........ www//orgggoooooooo hts//antoologg.. / 8-1776
Mavuto M Mukaka. A guide to appropriate use of correlation coefficient in medical research. Malawi Medicul Journal, 24(3):69-71, 2012.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. arXiv preprint arXiv: 2018.
Graham Neubig, Zi-Yi Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. compare-mt: A tool for holistic comparison of language generation systems. arXiv preprint 11Xiv:1903.07926, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic the 40th Annual Meeting the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115//0730033177333 URL https: / www. aclweb. http//antholooyyP222220
evaluation of machine translation. Proceedings the
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of NAACI, volume 1, pp. 2227-2237, 2018.
12

---

Under review as a conference paper at ICLR 2020
Maja Popović and Hermann Ney. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657-666 688, 2011.
Erik Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-indeeenden named entity recognition. arXiv preprint cs/0306050, 2003.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint 1 1.01603, 2016.
Ilya Sutskever, Oriol Vinyals, and Quoe VV Le. Sequence to sequence learning with neural networks. In Advances NIPS, pp. 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Jakob Parmar, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in NIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNIPP Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for purpose language understanding systems. arXiv preprint arXiv: arXiv:1905.00537, 2019.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. Ontonotes release 5.0 Idc2013t19 Linguistic Data Consortiun, Philadelphia, PA, 2013.
13

---

Under review as a conference paper at ICLR 2020
A HYPER-PARAMETERS
Character- Werd-Level
Sentence-Level and Training
Hyper-parameters
Value
Hyper-parameters
Value
Character emb. size
25
LSTM layer
1
CNN layer
LSTM hidden size
100
CNN kernel size
CNN layer
2
ELMo model
Original
CNN kernel size
3
BERT model
hert-base-casee
Learning rate
0.01
Flair model
forward & backward Learning decay
0.05
GloVe emb, size
100
Batch size
10
Word random emb. size
100
Optimizer
sgd
Table 6: Hyper-parameters for our evaluated models.
B MODEL-WISE ANALYSIS ON THE OTHER DATASETS
Spearman
Standard Deviation
ReeDen
R-00v
R-sLen
MF-et
F-ent
MF-1
F-tok
R-eLen
ReeDen
R-oov
R-sLen
MF-et
F-ent
MF-11
F-tok
R-el.em
F-tag
R-mor
Model
CRF+
-33
-75
82
80
80
100
50
100
6.3
7.4
5.4
16
13
10
2.5
4
13
5.9
Cnon WrandlstmCrf
-80
-61
57
80
100
100
30
100
4.9
6.6
4.1
25
22
19
6.5
7.3
10
3
Cenn WnonelstmCrff
-37 -69
86
80
100
100
20
100
6.2
7.5
6.3
15
9.2
2.2
5.2
6.8
Conn Crff
-48 -68
80
100
30
100
5.6
7.9
3.6
3.3
7.3
5.6
Cenn Wglovelstm Crf
-48 -80
86
100
20
100
60
100
9.5
12
6.2
16
13
22
7.3
14
20
12
Conn W WglovecnnCrf Crf
.43 -86
21
100
40
100
60
50
9.3
11
3.2
16
12
20
7
12
22
10
Cenn W WloovesstmMIp
-62 -81
75
100
20 100
60
-50
9.4
9.8
4.2
20
16
24
6.8
10
9.9
Celm WnonelstmCrff
-52 -70
68
100
20 100
-100
22
7.8
Celm W glovelstmCrff
-47 -79
43
100
20 100
70
-10
12
11
16
13
22
7.2
9.1
20
11
Cherr Ober:WoonelstmMip
-47 -84
79
80
20 100
-00
-50
12
12
5.7
17
14
21
9.4
9.4
22
13
Cflair WnonelstmCrf
-50
-74
39
80
40
100
700
100
12
9.7
7
16
13
21
7.4
8.9
23
12
CRairWloovessmme
-32 -81
86
100
32 100
-70
-100
11
11
8.5
16
8
8.8
21
10
Table 7: Model-wise measures
S and S on Wnut-16.
Spearman
Standard Deviation
R-eDen
R R-oov
R-sLen
MF-et
F-ent
MF-1t
F-tok
R-eLen
ReDDen
R-oov
R-sLen
MF-et
F-ent
MF-11
F-tok
R-elen
F-tag
R R-mor
Model
CRF++
43
40
-18
100
90
98
86
100
2.3
3.8
3.9
13
8.6
13
4.8
6,8
25
7.3
Cnon WrandlstmCCf
-70
20
-21
100
86
98
86
100
2.1
5.7
1.8
18
12
22
7.5
7.0
18
9.0
Cenn WnonelstmCrf
5.0
-40
0.0
80
88
75
100
2.3
3.5
3.5
14
10
14
4.9
8.1
25
7.3
Conn Wrrfd
0.0
100
86
100
2.5
4.3
2.8
8.9
5.8
Conn W WglovelstmCrr Crf
-12 -40
11
100
98
86 100
2.3
2.5
2.7
9.4
6.0
11
1.9
7.6
6.3
Conn W WglovecnnCrf Crf
-27 -40
0.0 100
62
95
54 -00
2.6
2.8
4.6
9.4
6.1
11
2.0
8.2
18
4.9
Conn W glovelstmMiip
-82 40
0.0 100
90
93
-50
2.7
3.5
3.5
12
7.8
13
3.5
5.8
22
7.6
Celm WnonelstmCrff
43 0.0
-11 100
98
-10
1.8
2.9
2.3
6.8
4.7
8.0
2.1
5.5
6.6
Celm glovelstmCrff
-40
100
98 -000
4.3
4.2
7.4
4.6
8.6
2.0
5.4
Cher WnonelstmMIppp
17 -20
7.1 100
29
98 86 100
2.1
3.8
2.6
7.7
4.6
8.0
2.0
3.4
15
4.7
Cilair WnonelsumCrf
-18 40
21
100
76
95
93 100
2.4
3.2
3.0
8.8
5.6
10
2.6
6.8
18
7.1
CflairW ClairWgloveeeee WglooelstmCrffff
-13 -40
21
100
55
95
86
100
1.5
3.5
2.3
8.9
5.2
9.2
2.3
6.3
17
6.0
Table 8: Model-wise measures S., and S on OntoNote-BN.
C BUCKET-WISE ANALYSIS
14

---

Under review as a conference paper at ICLR 2020
Spearman
Standard Deviation
R-eDen
R-oov
R-sLen
MF-et.
F-ent
MF-1t
F-tok
R-cLen
R-eDen
R R
R-sLen
MF-et
F-ent
MF-1t
F-iok
ReeLen
F-tag
R R-mor
Model
CRF++
-30
14
-46
100
95
29
100
2.6
4,9
3.9
19
13
18
5.7
8.3
26
14
Chon W Wra
-55
1.3
-57
80
43
93
67
100
3.5
5
2.5
21
14
23
8.6
7.2
25
17
Conn WnonelstmCrf
-52
37
-71
80
89
98
57
-100
3.6
5.5
3.7
18
13
19
5.9
8.3
26
12
Conn WrandlstmCrf
-63
98
52 -100
5.4
4.6
18
5.8
Conn WglovelstmCrff Crf
.48
52
-89
100
98
00000
3.7
5.6
5
13
9.1
16
5.1
8.8
24
11
Conn W Wglovecnn
-8.3
39
-96
80
64
93
12 -00
2.9
4.5
4.9
12
9.4
16
5.6
8.6
25
11
Conn W WglovelstmMip
-35
43
-96
100
50
100
-7.1 -50
4.3
5.7
5.8
15
12
19
8.4
7.7
24
13
Celm WnonelstmCrf
67
64
-96
60
46
98
12
-10
1.7
5.6
4.5
15
9.2
14
4.3
6.6
18
Celm WglovelstmCrf
48
-89
98
-38
-100
1.7
4.7
3.4
13
8.3
13
5.3
6.3
17
9.3
Chert WnonelstmMiPp
23
32
-82
80
43
98
77.1 50
2.2
3.7
3.1
13
7.9
13
5
4.3
19
9
ShuiiWnonelstmCr
-85
33
-89
80
54
100
12 100
2.2
4.3
1.2
11
8.7
13
5.2
6.9
22
6
Cftair MlaiirWgooelssmCrr
-33
22
-43
80
64
88
19
100
2.2
4.2
2.7
12
9.7
15
5.5
7
22
9.1
Table 9: Model-wise measures Si and S on OntoNote-BC.
Spearman
Standard Deviation
R-eDen
R-00v
R-sLen
MF-et
F-ent
MF-I
F-tok
R-eLen
R-eDen
R-oov
R-sLen
MF-et
F-ent
MF-tt
F-ook
R-eLen
F-tag
R-mor
Model
CRF++
80
100
93
64
-100
6.2
5.5
10
7.9
Cnon WrandIsttmrff
11
100
88
93
4.7
21
12
25
11
28
9.1
Conn Wnonelstm(rf
-10
-32
21
80
77
93
64
550
5.1
8.1
5.6
13
7.5
16
6.8
6.3
30
5
Conn WrandlstmCrf
-53
-43
71
80
94
90
74
-00
4.3
8.3
5.5
13
9.2
15
8
6.8
30
4.9
Conn WfloveessmCeff
-60
0
64
80
95
-100
4.6
5.1
2.7
8.8
5.8
15
4.3
7.6
28
8.6
Conn WgloveennCry Crf
-38
100
90
-100
4.2
5.4
3
7.9
5.2
6.4
27
Conn W WglovelstmMIp
-43
-45
64
100
60
95
64
-100
4.9
7.7
2.9
13
7.3
13
5.2
2.5
29
5.4
Celm WnonelstmCrf
-17
-37
57
100
94
95
71
100
4.6
6.5
2.4
9.6
5.6
12
4.6
4.6
28
5.7
Celm W glovelssmCCff
-37 -50
11
100
100
95 75
100
5.1
7.3
5.4
10
5.7
13
4.1
4.9
26
6.6
Chert WnonelstmMip
-4.5
21
80
79
50
4.5
5.6
3.6
4.6
Cflair WnonelstmCrf
15
80
100
90
71 100
4
7
5.1
13
5.6
3.6
26
5.5
Cllair W gglovelstmCrrr
-6.7
-37
61
80
100
86
71 100
4.4
6.8
3.9
12
6.1
12
5.6
4.4
26
5.2
Table 10: Model-wise measures Sh,, and S on OntoNote-MIZ.
Spearman
Standard Deviation
ReeDen
R-oov
R-sLen
MF-et
F-ent
MF-1
F-tok
R-eLen
ReeDen
R-oov
R-sLen
MF
F-ent
MF-1
F-tok
R-eLen
F-tag
R-mor
Model
CRF++
-20
43
-21
80
50
90
8.6
-100
8.1
11
9.3
22
22
22
12
5
20
8.2
Cnon WrandlstmCrf
-55
54
80
54
-26
-10
13
11
9.5
27
21
27
16
12
8.7
Cena WnonelstmCrf
-4.8
229
11
20
20
9.2
8.5
Conn WrandlstmCrf
-33
7.1
3.6
80
61
93
-37
-100
11
12
4.6
24
20
27
15
13
23
8.7
Cenn W glovelstmCrff Crf
-32
40
-86
80
50
98
-37
-100
11
11
5.6
18
14
22
12
9.9
23
9.9
Conn W gloveennCrf
-32
-2.4
-63
80
54
81
-43
-50
11
13
5.1
22
18
26
17
13
24
13
Cenn W MlovelstmMMI
-40
33
-71
80
68
-14
-50
11
9.2
5.2
21
26
16
Celm W WnonelstmCr
-37
29
-39
80
98
-43
-50
11
9.9
5.8
17
14
23
10
8.4
24
9.2
Celm W WglovelstmCif
-27
45
-14
80
71
98
-26
-50
9.7
10
3.1
17
13
21
11
8.8
26
9.5
Cherr WnonelstmMip
-25
2.4
-54
80
71
98
2.9
-50
8.7
5.1
5.7
16
13
20
12
7.1
28
10
Cflair WnonelstmCrf
-30
24
-79
80
71
95
-26
-50
10
11
18
16
21
10
8.9
24
8.9
Cflair WglovelstmCrff
-28
50
8.6
-50
10
16
9.2
Table 11: Model-wise measures S and S7 on OntoNote-Wi WB.
15

---

Under review as a conference paper at ICLR 2020
CONLL-2003
WNNU-2006
OmtoNote-MZ
OunoNot--BC
ReDDn Rows
Fee
moor
ReDDon
Reown
RelLan
Mi--0
Feo
M
Fook
ReeLan
Plg
ReDDen
Rowss
Romoo
RelDenn Recow
ReaLan Mi-ol
Fenn Mi
Overall F1
M1: 92,22; M2: 91.64
MI: 45.33; M2: 44.56
M1: 85 M2: 86.32
Mi: 78:78:1: 77.10
MI: ColnWgloneeenCof
M2: CelnWoondemmCC
10
MISC
DigF
LAN
Overall F1
92.22; M2: 91.11
45.33 42.00
85.70, MZ: 86-90
78.71: 81.0
MI: CrlmWgloneemnCd
-10
M2: ChertWnoneeimkkpp
OREC Digi
MOON Looer
Table 12: A supplement bucket-wing wise analysis results to Tab. 5. Each histogram is obtained based on subtracting the performance of Modell (M1) from Model2 (M2) bucket. For ease of presentation, we roughly classify some attribute values into three categories: small(S), middle(M) and large(L.). For example, the first column of the top left histogram represents M2 outperforms MI when the attribute R-eDen takes the small (S) values.
16

---

Under review as a conference paper at ICLR 2020
ReDhon
Mi-et
F-ont
MF-11
F-ook
F-ig R-moo
R-Donn
et
Flent
R-mor
ReDoon Roow
Milet Foon MF-al F-ok Rel.n F-ig R-mor
R-Dmn
Mi-et
Rmnor
OntaNote: BN
20
PIO
Tale
Overall F!
M1 96.8% M2:86.42
M1:86 78, M284407
MI 86.88 M2:83.59
M1:83 59, M2.80.36
Uppnr
TIME Diel
020 Other
M2156.11
M2152.14)
Co M2: Conn W gloveen
ComWaalovinnCCf
M2: Cow
M2 Cran CondlewCy
M2: ComWWrondemm
OntoNote-BN
Overall F1
M1:39.75: M2:86.78
M1:87.92 M2.8935
M1:89.25 35: M2:89.75
M1:89.35: M2.89.5
OntoNoto-WB
-10
-00
ORD Digli
Overall
60.54;
M1.63 M2:63.26
M1 36. M2:60.54
M1.63 M2.66.35
MI: Celm Cf
MI: gonnWWWloeettCC
MI: Celm WoloeeeeeCCC
MI: CeleeWalon cletmet
CoowWlooddnwCrr
Coloe of
ChertWannddsmMii
Table 13:
Illustaation of the bucket-wise measure on OntoNote-BN and OntoNote- WB. Each histogram is obtained based on subtracting the performance of Modell (MI) from Model2 (M2) on bucket. For ease of presentation, we roughly classify some attribute values into three categories: small(S), middle(M) and large(L...
17


Reviews:
[{"review_id": "rkxmuiLnYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe manuscript proposes an evaluation methodology to obtain deeper insights regarding the strength and weaknesses of different methods on different datasets. The method considers a set of methods addressing the task of Named Entity Recognition (NER) as case study. In addition, it proposes a set of attribute-based criteria, i.e. bucketization strategies, under which the dataset can be divided and analyzed in order to highlight different properties of the evaluated methods.\n\nAs said earlier, the manuscript proposes an evaluation methodology to obtain deeper insights regarding the strength and weaknesses of different methods on different datasets. The characteristic of being able to provided deeper insights on strength/weaknesses and relevant factors on the inner-workings of a given method is \nsomething very desirable for every evaluation. As such, in my opinion, the \"interpretable\" tag associate to the proposed method is somewhat out of place. Having said that, I would recommend removing the \"interpretable\" tag and stress the contribution of this manuscript as an evaluation protocol. \n\nIn Section 4.2, for the R-Bucket strategy it is stated as having the requirement of discrete and finite attributes. Based on the equations of the other two strategies (R-bucket and F-bucket), it seems that they also have the requirement of having discrete attributes. Is this indeed the case? if so, it should be explicitly indicated. \nHaving said that, this raises another question: Is this protocol exclusive to tasks/problems with explicit discrete attributes?\n\nThe goal of this manuscript is to propose a general evaluation protocol for NLP tasks.\nHowever, it seems to be somewhat tailored to the NER task. My question is: How well the proposed method generalizes to other NLP tasks without attributes? Similarly, how well the proposed bucketization strategies generalize beyond the NER task? Perhaps the generalization characteristics and limitations of the proposed evaluation methodology should be explicitly discussed in the manuscript.\n\nLast paragraph of Section 4.2 summarizes ideas that were just presented. It feels somewhat redundant. I suggest removing in in favor of extending the existing discussions and analysis.\n\nI may consider upgrading my initial rating based on on the feedback given to my questions/doubts.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skgm6QnaYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "TOWARDS INTERPRETABLE EVALUATIONS A CASE STUDY OF NAMED ENTITY RECOGNITION\n\n\n\nThe authors propose an evaluation methodology to study the relations between datasets and machine learning models. This methodology introduces the notion of attributes which describes different aspects of the samples and buckets which group samples according to the attributes. The goal is to give a better understanding of the strengths and weaknesses of an algorithm on a specific dataset according to the attributes, as shown on Fig4.\n\nThe article is very dense and the author chose to present the method from an abstract and generic point of view which makes the reading of the article difficult. In the end, the proposition is a formalisation of the simple error analysis which is commonly done when trying to improve a machine learning system. The advantage of the method could be to introduce some metrics to make the error analysis more automatic. These metrics are given in section 4 but here again only from a formal point of view : it is very difficult for the reader to understand how to interpret them and how to use them for a practical case. \n\nThe paper is 17 pages long with the annex : it would better fit a journal publication or the author should select some of the main results to present them in a conference paper. The aspects of the paper related to learning relations is no put forward enough. \n\n2. Related work\n\n2.1 :\n -supplementary exam : unclear\n\n2.2 : \n- methodological perspective : a bit a repetition of introduction\n- task perspective : not very clear, is the main message  \"it important to understand what in the dataset make the model work ?\"\n\n3 Task\n\nSection is too small to be a level 1 title\n\n4 Attributes\n\nfigure 2 : where are the links to levels ?\n\n4.2 :\n\n  * familiarity : test/train distribution should be the same. Fk computer on train set because it is bigger ? it allows to study the impact of the number of occurrences in the training set. Is it more interesting than a learning curve ?\n\n  * multi attribute familiarity : risk of metric explosion ? how to select the attributes ?\n\n  * eq 3 : spearman not defined\n\n* 4.3\n\n  * metric are defined by formula but it is difficult to understand what is the rationale behind each of them and therefore figure out how to interpret them\n\n  * \"Usually where a, b represent two different models and usually model a has a higher performance (by dataset-level metric)\" : unclear\n\n5 Experimental setting\n\nTable3 : \nthe encoding of the model name is not clear\na metric on all the dataset for each model could be computed to decide which one is the best overall\nhow did you choose the tested combinaisons ?\n\n6\\.2\n\nanalysis of Fig 4 : R-eLen does not existe (R-Ele). what is eta ?\n\ntable 4 : spearman\\**r*\\* ?\n\n6\\.4\n\n* CRF vs MLP : \"... a major factor for the choices of CRF and MLP: **if** a dataset with higher ζMF−et, in which longer entities can benefit more from CRF-based models.\" > missing words ?\n\nWriting :\n\n* \"Concretely\" isn't very natural at the beginning of sentences, same thing with \"Formally\", 'Intuitively' …\n\n* in 4.1 : \"We refer to E, P, K as the sets of entities (i.e. New York), entity attributes (i.e. entity length) and attributes values (i.e. 2).\" => \"We refer to the sets of entities (i.e. New York) as E, entity attributes (i.e. entity length) as P and attributes values (i.e. 2) as K\" would be better\n\n* same thing in 4.3 \"we refer to M = m1,··· ,m|M| as a set of **models** and P = p1,··· ,p|P| as a set of **attributes**\" doesn't really work, \"M = m1,··· ,m|M| is a set of **models** and P = p1,··· ,p|P| is a set of **attributes**\" maybe\n\n* in 4.2 page 5 : \"the familiarity Fk (p1 , p2 ) is a measure with intriguing explanation …\" : not clear\n\n* 6.3 (3) \"Only using character-level CNN is apt to overfit the feature of capital letters.\" **apt** doesnt work here", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByljmRZSqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses a methodology to interpret models and model outputs for Named Entity Recognition (NER) based on assigned attributes. The key idea is to bucketize the test data based on characteristics of attributes and then comment on effect of the attribute on the model, the task itself or the dataset bias. \n\nThe empirical evaluation is impressive. The authors have constructed a series of experiments to make their case. The paper is well-written and easy to understand, albeit some of the related work seems a little unrelated to the task at hand. While the authors have tried to state that the method is \"general\" and goes beyond NER, I am not sure if that is the case. The creation of attribute buckets is vital for any further analysis, its not clear how the method can be adapted to more general settings unless such attributes and buckets can be created easily (e.g. using domain knowledge). Furthermore, there is only one problem setting considered (i.e. NER), and for the paper is make claim to more general settings, I would expect evaluations on atleast one more problem setting. I would suggest the authors modify the claims accordingly. This is not to diminish from their contributions in the NER. \n\nThe bucketization idea is not something out of the park novel. It is probably something already being used in practice. However, delineating the procedure and suggesting quantifiable statistics and designing experiments to illustrate how these can be used to draw qualitative conclusions is something that is very  interesting and useful to the community as a whole. The strongest part of this paper is the empirical evaluation that allows drawing interesting conclusions, and suggests a methodology to reach that conclusion. While some of the claims made (e.g. regarding dataset biases) probably require further and deeper analysis, this is a good first step that should foster further research and discussion. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper diligently setup and conducted multiple experiments to validate their approach - bucketizating attributions of data and analyze them accordingly to discover deeper insights eg biases. However, reviewers pointed out that such bucketing is tailored to tasks where attributions are easily observed, such as the one of the focus in this paper -NER. While manuscript proposes this approach as ‘general’, reviewers failed to seem this point. Another reviewer recommended this manuscript to become a journal item rather than conference, due to the length of the page in appendix (17). There were some confusions around writings as well, pointed out by some reviewers. We highly recommend authors to carefully reflect on reviewers both pros and cons of the paper to improve the paper for your future submission. 

Decision: Reject