Towards Building More Robust NER datasets: An Empirical Study on NER Dataset Bias from a Dataset Difficulty View
Authors: Ruotian Ma
Xiaolei Wang
Xin Zhou
Qi Zhang
Xuanjing Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Robustness
OOD Generalization
Dataset Bias
NER

Abstract: Recently, many studies have illustrated the robustness problem of Named Entity Recognition (NER) systems: the NER models often rely on superficial entity patterns for predictions, without considering evidence from the context. Consequently, even state-of-the-art NER models generalize poorly to out-of-domain scenarios when out-of-distribution (OOD) entity patterns are introduced. Previous research attributes the robustness problem to the existence of NER dataset bias, where simpler and regular entity patterns induce shortcut learning. In this work, we bring new insights into this problem by comprehensively investigating the NER dataset bias from a dataset difficulty view. We quantify the entity-context difficulty distribution in existing datasets and explain their relationship with model robustness. Based on our findings, we explore three potential ways to de-bias the NER datasets by altering entity-context distribution, and we validate the feasibility with intensive experiments. Finally, we show that the de-biased datasets can transfer to different models and even benefit existing model-based robustness-improving methods, indicating that building more robust datasets is fundamental for building more robust NER systems.

Towards Building More Robust NER Datasets:
An Empirical Study on NER Dataset Bias from a Dataset Difficulty View
Ruotian Ma Xiaolei Wang", Xin Zhou', Qi Zhang', Xuanjing Huang!
School of Computer Science, Fudan University, Shanghai, China International Human Phenome Institutes, Shanghai, China (rtma 19mma1, 19, qz @fuddn. eud.. cn, xlwang22@m. cornn.udaa fudan. edu. cn
Abstract
Gold label: ORG
Predicted (Ouboo-vocabulrryy unseen entityy
Recently, many studies have illustrated the robustness problem of Named Entity Recog- nition (NER) systems: the NER models often rely on superficial entity patterns for predictions, without considering evidence from the context. Consequently, even state-of-the- art NER models generalize poorly to out- of-domain scenarios when out-of-distributiinn (OOD) entity pattems are introduced. Previous research attributes the robustness problem to the existence of NER dataset bias, where simpler and regular entity patterns induce shortcut learning. In this work, we bring new insights into this problem by comprehensively investigating the NER dataset bias from dataset difficulty view. We quantify the entity-context difficulty distribution in existing datasets and explain their relationship with findings.
Ms. Hall expects Cathay 's profii to grow around 13 % annually this year.
Gold label: PER
(Ambiguous entity
Predicted
President Bush told Mr.
Apple this week interview. "....
Figure 1:
Examples of two typical failures of NER models, copied from the OntoNotes dataset.
as CoNLLO3 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes 5.0 (Weischedel al., 2013). Despite the success, recent research has demon- strated the robustness problem in NER: the state- of-the-art NER systems often rely on superficial entity patterns for predictions, while disregarding evidence from the context. Consequently, the models show poor generalization ability in out- of-domain (OOD) scenarios where out-of-domain entity patterns are introduced (Lin al., 2020; Ghaddar et al., 2021; Wang al., 2022). For instance, Figure shows two typical OOD failures of SOTA NER models: (1) Instances with out-of- vocabulary entities (upper). The entity "Cathay" is unseen in the training data, however, the model fails to deduce correctly given obvious evidence "LJ's profit to grow" in the context. (2) Instances with ambiguous entities (bottom). Fu et al. (2020) shows when an entity is labeled differently across domains, the model fails to recognize well even with supportive context, such as mistaking the person name "Apple" in the example.
model robustness. Based we explore three potential ways to de-bias the NER datasets by altering entity-context distribution, and we validate the feasibility with intensive experiments. Finally, we show that the de-biased datasets transfer to different models and even benefit existing model-based robustness-improving methods, indicating that building more robust datasets is fundamental for building more robust NER systems.
1 Introduction
Named Entity Recognition (NER), aiming to recognize named entities from unstructured data, is widely studied by researchers as a fundamental task Natural Language Processing (NLP) and a crucial task in practical applications (Lample et al., 2016; Chiu and Nichols, 2016; Li et al., 2020). Recently, the advances in pre-trained language models (Devlin et al., 2019; Lewis et al., 2020; Liu al., 2019) have contributed to promising performance on standard NER benchmarks, such
Previous works have delved into the robustness problem of NER from different perspectives. (Agarwal et al., 2020; Ghaddar et al., 2021; Kim and Kang, 2022; Lin et al., 2020; Wang al., 2021) examine the NER models by constructing challenging test sets. Their experiments reveal that NER models fail to "lear from context information" during training. Agarwal et al. (2020) and Kim and Kang (2022) evaluate models
"Equal contribution. Conresponding authors.

---

on controlled entity sets and consider the poor robustness is due to the model's tendency to memorize entity patterns. Lin et al. (2020) further designs a series of randomization tests and demonstrates that the strong name regularity and high mention coverage rate in existing NER datasets might hinder model robustness. All these studies indicate that the poor robustness of NER models might be due
robust datasets is always fundamental for building more robust NER systems.
We hope our study can bring new insights into building more robust NER datasets, as well as developing more robust and general NER systems for real-world scenarios'.
Measuring the Difficulty of Entity and Context in NER datasets
hidden dataset bias of existing NER datasets: the entity pattern is "easier" for the model to learn, so the models are biased to learn the shortcut in entity names while paying less attention to the context.
2.1 Background of V-information
Recently, Xu et al. (2019) extends the mutual information Shannon (1948) to a concept of V- usable information under computational con- straints, which measures how much information about Y can be extracted from X with certain model family V. As defined in Xu et al. (2019):
In this work, we systematically locate the origin of NER dataset bias and investigate its effect on model robustness from dataset difficulty view. We try to answer two questions:
Definition 1. Let X, Y be two random variables laking values in X x V. Let V be predictive family that V - = {f X' {2) - P())}} The V-usable information is:2
Q1: How does the entity-context distribution in the training data induce dataset bias? To answer this question, we borrow recent concept "V-information" (Xu et al., 2019; Ethayarajh et al., 2022) to measure the difficulty of entity and context for the model learn in each dataset. We find that (1) In all NER datasets we examine, the y-information of the entity is obviously larger than that of the context, indicating the dataset distribution induces models to learn more entity than context. (2) We further design an instance- level metric to measure the difficulty of entity and context in every single instance. We find that the largest population of instances-with equality- informative entity and context-does not lead or even harm the models to learn context.
Iv(X Y) = -Hv(Y Hv(X
(I)
where
Hy(Y| = inf El log f( JEV
Hv(YIX) inf E log, [X)
JEV
More intuitively, V can be a pre-trained model family like BERT. HY can be computed with BERT model fine-tuned with null input 2 and Y, and Hy(Y)X) can be computed with a BERT model fine-tuned with (X
Ethayarajh et al. (2022) further extends V- information to measure dataset difficulty. Intu- itively, a higher Tv(X - Y) means V is able to extract more usable information from X about Y, thus indicating an easier dataset for V to learn. They also propose to compare different attributes of X by computing (x() + Y), where 7(-) is a transformation on X isolate an attribute a. For instance, we can transform the regular NLI (Bowman et al., 2015) inputs into hypothesis- only inputs to measure the V-information of the hypothesis attribute.
Q2: Based on the analysis in QI, are we able to build more robust NER datasets by altering the entity-context distribution in existing data? Based on Q1, we consider three potential ways to de-bias the NER datasets: (1) Reducing the overall V-information of the entity in the training data. (2) Enhancing the overall V V-information of the context the training data. (3) Enlarging the proportion of the hoblsul instances, i.e., instances with contexts easier than entities. By conducting extensive experiments, we verify the feasibility of all three approaches. These results also in turn confirm our analysis in Q1. Furthermore, validate the transferability of the model-specific constructed datasets to improve the robustness of other models. These de-biased data are even helpful for existing model-based robustness- improving strategies, showing that building more
Ethayarajh et al. (2022) also propose new measure based V-information to measure pointwise difficulty, which refers pointwise v- information (PVI):
I 'Our code is available at htttps:///ithule cot/ tmaww/ NERDataBias "We use log, to measure the entropies in hits of information following (Ethayarajh ct al., 2022).

---

y
y LOC? ORG?
model as Mg. For each instance (ar, y). we denote
the context PVI (measured by MMc) as PVIc(x). and the entity PVI (measured by ME) as PVIg(s). The CEIM is then calculated by:
Pre-trai aod mood
The tranned model
[MASK]
good person.
Rachel
Contoxt transfrmmtion
Endiy nome transfommtion
Rachel
good person.
CHIM (VIg(x) PVIc(a)
(4)
Figure 2: Illustaation of our method to decouple contexts and entity names. We respectively train context-only model and an entity-only model to examine the usable- information of each attribute.
Intuitively, high CEIM means the entity name in the instance is much easier to learn than the context.
3 How Does the Entity-context Distribution Induce Dataset Bias?
Definition 2. Let X, Y be two random variables and V be a predictive family, the pointwise V-
In order to answer Q1, in this section, we decouple the context and entity in NER datasets and calculate the V-information of each part, so as to obtain the context-entity difficulty distribution. Based on the results, we analyze the correlation between a context-entity distribution and dataset bias in NER datasets at both the whole-data level (Sec.3.2) and the instance level (Sec.3.3).
information (PVI) of an instance (x. y) is
PVI( - y) = log, (1) + log, 9 (x))
(3)
where g.g E V.
BERT model fine-tuned with (X,Y) and BERT model fine-tuned with (0,, Y) to calculate the PVI of every single instance, and higher PVI indicates the instance easier for V. We adopt PVI to measure instance difficulty.
Similarly, we
3.1 Experiment Setup
Dataset To obtain comprehensive knowledge of the entity-context distribution of different NER datasets, we first conduct experiments on commonly used datascts to calculate the V- information of each dataset: CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003), OntoNotes 5.0 (Weischedel et al., 2013), Bio-NER (Collier and Kim, 2004), ACE 2005 (Walker et al., 2006). Twitter (Zhang al., 2018) and WNUT 2017 (Derczynski et al., 2017).
2.2 Decoupling Context and Entity
Motivated by Ethayarajh et al. (2022), in this work, we propose to decouple the entity and context attributes in order to measure the difficulty of each part respectively. Specifically, we first transform the NER dataset into two separate entlty-only and context-only datasets. As shown in F2g.2, build the context-only dataset, we replace the entity with a "[MASKJ". To build the entity-only dataset, we simply use the entity as input. Then, we respectively train a context-only and an entity-only classification model based on a pre-trained model family (such as BERT) to predict the entity type based the inputs. Based the trained models, we can thus calculate the respective PVI of context and entity in each instance, as well as calculate the V-information of context and entity of the whole training data, which indicates the difficulty of context and entity in this dataset to the used pre- trained model. More implementation details are included in Appendix A.3.
In Section 3.3, we include experiments to further explore the correlation between the entity-context distribution and model robustness. Following previous works on NER robustness (Lin et al., 2020; Wang et al., 2022), we experiment on CONLL 2003 and ACE 2005 datasets. Except for evaluating standard performance the i.idd. test set (denoted Test), adopt two robustness test sets (Wang al., 2021) for each dataset, the OOV and CrossCategory test sets, as the measure of model robustness, also following (Wang et al., 2022). These two robustness test set corresponds to the two typical failures of NER models as described in Fig.1, respectively. We include more dataset details and example cases of these test sets in the Appendix A.2.
2.3 Context-entity Information Margin (CEIM)
In order better describe the difficulty discrepancy of context and entity in an instance, we further introduce a new measure: Context-entity Informa- tion Margin (CEIM). Formally, we refer to the context-only model as Mc. and the entity-only
Base Model We conducted all experiments in this section based on the BERR-baaee-cssed pre- trained model (Devlin et al., 2019). More imple- mentation details are included in Appendix A.3.

---

3.2 V-information Comparison between Context and Entity
Coried DV. Fafts
175
In Figure 3, show the isolated V-information of context and entity in common NER datasets. From the results, we can observe that (1) most the commonly-used datasets, the V-information of the entity is larger than 1.0. Such high values indicate the pre-trained model is able to learn and correctly classify most entities in the datasets without any information from the context. (2) In all datasets, the V-information of the entity is obviously higher than that of context, meaning that the entity much easier for the pre-trained model to learn than the context. Such V-information discrepancy in existing datasets means that the difficulty distribution of existing datasets induces the model to learn more from entities instead of contexts. Corresponding to previous studies (Agarwal et al., 2020; Kim and Kang, 2022; Lin al., 2020), this is intrinsic bias in NER datasets that harms model robustness.
Werbomatte
2.29
0.04
9.66
Dallaaeta
Figure The V-information of context and entity in 6
commonly used NER datasets.
contexts and entities. The low-CEIN instances consist of high-PVI context and low-PVI entities, where the entities are often rare, more complicated, or ambiguous. We include more discussion on how different CEIM instances might affect model prediction in Appendix A.I.
Low-CEIM
NZ-CEIM
High-CRIM
# mam
A.PVIg
A mmm
A-PVI
A.PVIc
CoNLL 2003
1962
-1.01
18266
1.90
1.95
OntoNNttes
-1.55
2.20
3.3 Understanding Entity-context Distribution with CEIM
ACE2005
1.99
1936
1.38
As V-information is the difficulty measure of the whole dataset, step further to understand the instance-level difficulty of context and entity with Context-entity Information Margin (CEIM) (Section 2.3). We calculate the CEIM of instances in each dataset and divide the instances into categories: (1) High-CEIM: instances with high CEIM, i.e., the entity is easier to learn than the context; (2) Low-CEIM: instances with low CEIM, i.c., the context is casier to learn; (3) NZ-CEIM: instances with near-zero CEIM (CEIM(2)) 0.5), i.e. the context and entity is equally-easy to learn.
Table 1: The number of instances with different CEIM levels in each dataset. A-PVIg denotes the Average- PVI entity names in different parts of data.
How do different CEIM instances affect model robustness? To further understand the impact of entity- entity-context distribution on the model robustness, we further conducted an experiment to compare the model behavior when trained on data with different CEIMs. We first randomly sampled 1000 instances from the whole dataset as baseline training set, denoted as Base. Next, we randomly sampled 1000 near-zero- CEM instances and add them to Base construct a new 2000-instance training set, denoted as Base+NZ. Similarly, we constructed another two training sets Base+High and Base+Low by adding 1000 sandomly-sampled high-CEIM and low-CEIM instances to Base, respectively. We then train a model on each training set and evaluate their performance on the test set and the robustness OOV and CrossCategory test set.
Table 1 shows the distribution of different CEIM instances and the average PVIg in each part. It's shown that the Bear-zero--EIM instances cover the largest proportion in all datasets, and next is the high-CEIM instances. Also, the Average-PVIg shows that both the near-zero-CEIM and high- CEIM instances mainly contain high-PVI (easy- to-learn) entities.
In Table 2, show cases of different CEIM instances. Generally speaking, the high- CEIM instances often contain low-PVI contexts that are ambiguous or misleading, while contain- ing informative entities. The near-zero-CEIM instances often consist of equally informative
Table 3 shows the effect of different CEIM data on the model performance. We can see that: (1) By adding 1000 extra instances to Base, the performance on the test set is improved on all training sets. Among all training sets, Base+High

---

Type
Context
Entity name
Labe
PVle
PVls
CELM
Tlgh
composed
[MASK]. United Nations.
European Union
2.04
6.77
CEIM
[MASK] is playing trick on us he says.
America
GPE
-5.25
1,78
7.03
Near-zero
And equally without doubt, [MASK] did nox want him, that Bush, to this,
Blair
PERSON
1.87
1.84
-0.03
CEIM
China and [MAS]] have become important mutual trade purtners.
Russia
GPE
1.82
1,85
0.03
Low
On Thesday; [MASK] directors announeed plans spin off rwo hig divisions...
Trellchorg
ORG
2.19
-0.12
-2.07
Mr.MAAK] the other night ADC's Nightline."
Apple
PERSON
-6.62
-8.25
Table 2: Examples of different CEIM instances. More detailed discussion can be found in Appendix A.I.
CoNLL 2003
4.1 Experiment Setup
Training set
Test
0OV
CrossCategory
Datasets Similar to Section 3, in this section, we conduct experiments on the CoNLL2003 and ACE2005 datasets, and evaluate the i.i.d. test performance on the Test set, as well as the robustness performance on the OOV and CrossCategory test sets (denoted as "Cate.").
Base
87.15
64.28
40.74
Base High
87.61 (40.46)
63.77 (-0.51)
40,91 (40.17)
Base
87.51 (+0.36)
64.05 (-0,23)
41.55 (+0.81)
Base Low
87.50 (+0.35)
67.96 (+3.68)
44,93 (+4.19)
ACE 2045
Training set
Text
0OV
CrusCategory
Base
81,44
73.91
39.04
Base High
83.56 (+2.12)
73.64 (-0.27)
40.80 (+1.76)
Base Model and Baselines In this section, we conduct all experiments based on two pre-trained LM, BERT-base-cased (Devlin et al., 2019) and RoBERTa- (Liu et al., 2019), To better verify the robustness improvement of the three de-bias methods, we include several pooustness-improinngg baselines: (1) Base (Devlin et al., 2019; Liu et al., 2019) The base token classification model based on BERT-base-casee and RoBERTa-large trained on the original training sets; (2) DataAug (Dai and Adel, 2020), which augments the training set by replacing entities with similar entities or typos entities; (3) MINER (Wang et al., 2022), which also creates samples with entity switching and trained with contrastive bobustness-improving loss. is also the SOTA method in NER robustness; (4) LPFT (Kumar et al., 2022), general OOD method that can effectively improve OOD model generalization. More implementation details can be found in Appendix A.4.
Base NZ
84.74 (43,30)
74,54 (40.63)
41.99 (+2,95)
Base
Low
83.36 (42.13)
76.56 (42.65)
42.86 (+3.82)
Table The effect different CEIM data on model generalization and robustness.
and Ba show slightly larger improvement than Base+Low, which corresponds to the findings in previous studies (Lin et al., 2020; Agarwal et al.. 2020; Zeng et al., 2020) that high-PVI entity names contribute more to the generalization on the i.i.d. set. (2) Base+Low shows notable improvement on both OOV and CrossCategory test sets, while Basc+High and Basc+NZ are less beneficial or even harmful to the robustness performance. These results demonstrate that the low-CEIM instances, i.e., instances with contexts easier than entities, contribute most to the model robustness. (3) The limited robustness performance of Base+NZ also indicates that although the context may be informative enough for label predicting. with equally easy entity and context, the pre-trained model still tends to make use of the entity information instead of context. Unfortunately, the near-zero-CEIN data constitutes the largest proportion in all datasets (Tab. thus having much larger effects the model training than the low- CEIM data (which largely improves robustness).
4.2 Enlarging the Low-CEIM Proportion
The experiments in Section 3.3 have shown that the high-CEIM data and near-zero- Zero-CEIM data contribute less to the model robustness, while the low-CEIM data do make the model learn more from context. However, the proportion of the low- CEIM data in the datasets is quite low, leading to limited influence on the model learning. In this section, reconstruct the training sets to alter the proportion of low-CEIM data, trying investigate: (1) If enlarging the proportion of low-CEIM data a feasible way to improve robustness; (2) If we can achieve a good balance of robustness and generalization with a certain proportion of low-
4 Can We Build More Robust Datasets by Altering Entity-context Distribution?
Based on the above analysis, in this section, we explore three potential ways to de-bias NER datasets by altering the entity-context distribution.

---

CONLLO3,
CroseCa
44.5
$500
09
Rammon 3000
Kandann 17000
98
43.5
Fl.core
43.0
RUTWOY 2002
42.5
42.0
COUnT 2016
41.5
1000 compused ISCA
componeed
coed
41.0
Proportion LOW Data (%6
Proportion of Low CaM Data (8)
Propertion of Low-CB Data
ACE20OS, Set
ACE2005,
ACE2005, Crra20sscrosCCategory
2013
78
Arndom 5050
Kondom 2000
Bandoo 2000
Flos
Renitore 2000
2500
75
2000
36
2005 domppooo 20G5
30
20
40
60
80
10 20
39
40
50
60
70
50
10
20
30
40
50
60
20
Proportion Lowe CEIM Dara 196)
Proportion of Low CEIM Data (9)
Proportion Low-CEIM Data (56
Figure 4: Performance of the BERT-base- models trained the reconstructed datasets of different CEWC proportions. We report the average result of 3 repeated experiments. We also include the results of the RoBERTa- large models Appendix A.5.
CEIM data.
keeps increasing, the poor generalization ability will also affect robustness and lead to declining results. (3) Compared with the results on 1000
4.2.1 Detailed Experimental Settings
As the number of low-CEIM instances is limited in both CoNLL22003 and ACE2005 datasets (Table 1), we decided to fix the instance number of the reconstructed training sets to 1000. We construct datasets with the proportion of low-CEIM data ranging from 0% to 90%. For example, when constructing training set with 20% low-CEIM data, randomly sample 200 instances from the low-CEIM data and 800 instances from the high- or Dear--zrroCEIM data.
random data, the reconstructed data, with certain data proportions, achieves significant improvement on the robustness test sets and comparable performance on the Test set. We conclude that for different datasets, there exists appropriate proportions when good balance of i..d. and OOD generalization can be achieved (e.g., 10%- 30% for CoNLLO3, 20%--40% for ACE05).
4.3 Reducing the V-information of Entity
4.2.2 Results
As discussed in Section 3.2, the V-information discrepancy between entity and context is the main factor of dataset bias in NER. In order to de-bias NER datasets, the most intuitive idea is increase the low-PVI entities decrease the high-PVI entities in the training set. In this section, we explore potential ways to reduce the V-information of the entity of the training set by dataset reconstruction:
Figure 4 shows the model performance trained on the reconstructed training sets on BERT-basc- cased 3 We can observe that: (1) With the proportion of low-CEIM data increasing, the performance on the test set keeps dropping, which corresponds to the discussion in Sec.3.3 that the high- and Year-zero-EEIM data contribute more i.idd generalization.
(2) As the proportion of low-CEIM data increases, the performance on the OOV and CrossCategory test sets generally shows a trend of first rising and then falling. The rising robustness performance at low low-CEIM proportion validates the effectiveness of enlarging the low-CEIM proportion for improving model robustness. However, the low-CEIM proportion
Increasing the low-PVI enti- ties by randomly replacing high-PV1 entities with low-PVI entities. The replaced sentences added the original dataset data augmentation.
A. Random2low:
Based on A, we further select instances with high-PVI context and high-PVI entity for entity replacement. This process
B. HighC2low:
'hhe results on RoBERTa-large are in Appendix A.5.

---

Base Model: DERT-buse-cused
Base Mudel: RoBERTu-lurge
Methods
2005
ACE:2005
CONIL12003
ACE2005
Test
OOV
Cate,
Avg.
Test
0OV
Cate.
Avg.
Test
00V
Cate.
Avp.
Test
OOV
Cate.
Avg.
91.18
70.89
44.48
68.85
87.48
79.80
92.10
72.54
50.01
71.55
71.94
DataAug
73:45
46.39
70.12
87.16
73.60
72.31
LPFT
71.12
72.11
72.39
MINER
49.97
22.61
87.41
75.83
88.83
72.76
48.24
71.71
87.51
54.95
74.95
74.35
Raandom2loww
40%
90.42
76.03
47.72
71.39
87.54
81.64
48.28
72.49
91.99
79.00
56.78
75,93
88.54
81.88
51.24
73.88
20%
91.19
75.84
48.45
71.83
87.43
$2.18
48.76
72.79
92.18
78.36
56,11
75.55
$8.56
82.81
51.90
74.42
B.HighC2L.ww
40%
90.55
77.11
48.92
72.19
87.11
82.32
49.97
73.13
91.96
79.37
57.17
76.10
88.42
83.55
52.72
74.90
20%
90.71
76.35
48.02
11.69
87.37
81.28
48.59
72.41
91.82
79.23
57.11
76.05
88.66
84.07
51.89
74.87
C-Redundann2Loo
40%
90,42
77.34
48.07
71.94
86.97
82.17
48.41
72.52
91.54
79.52
58.06
76.38
88.67
64.43
53.49
75.53
Dotavel Transfrnbiliit Study (Dixcuseed in Seciiion 4.5)
MINER HighC2Loww Low 40%
90.21
79.85
53.42
74.66
87.01
80.5
49.81
72.55
91:70
80.33
61.16
77.73
89.37
85.16
54.54
76.36
BEKT.HgghCLLow 40%
91.74
79.18
55.15
75.36
89.11
83.37
52.23
74.90
Table 4: Performance of different approaches to reduce the y-information of entity in the datasets. Each reported result averaged by 3 repeated experiments. More detailed results of different rates are reported in Appendix A.5.
ensures that the replaced instances have informative contexts for the model to learn and predict.
with low-PVI entities without considering the context might create instances with both difficult entity and context, which would not benefit robustness and generalization. B. HighC2LLww ensures introducing more low- CEIM instances. In most cases, B. H.HighC2Low can construct a dataset that achieves both higher robustness performance and comparable test performance. (3) C.Redundant2Low is also effective in improving model robustness. Although large proportion of high-PVI entities are replaced, the test performance only slightly hurt and comparable to A. Random2Loww and DataAug that increases data size with augmentation. It also achieves larger improvement on the robustness test sets than A.Random2Low and DataAug. These results demonstrate that the large number of redundant high-PVT entities in the NER datasets limitedly benefits model generalization yet would harm model robustness. (4) Apart from the three methods, MINER can also achieve good robustness. Nevertheless, we claim that the dataset reconstruction methods are generally orthogonal to the model-based bbbustress-mmprovin methods like MINER. More details of this point are discussed in Sec.4.5.
C. Redundant2low: As declared in (Lin et al.. 2020), existing datasets mainly consist of regularly- patterned entities that harm model robustness. We also find in our experiments that there exists large number of redundant high-PVI entities in the datasets. Therefore, we propose reduce these redundant entities by replacing them with low-PVI entities. Different from and B, this method doesn't t increase the total data size.
In contrast,
Note that all methods are actually trying to introduce more low-CEIM instances. In our experiments, we reconstruct the existing datasets based on the above methods and train new models on the reconstructed training sets, separately.
4.3.1 Results
Table 4 shows the results of the models trained on three types of reconstructed datasets. Here, m% means replacing m% of the total entities in each method. From the results, we can observe that: (1) All of the three methods show obvious improvement over Base and even DataAug on OOV and CrossCategory. As the replacement rate grows, the robustness performance generally increases, well the test performance decreases. This trend similar previously observed trends in Fig.4. (2) Compared with A. Randdo22Low, B. HighCLLw shows relatively higher robustness performance and test performance. This is because randomly replacing high-PVT entities
4,4 Enhancing the V-information of Context
Aside from lowering the V-information of the entity in Sec.4.3, another intuitive approach is enhance the Y-information of the context. Similar to Sec.4.3, in this section, we design two potential ways to increase the proportion of high-PVT

---

Base Model: BERT buse--bae-cused
Base Model: RoBERTa-Ing
CONEL22003
ACI2005
CONI.12003
ACE2005
Test
00V
Cafe.
Avg.
Test
00V
Cate.
Avp.
Test
OOV
Cate.
Avp.
Test
OOY
Cate:
Avg.
Rase
91.18
70.89
44.48
68.85
87.48
70.71
92.10
72.54
50.01
71.55
71.94
Data/ug
90.52
73.45
91.21
96.33
52.18
91.14
75.67
90.98
91.88
79.33
56.27
30%
90.95
75.31
54.52
40%
90.43
76.69
50.20
72.44
86.87
81.21
45.26
71.12
91.67
77.86
54.85
74.79
88.53
84,08
48.81
73.81
30%
90.42
76.87
52.15
73.15
86.99
81.38
46.78
71.71
90.76
79.12
60.52
76.80
88.86
83.60
50.57
74.34
11Low2Highh
40%
89.72
78.41
55.20
74.61
86.48
82.01
47.78
72.09
90.92
79.58
61.93
77.47
88.67
84,40
52.20
75.09
Dataset Transforabiliyy Study (Discussed in Secrion 4.5)
MINER Low2fligh 40%
90.18
79.69
56,03
75.30
87.13
80.16
48.72
22.00
91.72
50.05
61.00
77.60
89.33
84.67
50.07
24.69
BKKT...w2Highh4u 40%%
91.05
79
63
59.79
76.82
88. 79
83.28
$1.88
74.65
Table 5: Performance of different approaches to enhance the V-information of context in the datasets, Each reported result averaged by repeated experiments. More detailed results different rates are reported in Appendix A.5.
contexts in the training sets:
thus hindering generalization i.i.d. distribution. Nevertheless, the results show that a good trade- off between OOD and i.i.d. performance can be achieved with certain replacement ratios.
I.Rando2HHigh Randomly deleting a certain ratio of contexts and replacing them with high-PVI contexts from the retained set.
4.5 Transferability of the De-biased Datasets
Ho.g Deleting a certain ratio of low-PVI contexts, and replacing the deleted contexts with high-PVT contexts.
As the V-information calculation is model-specific, it is intuitive to wonder if the datasets reconstructed based on one model model. In both Table 4 and Table 5, we further conduct experiments investigate the transferability of the de-biased datasets. In both experiments, we trained MINER on the reconstructed datasets (MINER+HighC2L.o. 40% and MINER+Low2High 40%). We also use the BERT-based reconstrueted datasets to train RoBERTa-large models High 40% and BERT. 40%). Surprisingly, the reconstructed datasets can also benefit other models to a certain degree. These results not only validate the generalizability of the de-biased datasets, but also reveal that the dataset reconstruction methods are orthogonal to the model-based robustness- improving methods such as MINER. It also shows that building more robust datasets is fundamental for building more robust NER systems.
transfer to another
Similar to Section 4.3, we reconstruct the existing datasets based on the above methods and train new models on the reconstructed data, separately.
4.4.1 Results
Table shows the results of different methods, where m% means deleting (replacing) m% of the total context. We can observe that: (1) Randomly deleting a certain proportion of context, although decreases context diversity in the datasets, doesn't it show serious drop in the test performance. Interestingly, by replacing these contexts with high- PVI contexts, the robustness performance can be improved a certain degree. (2) As shown in Appendix A. low-PVI contexts are often noisy contexts and will harm performance. Therefore, is intuitive that replacing low-PVI contexts with high-PVI contexts effectively improves robustness performance. It also outperforms H.ihndom2High most replacement rates, validating that reducing context difficulty can largely improve model robustness. However, this method suffers from more decrease in the test performance, which might be because reducing the low-PVI contexts relatively reduces model attention on the entity,
Related Work
5.1 Analyzing the Robustness Problem in NER.
Many works have focused on analyzing the robust- ness problem NER. These works generally fall into two categories: (1) Constructing challenging sets to evaluate the model robustness (Ghaddar

---

al., 2021; Lin et al., 2020) such switching the entities in lest sets (Agarwal al., 2020), testing on out-of-dictionary entities (Lin al., 2020; Kim and Kang, 2022) or introducing typos to create OOV entities (Wang et al., ;021); (2) Investigating the impact of possible attribute through delicate design of experiments (Fu et al., 2020; Kim and Kang, 2022), such as conducting randomization experiments (Lin et al., 2020) measuring the impact of attributes with specifically-desineed metries (Fu et al., 2020). Our work is totally different from the existing studics. We provide a brand new view of considering the NER robustness problem by quantifying and comprehensively analyzing the correlation between context-entity distribution and model robustness and provide new insights into the NER robustness studies.
datasets can largely improve model robustness. As these datasets can also benefit model-based robustness-improoingg methods, we argue that building more robust datasets is fundamental for building more robust NER systems.
Acknowledgement
The authors wish thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069, 61976056) and Shanghai Academic Research Leader Program 22XD1 401 100.
Limitations
We summarize the limitations of this work as follows: (1) We regard our work as an exploratory study on the robustness problem of NER from the dataset view. Nevertheless, the method to measure
worth mentioning that Lin et (2020) and Peng et al. (2020) (work on Relation Extraction) also consider the "context or name" problem and design experiments to disentangle the impact of context and entity name. These experiments also motivate the experiment designs in this work.
the dataset difficulty is not quite efficient since it requires first training two individual models on an i.i.d. training set. There might be more efficient ways to measure the dataset and instance difficulty, thus further improving the efficiency and practicality the data reconstruction methods. (2) In this work, we consider the robustness problem of NER models with only small pre- pre-trained models (model size less than 1B). As the large language models have shown powerful ability in information extraction tasks, is in doubt that if the same conclusion can also generalize large language models. Nevertheless, we believe our study is worthful since building more robust datasets is always important for building a NER system required for real-world usage. We expect our work can prompt more valuable future works also on large language models.
5.2 Mitigating the Robustness Problem NER.
There are also many works that aim mitigating the robustness problem in NER. These works include methods to alleviate the OOV problem (Bojanowski et al., 2017; Peng et al., 2019), leveraging data augmentation (Dai and Adel, 2020; Zeng et al., 2020) or adopting adversarial training (Ghaddar et al., 2021) other training strategies (Wang et al., 2022) improve model robustness. In this work, we explore a new direction of approaches: improve model robustness through data reconstruction. We also argue that constructing robust datasets fundamental for building more robust NER systems.
References
6 Conclusion
Oshin Agarwal, Yinfei Yang. Byron Wallace, and Ani Nenkova. 2020. Entity-wwichhdd datasets: An approach auditing the in-domain robustness of named entity recognition models. arXiv preprint arXiv:2004.04123:
this work, we conduct an interesting study on the model robustness problem of NER. We quantify the difficulty distribution of context and entity in existing NER datasets and reveal how the comite- distribution affects model robustness. Based on analysis, we further explore three potential ways to de-bias the existing NER datasets by reconstructing the existing datasets and altering the cntity-context distribution. With extensive experiments, we validate that the reconstructed
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135- 146.
Samuel R. Bowman, Gabor Angeli, Christopher Potts.
and Christopher D. Manning. 2015. A large

---

annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, Association for Computational Linguistics.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine- tuning can distort pretrained features and underper- form -istr--distiiuutinn. In International Conference on Learning Representations.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional Transactions of the association for computational linguistics, 4:357-370.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 260-270, San Diego, California. Association for Computational Linguisties.
Nigel Collier and Jin- in-Dong Kim. 2004, Introduction to the bio-entity recognition task at JNL INA. In Proceedings of the International Joint Workshop on Natural Language Processing Biomedicine and its Applications (NLPBA/BiOLLP), pages 73-78, Geneva, Switzerland. COLING.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising requence-to-sequeen for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computational Linguistics.
Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3861-3867, Barcelona, Spain (Online). International Committee on Computational Linguistics.
pre-training
Leon Derezynski, Eric Nichols, Marieke van Erp. and Nut Limsopatham. 2017. Results of the WNUT2017 shared task novel and emerging entity recognition.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. 34(1):00-70.
Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Hongyu Lin, Yaojic Lu, Jialong Tang, Xianpei Han, Le Sun, Zhicheng Wei, and Nicholas Jing Yuan. 2020. A rigorous study on named entity recognition: Can fine-tuning pretrained model lead to the promised land? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 7291-7300, Online, Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1XXv:1907.11199
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6000 PMLR.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3661-3672, Online. Association for Computational Linguisties.
Jinlan Fu, Pengfei Liu, and Qi Zhang, 2020. Rethinking generalization of neural models: A named entity recognition case study. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34.
pages 7732-7739.
Minlong Peng. Qi Zhang, Xiaoyu Xing. Tao Gui, Jinlan Fu, and Xuanjing Huang. 2019. Learning task- specific representation for novel words in sequence labeling. arXiv preprint arXiv: wr.Xv::905..220 12277.
Abbas Ghaddar, Philippe Langlais, Ahmad Rashid, and Mehdi Rezagholizadeh. 2021. Context-aware adversarial training for name regularity bias in named entity recognition. Transactions of the Association for Computational Linguistics, 2556-604.
C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-223.
Hyunjae Kim and Jaewoo Kang. 2022. How do your biomedical named entity recognition models generalize to novel entities? leee Access, 10:31513- 31523.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task:

---

-ineeeeeennne named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning HILT-NACCI 2003, pages 142- 147.
Christopher Walker, Stephanic Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus 1dc2006 URI. https//ccttalog Idc. upenn, edu/LDC2006106.
Xiao Wang. Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang, Tao Gui, Liang Qiao. Zhanzhan Cheng, and Xuanjing Huang. 2022. MINER: Improving voc-bl-voya named entity recognition from an information theoretic perspective. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 5590-5600, Dublin, Ireland. Association for Computational Linguistics.
Xiao Wang. Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang. Ruotian Ma, Zichu Fei, Ruijian Cai, Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng. Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. 2021. TextFlint: Unified multilingual robustness evaluation toolkit Proceedings
for natural language processing. of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tt International Joint Conference on Natural Language Processing: System Demonstrations, pages 347-355, Online. Association for Computational Linguistics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 Idc2013(19, Linguistic Duta Consortium, Philadel- phia, PA, 23.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. 2019. A theory of usable information under computational constraints. In International Conference on Learning Representa- tions.
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020. Counterfactual generator: A weakly- supervised method for named entity recognition. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 7270-7280, Online. Association for Computa- tional Linguistics.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. Proceedings of the AAAI Conference Artificial Intelligence, 32(1).

---

A Appendix
OntoNotes 5.0 (Weischedel et al., 2013)* from the general domain, Bio-NER (Collier and Kim, 2004) from the biology domain, ACE 2005 (Walker al., 2006)* the general domain, Twitter (Zhang et al., 2018) from social media domain and WNUT 2017 (Derczynski et al., 2017) from social media domain. The dataset details can be found in Table 6.
A.1 Discussion of Different CEIM instances
In Section 3.3, we categorized each dataset into three categories based on the CEIM scores: high-CEIM instances, low-CEIM instances, and Near-zero-CEIM instances. Then, we conducted experiments to show the effect of different CEIM instances on model robustness. To better understand the concept of CEIM, we provide some examples of different CEIM instances Tab.2.
In Section and further exploration Section 3, we conduct experiments on CoNLL 2003 and ACE 2005 to investigate the relation between the entity-context distribution and model robustness. Following (Wang et al.. 2022), we adopt two robustness test sets from (Wang et al., 2021): the OOV test set and the CrossCategory test set (denoted as "Cate."). The OOV test set is constructed by replacing entities with out- of-vocabulary entities of the same category. The CrossCategory test set is constructed by replacing entities with entities from different categories. Therefore, these two robustness test sets respectively correspond to the two typical failures of SOTA NER models shown in Figure 1. In Table 9, show some cases of these two robustness test for better understanding. We also include quantity details of these two test sets in Table 6.
Generally, the high-CEIM instances often contain low-PVI contexts. These contexts are often ambiguous or misleading, thus might introduce noise in model learning. On the contrary, the entities in these high-CEIM instances are often informative enough for the model to predict. As result, the model will pay less attention to the context when trained on high-CEIM instances. Therefore, removing these low-PVI contexts is also helpful for de-bias the datasets, as shown in S.c.4.4.
In near-zero- instances, the context and the entity are usually equally supportive (the number of cases with equally-low context and entity is small). However, the entity pattern is casier to learn and memorize. We deduce that the model might still tend to memorize entity patterns instead of learning more context, corresponding the results in Table 3.
A.3 Implementation Details for V-information Experiments
In Section 3, we conducted experiments to calculate the V-information and PVIs of each dataset. For both context-only and entity-only models, we implemented based classification model based on the Huggingface library". To measure the PVI of the whole training set, we adopted 5-fold cross-validation. As the PVI of an instance ().,) only depends on the distribution of (X,Y), using 5-fold cross-validation will not affect the estimation of PVI (Ethayarajh et al., 2022). We detail the hyperparameters used for the experiments in Table 7. Other hyperparameters are the same as the default hyperparameters not noted.
The low-CEIM instances consist of high-PVI context and low-PVI entities. These entities are often ambiguous, more complicated frequent entities. Therefore, trained on low-CEIM instances, the model will tend to pay attention to the informative context for label predicting, which will benefit robustness. However, as the generalization to the i.i.d. test sets mainly relies on high-PVI entities, these low-CEIM instances, although helpful in OOD situations, might harm i..d. generalizations. In Section 4.2,4.3,4.4, we validate that an appropriate context-entity distribution can achieve better trade-off berween i..dd and OOD generalization.
the text
less
All experiments are conducted on NVIDIA GeForce RTX 3090 and NVIDIA Tesla V100. For each result, we report the average results of 3 repeated experiments.
A.2 Dataset Details
"https://catalog Idc. upenn. edu/license/ allemnon-nembeess-ggreeennn
In Section 3, we examine and analyze the Y-information of commonly used datasets, including: CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) from the newswire domain,
"https://catalog..dt upenn. edu/license/
aalemnon-eemberr--ggeemeet....
/ http::///gttaaaa.commmngngggaaae

---

Datasets
Domain
Language
Class
Train
Dev
# Test
0OV
# Cate.
CoNLL 2003
News
English
4
15.0k
3.5k
3.5k
3.5k
3.5k
OntoNotes 5.0
General
English
18
60.0k
8.5k
8.3k
ACE 2005
General
English
8
9.7k
2.3k
2.0k
1.3k
1.3k
WNUT 2017
Social media
English
3.4k
1.0k
1.3k
Twitter
Social media
English
4
4.0k
1.0k
3.3k
BioNER
Riology
English
5
18.5k
0.1k
3.9k
Table 6: Dataset details.
Dataset
Model
Epoch
Batch size
are the same the default hyperparameters if not noted.
Base model: BERT Saae---aa-csse
CoNLL2003
Context-only
2
5c-5
32
All experiments GeForce RTX 3090 and NVIDIA Tesla V100. For each result, we report the average results of 3 repeated experiments.
conducted NVIDIA
Entity-only
1/2
5o-5
Context-only
2
5e-5
32
OntoNotes 5.0
Entity-only
1/2
5e-5
32
Context-only
Se-5
ACE2005
Eniity-only
1/2
5e-5
32
Context-only
5
1e-4
32
WNUT
Dataset
Training set size
Epoch
Ir
Batch size
Entity-only
3
le-4
hase-hase-caseel
Context-only
5
le-4
Twitter
Entity-only
3
le-4
32
CoNEL22003
<5000
10
5e-5
Context-only
3
5c-5
>5000
5e-5
BioNER
Entity-only
5c-5
<5000
10
56.5
ACE2005
Base model: RoBH:Raa-laree
>5000
10
5c-5
16
Context-only
2
model: RoBERTa-large
ACE2005
Entity-only
1/2
le-5
<5000
10
le-5
CoNLL2003
CON1L22003
Context-only
2
1e-5
32
>5000
le-5
Entity-only
1/2
le-5
<5000
lo:5
ACI2005
>5000
lc-5
16
Table Hyperparameters used for V-information experiments in Section 3.
Table 8:
Hyperparameters used for all dataset
reconstruction experiments, DataAug, and LPFT baselines in Section 4. Also used for the experiments in Section 3.3. For LPFT, we train I.P models and L.FT models with the same hyperparameters as in the table.
A.4 Implementation Details for Dataset Reconstruction Experiments
We
implemented all BERT-base-cased and RoBERTa- large dataset reconstruction experiments based on the token classification model in the Huggingface library?. For DataAug, implemented based on the implementations from (Wang et al., 2022) For MINER, we implemented based on the released code". We used batch_ =ine__, gama=0.001 beta=0.01, 1r=0000001 for BERT-based base- lines, and used beta=0.0001, Ir=0.00001 for RoBERTa-based base- lines. For LPFT, we also implemented based on the token classification model in the Huggingface library.0. We include the hyperparameters used for the experiments in Table 8. Other hyperparameters
A.5 Detailed Experiment Results of Data Reconstruction
In Section 4, we only showed part of the experiment results due to space limitation. In this section, we include other detailed experiment results, including the results based on RoBERTa- large in Section 4.2 (Fig.5), the detailed results of different data reconstruction rates in Section 4.3 (Table 10) and Section 4.4 (Table 11).
gama=0.0001,
fomer olss:///otrrrhh/tokkecccssssss caa.eeeiiiilsssssiiiiiiiiiiii cination
whtppss/gggthoo.ooo//giiiineee/e formers/

---

Dataset
Test set
Example
Bazemore (PER) said the Commodore Books (ORG) 's credibility was at stake over the issue trade labour.
CoNLL2003
OOV
The Yellow (PER) missed his club last games after the "National (ORG)
CoNLL2003
Cate.
slapped worldwide ban on him...
ACE2005
OOV
Duygu (PER) happen to be at very nice Ginowan (LOC) by the beach Lotte Capital (LOC) this is a chance for Evon (PER) to get away from Sacred Heart Secondary school (ORG)
coverage everything, and kind of relax
ACE2005
Cate.
Reporter: Solomon (PER) is associated with Clive (ORG) since 1987 and been very successful.
Table 9: Examples of the robustness test sets. Cate, denotes the CrossCategory test set.
CoNLLO3,Nstt
CONLLG3,OOV
CoNULO0.CiossCtteprry
Randan S000
(00
1006 comeated dets
30
18 20
40
59
20 70
93
12
50 23
Proportion
Proportinn
Pripurtinn
ACE2005 Tist
ACE2005,001
ACE2005, CreesCategeen
Fabson 2000
2000
2000 compose
Proparton of LOW CER Lula (4)
Proportann of Law CEM Daca
Proportion of Low CEM Dota 181
Figure 5: Performance of the laBERTa-large models trained on the reconstructed datasets of different low-CEIM proportions. We report the average result of repeated experiments.
Base Model: BERT-base-csedd
Model: RobtETTblarre
Methods
CoNLL22003
ACE2005
CoNLL22003
ACE20005
Arg.
Avg.
Avg.
OOV
Avg.
Base
21.18
70.89
42.48
68:35
87.48
79,80
44.84
70.71
92.10
72.54
50.01
71.55
88.84
80.84
46.13
71.94
Data
90.52
73,45
46.39
70.11 12
87.16
80.99
44.58
70.91
91.71
7699
52. 18
73
88.96
81.95
46
72,31
LPFT
91.14
73.89
48.32
71.1
87.76
80.28
44.37
70:50
92.12
75.67
48.53
72.11
85.85
82.10
46.23
72.39
MINER
90.98
76.39
49.97
22.61
87.41
79.23
45.45
70.70
91.38
79.33
56.27
75.85
88.83
82.69
46.75
72.76
10%
90.99
17
70 18
87.04
71.45
91.97
$3.65
74.07
85.85
81.98
40. 10.43
73.42
201%
90.84
76.00 04
48.24
71.71
87.51
81.23
47.05
71.99
91.84
78.05
$4.05
74.95
88:23
82:69
$2.13
74.35
RandomZlow
30%
90.50
75.60
71.41
87.00
47.42
71.62
75.83
85.30
82.41
$1.66
40%
90.42
76.00
47.72
71.39
87.54
81.64
48.78
72:49
91.09
20.00
56.77 7M
75.93
88.54
81.88
$1.24
73.88
10%
91.12
47.53
11.56
87.38
71.74
91.83
26.99
$3.89
74. 24
51.13
201%
91.19
75.64
48:45
21.83
87.43
82.18
48.76
22:79
92.18
78.36
56.11 11
75.85
88.56
80.81
$1.90
74.42
B.High
30%
90.72
75.56
48.15
11
72:40
79.30
76
83.51
$2.13
40%
90.55
77.11
48.92
72.10 19
87.11
82.32
49.97
73.13
91.76
79.37
57.11
76. 10
88.42
83.85
52.72
74.90
90.52
47.31
70.96
97.37
47.72
72:13
91.78
54.62
74.76
83.12
50.14
20%
90.71
76.25
48.02
71.69
97.37
81.28
48.59
72:41
91.82
79.23
57.11
76.05
88.66
84.07
51.89
74.87
90.45
49.26
72.14
97.27
47.59
72.16
91.50
56.70
15.84
82.52
40%
90.42
27.34
48.07
71.94
36.97
82.17
48.41
72.52
91.54
79.52
58.06
26.38
88.67
84.43
53.49
75.33
Pastosst Transfrrabiii Study (Divaossd Sevtion 4.5)
MINER Loggh 40%
90.71
79.85
53.42
74.66
$7.01
80.35
19.87
72:55
91.70
80.33
61.16
77.73
89.37
85.16
54.54
76.36
BERTHIgghLLoww 40%
91.34
79.18
55.15
75.36
89.11
83.37
52.25
74.90
Table 10: Detailed performance of different approaches to reduce the V-information of entity in the datascts. Each reported result is averaged by 3 repeated experiments.

---

Rase Medel: BERT-bone-csee
Base Model: RoBERTs-aage
CONLL2003
ACE200S
CONLL2003
ACE20SS
Test
OOV
Cale.
Test
OOV
Cafe
Test
OOV
Cale.
Test
OOV
Cafe.
Bose
91.18
70.89
44.48
08.85
$7,48
79,80
44.84
70.71
92.10
72.54
50.01
71.55
$8.84
30.34
46.13
71.94
DataAun
73.45
30
72.31
LPFT
91.14
73.89
48, 30
71.12
$7.76
$0.25
44.37
70.80
92.12
75.67
48 53
72.11
85
$2. 10
46 23
72.39
10%
90.74
72.41
45,90
69.55
67.17
50.19
44
70.68
91.60
75.78
53.22
73.53
58.97
$1.51
45.96
72.15
47.33
46
72.96
Randdom2High
30%
90.95
75.31
48.27
71.51
86.74
50.70
45.03
70.83
91.88
77.66
54,52
74.69
58.67
544 56
47.70
73.67
77.86
48
10%
90,74
15.02
47.96
71.24
50.09
46.38
71.19
91.45
97.74
56.71
75.30
$9.05
$2.15
50.13
73.79
20%
90,48
96.51
49.90
72.30
$7.05
50.79
47.81
71.59
91.30
79.33
59.48
76.770
$8.87
$4.13
$1.93
74.99
HHig
30%
90,42
96,87
52.15
73.15
$6,99
46.78
71.71
90,76
79.12
00.52
76.30
$8,86
$3.00
20.52
74.34
40%
89.72
28.51
55.20
23.01
$6.48
$2.01
42.02
72.09
90.92
79.58
61.93
22,42
$8.67
$4.40
$2.30
75.09
Dataaset Transforabille Study (Diicossed Scctton 4.5)
MINER + Low.2Highh
2High 401
90.18
79.69
56.13
75.30
87.13
$0.16
48.72
22.00
91.72
$0.05
61.03
77.60
39.33
$4.67
50.07
74:60
2High 40%
70.63
76.82
$3.28
51.88
Table 11: Detailed performance of different approaches to enhance the V-information of context in the datasets. Each reported result is averaged by 3 repeated experiments.


Reviews:
[{"review_id": "pttKbjByH9", "reviewer": "Reviewer_kCPH", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough reading and constructive feedback!\n\n\n#### **Regarding the motivation:**\n\nSorry for claiming our motivation and main contribution less clearly. It is true that “the entity is much easier for the pre-trained model to learn than the context” has also been discovered and discussed in previous work. However, we want to clarify that this is not the “conclusion” of our work. As shown in line 076-082, we regard this point more as the motivation or verified support for our work. In fact, we start with previous findings on “the entity is easier to learn”, trying to provide a more systematic and more comprehensive investigation on this point. We want to reclaim the differences between this work and previous work as follows:\n\n1. While most previous work only verified or discussed this point experimentally (such as testing models on challenging test sets [1], conducting randomization experiments [2], etc. ), we are the first to systematically quantify this point from a theoretically-verified perspective (i.e., the V-information), providing a new angle of explanation for the NER robustness and the dataset bias.\n2. While previous work mainly focus on the impact of entity distribution on the model robustness, we consider the context distribution is also significant regarding the dataset bias, and the impact of entity cannot be isolated without considering the context. With the dataset-difficulty view, we are able to quantify and “compare” the difficulty of entity and context, and conduct a more comprehensive investigation into the joint distribution of entity and context, as well as how different joint distribution induces dataset bias. **We are the first to comprehensively consider the joint distribution of context and entity towards NER robustness.** (In Table 4, we verified that context distribution is indeed important for model robustness. We are the first to verify this and successfully improve the NER robustness by altering context distribution.)\n3. Besides how we provide new (and more comprehensive) explanation to this point, we also verified that our explanation can serve as a successful tool for constructing more robust datasets. While some previous works also conduct data augmentation for improving model robustness, their practices are relatively heuristic and empirical (such as switching entities to similar ones) and lack effective measures and targets. **We provide the dataset reconstruction with quantified measures (the PVI and the V-information) and verified targets (e.g., to enhance the V-information of context)**, and we believe our work is helpful for future work to evaluate the dataset robustness, refine existing datasets, and even build new robust datasets.\n\n\n\n#### **Regarding the Presentation:**\n\nSorry for presenting the paper less clearly and causing confusion. We would like to explain the confusing points as follows:\n\n1. Regarding “data bias”, “high-CEIM” and “entity-context distribution”:\n    1. The “data bias” (or “dataset bias”) refers to the intrinsic bias in the dataset that might induce shortcut learning. Specifically, in NER dataset, if the entity itself is adequate for recognizing the correct class, and the entity itself is much easier to learn and remember than the context, the model will not learn to deduce the answer using context information. This causes a shortcut learning problem of NER models, which will harm model robustness and generalization on other domains or OOD data. For example, if in the training data, “Washington” is mostly labeled as a Person name, the model will learn to directly regard “Washington” as a Person name without referring to the context. Then, if the model needs to recognize entities in the sentence “He lives in *Washington*”, it will incorrectly recognize “Washington” as Person, regardless that the context indicates “Washington” is a location.\n    2. By calculating and experimenting with the CEIM, we recognize two kinds of CEIM data that might induce dataset bias, i.e., the high-CEIM and near-zero-CEIM data. We regard the high proportion of high-CEIM and near-zero-CEIM data as a cause of NER dataset bias, and verified that deducing the proportion of these data helps alleviate dataset bias (corresponding to better model robustness). \n    3. The “entity-context distribution” refers to the “difficulty” distribution of the entity, the context, and the joint difficulty distribution of entity and context. (For instance, how many “easy” or “difficult” entities/contexts are there in the data? In each single sentence, how difficult is the entity and the context, and which one is easier to learn? In the whole dataset, how many instances contain an entity easier/more difficult than the context? As for the whole dataset, is the entity information easier for the model to learn than the context information?) By systematically examining the entity-context distribution towards difficulty, we are able to explain the NER dataset bias from a more comprehensive view, as discussed before.\n2. In Figure 1, we point out two typical failures, i.e., the OOV (out-of-vocabulary) failure and the ambiguous entity failure. **Both these two types of errors are discussed in the paper.** In line 237-243, we declared that the adopted two robustness test sets exactly correspond to the two failures, respectively. We also show some cases of the two robustness test sets in Appendix A.2. In Table 9, we can see that the OOV test set is designed to evaluate the OOV failure, which introduces many OOV entities. Similarly, the Cross-category (”Cate.”) test set introduces many ambiguous entities to challenge the model. All our robustness experiments are tested on these two test sets to show the model robustness towards both OOV and ambiguous entity failures.\n3. Thank you for pointing out the typos and mistakes in our paper! We will thoroughly review and correct all these typos/mistakes in the next version of this paper.\n\n\n\nWe apologize again for all our misleading and confusing writing and thank you very much for pointing out all these problems. These feedbacks are quite helpful for us to improve our paper!\n\n[1] Ghaddar, A., Langlais, P., Rashid, A., & Rezagholizadeh, M. (2021). Context-aware adversarial training for name regularity bias in named entity recognition. *Transactions of the Association for Computational Linguistics*, *9*, 586-604.\n\n[2] Lin, Hongyu, et al. \"A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?.\" *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a comprehensive study of NER dataset bias and model robustness from a dataset difficulty view, which may provide some new insights for NER research.\n2. The paper proposes three potential ways to de-bias the NER datasets by introducing more instances with low-CEIM. The experimental results demonstrate the effectiveness of the proposed methods.", "reasons_to_reject": "1. Motivation: The paper mentions there are many previous works that have delved into the robustness problem of NER, including the studies investigating how the contexts and entities affect the performance. So what are the specific differences between this paper and previous work? Has the conclusion of the paper, i.e., \"the entity is much easier for the pre-trained model to learn than the context\", been discovered in previous work？\n2. Presentation: (1) There are many terms in the text that are not explained very clearly. For example, Does the term \"data bias\" mean instances with high-CEIM? If yes, in real-world applications, there are many instances, e.g., \"Washington\", where we can directly determine the type of entity by its name. The term \"entity-context distribution\" is also not clear. (2)  In Figure 1, the paper points out two typical failures. However, the paper does not seem to discuss the second type of error (\"ambiguous entity\"). (3) There are many typos or mistakes in the paper. For example, there lack some citations in line 68. In sec. 3.3 line 285, Is the ''Table 2'' refers to \"Table 1\"?", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "The paper conducts an interesting study on the model robustness problem of NER. The paper quantifies the entity-context difficulty distribution in the existing dataset using V-information and CEIM and finds that the entity is much easier for the pre-trained model to learn than the context. Meanwhile, the paper explains the relationship between the entity-context difficulty and model robustness and finds the low-CEIM instances, i.e., instances with contexts easier than entities, contribute most to the model robustness. Based on the findings, the paper suggests three potential ways to de-bias the NER datasets by altering the entity-context distribution. The experimental results show that the de-biased datasets can benefit different models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See reasons to reject.", "Typos_Grammar_Style_and_Presentation_Improvements": "See reasons to reject."}, "extra_text": {}}, {"review_id": "sNZKspVOHu", "reviewer": "Reviewer_keJx", "summary": "", "questions": "\n(1) The used datasets are quite old. There are newer datasets like FewNerd which has more training examples and NER types. Using datasets with more types might be better for this paper's purpose.\n\n(2) In Table 4, it seems that the green line performance (even in the OOV case) can't match using 10,000 examples. I wonder how the proposed data augmentation method would perform when using more instances. Would there be diminishing returns?\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough reading and constructive feedback!\n\n#### **Response to Question (1):**\n\nThank you for your valuable advice about the newer datasets such as Few-NERD. To better illustrate our work, we further conduct several experiments on the Few-NERD dataset. The results are rather interesting, and we are glad to discuss our findings with you:\n\nThe Few-NERD has 66 fine-grained categories, which is more fine-grained than any of the existing NER datasets. By measuring the V-information on these fine-grained classes, we find that:\n\n1. Generally, the fine-grained dataset shows less dataset bias than the coarse-grained dataset: in most of the fine-grained classes, the V-information of the context is comparable and even larger than the V-information of the entity. This might be because: \n    1. Compared with categorizing coarse-grained classes (such as “person”, “location”, “organization”), categorizing different entities between fine-grained classes (such as “person-scholar”, “person-actor”, “person-artist”) is inherently harder, where the contexts include more clues for categorization. \n    2. The Few-NERD dataset is much larger than most existing datasets (about 130k training data) and contains more classes, and the entity patterns of each class are much more complicated to learn. For each class, it is harder for the model to predict the entity class through shortcut learning certain entity patterns. \n2. The analysis method we proposed allows us to delve into the distribution of different fine-grained classes, and we find that the V-information distribution is quite diverse across different classes:\n    1. In most classes, the V-information of entity and context are comparable. As for the CEIM distribution, the near-zero CEIM data still cover the largest proportion of most classes (the high-CEIM and low-CEIM proportion is relatively balanced). \n    2. In some classes, the V-information of entity is rather high, such as the “location-park” (V-info entity/context=6.04/4.23), “building-hospital” (V-info entity/context=6.84/5.93), etc. This might be because the entities in these classes generally show simple patterns. For example, most entities in “location-park” are ended with “park”. \n    3. Meanwhile, there exist some classes that heavily rely on the context information for prediction, such as the “person-athlete” (V-info entity/context=2.29/4.30), “person-actor” (V-info entity/context=2.79/4.80) class, etc. These classes, such as the “person-athlete” class, mostly consist of a normal person name, and it is quite hard for the model to deduce the correct class without the context. For instance, in the sentence “Honky then lost his championship at Summerslam”, one cannot tell “honky” is an athlete without the context. \n3. Based on findings 1,2, we conclude that Few-NERD is a much more robust dataset than most of the existing coarse-grained datasets. Nevertheless, there still exists severe bias problems in some classes (such as the biased “location-park”, “building-hospital” classes). Also, most classes that “have comparable V-information of entity and context” might also have potential robustness problems according to the CEIM distribution. **Based on the detailed analysis, we believe that our dataset reconstruction methods can be further applied to improve the robustness of Few-NERD in a class-wise manner.** Unfortunately, we didn’t conduct further robustness tests on the Few-NERD dataset as in Table 2 and Section 4 because **we didn’t have the robustness test sets as on CoNLL03 and ACE2005 (the robustness test sets we use now are constructed by (Wang et al., 2021)).** We agree with you that it will be more convincing to conduct the experiments on Few-NERD, and we also consider it will be interesting to delve into the potential robustness of Few-NERD. In future work, we might try to construct our own robustness test sets on Few-NERD and conduct more robustness studies. Thanks again for your advice!\n4. Another interesting little finding: We think Point 2 (3) also reveals that the model robustness (corresponding to learning more context) might also be important when we need to generalize the NER model to recognize more fine-grained classes that highly rely on context information. To verify this, we further conducted two simple transfer experiments: \n    1. Model A: first trained the NER model on the original CoNLL03 dataset, and then adapted the model to the Few-NERD dataset via few-shot training; \n    2. Model B: first trained the NER model on our reconstructed CoNLL03 datasets, and then adapted the model to the Few-NERD dataset via few-shot training. \n    \n    We find that model B that trained on the reconstructed robust dataset can achieve better few-shot results on the context-relied classes (for example, the average 5-shot f1-score of “person-x” classes is 20.01 (model A) and 24.23 (model B)) while maintaining similar results with model A on other classes (for example, the average 5-shot f1-score of “organization-x” classes is 40.07 (model A) and 40.18 (model B)). Although this experiment is rather simple, it indicates the importance of building more robust datasets for model generalization.\n\n\n#### **Response to Question (2):**\n\nIn Figure 4 (do you mean Figure 4 rather than Table 4?), we only conducted the data proportion altering studies on 1000 reconstructed data (this is because there are very few low-CEIM data in the datasets and it’s hard to achieve 90% low-CEIM proportion on the full datasets). Therefore, it is straightforward that the green line performance cannot match the performance of 10000 data. We compared the green line with more data here only to reveal the potential of the 1000 reconstructed data. We also admit the limitation of this proportion-altering method since it is hard to apply to larger data. \n\nThen, in order to further validate the data reconstruction methods when using more instances, in Section 4.3, 4.4, we designed two data-size-agnostic data reconstruction methods, and verified their effectiveness on full datasets. **All experiments in Table 3 and Table 4 are conducted on the full CoNLL03 and ACE 2005 datasets (15k and 9.7k training data, respectively)**. As shown in the tables, the proposed dataset reconstruction methods are still quite helpful when using more instances. Moreover, as discussed in both sections, these two methods actually also result in the increase of low-CEIM data proportion, which is similar to what we directly did in Figure 4. Therefore, these results indicate that enlarging the proportion of low-CEIM data can also be helpful when using more instances.\n\n#### **Response to “Reasons To Reject”:**\n\nWe agree with you that the idea that \"entity names cause shortcuts\" has been heavily explored, and our work is mainly on how to present the shortcut via the new V-information perspective. Despite this, we would like to further discuss our contribution:\n\n1. Previous work actually focus more on how the “entity distribution” affects NER robustness. While in this work, we are the first to comprehensively consider the joint distribution of entity and context on this problem. For one thing, the V-information-based measures we introduced allow us to systematically examine and “compare” the entity and context difficulty distribution, thus explaining the dataset bias in a more comprehensive manner. For another, we are the first to consider “improving robustness via altering context distribution” and validate the effectiveness of altering context distribution on model robustness (Section 4.4, Table 4). These results are rather intuitive based on our preliminary analysis in Section 2, yet are very interesting and provide new insights on how to build better NER datasets from the context view.\n2. Previous work on NER robustness mainly focus on discovering or verifying the robustness problem via experimenting, or proposing methods to improve model robustness. In this work, we offer a more systematic view for examining and analyzing the NER dataset bias and robustness problem, and verify that our new view can induce valid new methods for improving NER robustness (such as the context-based method). More importantly, we hope all these new views, quantified analysis, and exploratory experiments can provide useful references for “how to build more robust NER datasets”. For one thing, people can use the measures we used to assess existing NER datasets regarding how robust these datasets are. For another, people can refer to our measures and methods to improve existing datasets they have, and the improved datasets are verified to be transferable (Table 3,4). Furthermore, when building new NER datasets, our work might also be a good reference for raw data selection (e.g., to select which instances should be annotated in order to build more robust datasets. So far we haven’t explored methods toward this goal, yet we consider it will be interesting future work.) We believe our work is meaningful and hope it can provide distinctive insight into building more robust NER datasets.\n\n#### **Response to the Missing References:**\n\nThank you very much for reminding us of this missing reference! Actually, this interesting work, although not in the same area, also motivates us in the experiment designs. We will add it to the Related Work section in the next version.\n\n#### **Regarding typos errors:**\n\nThanks for pointing out these problems! We will carefully examine and revise our paper based on your advice in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The authors study the robust NER problem from a new dataset difficulty perspective.\n\n(2) The observation is drawn from comprehensive experiments (multiple NER datasets, detailed analysis, intuitive experiment design).\n\n(3) Based on the observation that easy-entity names caused the shortcut, the authors design data-augmentation methods to alleviate the problems. The proposed methods successfully improved the model's performance on robustness test sets.\n", "reasons_to_reject": "(1) Though the V-information angle is new, the idea that \"entity names cause shortcuts\" has been heavily explored. The contribution is mainly how to present the shortcut (via this new perspective). \n", "questions_to_authors": "\n(1) The used datasets are quite old. There are newer datasets like FewNerd which has more training examples and NER types. Using datasets with more types might be better for this paper's purpose.\n\n(2) In Table 4, it seems that the green line performance (even in the OOV case) can't match using 10,000 examples. I wonder how the proposed data augmentation method would perform when using more instances. Would there be diminishing returns?\n\n", "paper_topic_and_main_contributions": "This paper studies the named entity recognition (NER) robustness problem from a dataset difficulty perspective. Previous work shows that models tend to have short-cut learning via entity names and there is literature constructing adversarial datasets detecting such short-cut behavior. The authors of this paper study this problem via V-information (or dataset difficulty) instead.\n\nThe idea is straightforward: the authors measure how difficult it is to make predictions purely based on entities and purely based on context. And the results on several commonly-used NER datasets show that using entity names alone is even easier, which suggests shortcuts. With this observation, the authors propose several data-augmentation methods by replacing easy-entity / hard-context (to discourage the models to do short-cut learning) and the proposed methods improve models' performance on adversarial test sets.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\n\nThough not in the same area, the paper \"Learning from Context or Names? An Empirical Study on Neural Relation Extraction\" already studied a similar \"entity or context\" problem in relation extraction. \n", "Typos_Grammar_Style_and_Presentation_Improvements": "\n\n065: when a citation is used in the middle of the sentence, it should be \"Agarwal et al. examine...\" instead of \"(Agarwal et al.) examine...\"\n\nTable 2: the color use is a bit counter intuitive. Usually, people use green for improvement and red for degradation.\n"}}, {"review_id": "wyAKyFB1Vd", "reviewer": "Reviewer_zR24", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough reading and constructive feedback!\n\n**Regarding the presentation quality:**\n\nWe apologize for the presentation quality and all the mistakes we made. Thank you very much for your careful review and pointing out all these errors. We will thoroughly review and revise the paper based on your review. Also, the suggestions to include Appendix A.1 and Table 5 in the main page, and to increase the table/figure sizes are very helpful. We will take your advice once we have enough pages. Thanks again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The analysis methods, V-information and CEIM, are first used in the field of NER by this paper and are technically sound.\n\n2. The paper is firmly grounded in, and well-supported by relevant studies.\n\n3. The results are interesting. In Tables 3 and 4, although the proposed methods do not improve the performance on in-domain test sets, they are effective on out-of-domain or out-of-vocabulary test sets, even when applied to the MINER model.", "reasons_to_reject": "1. The presentation quality should be improved. Please see the comments in the section \"Typos Grammar Style And Presentation Improvements.\"\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides thorough analyses of dataset bias in named entity recognition (NER). It introduces V-information for measuring the difficulty of individual examples and entire datasets and context-entity information margin (CEIM) for assessing the difference in difficulty between entity- and context-level instances. Based on the analyses, the paper found that lexical information of entities is easier to learn compared to contextual information, which makes models learn shortcuts from entity strings. The paper argues that the difference in learning difficulty between entity and context information should be reduced to de-bias the model. The proposed ideas include increasing the portion of difficult entities or increasing the portion of easy contexts in the training data, which improved the performance of baseline models in several robustness tests on NER benchmark datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. I found many typos and grammatical errors throughout the manuscript. Please thoroughly review and revise the paper if the authors prepare the final version. Here are a few examples:\n\n- In the abstract, the abbreviations \"SOTA\" and \"OOD\" are used before defining them.\n\n- In Line 285, \"Table 2\" should be corrected to \"Table 1\".\n\n- In Line 308, \"the Test set\" should be corrected to \"the test set\".\n\n- In Line 368, \"(3) LPFT\" should be corrected to \"(4) LPFT\".\n\n- In Line 389, \"Tab.2\" should be corrected to \"Tab.1\".\n\n- In Line 473, \"might, create\" should be corrected to \"might create\".\n\n2. The tables and figures are too small. It would be better to increase the size of them.\n\n3. Appendix A.1 and Table 5 seem very helpful in understanding Section 3.3; therefore, consider including them on the main page if the paper is accepted and one additional page is allowed."}}]

Author rebuttal:

Meta-review: The paper conducts an interesting study on the model robustness problem of NER. It introduces V-information for measuring the difficulty of individual examples and entire datasets and context-entity information margin (CEIM) for assessing the difference in difficulty between entity- and context-level instances.
the paper found that lexical information of entities is easier to learn compared to contextual information, which makes models learn shortcuts from entity strings. The paper argues that the difference in learning difficulty between entity and context information should be reduced to de-bias the model.  In general, this paper represents a well-executed and systematic exploration. Its main limitation lies in the lack of novelty in the conclusions it draws and the proposed solutions, as they align closely with widely accepted findings within the field.
Decision: Accept-Main