Taxonomy Expansion for Named Entity Recognition
Authors: Karthikeyan K
Yogarshi Vyas
Jie Ma
Giovanni Paolini
Neha Anna John
Shuai Wang
Yassine Benajiba
Vittorio Castelli
Dan Roth
Miguel Ballesteros
Venue: EMNLP 2023 Main (2023)
Keywords: named entity recognition
taxonomy

Abstract: Training a Named Entity Recognition (NER) model often involves fixing a taxonomy of entity types. However, requirements evolve and we might need the NER model to recognize additional entity types. A simple approach is to re-annotate entire dataset with both existing and additional entity types and then train the model on the re-annotated dataset. However, this is an extremely laborious task. To remedy this, we propose a novel approach called Partial Label Model (PLM) that uses only partially annotated datasets. We experiment with 6 diverse datasets and show that PLM consistently performs better than most other approaches (0.5 - 2.5 F1), including in novel settings for taxonomy expansion not considered in prior work. The gap between PLM and all other approaches is especially large in settings where there is limited data available for the additional entity types (as much as 11 F1), thus suggesting a more cost effective approaches to taxonomy expansion.

Taxonomy Expansion for Named Entity Recognition
Karthikeyan K Yogarshi Vyas Jie Ma", Giovanni Paolini', Neha Anna John', Shuai Wang', Yassine Benajiba', Vittorio Castelli", Dan Roth', Miguel Ballesteros *Department of Computer Science, Duke University
AWSS AI Labs
karthikeyan kêduke. kew.kkeeaannkkkke..e. edu, poggarshi,jiemne pwwlppalligggoo. nehajohn, wshui, benajiy, vittorca, drot, ballemi wemaaoo.......nnm..moemm com
Abstract
Entity Types 6 PERSON, LOCATION
Training a Named Entity Recognition (NER) model often involves fixing taxonomy of tity types. However, requirements evolve and we might need the NER model to recognize additional entity types. A simple approach is to re-annotate entire dataset with both existing and additional entity types and then train the model the re-annotated dataset. However, this is an extremely laborious task. To remedy this, we propose novel approach called Partial Label Model (PLM) that uses only partially an- notated datasets. We experiment with diverse datasets and show that PMMosssistntyy per- forms better than most other approaches (0.5- 2.5 FI), including novel settings for taxon- omy expansion. The gap between PLM and other approaches especially large in settings where there is limited data available for the ad- ditional entity types (as much as 11 Fl), thus suggesting a more cost effective approach to taxonomy expansion.
Egg. James works BBC London
Dataset 50
PERSON
LOCATION
Entity Types 8 ORGANIZATION
Missing Annotations
Egg. Robertis
SDE NASA
Dataset
Final Entity Types & PERSON, LOCATION, ORGANIATION
Emma researcher ABC, New York
ORGANIZATION LOCATIN
Desired Model
Taxonomy Expansion for NER: Given Dataset DA annotated only with entity types &A and Dn annotated only with Ep, the task to train a model that recognize entities from both E and Eg.
Figure 1:
(ORGANIZATION), and then train single model to recognize all the entity types this data. If such re-annotation and re-training needs happen iteratively every time new entities are introduced, then this an expensive process, and re-annotation requires knowledge of all entity types both old and new. A secondary issue with this approach is that the original training data was chosen only to con- tain examples of the initial entity types. Thus, it may not even contain enough examples of the new entity types and it may be simpler obtain data that is rich in the new entities types for annota- tion (ORGANIZATION), Given these two disjoint datasets with two separate sets of annotations, we can train two different models to recognize the two sets of entity types - yet this requires heuristics to combine the predictions of the two models, es- pecially if the old and new entity types have some shared semantics (e.g. some of the entity types are subtypes of the existing entity types).
1 Introduction
Training Named Entity Recognition (NER) model typically involves presupposing a fixed taxonomy, that is, the entity types that are being recognized are known and fixed. However, NER models often need to extract new entity types of interest appear- ing in new text, thus requiring an expansion of the taxonomy under consideration. Consider an example (Figure 1), where NER model was ini- tially trained to recognize PERSON and LOCATION types using data annotated only for these two types. However, it later needs identify ORGANIZATION entities in addition to the two initial entity types.
There are several simple ways to solve this prob- lem, yet they are cumbersome in different ways. One possible solution is to re-annotate all data that was only annotated for the initial entity types (PERSON, LOCATION) with the new entity types
'Work done during an internship AWS Al Labs *Corresponding author
This problem of Taxonomy Expansion for NER

---

(henceforth, TE-NER) has been studied in several different settings in prior work. However, most prior work ($2) assumes that the old and the new entity types are completely disjoint and mutually exclusive. In contrast, we define general version of the TE-NER problem ($3) where the original training data can be used, and old and new en- tity types can be related in different ways. Our general setup allows for many practical scenarios encountered during taxonomy expansion such as new entity types being subtypes of existing entity types, or partially overlapping in definition.
2017) and continual learning (CL) based methods for NER (Xia et al., 2022; Monaikul al., 2021; Chen and Moschitti, 2019; De Lange et al., 2019), our setup differs in two aspects from previous tasks.
First, we assume access to both initial (Da) and new datasets (DB), while typical CL based meth- ods assume access only to the new data and to a model trained on the initial data. Xia et al. (2022) assumes access to a generative model trained on the initial dataset and trains the final model in two-stage process. In contrast, we systematically derive PLM using a latent variable formulation and train our final model in a single stage.
We then propose novel solution to TE-NER mo- tivated by the fact that the available training data are only partially annotated ($4.2). As shown in Figure BBC has not been annotated as ORGANI- ZATION in the initial dataset DA, since ORGANIZA-
Second, we do not constrain the new entity types to be disjoint from old entity types. For example, if PERSON is among the initial entity types, the new entity types can include ACTOR, which is a sub- type of PERSON; our desired final model should predict a mention (ACTOR, PERSON) if the men- tion an actor, and just PERSON if the mention is a person but not an actor. This setup is related to prior work on hierarchical classification (Silla and Fre- itas, 2011: 1; Arabie et al., 1996; Meng et al., 2019) or fine-grained NER (Ling and Weld, 2012; Choi et al., 2018; Mai et al., 2018; Ringland et al., 2019). However, the objective of such work is to use the hierarchical structure to train better tne-grained NER model; they do not deal with taxonomy ex- pansion. Abhishek et al. (2019) propose a unified hierarchical label set (UHLS) and use Partial Hier- archical Label Loss to train a unified model from multiple datasets with different (potentially over- lapping) label sets. However, our work differs in two key aspects: first, our approach works even if two labels are overlapping but neither one is sub- type supertype of other (i.e. even if the relation is non hierarchical) and second, our probabilistic approach theoretically grounded (PLM).
TION was not in the label taxonomy of A. Similarly, Robert is not annotated as PERSON in the new dataset Dg. Given such partial annotations, we treat the true, unobserved label as a latent variable and derive our approach called Partial Label Model (PLM). Intuitively, PLM uses model trained on DB to annotate BBC with a probability distribu- tion over entity types B. Similarly, it uses a model trained on DA obtain predictions for Robert. Finally, the desired model over all entity types is trained using both the observed annotation and the soft labels obtained via the predictions of the inter- mediate models. Minimizing the KL divergence loss (as used by Monaikul et al. (2021) and Xia et al. (2022)) corresponds to maximizing lower bound on the likelihood of our formulation, while the proposed PLM loss corresponds directly maximizing the likelihood.
Experiments with six English datasets show that PLM outperforms all baselines except one (cross annotation) by 0.5-2.5 F1 ($6). However, this competitive baseline, that involves simply cross- annotating the datasets using weak labels from a model trained on the other dataset, is less accu- rate in sho-shot settings compared to PLM. When only 200 examples are provided for each new entity type, PLM is F1 better than aooss-nnntttiion. We further test the robustness of these approaches across several data distributions and find that they perform robustly across the board ($7).
Another class of related approaches is based on positive-unlabelled (PU) learning (Bekker and Davis, 2020; Grave, 2014; Peng et al., 2019) or learning from partial/noisy training data (Mayhew et al., 2019) - if we combine the initial (D,) and new datasets (DB), then the combined training data can be viewed as positive-unlabelled or noisy dataset. However, unlike PU learning, the annota- tions in our setup are not randomly missing; instead, annotations of entity types and B are missing in datasets D,3 and D respectively.
2
Related Work
While this work is closely related to knowledge distillation (Hinton et al., 2015; Shmelkov et al.,

---

3 Problem Definition and Notation
Datanet SP Emiiiie
Dataset Entiiin Xg
Given datasets D and DB annotated with en- tity types Ea {e -x.."))) and Eg ( e..... respectively, the goal of TE-NER is learn a model to recognize entities from both E and Eg. For our initial definition and solution, we assume that all entity types are distinct and mu- tually exclusive, i.e., a token does not belong to more than one entity type. While this is in line with prior work, it is also shortcoming as entity types and definitions can often overlap in real world set- tings well academic datasets. For instance, in the FewNERD dataset (Ding al., 2021), an entity can be both a PERSON as well as a POLITICIAN. To handle such cases, we define a more general version of the problem and solutions in $5.
Model
Reannotated Dataset using Model
Reannotuted Dataset using Model
Figure 2:
PIM consists of three steps: ! Train Model A /Model to recognize entities in EalEB using dataset DA/DD @Use ModelA to annotate D3 with dis- tribution over entity types ε (repeat with Modelg and D + Use these annotations with the PLM loss train the final model. Cross- cross-annotatiin is similar except in @. we annotate the dataset with the hard predictions of the models instead of distribution, and use cross-entropy loss in 3. PLM KL replaces the PLM loss in 3 with a KL-divergence term.
Prior work on TE-NER is based on continual learning where the goal is to adapt the NER model continuously new entity types (Monaikul et al., 2021; Xia et al., 2022). Such work assumes access only the models trained on the original datasets (Model trained on DA. Modely trained on DB) and not the datasets themselves. However, in many scenarios, both the original dataset and the new dataset are available but are annotated using differ- ent sets of entity types due to an evolving taxonomy come from different sources. Hence, our defini- tion assumes full access both D and Dr. Also, unlike few-shot and transfer-leaming setups (Phang et al., 2018; Ma et al., 2022), our goal to train a model (Model finat) that does well on both ε and E (irrespective of the size of Dr: however, we return to the question of size in $6.3.2).
low are motivated by simple observation: in both D A and Dis, the observed labels not always the true labels. Specifically, if word is annotated as O (used to indicate tokens that are not part of entity mentions), we do not know its true label. Figure 2 provides an overview of all approaches.
4.1 Cross Annotation (X-Ann.)
Before discussing our solution, we suggest a naive solution to TE-NER. As discussed before, the main challenge with TE-NER is partial annotations. If the annotations are correct and exhaustive, i.c., DA was annotated for entity types En (and DD for EA), then we could simply combine these datasets and train an NER model on the combined dataset. Cross-annotation motivated by this observation instead of expecting the data to be fully labeled, uses model outputs to provide the missing labels. Under our setup, if a token in DA is annotated as O, it can still belong to either one of the entities in E (e.g. BBC in Figure can truly be O or it can be an ORGANIZATION) Therefore, in DA if a word is annotated O, replace it with the prediction of Modelg. Similarly, in DB, 3, if word is annotated O, we replace it with the prediction from Model A- We combine these re-annotated versions of DA and DB to train the final model.
Why is Taxonomy Expansion challenging? The central challenge with TE-NER is partial annota- tions - if a mention in D belongs to an entity in Eg. it will not be annotated (e.g. in Fig 1, BBC is not annotated ORGANIZATION). Similarly, DB is also partially annotated. Such partial annotation misleads model training, and prior work attempts to mitigate this issue on a single dataset (Mayhew et al., 2019; Jie et al., 2019). We focus on this problem in the context of TE-NER.
4 Methods for TE-NER
We first discuss our methods for the scenario where all entity types are disjoint; then, in $5.3, we dis- cuss modifications for a more general definition of TE-NER. We use the BIO scheme (Ramshaw and Mareus, 1995) for this work. All the approaches be-
4.2 Partial Label Model (PLM)
Cross annotation uses the hard labels obtained from the predictions of Model and Modelg to re-annotate D and Dzz. We present an alternate

---

approach that extends cross-annotation to use soft labels (i.e., the entire output distribution).
First, if a token w; E (for s E DA) is anno- tated as an entity e, we know that the annotated label the true label. In this case, gi(s)[e] = 1, and (s()]ee] - 0 Ve' + e. Second, if wy is not annotated as an entity (i.e., is assigned an O label), then we know that it does not belong to any entity in ε Therefore (s({ee)) 0 Ve Ex. However, wi could potentially be an entity Σg. So, we just need the probability distribution over Ep U (0), which we can directly estimate by using M. Analogously, for 8 E DB, we can use ModelB to estimate the distribution over εΛ. With these approximations, Equation can be split into two terms corresponding to the two cases above. For the first case, the loss is simply cross-entropy loss against the one-hot vector obtained from g (s). This gives the proposed loss function of PLM.
As in cross unnotation, we start with the obser- vation that the observed label for a token is not necessarily the true label, and conversely, the true label is unobserved. Thus, we treat the the true label as latent variable. First, to simplify the discussion, assume that the distribution of this latent variable is known and then solve for the opti- mal parameters of the model. Later, we relax this assumption and discuss how to approximate the distribution of this latent variable.
Let us denote our desired final model by Modelfinal: parameterized by 0 as f((()). The out- put of f()) probability distribution over the entity labels 'fimal = EA U Eg, for each token in the input and the output for 7t token is denoted by f.(.). We use a BERT-based sequence tagging model for (-) ($6.1).
Loss(s|0) Σ CE (E( vi)
First, let us calculate the likelihood for a sin- gle example s DA U DB consisting of n to- kens, = [w1, W2, ... Wn]. and its correspond- ing observed labels y [y1, ................................ vn). We denote the corresponding (latent) true labels as = [Z1, E2, Zn] and predicted labels as [Y, Y,.,.,, Yu] with Y; ~ f(((d) and Z; ~ (i()). Here g() is an oracle function that gives us the distribution of true labels. The likeli- hood of the predictions matching the true underly- ing label, P(Y = Z)0), is then calculated
Σ log [[(s))), gi(s) ) (2)
+
The loss term in Equation is similar to the knowledge distillation loss, where the second term is replaced by KL-divergence term:
PLM-KL:
Loss(s|0) = Σ
CD fa(ss(), wi)
CE
zy==O
Σ k0
(3)
This loss term has been used in prior work on tinuous learning in NER (Monaikul et al., 2021) and it simulates a student-teacher framework with g as the teacher, and f as the student. We prove, using a simple application of Jensen's inequality, that the loss in Equation 3 is an upper bound of the loss in Equation 2 (Appendix B). In $6, we experi- ment with both the exact loss function of PLM well the upper-bound of PLM-KL.
P(Y 0 inal) II P(Y Z.).
Further, P(Y 7:) can be decomposed as
P(Y: 2,) Σ fi(s)0]ee] x (()))e
e
= =(N((s00), a((s)
5 Revisiting Taxonomy Expansion
Finally, the negative log-likelihood yields the loss:
5.1 Definition
n Loss(s)0) = Σ log (i(()))) s((s) )
(1)
In $3, following prior work, we assumed that all the entity types are distinct and disjoint. We now
Approximating (.) Equation| 1 assumes access an oracle function 9(-) that gives us the distribu- tion of the true labels. In practice, this is exactly the information that we do not have access to. Thus, we approximate using the predictions model, based on two key observations.
More preeisely, token not annotated with entity type (e.g. PERSON),
combination of tag and the type (e.g. B-PERSON). We drop approach works identically regardless the tags.
tags for simplicity, but the
Model gives us the probability of not belonging to any of the Ep. but since we already know that the token does not helong to Es, the prohability of not belonging to En is equal to the probability of not belonging to E and Eg.

---

the SUBTYPE/SUPERTYPL example in Figure 3a. where a mention can belong to both (PERSON, ACTOR) or (PERSON, POLITICIAN), or just PER- SON (i.e. PERSON, but neither ACTOR nor POLITI- CIAN). Given this, we cannot directly train an NER model over EA U EB. Instead in final. we define entity type for each possible combina- tion (illustrated in the grey box in Figure 3a), and train model over the redefined output labels. For instance, we introduce three new entity types in the final label set corresponding to the PER- SON entity for the three cases discussed above- PERSON:ACTOR PERSON: PERSONNPOLITIE POLITICINN, and PER- SON:O). Similarly, for the OVERLAPPING case, we define three new types LOCATION: (entities that are locations only according to DA). LOCA- TION:B (entities that are locations only according to DB), and LOCATION:A, (entities that are loca- tions according to both D and DB).
(Crgartzzion, Locaton
Locatiny
Oigentation
Subnye/SSperrype
Overlapping
Lacation
Localin
Mourtans
Coeara
Finel Estlyy Typec: (Prgintaaiion Numbers, Persor Aetor Avrun Pobricas Awnon O. Actttonn Locatiec,, inceisesuut
(a) Illustration of redefined output space
Robert stured the thos esc
Origical crnotattr:
Freess
ACD3
PamossAcAss
in movi
(b) Illustration of allowed entity types
Figure 3: For the non-disjoint case, the output space of the combined entity types is not simply a union of the two original entity types. 3a illustrates how the final entity types look like given the original types and the relations between them. 3b shows the allowed entity types in the redefined output space for each token in the dataset, given their original annotations.
5.2 Allowed Entity Types
Before discussing the modifications to the methods for the non-disjoint case, we revisit the assumption that drove the methods in the disjoint case we observed that if token in D is annotated O, it belongs to one of εB U {0), else it belongs the annotated entity type ($4.1). However, this assumption does not hold in the non-disjoint setting. In Figure 3b, James is annotated as PERSON D A. so it can belong to one of (PERSON:ACTON PERSON: POLITICIAN, or PERSON:O)) in the final output space. Thus, for each token in the datasets. we define set of allowed entity types that the token can belong to. These allowed types are determined by the observed annotation of that token and combination of existing entity types or the newly- introduced types as discussed above. Figure 3b gives more examples of allowed entity types.
extend our definition of the problem well as methods to a more general version which allows for semantic overlap the entity types. Specifi- cally, we assume that along with AA, DB, E,, and Ep, we also know the relationship R, between the entity types (E = EA EB),, where R 8 & (DISIOINT, SUBTYPE, SUPERTYPE, OVERLAP- PING]. The SUBTYPE/SUPERTYYE relations allow for an entity in εg (e.g. Politician) to be subtype of an entity in A (e.g. Person), and vice versa. The OVERLAPPING relation allows for partial overlaps in the definitions of types (c.g. if both ε and ε have LOCATION type, but only a subset of LOCA- TIONS are common to both definitions)." Figure 3a illustrates these possible relationships.
5.3 Modifications to Proposed Methods
Given this mapping of the problem to detecting the entity types in redefined output space, the modification to the methods from $6.3.1 lies in ② from Figure 2, where instead of annotating a token with predictions from a model, we simply constrain this annotation by the allowed entity types for that token. The rest of the steps proceed as before. We defer further details to Appendix C.
Output: As in $3, our aim is train a model to recognize both С and Eg. However, now men- tion can get more than one entity label. Consider
WWe ignore the trivial case when an entity in E is exactly identical to Ex. Without loss of generality, we also assume that entities within &A (En) are disjoint; even if they are not, we can always convert them to disjoint.

---

Method
Ontonotes
FewNERD-SSupe
FewNERD-Sub
WNUT17
JNLPBA
I2B2
Naive Join
76.3 (2.1)
68.4 (2.1)
(1.6)
(3.9)
(3.5)
76.5 (2.8)
CL.
87.7 (0.2)
77.7 (0.1)
66.1 (0.2)
(1.3
(0.5)
91.2 (0.3)
AML
87.4 (0.3)
77.9 (0.1)
65.6 (0.2)
38.7
(0.4)
90.6
X-Ann.
87.8 (0.2)
78.3
(0.1)
41.3
(2.3 (0.5)
PLM-KL
88.2 (0.2)
78.4 (0.1)
(0.1)
(1.1
(1.5
92.5 (0.2)
PLM
88.1 (0.2)
78.4 (0.1)
67.2 (0.1)
43.3
(1.5)
92.3 (0.3)
Upper Bound
88.7 (0.2)
78.6(0.11
67.4 (0.1)
45.3 (0.8)
71.8 (0.4)
93.3 (0.1)
Table I: Results for the disjoint setup (mean and std. dev. micro-F1 across 25 runs). PLM, PLM-KL, and cross annotation are competitive across the board and close to the upper bound.
6 Experiments
Modified Continual Learning (CL++): Default CL does work when the entity types not disjoint. CL++ is a modified version of CL for our non-disjoint setup to account for allowed entity types.
6.1 Datasets, Sctup, and Hyperparameters
We study TE-NER using datasets covering diverse domains, entity types, and sizes-(1) Ontonotes (Weischedel et al., 2013) (2) WNUTI7 (Derczynski et al., 2017) (3) JNLPBA (Kim et al., 2004) and (4) 12B2 (Stubbs and Uzuner, 2015). Since these datasets are fully annotated, cannot them directly to study TE-NER. Instead modify each dataset to obtain partial annotations. For the non- disjoint setup, we only use the FewNERD dataset. We defer more details to Appendix A.
Adjusted Multilabel (AML): The AML base- line treats the NER problem as multi-label classi- fication (with sigmoid loss) problem for each token, instead of multi-class in standard sequence tag- ging approach. However, instead of all entities, the loss for multi-label considers the loss for only the allowed entity types.
Upperhound: Since DA and DB are derived from fully annotated datasets D, where every ex- ample is annotated for Efina- we train our model on the original, unmodified dataset D and use it hypothetical upperbound. In real scenarios, we do not have such fully annotated datasets.
For all experiments, we finetune BERT-base as our backbone models for all experiments with the exact setup from Devlin et al. (2019). We re- peat every experiment with 5 random splits of DA and D3 and 5 different seeds (for training) for each split and report mean and standard de- viation of micro-Fi scores averaged across all 5 runs. For each dataset, we use the vali- dation data choose the best learning rate from (5e le 5 2e 3e 5 5e $].
6.3 Research Questions and Experiments
We focus on three key research questions:
1. Is PLM more accurate than other approaches given sizeable training data (DB) for the new entity types En?
6.2 Baselines and Upperbound
2. Is PLM more accurate than other approaches given little training data (Dp)? Does PLM show robust performance when validation data is not exhaustively annotated?
Naive Join: A naive solution to TE-NER is to combine the two partially annotated datasets D and D3 and train model on this combined dataset using cross-entropy loss. We expect this approach perform poorly, but highlights the severity of issues caused by partial annotation.
6.3.1 Accuracy of PLM given sizeable DB We answer our first research question via experi- ments that focus on three different scenarios for TE- NER DISJOINT ($3), SUBtYPeSS/UPERTTYER and OVERLAPPING ($5.1).
Continual Learning (CL): This baseline uses knowledge distillation similar to Monaikul et al. (2021) and Xia et al. (2022). Briefly, first train ModelA teacher, then we train the student model (Model finat) on Bi a token annotated EB, we use the cross- coos-yy loss, else we calculate KL divergence with respect to the teacher's output.
Disjoint Entity types Table shows the results of all methods on the disjoint setup. First, Naive Join performs significantly worse than all other methods, with a drop of as much as 16 Fl the case of JNLPBA. This is an expected outcome,
E

---

Method
PER.
Loc.
ORG.
PROD.
PER.
Loc.
ORG.
PROD,
CL+
73.4 (0.2)
76.2 (0.1)
75.6 (0.1)
(0.0
74.7 (0.1)
75.6 (0.1)
75.1 (0.1)
76.8 (0.1)
AMI
72.7 (0.2)
(0.2)
74.5
76.5 (0.1)
74.8 (0.1)
75.6 (0.1)
75.1 (0.1)
76.8 (0.1)
X-Ann.
74.1 (0.1)
(.19
(..3
(0.1) (.11)
76.4 (0.1)
(..9
77.6 (0.1)
PLM-KL
74.1 (0.1)
(0.1)
(0.1)
75.5 (.11
(0.1)
75.1
(0.1)
PLM
74.2 (0.1)
77.0 (0.1)
(0.1)
77.8 77.8 (0.1)
75.6 (0.2)
76.4 (0.1)
76.0 (0.1)
77.6 (0.1)
Upper Bound
(0.1)
(0.1)
76.6 (0.1)
(8.1
76.1 (0.1)
76.9 (0.1)
(0.1)
77.9 (0.1)
Table 2: Results for the SUBTYPE/SUPERTYEE (left side), and OVERLAPPING (right side) setups (mean and std. dev. micro-Fl across 25 runs). Each column indicates the entity type manipulated to create datasets ($6.1).
and it highlights the severity of problem caused by partial annotation. Second, both CL and AMI are more accurate than the Naive Join approach as they approach the upper bound. Next, despite its simplicity, cross annotation offers a very strong solution to this problem. For 4 datasets, cross an- notation is within Fl of the Upper Bound and CL and AML are both behind cross annotation.
of examples in DB is limited, while maintaining the size of DA- A- We only focus cross annotation and PLM as these were most competitive in the experiments in 16.3.1. Further, we focus only on the Ontonotes and FewNERD datasets.
Table 3 indicates that in few-shot settings, PLM consistently outperforms cross annotation across the board. The gap between the two methods more severe as the number of examples reduce, with PLM scoring as much as Fl higher when DB contains only 100 examples for each type on Ontonotes, and 17 F1 higher on FewNERD with 300 examples per type. However, both approaches tend to be very unstable in such low data regimes, indicated by the high variance in results (more vere in Ontonotes). Despite this, PLM exploits the soft labels from the models to achieve better scores.
Finally, both PLM and PLM-KL reach scores that are on par, even slightly better than cross annotation, indicating that they offer alternative solutions to this problem. However, the differences of these methods with cross annotation are very small, and we take a closer look in future sections.
Non-disjont Entity Types: We now turn to the non-disjoint setup ($5), using the FeWNERD dataset ($6.1). Table 2 shows the results of these ex- periments, with the left-side showing the results for the SUBTYPES/SUPERTVYRR case, and the right- side showing the results for the OVERLAPPING case. either case, each column indicates the entity type manipulated to create datasets per Ap- pendix A. We leave out the Naive Join approach given its (expectedly) poor performance in the pre- vious experiment.
6.3.3 Robustness to partially annotated validation data
Experiments far have assumed access fully annotated validation set. In practice, it is likely that even validation sets are partially annotated (similar to training sets). How do models behave with such partially annotated validation sets? Specifically, we assume that we have validation sets, Duve and Dial corresponding to DA and DDs respectively. During the validation step, we evaluate the model being trained (Modeljimal
The observations for this set of results very similar to the previous experiments and also con- sistent across the SUBTY P and OVERLAPPING - cross annotation again proves to be a highly effective solution, both PLM and PLM-KL are also on par with cross annotation, and all three methods approach the Upper Bound.
both Dval and Dyna separately, masking out any predictions from &g on Dval and from ΣA on D We use the aver- age micro-F1 on these individual validation sets as our stopping criterion. Again, we focus on cross annotation and PLM on Ontonotes and FewNERD.
6.3.2 Accuracy of PLM given small DB A reasonable assumption in TE-NER is that the dataset with the new entities (DB) will be much smaller than the initial dataset (DA) since is im- practical to annotate many examples for each new entity type as exist for the old entity types. Our second set of experiments aims to test how various methods perform in scenarios where the number
Table 4 shows that F1 scores with partial valida- tion are similar to those of full validation. Thus, we do not even need validation data labeled with en- tities from both E and ÊB, as partially annotated validation sets are a reasonable proxy.

---

Dataset
Method
100
200
300
500
1000
2000
X-Amn.
24.8 (16.7)
6.2
(6.9 (.11
(1.4)
80.1 (1.0)
OntoNotes
PLM
(24.6)
16.5 (11.8)
(11.6)
60.5 (5.5)
75.8 (1.4)
81.2 (0.9)
X-Ann.
(10.0)
(4.1)
42.6
48.1 (4.3)
55.9 (1.4)
FewNERD
PLM
28.2 (4.0)
25.5
(7.8
54.2 (2.3)
60.1 (0.5)
Table 3: Experiments with varying sizes of DB (mean and std. dev. micro-Fl across 25 runs). Each column indicates the number of examples per type. As [Dp] reduces, PI.M scores higher compared to cross annotation.
Datasct
Val. type
X-Ann.
PLM
Exl: [Ea]
Model
XAnn.
PLM
Full val.
87.7 (0.2)
88.1 (0.2)
9:9
87.7 (2.4)
88.1
88.5 (2.1)
OntoNotes
Partial val.
87.7 (0.2)
88.1 (0.2)
12:6
87.3 (3.1)
87.8 (3.0)
88.3 (2.8)
Full val.
66.8 (0.2)
67.1 (0.2)
15:3
84.8 (8.1)
(8.3)
86.3 (7.1)
FewNERD
Partial val.
66.8 (0.2)
66.7 (0.2)
16:2
86.0 (8.9)
(7.2)
87.8 (6.4)
17:1
72.1 (13.3)
74.2 (12.4)
75.7 (11.9)
Table 4: Fl scores of PIM and cross-annotation are almost identical regardless of whether validation data are partially or fully annotated.
Table 6: The model from PLM recognizes En more accurately than Model As |EAl increases and |Egl decreases, this performance gap increases.
[Ea|: [E
X-Ann.
PLM
9:9
87.8 (0.2)
88.1 (0.3)
12:6
87.7 (0.2)
88.1 (0.2)
$6. Results in Table 5 indicate that even as the difference in number of entity types in E and & grows, cross Annotation and PLM do not diverge in behavior and continue be equally accurate.
15:3
87.7 (0.2)
88.1 (0.2)
16:2
87.7 (0.2)
88.0 (0.2)
17:1
87.9 (0.3)
88.1 (0.2)
Table 5: Results with varying |EAl and [Ee]: |; F1 scores cross annotation and PLM not vary with changes in && and E.
7.2 Performance of Model Vs Model
Another possible solution to TE-NER is to use Modelg to recognize the new entity types from Ep, Model to recognize the original entity types EA, and combine the predictions of these two mod- els via heuristics. However, this begs the question does Model recognizes entities from E well as (or better than) Model final derived from Cross Annotation or PLM? We answer this by testing these methods against Model on a test set anno- tated with just & If Model final predicts a mention as e E EA, we simply ignore it.
6.3.4 Summary of results
Overall, cross annotation and PLM similarly accurate when Dg is large enough. Both methods are also robust when partially annotated validation sets are used instead of fully annotated validation sets. However, as the size of DB reduces, PLM is increasingly more accurate.
7 Discussion and Further Analysis
Having seen the effectiveness of cross annotation and PLMM, we further analyze these methods. For these experiments, we use Ontonotes under the disjoint setting 15..3.1).
Results (Table indicate that (1) PLM and cross annotation are consistently more accurate than Modelg at recognizing entity types from &g. and (2) as |EBl decreases (and |&Al increases), the gap between all methods increases. We hypothesise that this is due to three reasons: (1) Cross Annota- tion and PLM use additional data corresponding to ΣA in dataset D hence their superior performance. (2) The more such additional information (about other entity types) present in DA, the larger the performance gain. Intuitively, this implies that the ability to recognize entity types in E is helping in better recognizing entities of interest in E
7.1 Effect of size of E and Eg
In experiments in $6, EA and Eg were set to have (almost) the same number of entity types. How- ever, more likely scenario is that the number of entity types to be added (EER) are fewer than the number of existing entity types ((EA)). We investi- gate the behavior of methods in such settings - for these experiments we keep the size of the datasets (D, and DR)) to be fixed and identical to those in

---

8 Conclusion
Arabie, L Hubert, De Socte, and A Gordon. 1996. Hierarchical classification. P. Arabie, L. Hubert, G. De Soete, & A. Gordon, Clustering and classification, pages 65-121.
We define and propose solutions for general ver- sion of the problem of taxonomy expansion for NER. Crucially, and unlike prior work, our defini- tion does not assume that the entity types that are being added are disjoint from existing types. We propose a novel approach, PLM, that is theoreti- cally motivated based on a latent variable formula- tion of the problem, and show that prior solutions based on sundent-teacher settings are approxima- tions of the loss arrived at by our method. PLM out- performs baselines on various datasets, especially in data scarce scenarios when there is limited data available for the new entity types. In such settings, it is as much as 10-15 Fl points more accurate than the next best baseline.
Jessa Bekker and Jesse Davis 2020. Learning from positive and unlabeled data: A survey. Machine Learning, 109(4):719 160
Lingzhen Chen and Alessandro Moschitti. 2019. Trans- fer learning for sequence labeling using source model and target data. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 6260-6267.
Eunsol Choi, Omer Levy. Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87-96.
Matthias De Lange, Rahaf Aljundi. Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019, Continual learning: A comparative study on how to defy forgetting in clas- sification tasks. arXiv preprint arXiv: 1XXiv:1900.000 190990833, 2(6):2.
9 Limitations
There are many other extensions of the definition and setup for TE-NER that this work does not ad- dress. For instance, the old and the new entity types / datasets, can belong to different domains and results in such settings are likely to different than those reported in this paper, where both old and new entity types are created from the same pre-annotated dataset. Studying this, however, re- quires creation of appropriate datasets, which is also something that this work does not attempt to do.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
For the more general definition of TE-NER that allows entity types to be related, We assume that we know these relations (i.e. they have been provided by an expert/user) and leave aside the problem of how to identify these relations in the first place. A separate body of work in taxonomy induction aims identify such a hierarchy between entities (e.g. (Snow et al., 2006)).
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recog- nition dataset. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3198-332 32133.
Finally, despite its stronger theoretical founda- tions, the key method proposed in this work, PLM, is not more accurate than the simple cross anno- tation bascline in data rich scenarios, and is more suitable for data scarce scenarios. Further investi- gation is required boost results for the former scenario.
Edouard Grave 2014. Weakly supervised named entity classification. Workshop on Automated Knowledge Base Construction (AKBC).
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in neural network. arXiv preprint arxiv 503.02531,, 2(7).
References
Zhamming Jie, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. 2019. Better modeling of incomplete anno- tations for named entity recognition. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Abhishek Abhishek, Amar Prakash Azad, Balaji Gane- san, Ashish Anand, and Amit Awekar. 2019. Collec- tive learning from diverse datasets for entity typing in the wild.

---

Human Language Technologies, Volume (Long and Short Papers), pages 739.734, Minneapolis, Min- nesota. Association for Computational Linguistics.
Nicky Ringland. Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris, and James R Curran. 2019. Nne: A dataset for nested named entity recognition in english newswire. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 5 5176-5181.
Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. Introduction to the bio-entity recognition task jnipba. In Proceed- ings of the international joint workshop on natural language processing in biomedicine and its applica- tions, pages 70-75. Citeseer.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. 2017. Incremental learning of object detec- tors without catastrophic forgetting, In Proceedings of the IEEE international conference on computer vision, pages 3400-3409.
Xiao Ling and Daniel S Weld. 2012. Vine-graines entity recognition. In Twenty-Sixth AAAI Conference on Artificial Intelligence.
Carlos N Silla and Alex A Freitas 2011. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1):31-72.
Jie Ma. Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 1956- 1971, Dublin, Ireland. Association for Computational Linguistics.
Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous ev- idence. In Proceedings of the 21 International Con- ference on Computational Linguistics and 44th An- nual Meeting of the Association for Computational Linguistics, pages 801-808, Sydney. Australia. Asso- ciation for Computational Linguistics.
Khai Mai, Thai-Hoang Pham, Minh Trung Nguyen, Tuan Due Nguyen, Danushka Bollegala, Ryohei Sasano, and Satoshi Sekine. 2018. An empirical study on fine-grained named entity recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 711-722, Santa Fe. New Mexico, USA. Association for Computational Linguistics.
Amber Stubbs and Özlem Uzuner. 2015, Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. Journal of biomedi- cal informatics, 28:S20-S29.
Stephen Mayhew, Snigdha Chaturvedi, Chen- Tsse-TTeeTe Tsai, and Dan Roth. 2019. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLLJ), pages 645-655, Hong Kong, China. Association for Computational Linguis- tics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, al. 2013. Ontonotes release 5.0 1d19013119. Linguistic Data Consortium, Philadelphia, PA, 23.
Yu Xia, Quan Wang, Yajuan Lyu, Yong Zhu, Wenhao Wu, Sujian Li, and Dai Dai. 2022. Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples. In Findings of the As- sociation for Computational Linguisticx: ACL 2022, pages 2291-2300, Dublin, Ireland. Association for Computational Linguistics.
Yu Meng. Jiaming Shen, Chao Zhang, and Jiawei Han. 2019. Weakly-superviss hierarchical text classifi- cation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6826-6833.
Natawut Monaikul, Giuseppe Castellucci, Simone Fil- ice, and Oleg Rokhlenko. 2021. Continual learn- ing for named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 33(05):1370
A Dataset creation process
We create partially annotated datasets from fully annotated dataset D with entity types & by first splitting the & into two (approximately) equal sub- sets (yielding & and Ei), and then repeating this with the examples in D (yielding DA and DB). Additionally, for the examples, we also scrub the annotations corresponding to the complementary entity types (e.g., remove entity annotations corre- sponding to ΣΛ from DB).
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly supervised named entity recognition using positive-unlabeled learning. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 2409- 2419, Florence, Italy, Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training intermediate labeled-data tasks. ArXiv, Ahs/1811.0100
For validation and test data, we use the corre- sponding fully annotated development and test sets as it is respectively. However, the assumption of fully annotated validation data being available is unrealistic, and we return to this in $6.3.3
Lance Ramshaw and Mitch Mareus. 1995. Text chunk- ing using transformation-bae based learning. In Third Workshop on Very Large Corpora.

---

Datasets for Non-disjoint TE-NER We study the non-disjoint version of the problem ($5.1) us- ing the FewNERD dataset (Ding al., 2021). This dataset consists of two levels of annotations corre- sponding to coarse-grained supertypes (e.g., PER- SON) and fine grained subtypes (e.g., ACTOR).
Using Jensen's inequality for concave functions (like log x)), we get log E[a] 2 E'log æ/. Hence,
Loss(s
Σ
CE (1(()))), mi)
====
Σ
log(f)(s10)), gi(s)
To study the SUBTYPE/SUPERTYE: scenario, after we split the garse- entity types into A and Eg as above, we randomly add a subset of the subtypes corresponding to coarse-grained entity type e E E to Eg. For instance, if the PERSON entity type is present in EA, we add subset of its eight subtypes (POLITICIAN. ACTOR, ARTIST) etc.) En.
<
Σ
CE (i((s))), 8i)
=x==
log( fi($|0)
Σ
9i(s)
ge(s)
Σ
CE (i((s))), Vi)
Σ KL KL gi(())
[I
(5)
To study the OVERLAPPING scenario, we first choose a coarse-grained entity type (e.g. PERSON), split its fin-grained subtypes into two overlapping subsets SA and Sy (e.g. SA ACTOR, POLITI- CIAN, ..., and SB ACTOR, ARTIST, ...). Then, if mention in DA annotated with subtype in SA we assign it new entity type PERSONA. Similarly, if a mention in DB is annotated with a subtype in SB, we assign a new entity type PERSONB. g- PERSON, y and PERSON y are added to A and E respectively.
C Modifications to proposed methods
C.1 Modified Cross Annotation
Recall that the idea behind ross-annotation is model predictions tions. For the non- non-disjoint case, if token in D is annotated as e, then we first get its allowed entity types following $5.2. If this set of allowed enti- ties contains just one element, then we annotate it as the allowed entity type. Otherwise, we use Model to chose the best label among the allowed entities. For the example in Figure 3b, James is annotated as PERSON in DA, allowed entity types are PPERSON:ATTON PER- SON: POLITCIAN,,, PERSON:O }. We use Model to get the probability of ACTOR, POLITICIAN and O and among them, choose the one with highest probability. After re-annotating both DA and DB. the rest of the steps proceed as in $4.1.
proxy for actual annota-
know that its
B Proof of lower bound
To show that the loss term for PLM in Equation is a lower bound on the Kl.-divergence loss used in prior work (Bquation 3), we start with the loss in Equation 2.
C.2 Modified PLM
For PLM, we are given the distribution of truc labels (g)(s)), the likelihood calculation does not change, therefore the loss in Equation remains the same for the non-disjoint case. However, simi- lar to cross annotation above, the estimation of the oracle function g(.) changes. For a data point in DA (similarly for Dix), if token is annotated as c, need g()) to give probability distribution over its allowed entity types (the probability of other en- tity types is 0). Therefore, we compute softmax over the logits of the allowed entity types. Fig- ure 3b, James is annotated as PERSON, so we use Model to get the logits of ACTOR, POLITICIAN,
Loss(s|0)= - - -- CE [i(s]0), u))
:===
Σ
log f((s)0), 9i(s)) ) (4)
#0
*For the disjoint version of the problem (6).3.1), we simply treat these two different levels of the dataset as two different datascts - FowNERD-Super and FewNERD-Sub.

---

and O, and then take softmax over just these (the
probability of other entities in Fonal 0)


Reviews:
[{"review_id": "bLBxHmBo0u", "reviewer": "Reviewer_xZ9p", "summary": "", "questions": "1. Does your method also consider the dynamic scenario where the number of categories continuously increases? Does the model experience the problem of forgetting during the continuous learning process?\n2. In the experiments, is the sample category distribution kept balanced? How stable and robust is the model when the categories are imbalanced?\n3.  If the performance of PLM is similar to simple cross-annotation, then where lies the advantage of PLM? Are there any comparison experiments in complex scenarios?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. \n\n* “category relationship judgment module” : We would argue that it is not necessary to model this problem. This can be a simple input from users. Identifying such relationships is a far simpler task than fully annotating all datasets for all types.\n\n\n \nAnswers for the questions: \n1. A simple solution to the dynamic scenario is to train the models incrementally – first train a model using PLM to recognize entity A and B, call it Model-AB, then train a model C to recognize entity type C, now again use PLM to train the model-ABC. In our experiments, this s the model’s probabilistic annotations are similar to gold annotations. In data scarce scenarios, this may not work, because of error propagation; instead we can train Model-AB, Model-BC and Model-CA and use then use them to (probablistically) annotate Dataset C, A and B respectively and then train the final model-ABC. However, the caveat with this approach is scalability. We would be happy to discuss and include preliminary results in the appendix of the next version of the paper, with both these approaches. \n2. In Table 5 / Section 7.1, we analyze the effect of difference in the number of entity types in set A and B, which naturally leads to an imbalance in the number of mention in the two sets. The results are very robust and an imbalance in the distribution across the two sets does not lead to a performance degradataion\n3. While PLM and cross-annotation perform similarly in data abundant scenarios, PLM is more accurate in data scarce scenarios. Furthermore, we expect that PLM could lead to a potential solution to minimize error propagation or catastrophic forgetting issues of continual learning (basic idea of the solution is discussed in 1), we left it for the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A Partial Label Model (PLM) method based on latent variable modeling has been proposed. PLM uses individually trained models to provide soft labels for another dataset, and then trains the final model. This method is quite reasonable.\n\n2.  Compared to other methods, it is shown that PLM performs well when there is limited data for new categories, and PLM is also suitable for situations where categories overlap.", "reasons_to_reject": "1. The paper assumes that the relationship between new and old categories (non-overlapping, subclasses, etc.) is known. However, in reality, a category relationship judgment module is also needed.\n2. The theoretically more optimized PLM algorithm is similar in performance to simple cross-annotation and does not show a significant advantage in large data scenarios.", "questions_to_authors": "1. Does your method also consider the dynamic scenario where the number of categories continuously increases? Does the model experience the problem of forgetting during the continuous learning process?\n2. In the experiments, is the sample category distribution kept balanced? How stable and robust is the model when the categories are imbalanced?\n3.  If the performance of PLM is similar to simple cross-annotation, then where lies the advantage of PLM? Are there any comparison experiments in complex scenarios?", "paper_topic_and_main_contributions": "\nThis paper presents a comprehensive study on the taxonomy expansion problem for named entity recognition. The authors formally define the expansion problem allowing overlap between old and new entity types, and propose the Partial Label Model (PLM) algorithm based on partially labeled modeling. Experiments show that PLM can improve over baselines in low-resource scenarios, and achieve competitive results even with complex subtype and overlapping relations between new and old types. Analysis demonstrates models trained with PLM recognize new types more accurately than those trained only on the new type data, especially when the number of new types is small.  In summary, the paper provides a formal definition for the NER taxonomy expansion problem, and proposes a theoretically grounded PLM solution that shows significant advantages in low-resource settings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vhMT6Hyomz", "reviewer": "Reviewer_nj9h", "summary": "", "questions": "1. Why are the experiments performed separately for subtype/supertype and overlapping relations among entity types?\n2. Why the choice of baseline methods was kept limited?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback.\n\n* “Limited baseline”: Please see the response to question 2 below. \n\n* “manual intervention”:  In cases of thousands of labels, we can probably use LLM if the label names are meaningful, or even without any external knowledge, we can probably estimate the relation based on the model’s predictions and gold annotations. Regardless, the effort to identify such relation is still significantly smaller than actually annotating the data for all the entity types, and reducing such effort is one of the goals of the paper. \n\n* “Limited set of entity types”: We wonder why the reviewer thinks the set of entity types is limited. FewNerd consists of 8 coarse-grained entity types, and 66 fine-grained types. Ontonotes consists of 18 entity types, while i2b2 also consists of 18 entity types spanning PII entities in the medical domain. In fact, we consciously included multiple datasets from different domains to ensure that out conclusions are broadly applicable, so we would like more clarity on the specific limitations. \n\n* “Related Work” : Thank you for pointing us to a related paper, we would be happy to cite and discuss their approach in related works. However, we would like to point out that there are also  significant differences with that work. In particular, we propose a theoretically grounded probabilistic approach, whereas their approach is similar to CL++  (not exactly same but comparable; snippet from the reference “From this normalized distribution, we select a label which has the highest probability and is also a member of the mapped labels Ym. We assumed the selected label to be correct and propagate the log-likelihood loss.” )\n\nBelow are the answers to your questions. \n\n1. We consider subtype/supertype and overlapping separately because it is a relatively controlled set of experiments with results that are easier to analyze. If we combine both cases, then in each experiment we have two categories like PER for subtype and LOC for overlapping. In addition, there is also randomness in how we create overlapping categories in LOC. Visualizing all of them along with different solutions could become quite challenging. Nonetheless, our solution is applicable in combination too, and we expect it to work as well as it does in individual cases. \n\n2. The exact problem definition of our work is novel, even though it is similar to problems in continual learning space (which we compared), it's not exactly the same. Moreover, the seemingly simple baselines we choose, indeed performs very well on the task on a large data scenario. If you can further point out in what ways the baselines are limited, we would be happy to discuss this in future iterations of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Relevant problem\n", "reasons_to_reject": "1.\tLimited set of baseline methods was selected\n2.\tExperiment setup is basic and not matching with the ideal situation. Why the experiments are performed separately for subtype/supertype and overlapping relations among entity types?\n3.\tLimited set of entity types is considered. Perhaps, that is the reason, even the basic X-Annot. method has similar performance as with other methods.\n4. Manual intervention to get the final sets of entity types if relations among them exist. It may not be a good idea when entity types increase from 100s to 1000s.\n", "questions_to_authors": "1. Why are the experiments performed separately for subtype/supertype and overlapping relations among entity types?\n2. Why the choice of baseline methods was kept limited?\n", "paper_topic_and_main_contributions": "Several tasks in NLP and other areas often face the challenges of handling a greater number of classes than any single labeled dataset provides. This paper focuses on the problem of taxonomy extension or increase in the number of classes for Named Entity Recognition (NER) task. Toward this objective, it proposes models for two distinct scenarios:\n1.\tEntity types coming from different datasets are disjoint. For this scenario, they suggested two solutions. The first solution is based on cross-annotation (X-Ann.), where models trained on one dataset is used to get annotation of additional entity types for the other dataset and vice-versa. Then the combined set of re-annotated datasets is used to train the final model. X-Ann. approach is quite trivial and intuitive. The second solution (Partial Label Model or PLM) treats the prediction of additional entity types as soft-label. In the PLM framework, a modified loss function resembling the student-teacher framework is used.\n2.\tEntity types in different datasets may be related. In this scenario, entity types present in different datasets are assumed to be related. In particular, four relations between a pair of entity types were considered: disjoint, subtype, supertype and overlapping. Based on these relations, entities are relabeled with a modified set of final entity types. The final set of entity types, however, is made flat, even if one of the datasets has hierarchical labels.\n\nThe experiments are performed on six datasets covering diverse domains and entity types.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Authors may want to check with the following related work, where entity type expansion is considered with the assumption that complex relations among types may exist.\n\nAbhishek, A P Azad, B Ganesan, A Anand, A Awekar. Collective Learning From Diverse Datasets for Entity Typing in the Wild. 2nd International Workshop on Entity Retrieval (EYRE), CIKM 2019. [https://ceur-ws.org/Vol-2446/paper3.pdf]"}}, {"review_id": "ZlNoaTGxng", "reviewer": "Reviewer_CUSJ", "summary": "", "questions": "Question A: Your examples are always (probably realistically) on the way to more specification of classes. Have you considered the possibility of ambiguating? That is, having writers, actors and directors in an initial annotation, and people in a second corpus, could you convert everything just to people?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\nIn the final predictions, if the model predicts “actor”, we can still deterministically assign this entity as “person”, i.e, by predicting “actor” we do not lose any information since “actor” is the more specific tag. In general, if annotations go from specific to generic, we can always do such a deterministic mapping.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The issue is very well explained, the methods are robust and provide good results.", "reasons_to_reject": "I can see none.", "questions_to_authors": "Question A: Your examples are always (probably realistically) on the way to more specification of classes. Have you considered the possibility of ambiguating? That is, having writers, actors and directors in an initial annotation, and people in a second corpus, could you convert everything just to people?", "paper_topic_and_main_contributions": "This paper is about extending named entity annotation with newer categories, called \"taxonomy expansion\". It proposes a method that composes two annotated resources to provide a better system for annotation with the merge of the categories. It provides extensive comparuison with other methods, as well as several empirical tests for the methods it suggests, on the size of the second corpus and on partial annotation.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": " "}, "extra_text": {}}, {"review_id": "tlTGtypcJD", "reviewer": "Reviewer_451n", "summary": "", "questions": "- Question A: Are the results reported token- or entity-level micro F1?\n\n- Question B: This is more a suggestion, but there was no entry field for suggestions, so I put it here: Due to the way the dataset splits are constructed from a single underlying fully annotated dataset, the results reported in this paper present an optimal case where the underlying distribution of entity labels is the same for the two datasets DA and DB. As the authors acknowledge in the introduction, this is typically not the case, as the \"old\" dataset DA is often sampled with respect to the original entity type set, and the new dataset DB is (most likely) focused on the new entity type set, even if it is drawn from the same general 'domain', e.g. news text. It would have been interesting the see results for this scenario as well, e.g. by using ConLL-03 (news) and Ontonotes (news + dialogue). Did you do any experiments in this direction?\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. All typos/presentation comments will be addressed in the next version of the paper. \n\nAnswers to questions: \n1. We use default “seqeval” evaluation metrics so these would be entity-level metrics. We will clarify this in the next version of the paper. \n2. Yes, that is an interesting experiment to analyze and we considered it but left it out of scope (see Limitations section). However, one of the major challenges is the test set. Say we have Domain A data annotated with entity types A and Domain B data annotated with entity type B. To test our approach we need to reliable test set set in Domain B annotated with both entity types A and B; such data is relatively harder to find. A possible workaround could be to remove some of the common entity types like PER, LOC from a dataset and this forms the Domain B and use a dataset from a different domain for those common entity types. However, there could still be differences in annotation guidelines across different datasets, so the validity of findings would be slightly questionable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting problem and a real-world issue, with the proliferation of “disjoint” datasets that often need to be combined\n- paper is well written and easy to follow \n- interesting extension to 'hard' cross-annotation \n- experiments are extensive, and results are convincing", "reasons_to_reject": "- I can't find any", "questions_to_authors": "- Question A: Are the results reported token- or entity-level micro F1?\n\n- Question B: This is more a suggestion, but there was no entry field for suggestions, so I put it here: Due to the way the dataset splits are constructed from a single underlying fully annotated dataset, the results reported in this paper present an optimal case where the underlying distribution of entity labels is the same for the two datasets DA and DB. As the authors acknowledge in the introduction, this is typically not the case, as the \"old\" dataset DA is often sampled with respect to the original entity type set, and the new dataset DB is (most likely) focused on the new entity type set, even if it is drawn from the same general 'domain', e.g. news text. It would have been interesting the see results for this scenario as well, e.g. by using ConLL-03 (news) and Ontonotes (news + dialogue). Did you do any experiments in this direction?\n\n", "paper_topic_and_main_contributions": "The paper addresses the problem of taxonomy expansion for NER, i.e. increasing the set of entity labels that the NER model should recognize, under the assumption that re-annotation of the original dataset is impractical, and instead a different dataset with annotations for the novel entity labels is available. To address this problem of training on two disjoint, partially labeled datasets, the paper proposes a variant of cross-annotation that uses the output distribution as soft labels and a latent variable formulation of the loss. The authors also introduce a novel variant that allows for semantic overlap of the entity sets (e.g. sub/supertypes). Experiments on 6 datasets show that their approach is on-par with/outperforms standard cross-annotation, and is especially beneficial in low-resource settings with few examples per entity type. (main contribution are NLP engineering experiment / approach for data efficiency).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 13, whitespace between PLM and consistency\n- line 21, ‘a more cost effective approach ‘  \n- line 463, ‘Non-disjoint’    \n- line 606, ‘we’    \n- line 786. end-of-sentence period"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the task of NER with newer categories, named taxonomy expansion. The difficulty is how to train the NER model with two disjoint, partially labeled training sets. To this end, the paper proposed a new approach that employs a modified loss function to capture the relatedness between the original labels with the new ones. The experimental results on six datasets show the proposed methods could cover more domains and types. The paper is clear and easy to follow. The cross-annotation idea is interesting.
Decision: Accept-Main