Enhancing Low-resource Fine-grained Named Entity Recognition by Leveraging Coarse-grained Datasets
Authors: Su Ah Lee
Seokjin Oh
Woohwan Jung
Venue: EMNLP 2023 Main (2023)
Keywords: Named Entity Recognition
Fine-grained NER
Low-resource scenario

Abstract: Named Entity Recognition (NER) frequently suffers from the problem of insufficient labeled data, particularly in fine-grained NER scenarios.
Although $K$-shot learning techniques can be applied, their performance tends to saturate when the number of annotations exceeds several tens of labels. 
To overcome this problem, we utilize existing coarse-grained datasets that offer a large number of annotations.
A straightforward approach to address this problem is pre-finetuning, which employs coarse-grained data for representation learning.
However, it cannot directly utilize the relationships between fine-grained and coarse-grained entities, although a fine-grained entity type is likely to be a subcategory of a coarse-grained entity type. 
We propose a fine-grained NER model with a Fine-to-Coarse(F2C) mapping matrix to leverage the hierarchical structure explicitly. 
In addition, we present an inconsistency filtering method to eliminate coarse-grained entities that are inconsistent with fine-grained entity types to avoid performance degradation.
Our experimental results show that our method outperforms both $K$-shot learning and supervised learning methods when dealing with a small number of fine-grained annotations.

Enhancing Low-resource Fine-grained Named Entity Recognition by Leveraging Coarse-grained Datasets
Su Ah Lee", Seokjin Oh", and Woohwan Jung!?
'Dpaarment of Applied Artificial Intelligence, Hanyang University
'Ramply Inc.
(sue991, seokjinoh, (hjjung)@hayyang.aa kc.
Abstract
detailed information extraction. Nonetheless, de- tailed labeling required for large datasets in the context of fine-grained NER presents several sig- nificant challenges. It is more cost-intensive and time-cossuming than coarse-grained NER. In addi- tion, it requires high degree of expertise because domain-specific NER tasks such as financial NER (Loukas et al., 2022) and biomedical NER (Sung et al., 2022) require fine-grained labels. Thus, fine- grained NER tasks typically suffer from the data scarcity problem. Few-shot NER approaches (Ding et al., 2021) can be applied to conduct fine-grained NER with scarce fine-grained data. However, these methods do not exploit existing coarse-graine datasets that can be leveraged to improve fine- grained NER because fine-grained entities are usu- ally subtypes of coarse-grained entities. For in- stance, model knows what Organization is, it could be easier for it to understand the concept of Government or Company. Furthermore, these meth- ods often experience early performance saturation, necessitating the training of a new supervised learn- ing model if the annotation extends beyond several tens of labels.
Named Entity Recognition (NER) frequently suffers from the problem of insufficient labeled data, particularly in fine-grained NER scenar- ios. Although K-shot learning techniques can be applied, their performance tends to saturate when the number of annotations exceeds sev- eral tens of labels. To overcome this problem, we utilize existing coarse- caarse-rained datasets that offer large number of annotations. A straight- forward approach to address this problem is pre- finetuning, which employs coarse- coarse-grained data for representation learning. However, it cannot directly utilize the relationships between fine- grained and coarse- garsse-rrained entities, although fine-grained entity type is likely be sub- category of a coarse-grained entity type. We propose a fine-grained NER model with a Fine- Io-Coarse(F2C) mapping matrix to leverage the hierarchical structure explicitly. In addition, we present an inconsistency filtering method to eliminate coarse-grained entities that are incon- sistent with fine- grained entity types to avoid performance degradation. Our experimental re- sults show that our method outperforms both K K-shot learning and supervised learning meth- ods when dealing with small number of fine-grained annotations. Code is available at https://gi thub. thub. cww/sue991 /CoF iNER.
A pre-fineee rreffinetunig strategy (Aghajanyan et al., 2021; Ma et al., 2022a) proposed to overcome the aforementioned problem. This strategy first learns the feature representations using a coarse- grained dataset before training the fine-grained model on a fine-grained dataset. In this method, coarse coarse-grained data are solely utilized for repre- sentation learning; thus, it still does not explicitly utilize the relationships between the coarse- and fine-grained entities.
1 Introduction
Named Entity Recognition (NER) is a fundamen- tal task locating and categorizing named enti- ties in unstructured texts. Most research on NER has been conducted on coarse-grained datasets, including CONLL'03 (Tjong Kim Sang, 2002), ACE04 (Mitchell et al., 2005), ACEOS (Walker et al., 2006), and OntoNotes (Weischedel et al., 2013), each of which has less than 18 categories. As the applications of NLP broaden across di- verse fields, there increasing demand for fine- grained NER that can provide more precise and
Our intuition for fully leveraging coarse-grained datasets comes mainly from the hicrarchy between coarse- and fine-grained entity types. Because a coarse-grained entity type typically comprises mul- tiple fine-grained entity types, we can enhance low- resource fine- -ne-grained NER with abundant coarse- grained data. To jointly utilize both datasets, we
Major in Bio Artificial Intelligence

---

devise method to build mapping matrix called F2C (short for PFine-t-Coaarse) ) which connects fine-grained entity types to their corresponding coarse-grand coarse-grained entity types and propose a novel approach to train a fine-grained NER model with both datasets.
et al., 2022; Ma et al., 2022b; Wang al., 2022). The objective of this approach to train a model that can correctly classify new examples into one of N classes, using only K examples per class. To achieve this generalization performance, a large number of episodes need to be generated. Fur- thermore, to sample these episodes, the training data must contain significantly larger number of classes than N. In contrast, in our problem con- text, the number of fine-grained classes we aim to identify substantially larger than the number of coarse-grained classes the existing dataset. Consequently, the episode-based few-shot learn- ing approaches are unsuitable for addressing this problem.
Some coarse-graine granse-raainddddeeeee entities improperly match fine-grained entity type because datasets may be created by different annotators for different pur- poses. These mismatched entities can reduce the performance of the model during training. To miti- gate this problem, coarse-grained entities that can degrade the performance of fine-grained NER must be eliminated. Therefore, we introduce filter- ing method called 'Inconsistency Filtering'. This approach is designed to identify and exclude any in- consistent coarse-to-fine entity mappings, ensuring a higher-quality training process, and ultimately, better model performance. The main contributions of our study are as follows:
Leveraging auxiliary data for information ex- traction. Several studies have employed auxil- iary data to overcome the data scarcity problem. Jiang al. (2021); Oh et al. (2023) propose NER models trained with small, strongly labeled, and large weakly labeled data. Jung and Shim (2020) used strong and weak labels for relation extraction. However, in the previous studies, the main data and auxiliary data shared the same set of classes, which is not the case for fine-grained NER with boarse-grained labels. While the approaches of Aghajanyan et al. (2021) and Ma al. (2022a) be applied our problem setting, they utilize aux- iliary data for representation learning rather than explicitly utilizing the relationship between the two types of data.
We propose an F2C mapping matrix to directly leverage the intimate relation between coarse- and fine-grained types.
We present an inconsistency filtering method to screen out coarse- grrsse-graindd data that are in- consistent with the fine-grat fine-grained types.
The empirical results show that our method achieves state-of-the-art performance by uti- lizing the proposed F2C mapping matrix and the inconsistency filtering method.
3 Proposed Method
2 Related Work
In this section, we introduce the notations and de- fine the problem of fine-grained NER using coarse- grained data. Then, we introduce the proposed CoFiNER model, including the creation of the F2C mapping matrix.
Fine-grained NER. NER is a key task in informa- tion extraction and has been extensively studied. Traditional NER datasets (Tjong Kim Sang, 2002; Ritter al., 2011; Weischedel al., 2013; Derezyn- ski et al., 2017) address coarse-grained entity types the general domain. Recently, domain-specific NER has been investigated in various fields (Hofer et al., 2018; Loukas et al., 2022; Sung et al., 2022). The domain-specific NER tasks typically employ fine-graing iineggrineed entity types. In addition, Ding et al. (2021) proposed a general domain fine-grained NER.
3.1 Problem definition
Given a sequence of n tokens X {x1, 22, Fn). the NER task involves as- signing type y: E to each token where E is predefined set of entity types. In our problem setting, used fine-grained dataset DF [(X ...,.., (x= DDF X with a predefined entity type EF. Addi- tionally, we possess a coarse grasee-raaied dataset DC = {(XC, Y(), (X ppp xC
>>))
N-way K-shot learning for NER. Since label- ing domain-specific data an expensive process, few-shot NER has gained attention. Most few- shot NER studies are conducted using an N-way K shot episode learning (Das et al., 2021; Huang
( charac- terized by a coarse- grained entity set E (i.e. EF > LXD A coarse -aarine dataset typically

---

has a smaller number of types than fine-grained dataset. Throughout this study, we use JF and C to distinguish between these datasets. It should be noted that our method can be readily extended to accommodate multiple coarse-grained datasets, incorporating an intrinsic multi-level hierarchy. However, our primary discussion revolves around a single coarse-grained dataset for simplicity and readability.
fine-grained label l E (i.e., Mc - plyf~ sly ()).
Given a fine-grained probability distribution 5 computed using the proposed model, the marginal probability of a coarse-grained type s can be com- puted as
pf = D6fff=sly p 10.
EEE
3.2 Training COFiNER
Thus, the coarse-grained output probabilities are simply computed as follows:
We aim utilize both coarse- and fine-grained datasets directly in single model training. Figure I illustrates an overview of the fine-grained NER model training process using both coarse- and fine- grained datasets. The CoFiNER training process consists of the following four steps:
- P M
(3)
where M RI x| the F2C mapping matrix whose row-wise sum is 1. By introducing this F2C mapping matrix, we can train a single model using multiple datasets with different granularity levels. Manual annotation is straightforward approach that can be used when hierarchical information is unavailable. However, it is not only cost-intensive but also noisy and subjective, especially when there are multiple coarse-grained datasets or a large num- ber of fine-grained entity types. We introduce an efficient method for automatically generating an F2C matrix in $3.3.
Step 1- Training fine-grained model. In the first step, we train fine-gran fine-grained model f (0) with the low-resource fine-grained dataset D-F This pro- cess follows typical supervised learning approach for NER. For training example (X=,, Y²). X is fed into a PLM (Pre-trained Language Model), such as BERT, RoBERTa, to generate contextual representations h, E Rd of each token Zg.
h1 hn] PLM([=T,
Step Filtering inconsistent coarse labels. Although a fine-grained entity type is usually a subtype of coarse-grained type, there can be some misalignments between the coarse- and fine- grained entity types. For example, an entity "Mi- crosoft" in financial document can either be tagged as Company or Stock which are not hierar- chical. This inconsistency can significantly degrade the model's performance.
Then, we apply a softmax layer to obtain the label probability distribution:
P softmax (Wh; b)
where W T | and b R represent the weights and bias of the classification head, respec- tively. To train the model using a fine-graine fine-grained dataset, we optimize the cross-entropy loss func- tion:
To mitigate the effect of inconsistent labeling. we devise an inconsistency filtering method aimed at masking less relevant coarse labels. By automat- ically filtering out the inconsistent labels from the coarse-rrained dataset, grained labels using the finc- inee-grained NER model trained in Step For each token in the coarse- grained dataset, we predict the coarse-grainee label using the fine-grained model and the mapping ma- trix as follows:
Lx n
log p
p [of]]
(2)
investigate the coarse-
where Yi E F is the fine-grained label for the token Ti.
Step Generating an F2C matrix. To fully leverage the hierarchy between coarse- and fine- grained entity types, we avoid training separate NER models for each dataset. Instead, we utilize single model that incorporates F2C mapping matrix that transforms fine-grained output into a corresponding -oaise-g output. The F2C mapping matrix assesses the conditional probabil- ity of coarse-gee -rriined entity type s E E given
C Jf arg Max P
(4)
If the predicted label is the same as the coarse- grained label (i.e., yf = iff), we assume that the coarse-grained label is consistent with the fine- grained one and can benefit the model. Otherwise,

---

Step 1) training process using fine-grained dataset
FG label of
Fine-grained training imput.x" x
FG-Model
organization-company
MMicrosoft new Bing.
Sofuan
FG Probubiliyy
FG predict
Loss Lp
PLM
p
organization-compan
Step training process using grained dataset
CG prediet ORG
Coarse-grained training input.x" Microsoft was founded in 1975.
666 uc
Step 2)
CG Probability
Step 3)
F2C Mapping Matrix
OG label y
consistent Inconsistency Filtering Filter(y)
Loss Le
M E R (xxiF
Filtering predict y ORG
Probability
FG-Model
y
Probability y
label y
ORG
inconsistent.
fine-grained training process
Inconsistency
coarse-grained training process
Filtering
dot product
predict
Filter(y)
Figure 1: Overall process of our proposed method.
we regard the label as inconsistent with fine- fine-grained types and do not utilize the coarsc-graid coarsc-grained grained label in Step 4. Note that the fine-grained NER model is frozen during this phase.
3.3 Construction of the F2C mapping matrix
The F2C mapping matrix assesses the conditional probability of coarse-grainet entity type sE E given fine-grained label & E E (i.c., M p(ff sly +)). As there are only few iden- tical texts with both coarse- and fine- fine-graine rre-ggaaidd an- notations simultaneously, we can not directly cal- culate this conditional probability using the data alone. Thus, we approximate the probability us- ing coarse-grained NER model and fine-grained labeled data as follows:
Step - Jointly training CoFiNER with both datasets.. The model is trained by alternating be- tween epochs using coarse- and fine- mne-grained data. This is an effective learning strategy for training models using heterogeneous datasets (Jung and Shim, 2020).
For the fine-grained batches, CoPiNER is trained by minimizing the loss function defined in Equa- tion (2), as described Step Meanwhile, we use the F2C mapping matrix to generate coarse- grained outputs and utilize inconsistency filtering for boarse-grained batches. Thus, we compute the cross-entropy loss between the coarse grarseegrained la- bel of and the predicted probabilities P when the coarse-grained label is consistent with our model (i.e..9f off) follows:
M = ply = sly==00 ~ p(T =|y = E).
To generate the mapping matrix, we first train coarse-grained NER model fC(() with the coarse- grained dataset DC Then, we reannotate the fine- grained dataset D by using the coarse-graine model (O)).. As a result, we obtain parallel an- notations for both coarse- and fine-grained types in the fine-grained data D.F. By using the paral- lel annotations, matrix C N where each cell Cg is the number of tokens that are labeled as fine-grained type f and boarse-grained type together.
compute the co-occurrence
Le log p [f
Thv
(5)
Because some labels generated by the coarse- grained model (C) can be inaccurate, the co-occurrence matrix by retaining only the top- counts for each fine-grained type and setting the rest to zero. Our experiments show that our model performs best when & = 1. This process effec-
refine
where is length of X For example, suppose that the coarse label of token z; is of =ORG. the estimated coarse label of is ORG, the loss for the token is - log [ff]. Otherwise, it is zero.

---

tively retains only the most frequent coarse-grained categories for each fine-gratn ane-grained type, thereby im- proving the precision of the resulting mapping. Fi- nally, we compute the conditional probabilities for all E E², 8 E EC by using the co-occurrence counts as follows:
K~(K + 50-shot setting, In the K (~(K+))shhr
setting, there are at least K examples and at most K K+5 examples for each entity type. See Ap- pendix A for more details. In our experiments, we sampled fine-grained training data for K 10, 20, 40, 80, and 100.
Cr.
M - = =()= Excrs Excrs Ce.
4.2 Experimental Settings
In experiments, we use transformer-based PLM, including BERT RoBERTaBASE and follow ROBERTaLARGE In COFiNER, RoBERTaLARG to build baseline model. The maximum sequence length is set to 256 tokens.
The F2C mapping matrix M is used to predict the coarse-grained labels using Equation (3).
4 Experiments
The AdamW optimizer (Loshchilov and Hutter, 2019) is used to train the model with a learning rate of 2 and a batch size of 16. The number of epochs is varied for each model. We train the fine-grained model, CoFiNER, over 30 epochs. To construct the F2C mapping matrix, the coarse-grained model is trained for 50 epochs using both CoNLL'O3 and OntoNotes. To train the inconsistency filtering model, we set different epochs based on the number of shots: For 10, 20, 40, 80, and 100 shot settings, the epochs are 150, 150, 120, 50, and 30, respectively. We report the results using span-level FI. The dropout with a probability of 0.1 is applied. All the models were trained NVIDIA RTX 3090 GPUs.
4.1 Datasets
# Sentences
Dataset
# Types
Train
Dev
Test
Coarse-grained grained datasets
CoNLL'03
14k
3.3k
3.4k
4
OntoNotes
59.9k
8.5k
8.3k
18
Fine- Fine-grained dataset Fow-NERD
10-Shot
0.3k
20-Shot
0.6k
40-Shot
1.1k
18.8k
37.6k
66
80-Shot 2.2k
100-Shot
2.7k
Table Statistics of each dataset.
We conduct experiments using fine-grained NER dataset, Few-NERD (SUP) (Ding al., 2021), as well as two searse-grained datasets, namely, OntoNotes (Weischedel et al., 2013) and CoNLL'03 (Tjong Kim Sang, 2002). The fine- grained dataset Few-NERD comprises 66 en- tity types, whereas the coarse-grained datasets CoNLL'03 and OntoNotes consist of 4 and 18 entity types, respectively. The statisties for the datasets are listed in Table 1.
4.3 Compared Methods
In this study, we compare the performance of CoFiNER with that of both the supervised and few-shot methods. We modified the existing meth- ods for our experimental setup and re-implemented them accordingly.
Supervised method. We use BERT ROBERTaaASE. and ROBERTa vised baselines, each including a fine-grained clas- sifier on the head. In addition, PIQN (Shen et al., 2022) and PL-Marker (Ye et al., 2022) are meth- ods that have achieved ofate-of-the--at performance in a supervised setting using the full Few-NERD dataset. All models are trained using only a Few- NERD dataset.
the super-
K-shot sampling for the fine-grained dataset. Because we assumed a small number of examples for each label in the fine-gras -eeaiiea dataset, we evalu- ated the performance in the K-shot learning setting. Although Few-NERD provides few-shot samples, they are obtained based on an N-way K-shot sce- nario, where N is considerably smaller than the total number of entity types. However, our goal is to identify named entities across all possible entity types. For this setting, we resampled K-shot exam- ples accommodate all-way K-shot scenarios.
Few-shot method. The LSFS (Ma et al., 2022a) leverages a label encoder to utilize the semantics of label names, thereby achieving state-of-ther thethrrr results in low-resource NER settings. The LSFS ap- plies a pre-finetuning strategy to learn prior knowl- edge from the coarse- -aanscegranned dataset, OntoNotes. For a fair comparison, we also conducted pre-
Since multiple entities exist in single sentence, we cannot strictly generate exact K K-shot samples for all the entity types. Therefore, we adopt the

---

Method
Model
10-shot
20-shot
40-shot
80-shot
100-shot
BERT (Devlin et al., 2019)
23.101
34.718
43.138
46.182
46,449
RoBERTaaas (I.iu al., 2019)
21.073
34.157
39.226
45.735
46.942
Supervised Method
RoBERRTAuaEE (Liu et al., 2019)
29.137
41.425
51.137
55.057
54.172
PIQN (Shen et al., 2022)
21.750
22.007
28.533
29.339
38.658
PL-Marker (Ye et al., 2022)
40.902
48.064
52.395
$3.249
53.061
Few-shot Method
LSFS (Ma et al., 2022a)
47.998
43.269
50.595
$1,420
50.366
Proposed Method
CoFiNER
44.951
51.142
56.409
56.847
57.178
Table 2: Results on few-shot NER. The best scores across all models are marked bold.
finetuning on OntoNotes and performed fine-tuning on each shot of the Few-NERD dataset.
10-shot
20-shot
80-shot
100-shot
CoFINER
44,95
51.14
56.41
56.85
57.18
wlo filtering
41.38
43.87
53.42
$4,70
55.13
Proposed method. We trained our COFiNER model as proposed in $3. In each epoch, CoFiNER is first trained on two coarse-grained datasets: OntoNotes and CONLL'03. Subsequently, it is trained on the fine-grained dataset Few-NERD. We used ROBERTaLARGE as the pre-trained language model for CoFiNER in Equation (1).
wio OntoNotes
42,63
44.51
57.18
56.97
55.54
CoNLL'03
43.99
50.07
w/o course
29.14
41.43
51.14
55.06
54.17
Table 3: Performances of ablation study over different components.
4.5 Ablation Study
4.4 Main Results
An ablation study is conducted to validate the ef- fectiveness of each component of the proposed method. The results are presented in Table 3. First, we remove the inconsistency filtering (w/o filtering) and observe a significant decrease the Fl score, ranging from 2.05 to 7.27. These results demonstrate the effectiveness of our fil- tering method, which excludes mislabeled enti- ties. Second, we provide the results using sin- gle coarse-grained dataset (w/o OntoNotes and w/o CoNLL'03). Even with single coarse-grained dataset, our proposed method significantly out- performs w/o coarse, which trained solely on the fine-grained dataset (i.e. ROBERTaLARGE in Ta- ble 2).
Table 2 reports the performance of COFiNER and existing methods. The result shows that CoFiNER outperforms both supervised learning and few-shot learning methods. Because supervised learning typically needs a large volume of training data, these models underperform in low-resource set- tings. This demonstrates that our method effec- tively exploits coarse-grained datasets to enhance the performance of the fine-gra grained NER model. In other words, the proposed F2C mapping matrix significantly reduces the amount of fine-grained dataset required to train supervised NER models. particular, CoFiNER achieves significant per- formance improvements compared to the state-of- the-art model PL-Marker, which also utilizes the same pre-trained language model RoBERTa CoFiNER.
This indicates the effectiveness of using a well- aligned hierarchy through the F2C mapping matrix and inconsistency filtering. Although we achieve a significant improvement even with single coarse- grained dataset, achieve more stable result with coarse-grained datasets. This implies that our approach effectively utilizes multiple coarse- grained datasets, although the datasets contain dif- ferent sets of entity types.
The few-shot method LSFS yields the highest FI score for the 10-shot case. However, this few- shot method suffers from early performance sat- uration, resulting in less than 2.4 Fl improve- ment with an additional 90-shot. By contrast, the Fl score of CoFiNER increases by 12.2. Conse- quently, COFiNER outperforms all the compared methods except for the 10-shot case. In summary, the proposed method yields promising results for a wide range of data sample sizes by explicitly leveraging the inherent hierarchical structure.
4.6 Analysis
In this section, we experimentally investigate how the F2C mapping matrix and inconsistency filtering improve the accuracy of the proposed model.

---

GR
LOC
INC
ORG
PERSON
ART
PRODUCT
EVENT
LANGUSE
Lnn
Putt Ponnop
1022
Inssspp
Bua aeaaaa aeappa
other
location
building
organization
person
art
product
event
Figure 2: The F2C mapping matrix between OntoNotes and 100-shot of the Few-NERD dataset. During the generation of the F2C mapping matrix, some coarse-grained types are not mapped to any fine-grained types. The 7 unmapped types are not represented: DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, and CARDINAL.
LOC
ORG
PER
MISC
Wn Wiidiiona
340
Jappo
piod
Nuaqi
Seew poods
Jois
Rermeed
indrows
spor
0
other
location
building
organization
person
art
product
event
Figure 3: The F2C mapping matrix between CoNLL'03 and 100-shot of the Few-NERD dataset.
4.6.1 F2C Mapping Matrix
Mapping the entity types between the coarse- and fine-grained datasets directly affects the model per- formance. Hence, we investigate the mapping outcomes between the coarse- and fine-grained datasets. Figures and show the F2C matrices for FewNERR-OntoNotes and NewNERD-CoNDH '03, respectively. In both figures, the x- and y-axis rep- resent the fine-grained and coarse-grained entity types, respectively. The colored text indicates the corresponding coarse-grained entity types in Few- NERD, which not used to find the mapping matrix. The mapping is reliable if we compare the y-axis and the colored types coorre--rringe types in Few-NERD). Even without manual annotation of the relationship between coarse- and fine-grained entities, our method successfully obtains reliable mapping from fine-grained to coarse- coarse-grained types. Our model can be effectively trained with both types of datasets using accurate F2C mapping.
96
1
all
k
Figure 4: Impact of top-k values in the F2C mapping matrix. is reported on the test data.
F2C mapping matrix. We observe that the highest performance is achieved when is increases, the performance gradually decreases. Although the optimal value can vary depending on the quality of the coarse Soarse-grained data and the performance of the coarse -rasined NER model, the results indicate that a good F2C mapping matrix can be obtained by ignoring minor co-occurrences.
and
Figure 4 provides the Fl scores by varying the hyperparameter i to refine the F2C mapping matrix described in $3.3. 'all' refers to the usage of the complete frequency distribution when creating an
Additionally, We conducted an experiment by setting the F2C mapping matrix be learnable and comparing with our non-lcarnbbee F2C ma- trix. The non-learnable approach showed better

---

Type
Example
predict
they may be offshoots of the intifadaheveNr. Palestinian rebellion:veNt
target
they may be offshonts of the intifadah, the Palestinian rehellion in
predict
players heading to Canadacre- particularly Torontogre for the All-stanavENT
target
...
players heading Canadacrs, particularly TorontoGPe for the All-star
predict
Judges at Flight 103 Lockerbie trial are expected...
target
Judges Flight 103 Lockerbie.re trial expected
Table 4: Inconsistent examples:
The labeled sentence in the original coarse-grained dataset is the "target" type, while the label predicted by the model is the "predict" type. Consistent entity types are indicated in blue, while inconsistent entity types are indicated red.
performance, hence we adopted this approach for COFiNER. Detailed analysis and experiment re- sults are in Appendix B.2.
4.6.2 Inconsistency Filtering
Fl improvement
We aim to examine whether inconsistency filtering successfully screened out inconsistent entities be- tween OntoNotes and Few-NERD datasets. To con- duct this analysis, we describe three inconsistent ex- amples. Table 4 shows the predicted values and tar- get labels, which correspond to the coarse- grained output of the fine-gran tieeggrined NER model trained as de- scribed in $3.2 and the golden labels of the coarse- grained dataset.
EVENT
PERGON
0.2
0.4 Filtering proportion
0.6
Figure 5: Performance with increasing filtering propor- tion between Few-NERD and OntoNotes datasets. The correlation coefficient between filtering proportion and performance improvement is 0.29.
The first example illustrates the inconsistency in entity types between mislabeled entities. In the original coarse-grained dataset, "Palestinian" la- beled as NORP, but the model trained on the fine- grained dataset predicts "Palestinian rebellion" as its appropriate label, EVENT. However, annotators of the OntoNotes labeled "Palestinian" NORP, whereas the fine-grained NER model correctly pre- dicts the highly informative actual label span. The inconsistency caused by a label mismatch between the coarse-ae coarse-grained and fine-grain greiner datasets can result performance degradation.
illustrates the correlation between the filtering pro- portion and the performance improvement for cach course-grained label mapped to the ime-grained labels. In this figure, a higher filtering proportion indicates a greater inconsistency between the two datasets. F1 improvements indicate the difference in performance when filtering is applied and when not. Each data point refers to fine- fine-grained label mapped onto coarse coarse-grained label. As the proportion of the filtered entities increases, the scores also increase. These improvements indicate that inconsistency filtering effectively eliminates noisy entities, enabling the model to be well-traine on consistent data only.
In the second example, both "Canada" and "Toronto" are consistently labeled as GPE; thus, the model not confused when training on these entities. However, in the case of "All-star", we can observe mismatch. This example in the coarse-grained dataset labeled 0 instead of the correct entity type EVENT, indicating a mismatch. Through inconsistency filtering, unlabeled "All- star" is masked out of the training process.
Conclusion
As shown in the examples, inconsistency filter- ing is necessary to mitigate the potential noise aris- ing from mismatched entities. We analyze the fil- tering results for each coarse-grained label to as- sess its impact on model performance. Figure
We proposed COFiNER, which explicitly leverages the hierarchical structure between coarse- and fine- grained types to alleviate the low-resource problem of ane-grained NER. We devised the F2C map- ping matrix that allows for fine- mme-graieed NER model

---

training using additional coarse-grained datasets. However, because not all coarse-grained entities are beneficial for the fine-grained NER model, the proposed inconsistency filtering method is used to mask out noisy entities from being used in the model training process. We found through exper- iments that using smaller amount of consistent data is better than using a larger amount of data without filtering, thus demonstrating the crucial role of inconsistency filtering. Empirical results confirmed the superiority of CoFiNER over both the supervised and few-shot methods.
References
Armen Aghajanyan. Anchit Gupta, Akshat Shrivastava. Xilun Chen, Luke 7ettlemoye, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with ure-finetuning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2021. Container: Few- shot named entity recognition via contrastive learn- ing, arXiv preprint arXiv: arXiv:2109.07589
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy eser-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguis- ties.
Limitations
Despite the promising empirical results of this study, there is still a limitation. The main drawback of CoFiNER is that the token-level F2C mapping matrix and inconsistency filtering may not be di- rectly applicable nested NER tasks. Nested NER involves identifying and classifying certain over- lapping entities that exceed the token-level scope. Because CoFiNER addresses fine-grained NER at the token level, it may not accurately capture en- tities with nested structures. Therefore, applying our token-level approach to nested NER could pose challenges and may require further adaptations or other modeling techniques effectively handle the hierarchical relations between nested entities.
Jacob Devlin, Ming-Wai Beig-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Haman Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis. Minnesota. Association for Computational Linguistics.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 3198-3213, Online. Association for Computational Linguistics.
Acknowledgments
This work was supported by the National Re- search Foundation of Korea(NRF) grant funded by the Korea govermmen(MMIIT) (No. NRF- 1122RIGIA11135999 This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP)) grant funded by the Korea (overnment(MSIT) (No. RS- 2023-00261068. Development of Lightweight Mul- timodal Anti-Phishing Models and Split-Lcaming Techniques for Privacy-Preserving Anti-Phishing! and
Maximilian Hofer, Andrey Kormilitzin, Paul Goldberg. and Alejo Nevado-Holgado. 2018. Few-shot learning for named entity recognition in medical text.
Yucheng Huang, Kai Hc. Yige Wang, Xianli Zhang, Tieliang Gong, Rui Mao, and Chen Li. 2022. Copner: Contrastive learning with prompt guiding for few- shot named entity recognition. In Proceedings of the 29th International conference on computational linguisticx, pages 2515-2527.
Artificial Intelligence Convergence Innovation Human Resources Devel- opment (Hanyang University ERICA)). This re- search was also supported by the Ministry of Trade, Industry, and Energy (MOTIE), Korea, under the "Project for Research and Development with Mid- dle Markets Enterprises and DNA(Data, Network, AI) Universities" (Smart Home Based LifeCare Service & Solution Development((0022555)) su- pervised by the Korea Institute for Advancement of Technology (KIAT).
Haoming Jiang, Danqing Zhang, Tianyu Cao, Bing Yin, and Tuo Zhao. 2021. Named entity recognition with small strongly labeled and large weakly labeled data.
Woohwan Jung and Kyuscok Shim. 2020. Dual super- vision framework for relation extraction with distant supervision and human annotation. Proceedings of the 28th International Conference on Computational Linguistics, pages 641 1-6423, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: (tt::997..1122.
Conference on Natural Language Learning 2002 (CoNLL-2002).
Christopher Walker, Stephanie Strassel, Julie Medero. and Kazuaki Maeda. 2006. Ace 2005 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Peiyi Wang, Runxin Xu, Tianyu Liu, Qingyu Zhou, Yunbo Cao,, Baobao Chang, and Zhifang Sui. 2022. An enhanced span-based decomposition method for few-shot sequence labeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5012-5024.
Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou. Prodromos Malakasiotis, Ion An- droutsopoulos, and Georgios Paliouras. 2022, Finer: Financial numeric entity recognition for xhrl tagging. arXiv preprint arXiv:2203.0m 064882.
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022a. Label semantics for few shot named entity recognition. Findings of the Association for Computational Linguistics: ACL. 2022, pages 1956- 1971, Dublin, Ireland. Association for Computational Linguisties.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 1t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers), pages 4904-4917 Dublin, Ireland. Association for Computational Linguistics.
Tingting Ma, Huiqiang Jiang. Qianhui Wu, Tiejun Zhao, and Chin- Yew Lin. 2022h. Decomposed meta- learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1584-1596.
Appendix
Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2005. Ace 2004 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 1:1-1.
A Sampling Algorithm
Algorithm 1 K~(++5) sampling algorithm
Seongwoong Oh, Woohwan Jung. and Kyuseok Shim. 2023. Thunder: Named entity recognition using a teacher-student model with dual classifiers for strong and weak supervisions. In Proceedings of 26th Euro- pean Conference on Artificial Intelligence.
Input: Dataset X, labeled set Y, K
Output: Fine-grained dataset D=
1: Sort classes in based on their freq. in X D Init the train dataset b Init the count of entity classes in D
2: D= <- 0:
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An ex- perimental study. In Proceedings of the 2011 Con- ference on Empirical Methods in Natural Language Processing, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
3: for i = to Y do
4:
Count [t] 0;
5: for i to Y do
if V Count(i"] > K then
Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yuet- ing Zhuang. 2022. Parallel instance query network for named entity recognition. 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 947-961, Dublin, Ireland. Association for Computa- tional Linguistics.
break;
8: Randomly sample (x, E s.t. Y, E
Proceedings of the
9:
for j = 1 to |y| do
10:
Count [i] 1;
11:
if 3 Count + Count Y [1] > K + 5 then
12:
Continue;
13:
else
Mujeen Sung, Minbyul Jeong, Yonghwa Choi, Donghyeon Kim, Jinhyuk Lee, and Jaewoo Kang. 2022. Bern2: an advanced neural biomedical named entity recognition and normalization tool. Bioinfor- maties, 38(20):4837-483
14:
D U v),
15:
Count <- Count + Count
16:
return D
Unlike other tasks, NER involves multiple en- tity occurrences within a sentence, making it too restrictive to sample an exact count. We adopt a
Brik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th

---

-~(K-5)-shht setting to minimize differences in the number of entity types. Additionally, we sample the entity types, starting from those with fewer occurrences to ensure a balanced distribution of multiple entity types within the sentences. Al- gorithm presents the K-shot sampling algorithm used in this study.
Model
F1
RoBERTA LARGE
75.15
PL-Marker
74.03
CoFiNER 80.44
w/o filtering
78.20
w/o coarse
75.15
Table 6: Performances of different models and ablation studies on our model. RoBERTaLARE and w/o coarse are identical.
B
Additional Experiments
B.1 Generalization of Inconsistency filtering in diverse dataset settings
set up the CoNLL'O3,, which has entity types, as the coarse-grained dataset, and the 100-shot OntoNotes, which has a finer label with 19 entity types, as the fine-grained dataset. In Table 6, when compared to the two pop-performing models in the main results, RoBERTa and the state-of-the- art PL-Marker, CoFiNER show consistently higher performance. Furthermore, as shown in the abla- tion study that was conducted following the same methodology in $4.5, CoFiNER exhibited superior performance.
Coarse-grained datasets
Setting
Few Pew-NERD
OntoNotes CoNLL'03
w/ filtering
56.63
57.18
w/o filtering $6.46
55.13
Table 5: Results different coarse genseeerained datasets. fine-grained dataset is 100-shot of the Few-NERD.
To assess the generalization of the proposed method, conducted experiments under various coarse- and fine-grate fine-grained dataset settings.
In conclusion, through above the two experi- ments, our method has been demonstrated to work robustly across different coarse- and fine-grained dataset settings.
First, we verified the robustness of inconsis- tency filtering through experiments using differ- ent coarse-grained datasets. The fine-grained dataset, Few-NERD, remains unchanged. Since Few-NERD has both coarse- and fine-grained labels, we constructed coarse-grained dataset Few-NERD _oarse using the coarse-grained labels from Few-NERD. When compared to Few-NERD__comd coarse, entities in OntoNotes and CoNLL'03 are inconsistently labeled with Few- NERD because they were independently created. Table 5, Few-NERD_ _ exhibits a higher Fl score in the w/o filtering setting due to its con- sistency with fine-grained labels of Few-NERD. However, the performance improvement achieved with filtering is more substantial in the inconsistent datasets, when compared to the consistent dataset. This result indicates that inconsistency filtering im- proves performance by filtering out the mismatch- ing labels. Therefore, we have demonstrated the importance of using inconsistency filtering to filter out noise when working with datasets that employ different labeling schemes. Furthermore, by achiev- ing effective performance improvements across ious coarse- coarse-grained datasets, we have provided evi- dence of the robustness of the filtering method. Second, we validate the generalization perfor- mance through experiments conducted in differ- ent coarse- and Ene-grained dataset settings. We
B.2 Comparison with learnable F2C mapping matrix
Matrix Type
F1
non-learnable
56.27
learnable 53.25
Table 7: Results learnable and non-learnable F2C mapping matrix on 100-shot of Few-NERD.
find the optimal F2C mapping matrix, we con- ducted experiments to explore the impact of mak- ing the F2C mapping matrix learnable. We use Few- NERD as a fine-grained dataset and OntoNotes as coarse-grained dataset. Table 7 shows no perfor- mance gains when the F2C mapping matrix was set to be learnable. We found that the learnable matrix tends to form a pattern similar to what is shown in Figure 4 with k=all. This result suggests that taking minor co-occurrences into account leads to an overall decrease in performance. Based on this analysis, the non-learnable mapping matrix is used in our experiments.


Reviews:
[{"review_id": "2VZdf7Nl7V", "reviewer": "Reviewer_D3qZ", "summary": "", "questions": "1) The method only back propagates loss for the examples for which the coarse label matches the truth. Why do you think this is helpul in training the model? Seems like the model does not get to learn from its mistakes.\n2) I am not sure if I fully understand the w/o course baseline. is it not a supervised fine-tuned model then?\n3) It could not sing this detail in the paper - which setting of Few NERD is used - SUP, INTER or INTRA?\n4) Did the authors re-implement/ retrain all baselines with new splits? The PIQN numbers are much lower, would you know why?\n5) OntoNotes has 18 types, the paper mentions 11. Which types are not used. Pls include details.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer D3qZ\n\nWe sincerely appreciate the time and effort that the reviewer dedicated to reading our paper and providing insightful feedback. Please find below our responses to the points raised in the review.\n\n---\n\n**The method only back propagates loss for the examples for which the coarse label matches the truth. Why do you think this is helpul in training the model? Seems like the model does not get to learn from its mistakes.**\n\n**Response:**  We would like to point out a key advantage of the inconsistency filtering through selective backpropagation: it mitigates potential confusion that could arise from label inconsistencies across datasets. If the inconsistency filtering is not employed, the model could learn wrong correlations from misaligned labels, potentially diminishing the benefits of leveraging multiple datasets. \n\nTo empirically assess the impact of label inconsistencies on model performance, we conducted additional experiments using coarse-grained labels of Few-NERD as an auxiliary coarse-grained dataset. \n\n|  | Few-NERD_coarse | OntoNotes + CoNLL |\n| --- | --- | --- |\n| w/ Filtering | 56.63 | 57.18 |\n| w/o Filtering | 56.46 | 55.13 |\n\nSince Few-NERD dataset has both fine-grained and coarse-grained labels for every input text, the coarse-grained labels are consistent with the fine-grained labels. Meanwhile, OntoNotes and CoNLL, which were created independently with Few-NERD, have many inconsistent labels. The performance improvement due to the inconsistency filtering is greater in the inconsistent datasets, CoNLL and OntoNotes than in the consistent dataset, Few-NERD_coarse. \n\nThe result implies that the inconsistency filtering improves performance by filtering out the inconsistent labels via selective backpropagation. Hence, we believe that erroneous labels could lead to confusion, as they deviate from the labeling scheme of Few-NERD, rather than helping the model learn from its mistakes.\n\n---\n\n**I am not sure if I fully understand the w/o course baseline. is it not a supervised fine-tuned model then?**\n\n**Response:**  Yes, it is a supervised fine-tuned model. Thank you for pointing out the ambiguously described part of our paper. To clarify, *w/o coarse* baseline is a supervised fine-tuned model that is trained only on the fine-grained dataset, without incorporating any data from the coarse-grained datasets. Thus, the *w/o coarse* baseline is identical to RoBERTa_LARGE in Table 2 since we use RoBERTa_LARGE as the base model.\n\nWe will make clear this part in the revised version.\n\n---\n\n**It could not sing this detail in the paper - which setting of Few NERD is used - SUP, INTER or INTRA?**\n\n**Response:**  We used the dataset of the Few-NERD SUP setting. The INTER and INTRA settings described in the Few-NERD[1] use an $N$-way $K$-shot setting where the types in the train, dev, and test sets are all different. \n\nIn contrast, we conducted our experiments in an **a*ll*-way $K$-shot setting**. We believe this approach is more **realistic** and **challenging** compared to $N$-way $K$-shot setting. $N$ usually represents a small, restricted number of classes, making it inadequate to handle a large number of entities. In our approach, we expand $N$ to include a much larger set of fine-grained labels, such as the 67 types in Few-NERD. This **a*ll*-way $K$-shot setting** makes our experimental setting more representative of real-world scenarios.\n\n---\n\n**Did the authors re-implement/ retrain all baselines with new splits?**\n\n**Response:**  Yes, we re-implemented and retrained all baselines using new splits.\n\nWe used the Hugging Face library to obtain pre-trained models for BERT and RoBERTa. For other baselines(PIQN, PL-Marker, and LSFS), we adapted the original code available on each paper's GitHub repositories, modified it for our experimental setup, and retrained them accordingly.\n\n**The PIQN numbers are much lower, would you know why?**\n\n**Response:** PIQN was originally proposed for a standard supervised setting, as demonstrated in their paper[2]. In this context, PIQN is trained on all available training data in the Few-NERD SUP setting. On the other hand, our work operates in an *all*-way $K$-shot setting, where we have limited examples for each class. PIQN, which is designed to perform well with a substantial amount of data, exhibits lower performance in our few-shot setting.\n\n---\n\n**OntoNotes has 18 types, the paper mentions 11. Which types are not used. Pls include details.**\n\n**Response:**  As you mentioned, Figure 2 illustrates 11 out of 18 entity types.\n\nFor clarity, we excluded certain coarse-grained types that were **not mapped** to any fine-grained types during the generation of the F2C mapping matrix using OntoNotes and Few-NERD datasets. The 7 unmapped types from OntoNotes are as follows: `DATE`, `TIME`, `PERCENT`, `MONEY`, `QUANTITY`, `ORDINAL`, and  `CARDINAL`. Because the correlation between coarse- and fine-grained types makes mapping decisions, consequently affects the types within the F2C mapping matrix.\n\nWe hope this clarifies any confusion. We will also include these details in the revised manuscript.\n\n---\n\n**References**\n\n[1] [Few-NERD: A Few-shot Named Entity Recognition Dataset](https://aclanthology.org/2021.acl-long.248) (Ding et al., ACL-IJCNLP 2021)\n\n[2] [Parallel Instance Query Network for Named Entity Recognition](https://aclanthology.org/2022.acl-long.67) (Shen et al., ACL 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task is interesting and useful - to leverage existing data for few-shot fine-grained \nThe paper is generally well written, with easy to understand figures and table. Improvements could be made as suggested.\nThe results show the method works. If the authors make the code available, this could be a useful repo.", "reasons_to_reject": "Pls see the questions section - I have questions over the evaluation and also on why the method works when the loss erroneous labels is not back-propagated.", "questions_to_authors": "1) The method only back propagates loss for the examples for which the coarse label matches the truth. Why do you think this is helpul in training the model? Seems like the model does not get to learn from its mistakes.\n2) I am not sure if I fully understand the w/o course baseline. is it not a supervised fine-tuned model then?\n3) It could not sing this detail in the paper - which setting of Few NERD is used - SUP, INTER or INTRA?\n4) Did the authors re-implement/ retrain all baselines with new splits? The PIQN numbers are much lower, would you know why?\n5) OntoNotes has 18 types, the paper mentions 11. Which types are not used. Pls include details.", "paper_topic_and_main_contributions": "The paper details a method to use existing coarse-grained data to improve the probability of few-shot fine grained system. The system consists of a pipeline where fine-grained fine tuned model is used to produce the labels, which are then mapped to a coarse-grained label, these labels are filtered for inconsistency and loss is computed over only the consistent labels. \nThey use CoNLL and OntoNotes as the source coarse grained datasets to improve performance over few NERD. Results show they achieve improvements over baselines.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "66d7n8Fn6d", "reviewer": "Reviewer_ZwPd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ZwPd\n\nWe sincerely appreciate the time and effort that the reviewer dedicated to reading our paper and providing insightful feedback. Please find below our response to the points raised in the review.\n\n---\n\n**The hierarchical structure is not well-described in the paper. It seems it is only the direct mapping between coarse- and fine-grained types instead of hierarchy.**\n\n**Response:**  We agree with your assessment that the paper may give the impression of direct mapping between coarse- and fine-grained types because we mainly introduce our model with a pair of coarse- and fine-grained datasets for readability. Furthermore, fully exploiting the hierarchical structure is important to achieve a high accuracy in fine-grained NER.\n\nIn fact, our approach can straightforwardly leverage a multi-level hierarchy by incorporating multiple coarse-grained datasets during the training process as discussed in Section 3.1. In addition, we demonstrated performance improvements by using a multi-level hierarchy that incorporates both OntoNotes and CoNLL datasets, outperforming the results when employing a two-level hierarchy using either OntoNotes or CoNLL.\n\nWe acknowledge that this could have been more clearly described in our paper, and we will clarify it in the revised manuscript.\n\nWe would like to explain the multi-level hierarchy with `LOC` as an example. `LOC` type in the most coarsely grained CoNLL dataset, exhibits hierarchical relations with `GPE`, `LOC`, and `FAC` in the finer dataset, OntoNotes. These types are further divided into more specific types and establish correlations with the most finely grained dataset, Few-NERD. As illustrated in Figure 2,  `GPE`, `LOC`, and `FAC` in OntoNotes show hierarchical relations with 10 Few-NERD types, including `bodiesofwater`, `island`, `mountain`, `park`, and `road`. \n\nDespite the datasets exhibiting multi-level hierarchical relationships, we do not explicitly consider the relationships between the coarse-grained datasets, since our primary goal is to improve fine-grained NER.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Regarding the writing: The paper is well-written and easy to follow.\n- Regarding the methodology: The method is well-designed to improve the low-resource problem in fine-grained NER, where a mapping matrix is constructed to measure the relations between the coarse- and fine-grained NE types and a filtering method is used to filter those inconsistent labels.\n- Regarding the experiments: The experiments are sufficient and show the effectiveness of the proposed approach.", "reasons_to_reject": "- The **hierarchical structure** is not well-described in the paper. It seems it is only the direct mapping between coarse- and fine-grained types instead of hierarchy.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Paper Topic: Fine-grained NER\n\nMain Contribution (NLP engineering experiment): The authors propose to utilize the hierarchical structure between coarse-grained and fine-grained types to alleviate the low-resource problem in fine-grained NER. Experiments demonstrate the effectiveness of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8fSaQawRlU", "reviewer": "Reviewer_Y7sL", "summary": "", "questions": "- I wonder whether the F2C matrix is learnable or not during the final training stage. Can you justify why it is or is not?\n- Table 3 shows that more data does not always improve the final results and the behavior varies for different number of shots. Is there any good explanation on this?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Y7sL\n\nWe sincerely appreciate the time and effort that the reviewer dedicated to reading our paper and providing insightful feedback. Please find below our responses to the points raised in the review.\n\n---\n\n**It would be interesting to see results on more fine-grained datasets.**\n\n**Response:** Thank you for your thoughtful suggestion to evaluate our model on different fine-grained datasets. We agree that doing so would strengthen the paper's contributions.\n\nTo demonstrate the robustness of our model across diverse settings, we conducted additional experiments using the following datasets:\n\n- Fine-grained dataset: OntoNotes (19 entity types)\n- Coarse-grained dataset: CoNLL (4 entity types)\n\nWe used the same sampling algorithm as presented in our paper to sample 100-shot examples from OntoNotes and conducted an ablation study. Here are the results for different models:\n\n| Model | 100-shot |\n| --- | --- |\n| CoFiNER | 80.44 |\n|   w/o filtering | 78.20 |\n|   w/o coarse | 75.15 |\n\nCoFiNER consistently improves performance across different fine-grained NER tasks, and we believe these results support that CoFiNER is generally applicable to various fine-grained datasets.\n\n---\n\n**I wonder whether the F2C matrix is learnable or not during the final training stage. Can you justify why it is or is not?**\n\n**Response:** Thank you for your question about the learnability of the F2C mapping matrix during the final training stage.\n\nIn response to your question, we conducted supplementary experiments to explore the impact of making the F2C mapping matrix learnable. Unfortunately, our subsequent testing showed no performance gains when the F2C mapping matrix was set to be learnable. We conducted experiments under the following settings:\n\n- Fine-grained dataset: Few-NERD\n- Coarse-grained dataset: OntoNotes\n\n| F2C Mapping Matrix | 100-Shot |\n| --- | --- |\n| non-learnable | 56.27 |\n| learnable | 53.25 |\n\nWe found that when the matrix is learnable, it tends to form a pattern similar to what is shown in Figure 4 with $K$=all. These results suggest that taking minor co-occurrences into account leads to an overall decrease in performance.\n\nBased on this additional analysis, we would like to maintain our original approach of using a fixed F2C mapping matrix. Even without the training through backpropagation, the F2C mapping matrix can capture the adequate relation between types, and in doing so, it contributes to a more stable and improved performance.\n\nYour question gave us the opportunity for further investigation, and we are grateful for that. \n\n---\n\n**Table 3 shows that more data does not always improve the final results and the behavior varies for different number of shots. Is there any good explanation on this?**\n\n**Response:** As you pointed out, some models do not show a consistent improvement in performance with an increase in the number of shots. We believe the following are possible reasons for the behavior:\n\n**1. Overfitting**\n\nIn few-shot settings, the risk of overfitting is inherently higher due to the limited amount of data. This is especially apparent in our experiments that exclude the OntoNotes or CoNLL datasets, making the models more prone to overfitting. This could possibly explain the fluctuating performance as the number of shots changes.\n\nOn the other hand, CoFiNER leverages a more diverse set of training datasets, which helps to alleviate the risk of overfitting to some degree. Thus, it shows a consistent performance improvement as the number of shots increases.\n\n**2. A Mismatch of train and test data distributions**\n\nWith a low number of shots, the limited training samples might be insufficient to capture the full range of the test data distribution.\n\nIn the test dataset, the frequency of each entity type varies considerably. For instance, in the Few-NERD test set,  the fine-grained type `location-GPE` appears 20,409 times, while the type `art-painting` shows up just 58 times. However, when generating training data in few-shot setting, we try to make the occurrence of each entity type as uniform as possible. This can lead to a mismatch between the train and the test set. Consequently, as the amount of training data increases, there might be fluctuations in performance improvements on the test set.\n\nWe hope that we have addressed your concerns regarding the observed behavior and provided a reasonable explanation.\n\n---\n\n**line 43: 'cost' -> 'costly' line 88: 'properly' -> ? line 239: 'a number' -> 'a large number'?**\n\n**Response:** Thank you for the detailed comments. We will rectify them in our revision. Furthermore, we thoroughly proofread the manuscript and corrected grammatical errors and ambiguous expressions to improve readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a way to utilize a coarse-grained labeled data to help fine-grained typing by explicitly modeling the coarse-to-fine relations which are then used to help integrate training on both coarse- and fine-grained data.\n- There are insights into the coarse-to-fine correlations and the inconsistency filtering parts to demonstrate the motivations. The illustration of F2C mapping is intriguing.\n- The proposed method achieves strong performance compared with baselines.", "reasons_to_reject": "- It would be interesting to see results on more fine-grained datasets. ", "questions_to_authors": "- I wonder whether the F2C matrix is learnable or not during the final training stage. Can you justify why it is or is not?\n- Table 3 shows that more data does not always improve the final results and the behavior varies for different number of shots. Is there any good explanation on this?", "paper_topic_and_main_contributions": "This paper studies the task of few-shot fine-grained entity typing. It proposes to utilize widely available coarse-grained datasets as additional knowledge to help the fine-grained task. By explicitly modeling the coarse-to-fine relations, the proposed methods achieve strong performance to baseline few-shot methods and another method also takes external coarse-grained data. Extensive experiment analysis shows the effectiveness of each proposed part and provides insights to the proposed model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 43: 'cost' -> 'costly'\nline 88: 'properly' -> ? \nline 239: 'a number' -> 'a large number'?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper utilizes coarse-grained datasets for fine-grained NER with a small number of fine-grained annotations. The research questions have significant value and this paper gives the effective method. I suggest the authors give detailed descriptions for the method, and make precise revision accroding to comments of reviewers and responses.
Decision: Accept-Main