Coarse-to-fine Few-shot Learning for Named Entity Recognition
Year: 2023

Abstract:
Recently, Few-shot Named Entity Recognition has received wide attention with the growing need for NER models to learn new classes with minimized annotation costs. However, one common yet understudied situation is to transfer a model trained with coarse-grained classes to recognize fine-grained classes, such as separating a product category into sub-classes. We find that existing few-shot NER solutions are not suitable for such a situation since they do not consider the sub-class discrimination during coarse training and various granularity of new classes during few-shot learning. In this work, we introduce the Coarse-to-fine Fewshot NER (C2FNER) task and propose an effective solution. Specifically, during coarse training, we propose a cluster-based prototype margin loss to learn group-wise discriminative representations, so as to benefit fine-grained learning. Targeting various granularity of new classes, we separate the coarse classes into extra-fine clusters and propose a novel prototype retrieval and bootstrapping algorithm to retrieve representative clusters for each fine class. We then adopt a mixture prototype loss to efficiently learn the representations of fine classes. We conduct experiments on both in-domain and cross-domain C2FNER settings with various target granularity, and the proposed method shows superior performance over the baseline methods.

heading: 1 Introduction
text: Named Entity Recognition (NER), aiming at recognizing named entities such as person names, locations, and organizations from unstructured text, is a fundamental task for NLP applications (Mintz et al., 2009; Li et al., 2013; Rajpurkar et al., 2016). With the emerging of knowledge from various domains and the emerging of new entity types, it
∗Equal contribution. †Corresponding authors.
is difficult to manually annotate a large-scale highquality dataset for each new application scenario (Ding et al., 2021; Yang and Katiyar, 2020). Under this circumstance, Few-shot NER has become a crucial task, which studies NER systems that can fast adapt to new domains or unseen entity types with minimized annotation efforts. Recently, an increasing number of researchers have focused on Few-shot NER and made contributions to this task (Yang and Katiyar, 2020; Huang et al., 2021; Cui et al., 2021; Ma et al., 2022; Tong et al., 2021).
Unfortunately, previous works on few-shot NER typically assume the new classes are of the same or similar level of granularity to the known classes (Ding et al., 2021; Yang and Katiyar, 2020). In practice, a common situation is that the model needs to extend to more fine-grained classes from existing coarse-grained classes (Yang et al., 2021; Ni et al., 2021). Such a situation typically occurs during the lifespan of the model when the application requires separating some sub-classes from the current classes and yet when the model is pre-trained without any knowledge of the subclasses. For example, a product recognizer initially built with a "Computer" class may need to separate into sub-classes such as "Notebook computer" and "Tablet computer" as the requirement updates.
4115
Naturally, relabeling a high-quality dataset each time is much too costly to be an option (Bukchin et al., 2021).
In this work, we introduce the Coarse-tofine Few-shot NER (C2FNER) task. We find that existing few-shot NER solutions are less suitable for C2FNER for two reasons: (i) When training on the coarse-grained classes, most of the existing methods aim at maximizing the interclass boundaries yet neglect the intra-class variance (Bukchin et al., 2021; Chen et al., 2022b), which might result in reducing separability of sub-classes, thus hindering few-shot fine-grained learning. (ii) As the granularity of new classes is unknown during coarse training, it is challenging to provide a solution that can benefit various granularity of new classes. This requires specific designs for cross-granularity few-shot learning, which is not considered in general few-shot NER methods.
In this work, we propose a novel and effective solution for C2FNER. Firstly, in order to promote the intra-class variance in a way that best benefits cross-granularity learning, we propose a cluster-based prototype margin loss that improves group-wise discrimination in the coarse classes. Specifically, we first perform clustering on the representations of coarse-grained model to obtain entity clusters. Then, we minimize the distance of each entity with its prototype while maximizing its distance with negative prototypes from other coarse classes. Second, in order to benefit various granularity of new classes, we separate each coarse class into extra-fine clusters and propose a novel prototype retrieval algorithm to retrieve representative prototypes (i.e., representative clusters) for each fine-grained classes. By initialize each fine-grained class with multiple prototypes, we then adopt a mixture prototype loss for efficiently learning the representations of fine classes with only few samples. We also bootstrap the prototypes of each fine class with a kNN-based method during few-shot training, so as to make full use of the coarse model.
As the first attempt on C2FNER, we evaluate the proposed method and baselines on various practical C2FNER settings, including (1) the in-domain coarse-to-fine setting where the fine-grained classes are strictly included in and inferior to each coarsegrained class; and (2) the cross-domain coarseto-fine setting where the fine classes might not strictly included in the coarse classes. Additionally,
we evaluate the methods with various granularity of new classes. Experimental results show the superiority of the proposed method over baseline methods on multiple C2FNER settings. 1
To sum up the contribution of this work:
• In this work, we introduce the Coarse-to-fine Few-shot NER (C2FNER) task, a practical yet understudied task that is hard to solve with existing few-shot solutions.
• We propose a novel and effective method for C2FNER, which not only provide adaptive representations for fine classes during coarsegrained training, but also benefit various granularity of new classes through novel mixture prototypes retrieving and learning.
• We provide comprehensive benchmarks for C2FNER on both in-domain cross-domain settings and for various granularity of new classes.

heading: 2 Problem setup
text: In this work, we introduce the Coarse-to-fine Fewshot NER task (C2FNER). Formally, we denote by Ycoarse = {c1, . . . , cR} a set of R coarse classes and let Yfine = {c1, . . . , cF } be a set of fine classes. In our experiments, we explore both the situations when the fine classes are strictly included or not in the coarse classes. When training on the coarse classes, we assume an adequate number of N training instances Scoarsetrain = {(Xj , Yj)}Nj=1 annotated (only) with Ycoarse. When tested on the fine classes, only a K-shot training set Sfinetrain = {(Xj , Yj)}NKj=1 and the test set S fine test = {(Xj , Yj)}Mj=1 is provided, where K-shot means each fine class is annotated with only K entities. Our goal is to train a model M on the coarse training set Scoarsetrain and then fine-tune it on the K-shot fine training set Sfinetrain to obtain maximal performance on the fine test set Sfinetest .

heading: 3 Methodology
text: As shown in Figure 2, in this work, we propose a novel and effective method for C2FNER, which improves the performance from both the coarse and fine training phases: (1) During coarse training, we propose a cluster-based prototype margin loss to learn group-wise discriminative representations
1Our code is publicly available at https://github. com/rtmaww/C2FNER.
that benefit fine-grained learning. (2) During fine training, we improve performance for various fine granularity by retrieving representative clusters for each fine class with a prototype retrieval and bootstrapping algorithm and then training with a mixture prototype loss.

heading: 3.1 Coarse-grained Representation Learning: Cluster-based Prototype Margin Loss
text: The goal of coarse training is to provide a finefriendly representation for future fine-grained adaptation when ensuring the coarse performance. However, general supervised learning or few-shot learning methods aims at maximizing the interclass margin of coarse classes without considering the sub-classes included in the coarse classes. Such maximization of inter-class variance will harm the intra-class variance, leading to less separable subclasses and unfriendly representations for coarseto-fine transfer.
In this work, we propose a cluster-based prototype margin loss for coarse training, which increases the intra-class variance of each coarse class during training as well as learning a good semantic structure of features. Specifically, semantically similar entities are pulled together while being separable from other entities. Moreover, the discrimination of coarse classes should be retained, thus not hindering the performance of the coarse classes. Therefore, we use a prototype for representing a group of semantically similar entities and achieve the above goal with a prototype margin loss. Formally, we first assign the entities in each coarse class cr to a set of groups Gcr = {gi}NGi=1,Gcr ⊂ G by performing clustering on the representations of all entities from cr. We denote M(ei) as the assigned group (or cluster) of entity ei (i.e., ei ∈ M(ei)), where M(·) is a mapping function. Then, we obtain the prototype representation pk of each group k by:
pk = 1 |gk| ∑
ei∈gk h(ei) (1)
where h(ei) is the default representation of ei from the pre-trained PLM encoder. Then, the prototype margin loss for ei is calculated by:
Lpm,i = max(0,m+ d(h(ei),pM(ei))
− min pn∈G\Gyi d(h(ei),pn))
(2)
where m is a pre-defined margin. pM(ei) is the assigned prototype of ei. pn is a negative prototype selected from other coarse classes. d(·) denotes a distance function, which we use the Euclidean distance in our implementation. To optimize the prototype margin loss, each entity is forced to get closer to its prototype as well as keep a distance from prototypes from other classes. Consequently, semantically similar entities are pulled together and are discriminative to other groups of entities, thus contributing to friendly representations for coarseto-fine transfer learning.
As the prototype margin loss is calculated only for entity representations, we also train a classifier with the Cross-Entropy Loss for token classification (including the none-entity class):
Lce = − 1 N
N∑
i
log exp(W⊤c h(xi) + bc)∑|Ycoarse|
j=1 exp(W ⊤ j h(xi) + bj)
(3)
where Wc and bc is the weight and bias of class c. The final training loss can be written as:
Lcoarse = Lce +
Ne∑
i
Lpm,i (4)
where Ne is the entity number in the training set.

heading: 3.2 Fast Adaptation for Various Fine-Granularities
text: One challenge of adapting a coarse model to the fine classes is the unknown granularity of target fine classes. As the model trained on the coarse classes has no prior information about the fine classes, one intuitive solution is to provide adaptive representations that is suitable for various fine granularities. Also, it is essential to make full use of the coarse model and few-shot samples for fast adaptation during the few-shot training phase, which is unexplored neither in previous few-shot methods nor coarse-to-fine few-shot methods.
In this work, we solve the above problem by learning extra-fine prototypes on the coarse classes and performing multi-prototype learning on the fine classes.

heading: 3.2.1 Over-clustering for Unknown Fine Granularity
text: In order to learn adaptive representations for unknown granularity of fine classes, we propose to divide the entities in the coarse classes into extrafine clusters through over-clustering and represent
each extra-fine cluster with a prototype. As a result, after coarse training, we have:
G = R⋃
r=1
Gcr , |G| ≫ |Ycoarse|
P = {pi}|G|i=1
(5)
By assuming that |G| > |Yfine| for most of the fine granularity, the trained representations are adaptive to various fine granularity as long as we perform appropriate training with few-shot samples.

heading: 3.2.2 Prototype Retrieval with Few Samples
text: As the coarse-trained representations are divided into |G| extra-fine clusters, for arbitrary granularity of fine classes that |Yfine| < |G|, we propose to retrieve representative clusters for each fine class from the coarse-trained representations. This idea is achieved by a prototype retrieval algorithm. Specifically, for a given fine-grained class cf ∈ |Yfine| with K sampled entities {(efi , y f i )}Ki=1 (where efi denotes the i-th sample of class cf ), we retrieve one prototype for each sample efi by:
P̂ (efi ) = arg min t∈{1..|P|} d(h(ei),pt) (6)
where P̂ (efi ) is the index of the retrieved prototype, i.e., the retrieved cluster. As the retrieved prototypes of different samples might conflict, it is essential to ensure that each prototype is assigned to only one class. Therefore, for each prototype pj , we assign it to a possible fine class by:
Y (pj) = { None, if ∀efi , P̂ (efi ) ̸= j argmin
cf d(h(efi ),pj), else
(7)
The detailed process of the Prototype Retrieval Algorithm can be found in Alg.1.
Prototype Bootstrapping In order to make the fullest use of the pre-trained representations and discover more representative clusters for each fine class, we also propose to bootstrap the prototypes of each class. To ensure appropriate prototype assignment, we borrow the idea of K-NearestNeighbor, i.e., one prototype is most likely to belong to class cf if most of its nearest neighbors belong to class cf . Therefore, we bootstrap the prototypes after every M epochs by:
ηk(cf | pi) = |P|∑
j=1
1[Y (pj) = cf ,pj ∈ topK(pi)]
Ŷ (pi) = argmax cf
ηk(cf | pi)
Y (pi) = { Ŷ (pi), if ηk(Ŷ (pi) | pi) > T None, else
(8)
where 1[·] is an indicator function and topK(·) is obtained by selecting the K most nearest neighbor of pi by calculating d(pi,pl),pl ∈ P . Also, as shown in the equation, we assign a prototype to class cf only when the number of prototypes belonging to cf is larger than a threshold T , in order to keep the quality of the prototype assignment.

heading: 3.2.3 Mixture Prototype Learning
text: By retrieving prototypes from coarse representations, we initialize the few-shot training of fine classes by regarding each fine class cf ∈ Y{⟩\⌉ as a mixture distribution including multiple subclusters represented by multiple prototypes {pi | pi ∈ P(cf )}. In order for faster adaptation, we make full use of the retrieved prototypes with a Mixture Prototype Loss. Specifically, for each
Algorithm 1 Prototype Retrieval Require: The set of prototypes of the coarse model P ,
the few-shot sample set of the fine-grained classes {{(e1i , y1i )}Ki=1, . . . , {(e |Cfine| i , y |Cfine| i )}Ki=1}.
1: for each prototype pi ∈ P do 2: // Initialize prototype-class assignments 3: Y (pi)← None 4: // Initialize the nearest sample lists 5: NS[i]← ∅ 6: end for 7: // Retrieve the nearest prototype for each sample 8: for each class cf ∈ Yfine do 9: for each entity efi ∈ {ef1 , . . . , efK} do
10: P̂ (efi )← (Eq.6) 11: NS[P̂ (efi )]← NS[P̂ (efi )] ∪ {efi } 12: end for 13: end for 14: // Prototype-class assignment with conflict 15: for each prototype pi ∈ P do 16: if NS[i] ̸= ∅ then 17: Y (pi)← argmin
cf {d(h(efj ),pi) | efi ∈ NS[i]}
18: end if 19: end for 20: return {Y (p1), . . . , Y (p|P|)}
entity (ei, yi), we estimate its probability to yi by:
p(yi|ei) = exp(−∑pj∈P(ci) d(h(ei),pj))
|Yfine|∑ k exp(−∑pj∈P(ck) d(h(ei),pj)) (9)
Then, the loss is computed as the negative loglikelihood Lmixpt = −log p(yi|ei) of the true class yi. In this way, the representations of multiple clusters that belongs to each fine class are forced to gather together and draw closer to the representations of few-shot samples, while become more discriminative to the representations from other clusters. To further adapt the representations, we add an additional prototype distance loss for the prototype learning:
Lpt =
|Yfine|∑
k
∑
pi,pj∈P(ck) i ̸=j
d(pi,pj) (10)
As the mixture prototype loss and prototype distance loss are only calculated for entity representations, we also calculate the CrossEntropy Loss as in Eq.3. Finally, the training loss can be written as:
Lfine = Lce + Lmixpt + Lpt (11)

heading: 4 Experiments
text: In C2FNER, the models are first trained on a coarsegrained dataset and then adapted to a fine-grained
dataset. To comprehensively evaluate the C2FNER task, we conduct experiments with multiple choice of coarse- and fine-grained settings. Two fine granularity To evaluate the model with unknown fine granularity, we construct datasets of two fine granularity based on the Few-NERD dataset (Ding et al., 2021): (a) Fine: containing the original 66 fine-grained classes; (b) Medium-fine: we combine similar classes in the 66 classes and construct a new granularity including 38 classes, referred to as the Medium-fine dataset. 2 Multiple coarse-grained datasets We adopt three widely-used datasets for coarse-grained training, corresponding to two C2FNER settings: (a) Indomain: We construct a coarse-grained dataset based on the Few-NERD dataset with the 8 coarse classes hierarchical to the 66 fine-grained classes. (b) Cross-domain: We adopt the CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes 5.0 (Weischedel et al., 2013) datasets from the newswire domain as the coarse-grained datasets for the cross-domain C2FNER setting.

heading: 4.1 Few-shot Experiment Settings
text: For different coarse- and fine-grained choices, we follow the evaluation settings from (Simple, container) and evaluate the methods with all-way 1-shot and 5-shot support set and the standard test set. For both settings, we sampled 3 different fewshot training sets. For each sampled set, we repeat experiments 3 times. Each reported result is the average result of 3×3 experiments. Evaluation Metric General NER researches adopt the entity-level micro-f1 score for evaluating the models. However, due to the long-tailed class distribution of the fine-grained test set, evaluating with the micro-f1 score less reflect the performance on the small classes. Therefore, we report both the macro-f1 and micro-f1 scores for a more
2The details of the dataset construction are in Appendix A.3.
comprehensive evaluation.

heading: 4.2 Baselines
text: We evaluate the C2FNER task with various baseline methods: 3. General few-shot methods: We include directly fine-tuning on BERT (Devlin et al., 2019) with a token classifier as a general baseline, referred to as BERT. Besides, the Prototypical Network (Snell et al., 2017), denoted as ProtoNet, is a typical few-shot learning method. We also include SCL (Khosla et al., 2020; Gunel et al., 2020), the supervised contrastive learning method that has shown great advantages in few-shot NLP tasks (Zhang et al., 2021; Li et al., 2021). Few-shot NER methods: NNShot and StructShot (Yang and Katiyar, 2020) is two few-shot NER methods based on token-level nearest neighbor classification. CONTaiNER (Das et al., 2022) is an up-to-date few-shot NER method that leverages a novel contrastive learning technique. Coarse-to-fine method: FCDC (An et al., 2022) is a recently published research on coarse-tofine category discovery for text classification.
3The implementation details are in Appendix A.1
Specifically, it proposes a hierarchical weighted self-contrastive network for learning a fine-friendly representation during the coarse training.

heading: 4.3 Main Results
text: Table 2 and Table 3 show the results of the proposed method and the baseline methods on the C2FNER task on the in-domain and cross-domain settings, respectively. From the results, we can observe that: (1) The few-shot NER methods, although effective on previous few-shot NER settings, are less advantageous on the C2FNER task. This might be because these methods are mainly designed for generalizing to classes of the same or similar level of granularity, thus are less beneficial for coarse-to-fine generalization. (2) Surprisingly, the FCDC method, which is proposed for discovering fine-grained classes with coarsegrained annotations on text classification, is less effective on C2FNER. This might be because the hierarchical self-contrastive method assumes different granularities exist in different layers of BERT, while in token-level tasks like NER, the token-level representations are naturally more finegrained than the sentence-level representations. (3)
The proposed method shows significant advantages on both in-domain and cross-domain C2FNER settings. On the in-domain setting, the proposed method achieves up to 12.67% micro-f1 score and 13.31% macro-f1 score improvement on 1- shot learning over the baseline methods. On the cross-domain setting, all the methods show decreased results due to the data distribution shift across domains, among which the proposed method consistently outperforms all baseline methods. (4) It is also worth noting that the proposed method shows more advantages regarding the macro-f1 results, indicating that it might better benefit the long-tailed classes. We further illustrate this phenomenon in Sec. 4.6.

heading: 4.4 Ablation Study
text: We conduct an ablation study on the proposed method, as shown in Tab.4. w/o CPML means we didn’t adopt the Cluster-based Prototype Margin Loss during coarse training, and directly performed clustering on BERT and then used Prototype Retrieval and Mixture Prototype Learning. w/o PR&MPL means we didn’t adopt the Prototype Retrieval and Mixture Prototype Learning and w/o CPML w/o PR&MPL is the pure BERT model without any of our methods. We can see that both CPML and PR&MPL do contribute to the model improvement. The CPML is able to learn appropriate representations for fine classes. The PR&MPL is more effective regarding the macro-f1 results, and more interesting, it can improve the results limitedly when performing without CPML, since it is hard to retrieve appropriate clusters for fine classes without CPML.

heading: 4.5 Effect of the Cluster Number
text: To train the model with Cluster-based Prototype Margin Loss, we first perform clustering on the coarse classes with K-means clustering. In our implementation, we set the cluster number in each class to 50. Figure 3 shows the effect of different
cluster numbers on the Fine dataset. We can see that the performance grows as the cluster number increases. This might be because the proposed prototype retrieval and mixture prototype learning method are based on regarding each fine class as a mixture distribution, which is ensured by performing over-clustering on the coarse classes. Additionally, when the cluster number is larger than 50, the performance is relatively stable, showing the robustness of the prototype retrieval and mixture prototype learning method towards different over-clustered representations.

heading: 4.6 Benefiting Long-tailed Classes
text: As observed in Table 2 and Table 3, the proposed method is more advantageous regarding the macrof1 scores, which might be related with the performance of long-tailed classes. In Table 5, we show the results on the smallest classes of different methods. We can see that the proposed method indeed outperforms the baseline methods by a large
margin on these long-tailed classes, which might credit to the over-clustering and Prototype Retrieval method that learn and align small clusters to their corresponding fine classes.

heading: 4.7 Visualization of Coarse Representations
text: In figure 4, we visualize the representations of different models after coarse training. We can observe that both the NNShot and BERT show less separable representations after coarse training due to disregard for the structure of sub-classes. By training with Cluster-based Prototype Margin Loss, we obtain group-discriminative representations that benefit coarse-to-fine learning. Furthermore, we visualize the prototypes (⋆) retrieved through Prototype Retrieval of each fine class. We can see that the Prototype Retrieval algorithm can retrieve various representative clusters for each fine class with a certain accuracy.

heading: 5 Related Works

heading: 5.1 Coarse and Fine Learning
text: The coarse and fine learning problem has been studied in many works. Early works (Ristin et al., 2015; Guo et al., 2018; Taherkhani et al., 2019; Hsieh et al., 2019; Robinson et al., 2020) on coarseto-fine learning often assume the knowledge of fine classes and even a number of samples of fine classes is available during coarse training, which is less practical to meet the continually-updating user requirements. Recently, several works have explored similar scenarios with C2FNER (Bukchin et al., 2021; An et al., 2022; Yang et al., 2021) and proposed effective solutions. These methods typically focus on learning adaptive representations during coarse training by leveraging contrastive
learning (Bukchin et al., 2021; An et al., 2022), meta-learning (Yang et al., 2021) or cluster-based (Ni et al., 2022) methods. In this work, we propose a new solution to learn good representations during coarse training. Moreover, we are the first to consider different target granularity and provide solutions to improve the few-shot learning part.

heading: 5.2 Few-shot Learning for NER
text: Recently, many works have focused on few-shot NER. Yang and Katiyar (2020) improved the prototypical network with a nearest-neighbor-based learning method. Huang et al. (2021) conducted intensive experiments on different methods for fewshot NER. Tong et al. (2021) also leverages a cluster-based method for few-shot NER, yet it only focuses on clustering the none-entity class. More recently, (Das et al., 2022) leverages contrastive learning for few-shot NER. Many works (Cui et al., 2021; Ma et al., 2022; Chen et al., 2022a,c; Huang et al., 2022b) explore better leveraging the pretrained models with prompt learning or generationbased methods. Generally, these studies only consider few-shot generalization across classes of similar granularity, and are less effective when facing coarse-to-fine situations. To the best of our knowledge, this work is the first to consider and improve coarse-to-fine transfer learning for the NER task.

heading: 5.3 Relation with Entity Typing
text: The proposed C2FNER task is partially related to Entity Typing (Choi et al., 2018; Ma et al., 2016; Del Corro et al., 2015), a task that classifies recognized entities into fine entity types. Recently, several works on few-shot entity typing have
achieve great performance on various benchmarks (Huang et al., 2022a; Ding et al., 2022; Eberts et al., 2020). In C2FNER, we aim to adapt a NER model trained on coarse classes to recognize finer classes. When the fine classes are strictly included in the coarse classes, we can simply train a fewshot fine-grained entity typing model to accomplish the coarse-to-fine transfer. However, such practice is restricted to the in-domain setting and cannot be applied to more general situations when the fine classes are not strictly included in the coarse classes. Also, the practice requires the deployment cost of an extra model. Therefore, the proposed method to adapt the NER model is more efficient and more generally applicable.

heading: 6 Conclusion
text: In this work, we introduce the Coarse-to-fine Fewshot NER (C2FNER) task, which considers the practical situation when the models are required to be updated to more fine-grained classes. To solve the C2FNER task, we propose a novel method, which learns fine-friendly representations via a Cluster-based Prototype Margin Loss, and further benefits few-shot learning for various fine granularity via a Prototype Retrieval and Mixture Prototype Learning method. We provide a comprehensive benchmark for C2FNER with different settings, where the proposed method consistently outperforms all baseline methods.

heading: 7 Limitations
text: There are several limitations of this work: (1) In this work, we leverage a cluster-based method for improving the representations during coarse training. Regarding the clustering, we simply perform a K-means process to obtain cluster assignment. The problem is, the cluster assignment with one-time clustering, especially based on the fine-tuned BERT model, might be incorrect. Training with incorrect cluster assignments will pull together the instances from different fine classes, thus hindering the performance. Therefore, the proposed method can be further improved by: (a) leveraging a more robust learning method that can tolerate incorrect cluster assignments; (b) Improving the clustering process to obtain more accurate cluster assignments. (2) In this work, we aim to extend existing few-shot NER to a practical yet understudied setting. However, we limit the setting to the few-shot coarse-to-
fine transfer learning setting as in previous works (Bukchin et al., 2021; An et al., 2022; Yang et al., 2021; Ni et al., 2022). In practice, there might be more complex situations that require to be explored in further works.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by the National Natural Science Foundation of China (No.62076069,62206057,61976056), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A Appendix
text: A.1 Implementation Details
For the proposed method and all baselines, we implemented based on the BERT-base-cased pretrained model4. We trained the models for 3 epochs during coarse training and trained the models for 20 and 30 epochs during 5-shot and 1-shot, respectively. For our method and BERT, we set batchsize=32 and learning_rate=5e-5 on coarse training and batchsize=4 and learning_rate=1e-4 on few-shot learning. For other methods, we used their default parameters. For all methods, we choose the last model after few-shot training for evaluation. All our experiments are conducted on NVIDIA GeForce RTX 3090.
For our method, we performed clustering using the K-means clustering (Hartigan and Wong, 1979) 5 on the BERT model trained after the first epoch, and then re-initialized and trained on the coarse classes for 3 epochs. We set the cluster number of each coarse class to 50 for both indomain and cross-domain settings. We set the margin m = 2 for the in-domain setting and m = 0.2 for the cross-domain setting. For the prototype boosting, we set T = 3,K = 6.
A.2 Visualization of More Baselines
In Figure 5, we show the t-SNE visualization of more competitive baselines. As we can see, all baseline models show less separable representations after coarse training due to disregard for the structure of sub-classes, while the proposed model can achieve quite fine-friendly
4https://github.com/huggingface/transformers 5https://scikit-learn.org/stable/modules/generated/
sklearn.cluster.KMeans.html
representations.
A.3 Details of the Medium-fine dataset In order to evaluate different methods with various fine granularity, we constructed a Medium-fine dataset (39 entity classes) based on the original Fine dataset (66 entity classes) by merging similar classes into new classes. The details of the merged classes can be found in 6. For other details of the Fine dataset, please refer to (Ding et al., 2021).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 4,Appendix A
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? inapplicable

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we use are publicly available. And we need to perform NER tasks that involve identifying the names of people, thus the data are usually not anonymized.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4, Appendix B
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: The goal of coarse training is to provide a finefriendly representation for future fine-grained adaptation when ensuring the coarse performance. However, general supervised learning or few-shot learning methods aims at maximizing the interclass margin of coarse classes without considering the sub-classes included in the coarse classes. Such maximization of inter-class variance will harm the intra-class variance, leading to less separable subclasses and unfriendly representations for coarseto-fine transfer. In this work, we propose a cluster-based prototype margin loss for coarse training, which increases the intra-class variance of each coarse class during training as well as learning a good semantic structure of features. Specifically, semantically similar entities are pulled together while being separable from other entities. Moreover, the discrimination of coarse classes should be retained, thus not hindering the performance of the coarse classes. Therefore, we use a prototype for representing a group of semantically similar entities and achieve the above goal with a prototype margin loss. Formally, we first assign the entities in each coarse class cr to a set of groups Gcr = {gi}NGi=1,Gcr ⊂ G by performing clustering on the representations of all entities from cr. We denote M(ei) as the assigned group (or cluster) of entity ei (i. e. , ei ∈ M(ei)), where M(·) is a mapping function. Then, we obtain the prototype representation pk of each group k by:
pk = 1 |gk| ∑
ei∈gk h(ei) (1)
where h(ei) is the default representation of ei from the pre-trained PLM encoder. Then, the prototype margin loss for ei is calculated by:
Lpm,i = max(0,m+ d(h(ei),pM(ei))
− min pn∈G\Gyi d(h(ei),pn))
(2)
where m is a pre-defined margin. pM(ei) is the assigned prototype of ei. pn is a negative prototype selected from other coarse classes. d(·) denotes a distance function, which we use the Euclidean distance in our implementation. To optimize the prototype margin loss, each entity is forced to get closer to its prototype as well as keep a distance from prototypes from other classes. Consequently, semantically similar entities are pulled together and are discriminative to other groups of entities, thus contributing to friendly representations for coarseto-fine transfer learning. As the prototype margin loss is calculated only for entity representations, we also train a classifier with the Cross-Entropy Loss for token classification (including the none-entity class):
Lce = − 1 N
N∑
i
log exp(W⊤c h(xi) + bc)∑|Ycoarse|
j=1 exp(W ⊤ j h(xi) + bj)
(3)
where Wc and bc is the weight and bias of class c. The final training loss can be written as:
Lcoarse = Lce +
Ne∑
i
Lpm,i (4)
where Ne is the entity number in the training set