Exploiting Hierarchically Structured Categories in Fine-grained Chinese Named Entity Recognition
Year: 2023

Abstract:
Chinese Named Entity Recognition (CNER) is a widely used technology in various applications. While recent studies have focused on utilizing additional information of the Chinese language and characters to enhance CNER performance, this paper focuses on a specific aspect of CNER known as fine-grained CNER (FG-CNER). FG-CNER involves the use of hierarchical, fine-grained categories (e.g. PersonMovieStar) to label named entities. To promote research in this area, we introduce the FiNE dataset, a dataset for FG-CNER consisting of 30,000 sentences from various domains and containing 67,651 entities in 54 fine-grained flattened hierarchical categories. Additionally, we propose SoftFiNE, a novel approach for FG-CNER that utilizes a custom-designed relevance scoring function based on label structures to learn the potential relevance between different flattened hierarchical labels. Our experimental results demonstrate that the proposed SoftFiNE method outperforms the state-of-theart baselines on the FiNE dataset. Furthermore, we conduct extensive experiments on three other datasets, including OntoNotes 4.0, Weibo, and Resume, where SoftFiNE achieved stateof-the-art performance on all three datasets.

heading: 1 Introduction
text: Named Entity Recognition (NER) (Li et al., 2020a; Nasar et al., 2021) is a fundamental component of natural language processing and has been widely studied to overcome the challenges brought by realworld text data. Chinese Named Entity Recognition (CNER) (Liu et al., 2022), as an important subfield of NER, has also drawn wide interests recently (Hao et al., 2013; Zheng et al., 2021; Liu et al., 2021a).
Fine-grained Named Entity Recognition (FGNER) is a key challenge in current NER research, motivated by the growing demand for more detailed
∗These authors contributed equally to this work.
categorization of named entities. Compared to traditional Coarse-grained Named Entity Recognition (CG-NER) (Sun et al., 2002; Mansouri et al., 2008), FG-NER requires identifying the correct label from a larger number of categories for each entity, making it more difficult than CG-NER (Ekbal et al., 2010; Ling and Weld, 2012). These challenges also apply to Fine-grained Chinese NER (FG-CNER). However, most recent research on FG-CNER has focused on addressing the difficulties posed by the nature of Chinese characters (such as word segmentation (Ye et al., 2021)) and leveraging special information that can be extracted from the Chinese language, such as glyphs (Xuan et al., 2020), phonetics (Pinyin) (Sun et al., 2021), and rich lexicons (Zhang and Yang, 2018; Li et al., 2020b). The challenge posed by the increasing amount of corpus lacks attention. In other words, these are languagecentric approaches that focus on leveraging special characteristics offered by the language, rather than data-centric approaches that exploit the potential advantages that the data itself can provide.
On the other hand, as the number of named entities increases, there is an increasing need to introduce hierarchical relations between different levels of categories. For example, “Jackie Chan” can be categorized as both a “Person” and a “Movie Star”, which can be intuitively represented by a flattened hierarchical label “Person-MovieStar”, where “Person” is the parent category and “MovieStar” is the child. Similarly, “Taylor Swift” can be labeled as “Person-Singer”. It is clear that “Person-MovieStar” and “Person-Singer” are relevant to each other by sharing the common category of “Person”. However, since NER tasks mostly apply sequence labeling to words (characters) (such as BIOES), these flattened hierarchical labels are usually encoded in a one-hot format (Peng and Dredze, 2015), which assumes that labels are mutually exclusive, thus ignoring the potential relevance among labels. A data-centric approach should leverage the relevance
3407
between hierarchical labels, such as the fact that "Jackie Chan" as a "Person-MovieStar" is closer to "Person-Singer" than many other categories, and use this knowledge to improve fine-grained NER performance.
Despite the importance of the research agenda outlined above for CNER applications, it has not received much attention in recent CNER literature. We believe this is due to a lack of a high-quality FGCNER dataset. Existing CNER datasets are either outdated (Levow, 2006; Weischedel et al., 2011) or small-scale (Peng and Dredze, 2015; Zhang and Yang, 2018), and also lack fine-grained hierarchical categories commonly used in CNER applications today. These limitations in datasets have become a bottleneck for current CNER research, making it difficult to develop more advanced approaches that can meet real-world challenges.
To address this bottleneck in Fine-grained Chinese Named Entity Recognition, we introduce the FiNE dataset, a FG-CNER dataset consisting of 30,000 sentences sampled from various domains, containing 67,651 entities falling into 54 finegrained flattened hierarchical categories. To the best of our knowledge, FiNE is currently the largest fully open-sourced (and test-dataset-included) FGCNER dataset with hierarchical labels.
To exploit the relatedness of fine-grained labels in FG-CNER and facilitate future research, we propose the SoftFiNE method, a novel approach for solving FG-NER problems based on soft labels. SoftFiNE removes the mutual exclusive assumption and captures the hidden relevance between flattened hierarchical labels. Additionally, we also introduce a properly designed entity augmentation method in SoftFiNE, which can further boost FG-CNER performance by teaching the model to gather more context information about a character.
Experimental results on the FiNE dataset show the effectiveness of SoftFiNE compared to other baselines, such as LEBERT (Liu et al., 2021b), HMCN-F (Sun et al., 2021), LambdaRank (Burges et al., 2006), etc. To address the bottleneck caused by the limited CNER dataset, we conduct extensive experiments on the Weibo (Peng and Dredze, 2015), Resume (Zhang and Yang, 2018) and OntoNotes 4.0 (Weischedel et al., 2011) datasets, which are widely studied CNER datasets released in the past decade. Although SoftFiNE method achieves new state-of-the-art (SOTA) performance, the improvement compared to the previous SOTA is limited
in terms of numbers, which also appears in recent CNER research (Mai et al., 2022; Sun et al., 2021; Liu et al., 2021b). This indicates that current datasets may no longer be able to support the design of more advanced CNER techniques due to their limitations, as mentioned earlier. To facilitate future research on FG-CNER, both the FiNE dataset and the source code of the SoftFiNE method are publicly available at https://github.com/XpastaX/SoftFiNE.

heading: 2 Related Work
text: CNER datasets. There are many open-source high-quality CNER datasets available. For example, People’s Daily is one of the earliest CNER datasets, with over 20,000 samples in four categories (person, location, organization, and date). MSRA (Levow, 2006) is one of the largest CNER datasets, with over 50,000 samples in three categories (person, location, and organization). These datasets have greatly contributed to the early research on CNER, however, their utility in current CNER development is limited due to their outdatedness. Furthermore, real-world applications are increasingly demanding more fine-grained categories to classify named entities.
OntoNotes Release 4.0 (Weischedel et al., 2011) is a large-scale annotated corpus that also includes annotated CNER data with 18 types of categories. However, the commonly used version is processed by Che et al. (2013), where only four categories are available. Resume (Zhang and Yang, 2018) is a smaller dataset with more categories. It contains about 4,500 samples from real resumes and classifies entities into eight categories. Weibo (Peng and Dredze, 2015) has about 2,000 samples with four main categories and two sub-categories for each, indicating whether the entity is a name mention or nominal mention, except “Geo-political”, which has no nominal mention. Although it has a hierarchy of labels, the structure is too simple. Thus, Peng and Dredze (2015) treat them as seven different labels while developing their model, which is also followed by later research.
Due to the limitations of existing CNER datasets, we decided to construct the FiNE dataset to facilitate the study of hierarchical FG-CNER.
Recent research on CNER. The development of neural networks has led to a growing interest in combining well-trained word embeddings with deep learning models to support CNER. Various
pictograph information, such as strokes (Sun et al., 2021), radicals (Dong et al., 2016; Xu et al., 2019), and glyphs (Meng et al., 2019), have been utilized to improve the performance of CNER. Additionally, the large lexicons in Chinese, where each word is a combination of characters, have been collected and leveraged to further improve CNER performance (Zhang and Yang, 2018; Li et al., 2020b; Liu et al., 2021b).
Recently, with the success of pre-trained language models in NLP, such as BERT (Devlin et al., 2019), extracting semantic information from target data has become easier, shifting research attention towards utilizing pre-trained knowledge to support advanced studies in CNER (Mai et al., 2022; Zhang et al., 2022a).
There is also research on FG-CNER (Zhang et al., 2020; Liu et al., 2020) that tries to handle more categories, but the number of categories in these works is still relatively small. Additionally, although Gong et al. (2020) utilized hierarchy information to facilitate their CNER task, the hierarchies they used are based on the tree structures of how characters construct a Chinese word. The hidden relevance between flattened hierarchical labels needs to be further studied. Therefore, based on the FiNE dataset, we propose the SoftFiNE method to provide a better solution for FG-CNER tasks.

heading: 3 Dataset
text: To facilitate research on Fine-grained Chinese Named Entity Recognition (FG-CNER) with hierarchical structured labels, we construct the FiNE dataset. A brief introduction of the CNER task in the BIOES (begin, inside, outside, singleton, end) labeling scheme is given in Appendix A.
Construction of hierarchical categories. The hierarchies of the categories in FiNE are collected and refined from Chinese Wikipedia1. Based on their released knowledge graph, we collect the labels of every entity and keep those with the highest frequencies. However, since the knowledge graph is generated from the open domain, these collected labels often have redundant meanings. We manually fine-grained the categories into at most 3 levels of hierarchies and removed those that are less acknowledged by most people. Then, following the sequence labeling method for the NER task, we flatten the hierarchies and obtain 54 categories with different numbers of hierarchy levels. A list of all
1https://dumps.wikimedia.org/zhwiki/
categories is provided in Figure 1 which includes 10 sub-labels for the first level, 29 sub-labels for the second level, and 32 sub-levels for the third level.
Annotation. We created a dataset that simulates real-world scenarios by collecting 30,000 passages from QQ Browser, which provides daily content to a general audience of over 300 million people. We selected passages from 27 different domains such as Medicine, Sports and E-Sports based on their click rates and used TextRank4ZH2 to extract key sentences. Four experts with professional knowledge in NLP were hired to label and classify the entities into hierarchical categories. To ensure accuracy, the 30,000 sentences were divided into three packs and labeled by three annotators, with the fourth double-checking for correctness. Additionally, 100 samples were manually labeled for each pack, and any discrepancies were resolved through re-labeling the whole pack. In the final version of FiNE, the fourth annotator agreed with 89.93 percent of all labels, with the remaining 10.07 percent
2https://github.com/letiantian/TextRank4ZH
removed from the dataset. As shown in Table 1, our dataset covers all categories in the validation and test sets, and has more fine-grained categories in hierarchical structures compared to existing CNER datasets. Additionally, our dataset is constructed with online passages from multiple domains, making it more representative of real-world FG-CNER implementation. A detailed distribution of all entities is provided in Appendix B.

heading: 4 Methodology
text: In this section, we present SoftFiNE, a solution to the hierarchical labeled fine-grained CNER problem that effectively utilizes the hidden relevance between flattened hierarchical labels to gain a better understanding of them. Before introducing our approach, we will discuss two alternative methods for solving hierarchical labeled FG-CNER.

heading: 4.1 Alternative Methods
text: There are two existing types of approaches to learning from the hierarchies.
Hierarchical Multi-label Classification. An alternative approach for solving FG-CNER with hierarchical labels is to treat it as a hierarchical multi-label classification task, where the model makes predictions on each sub-label to determine if it is related to the character and then reconstructs the hierarchies based on the predicted sub-labels (Giunchiglia and Lukasiewicz, 2020; Zhang et al., 2022b). However, this method can lead to accumulation of errors in the predictions of each sub-label, resulting in degraded performance. To evaluate the
effectiveness of this approach, we implemented the method introduced by Wehrmann et al. (2018) on the FiNE dataset.
Learning to Rank. Another approach for solving hierarchical FG-CNER is to treat it as a Learning to Rank problem, where the flattened hierarchical labels are ranked by relevance, and the most relevant label is used as the prediction (Cao et al., 2007; Liu et al., 2009). To enable this method, we use the rules in Table 2 to obtain a rank list for each character. For a given ground truth, we calculate its relevance in comparison to other labels by counting the number of missed levels (“miss”) and the number of wrong sub-labels (“wrong”). The relevance of the ground truth is set to 0, and for every other label, the relevance is calculated as 1 − miss − 2 × wrong. The score of the label "O" is set to negative infinity. To leverage the relevant information in the rank list, we implemented RankNet (Burges et al., 2005) and LambdaRank (Burges et al., 2006) on the FiNE dataset. However, if two labels have the same score, they cannot be compared using these methods and such label pairs are excluded during training.

heading: 4.2 SoftFiNE
text: In order to better leverage the relevance between labels, we propose SoftFiNE. Our approach is designed to be simple and focuses on capturing the hidden relevance between flattened hierarchical labels through proper augmentation. The detailed structure of SoftFiNE is illustrated in Figure 2.
Scoring Label. To better represent the relevance between flattened hierarchical labels, we design a label scoring method based on their hierarchical structures. For the ith character ti in a sentence T = {ti}1fifM , where M is the length of T , denotes its ground truth label with zi ∈ Z, where Z is the set of all possible flattened hierarchical labels with the total number of N . Z̄i = {zj}j ̸=i,1fjfN represents all other flattened hierarchical labels that are not the ground truth. zi has ¼i levels in the
format:    O, ¼i = 0 l seg i − l1i − non − non, ¼i = 1 l seg i − l1i − l2i − non, ¼i = 2 l seg i − l1i − l2i − l3i , ¼i = 3 ,
where “non” means no hierarchy at the corresponding level, l1i , l 2 i , l 3 i are the corresponding sub-labels of entity categories and lsegi ∈ {B, I, E, S} is the sub-label of BIOES segmentation. Notice characters labeled with “O”have no hierarchy. For each non-“O” label zx, we use cx to denote the set of its label components, “non” will not be included.
We first calculate how many levels are missing for each zi,j compared with zi by
missi,j = max(¼j − ¼i, 0).
Then, we count the number of wrong label components for each zj compared with zi:
wrongi,j = ∑
k
1ki,j ,
where 1ki,j = 1 if the k th component of cj not exists in ci, otherwise is zero. Finally, we employ the following designed scoring function to obtain the relevance to the ground
truth of all labels: ỹi = {ỹi,j}1fjfN , where
ỹi,j =   
1 β(missi,j+γ×wrongi,j) , i ̸= j, zj ̸= O
0, zj = O
1, otherwise
.
´ > 1 and µ > 1 are two hyperparameters. The larger µ will lead to a harder punishment on wronglevel components when scoring, the larger ´ will decrease the value of the scores of all flattened hierarchical labels zi,j .
The above scoring method can evaluate the degree of how a label is relevant to the ground-truth label, with a maximum degree of 1. The relevance difference is then integrated into those soft labels, which helps the model understand the relations between them. An example of evaluating the relevance between labels is provided in Table 2.
Weighted cross-entropy. To keep the model simple, we employ BERT with a softmax classifier to handle the CNER task.
For a given ti ∈ T , we encode the sequence with BERT (Devlin et al., 2019):
H = {hi}1fifM = BERT(T ), (1)
where hi is the feature of ti. H = {hi}1fifM is the features of all characters in sentence T = {ti}1fifM .
The result prediction is then
pi = softmax(Whi + b),
where W and b are the parameters of a single-layer linear projector. We let P = {pi}1fifM to represent the prediction of all characters in T , as shown in the bottom-left of Figure 2. The softmax(·) we used here is the softmax function for normalization.
Given the soft label ỹi, an intuitive way is to use the cross-entropy to build the training object:
ℓi = − N∑
j=1
p̃i,j log pi,j ,
where p̃i,j =
ỹi,j∑N k=1 ỹi,k , (2)
p̃i,j represents the normalized ground truth relevance score between ti and jth hierarchical label, while pi,j ∈ pi is the corresponding predicted normalized relevance score. However, the groundtruth label of different characters may have different numbers of hierarchical levels (¼), which
leads to a different scaling factor ∑N
k=1 ỹi,k. After the normalization in Equation (2), the value of the ground truth may be too low if the scaling factor is large, which makes it harder to learn from the sample. Thus, we simply use the scaling factor to weight the loss term of ti:
ℓ̃i = −( N∑
k=1
ỹi,k)
N∑
j=1 ỹi,j∑N k=1 ỹi,k log pi,j
= − N∑
j=1
ỹi,j log pi,j .
(3)
The loss L̃ for the whole sentence is then
L̃ = M∑
i=1
ℓ̃i
= − M∑
i=1
N∑
j=1
ỹi,j log pi,j .
(4)
Augmentation. The above approach helps the model understand the relation between labels from the hierarchies. To better leverage the information provided by the input samples, we propose a novel augmentation method that can help the model better understand the entity classification of CNER.
Table 3 gives an example of our augmentation method. For a given sentence T = {ti}1fifM , we generate its augmentation T aug = {taugi }1fifM . Recall the segmentation label types in BIOES are begin (B), inside (I), outside (O), singleton (S), end (E), denote the segmentation label of ti as segi, we have
t aug i = { segi, segi ∈ {B, I, E, S} ti, segi = O ,
Following Equation (3), we obtain the loss term of the augmented sentence T aug by
L̃aug = M∑
i=1
ℓ̃ aug i
= − M∑
i=1
N∑
j=1
ỹi,j log p aug i,j
where
p aug i = softmax(Wh aug i + b),
Haug = {haugi }1fifM = BERT(T aug). We use P aug = {paugi }1fifM to represent the predictions of all characters in the augmented sentence T aug. The segmentational labels are added to the vocabulary of the tokenizer of BERT. By doing this, our model can learn a general embedding of those labels, which grants a better understanding of the segmentation.
To teach BERT to gather more context information when predicting the labels of the character in entities for a given sentence, denote the features of all characters in all entities of T as He, and H aug e are those for the augmented sentence, we employ KL-divergence to close the distributions of ĥ aug x ∈ Hauge to ĥx ∈ He, where 1 f x f N̂ , and N̂ is the number of characters of all entities in the sentence T . Denote the distribution of ĥi as d̂i = softmax(ĥi), and the distribution of ĥ aug i as d̂ aug i = softmax(ĥ aug i ), we can get the feature loss term for ti:
ℓ̃feai = N̂∑
j=1
d̂i,j log d̂i,j
d̂ aug i,j
,
where d̂i,j and d̂ aug i,j are the j
th element in d̂i and d̂ aug i . The feature loss for the whole sequence is then:
L̃fea = M∑
i=1
ℓ̃feai
= M∑
i=1
N̂∑
j=1
d̂i,j log d̂i,j
d̂ aug i,j
.
By performing KL-divergence on the original feature and the augmentation feature, BERT learns to generate more robust outputs of the characters of the entities, thus making the classification much easier. Also, when encoding a segmentation label
in the augmented sentence, the model is taught to generate the feature that is close to the original one and make the correct prediction, which forces BERT to integrate the information of the whole sentence better to understand what to generate during the feature extraction.
Training object. Combing the above components, we form the training object of SoftFiNE with
L = L̃+ ³augL̃aug + ³feaL̃fea,
where ³aug and ³fea are two hyper-parameters we utilize to control the contribution of each kind of loss.
Combining the soft labels with the proposed augmentation approach, SoftFiNE can better understand the flattened hierarchical labels by learning the hidden relevance between them. The augmentation further grants a better understanding of sequence labeling by learning the embeddings of segmentation labels and using them to predict the categories. Moreover, our method has simple model architecture, and it only uses a BERT model with a linear projector for prediction, which grants a fast inference speed.

heading: 5 Experiments
text: We carry out an extensive set of experiments to investigate the effectiveness of SoftFiNE on FiNE and three other commonly used CNER datasets: OntoNotes 4.0, Weibo, and Resume. The statistics of those datasets are given in Table 1. Following previous literature on CNER, we report standard precision (P), recall (R), and micro F1-score (F1) to evaluate the performance. We compare SoftFiNE with baselines having valid official code on the FiNE dataset. Detailed experimental settings are given in Appendix C.

heading: 5.1 Results
text: Table 4 gives the results of SoftFiNE and selected baselines on our FiNE dataset. From the table, we can conclude some interesting findings: the BERT baseline outperforms all other baselines on FiNE; methods with a CRF decoder (Liu et al., 2021b) usually have lower performance than their implementation without CRF. We believe these are caused by the hierarchical structures of the labels, which are much harder compared with coarse-grain labels. Those flattened hierarchical labels are very likely to be relevant to each other, which challenges
the models more in distinguishing different categories, thus causing the degradation of the performance of CRF. Instead, by employing softmax normalization as decoders, BERT learns more directly from the samples and their labels. As a transformerbased PLM, its ability to understand tasks is much higher than most designed decoders on our FiNE, which leads to better performance. The hierarchical multi-label classification and learning-to-rank methods underperform the BERT baseline, proving our assumptions above. Moreover, we can observe that BERT does not outperform all baselines on the other three datasets, which have lower number of categories. Also, by comparing the results of RankNet and LambdaRank, we can conclude that it is efficient to include global relevance (ranking) information while training. By keeping the model in a simple structure and leveraging a welldesigned supervise method, our method achieves the best overall performance (F1) compared to all other models by at least 0.78.
With the supervision of generated soft labels, SoftFiNE can better understand the relevance of hierarchies to distinguish different flattened hierarchical labels. Moreover, the augmentation method also grants SoftFiNE a better vision of word segmentation and classification.
We implement our method on the other three commonly used datasets to further test the capability of SoftFiNE, and address the bottleneck of existing CNER datasets. The experimental results are shown in Table 5, from where SoftFiNE achieves better overall performance than all other state-of-
the-art baselines. This is because the BIOES sequence labeling method brings hierarchy information into the labels, which SoftFiNE can utilize to understand different labels better. However, the numerical improvement in performance is relatively small, which also commonly exists on other baselines. For example, SoftFiNE outperforms Mai et al. (2022) by 0.11 on OntoNotes 4.0, Mai et al. (2022) outperforms Ma et al. (2020) by 0.11 on Resume, and Sun et al. (2021) outperforms Ma et al. (2020) by 0.30 on Weibo. Although the improvements are numerically small, it does not mean the methods are not good. Instead, this is caused by the limitation of the current CNER dataset: they might be too easy or too small, while the models today are becoming more and more advanced in understanding languages and making precise predictions.

heading: 5.2 Ablation Studies
text: To examine the effectiveness of each component of SoftFiNE, we conduct ablation studies on the four datasets. The results are shown in the corresponding tables, where “-soft” represents SoftFiNE without using generated soft labels, and “-aug” represents SoftFiNE without augmentation.
From the results, we can conclude that both the soft label and the entity augmentation are keys for SoftFiNE to achieve better performance. The augmentation method can greatly boost the precision of the prediction, and it outperforms all baselines on the four datasets on precision. With such an augmentation method, SoftFiNE can learn the embedding of the segmentation labels by using their encoded features to classify recognized entities. Combined with the design of closing their encoded
features to those of their original characters, the BERT encoder in SoftFiNE is forced to gather more information from the context of an entity to make correct classification, which grants a high performance on precision.
The soft label method can outperform most baselines by leveraging the hierarchy information between flattened labels. SoftFiNE can leverage the hierarchy information brought by the BIOES sequence labeling method to find the hidden relevance between the different characters of an entity (e.g., “B-person” and “E-person”), and between the characters from different entities that have the same segmentation labels (e.g., “B-person” and “B-location”), which can help the BERT encoder achieve better performance even without finegrained hierarchical categories.
Combining the advantages of those two novel designs, our SoftFiNE can better understand both CNER and FG-CNER, and achieves the new stateof-the-art of the four datasets.

heading: 6 Conclusion
text: In this paper, to facilitate research on Fine-Grained Chinese Named Entity Recognition (FG-CNER) with hierarchical labels, we construct the FiNE dataset, which contains 30,000 high-quality samples with 54 inter-related categories organized in a hierarchical structure. The data are sampled from various domains of online passages and are closer to the requirements of real-world CNER implementation today. To exploit the hierarchical information present in fine-grained NER labels, we further propose the SoftFiNE method, a novel method aiming at enhancing the performance of FG-CNER by leveraging the hidden relevance
between flattened hierarchical labels and learning with relevance-aware soft labels. The proposed SoftFiNE method also uses entity augmentation to understand samples better. Experimental results show that SoftFiNE has achieved the best performance on four datasets, including FiNE, Weibo, Resume, and OntoNotes 4.0.
Limitations
Our constructed dataset, FiNE, has limitations in terms of entity category balance. Some categories have a lower number of online passages and less user attention, resulting in an unbalanced distribution of entities across categories. We aimed to simulate a real-world situation by sampling passages based on their click rates. But this may have contributed to the imbalance.
Additionally, our proposed method, SoftFiNE, is specifically designed for fine-grained Chinese named entity recognition with hierarchical categories, and thus has its own limitations. The model is kept simple in structure, with most efforts focused on developing supervision methods, which result in more hyperparameters and require a grid search to find the optimal hyperparameters. This can be resource-intensive. However, it has fast inference speed that is comparable to the BERT baseline in real-world applications. To address these limitations, future research could explore methods to automatically balance the loss ratio and dynamically score relevance between flattened hierarchical labels.
Ethics Statement
We sample all data in FiNE from QQ Browser, while the platform has already censored all samples. Therefore, the sensitive or personal information is neither present to the annotator nor included in the released dataset. The annotation of FiNE cost over 25,000 CNY. We hired four experts in NLP data annotation (through a third party), each of whom has at least one year of working experience in NLP data annotation. It took 672 work hours (8 hours per day for 21 days) to complete the annotation process. The hourly rate is about 37.20 CNY, higher than the standard local rate of 25.30 CNY.

heading: B Distribution of FiNE
text: Table 6 gives the distribution of all 67,651 sample in FiNE. We ensure there are at least 100 entities for each flat categories. Notice the number of entities in “Person” is not the summation of all entities whose flat categories has “Person” as their first level sub-label. Instead, they are the entities that can not be classified into any sub-level of “Person”. Same for “Location-Site” and “OrganizationCompany”. The categories in FiNE are designed to cover general online passages. Three levels of categories are enough to cover most topics without increasing the annotation cost or making it difficult to distinguish between very similar sub-categories.

heading: C Experimental Setting
text: For FiNE and OntoNotes 4.0, we set the batch size to 16, the learning rate to 2e−5, dropout rate to 0.5 for all models. We train the models for 20 epochs for FiNE and 5 epochs for OntoNotes 4.0 and use 5% training steps to warn up. For Weibo and Resume, we set the batch size to 4. The decay rates are set to 0.5 and 0.1 for Resume and Weibo, with the learning rates of 3e−5 and 2e−5 correspondingly. The dropout rates are 0.1 and 0.5. We train 12 epochs for those two datasets and utilize 10% of training steps to warm up.
For the hyper-parameters of SoftFiNE, we perform grid search to find the best combination. Specifically, for all datasets, we search (³aug, ³fea) among (0.1,0.1), (0.1,0.05), (0.01,0.01) and (0.01,0.005), ´ in {2, 3, 4} and µ in {1.5, 2, 2.5}. The result combination of (³aug, ³fea, ´, µ) are (0.1, 0.1, 4, 2) on FiNE and OntoNotes 4.0, (0.01, 0.01, 3, 2) on Weibo and (0.01, 0.005, 2, 2) on Resume.
For reproduced baselines in Table 4, we only use “bert-base-chinese”3 to extract the embeddings of tokens for all methods for a fair comparison, other experimental settings are same with their original paper.
We train all methods on a single NVIDIA Tesla A100 GPU. Similar to BERT, our parameter amount is less than 120M. Each epoch of FiNE takes 6 minutes to train, for the other datasets, it takes at most 4 minutes to train an epoch.

heading: D Additional Experimental Results
text: Table 7 gives the test performance of SoftFiNE on FiNE with different label levels. “SoftFiNElevel-1” gives the performance of only checking whether the first level is correctly predicted, “SoftFiNElevel-2” only checks the first two levels, and “SoftFiNElevel-3” checks all levels. For example, if a character has the true label “B-WorkApp-Software” but is wrongly predicted as “BWork-APP-Game”, it will be seen as a correct prediction by “SoftFiNElevel-1” since the first levels of the true label and the prediction are all “B-Work”. Similarly, it will also be seen as correct by “SoftFiNElevel-2” with both having “BWork-App”. However, the prediction is false for “SoftFiNElevel-3”, since the third levels of the two labels are different.
From Table 7, we can observe an apparent degra3https://huggingface.co/bert-base-chinese
dation in performance when considering more levels. This demonstrates the challenge brought by having hierarchical labels in the CNER task, which commonly exist in CNER applications today. We believe FiNE can greatly support future CNER research by providing entities with hierarchical structured fine-grained labels.

heading: E Case Study
text: Table 8 gives a real-case example in FiNE, together with the ground truth and predicted (by SoftFiNE) relevance scores of some hierarchical labels. From the table, we can observe that SoftFiNE can successfully predict the relevance between the ground truth label and other labels. For example, for character “s”, SoftFiNE can find the degree of relevance to “S-Work-Video-TVShow”, which has the same categorical label but different segmentational sub-label to its ground truth label “B-Work-VideoTVShow”. Such an ability to judge relevance between flattened hierarchical labels can help SoftFiNE better understand CNER tasks and perform better.
From Table 8, we can also find an interesting phenomenon that our model thinks labels with wrong segmentational sub-labels (i.e., B, I, O, E, S) are more relevant to the true label than labels with
wrong categorical sub-labels. For example, for character “s”, “B-Work-Video-VarietyShow” is less relevant to “B-Work-Video-TVShow” compared with “S-Work-Video-TVShow”. This may suggest that the segmentational sub-labels and the categorical sub-labels should have different weights when scoring the relevance between flattened hierarchical labels. Future research on FGCNER with hierarchical labels could follow the suggestions and either design a better relevance scoring function or iteratively refine relevance based on the predictions of their models.
Table 9 gives a bad case that commonly exists on SoftFiNE and all baselines. In the table, SoftFiNE predicts the relevance of all related flattened labels as zero, where, intuitively, it should at least give some scores greater than zero. One reason may be the polysemy in the Chinese language. While the original meaning of the sentence is “filming (the poster of) Better Days by using sachima (as background with photo editing techniques)”, in Chinese, it can also mean “slap the young you with sachima”, since “į” can either mean “film” or “slap” and “m Òú¯” means “young you” if directly translated into English. The models may incorrectly understand the sentence and thus fail to recognize “Better
Days” even if the entity indeed exists in the training set of FiNE. Moreover, the sentence itself is short, which contains less information to help the model correctly understand its meaning.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
3 A4. Have you used AI writing assistants when working on this paper? use Grammarly to check grammar
B 3 Did you use or create scientific artifacts? 3,5
3 B1. Did you cite the creators of artifacts you used? 3,5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethic
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 1
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Ethic statement
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future research on FGCNER with hierarchical labels could follow the suggestions and either design a better relevance scoring function or iteratively refine relevance based on the predictions of their models. Table 9 gives a bad case that commonly exists on SoftFiNE and all baselines. In the table, SoftFiNE predicts the relevance of all related flattened labels as zero, where, intuitively, it should at least give some scores greater than zero. One reason may be the polysemy in the Chinese language. While the original meaning of the sentence is “filming (the poster of) Better Days by using sachima (as background with photo editing techniques)”, in Chinese, it can also mean “slap the young you with sachima”, since “į” can either mean “film” or “slap” and “m Òú¯” means “young you” if directly translated into English. The models may incorrectly understand the sentence and thus fail to recognize “Better
Days” even if the entity indeed exists in the training set of FiNE. Moreover, the sentence itself is short, which contains less information to help the model correctly understand its meaning. ACL 2023 Responsible NLP Checklist