An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition
Year: 2023

Abstract:
Named entity recognition (NER) is the task to detect and classify entity spans in the text. When entity spans overlap between each other, the task is named as nested NER. Span-based methods have been widely used to tackle nested NER. Most of these methods get a score matrix, where each entry corresponds to a span. However, previous work ignores spatial relations in the score matrix. In this paper, we propose using Convolutional Neural Network (CNN) to model these spatial relations. Despite being simple, experiments in three commonly used nested NER datasets show that our model surpasses several recently proposed methods with the same pre-trained encoders. Further analysis shows that using CNN can help the model find more nested entities. Besides, we find that different papers use different sentence tokenizations for the three nested NER datasets, which will influence the comparison. Thus, we release a pre-processing script to facilitate future comparison. 1

heading: 1 Introduction
text: Named Entity Recognition (NER) is the task to extract entities from raw text. It has been a fundamental task in the Natural Language Processing (NLP) field. Previously, this task is mainly solved by the sequence labeling paradigm through assigning a label to each token (Huang et al., 2015; Ma and Hovy, 2016; Yan et al., 2019). However, this method is not directly applicable to the nested NER scenario, since a token may be included in two or more entities. To overcome this issue, the spanbased method which assigns labels to each span is introduced (Eberts and Ulges, 2020; Li et al., 2020; Yu et al., 2020).
∗Equal contribution. †Corresponding author.
1Code is available at https://github.com/yhcc/CNN_ Nested_NER
Eberts and Ulges (2020) use a pooling method over token representations to get the span representation, and then conduct classification on this span representation. Li et al. (2020) transform the NER task into a Machine Reading Comprehension (MRC) form, they use the entity type as the query, and ask the model to select spans that belong to this entity type. Yu et al. (2020) utilize the Biaffine decoder from dependency parsing (Dozat and Manning, 2017) to convert the span classification into classifying the start and end token pairs. However, these work does not take advantage of the spatial
1442
correlations between adjacent spans. As depicted in Figure 1, spans surrounding a span have special relationships with the center span. It should be beneficial if we can leverage these spatial correlations. In this paper, we use the Biaffine decoder (Dozat and Manning, 2017) to get a 3D feature matrix, where each entry represents one span. After that, we view the span feature matrix as a spatial object with channels (like images) and utilize Convolutional Neural Network (CNN) to model the local interaction between spans.
We compare this simple method with recently proposed methods (Wan et al., 2022; Li et al., 2022; Zhu and Li, 2022; Yuan et al., 2022). To make sure our method is strictly comparable to theirs, we ask the authors for their version of data. Although all of them use the same datasets, we find that the statistics, such as the number of sentences and entities, are not the same. The difference is caused by the usage of distinct sentence tokenization methods, which will influence the performance as shown in our experiments. To facilitate future comparison, we release a pre-processing script for ACE2004, ACE2005 and Genia datasets.
Our contributions can be summarized as follows.
• We find that the adjacent spans have special correlations between each other, and we propose using CNN to model the interaction between them. Despite being very simple, it achieves a considerable performance boost in three widely used nested NER datasets.
• We release a pre-processing script for the three nested NER datasets to facilitate direct and fair comparison.
• The way we view the span feature matrix as a spatial object with channels shall shed some light on future exploration of span-based methods for nested NER task.

heading: 2 Proposed Method
text: In this section, we first introduce the nested NER task, then describe how to get the feature matrix. After that, we present the CNN module to model the spatial correlation on the feature matrix. A general framework can be viewed in Figure 2.

heading: 2.1 Nested NER Task
text: Given an input sentence X = [x1, x2, . . . , xn] with n tokens, the nested NER task aims to extract all
entities in X . Each entity can be expressed as a tuple (si, ei, ti). si, ei are the start, end index of the entity. ti ∈ {1, . . . , |T |} is its entity type and T = {t1, ..., tn} is entity types. As the task name suggests, entities may overlap with each other, but different entities are not allowed to have crossing boundaries. For a sentence with n tokens, there are n(n+ 1)/2 valid spans.

heading: 2.2 Span-based Representation
text: We follow Yu et al. (2020) to formulate this task into a span classification task. Namely, for each valid span, the model assigns an entity label to it. The method first uses an encoder to encode the input sentence as follows:
H = Encoder(X),
where H ∈ Rn×d, and d is the hidden size. Various pre-trained models, such as BERT (Devlin et al., 2019), are usually used as the encoder. For the word tokenized into several pieces, we use maxpooling to aggregate from its pieces’ hidden states.
Next, we use a multi-head Biaffine decoder (Dozat and Manning, 2017; Vaswani et al., 2017) to get the score matrix R as follows:
Hs = LeakyReLU(HWs),
He = LeakyReLU(HWe),
R = MHBiaffine(Hs,He)
where Ws,We ∈ Rd×h, h is the hidden size, MHBiaffine(·, ·) is the multi-head Biaffine decoder2, and R ∈ Rn×n×r, r is the feature size. Each cell (i, j) in the R can be seen as the feature vector v ∈ Rr for the span. And for the lower triangle of R (where i > j), the span contains words from the j-th to the i-th (Therefore, one span will have two entries if its length is larger than 1).

heading: 2.3 CNN on Feature Matrix
text: As shown in Figure 1, the cell has relations with cells around. Therefore, we propose using CNN to model these interactions. We repeat the following CNN block several times in our model:
R′ = Conv2d(R),
R′′ = GeLU(LayerNorm(R′ +R)),
where Conv2d, LayerNorm and GeLU are the 2D CNN, layer normalization (Ba et al., 2016) and GeLU activation function (Hendrycks and Gimpel, 2016). The layer normalization is conducted in the feature dimension. A noticeable fact here is that since the number of tokens n in sentences varies, their Rs are of different shape. To make sure results are the same when R is processed in batch, the 2D CNN has no bias term, and all the paddings in R are filled with 0.
2The detailed description is in the Appendix A.1.

heading: 2.4 The Output
text: We use a perceptron to get the prediction logits P as follows: 3
P = Sigmoid(Wo(R+R ′′) + b),
where Wo ∈ R|T |×r, b ∈ R|T |, P ∈ Rn×n×|T |. And then, we use golden labels yij and the binary cross entropy to calculate the loss as:
LBCE = − ∑
0≤i,j<n yij log(Pij),
More special details about our proposed method during training and inference procedure are described in Appendix A.

heading: 3 Experiment

heading: 3.1 Experimental Setup
text: To verify the effectiveness of our proposed method, we conduct experiments in three widely used nested NER datasets, ACE 20044 (Doddington et al., 2004), ACE 20055 (Walker and Consortium, 2005) and Genia (Kim et al., 2003).
3We did not use the Softmax because in the very rare case (such as in the ACE2005 and Genia dataset), one span can have more than one entity tag.
4https://catalog.ldc.upenn.edu/LDC2005T09 5https://catalog.ldc.upenn.edu/LDC2006T06
Besides, we choose recently published papers as our baselines. To make sure our experiments are strictly comparable to theirs, we ask the authors for their versions of data. The data statistics for each paper are listed in the Appendix B. For ACE2004 and ACE2005, although all of them use the same document split as suggested (Lu and Roth, 2015), they use different sentence tokenizations, resulting in different numbers of sentences and entities. To facilitate future research on nested NER, we release the pre-processing code and fix some tokenization issues to avoid including unannotated text and dropping entities. While for the Genia data, there are some annotation conflicts. For examples, one document with the bibliomisc MEDLINE:97218353 is duplicated in the original data, and different work has different annotations on it. We fix these conflicts. We replicate each experiment five times and report its average performance with standard derivation.

heading: 3.2 Main Results
text: Results for ACE2004 and ACE2005 are listed in Table 1, and results for Genia is listed in Table 2. When using the same data from previous work, our simple CNN model surpasses the baselines with less or similar number of parameters, which proves
that using CNN to model the interaction between neighbor spans can be beneficial to the nested NER task. Besides, in the bottom block, we reproduce some baselines in our newly processed data to facilitate future comparison. Comparing the last block (processed by us) and the upper blocks (data from previous work), different tokenizations can indeed influence the performance. Therefore, we appeal for the same tokenization for future comparison.

heading: 3.3 Why CNN Helps
text: To study why CNN can boost the performance of the nested NER datasets, we split entities into two kinds. One kind is entities that overlap with other entities, and the other kind is entities that do not. We design 4 metrics NEPR, NERE, FEPR and FERE, which are flat entity precision, flat entity recall, nested entity precision and nested entity recall, respectively.6, and list the results in Table 3. Compared with models without CNN, the NERE with CNN improve for 2.2, 2.8 and 10.7 on ACE2004, ACE2005 and Genia respectively. Namely, much of the performance improvement can be ascribed to finding more nested entities. This is expected as the CNN can be more effective for exploiting the neighbor entities when they are nested.

heading: 4 Related Work
text: Previously, four kinds of paradigms have been proposed to solve the nested NER task.
The first one is the sequence labeling framework (Straková et al., 2019), since one token can be
6The detailed calculation description of the 4 metrics locates in the Appendix D.
contained in more than one entities, the Cartesian product of the entity labels are used. However, the Cartesian labels will suffer from the long-tail issue.
The second one is to use the hypergraph to efficiently represent spans (Lu and Roth, 2015; Muis and Lu, 2016; Katiyar and Cardie, 2018; Wang and Lu, 2018). The shortcoming of this method is the complex decoding.
The third one is the sequence-to-sequence (Seq2Seq) framework (Sutskever et al., 2014; Lewis et al., 2020; Raffel et al., 2020) to generate the entity sequence. The entity sequence can be the entity pointer sequence (Yan et al., 2021; Fei et al., 2021) or the entity text sequence (Lu et al., 2022). Nevertheless, the Seq2Seq method suffers from the time-demanding decoding.
The fourth one is to conduct span classification. Eberts and Ulges (2020) proposed to enumerate all possible spans within a sentence, and use a pooling method to get the span representation. While Yu et al. (2020) proposed to use the start and end tokens of a span to pinpoint the span, and use the Biaffine decoder to get the scores for each span. The span-based methods are friendly to parallelism and the decoding is easy. Therefore, this formulation has been widely adopted (Wan et al., 2022; Zhu and Li, 2022; Li et al., 2022; Yuan et al., 2022). However, the relation between neighbor spans was ignored in previous work.

heading: 5 Conclusion
text: In this paper, we propose using CNN on the score matrix of span-based NER model. Although this method is very simple, it achieves comparable or better performance than recently proposed methods. Analysis shows exploiting the spatial correlation between neighbor spans through CNN can help model find more nested entities. And experiments show that different tokenizations indeed influence the performance. Therefore, it is necessary to make sure all comparative baselines use the same tokenization. To facilitate future comparison, we release a new pre-processing script for three nested NER datasets.
Limitations
While we discover that simply applying CNN on top of the score matrix of span-based NER model performs well on the nested NER scenario, there are still some limitations that are worth discussing. Firstly, we mainly choose three commonly used
nested NER datasets, which may lack generalization. Secondly, we only focus on nested NER tasks for the spatial relations between spans are more intuitive and common in nested scenario than those in flat NER. However, the principle of using CNN to model the relations is also applicable to spans in the flat NER task. Future work can take flat NER into consideration based on our exploration, and experiments on more datasets.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. We also thank the developers of fastNLP7 and fitlog8. This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027) and CCF-Baidu Open Fund.

heading: A Detailed Proposed Method
text: A.1 Multi-head Biaffine Decoder
The input of Multi-head Biaffine decoder is two matrices Hs,He ∈ Rn×h, and the output is R ∈ Rn×n×r. The formulation of Multi-head Biaffine decoder is as follows
S1[i, j] = (Hs[i]⊕He[j]⊕w|i−j|)W, {H(k)s }, {H(k)e } = Split(Hs), Split(He),
S (k) 2 [i, j] = H (k) s [i]UH (k) e [j] T ,
S2 = Concat(S (1) 2 , ...,S (K) 2 ),
R = S1 + S2,
where Hs,He ∈ Rn×h, h is the hidden size, w|i−j| ∈ Rc is the span length embedding for length |i − j|, W ∈ R(2h+c)×r, S1 ∈ Rn×n×r, r is the biaffine feature size, Split(·) equally splits a matrix in the last dimension, thus, H(k)s ,H (k) e ∈ Rn×hk ; hk is the hidden size for each head, and U ∈ Rhk×rk×hk , S2 ∈ Rn×n×r, and R ∈ Rn×n×r.
We do not use multi-head for W , because it does not occupy too many parameters and using multihead for W harms the performance slightly.
A.2 Training Loss
Unlike previous works that only use the upper triangle part to get the loss (Yu et al., 2020; Zhu and Li, 2022), we use both upper and lower triangles to calculate the loss, as depicted in section 2.4. The reason is that in order to conduct batch computation, we cannot solely compute features from the upper triangle part. Since features from the lower triangle part have been computed, we also use them for the output. The tag for the score matrix is symmetric, namely, the tag in the (i, j)-th entry is the same as that in the (j, i)-th.
A.3 Inference
When inference, we calculate scores in the upper triangle part as:
P̂ij = (Pij + Pji)/2,
where i ≤ j. Then we only use this upper triangle score to get the final prediction. The decoding process generally follows Yu et al. (2020)’s method. We first prune out the non-entity spans (none of its scores is above 0.5), then we sort the remained spans based on their maximum entity score. We pick the spans based on this order, if a span’s boundary clashes with selected spans’, it is ignored.

heading: B Data
text: We list the statistics for each dataset in Table 4.10 As shown in the table, the number of sentences and even the number of entities are different for each paper on the same dataset. Therefore, it is not fair to directly compare results. For the ACE2004 and ACE2005, we release the pre-processing code to get data from the LDC files. We make sure no entities are dropped because of the sentence tokenization. Thus, the pre-processed ACE2004 and ACE2005 data from this work in Table 4 have the most entities.
10The number of entities is different from that reported in their paper, because we found some duplicated sentences in their data.
And for Genia, we appeal for the usage of train/dev/test, and we release the data split within the code repository. Moreover, in order to facilitate the document-level NER study, we split the Genia dataset based on documents. Therefore, sentences from train/dev/test splits are from different documents, the document ratio for train/dev/test is 8:1:1. Besides, we find one conflicting document annotation in Genia, we fix this conflict. After comparing different versions of Genia, we find the W2NER (Li et al., 2022) and Triaffine (Yuan et al., 2022) drop the spans with more than one entity tags (there are 31 such entities). Thus, they have less number of nested entities than us. While SG (Wan et al., 2022) includes the discontinuous entities, so they have more number of nested entities than us.
C Implementation Details
We use the AdamW optimizer to optimize the model and the transformers package for the pretrained model (Wolf et al., 2020). The hyperparameter range in this paper is listed in Table 5.

heading: D FEPR FERE NEPR NERE
text: We split entities into two kinds based on whether they overlap with other entities, and the statistics for each dataset are listed in Table 6. When calculating the flat entity precision (FEPR), we first get all flat entities in the prediction and calculate their
ratio in the gold. For the flat entity recall (FERE), we get all flat entities in the gold and calculate their ratio in the prediction. And we get the nested entity precision (NEPR) and nested entity recall (NERE) similarly.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations" (5th section)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section "Limitations" (5th section)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: "Abstract" and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 and Appendix C

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To facilitate future comparison, we release a new pre-processing script for three nested NER datasets. Limitations
While we discover that simply applying CNN on top of the score matrix of span-based NER model performs well on the nested NER scenario, there are still some limitations that are worth discussing. Firstly, we mainly choose three commonly used
nested NER datasets, which may lack generalization. Secondly, we only focus on nested NER tasks for the spatial relations between spans are more intuitive and common in nested scenario than those in flat NER. However, the principle of using CNN to model the relations is also applicable to spans in the flat NER task. Future work can take flat NER into consideration based on our exploration, and experiments on more datasets