Rethinking Negative Instances for Generative Named Entity Recognition (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities for generalizing in unseen tasks. In the Named Entity Recognition (NER) task, recent advancements have seen the remarkable improvement of LLMs in a broad range of entity domains via instruction tuning, by adopting entity-centric schema. In this work, we explore the potential enhancement of the existing methods by incorporating negative instances into training. Our experiments reveal that negative instances contribute to remarkable improvements by (1) introducing contextual information, and (2) clearly delineating label boundaries. Furthermore, we introduce an efficient longest common subsequence (LCS) matching algorithm, which is tailored to transform unstructured predictions into structured entities. By integrating these components, we present GNER, a Generative NER system that shows improved zero-shot performance across unseen entity domains. Our comprehensive evaluation illustrates our system’s superiority, surpassing state-of-the-art (SoTA) methods by 9 F1 score in zero-shot evaluation.1.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a critical and challenging task in the field of Natural Language Processing (NLP). Previous NER models are constrained by a pre-defined label set and require extensive human annotations, which limits their flexibility and adaptability to unseen entity domains. Recent advantages in LLMs have enabled the models to be capable of generalizing to unseen tasks (Ouyang et al., 2022; Achiam et al., 2023) in an auto-regressive generation manner, making it possible to construct powerful NER systems. However, despite these advancements, recent studies (Wei et al., 2023; Li et al., 2023) show that
∗Corresponding author 1Code, datasets, and models are publicly available: https:
//github.com/yyDing1/GNER
the zero-shot performance of LLMs still falls behind the supervised training state-of-the-art (SoTA) methods, as LLMs train with limited NER data.
To bridge this gap, recent works have fine-tuned open-sourced LLMs on diverse NER datasets, enhancing their domain adaptability for NER tasks. They utilize varied task schemas to handle NER tasks across multiple domains. Specifically, InstructUIE (Wang et al., 2023) is fine-tuned on a wide range of IE datasets using a single-round conversation manner. Meanwhile, UniversalNER (Zhou et al., 2023) found that querying all entities at once is less effective than making multiple inquiries, with each inquiry focusing on one entity type at a time. Additionally, GoLLIE (Sainz et al., 2023) enhances zero-shot performance with well-crafted code-style guidelines. However, these approaches primarily adopt an entity-centric training strategy, focusing on recognizing entities while overlooking the non-entity text, which is crucial as negative instances. Actually, negative instances play an important role in traditional classification models like
3461
BERT Tagging (Devlin et al., 2018). For generative models, the role of negative instances in the training process has not yet been fully explored.
To calibrate the potential enhancement of including negative instances in training, we first conduct a preliminary study. Through experiments, we show that negative instances can significantly boost the model’s performance by (1) incorporating the contextual information, and (2) enhancing the label boundary between entities and non-entities. The possible drawback of introducing the negative instances is the increase of the prediction length, leading to inaccurate predictions, reflected by the word addition, omission and substitution. To tackle the inaccuracy drawbacks, we aim to design a more accurate and efficient algorithm to convert unstructured text into structured entities.
Inspired by the above observations, we design an effective and efficient Generative NER framework named GNER. We first design a proper task schema integrating negative instances into the instruction tuning process. Additionally, we design an LCS Matching algorithm to tackle the issues in the structuring process efficiently. This innovation ensures accurate categorization and alignment of extracted entities. We also demonstrate that zero-shot performance can be enhanced with beam search through a self-correction mechanism. These strategic developments collectively advance the GNER framework, setting a new standard for accuracy and efficiency in the field of NER.
We conduct experiments on two representative generative models, Flan-T5 and LLaMA. The resulting models, GNER-T5 and GNER-LLaMA, outperform SoTA by a large margin. As stated in Fig. 1, GNER-LLaMA-7B outperforms the GoLLIE (Sainz et al., 2023) trained on Code-LLaMA7B by 6 F1 score. Furthermore, compared to the similarly configured model UniversalNER, GNERLLaMA-7B shows an improvement of 12.7 F1 score, with a 2.5× boost in inference speed. We also showcase the potential of smaller models with our 780M GNER-T5-large model, which outperforms all baseline models in both zero-shot and supervised scenarios.

heading: 2 Related Work
text: Named Entity Recognition Early works format Named Entity Recognition (NER) as a sequence labeling problem (Chiu and Nichols, 2016; Huang et al., 2015; Akbik et al., 2018; Qin et al., 2019; De-
vlin et al., 2018), utilizing the BIO-Tagging scheme. Then, different methods are proposed to address more complex scenarios, i.e., nested and discontinuous NER. These methods regard NER as question answering (Li et al., 2020a; Mengge et al., 2020), span classification (Fu et al., 2021; Li et al., 2020b), dependency parsing (Yu et al., 2020), word-level relation classification (Li et al., 2022a), and so on. In most of these approaches, negative instances have played a crucial role in the training process, either by integrating all negative instances or employing sampling methods to select part of them (Li et al., 2022b). However, the performance of the above-mentioned supervised models significantly decreases in zero-shot settings (Liu et al., 2021), especially when the data and domain distribution significantly diverge from those seen of training.
Zero-shot NER Instruction tuning (Wei et al., 2021; Chung et al., 2022), also known as multitask fine-tuning, has emerged as a leading method to achieve generalization to unseen tasks by finetuning pre-trained LLMs on a diverse collection of tasks phrased as text-to-text problems (Longpre et al., 2023). In NER, numerous works have explored the potential of LLMs across diverse domains. For instance, InstructUIE (Wang et al.,
2023) is fine-tuned on a wide range of IE datasets and achieves impressive results in both zero-shot and supervised settings. UniversalNER (Zhou et al., 2023) explores the effectiveness of knowledge distillation and multi-round conversational training paradigms in enhancing model generalization, achieving superior results. GoLLIE (Sainz et al., 2023) introduces an innovative strategy by integrating well-crafted code-style guidelines into instructions, which has been found to further improve the model’s zero-shot performance. A simplified version of the task schema of the mentioned methods above is shown in Fig. 2. It can be concluded that the task schema plays an important role in determining the learning paradigm of models, significantly influencing their performance. We observe that these methods are entity-centric, meaning only the entity portions are involved in training and used for backpropagation to train the model.

heading: 3 Incorporating Negative Instances
text: In this section, we start from the entity-centric schema and explore the possible improvements by incorporating negative instances. Through experiments, we demonstrate the impact of negative instances in mitigating Unlabeled Errors (UE), Noisy Errors (NE), and Boundary Errors (BE) by (1) introducing contextual information and (2) enhancing entity boundaries.

heading: 3.1 Definition & Settings
text: The Named Entity Recognition (NER) task can be formally defined as a function mapping of input tokens X = {x1, x2, . . . , xn} and a pre-defined set of entity types L = {l1, l2, . . . lm} to entity labels Y = {y1, y2, . . . , yn}. The positive and negative instances can be formulated as follows:
P = {(xi, yi) | i ∈ {1, . . . , n}, yi ∈ L} , N = {(xi, yi) | i ∈ {1, . . . , n}, yi = O} , (1)
where O represents non-entity text. For the experimental setup, we choose Flan-T5large, a model with 780M parameters, as our backbone. Additionally, we sample 10K samples from the Pile-NER (Zhou et al., 2023) dataset as our training set and 200 samples for each subtask of CrossNER (Liu et al., 2021) validation set to evaluate the model’s zero-shot performance. To conduct a more detailed evaluation, in addition to the F1 score, we introduce the following three metrics to assess model performance:
Unlabeled Error (UE) The model fails to recognize the entity and labels it as “O”.
Noisy Error (NE) The model mistakenly label an entity with another incorrect entity tag.
Boundary Error (BE) The model correctly predicts the entity type but fails to identify its full extent, either capturing only a portion of the entity or resulting in overlaps.

heading: 3.2 Learning with Entity Context
text: We integrate the contextual information before and after an entity into our training process to explicitly enable the model to recognize entities based on their surrounding context. Specifically, we introduce negative instances that are closest to the entity, extending up to a length L, until encountering the boundary of the sentence, as part of our training instances. An example of our constructed training prompt is shown in Fig. 3, and the corresponding results are summarized in Table 1 and Fig. 4.
The context surrounding entities plays a signif-
icant role in determining their categories, with a notable improvement observed when increasing the contextual length from 0 to 1. As the contextual length increases, performance progressively improves, showing that the model benefits from the context, and the ratio of noisy error (NE) significantly decreases, largely contributing to the improvement in the final F1 score. Qualitatively, we further analyze through case studies why the closest negative instances contribute to improvement. We discover that the model tends to learn more from the context, such as “flew to” prompting the model to focus more on the following entity “New York” instead of merely memorizing “New York” as a location. We also experiment by placing terms with multiple meanings, such as Jordan, Amazon, and Mercury, after “flew to” and observe that the model consistently identified them correctly.

heading: 3.3 Entity Boundary of Generative Model
text: The above analysis has demonstrated the effectiveness of entity contexts, where labels are applied exclusively to entity parts. We then adopt BIOtagging (Huang et al., 2015) to enrich all label information, with the training prompt as shown in Fig. 3. The results in Table 1 indicate that the introduction of BIO tagging effectively strengthens the boundaries around entities, leading to a significant reduction in boundary errors. As shown in Fig. 4, compared to training with contexts alone, incorporating the BIO-tagging strategies results in consistent improvements across various contextual lengths. In addition to BIO tagging, we also try the BIOES tagging scheme, based on the intuition that the BIOES tagging method provides a stronger delineation of entity boundaries. However, we find
that the performance of BIOES tagging is not as good as BIO tagging. The results are listed in Table 1. Upon further analysis, we discover that the BIOES tagging seems harder to learn under the auto-regressive generation manner, where each token is predicted sequentially. For instance, both “B-” and “S-” can serve as the beginning of an entity, but only “B-” can be followed by “I-”, which may confuse the model for subsequent words.

heading: 3.4 The role of negative instances
text: The improvement in model performance (F1 score) can be explained as follows:
Precision and Recall To put it more directly, the improvements over entity-centric approaches are primarily reflected in (1) Precision: The context surrounding an entity often leads to a more accurate determination of its type, and (2) Recall: The model is guided to make judgments on every token in a sentence (including those in non-entity texts), which helps recall more entities. We observe improvements in both recall and precision, which in turn lead to an increase in the F1 score.
Less Unlabeled, Noisy and Boundary Error As indicated in Table 1, context helps mitigate Unlabeled and Noisy Errors, while BIO tagging strengthens entity boundaries, thereby reducing Boundary Errors. These reductions in errors directly contribute to improved performance.

heading: 4 Method
text: In this section, we present our GNER framework. We start by describing our task schema, which integrates negative instances into the training process for better usage of contextual information (section 3.2) and sensitivity to the entity boundaries (section 3.3), followed by the correlated tuning strategies. We also propose an effective longest common subsequence (LCS) matching algorithm, to convert the model’s unstructured text outputs into structured data efficiently, thereby enhancing the accuracy of our system.

heading: 4.1 Task Schema
text: Integrating negative instances, specifically those parts of the sentence labeled as “O” to indicate non-entity text, enhances the generative process by including contextual information and the discrimination of entity boundaries, thereby boosting the model’s performance, as detailed in section 3.
Due to the token-by-token generation paradigm of generative models, we design a token-by-token prediction task schema, where the model predicts the category of each token as it generates them, either entities or non-entities. This schema offers a more direct and focused way, where each token is annotated individually and assigned a specific entity label based on its context within the sequence.

heading: 4.2 Instrucion Tuning
text: Instruction Format As shown in Fig. 5, our designed instruction prompt includes four parts: task description, guideline, input, and output. To enhance our model’s ability to generalize across diverse labels and effectively handle real-world data, we implement some regularization strategies: (1) class order shuffling, where the order of entity classes is randomly shuffled, and (2) external entity sampling2, involving the entity types that are absent in the given sentence in the training prompt.
Task Adaption & Supervised Fine-tuning Zeroshot capabilities of LLMs in NER are limited due to their exposure to relatively little NER data during training. To equip the model with capabilities specific to NER tasks, we first perform task adaptation on NER data spanning various domains. Subsequently, to assess the model’s zero-shot capabilities, we evaluate it against unseen entity types. We proceed to extensively fine-tune our models on a wide range of publicly available NER data, aiming to enhance our model’s effectiveness in supervised settings, followed by supervised evaluations.
2Zhou et al. (2023) refers to this as negative entity sampling, which is different from the negative instances discussed in this work. We term it “external” to differentiate it.

heading: 4.3 LCS Matching Algorithm
text: The prediction length of the model increases with the integration of entity contexts and BIO tagging strategies. A longer generation sequence might bring challenges to the popular generative LLMs. The model’s output may include omissions, additions, and substitutions of words. We launch a detailed case study and find the potential causes of these issues: (1) noise in the original text, (2) missing words in the vocabulary, and (3) accumulative exposure bias. The representative examples, issue proportion, along with detailed analysis, are documented in Appendix C.
To handle these problems, we develop a LCS Matching algorithm that provides a straightforward and effective solution to these challenges. Formally, given a sentence X = {x1, x2, . . . , xn}, the generated outputs can be formatted as “x̃1(ỹ1) x̃2(ỹ2) . . . x̃m(ỹm)”. Firstly, we utilize regular expression matching to obtain the predicted sequence X̃ = {x̃1, x̃2, . . . , x̃m} and the corresponding answers Ỹ = {ỹ1, ỹ2, . . . , ỹm}. Due to the inherent uncertainties in generation, X̃ often differs from X . Next, we establish a oneto-one correspondence between the words in the original sequence X and the generated sequence X̃ , then map the labels of the corresponding words Ỹ back to obtain the final prediction results Ŷ . A common method involves calculating the Longest Common Subsequence (LCS) between X and X̃ to identify the correspondence between words, using the classic dynamic programming algorithm with time complexity of O(N2). Combined with the actual NER task scenarios and our task schema, we make the optimizations in the matching algorithm and condition. The resulting algorithm can handle these issues effectively with a time complexity of O(N logN). The optimization in matching conditions (i.e., Back Tokenization) further enhances the robustness across various models in our system. More details concerning the optimization can be seen in Appendix D.

heading: 5 Experiments

heading: 5.1 Settings
text: Datasets The datasets used in our experiments include: (1) Task Adaptation Datasets: Following the setting of Zhou et al. (2023), we first train our model with Pile-NER, which consists of approximately 240K entities across 13K distinct entity categories. These passages are sampled from the
Pile Corpus (Gao et al., 2020) and subsequently processed using ChatGPT to generate the inherent entities openly. To evaluate the model’s zero-shot performance in unseen entity types, we adopt two widely-used datasets, i.e., CrossNER (Liu et al., 2021) and MIT (Liu et al., 2013). (2) Supervised Datasets: Following the task adaptation phase, the performance of the model can be further enhanced by training across a wide range of well-annotated NER datasets (Zhou et al., 2023). To achieve this, we compile 18 public NER datasets in the BIO format for additional training, subsequently assessing performance on the test splits of these 18 datasets. From the 20 datasets used in Wang et al. (2023), we exclude two nested NER datasets, ACE2005 and GENIA, due to their incompatibility with the BIO format. Following the settings of Wang et al. (2023), we randomly select 10K data points from each dataset to create a mixed set. In cases where a dataset contains fewer than 10K samples, we incorporate its entire dataset. Additional information regarding the datasets is available in Appendix A.
Compared Baselines Our main point of comparison is UniversalNER (Zhou et al., 2023) as it is the approach closest to our system, with similar data and training procedures. Another baseline considered for comparison is GLiNER (Zaratiana et al., 2023), which utilizes bi-directional models to match entity types with textual spans in a latent space. We also include some strong Information Extraction (IE) systems like InstructUIE (Wang et al., 2023), which is based on Flan-T5-xxl (Chung et al., 2022) and fine-tuned on diverse informa-
tion extraction datasets, and GoLLIE (Sainz et al., 2023), which is based on Code-LLaMA (Roziere et al., 2023), and use guidelines to improve model’s zero-shot performance. We use strict entity-level micro-F1 as the evaluation metric for comparison. Previous work lack a uniform setting; UniNER removed the “else” entity type in the CrossNER dataset, while InstructUIE, GoLLIE, and GLiNER retained it. To standardize the settings, we reevaluated their methods using their publicly released checkpoints and code, ensuring that all test set configurations are consistent with UniNER.
Backbones & Implementation Generative models typically consist of two types of architectures, i.e., the encoder-decoder architecture and the decoder-only architecture. We conduct experiments on both of these architectures. Specifically, we select Flan-T5 (encoder-decoder) and LLaMA (decoder-only) as our backbone models. To ensure a fair comparison, our training settings for GNERT5 align with those of InstructUIE (Wang et al., 2023), and those for GNER-LLaMA are consistent with UniversalNER (Zhou et al., 2023). Due to our model producing longer output sequences, we implement longer length limits for both input and output. More details can be found in Appendix B.

heading: 5.2 Zero-shot Evaluation
text: We evaluate the zero-shot performance of our models after the domain adaptation phrase. Table 2 summarizes the results. Our model demonstrates significant improvements compared to other models. Significantly, although our GNER-LLaMA
model shares the same backbone model (LLaMA7B) and dataset (Pile-NER) with UniNER (Zhou et al., 2023), it demonstrates a notable improvement. Our results show that our 7B model outperforms the UniNER model of the same scale by approximately 12.7 F1 score points on average, and exhibits improvements across every dataset. Remarkably, our 7B model surpasses the UniNER 13B model by 10.5 points. When considering smaller backbone models such as GNER-T5-base and GNER-T5-large, it’s noteworthy that they also outperform all the aforementioned strong baselines.

heading: 5.3 Supervised Evaluation
text: To test our model’s performance on supervised data, we conduct supervised multi-task fine-tuning based on the NER-specialized model. The results are
summarized in Table 3 and 4. We first compare our approach with two closely related baselines, InstructUIE and UniNER, as we share the same backbone model and train with similar data. As a result, our method demonstrates significant improvements over these baselines: GNER-T5 achieves a 4.6-point increase in the F1 score, while GNERLLaMA shows a 1-point F1 score improvement. Moreover, we observe consistent enhancements across almost all datasets. We also experiment with smaller models, considering both effectiveness and inference efficiency. As shown in Table 4, our GNER-T5-large model, with only 10% the parameter size of UniNER, achieves superior performance and boasts 10× the inference efficiency.

heading: 5.4 Ablation Results
text: We have demonstrated the effectiveness of negative instances in section 3. In this part, we conduct the ablation study to evaluate the performance of our LCS Matching Algorithm. Our focus lies in two aspects: (1) how optimization in the algorithm
increases the model’s inference efficiency, and (2) how optimization in the matching condition improves the model’s performance. For the former, we evaluate the algorithm’s acceleration across various sentence length ranges, as shown in Table 5. As sentence length increases, the acceleration effect of our algorithm becomes more pronounced. For sentence lengths between 100 and 200, it achieves an average acceleration factor of 17.3. For the latter, we remove the Back Tokenization procedure from LCS and eliminate all LCS processes. The results, presented in Table 6, indicate that removing Back Tokenization and the whole LCS algorithm leads to a decrease in effectiveness, underscoring the efficacy of our LCS Matching algorithm.

heading: 6 Analysis
text: Scaling Law of Generative NER Models Our experiments show that even smaller models like
Flan-T5-large possess significant potential. We investigate the scaling law of Generative NER tasks in both zero-shot and supervised settings. The results are illustrated in Fig. 7. In the zero-shot setting, our methods scale well with model size. As the model size increases, the zero-shot capability of the model continues to rise, showing ample potential for further improvement with even larger models. In the supervised setting, our 783M model already demonstrates strong multi-task generalization abilities, and as the model size increases further, the improvements tend to converge.
Self-Correction Mechanism via Beam Search Beam search can enhance the performance of generative models by expanding the search space to include multiple hypotheses at each generation step. Previous research (Yan et al., 2021) has demonstrated that applying beam search in an entitycentric generation does not improve the model’s performance or even degrade it. We conduct experiments on UniNER and our model, the results of which are shown in Table 7. We discover that as the beam size increases, the performance of the UniNER model decreases. In contrast, we observe a consistent improvement with beam search under our task schema. Upon a detailed case study of the model’s generated results, we found that our task schema possesses a self-correction mechanism. The model retains some other hypotheses while generating subsequent results. In the decoding process that follows, the model can correct earlier mistakes. As demonstrated in Fig. 6, the model revises its previous incorrect prediction of “O” for “the fog” upon encountering the subsequent token “rated”. This token is crucial for identifying the entity type associated with “the fog”. A detailed analysis is provided in Appendix E.

heading: 7 Conclusion
text: This paper explores the potential of a strong Generative Named Entity Recognition (NER) system based on pre-trained LLMs by integrating the negative instances into training. Through experiments, we have demonstrated significant advancements. Our approach, which combines the inclusion of contextual information and a clear definition of entity boundaries through negative instances, has proven to be highly effective in improving the model’s performance, especially in zero-shot scenarios where prediction uncertainty is high. The introduction of an LCS Matching algorithm further addresses the challenges of converting unstructured text into structured entities, ensuring accurate categorization and alignment. These findings highlight the crucial role of negative instances in NER tasks and the potential of generative models to revolutionize the field.

heading: 8 Limitation
text: Despite our system achieving impressive results, there remain limitations and space for improvement. In task settings, our approach focuses on the main-stream Flat-NER settings, where entities appear as continuous text segments, without addressing the discontinuous forms, i.e., discontinuous NER. Actually, it has always been challenging for generative models to adopt a unified paradigm to resolve all the complex settings. Previous entitycentric methods can address the discontinuous settings but fail to manage polysemy, where a phrase corresponds to different entity types in different sentence parts. The primary focus of this paper is to explore the impact of negative instances in the training process, and we will explore a unified framework for generative models in future work.
Ethics Statement
In this paper, we utilize the pre-trained large language models, i.e., Flan-T5 and LLaMA, as the foundational models. It’s important to acknowledge that these models may contain inherent biases resulting from their pre-training processes. However, this issue is mitigated through our finetuning process, which refines the models to specifically concentrate on the Named Entity Recognition (NER) task, thereby reducing potential biases. Moreover, we strictly use all datasets and corpora in our study for scientific research purposes only.

heading: Acknowledgements
text: We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and Supercomputing Center in Yancheng, Grant No. 20231001.

heading: A Data Statistics and Pre-processing
text: We show the full dataset statistics in Table 8, including the domain of datasets, the number of instances in train/valid/test data, and their download address. In particular, we have to pre-process the Pile-NER (Zhou et al., 2023) dataset to fit in our task schema. We observe nuances between our compiled datasets and those referenced by Wang et al. (2023) and Zhou et al. (2023). Specifically, their MultiNERD and PolyglotNER datasets omit the last 10,000 training samples. Furthermore, they miss the last sample in some datasets for the validation and test sets, such as CrossNER politics, MIT Movie, and MIT Restaurant. We have included these omitted instances in our dataset, adhering to the original dataset compositions. The modifications have a negligible impact on our results. This is because our sampling approach aligns with those used in the referenced studies, ensuring that the number of data instances sampled from each training set, up to 10,000 samples, is consistent. Moreover, adding a single extra sample in the test sets hardly affects the final results.

heading: B Hyper-parameters settings
text: In our experiments, we train all models using a batch size of 256, employing the AdamW optimizer (Loshchilov and Hutter, 2018) for optimization. For the T5 model, we set a constant learning rate of 5 × 10−5 and impose a length limitation of 640 tokens for both the encoder and decoder. For the LLaMA model, we adopt a cosine learning rate schedule, initiating with a warm-up phase that covers 4% of the training steps, ramping up to a learning rate of 2×10−5, followed by a decay phase for the remainder of the training steps. The length limitation is set to 1280. Due to our prediction sequences being longer, more training steps are required. The number of training epochs for our models varies by size: 20 epochs for both the small and base models, 10 epochs for the large and xl models, and 6 epochs for the xxl model. For the LLaMA model, we set the number of epochs to 3. We observe an interesting phenomenon that the T5 model often requires more training steps to converge. A possible explanation is that the backbone model, Flan-T5, an instruction-tuned model without any Named Entity Recognition (NER) related data in the instruction-tuning process, requires more training steps to adapt to the NER task.

heading: C Problems in long sequence
text: In response to the issues in long predictions mentioned in section 4.3, we conduct a detailed case study. The representative examples are presented in Table 9. The problems can primarily be categorized into word omission, addition, and substitution, with omission and substitution accounting for the majority. We can conclude the following causes:
Noise in the original text Some of the issues can be attributed to noise in the original text. For example, in case 4, the model corrects “manattan” to “manhattan”, and in case 7, it corrects the misuse of “the”. However, we also observe that the model can introduce errors, as seen in cases 6 and 8, where the entities with repeated words, “norz norz norz” and “wet wet wet”, confuse the model.
Missing words in the vocabulary Furthermore, we find that a certain proportion of issues can be derived from missing words in the model’s vocabulary. As a result, these words naturally do not appear in the model’s output. For instance, in case 10, “brontë” was replaced with “bront” because “brontë” does not exist in the vocabulary. We also discovered that several special characters do not exist in the T5 vocabulary, leading to more occurrences of omission and substitution.
Accumulative exposure bias The issue of repetitive generation of words and phrases is common in long text generation (LTG) due to the accumulative exposure bias as the prediction length increases. As illustrated by cases 3 and 9, the model produces meaningless and repetitive information.

heading: D Optimization in LCS Matching
text: Optimization in Matching Algorithm We optimize the complexity of the LCS algorithm using a hierarchical divide-and-conquer approach through the following steps: (1) If the sequence does not have the above problem, i.e., X̃ = X , it is obvious that Ŷ = Ỹ . The time complexity is O(N), (2) For the omission case, where X̃ is a subsequence of X , the matching process can be accomplished in O(N) through greedy matching. (3) In other cases, we have implemented a fast version of the LCS algorithm (Hunt and Szymanski, 1977) within O(N logN), based on the nature of the small number of duplicate words in X̃ . Our experimental results in Table 6 demonstrate that the optimization can significantly enhance efficiency, achieving up
to a 17.3 times speedup for long sequences compared to the naive O(N2) implementation.
Back Tokenization One notable problem in the matching process is the missing words in the vocabulary, as detailed in our case study in Appendix C. For example, “antropología” in the original text becomes ‘antropologa’ in the model’s predictions, resulting in an inaccurate match in the matching process. To address this, we employ back tokenization, which involves tokenizing each word in the original text and then detokenizing it to match a word in the model’s vocabulary, thereby creating a more resilient matching condition.

heading: E Self-correction Mechanism
text: In this section, we conduct a case study to explore (1) the reasons behind the reduced effectiveness of entity-centric methods like UniNER (Zhou et al., 2023) when beam search is applied, and (2) the specific enhancements of the self-correction mechanism in our task schema. Upon comparing UniNER’s performance with and without beam search, we observe that beam search leads to the model responding with the same answers across a variety of entity-type queries. For our models, we provide representative examples in Table 10 to
illustrate the self-correction mechanism’s impact, showcasing (1) enhanced precision in determining entity boundaries (cases 1 and 2), (2) the use of contextual clues to recognize inherent entities (cases 3, 6 and 7), and correct mistakes (cases 4 and 5).

heading: F Detailed Evaluation Results
text: We detail the performance of our models across all datasets in Table 11, including error bars for zeroshot performance derived from the variance of five separate runs. For the supervised settings, we do not conduct multiple runs due to the extensive size of the datasets, where the training and inference process can be very time-consuming. Our trials with smaller models indicate that the variability, or error bars, for models in the supervised settings is minimal, approximately around 0.15.

heading: G Environmental Impact
text: Training huge models can have a negative impact on the environment. All our models are trained on the hardware of a single A100 node (8× NvidiaA100-80G-SXM4) with approximately 800 GPU hours in total. The carbon footprint estimation is 135.3 kg CO2eq according to Wu et al. (2022).

heading: Future Work
text: The primary focus of this paper is to explore the impact of negative instances in the training process, and we will explore a unified framework for generative models in future work