PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition
Authors: Jinghui Lu
Yanjie Wang
Ziwei Yang
Xuejing Liu
Brian Mac Namee
Can Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: NER
inference speedup
LLM
Primary area: natural_language_processing

Abstract: In this study, we aim to reduce generation latency for Named Entity Recognition (NER) with Large Language Models (LLMs). The main cause of high latency in LLMs is the sequential decoding process, which autoregressively generates all labels and mentions for NER, significantly increase the sequence length. To this end, we introduce Parallel Decoding in LLM for NE} (PaDeLLM-NER), a approach that integrates seamlessly into existing generative model frameworks without necessitating additional modules or architectural modifications. PaDeLLM-NER allows for the simultaneous decoding of all mentions, thereby reducing generation latency. Experiments reveal that PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22 times faster than the autoregressive approach for both English and Chinese. Simultaneously it maintains the quality of predictions as evidenced by the performance that is on par with the state-of-the-art across various datasets. All resources are available at https://github.com/GeorgeLuImmortal/PaDeLLM_NER.

PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition
Jinghui Lu 1x, Ziwei Yang Yanjie Wang 1x
Xuejing Liu Brian Mac Namee Can Huang
ByteDance
2 University of Chinese Academy of Sciences, China
School of Computer Science, University College Dublin
(lujinghui, yangziwei. 1221, wangyanjie ie prince, can hww.ghhbyteaancceee.....o xue
.co com bri wwt.mmaaameeeuc......
Abstract
this study, we aim to reduce generation latency for Named Entity Recognition (NER) with Large Language Models (LLMs). The main cause of high latency in LLMs is the sequential decoding process, which autoregressively generates all labels and mentions for NER, significantly increase the sequence length. To this end, we introduce Parallel Decoding in LLM for NER (PaDeI.LM-EER)) approach that integrates seamlessly into existing generative model frameworks without necessitating additional modules or architectural modifications. PaDeLLM- NER accelerates decoding by simultaneously generating all mentions once, i.e. a label-mention pair per sequence. This results in shorter sequences and faster inference. Experiments reveal that PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22 times faster than the autoregressive approach for both English and Chinese. Concurrently, maintains the prediction quality as evidenced by the micro F-score that is on par with the state-of-the-r approaches under both zero-shot and supervised setting. All resources are available httpp //github. com/
Introduction
Named Entity Recognition (NER), fundamental task in Natural Language Processing (NLP), aims to extract structured information from unstructured text data. This includes identifying and categorizing key elements such as Organization, Geopolitical Entity and so on (referred to as "labels") ") in inputs, and pairing them with relevant text spans extracted from the text (termed "mentions") "), Conventionally, NER tasks are carried out through an extractive paradigm that entails token-level classification and the subsequent extraction of identified tokens
have revolutionized numerous foundational tasks in NLP. including NER tasks 7 through the adoption of generative paradigm. This paradigm involves instruction-tuniis conniinnonnuinnggggg a sequence-to-sequees The model takes sequence of unstructured text as input and produces a sequence of structured label-mention pairs as output. Generally, the output structured string should be formatted to meet two criteria: (1) it should have clear and straightforward structure that facilitates post-processing for label and mention extraction, and (2) it needs to generated fluidly and efficiently from the perspective of language models [T8].
Recent advancements in Large Language Models (LLMs)
(seq2seq) model.
Equal contribution. X Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Variant
Input Unstructured Text
Output Structured Label-mention String
Augmented Language
1
Japan, cn-hosts of the World Cup in 2002 and ranked 20th in the world by FIFA, are favourites reguin their title here.
[Jaan [LOC), co co-hosts of the [World Cup MISC] in 2002 and ranked 20th in the world by [FIFA ORG). are favourites regain their title here.
Structured Annotation
7
Cuttitta announced his retirement after the 1995 World Cup where he took issue with
(PER)) (Curtitta), (MISC): (1995 World Cup). (LOC); (Iuly), (LOC): (England), (ORG)) (NULL.)
being dropped from the Italy side that faced England in the pool stages.
Table I: Structured output string format used in the literature. The examples come from CONLL20003 dataset.
In Table I we list two typically used autoregressive output formats found in the literature (1) accommodate original input text to contain label information, which is referred as "augmented language" B 4] (2) directly using a customized, easily-parsed structured format to output all labels and mentions, which is called "structured annotation" (5H7]. These formats present certain challenges. For example, augmented language necessitates duplicating all original input text, thereby increasing output length and resulting in inference inefficiency. While structure annotation avoids replicating the entire input, it produces all labels and mentions in an autoregressive manner. This implies that each subsequently generated pair depends on its preceding pairs, and when the number of label-mention pairs is large, it will lead to longer sequences. As demonstrated in Chen et al. [99, Ning et al. (20). high lateney in LLMs mainly stems from lengthy sequence generation, we believe that by reducing the length of sequence, more efficient inference scheme can be provided for NER tasks.
In light of this, we propose Parallel Decoding in LLM for NER (PaDeLLM-NER), a novel approach to accelerate the inference of NER tasks for LLMs. PaDeLLM-NNE empowers the model with the capability to predict a single label-mention pair within a single sequence, subsequently aggregating all sequences generate the final NER outcome. Specifically, in the training phase, we reconstruct the instruction tuning tasks, enabling LLMs to predict the count of mentions for a specific label and to identify the th mention within the entire input for that label (Figure 1p. In the inference phase, L.IMs first predict the number of mentions for all labels, then predict all Tabel-mentiog pairs in parallel (Figure 2.. Finally, results from all sequences are aggregated and duplicate mentions across labels are eliminated based on prediction probability. This approach results in more efficient inference method, producing shorter sequences and enabling parallel decoding label-mention pairs in batches.
Comprehensive experiments have been conducted, demonstrating that PaDeLLM-NER effectively reduces the number of tokens produced in each sequence, thereby decreasing inference latency. Additionally, it maintains or even enhances prediction quality in both flat and nested NER for English and Chinese languages, compared to existing methods in the literature under both zero-shot and supervised setting. To conclude, our contributions are as follows:
We present PaDeLLM- NER, a novel approach tailored for NER using LLMs. This approach can predict all tabel-mention pairs in parallel, effectively reducing inference latency.
Extensive experiments have been conducted, revealing that PaDeLLM-NER significantly improves inference efficiency. By completely decoupling the generation of fabel-mention pairs, the average sequence length is reduced to around 13% of that produced by conventional autoregressive methods. Correspondingly, the inference speed is 1.76 10.22 times faster than these previous approaches.
Comprehensive experiments demonstrate that, in addition to its enhanced prediction speed, PaDeLLMM-NER also maintains or surpasses the prediction quality of conventional autore- gressive methods, on par with state-of-the-ard (SOTA) performance on many NER datasets, including zero-shot as well as the supervised scenarios.
To the best of our knowledge, our technique stands as pioneering approach in accelerating NER inference in L.Mss by parallel decoding all label-mention pairs. This unique characteristic makes complementary to other inference acceleration methods such as i.tM. [2]] and speculative sampling (22 23 Thus, it can be efficiently integrated with these methods.

---

An training example
eatity type:jaPPRRoolnccums
Innemention I>Cutilta<00os>>
text:in Cuttitta
antounced. his
entity bype://M//SSitntnemmmm
I'm-cmention 1>1995 World Cup-eos>
retirement after
the 1995 World
Cup, where he
eatity uuu
Zyvemettion 1>italy 1205>
took issue with
being dropped
from Italy that faced
entity Cunnioogmmms..
Zyn-mmention 2=England<ees>>
England in the
pool stages.ja Jn
entity type:'nO Ral
0-2005>
Ingut: unstructured text and target label
Output: count of mentions and the o mention
Figure I: PaDeLLM-NER training paradigm: texts within frames of the same color represents one training example, where texts inside the solid-line frame are the input, and those inside the dashed-line frame are the output. Italic texts prompt templates. The "enfity type" signifies the label being predicted. The "<num>" indicates count of mentions for that label, and "<mention n> refers to the th mention of label in the input.
Related Work
2.1 Generative Models for NER
Before the era of LLMs, most research approached NER sequence labeling task, where each token is assigned a pre-defined tag (e.g., BIO scheme). In this line of work, usually pre-trained transformer-basee language models [1] 2 is combined with tailored prediction head to perform token-level classification, followed by the extraction of identified tokens.
Encouraged by the success of unifying multiple NLP tasks into single seq2seq paradigm [24] 25 especially with the evolution of LLMs 10
13 26 27,, the trend of applying seq2seq models to NER tasks is gaining momentum 228 with both inputs and outputs being represented as sequences of text 7 Recently, the focus of work on NER using LLMs has shifted towards zero-shot
29 or
few-shot learning 4 18
31 32 utilizing in-context learning [8 32
ceff-consistency
29 36 or
learning programming (30
Unlike previous studies emphasizing few-shot performance with training-free prompt learning, our work focus fully supervised setting. More importantly, our primary objective is to speed up NER inference.
2.2 Inference Speedup in LLMs
Modern LLMs employ a sequential decoding strategy for token generation, which poses a significant challenge in terms of parallelization, especially model size and sequence length increase [20]. There plenty of work in the literature address this challenge 35 One line of work falls into
training- free category such as introducing extra modules for speculative sampling 22 23 Another
approaches explore modifying model architecture to accelerate inference, such as exiting at earlier
layer [39 40), or designing entirely different training and inference mechanisms
4 Different
from previous works, we focus on exploring the inference speedup LLMs with focus on the NER task without the change of model architecture introducing extra modules.
3 Method
In this section, we delve into the details of PaDeLLM-NER. First, we focus on reframing the instruction tuning tasks as outlined in Section 3.T Second, we explore the two-step inference process, detailed in Section 3.2 Finally, we discuss the aggregation of results and the technique for

---

Step prodict the count of mentions
Step 2: predict the o² mention and deduplicate
eattyy Syperi Spwe//ePPR WPPP Weweweeremm
Sinaationn I>
1905 Wirld Capricos
Prod: 0.99
Cuntiro
sonit tsuse with
Cup whhne he
announeed
the 1995 World
retireenr
runty oyen Weeeeeeeeeeeeeeeeeee
PaleLLLM NER
Sincationn 2
PateeL LM
NER
Ialysaaee
Proh 6.6
06
dup
being deopped
from the Inil
eutty Pepp://w./CCiieeeeeee
Amention 1>
Itllseoss
Phobi 0.99
Suat faced
Zi
England
pool sages
Simcation
Englandeose
rnttly wwwpp//Oeeeeieeeeeeeeee
Figure 2: PaDeLLM-NER inference paradigm: texts enclosed frames with identical colors indicate sequences of the same label. Specifically, the texts within solid-lined frames represent the added templates, while those within dashed-lined frames denote the prediction. In Step I, the model predicts the number of mentions for all labels while in Step 2, it predicts the mentions. By aggregating mentions and labels from all sequences, the final NER results are obtained. Duplicate mentions appearing in different labels are resolved using prediction probabilities.
eliminating duplicate mentions across labels, which is elaborated in Section 3.3 An illustration of PaDeLLM-NER is shown in Figure I and Figure 2
3.1 Reframing of Instruction Tuning
an example, we use a case from the CONIJ.2003 dataset including four labels: person (PER), miscellaneous (MISC), location (L.OC), and organization (ORG). The specifics of the input text and the corresponding ground truth are provided in the second row of Table 0
Illustration of the reframing presented in Figure 1
During reformulation, single unstructured sequences. Each new sequence's output includes the count of mentions for specified label (denoted as "entity type"), followed by the nh mention of that label (denoted as "<mention nz"). Note that the count of mentions and their respective indices are represented using corresponding digit tokens from the LLM'' vocabulary. Specifically, there are no mentions, the model is trained to immediately predict the "<eas>" token, bypassing the need to predict mentions.
containing all label-mention pairs is split into several
Therefore, in this example, one original training data is transformed into five new training data entries. These include two for predicting "LOC" (with 2 mentions), one for predicting "MISC" (with I mention), one for predicting "PER" (with 1 mention), and one for predicting "ORG" (with mentions, directly predicting "<eos>") "). Moreover, the number of mentions for each label and the text corresponding each mention index can be easily obtained from the original ground truth, meaning that the number of new examples depends on the ground truth of that particular example.
With the newly reformulated training examples, then apply the standard instruction tuning procedure. The model takes a sequence of text (,, .2,...,44 lr consisting of input unstructured text and output structured label- mention pair. The optimization objective is cross-entropy loss L which can be defined as follows:
Σ log P (4: 41 b2, (2,....4-11
where P (t: 12,,,....... 1 represents the probability of 4th token t; given the sequence of pre- ceding tokens (,, 12,... 4-1. 1.-1.am as predicted by the model. Note that loss calculation begins from the number of mention tokens (i.e., texts enclosed by dashed-line frames). Theoretically, loss from text spans such as "<mention n> could be ignored during this calculation, since they simply prompt the mention's 's order, which does not necessarily need to be generated by the model. However, our

---

ablation studies show that ignoring these texts has negligible impact model performance, point further discussed in Appendix [ Therefore, we adhere to the standard instruction tuning procedure.
This reformulation allows the model to focus one label-mention pair at a time, shortening the generated length per sequence. More details are shown in Appendix C
3.2 Inference of Label-Mention Pairs
Given a trained LLM, we propose a two-step inference approach: firstly, to predict the number of mentions for specific label based the prompt; and secondly, given the label and provided index to precisely identify the corresponding mention.
Figure 2 shows the overview of PaDeLLM-NER inference. In Step 1, the model predicts the total number of mentions for each label in the input, based on the label prompt. A separate token "vr" signals the completion of this count prediction. If no mentions of the given label exist, the model generates an "<eos>" token, skipping Step 2 for that label. In Step 2, following adding the predicted mention count to the input, mention indexes templates are appended. Formally, if the predicted number of mention m, then "<mention n> indicating the n " mention of the specified label, is appended for each within the set {1, 2, 3, ........ m) and is an integer. Subsequently, the corresponding mention is generated by the model conditioned on preceding tokens. Note that the decoding of all label-mention pairs occurs in parallel, allowing for their simultaneous generation. Additionally, to justify the efficacy of the proposed two-step inference approach, we also implement one-step parallel decoding method. In this approach, multiple mentions of the same label are predicted in a single sequence and compared to the two-step method in preliminary experiment. Further details are provided in the Appendix A
In practice, if there are sufficient GPU resources, the inference for the number of mentions for each label, as well as the subsequent inference for the mention text spans, can be allocating on separate GPUs. If GPU resources are limited, the inference can also be deployed on a single GPU using batch inference, facilitating parallel decoding. Using Figure 2 as example, in Step 1, the batch size four, as there are four labels the dataset. Step 2, the batch size is five, reflecting the five label-mention pairs determined in Step (i.e., in "PER", 2 in "MISC", 2 in "LOC"). This parallel decoding strategy is effective in reducing inference latency, especially in scenarios where inputs are received in a streaming manner.
3.3 Removal of Duplicate Mentions
Unlike autoregressive decoding, where subsequent label-mention pairs can attend preceding ones, PaDeLLM-NER generates each label-mention pair independently. This inference strategy means that the model might generate mentions erroneously repeated in multiple labels. As exemplified in Figure the model correctly predicts the first mention of "LOC" as "Ituly", but it also incorrectly predicts the second mention of "MISC" as "Italy".
To address the issue of duplicate mentions, we suggest employing prediction probability to remove repeated mentions. Specifically, we calculate the prediction probability for each instance of the mention. This is done using the formula: P = II P(t [t, t2, f4-1) where b represents the starting token index of the mention text, and e denotes the ending token index. Then, for mention that appears in multiple labels, the mention instance with the highest probability will be preserved. As illustrated in Figure2] "Italy" is categorized as "MISC" with only 0.61 probability, which is lower than that for "LOC", resulting in its removal. In practice, the probability of each token can be calculated concurrently with token generation. Consequently, this method enables an efficient and accurate ddentification of duplicate mentions without incurring additional costs. The effectiveness of this de-duplication approach is further explored in Appendix A
Experiments
In this section, we showcase the effectiveness of P-DELL.M-NER terms of prediction quality and inference acceleration through experiments.
5

---

4.1
Setup
Datasets
The datasets used in our experiments include:
Zero-shot Datasets: To align with the methodology proposed by 44 train PaDeLLM using the Pile-NER dataset [45). This dataset comprises around 240,000 entities categorized into 13,000 distinct types, derived from the Pile Corpus [46]. The passages in Pile-NER
enhanced through processing with ChatGPT, which facilitates the transparent generation of inherent entities. For assessing the model's zero-shot capabilities on previously unseen entity categories. following (30 45
we select two established benchmarks: CrossNER 47
and MIT 48).
Supervised Datasets: we evaluate our method on supervised English and Chinese NER datasets. Following [30
49
50) English datasets include the general domain flat NER
CoNLL2003 [51), the nested NER ACE2005 (52), and the biomedical nested NER GE-
NIA [53]. Following [6 154 55 Chinese datasets include four commonly used general
domain flat NER benchmarks Resume (56), Weibo [57]. MSRA [58] and Ontonotes 4.0 [59
and two vertical industrial domain flat NER datasets YouKu [60] and Ecommerce [61]. The
statistics of all datasets are shown in Appendix B
Training setup We employ pre-trained version of Llama2-7b [ and Baichuan2-77 as
base models for English and Chinese study respectively. Additional implementation details are in Appendix [
Inference setup For all generative models, we use greedy search with beam size of a maximum of 512 new tokens, and a temperature of 1.0. As described in Section 3.2 for PaDeLLM-NER, we adopt two inference settings: (1) each example is inferred on multiple GPUs to implement parallel decoding (i.e., each sequence is assigned on one GPU), termed as PaDeLLMMMuti and (2) each example inferred on a single GPU, employing batch decoding for parallel decoding, termed as PaDeLLMBachh Note that for PaDeLLMMatii we sequentially predict each sequence of one example to simulate parallel decoding on multiple GPUs.
Baselines
The baseline used in our experiments include:
Inference Latency Baselines: As the primary focus of this work is on reducing inference latency in NER tasks using LLMs, we compare our method, PaDeLLM-NER, with traditional autoregressive approaches. As mentioned in Section T the main points of comparison are autoregressive structured output formats used in 3 4 and [37 referred to respectively as AutoRegaug and AutoReg as these are the approaches very close to our system. We reimplemented all these methods for both English and Chinese datasets, utilizing the same pre-trained L.LMs as PaDELLM-NER.
Zero-shot Baselines: LLMS are known for their generalizability, therefore, following Ding et al. [44], we also evaluate the zero-shot performance of PaDeLLM. Several most recent SOTA LLM- approaches are selected as strong basclines as their great generalizability in zero-shot NER scenarios including GoLLIE-7B (30], UniNER-7B 45), GLiNER-L 62
GNER-LLaMM--7
44
Supervised Baselines:
We compare our approach with other recent SOTA supervised
approaches, including BINDER (50), Gollie (30), and DeepStruct [49] for English bench-
marks, as well as W2NER (63, NEZHA-BC (54), and SSCNN (55) for Chinese bench- marks, show PaDeLLM-NER's efficacy in prediction quality.
More details on the re-implementationn and model size of each method are provided in Appendix
Our evaluation encompasses two dimensions: prediction quality and acceleration of NER inference. For assessing prediction quality, in line with Lu et al. [5), Wang et al. 77. we employ the micro F-score.
Evaluation
Base

---

English Dataset
Chinese Dataset
AutoReg
CoNLL303
ACE05
Weibo
Onto4
Resume
Youku
Eeom
Avg.
AutoRegaug
992.70
944.90
1,515.35
1,276.32
812.78
1009.68
982.39
579.99
845.42
995.50
AutoRogaius
753.36
1,293,87
1,266.31
1,630.62
609.34
783.28
1,462.56
598.59
738.20
1,015.12
Ours
M
229.74
255.53
316.90
159.57
143.47
171.67
238.27
2033663
293.40
223.57
MaDeLLMnach
333.89
498.50
616.01
344.75
288.43
459.20
241.25
419.40
378.40
Table Comparison of inference latency (in milliseconds) between PaDeLLM-NER and baseline methods. Underscored font is the second-best method, while bold font is the best method, also applied to subsequent tables.
Welbo
8.00c 00x
Welbo
10.22X
MSR
3.98x
Weibo
4.7 4.73x
Ontod
5,88x
Resume
F. F.14x
Welbo
3,70x
Resume
3.19x
MSRA
5.67x
ACE05
.066x
Oriof
3. 3.50x
MSRA
2.98x
GENIA
4,78k
Onto4
4.56x
CoNiL
2.97x
Onlo4
2.72x
CoNLLO3
4.22X
MSRA
4 4.25x
GENIA
2.46x
ACE0S
2.60x
Resume
4.12X
GENIA
100x
Youku
2 2.40x
Youku
2 2.48x
ACE0S
70x
CoNL103
28x
Resume
2.14x
CoNLLO3
2.26x
Ecom 88x
Youku
2 2.94X
Ecom
7,02x
GENIA
2. 2.06x
Youku
2,85x
Ecom
52x
ACEOS
1.90x
Ecom
1.76x
10
2
0.0
2.5
5.0
(a) ARaug vs. PDLMMai (b) ARsimac vs. PDLMM
ARAug vs. PDLMRach (d) ARsimat vs. PDLMaach
Figure Speedup of PaDeLLM-NER compared to Autoregressive methods.
Following Ning et al. [20]. we evaluate inference speed using latency (in milliseconds). We record the latency with the code: start time.time(). model.generatie latency time.time() start. In PaDeLLM-NER, we add the latency of mention counting and label-mention pair generation as the latency of each sequence. The final latency for the example determined by the highest latency across sequences, as the user can only obtain the result of an example when the slowest sequence generated. We conduct experiments three times and use the average result alleviate the effect of randomness. We also report the average sequence length (tokenized) to clearly demonstrate the extent of sequence length reduction Appendix E Evaluations all models were performed on the same NVIDIA A A10 GPU.
4.2
Main Results
Model
AT
Literature
Music
Politics
Science
Movie
Restaurant
Avg.
SOTA
GoLLIE-7B
59.1
62.7
67.8
57.2
67.0
63.0
43.4
60.02
UniNER-7B
53.5
59,7
65.0
60.8
61.1
42.4
31.7
53.45
GLiNER-L
57.2
64.4
69.6
72.6
$7.2
42.9
60.92
GNER-LLaMA-7B
63.1
68.2
75.7
69.4
69.9
68.6
47.5
66.05
Ours
PaDeLLM-NER-7B
60.7
66.1
67.6
68.1
64.4
61.3
43.6
61.68
Table Comparison of prediction quality with recent SOTA models zero-shot setting.
Evaluation inference latency We investigate how PaDeLLM-NER reduces the end-to-end latency compared to baseline methods. Table pressents the average latency for each method across all datasets. First, it's clear that both PaDeLLMMunt and PaDeLLMpuuk significantly reduce inference latency when compared to baseline methods, as highlighted by the substantial reduction in mean latency. For example, the mean latency reduction achieved between PaDeLLMMuti and AutoRegsuue stands at an impressive 791.55 ms, underscoring the significant improvement.
To more intuitively quantify the latency reduction of PaDeI.LM-NER we break down its speedup across different datasets in comparison baseline methods Figure The speedup is computed by dividing the lateney of basclines by the latency of PaDeLLM-NER, We can observe that PaDeLLM-

---

SOTA
CoNLL03
ACE05
GENIA
Avg.
BINDER 50
93.33
89,50
80.50
87.77
Gollie 50
93.10
89.60
StepStruct
93.00
86.90
80,80
86.90
AutoReg
AutoRegAug
93.08
83.04
70.16
82.09
AgtoRegsiuat
91.87
82.99
77.90
84.25
Ours
PaDeLLM-NER 92.52
85.02
77.66
85.07
Table 4: Comparison of prediction quality with recent SOTA methods on English supervised datasets.
NER consistently show a speedup over baseline methods across all datasets. The highest speedup observed in the Weibo dataset when comparing AutoRegsmer vs. PaDeLLMMunt: with a speedup of 10.22x. When we narrow our focus to the comparison between PaDeLLMRaach and the baseline methods, considering these methods utilize a single GPU for inference, we can still observe substantial speedup ranging from 1.76x to 4.73x. The speedup factor varies across different datasets, suggesting that the efficiency gains of PaDeLLM-NO Interestingly,
may be influenced by the characteristics cach dataset. observe that the PaDeLLMRaac is slower than PaDeLLMMund (378.40 40 ms vs. 223.57 ms), more analysis about this shown in Section []
Overall, the T Figure 3 suggest that PaDeLLM-NERR significantly reduces latency compared to autoregressive methods, though the extent of this reduction varies by dataset and the specific baseline method it's compared to.
Evaluation zero-shot prediction quality Table compares the prediction quality of different models across various domains like AI, Literature, Music, Politics, Science, Movie, and Restaurant in zero-shot setting. Among all these methods, GoLLIE-7B scores range from 43.4 in Restaurants to 67.8 in Music, with an average of 60.02. UniNER-7B has lower scores, particularly in Restaurants (31.7), and averages 53.45. GLiNER-L shows fairly balanced performance with high of 72.6 in Polities and an average of 60.92. GNER-LLaMA-1E excels in Music with a 75.7 score and has the highest average of all 66.05. Our model, PaDeI.LMMNER. which consistently performs well across all domains. It has the second-best average score of 61.68, following GNER-LLaMA- not the top performer, offers robust and balanced prediction
7B. This highlights that while capabilities across a diverse set of topics in zero-shot setting. Note that the training of PaDeLLM-NER does not incorporate the additional task scheme prompt for describing unseen entities used in GNER-LLaMA-78 [4], which may account for the observed differences in performance.
Evaluation on supervised prediction quality Table and PaDeLLM-NER in comparison to other SOTA methods on supervised datasets. Notably, the miero F-scores for both PaDeLLMMons and PaDeLLMsach based methods surpass LLMe-aadd approaches, such as AutoReg and PaDeLLM-NER, within the supervised context. Nonetheless, the strength of LLM-based methods lies not in their performance under task-specific supervised settings, but rather in their superior zero-shot capabilities, which compensates for their relative shortcomings in supervised scenarios. Nevertheless, PaDeI.LM-NER demonstrates SOTA performance on certain task-specific datasets, exemplified by its exceptional results on the Youku dataset.
and Table present the micro F-scores of
identical. Initially, it is evident that encoder-
Upon comparing PaDeLLM-NER with AutoReg, both which are LLM-based methods, it becomes evident that PaDeLLM-NER outperforms AutoReg across both English and Chinese supervised datasets, as evidenced by its superior mean F-score. This outcome indicates that PaDeLLM-NER not only achieves lower inference latency but also maintains higher level of prediction quality when contrasted with baseline methods.
In summary, the results presented in Table only maintains superior prediction quality both zero-shot and supervised environments but also significantly reduces inference latency.
and 5 demonstrate that our approach not
8

---

SOTA
Weibo
MSRA
Onto4
Resume
Youku
Ecom
Avg.
NEZHA-BC [54]
82.98 81.80
SSCNN 55
71.81
82.99
96.40 96.65
86.
WER
63
72.32
96.10
83.08
AutoReg
AutoReg
59.04
95.56
79.20
95,80
86.07
76.02
81.94
egsraee
56.07
90.92
80.97
95.74
86.85
81.57
82.02
Ours
M-DEI.LM-NER
67.36
95.03
80.81
94.98
87.91
81.85
84.66
Table 5: Comparison of prediction quality with recent SOTA methods on English supervised datasets. "** indicates that results are not directly comparable.
Speedup Analysis
One concern noted is that batch inference does not speed up much as inference distributed across multiple GPUs. This observation is consistent with our expectations and supported by Chen et al. [9 who found that batch inference in LLMs tends to be slower than single sequence inference under identical conditions, likely due to limitations in GPU memory bandwidth (4
Transitioning from these performance considerations, it's noteworthy that PaDeLLM-NER is self- contained and can be seamlessly integrated with various generative architectures, including well- established decoder-only models [8-13] and recent innovations like RWKV [65], well multi- modal LLMs 666, 67] for tasks like Key Information Extraction tasks 68 all without needing architectural changes additional data/modules. Also, it could be incorporated with off-the-shelf LLMs such as ChatGPT (27] and Claude thhoough prompt engineering without the need for further training, aspect we plan to explore in future research.
6 Data Contamination Concerns
Since we are using LLMs as our foundational models, trained on extensive datasets from various online sources
there is a chance that the models may have encountered parts of our evaluation sets during their pre-training phase, albeit unintentionally This could potentially affect our experimental results. However, the primary focus of our experiments is the comparison of our proposed method with baseline methods. Given that these methods employ the same LLM as the base model, data contamination is unlikely significantly impact the results.
Limitations
One clear disadvantage of LMDELL is the multiplication of training examples from one to m * n, where m is the label count and n the mention count. Despite this, given that low latency is a major bottleneck in LLMs, trading longer training for lower latency is justifiable. Also, given the impressive generalization ability of LLMs, we believe that this method can be smoothly adapted to few-shot scenarios requiring less computation resources, which will be explored in future work.
Additionally, accurately counting the number of mentions remains challenge for LLMs as discussed in Appendix(F [F] This issue could be alleviated by implementing a specialized counting model dedi- cated this task (69] Another drawback that reformulating lahel-mention pairs loses location information, which hinders tasks like downstream editing. We will address this in future work. Additionally, the de-duplication mechanism is overly aggressive, potentially removing mentions that appear under different labels-a common issue in real-world applications (see Appendix A for more details).
Finally, there are several instances of re-computation within the pipeline that can be optimized. Specif- ically, input texts are encoded multiple times throughout the process. During batch decoding, certain
https://wwwi httpps//www.anttrrpc....o.oo///a httpaaroppc..com///////ll/auuule

---

sequences may encounter the "<eos>" token earlier, but due to the nature of batch inference, these sequences continue to predict. We plan improve this the future by implementing enhancements like KV cache reuse and batch inference with an early quit mechanism, among other strategies.
8 Conclusion
In this study, we present PaDeLLM-NER, a parallel decoding framework for NER within LLMs. This approach enables batch parallel decoding of all inference time by 1.76 to 10.22 times without sacrificing prediction accuracy.
pairs, significantly cutting down
References
[1] Ruotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. Simplify the usage of lexicon in Chinese NER. In Proceddings of the ACI. 2020, pages 5951-5960, Online, July 2020. doi: 10.18653/112222...o aat.acclmma........ URL
wwwp:nttthooooooooo/.o.o 202
acl mainn.n.... 528
[2] Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. Lexicon enhanced Chinese sequence labeling using BERT adapter. In Proceedings of the ACL. 2021 and the I 1Ith International Joint Conference Natural Language Processing (Volume I: Long Papers). pages 5847-5858. Online, August 2021. doi: 10.18653/v1/20211 acl-lon.--- URL https: http:///claathooogoo
brg/2021.ac1-1eer 454
[3] Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma. Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. Structured prediction as translation between augmented natural languages. In International Conference on Learning Representations, 2020.
[4] Sarkar Snigdha Sarathi Das, Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 6998-7010, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v 1/2023 www.:22.emlppman...... 433. URL https://
aclanthology. org/2023. manlp-main. 433
[5] Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 5755-5772, 2022.
[6] Jinghui Lu, Rui Zhao, Brian Mac Namee, and Fei Tan. Punifiedner: A prompting-based unified ner system for diverse datasets. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1)):13337771333 Jun. 2023. doi: 10. (tt09//aa...37/11..... //ojs. aaai org/index..i.ooooooooogorgregree index. pttp:/pppp//paa Ptlp//AAI///
https
[7] Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjic Ye, Qi Zhang, Tao Gui, et al. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint urXiv:2304.080S5 2023.
[8] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 66,, 2020. URL http://jmlr. http:///mllrrrr//eee/mm
htmm
[9] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, al. Crosslin- gual generalization through multitask finetuning. arXiv preprint 1.Xiv:22/1.017880 2022.
[10] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
10

---

[11] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urXiv:2307.0922:. 2023.
[12]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[13] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang. Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint 230.v:2309.10303 2023.
[14] Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, et al. A bounding box is worth one token: Interleaving layout and text in a large language model for document understanding. arXiv preprint arXiv:2407.01976. 2024.
[15] Jinghui Lu, Dongsheng Zhu, Weidong Han, Rui Zhao, Brian Mac Namee, and Fei Tan. What makes pre-trained language models better zero-shot learners? In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2288-2303, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1025333/112222acl---ooo 128. URL https: atttps:/aacaanthologg hww./aaattoooooooooooooooooo
ac1- long. 128
[16] Jinghui Lu, Linyi Yang, Brian Namee, and Yue Zhang. rationale-centric framework for cuman-in-thh--lle machine learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 6986-6996, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.186653//20022.ccllle 481. URL https://aclantholl
org/ 2012 aow.lll......4.. 4881
[17] Hao Feng, Zijian Wang. Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308. 112000..1199 11522, 2023.
[18) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428. 2023.
[19] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving, arXiv preprint arXiv:2310 20..:22300.88. 18547, 2023.
[20] Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. Skeleton-of Large language models can do parallel decoding. arXiv preprint arXiv:2307. 15337, 2023.
[21] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, Advancc in Neural Information Processing Systems, 2022. URL http:///ppparee..ee.eetftrrrrrrp
[22] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint 230. 2023.
[23] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286.
PMLR, 2023.
[24] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[25] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.

---

[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 21()):5485-551) 2020.
[27] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2333::23003.8777. 2023.
[28] Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. Large language models for generative information extraction: A survey, arXiv preprint arXiv:2312.17611 2023.
[29] Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. Empirical study of zero-shot ner with chatgpt. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages
2023.
[30] Oscar Sainz, Iker Garcia-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. Gollie: Annotation guidelines improve zero-shot information-extractin arXiv preprint 2rXiv:2310.03668, 2023.
[31] Dhananjay Ashok and Zachary C Lipton. Promptner: Prompting for named entity recognition. arXiv preprint 2305:2200..5444 2023.
[32] Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. Learning in-context learning for named entity recognition. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13661-13611 Toronto, Canada, July 2023, Association for Computational Linguistics. doi: (0.18653/1/2023. acI- acl-long.764. URL / http://lolaaantooogy.ooooo .rg. 2023 acl-long.......... 764
[33] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learming Representations, 2022.
[34] Dan Friedman, Alexander Wettig, and Danqi Chen. Learning transformer programs. arXiv preprint arXiv:2306..00 01 /28, 2023.
[35] Hanrui Wang. Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High- PigfoPeerfommnce Computer Architecture (HPCA), pages 97-110. IEEE, 2021.
[36] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers, In The Eleventh International Conference Learning Representations, 2023. URL httttt::///oeereiiieeee nwt weet//orummdd-tcbBBPeeee td=tcbBPnfxxS
[37] Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. Accelerating transformer inference for transla- tion via parallel decoding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12336-12355, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023. acl-lo URL https://aclanthol/
org/ 2023/222...0 /tl/:/2221111....///...eeecllcaa long. 689
[38] Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[39] Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2019.
[40] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing 3 35:17456 2022.
12

---

[41] Tian Lan, Deng Cai. Yan Wang. Heyan Huang, and Xian-Ling Mao. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023. URL https /ooenteevee..ee//oonree wwtp:/oooummmddd00110000000
[42] Nan Yang, Tao Ge, Liang Wang. Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023.
[43] Xinpeng Zhang, Ming Tan, Jingfan Zhang, and Wei Zhu. Nag-ner: unified non-autoregressive generation framework for various ner tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 676-686, 2023.
[44] Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, Bowen Yan, and Min Zhang. Rethink- ing negative instances for generative named entity recognition. arXiv preprint arXiv:2402.1 16602. 2024.
[45] Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition. ICLR 2024, 2023.
[46] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2 1e:::2211000002. 2020.
[47] Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligncee volume 35, pages 13452-13460, 2021.
[48] Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. Asgard: A portable architecture for multilingual dialogue systems. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8386 8390. IEEE, 2013.
[49] Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong. Jie Tang, and Dawn Song. DeepStruct: Pretraining of language models for structure prediction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 803-823, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.18653/v1/2022 h/220222222indngg-aa...77 URL URL wttphhttp//// tetttps:/acclatthont http:///aaaaathogg....ooo org. org/2022 findings-acl.67
[50] Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. Optimizing bi-encoder for named entity recognition via contrastive learning. In The Eleventh International Conference on Learning Representations, 2022.
[51] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the con1l-2003 shared task. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 Jan 2003. doi: 10:1115/119176..119999
htt:ttp:: http::://xxxx http::/xxxdddoiddoo/oooom www..ooo.orgg.... 1ttp://///./..111111111111 3115/1119176
1119195
[52] Andy Kirkpatrick. Researching english as lingua franca in asia: The asian corpus of english (ace) project. Asian Englishes, 13(1):4-14 2010.
[53] Tomoko Ohta, Yuka Tateisi, Jin-Dong Kim, Hideki Mima, and Junichi Tsujii The genia corpus: An annotated research abstract corpus in molecular biology domain. In Proceedings of the human language technology conference, pages 73-77. Citeseer, 2002.
[54] Xin Zhang, Yong Jiang, Xiaobin Wang, Xuming Hu, Yueheng Sun, Pengjun Xie, and Meishan Zhang. Domain-specific ner via retrieving correlated samples. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2398-2404, 2022.
[55] Miao Zhang and Ling Lu. A local information perception enhancement based method for chinese ner. Applied Sciences, 13(17):9948, 2023.
13

---

[56] Yue Zhang and Jie Yang. Chinese NER using lattice LSTM. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5554-1564, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:
10.18653/v1/P18-- 1144.
[57] Nanyun Peng and Mark Dredze. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Jan 2015. doi: 10.1865311 httpl////11111111 1064. URL http:/ pett::/dx... doi org/ ////// 18653/v1/015-1066
[58] Gina-Anne Levow. The third international Chinese language processing bakeoff: Word seg- mentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 107-117, Sydney, Australia, July 2006. Association for Computational Linguistics. URLphttps http://ccaaathhoooo www.ooogyoogg//o-111.11
[59] Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng. Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL org/N13-3355
https://aclanthoot
[60] Zhamming Jic, Pengjun Xie, Wei Lu. Ruixue Ding, and Linlin Li. Better modeling of incomplete annotations for named entity recognition. In Proceedings of NAACL, 2019.
[61] Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. A neural multi- digraph model for chinese with gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1462-1467, 2019.
[62] Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. Gliner: Generalist model for named entity recognition using bidirectional transformer. arXiv preprint (rXiv:2311.08522 2023.
[63] Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. Unified named entity recognition as word-word relation classification. Proceedings of the AAAI Conference on Artificial Intelligence, 10661-0):066-10000 v36i10.21344. URL https:://///// wwwpo/s..aaai..oo.. wwg.aaaa...aaaaaaaaaaa.aa.aaaaae index. attpaii.rgg..eee..pp/p///aaaaaaa htt///AAAAArrriieevvvie/eeee//e/ 21344
Jun. 2022. doi: 1009/aia...
[64] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10771 2024.
[65] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305. 2023.
[66] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[67] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.
[68] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar, lcdar2019 competition on seanned receipt ocr and information extraction. In 2019 International Conference Document Analysis and Recognition (ICDAR), pages 1516-1520. IEEE, 2019.
[69] Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned laama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305. 14200, 2023.
[70] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
14

---

[71] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL pectps://openrevieee net/forum? set/forrum1ddSkq88855 1d-Skg8
[72] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11008-110000 1104, 2022.
[73] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pages 2717-2739, Toronto, Canada, July 2023, Association for Computational Linguisties. doi: 10.18653/11/023..ck acl--lng..11... URL https: http://llaaaahooooooo org/ 2023 acl- moll-long.o. 153
[74] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. https:::::// htottps://openreviert wwwpiiiwwww.eeeeeee www./efouuunneee http://rrunnnnKKKooo.o.. Kd-KBBOeeX2hhe
15

---

A
Ablation study
Variant
CoNLL03
ACE05
GENIA
Mean
P.DeILMM-NER
92.52
85.02
77.66
85.06
Loss ignoring
92.01
85.18
73.47
83.55
De-duplication
92.44
84.80
77.54
84.92
De-duuliictiooneevne
92.38
84.44
77.38
84.73
Table 6: Ablations ignoring loss and de-duplication.
In this section, we set out to investigate the effects of the different aspects of PaDeLLM-NER.
Ignoring text spans in loss As discussed in Section 3.1 during training, it is permissible to overlook the loss of text span "<mention n>" ", as the model does not need to generate this specific text, which is appended during inference. However, as shown in Table illustrate, omitting these texts has minimal impact on prediction quality.
One possible explanation is that during training, the more significant challenge for LLMs lies in predicting the appropriate mention texts, rather than their format. As the model can readily learns correctly position the format "<mention n> ", this aspect contributes minimally to the loss computation in training. In this case, computing the loss for all text almost equivalent to "neglecting" the computation of loss for "<mention n>".
De-duplication To demonstrate the effectiveness of the de-duplication technique, we established two configurations as detailed in Table The -De-duplication denotes the pipeline operating without the de-duplication technique; +De-duplicationnowe indicates the pipeline that removes mentions with the highest probability, opposite to the original de-duplication technique.
Theoretically, PaDeLLM-NER should be the pop-perfomming method, as its de-duplication climinates noisy mentions, enhancing precision. Following closely is the -heduuliication, allows duplicate mentions to persist. + +Deeduplicationneeens ranks lowest since it removes correct mentions and retains incorrect ones, lowering recall and precision simultaneously. As shown in Table the results consistently align with our expectations, thereby verifying the effectiveness of the de-duplication process. Moreover, the difference among these variants is subtle, which can be attributed the rare cases where duplicate mentions exist. This further highlights the robustness of proposed method.
We also report statistics in Table 7 and g showing that mentions under multiple labels are rare for both ground truth and PaDeLLM predictions. However, we recognize that the de-duplication mechanism can be overly aggressive, potentially removing mentions that appear under multiple labels-a common scenario in real-world applications. In such cases, opting not to use the de-duplication mechanism may be preferable.
Dataset
Count
Ratio
ACE05
0.00034
ConLLO3
0.00017
GENIA
0
Ecom
O
0
MSRA
0.00013
Weibo
0
Youku
0.0012
Resume
0
Table 7: Mentions appear under multiple labels ground truth.
Preliminary experiments for justifying the importance of two-step prediction We conducted preliminary experiment using one-step prediction, where all mentions of the same label are predicted single sequence, which referred to as OneStep this paper. An example of OneStep parallel decoding is shown in Table 9 Note that the order of mentions is preserved as in the ground truth.
16

---

Dataset
Count
Ratio
ACE05
22
0.0074
ConLLO3
10
0.0017
GENIA
0.0034
Ecom
0.0012
MSRA
0.00089
Weibo
0
0
Youku
0.0019
Resume
0
Table 8: Mentions appear under multiple labels PaDeLLM prediction.
following the data from the corresponding dataset. The overall latency for each example determined by the latency of the slowest sequence. The preliminary experiment is conducted on three English dataset, i.c., CoNLLO3, ACE05 and GENIA.
Entity
Text
NER Result
(1C Microscopic microcap Everlast, mainly maker of boxing equipment, has soared over the last several days thanks to licensing deal with Jacques Moret allowing Moret to buy out their women apparel license for million on top 12.5 million payment now
("Microscopic microcap Everlast", "a maker of boxing equipment", "their"]
ORG
<onitt-PPER:tcxtt2200-122 00000t2220011122200533.00..00 million pay-
("Jacques Moret", "Moret", "thcir", "their women")
ment now.
GPE
million pay-
ment now.
LOC
million pay-
ment now.
Table Illustration of one-step parallel decoding NER approach.
Method
ACE05
CoNLLO3
GENIA
Mean
PaDeLLMMauti
255.53
229.74
316.90
267.39
OneStepMult
386.93
272.22
513.63
390.93
AutoRegAug
944.90
992.70
1,515.35
1150.98
AutoReg
1,293.87
753.36
1,266.31
1104.51
Table 10: Comparison inference latency.
Method
ACE05
CoNLLO3
GENIA
Mean
PaDeLLMMuli
85.02
92.52
77.66
85.06
OneStepMuli
80.98
91.36
76.27
82.87
AutoReg
83.04
93.08
70.16
82.09
AutoReg
82.99
91.87
77.90
84.25
Table 11: Comparison of prediction quality.
The results are reported in Table 10 and Table 11 As expected, the inference speed of one-step approach falls between that two-step prediction (i.e., PaDeLLM) and the purely autoregressive model. However, the prediction quality is lower compared to the two-step prediction. In other words, PaDeLLLM outperforms the one-step approach both inference speed and prediction quality, which again verifies the efficacy of PaDeLLM.
17

---

Preliminary experiments for zero-shot autoregressive baseline We did not report zero-shot re- sults for AutoRegang and AutoRegamet as they are unsuitable for this setting. Preliminary experiments show higher latency and lower F-scores compared to PaDeILLM (see Table 12 for details).
A1
Literature
Music
Politics
Science
Avg
Latency (ms)
PaDeLLM
398.37 1529.95
357.45 2096.08
352.85 2545.20
366.76
375.02
370.09 2174.03
Auto Aug
2364.87
2334.05
F-score PaDeLLM
60.7 0.19
66.1
67.6
68.1
64.4
65.38
AutoRegAug
0.15
0.94
0.13
0.21
0.324
Table 12: Comparison of Latency and F-score between PaDelLLM and AutoRegAug under zero-shot scenarios.
B Dataset Statisties
Variant
CoNLLO3
ACE05
GENIA
Mean
PaDeLLM-NER
92.52
85.02
77.66 78.84
85.06
Model scale up to
93.02
84.37
85.45
Table 13: Ablations on model scaling up.
Count Mismatch
Index Inaccuracy
Ground Truth Error
19.3%
23.9%
56.8%
Figure 4: Percentage of different error types.
We evaluate our framework on 3 English and 6 Chinese flat/nested NER datasets. In Table 15 present the detailed statistics. Note that while the statistics of the development set are reported, our training process does not involve the development set.
For the MSRA dataset, we excluded four outlier instances from the test set due to their excessively high number of names, significantly deviating from typical examples. These outliers not only posed challenges for model inference but also risked distorting the evaluation metrics, potentially leading to an inaccurate assessment of the model's performance representative data.
18

---

English Dataset
Chinese Dataset
AutoReg
CoNLLO33
ACE05
GENTA
Weibo
MSRA
Onto4
Resume
Youku
Ecom
Mean
AutoRegAug
33.85
37.10
60.50
45.02
27.42
35.90
30.39
18.21
31.50
35.54
AutoRegsmuc
28.36
49.95
49.03
62.45
18.97
25.53
53.02
18.56
22.51
36.48
Ours
P-DeLLM-NER
6.54
8.29
10.05
2.19
2.23
2.68
4.87
3.66
3.27
4.86
Table 14: Comparison of the number of generated tokens per sequence by PaDeILLM-NER with baseline methods.
Also, we perform label mapping to convert ground truth from special tokens to Chinese words following (6]. Further details are provided in Table
Sentence
Mention
Dataset
#AIl
#Train
#Dev
#Test
#All
#Train
#Dev
#Test
CoNLL2003
20,744
14,041
3,250
3,453
35,089
23,499
5,942
5,648
ACE2005
9,210
7.194
969
1,047
30,634
24,441
3,200
2,993
GENIA
18,546
15,023
1,669
1,854
56,015
46,142
4,367
5,506
Weibo
1,890
1.350
270
270
2,701
1,894
389
418
MSRA"
50,725
44,364
4,361
80,214
74,703
5,511
OntoNotes 4.0
24.371
15,724
4,301
4,346
28,006
13,372
6,950
7,684
Resume
4,759
3,819
463
16,565
13,438
1,497
1,630
Youku 10,002
8,001
1,000
1,001
15,905
12,754
1,581
1,570
Ecommerce
4,987
3,989
500
498
15,216
12,109
1,540
1,567
Table 15: Dataset Statisties. "#" denotes the amount. For MSRA, we remove four outlier examples in
test set.
Dataset
#Entity
Entity
Weibo
Speccffi " (MER.No
Na
"GSeecAMiSpccfffc Geo-Political
Entity)":"
XX
"GPE.NOM CEL.NOMGGenerrie Geo-Political
:ntity)":"*FT
1
K
Loca- "L.Ce.nMMiGeneric Location)". #
tion)":"1t
,ti
f JES.
10 "ORG.NAM(Specfffc Organization) " $R H
"OreeeoM(Generic Organization)" """
MSRA
["L "PER"." Nir. "ORG"."A13""
OntoNotes 4.0
["GPE". "U "LOCC ".."" "PER" " "
Resume
("NAME"." tt. (Ntonaaaiinnltyy"::
"RACE"
A " ftt "EDU"." Di" "ORG":"2 7
"PRO(Profession)" " LLCC(Place of Birth) " "t
Youku
("TELEVISOO":* a Er. "PER(Celebrit)""."" 1
"MISC" " fe")
Ecommerce
("HP(brand)):: He "HC(coomodity""""
Table 16: Entity tag of each dataset and the conversion from tag used in dataset corresponding Chinese natural language. For some tags that are hard to understand, we provide their meaning in brackets. "#" denotes the amount of entity types.
C Reformulation Examples
Two compete reformulated examples are presented in Table 17 for English and Chinese, respectively.
19

---

Language
Input
Output
English
text:
But Fischler agreed review his proposal after the EU 's standing veterinary committee, mational animal health officials, questioned if such action was justified as there was only a slight risk to human health.
<mention 1>Fischler
entity type:
PER
<num>
Chinese
(text)
A BAERRUS $ B 119 - (The communique concluded by stating that Mexican government considers drug trafticking and related activities such as money laundering to be global issuc that threatens national sovereignty and security.)
> (<mention
1>) 2 (Mexican)
(n type):
# (L.OC) <Rt M (<num>>
Table 17: Reformulated examples for English and Chinese dataset, respectively. We provide transla- tions facilitate understanding. The examples come from CoNLL2003 and MSRA dataset.
D Implementation Details
We train our model all datasets for 4 epochs, using batch size of 128 and learning rate of le - 5, with the AdamW optimizer [70] and a cosine scheduler (71). The maximum input and output sequence lengths
to 2048 and 512, respectively. Training is conducted on 8 NVIDIA A100 GPUs. This configuration is applied across all PaDeLLM-NER models, as well as three baseline models: AutoRegaug- AutoRegsuuet as well as Onestep baseline reported in preliminary experiment. We also report the model size of each NER method in Table 18
English Method
Base Language Model
Chinese Method
Base Language Model
BINDER
BERT-base 110M
NEZHA-BC
NEZHA 110M
Gollic
Code-llama 34B
SSCNN W2NER
report
DeepStruct
GLM10B
Transformer- 110M
AutoRegAug
LLaMA-2-77
AutoRegAug
Baichuan2-77
AutoRegStruct
LLaMA-2-7B
AutoRegStruct Baichuan2-7B
PaDeILLM-NER
LLAMA-2-7B
PaDeLLM-NER
Baichuan2-7B
Table 18: Model size of each NER method.
E
Sequence Length Reduction
Results of average sequence length produced by different approaches presented in Table 14 Most notably, PaDeLLM-NER generates much shorter sequences than the other models across datasets. The lengths range from 6.54 CoNLL20023 to 10.05 GENIA for English datasets, and from 2.19 on Weibo to 4.87 on Resume for Chinese datasets. The mean length for PaDeLLM-NER 4.86, which is significantly lower than the means of the other approaches: 35.54 for AutoRegAug and 36.48 for AutoRegsmmet-
In summary, the result shows that PaDeLLM-NER produces much shorter generated sequences compared to the other methods, which is around 13.19% to 13.67% of the original length, respectively, indicating higher efficiency in its inference.
20

---

F Error analysis
PaDeLLM-NER error analysis For our error analysis. we utilize the ACE2005 dataset. We sample and manually examine 50 erroneous examples for analysis. We seek to identify the root causes of errors, which we have categorized into three types: (1) incorrect mention count, referred as Count Mismatch; (2) inaccuracies in the mention corresponding to specific index, termed Index Inaccuracy; and (3) errors in the ground truth data, known as Ground Truth Errors.
The distribution of each error type is illustrated in Figure It is important to note that significant portion of the errors stem from inaccuracies in mention counts (i.e., Count Mismatch, about 56.8%) underscoring the necessity for enbancements in the model's s counting capabilities. Accurate mention counts are pivotal for the quality of predictions. Overestimating the mention count often leads the model to either repeat the last entity or, more problematically, fabricate an entity, thereby escalating the rate of false positives. Conversely, underestimating the mention count results in the model's inability to identify some entities, thus increasing the incidence of false negatives. Following closely the Index Inaccuracy error, indicating that the model sometimes struggles to accurately pinpoint the correct mention for given index, further emphasizing areas for improvement.
Interestingly, our analysis reveal that significant portion of the model's predictions, specifically 19.3%, are actually correct, challenging the accuracy of the ground truth data. This observation suggests the presence of inaccuracies within the ground truth, contributing an elevated rate of false positives. Prior research, as noted in studies by Min et al. [72]. Wang et al. [73]. Zhou et al. [74]. has demonstrated that LLMs predominantly acquire their knowledge during the pre-training phase. These models develop certain "core beliefs" that tend to align more closely with human judgment. In this context, appears that the models possess an inherent capability to rectify errors in the ground truth data, demonstrating their potential to improve data accuracy beyond initial human annotation.
G Model Scaling Up
As we increase the model size to 13B, Table 33 presents mix of results. datasets like CoNLL2003 and GENIA, the model shows significant improvement in predictions. In contrast, the results on ACE2005 are slightly worse. Note that the improvement GENIA is substantial, at approximately 1.18%. Based on these findings, it seems reasonable to suggest that continuously scaling up the model size has the potential to maintain the performance that is at least on par, or even superior, especially in specifie industrial domains. However, this hypothesis warrants further investigation, involving more families of models 1 and a broader range of datasets. We leave this exploration for future work.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have detailed the contributions accurately in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
we have listed some limitations of our work and shown corresponding failure cases in the supplementary material.
Justification: As discussed in Section. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4. and Appendix D we have described the details of implementing and training the proposed model to ensure the reproducibility of our work.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
We have released the model and code, available at URL masked for anonymous review.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https wwwoooeuwwiiiiiooooolloooeee
7 //iips.cc/pulii//uudddd. http:pppl......iddeddddddeeubbii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed the experimental setting and implementation details in Appendix B and Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our deep learning model is designed for a complex task (requiring huge computing resources) where traditional error bars are less informative due to the high variability in model training and initialization. We ensured the robustness of our model by fix the random seed during inference. In addition, comparative analysis with baseline models demonstrated improvements in key performance areas, underscoring the practical effectiveness our approach. We acknowledge the limitation of not using traditional statistical tests and suggest that future work could explore statistical significance in more controlled settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have reported the needed computer resources in Section 4.1 of the supple- mentary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/s//nouiipp..cpppii/////ie
Answer: [Yes]
Justification: respect.
The research in this paper conforms with the NeurlPS Code of Ethics in every
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All the datasets used in this paper are publicly available and they contain no unsafe images.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
For the used datasets and pre-trained models, we have cited their corresponding
works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "vZKjT60W3a", "reviewer": "Reviewer_GKXR", "summary": "This paper proposes an interesting extension of the parallel text generation paradigm, where the authors tackle the NER task and propose to generate the labels independently. For each label prediction, the proposed method first predicts the number of mentions and then predicts the exact entity. The results show that the proposed model performs reasonably well, while achieving faster inference.", "questions": "1. Is the duplication issue, as mentioned in Figure 2, very common? Do you have statistics for this? \n2. Do you know if the testing datasets are in the training data of the LLM?", "limitations": "The authors make earnest efforts to discuss the limitations. In addition, the previously mentioned Weakness 3 is another potential limitation. The authors may include further discussion in this regard.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is a pioneer work to accelerate LLM generation following the parallel generation paradigm. \n2. We do observe significant speed-up empirically, which suggests the proposed method may be of value in real word applications.", "weaknesses": "1. The importance of the two-step prediction for each entity is not justified. I feel there should be a baseline such that the multiple mentions can be predicted together in an autoregressive fashion. For example, I can predict \"entity type: LOC Italy English\" as a whole.\n2. Fundamentally, parallel predictions should be weaker than autoregressive predictions due to the drop in dependency capturing. However, we observe from Table 4 that AR models are noticeably worse than the parallel approach. Since these results contradict common wisdom, there needs more effort to justify them. For example, the authors may need to reveal the full training/testing configurations of both the AR and parallel models, and there could be some more detailed error analysis to show how AR models are making more mistakes than the parallel approach.\n3. The proposed approach may face difficulty when a word is used multiple times with different types. For example, in \"Washington lives in Washington,\" the proposed approach may predict \"LOC\" and \"PER\" for both \"Washington\"; however, it can not align them because the parallel approach is ordered agnostic among entities.    \n4. The proposed method needs finetuning to adjust the LLMs, which can be difficult when it comes to very large LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions. We are grateful for the opportunity to address the concerns raised.\n\n**W1-Justification of the importance of two-step prediction**:\n\n We conducted an additional experiment using one-step prediction, where all mentions of the same label are predicted in a single sequence. As shown in the results below, the inference speed of this approach falls between the two-step prediction and purely AR model, which is as expected. However, the prediction quality is lower than that of the two-step prediction. In other words, two-step prediction outperforms one-step prediction in both inference speed and prediction quality. We will include these new results in the updated version as strong justification for two-step prediction.\n\n|           Latency     | ace05 | conll03 | genia |\n|----------------|-------|---------|-------|\n| PaDeLLM_multi  | **255.53**|  **229.74**  |  **316.90**|\n| OneStep_multi  | 386.93| 272.22  | 513.63|\n\n|        f-score        | ace05 | conll03 | genia |\n|----------------|-------|---------|-------|\n| PaDeLLM_multi  | **85.02** | **92.52**   | **77.66** |\n| OneStep_multi  | 80.98 | 91.36   | 76.27 |\n\n\n**W2-Reason that AR model is worse**: We acknowledge that AR methods, which rely on dependency tokens, offer certain advantages. However, they also encounter challenges in specific scenarios:\n  - Nested NER: For example, in the ACE dataset, the term \"human\" is nested within the \"Human rights group.\" All AR methods struggle to identify this nested structure. In contrast, PaDeLLM, which decomposes from dependencies, effectively addresses this issue.\n\n||||\n|-------|---------------------------------------------------------------------------------------------------------------------------------|---------|\n| **Input** | Human rights group Amnesty International said Friday 's verdict `` represents another step in the further deterioration in the human rights situation in the country . | f-score |\n| **GT** | \"PER\": [\"Human\", \"human\"], \"ORG\": [\"Human rights group\", \"Human rights group Amnesty International\"], \"GPE\": [\"the country\"]  | 1.0     |\n| **PaDeLLM** | \"GPE\": [\"the country\"], \"PER\": [\"human\"], \"ORG\": [\"Human rights group\", \"Human rights group Amnesty International\"]         | 0.88    |\n| **AutoStruct** | ((ORG:Human rights group), (ORG:Human rights group Amnesty International), (GPE:the country), (VEH:null), (FAC:null), (LOC:null), (PER:null), (WEA:null)) | 0.74    |\n| **AutoAug** | [ [ Human rights group /ORG] Amnesty International /ORG] said Friday 's verdict `` represents another step in the further deterioration in the human rights situation in [ the country /GPE] . | 0.74    |\n\n  - Parse Errors in AR Output: Additionally, AR models can encounter format errors as shown below.\n    - Overlong Example (MSRA): ((组织:邓小平同志治丧委员会),(名称:江泽民),...,(名称\n      - This is due to the prediction being cut off mid-output.\n    - Error Format (Resume): ((公司:环三)+...+(名称:null)+(籍贯:null)+(学历:null)+(国籍:null))\n      - Incorrect use of \"+\" causes parsing issues.\n    - Invalid Null Entries: ((组织:亚协会长),...,(组织:庆祝东方新年),(组织:null),(null))\n      - Entries with \"(null)\" lacking an entity type lead to parsing failures.\n  - Training/test configuration: for AR models and PaDeLLM, we use the same training and testing configuration as reported in Appendix D (i.e., line 581-586)\n  - We will release all resources including inference results for the community to replicate and analyze the results. And we will add all this analysis to the updated version.\n\n**W3-same mention under multiple labels**: \n - We acknowledge the method loses token location information, which will be explored in future work. \n - We have identified instances where the same mention is assigned different entity types in the ground truth, as shown in the table. Although such cases are rare, addressing this issue is important for real-world applications. In these situations, we can disable the de-duplication, allowing 'Washington' to be predicted as both 'LOC' and 'PER' simultaneously.\n\n| Dataset | Count | Ratio|\n|---------|-------|------------|\n| ACE05   | 1     | 0.00034    |\n| ConLL03 | 1     | 0.00017    |\n| Genia   | 0     | 0          |\n| ecom    | 0     | 0          |\n| msra    | 1     | 0.00013    |\n| weibo   | 0     | 0          |\n| youku   | 2     | 0.0012     |\n| resume  | 0     | 0          |\n\n\n**W4-need fine-tune**:\n  - First, we tested the zero-shot generalizability of PaDeLLM, as detailed in lines 249-260, demonstrating its efficacy without further SFT to handle out-of-domain entity types, as appreciated by Reviewer XzjF.\n  - Additionally, we can employ resource-friendly fine-tuning techniques such as LoRA, P-tuning, and others to achieve the SFT.\n  - For larger LLMs, it is feasible to use training-free methods like in-context learning to enable the LLM to make the two-step prediction, as highlighted in line 287-289. This approach will be explored in future work. We will add this discussion to the updated version.\n\n**Q1**: Such situation is too rare, the statistics of prediction is shown below:\n\n| Dataset | Count | Ratio|\n|---------|-------|------------|\n| ACE05   | 22 | 0.0074   |\n| ConLL03 | 10     | 0.0017|\n| Genia   | 18     | 0.0034         |\n| ecom    | 2 | 0.0012|\n| msra    | 5     | 0.00089|\n| weibo   | 0     | 0          |\n| youku   | 3     | 0.0019     |\n| resume  | 0     | 0          |\n\n**Q2**: this is discussed in Section 6,  the primary focus of our experiments is the comparison of our proposed method with baseline methods (Auto_Struct and Auto_Aug). Given that these methods employ the same LLM as the base model, data contamination is unlikely to significantly impact the results.\n\nWe kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would be fully appreciated if you would consider to improve the **rating**. We look forward to your response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yhyCeH4zVj", "reviewer": "Reviewer_XzjF", "summary": "This paper presents PaDeLLM-NER, a novel approach for accelerating Named Entity Recognition (NER) inference in Large Language Models (LLMs) through parallel decoding. A reformulation of the NER task that enables parallel generation of label-mention pairs, significantly reducing inference latency. A two-step inference process involving mention count prediction and parallel mention generation.\nExtensive experiments demonstrated significant speedups (1.76x to 10.22x faster) compared to autoregressive approaches while maintaining or improving prediction quality across multiple datasets and two languages.", "questions": "No", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The parallel decoding strategy for NER is innovative and addresses a significant bottleneck in LLM inference speed, which is important in some speed-sensitive applications. The authors conduct extensive experiments across multiple datasets, languages, and settings (zero-shot and supervised), proving the method's effectiveness. The reported speedups are substantial and could have a meaningful, practical impact on NER applications. The method is compatible with existing LLM architectures and can be integrated with other acceleration techniques. The methodology is well-explained with helpful diagrams and examples.", "weaknesses": "Some details and corner cases are not well explained. For example, I didn't see the token location information in Figure 2. If the input has multiple and same mentions (e.g., \"Donald Trump owns the Trump Organization\" ), how does this framework distinguish with the same mentions? (e.g. Trump in the above example)\n\nIn addition, it is not clear how the de-duplicate model processes the partially duplicated mentions. For example, in the above case,  the \"Trump organization\" was recognized as ORG, and what if the person module predicted the \"Trump\" in the \"Trump organization\" as a person? Will the de-duplicate model filter this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions, writing and novelty. We are grateful for the opportunity to address the concerns raised.\n\n**W1-token location information**:\n  - NER traditionally relies on sequence labeling, where each token is assigned a class based on its position. This requires token location information. However, in the seq2seq approach such as PaDeLLM, Autoregressive_struct and autoregressvie_aug, token location information is not mandatory.\n  - For the sentence \"Donald Trump owns the Trump Organization,\" PaDeLLM can differentiate and label the entities as \"PER: Trump\" and \"ORG: Trump\" if the de-duplication mechanism is not employed, but the token location information is losing. We recognize that including token location might improve performance and will consider this in future work, as suggested in [1].\n\n[1] A Unified Generative Framework for Various NER Subtasks (Yan et al., ACL-IJCNLP 2021)\n\n**W2-drawback of de-duplication strategy**: Yes, the de-duplication mechanism will filter this case. In our dataset, instances where two mentions appear under multiple labels are rare, as evidenced by our statistics. This is why we propose using the de-duplication mechanism. However, in real-world applications where a mention might be allowed to appear under multiple labels, we can choose not to use the de-duplication strategy. This decision is a trade-off.\n\n| Dataset | Count | Ratio|\n|---------|-------|------------|\n| ACE05   | 1     | 0.00034    |\n| ConLL03 | 1     | 0.00017    |\n| Genia   | 0     | 0          |\n| ecom    | 0     | 0          |\n| msra    | 1     | 0.00013    |\n| weibo   | 0     | 0          |\n| youku   | 2     | 0.0012     |\n| resume  | 0     | 0          |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKA2XWPwHg", "reviewer": "Reviewer_14Sq", "summary": "They create an NER system where an LLM first outputs the number of mentions there are of a given type (for all possible types). Then all mentions can be generated in parallel.\n\nThis results in faster inference times as each generation is short, and they can be done in parallel.", "questions": "How do you expect this to scale to NER datasets like Ontonotes where a there are 20+ different mention category? Similarly what about long documents that could have 10-30+ mentions of a given type?\n\nDid you see inconsistencies in the model outputs? For example a model that output that there is `1` person but then generated <mention 2> ${person name}?", "limitations": "yes", "rating": 9, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "They compare to several different baseline on multiple NER datasets in multiple different settings.\n\nTheir method is much faster than others.", "weaknesses": "Their reformulation of NER as predicting (label, mention) pairs removes a critical component of classical NER, the actual alignment of the mention to the tokens. Polysemous words are often mentions in some context and not in others and it if often important to know which one was the actual mention, especially if it is used for things like editing downstream.\n\nThe deduplication strategy is very aggressive and removes the possibility that some surface text is a label for multiple types in a single sentence. For example, \"It is England vs. Italy on this sunny day in England\", England is both a place (LOC) and a sports team (ORG) this would get filtered  by their setup.\n\nThe prose's definition of \" prediction quality [...] that is on par\" is rather loose, with their model being 6 points behind on average for zero-shot (table 3) and behind by a point of two on most supervised datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions. We are grateful for the opportunity to address the concerns raised.\n\n**W1-loss aligment of mention to the tokens**: We acknowledge that PaDeLLM loses token position information, and we recognize that incorporating token location (like start and end index) might enhance performance. We will consider this in future work, as suggested in [1].\n\n[1] A Unified Generative Framework for Various NER Subtasks (Yan et al., ACL-IJCNLP 2021)\n\n**W2-dedpulication is aggressive**: Yes, we acknowledge the de-duplicate strategy is aggressive. In our dataset, instances where two mentions appear under multiple labels are rare, as evidenced by our statistics. This is why we propose using the de-duplication mechanism. However, in real-world applications where a mention might be allowed to appear under multiple labels, we can choose not to use the de-duplication mechanism. This decision is a trade-off.\n\n| Dataset | Count | Percentage |\n|---------|-------|------------|\n| ACE05   | 1     | 0.00034    |\n| ConLL03 | 1     | 0.00017    |\n| Genia   | 0     | 0          |\n| ecom    | 0     | 0          |\n| msra    | 1     | 0.00013    |\n| weibo   | 0     | 0          |\n| youku   | 2     | 0.0012     |\n| resume  | 0     | 0          |\n\n**W3-improper wording**: We will tone down the wording in the updated version.\n\n**Q1-large number of entity types or mentions**:\n\n- Theoretically, this method is effective regardless of the number of mention categories because each label-mention pair is handled by a different string. If you have more than 20 different mention categories, you can process them in parallel using over 20 different sequences to predict the number of mentions. Once that is complete, proceed to Step 2 to predict the label-mention pairs.\n- In our in-house NER tasks, where there are often numerous mentions of a given type, we found that the method can manage tens of mentions without significantly losing accuracy. However, in practice, we typically limit the input to 512 tokens to prevent potential issues.\n\n**Q2-generating inconsistency**: During inference,  <mention x> is added to the prompt, which is not generated by the model. For instance, if the model outputs that there is `1` person at Step 1, we include <mention 1> in the prompt. If the model indicates there are  `2` person at Step1, we duplicate the input string and add  <mention 1> and <mention 2> to two strings, respectively. This approach effectively avoids inconsistencies. See Figure 2 caption the dashed box vs the solid box, and line 139-144.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spexdGTOXl", "reviewer": "Reviewer_1fG5", "summary": "This paper introduces a novel approach to reduce generation latency in Named Entity Recognition (NER) using Large Language Models (LLMs). The primary issue addressed is the high latency caused by the sequential decoding process in LLMs, which significantly lengthens the sequence by autoregressively generating all labels and mentions for NER. To tackle this, the authors propose Parallel Decoding in LLM for NER (PaDeLLM-NER), which integrates into existing generative model frameworks without requiring additional modules or architectural changes. PaDeLLM-NER enables simultaneous decoding of all mentions, effectively reducing generation latency. Experimental results show that PaDeLLM-NER can improve the inference speed than the traditional autoregressive approach.", "questions": "Please see the weakness.", "limitations": "The authors provide one limitation section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The parallel decoding strategy is well-designed and experimental results prove the effectiveness.\n- The authors provide comprehensive experiments with different setting and with furthre analysis.\n- The paper is easy to follow.", "weaknesses": "- The proposed method cannot improve the inference speed in scenarios where only one type of entity is predicted.\n- Since the method focuses on the inference efficiency of LLMs-based NER, it is better to report both inference speed and performance compared to zero-shot (Table 3) and supervised (Tables 4 and 5) methods. Notably, Table 3 only reports performance without considering the efficiency of different LLMs. Furthermore, why not report the performance of AutoReg_aug and AutoReg_struct in Table 3?\n- For better understanding of the training resource usage when compared with other methods, it is better to report the base language models used (SOTA methods) in Tables 4 and 5.\n- The writing of this paper could be further improved. For example, Line 219, “As per Ning et al...” appears to be a typo; the meanings of the underline (second performance) and bold (best performance) are not provided; and there is no explanation for why “*” indicates that results are not directly comparable in the Table 5 caption.\n- Comparing with fixed few-shot in-context learning of LLMs may also be worth considering, as caching the fixed prompt could improve the inference speed of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s novelty and writing. We are grateful for the opportunity to address the concerns raised.\n\n**W1-speedup is weak when only one entity type**: Even if the entity type is one, if there are many mentions in that entity type, PaDeLLM can still improve inference speed through Step 2, because we predict all label-mention pair in parallel. As compared in the table,  the string length of PaDeLLM still shorter than AR method (<mention x> is the templated added not generated by the model).\n\n\n|                   | Prediction                                                                                                                                                      |\n|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Autoregressive    | str1: \"LOC: England, India, China, the US\"                                                                                                                      |\n| PaDeLLM           | str1: \"<mention 1>England\"\n|         | str2: \"<mention 2>India\" \n|            |  str3: \"<mention 3>China\"\n|          |  str4: \"<mention 4> the US\"                                         |\n\n\n**W2-reporting other method latency**: \n - Our primary comparison for inference efficiency is between two typical AR methods, AutoReg_aug and AutoReg_struct. Other methods, such as BINDER, are encoder-only models and are not comparable. Methods like GoLLIE and DeepStruct use larger LLMs and different prompt engineering, making direct inference speed comparisons unfair.\n- We did not report the results for AutoReg_aug and AutoReg_struct in Table 3 because these methods are not immediately suitable for the zero-shot setting. Our preliminary experiments show that they have significantly higher latency compared to PaDeLLM, and their F-score is quite limited. We will include these experimental results in the updated version.\n\n| Latency  | AI     | Literature | Music  | Politics | Science | Avg    |\n|----------|--------|------------|--------|----------|---------|--------|\n| PaDeLLM  | 398.37 | 357.45     | 352.85 | 366.76   | 375.02  | 370.09 |\n| Auto_Aug | 1529.95| 2096.08    | 2545.20| 2364.87  | 2334.05 | 2174.03|\n\n| F-score  | AI   | Literature | Music | Politics | Science | Avg   |\n|----------|------|------------|-------|----------|---------|-------|\n| PaDeLLM  | 60.7 | 66.1       | 67.6  | 68.1     | 64.4    | 65.38 |\n| Auto_Aug | 0.19 | 0.15       | 0.94  | 0.13     | 0.21    | 0.324 |\n\n\n**W3-report base model**: We have added a column specifying the base language model used by each method and will update these tables in the next version. Except for GoLLIE and DeepStruct, all other methods are encoder-only, which results in a smaller number of parameters. GoLLIE and DeepStruct use backbones of 34B and 10B parameters, respectively, which are larger than PaDeLLM based on LLaMA-2 with 7B parameters.\n\n| Method        | Base Language Model |\n|---------------|---------------------|\n| BINDER        | BERT-base 110M      |\n| Gollie        | Code-llama 34B      |\n| DeepStruct    | GLM10B              |\n| AutoRegAug    | LLaMA-2-7B          |\n| AutoRegStruct | LLaMA-2-7B          |\n| PaDeLLM-NER   | LLaMA-2-7B          |\n\n| Method        | Base Language Model     |\n|---------------|-------------------------|\n| NEZHA-BC      | NEZHA-base 110M         |\n| SSCNN         | not report              |\n| W2NER         | Transformer-based 110M  |\n| AutoRegAug    | Baichuan2-7B            |\n| AutoRegStruct | Baichuan2-7B            |\n| PaDeLLM-NER   | Baichuan2-7B            |\n\n\n**W4-writing improvement**\n\n  - Thank you for pointing out the typos. We will correct them in the updated version.\n  - The meanings of underline and bold formatting will be clarified in the updated version.\n  - The reason for the asterisk (*) is explained in  the caption of Table 9. We will provide a more explicit explanation in the updated version.\n\n**W5-considering ICL**: This is a good suggestion. We have acknowledged it in lines 287-289, and we will rephrase the wording to make it clearer. We'll also list caching mechanism as a potential area for future exploration (line 310).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a two-step parallel NER prediction scheme, which predicts the number of entities and then parallelises their label prediction.  The method achieves a significant speedup over sequential models and reasonable performance on benchmarks.  Weaknesses outlined by reviewers were all addressed during the rebuttal/discussion period.
Decision: Accept (poster)