Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood
Year: 2023

Abstract:
Recognizing flat, overlapped and discontinuous entities uniformly has been paid increasing attention. Among these works, Seq2Seq formulation prevails for its flexibility and effectiveness. It arranges the output entities into a specific target sequence. However, it introduces bias by assigning all the probability mass to the observed sequence. To alleviate the bias, previous works either augment the data with possible sequences or resort to other formulations. In this paper, we stick to the Seq2Seq formulation and propose a reranking-based approach. It redistributes the likelihood among candidate sequences depending on their performance via a contrastive loss. Extensive experiments show that our simple yet effective method consistently boosts the baseline, and yields competitive or better results compared with the state-of-the-art methods on 8 widelyused datasets for Named Entity Recognition.

heading: 1 Introduction
text: Recently, recognizing flat, overlapped and discontinuous entities in a unified manner has been paid increasing attention. Among the existing works for unified Named Entity Recognition (NER), Seq2Seq formulation prevails for its flexibility and effectiveness in unified modeling (Yan et al., 2021; Lu et al., 2022; Ye et al., 2022). Typically, it arranges the output entities into a fixed order to form a target sequence, and trains the generative model by maximum likelihood estimation (MLE).
However, this estimation introduces bias by assuming a deterministic target distribution, where the model learns to assign all the probability mass to the observed target sequence. The biased estimation hurts the performance during decoding where predicted sequence likelihoods often do not accurately rank the performance of the generated sequences. To alleviate the bias, (Zhang et al., 2022) propose two data augmentation methods that sample possible sequences from the target space.
Others resort to other formulations, e.g., W2NER (Li et al., 2022) reformulates NER as a word-word relation classification. In this study, we stick to the Seq2Seq formulation and explore how to mitigate the bias from another perspective orthogonal to (Zhang et al., 2022).
Beam search decoding algorithms maintain B candidates in descending likelihoods and output the highest one. However, the rest candidates could contain predictions with better performance. We measure this phenomenon with oracle scores. As shown in Table 1, the beam candidates contain predictions with up to 8.1 points higher F1 over the outputted one, averaged on eight datasets. Doubling the beam size further increases the advantage to 9.38 points.
Recently, reranking-based methods proposed for the abstractive summarization task offer a potential technique (Liu and Liu, 2021; Ravaut et al., 2022). They train a discriminator on the candidates to predict a score for picking out the best candidate. For example, SimCLS (Liu and Liu, 2021) regards the cosine similarity between the input and candidate representations as the score. However, when applying reranking-based methods to our task, we find a challenge originating from the nature of information extraction. Candidates of the same input share most of the words and the discriminators trained from scratch have difficulty differentiating them
1137
(detailed in Sec. 3.3). To address the above issue, we propose RerankNER to debias generative NER based on a reranking framework adapted for the task. Specifically, we first train the generative model in the standard way, resulting in a biased model. Then, we generate several candidates for each input with beam search. Instead of training a separated discriminator on the candidates sharing most of the words, we calibrate the generative model with a contrastive loss defined on the candidates. The contrastive loss aims to make the estimated sequence likelihoods consistent with their relative task performance as shown in Figure 1. This objective softens the target distribution and thus alleviates the bias.
Our contributions are summarized as follows: 1. To the best of our knowledge, we are the first
to explore reranking-based methods in the field of generative information extraction (Ye et al., 2022). 2. We propose a method for generative NER tackling the bias problem. 3. Experimental results show that our method consistently boosts the baseline, and yields competitive results compared with the stateof-the-art methods on 8 widely-used datasets for NER.

heading: 2 Method

heading: 2.1 Task Formulation
text: We unify three NER subtasks (i.e. the flat, overlapped, and discontinuous NER) as follows. Given an input sentence of n tokens X = x1x2 . . . xn, the m output entities are arranged into a target sequence Y = E1E2 . . . Em, Ei = y1i y 2 i . . . y j−1 i y j i li, where y 1 i , ..., y j i denotes the tokens in the i-th entity and li denotes the label of the i-th entity. Our goal is to model the conditional probability P (Y |X), which is factorized
auto-regressively into ∏|Y |
t=0 P (yt|X,Y<t).

heading: 2.2 Overview
text: Given a generative NER model trained on the target sequences with the standard MLE, we perform sequence likelihood calibration to alleviate the bias. First, we generate several candidates for each input with beam search and evaluate their task performance (F1 score is used). Then, we continue training the model with the contrastive loss to make the estimated sequence likelihoods consistent with their relative task performance. Finally, we generate the answer with the standard beam search by the calibrated model.

heading: 2.3 Sequence Likelihood Calibration
text: The contrastive loss depicted in Figure 2 is composed of three terms LMLE,LRank,LGold. LMLE is identical to the standard MLE used in the first training stage. It maintains the generating ability of the model during the calibration process.
LMLE maximizes the sequence likelihood of the gold target sequence Y , where the sequence likelihood is calculated as the product of token-level likelihood:
LMLE = −S(Y ) S(Y ) = ∑
t
logPθ(yt|X,Y<t)
and θ denotes model parameters. LRank improves the consistency between the estimated sequence likelihoods and the task performance of the candidate sequences. We adopt the margin ranking loss (Hopkins and May, 2011) for this term, i.e.,
LRank = ∑
i,j
max ( 0, S(Ŷj)− S(Ŷi) + λ )
where Ŷi, Ŷj is a pair of candidates generated by beam search, provided that Ŷi has a higher F1 score than Ŷj . λ denotes the margin, a hyper-parameter.
Apart from the supervision of relative order in the candidates, we utilize the supervision of the gold sequence as well. LGold ensures the sequence likelihoods of the generated candidates do not overstep the likelihood of the gold.
LGold = ∑
i
max ( 0, S(Ŷi)− S(Y ) + λ )
where Ŷi denotes a candidate sequence, provided that it is not an equivalent of the gold.
The contrastive loss is the sum of the terms:
L = LMLE + αLRank + ᾱLGold where α and ᾱ are coefficients.

heading: 3 Experiments

heading: 3.1 Main Results
text: We conduct experiments on eight datasets of three NER subtasks in total. Precision (P), Recall (R) and Micro F1 score (F1) are reported as previous works. We use BART-large as our backbone. For fair comparison, we reproduce BARTNER (Yan et al., 2021) using the public code 1 and get similar results reported in the paper. We compare our model principally with SOTA generative NER models, including (Yan et al., 2021; Zhang et al., 2022; Lu et al., 2022). Performances of SOTA discriminative NER models (Li et al., 2022) are also listed for reference. Refer to Appendix A for more details.
1https://github.com/yhcc/BARTNER/
The results for flat, overlapped and discontinuous NER are shown in Table 2, Table 3 and Table 4 respectively. On eight datasets, our proposed sequence calibration consistently boosts the baseline. It achieves SOTA performance among the generative methods. Noting that our method gets competitive results even compared with discriminative methods that use extra embedding and domain pretrained model, which shows the potential of generative models.

heading: 3.2 Analysis of Improvement
text: We manually analyze the predictions corrected by the calibration. Apart from reranking the correct candidate to the top beam, RerankNER can generate new candidates with boundary or type corrected. More cases can be found in Appendix B.
In addition to manually observing examples, we also quantitatively analyze the sources of gain. We find that the gain mostly comes from samples with low likelihood, which means sequence likelihood calibration is more effective for samples with higher difficulty. Specifically, we group the samples in the test set into ten groups according to their original sequence likelihood and evaluate their performance before (colored in orange) and after (colored in blue) calibration. It can be seen from Figure 3 that the F1 scores of most groups get improved after calibration, and the improvement is greater for samples with lower likelihoods.
We also conduct the hit@top-k evaluation. Specifically, we iterate over the test samples and increase the number of hits when a gold answer exists among the top-k candidates. Table 5 shows that calibration slightly increase the hit@top-k across various datasets.

heading: 3.3 Variants of Reranker
text: As stated in Section 1, we observe that previous methods have difficulty capturing the subtle nuance among the candidates. We have investigated three variants: (1) SimCLS (Liu and Liu, 2021). (2) SimCLS with our modification which concatenates the input and the candidate representation and projects it to a score to replace the cosine similarity. (3) Picking out the best candidate based on the estimated likelihood of our model. Overall, we find their training losses fluctuate and their performance consistently lower than the baseline which selects the top beam with the highest likelihood. Future work could investigate this phenomenon in more depth.
CoNLL03 OntoNotes5.0 ACE04 ACE05
hit@3 3196/3119 7732/7734 559/566 759/779 hit@5 3240/3138 7858/7869 582/586 786/797
GENIA CADEC ShARe13 ShARe14
hit@3 1135/1161 981/962 8046/8077 14405/14578 hit@5 1245/1254 1005/980 8085/8124 14481/14659
Table 5: Hit@top-k evaluation. Each element in the table denotes the hit count among top-k candidates before/after calibration.

heading: 4 Related Work
text: Named Entity Recognition The existing methods for NER can be broadly classified into sequence labeling formulation, span-based formulation and generative-based formulation. A majority of initial works adopt sequence labeling formulation which assigns each token a tag from a predefined tagging scheme (Huang et al., 2015; Lample et al., 2016). Then, the span-based formulation is proposed which enumerates all possible spans and
performs classification at the span-level (Wang and Lu, 2019). Recently, researchers have grown more interest in tackling the three subtasks uniformly, i.e., flat NER, overlapped NER and discontinuous NER. We refer to them as unified NER in the rest of the passage. The above two formulations have major drawbacks in modeling unified NER. For example, sequence labeling methods need to design different tagging schemas for each subtask (Dai et al., 2020). While span-based methods have to trade-off between maximal span length and computation efficiency due to the enumeration operation (Luan et al., 2019). Generative-based formulation prevails in unified NER for its flexibility in generating variable-length entities (Lu et al., 2022; Yan et al., 2021). In this paper, we adopt BARTNER (Yan et al., 2021) as our backbone generative model.
Bias in Generative NER Since the generative model generates outputs in an autoregressive manner which differs largely from the extraction objective of NER, it introduces incorrect biases during training. (Zhang et al., 2022) analyze these biases from the causality perspective and attribute them to two confounders namely pre-context confounder (the model may be biased to pre-generated words which have no causal relation with the word to be generated) and entity-order confounder. They propose two data augmentation methods to address them respectively. (Tan et al., 2021) observe that overlapped NER is essentially an unordered task and propose a sequence-to-set network to predict entity spans simultaneously in a non-autoregressive manner. W2NER (Li et al., 2022) abandons the generative-based formulation and model unified NER as a word-word relation classification based on the proposed relation schema. In this paper, we improve the generative-based method by exploiting
the candidate information and get comparable or better results.
Reranking Reranking has been explored in various tasks of Natural Language Processing for long. In question answering, passage reranking is used as the first stage to retrieve relevant passages where the answer might locate and reorder them according to their scores. Similarly, answer reranking is used as the last stage to refine the answer selection. In neural machine translation, (Bhattacharyya et al., 2021) apply an energy-based model on the top of BERT to reorder candidates according to their BLEU scores. In abstractive summarization, SimCLS (Liu and Liu, 2021) trains a separate secondstage model with discriminative ranking loss to select the best summary candidate. BRIO (Liu et al., 2022) optimizes the autoregressive language model by a contrastive loss over the discrete space of the generated texts. SummaReranker (Ravaut et al., 2022) adopts a mixture-of-expert architecture as the reranker to measure the quality of the candidates with multiple metrics. To the best of our knowledge, there is no work exploring reranking methods on generative IE.

heading: 5 Conclusion
text: Through pilot experiments, we find the decoded candidates provide potential supervision. Based on this finding, we propose RerankNER to debias generative NER based on a reranking framework adapted for the task. It consistently boosts the baseline and achieves competitive results with state-ofthe-art generative methods on eight NER datasets, which verifies the effectiveness of candidate order supervision. Future work could consider extending this method to other generative IE tasks. Another meaningful direction is to consider incorporating Large Language Models into the reranking process.
Limitations
RerankNER conducts calibration after the regular training, which introduces extra computational overhead. This drives us to further improve the overall efficiency of our method. Recent works find that few-shot learning serves as an effective finetuning method of pretrained language models. It is reasonable to investigate our model under fewshot learning to reduce the overhead. Although we get competitive results with the state-of-the-art methods, there is still a gap between the oracle score and the best results. We leave them as our future work.

heading: Acknowledgement
text: We thank the anonymous reviewers for their helpful comments on this paper. This work was partially supported by National Key R&D Program of China (No. 2022YFC3600402) and National Social Science Foundation Project of China (21&ZD287). The corresponding author of this paper is Sujian Li.

heading: A Details
text: A.1 Dataset Statistics
The statistics of the datasets are listed in Table 6.
Flat NER subtask We conduct experiments on CoNLL-2003 (Sang and Meulder, 2003) and OntoNotes5.0 (Pradhan et al., 2013) in English. We follow the experimental settings as previous works (Lample et al., 2016; Yan et al., 2021).
Overlapped NER subtask We conduct experiments on ACE 2004 (Doddington et al., 2004), ACE 2005 (Walker et al., 2006), and GENIA (Kim et al., 2003). For ACE 2004 and ACE 2005, we shuffle and split the documents into training, development, and testing in a ratio of 8:1:1 following (Yu et al., 2020). For GENIA, the ratio is set to 8.1:0.9:1.0 following (Yan et al., 2021).
Discontinuous NER subtask We conduct experiments on CADEC (Karimi et al., 2015), ShARe13 (Mowery et al., 2013a), and ShARe14 (Mowery et al., 2013b). These datasets contains approximately 10% discontinuous entities. We follow the experimental settings from (Dai et al., 2020).
A.2 Implementation Details
For the fine-tuning stage, we use the code, the hyper-parameters, the package version from (Yan et al., 2021) and get comparable results on all datasets reported in the paper. We set the max
epoch as 30 with early stop (patience=5). We use AdamW optimizer with the same learning rate as (Yan et al., 2021). Linear learning rate scheduling is employed. For all subtasks, we do predictions on the word-level, i.e., only the position index of the first BPE of each entity word is used.
For the calibration training, we use the standard beam search to generate 5 candidates for each input sentence. We adopt the hyper-parameters as the fine-tuning stage except for the newly added ones. We implement both the fixed margin and the linear margin. The linear margin λ = λ̄(j − i) denotes the linear margin depending on the order difference of the candidates, and λ̄ is a hyper-parameter. We search the value of the margin λ̄ within [0.01, 0.1]. We search the value of coefficient α within [0.1, 1]. Table 7 “mask out tie” means whether we mask out the comparison between candidates with the same F1 score in the contrastive loss. Effects of “add LGold” and “mask out tie” differs across 8 datasets, so we view them as hyper-parameters. All experiments are conducted on the NVIDIA RTX 3090 GPU with 24G memory.
A.3 Baselines
The following methods can adapt to all NER subtasks. Please refer to the original papers for the other methods designed specifically for a certain NER subtask.
BERT-MRC (Li et al., 2020) reformulates NER as a machine reading comprehension (MRC) task and extract entities by answering questions such as "find locations in the text".
UIE (Lu et al., 2022) represents various information structures with a structured extraction language and tackles general information extraction tasks with a unified text-to-structure generation framework.
(Zhang et al., 2022) analyzes incorrect biases in the generative NER models from the causality perspective and proposes two data augmentation methods to address them. Note that T5-Base they use has the same number of Transformer layers as BART-Large.
W2NER (Li et al., 2022) reformulates unified NER as a word-word relation classification task based on the proposed relation schema.

heading: B Case Study
text: Table 8 shows some examples corrected by the sequence likelihood calibration.

heading: C Generative Model
text: Our method is agnostic to the generative model. In this study, we adopt BARTNER (Yan et al., 2021), an Encoder-Decoder framework with pointer mechanism, to model the probability P (Y |X):
Encoder encodes the input sentence X into vectors HEnc, which can be denoted as:
HEnc = Encoder(X) (1)
where HEnc ∈ Rn×d and d is the dimension of the hidden state.
Decoder predicts the index probability distribution step-by-step according to P (yt|X,Y<t). Since Y<t consists of the indices of the pointers and tags, it needs to be mapped to the vocabulary indices before inputted to the Decoder. We get the hidden state at the t-th step by:
hDect = Decoder(H Enc; Ŷ<t) (2)
Finally, we get the index probability distribution Pt by:
GDec = Embed(G)
EEnc = Embed(X)
ĤEnc = α ∗HEnc + (1− α) ∗ EEnc
P (yt|X,Y<t) = Softmax([ĤEnc ⊗ hDect ;GDec ⊗ hDect ]) (3) where Embed(·) is the embedding layer shared between the Encoder and Decoder, G denotes the label token while X denotes the entity words. ĤEnc denotes the input representation. ⊗ denotes the dot product. For training, we use the cross-entropy loss with teacher forcing. During inference, we generate the target sequence auto-regressively.
CADEC (2=ADR) Muscle twitching, stiff neck, constant lightheadedness, always worrying about a brain tumor or something.
0.50,-0.05,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying about a brain 2 0.50,-0.12,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying about a brain tumor 2 0.75,-0.15,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying 2 0.50,-0.18,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying about a 2 0.50,-0.23,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying about a tumor 2
0.86,-0.07,Muscle twitching 2 stiff neck 2 lightheadedness 2 0.75,-0.22,Muscle twitching 2 stiff neck 2 lightheadedness 2 always worrying about a brain tumor 2 0.50,-0.33,Muscle twitching 2 stiff neck 2 constant lightheadedness 2 always worrying about a brain tumor 2 0.75,-0.34,Muscle twitching 2 stiff neck 2 lightheadedness 2 always worrying about a brain tumor or something 2 0.75,-0.39,Muscle twitching 2 stiff neck 2 stiff neck 2 lightheadedness 2 always worrying about a brain tumor 2
Possibly diarrhea and stomach pain, but most likely none because I am taking this with a nasty antibiotic for a sinus infection that definitely causes diarrhea and nausea. I stopped taking it the next day and within 72 hours my swelling decreased significantly, muscle aches and joint pain disappeared, memory loss is not as severe, breathing is easier, stamina is back etc.
0.67,-0.05,diarrhea 2 stomach pain 2 0.80,-0.004,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 1.00,-0.14,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.80,-0.16,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 0.86,-0.49,diarrhea 2 stomach pain 2 nausea 2 0.89,-0.22,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 0.88,-0.57,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.89,-0.30,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 0.86-0.76,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.89,-0.41,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2
1.00,-0.08,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 1.00,-0.09,swelling 2 muscle aches 2 joint pain 2 memory loss 2 0.86,-0.23,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.80,-0.23,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 0.86,-0.29,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.80,-0.25,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 1.00,-0.44,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 0.89,-0.26,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina is 2 1.00,-0.53,diarrhea 2 stomach pain 2 diarrhea 2 nausea 2 stomach pain 2 0.80-0.27,swelling 2 muscle aches 2 joint pain 2 memory loss 2 breathing is 2 stamina 2
CONLL2003 (2=LOC,3=PER,4=ORG,5=MISC) POLAND GOT MONEY FROM POST-WAR SWISS ACCOUNTS. Mike Cito, 17, was expelled from St Pius X High School in Albuquerque after an October game in which he used the sharpened chin strap buckles to injure two opposing players and the referee.
0.80,-0.08,POLAND 2 POST-WAR 5 SWISS 5 0.67,-0.01,Mike Cito 3 St Pius X 4 Albuquerque 2 1.0,-0.23,POLAND 2 SWISS 5 1.0,-0.19,Mike Cito 3 St Pius X High School 4 Albuquerque 2 0.50,-0.37,POLAND 2 POST-WAR SWISS 5 0.67,-0.48,Mike Cito 3 St Pius X School 4 Albuquerque 2 1.0,-1.33,POLAND 2 POST-WAR POST-WAR 5 SWISS 5 0.67,-0.58,Mike Cito 3 St Pius X 3 Albuquerque 2 0.80,-1.41,POLAND 2 POST-WAR ACCOUNTS 5 SWISS 5 0.67,-0.61,Mike Cito 3 St Pius X 2 Albuquerque 2
1.0,-0.1.0,POLAND 2 SWISS 5 1.0,-0.11,Mike Cito 3 St Pius X High School 4 Albuquerque 2 0.80,-0.25,POLAND 2 POST-WAR 5 SWISS 5 0.67,-0.14,Mike Cito 3 St Pius X 4 Albuquerque 2 0.80,-0.74,POLAND 2 POST-WAR 5 SWISS 5 SWISS 5 0.80,-0.41,Mike Cito 3 St Cito X High School 4 Albuquerque 2 1.0,-0.76,POLAND 2 SWISS 5 SWISS 5 0.67,-0.44,Mike Cito 3 St Pius X School 4 Albuquerque 2 0.80,-0.78,POLAND 2 POST-WAR 5 SWISS 5 POST-WAR 5 0.86,-0.48,Mike Cito 3 St Pius X High School 4 Albuquerque 2 St Pius X 4
There is the international prestige Singapore would enjoy, but "more importantly there is a genuine national interest in fostering better global free trade and an open market", said Tan Kong Yam, head of Business Policy at the National University of Singapore.
0.86,-0.04,Singapore 2 Tan Kong Yam 3 Business Policy 4 National University of Singapore 4 1.0,-0.07,Singapore 2 Tan Kong Yam 3 National University of Singapore 4 0.86,-0.36,Singapore 2 Tan Kong Yam 3 Business Policy 5 National University of Singapore 4 0.67,-0.68,Singapore 2 Tan Kong Yam 3 Business Policy 4 National University of Singapore 4 0.80,-0.68,Singapore 2 Tan Kong Yam 3 Business Policy of National University of Singapore 4
1.0,-0.06,Singapore 2 Tan Kong Yam 3 National University of Singapore 4 1.0,-0.34,Singapore 2 Tan Kong Yam 3 National University of Singapore 4 National University of Singapore 4 1.0,-0.44,Singapore 2 Tan Kong Yam 3 National University of Singapore 4 Tan Kong Yam 3 0.86,-0.44,Singapore 2 Tan Kong Yam 3 National University of Singapore 4 Singapore 2 0.80,-0.45,Singapore 2 Tan Kong Yam 3 National University of Singapore 4
ACE04 (2=LOC,3=GPE,4=WEA,5=VEH,6=PER,7=ORG,8=FAC) I believe our issues do relate directly to the appointing of electors for the state of Florida.
0.67,-0.06,I 6 our 3 electors for the state of Florida 6 the state of Florida 3 0.80,-0.31,I 6 our 3 electors for the state of Florida 6 the state of Florida 3 Florida 3 0.89,-0.32,I 6 our 6 electors for the state of Florida 6 the state of Florida 3 0.50,-0.32,I 6 our 3 electors for the state of Florida 6 the state of Florida 3 0.50,-0.33,I 6 our 3 electors for the state of Florida 6 the state of Florida 3
1.0,-0.01,I 6 our 6 electors for the state of Florida 6 the state of Florida 3 Florida 3 0.89,-0.10,I 6 electors for the state of Florida 6 the state of Florida 3 Florida 3 0.80,-0.20,I 6 our 3 electors for the state of Florida 6 the state of Florida 3 Florida 3 0.91,-0.47,I 6 our 6 electors for the state of Florida 6 the state of Florida 3 state 3 Florida 3 0.89,-0.48,I 6 our 6 electors for the state of Florida 6 the state of Florida 3
One hundred South Koreans will be in the northern capital Pyongyang, to meet their North Korean relatives.
0.83,-0.08,One hundred South Koreans 6 the northern capital 3 the northern capital Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.77,-0.09,One hundred South Koreans 6 South 3 the northern capital 3 the northern capital Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.77,-0.17,One hundred South Koreans 6 South 2 the northern capital 3 the northern capital Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.67,-0.20,One hundred South Koreans 6 the northern capital 3 the northern capital Pyongyang 3 their 3 their North Korean relatives 6 North Korean 3 0.62,-0.20,One hundred South Koreans 6 South 3 the northern capital 3 the northern capital Pyongyang 3 their 3 their North Korean relatives 6 North Korean 3
0.92,-0.03,One hundred South Koreans 6 South 3 the northern capital 3 Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 1.0,-0.03,One hundred South Koreans 6 the northern capital 3 Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.92,-0.12,One hundred South Koreans 6 South Koreans 6 the northern capital 3 Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.77,-0.17,One hundred South Koreans 6 South Koreans 6 the northern capital 3 the northern capital Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3 0.92,-0.23,One hundred South Koreans 6 South Koreans 3 the northern capital 3 Pyongyang 3 their 6 their North Korean relatives 6 North Korean 3
Netanyahu supporters are calling either for a change in the law or for simultaneous elections for the Knesset and Prime Minister, which would allow their candidate to run.
0.73,-0.07,Netanyahu 6 Netanyahu supporters 6 the Knesset and Prime Minister 6 their 6 their candidate 6 1.0,-0.17,Netanyahu 6 Netanyahu supporters 6 the Knesset 7 Prime Minister 6 their 6 their candidate 6 0.5454545454544859,-0.347797691822052,Netanyahu 6 Netanyahu candidate 6 the Knesset and Prime Minister 6 their 6 their candidate 6 0.7999999999999359,-0.3507174551486969,Netanyahu 6 Netanyahu supporters 6 the Knesset and Prime Prime Minister 6 their 6 their candidate 6 0.8333333333332694,-0.35162267088890076,Netanyahu 6 Netanyahu supporters 6 Knesset 7 Prime Minister 6 their 6 their candidate 6
0.9999999999999332,-0.010539926588535309,Netanyahu 6 Netanyahu supporters 6 the Knesset 7 Prime Minister 6 their 6 their candidate 6 0.7272727272726645,-0.10700102150440216,Netanyahu 6 Netanyahu supporters 6 the Knesset and Prime Minister 6 their 6 their candidate 6 0.7272727272726645,-0.37671196460723877,Netanyahu 6 Netanyahu supporters 6 the Knesset and Prime Minister 7 their 6 their candidate 6 0.7272727272726645,-0.6056239604949951,Netanyahu 6 Netanyahu supporters 6 the Knesset and Prime Minister candidate 6 their 6 their candidate 6 0.8333333333332694,-0.6295873522758484,Netanyahu 6 Netanyahu supporters 6 the Knesset 6 Prime Minister 6 their 6 their candidate 6
Table 8: Case Study. Candidates before (upper) and after (lower) calibration. Each candidate is formatted as "F1, log-probability, target sequence". The number denotes the corresponding entity type.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After Conclusion.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The first.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix A
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We obtained proper licencing and will not distribute the artifacts.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We stick to the intended use only.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? A.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? A.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? A.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could consider extending this method to other generative IE tasks. Another meaningful direction is to consider incorporating Large Language Models into the reranking process. Limitations
RerankNER conducts calibration after the regular training, which introduces extra computational overhead. This drives us to further improve the overall efficiency of our method. Recent works find that few-shot learning serves as an effective finetuning method of pretrained language models. It is reasonable to investigate our model under fewshot learning to reduce the overhead. Although we get competitive results with the state-of-the-art methods, there is still a gap between the oracle score and the best results. We leave them as our future work