Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing
Authors: Yanlin Feng
Adithya Pratapa
David R Mortensen
Venue: EMNLP 2023 Findings (2023)
Keywords: entity typing
information extraction
probability calibration

Abstract: Ultra-fine entity typing plays a crucial role in information extraction by predicting fine-grained semantic types for entity mentions in text. However, this task poses significant challenges due to the massive number of entity types in the output space. The current state-of-the-art approaches, based on standard multi-label classifiers or cross-encoder models, suffer from poor generalization performance or inefficient inference speed. In this paper, we present CASENT, a seq2seq model designed for ultra-fine entity typing that predicts ultra-fine types with calibrated confidence scores. Our model takes an entity mention as input and employs constrained beam search to generate multiple types autoregressively. The raw sequence probabilities associated with the predicted types are then transformed into confidence scores using a novel calibration method. We conduct extensive experiments on the UFET dataset which contains over $10k$ types. Our method outperforms the previous state-of-the-art in terms of F1 score and calibration error, while achieving an inference speedup of over $50$ times. Additionally, we demonstrate the generalization capabilities of our model by evaluating it in zero-shot and few-shot settings on five specialized domain entity typing datasets that are unseen during training. Remarkably, our model outperforms large language models with 10 times more parameters in the zero-shot setting, and when fine-tuned on 50 examples, it significantly outperforms ChatGPT on all datasets.

Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing
Yanlin Feng"
Adithya Pratapa David Mortensen
I "Language Technologies Institute, Carnegie Mellon University
Megagon Labs
yanlin@megagon.at (vpratapa, dmortens]@cs. cmu.
Abstract
Input:
In addition, Greer said, scientists do not know if chemically treated oil will degrade quickly oil that 's dispersed through wind and wave, and it 's more toxic.
Ultra-fine entity typing plays crucial role in in- formation extraction by predicting fine-grained semantic types for entity mentions in text. How- ever, this task poses significant challenges due the massive number of entity types in the output space. The current state-of-the-art ap- proaches, based on standard multi-label classi- fiers cross-encoder models, suffer from poor generalization performance or inefficient infer- ence. In this paper, we present CASENT, seq2seq model designed for ultra-fine entity typing that predicts ultra-fine types with cal- ibrated confidence scores. Our model takes entity mention input and employs con- strained beam search to generate multiple types autoregressively. The raw sequence probabil- ities associated with the predicted types are then transformed into confidence scores using novel calibration method. We conduct exten- sive experiments on the UFET dataset which contains over 10k types. Our method outper- forms the previous state-of sfate-of-the--at the-art in terms of Fl score and calibration error, while achieving an inference speedup of over times. Addi- tionally, we demonstrate the generalization ca- pabilities of our model by evaluating it in zero- shot and few-shot settings on five specialized domain entity typing datasets that are unseen during training. Remarkably, our model out- performs large language models with 10 times more parameters in the zero-shot setting, and when fine-tuned on 50 examples, it significantly outperforms ChatGPT all datasets.'
Box4 Types (Onne et al., 2021):
object
LITE (Li et al., 2022):
petrolcum (0.00) joazi
object rosst
10
liquid toast
material jo
CASENT (ours):
10.551
liuuid jassi substance
object
petroleum (0:30)
Ruid (2:22)
Figure 1:
Comparison of predicted labels and confi- dence scores for UFET test example using Box4Types (Onoe et al., 2021), LITE (Li et al., 2022), and our ap- proach, CASENT. Predictions are sorted in descending order based on confidence. Box4Types fails gener- alize to rare and unseen types, while LITE does not predict calibrated confidence scores and exhibits slow inference speed.
type inventories (Ling and Weld, 2012) which im- posed severe limitations on the practical value of such systems, given the vast number of types in the real world. For example, WikiData, the cur- rent largest knowledge base in the world, records more than 2.7 million entity types?. As a result, fully supervised approach will always be ham- pered by insufficient training data. Recently, Choi et al. (2018) introduced the task of ultra-fine entity typing (UFET), multi-label entity classification task with over 10k fine-grained types. In this work, we make the first step towards building an efficient peneral-purpose entity typing model by leveraging the UFET dataset. Our model not only achieves state-of-ther -hee-arr performance on UFET but also gen- eralizes outside of the UFET type vocabulary. An
1 Introduction
Classifying entities mentioned in text into types, commonly known as entity typing, is a fundamen- tal problem in information extraction. Earlier re- search on entity typing focused relatively small
This work was done while the first author was Carnegie Mellon University.
'Our code, models and demo are available at https:// gi github. Comm/yanninf/AASEM
"sstmmttdd from the unique children in the subclassOf (P279) relations using the Fehruary 2023 Wikidata dump.

---

example prediction of our model is shown in Fig- ure
employ beam search to decode multiple types and use a probability threshold select types. How- ever, we show that this approach fails to achieve optimal performance as the raw conditional prob- abilities do not align with the true likelihood of the corresponding types. In this work, we propose to transform the raw probabilities into calibrated confidence scores that reflect the true likelihood of the decoded types. To this end, extend Platt scaling (Platt et al., 1999), a standard technique for calibrating binary classifiers, the multi-label setting. To mitigate the label sparsity issue in ultra- fine entity typing, we propose novel weight sharing and efficient approximation strategies. The ability to predict calibrated confidence scores not only im- pacts task performance but also provides flexible means of adjusting the trade-off between precision and recall in real-world scenarios, For instance, in applications requiring high precision, predictions with lower confidence scores can be discarded.
Ultra-fine entity typing can be viewed as multi- label classification problem over an extensive label space. A standard approach to this task employs multi-label classifiers that map contextual represen- tations of the imput entity mention to scores using a linear transformation (Choi et al., 201 8; Dai et al., 2021: Onoe et al., 2021). While this approach of- fers superior inference speeds, it ignores the type semantics by treating all types as integer indices and thus fails to generalize to unseen types. The current statc-of-the of--th--ttt-tttt approach (Li et al., 2022) reformulated entity typing as a textual entailment task. They presented cross-encoder model that computes an entailment score between the entity mention and candidate type. Despite its strong generalization capabilities, this approach is ineffi- cient given the need to enumerate all 10k types in the UFET dataset.
We carry out extensive experiments on the UFET dataset and show that filtering decoded types based on calibrated confidence scores leads to state-of- the-art performance. Our method surpasses the previous methods in terms of both F1 score and cal- ibration error while achieving an inference speedup of more than 50 times compared to cross-encoder methods. Furthermore, we evaluate the zero-shot and few-shot performance of our model on five specialized domains. Our model outperforms Flan- T5-XXL (Chung et al., 2022), an instruction -uuned large language model with 11 billion parameters in the zero-shot setting, and surpasses ChatGPT when fine-tuned 50 examples.
Black-box large language models, such as GPT-3 and ChatGPT, have demonstrated impressive zero- shot and few-shot capabilities in a wide range of generation and understanding tasks (Brown et al.. 2020; Ouyang et al., 2022). Yet, applying them to ultra-fine entity typing poses challenges due to the extensive label space and the context length limit of these models. For instance, Zhan et al. (2023) reported that GPT-3 with few-shot prompting does not perform well on a classification task with thou- sands of classes. Similar observations have been made in our experiments conducted on UFET.
In this work, we propose CASENT, a Calibrated Seq2Seq model for Entity Typing. CASENT pre- dicts ultra-fine entity types with calibrated con- fidence scores using seq2seq model (T5-large (Raffel et al., 2020)). Our approach offers sev- eral advantages compared to previous methods: (1) Standard maximum likelihood training without the need for negative sampling or sophisticated loss functions (2) Efficient inference through a single autoregressive decoding pass (3) Calibrated confi- dence scores that align with the expected accuracy of the predictions (4) Strong generalization perfor- mance unseen domains and types. An illustration of our approach is provided in Figure 2.
2 Related Work
2.1 Fine-grained Entity Typing
Ling and Weld (2012) initiated efforts to recognize entities with labels beyond the small of classes that is typically used in named entity recognition (NER) tasks. They proposed to formulate this task as a multi-label classification problem. More re- cently, Choi et al. (2018) 8) extended this idea to ultra- fine entity typing and released the UFET dataset, expanding the task to include an open type vocab- ulary with over 10k classes. Interest in ultra-fine entity typing has continued to grow over the last few years. Some research efforts have focused on modeling label dependencies and type hierarchies, such as employing box embeddings (Onoe et al., 2021) and contrastive learning (Zuo et al., 2022).
While seq2seq formulation has been successfully applied NLP tasks such entity linking (De Cao al., 2020, 2022), its application to ultra-fine entity typing remains non-trive trivial due to the multi-labe prediction requirement, A simple adaptation would

---

investor
<MM-Mr. Dorfmans/MM: states that invested $100,000 year ago the first four...
businessman
millionaire
seq2seq
Plavsic's Interior Ministry said that <Morhhys</Mb would take all
organization
government
Training
Inference
type
pit
constrained beam search w/ profix
person:
sentenced <Maa Palestinlans/Mss forcing a cliff
criminal
0.96
criminal:
0.7
seq2seq
adoleccent
-1.37
adult: 0.2
type
seq2seq
log elt B3
-1.04
5.31
-2.23
ppre-computed madel blas)
Figure 2: Overview the training inference process of CASENT. We present an example output from our model.
Another line of research has concentrated on data augmentation and leveraging distant super- vision. For instance, Dai et al. (2021) obtained training data from pretrained masked language model, while Zhang et al. (2022) proposed de- noising method based on an explicit noise model. et (2022) formulated the task natural lan- guage inference (NLD) problem with the hypothesis being an "is-a" statement. Their approach achieved state-of-the-art performance on the UFET dataset and exhibited strong generalization to unseen types, but is inefficient at inference due to the need to enu- merate the entire type vocabulary.
Although probability calibration has been exten- sively studied for single-label classification tasks (Jiang et al., 2020; Kadavath et al., 2022), it has rarely been explored in the context of fine-graines entity typing which is multi-label classification task. To the best of knowledge, the only excep- tion Onoe et al. (2021), where the authors applied temperature scaling to BERT based model trained on the UFET dataset and demonstrated that the re- sulting model was reasonably well-calibrated.
Methodology
In this section, we present CASENT, a calibrated seq2seq model designed for ultra- fine entity typing. We start with the task description ($3.1) followed by overview of the CASENT architecture ($3.2). While the focus of this paper is the task of entity typing, our model can be easily adapted other multi-label classification tasks.
2.2 Probability Calibration
Probability calibration is the task of adjusting the confidence scores of a machine learning model to better align with the true correctness likelihood. Calibration is crucial for applications that require interpretability and reliability, such as medical di- agnoses. Previous research has shown that modern neural networks while achieving good task per- formance, are often poorly calibrated (Guo et al., 2017; Zhao al., 2021). One common technique for calibration in binary classification tasks Platt scaling (Platt al., 1999), which fits a logistic re- gression model on the original probabilities. Guo et al. (2017) proposed temperature scaling as an extension of Platt scaling in the multi-class setting.
3.1 Task Definition
Given an entity mention e, we aim to predict set of semantic types t = (t....., fn) C T, where T is predefined type vocabulary (T 10331 for the UFET dataset). We assume each type in the vocabulary noun phrase that can be represented by sequence of tokens assume the availability of a training set Dtrain with annotated (c, t) pairs as well as a development set
(V1- ..,...,,ll. 2kk. We

---

for estimating hyperparameters.
tive sampling, which is required by previous cross- encoder approaches (Li et al., 2022; Dai al., 2021).
3.2 Overview of CASENT
Figure 2 provides an overview of our system. consists of seq2seq model and calibration mod- ule. At training time, we train the seq2seq to output a ground truth type given an input entity mention by maximizing the length-normalized log-likelihood using an autoregressive formulation
3.4 Calibration
At the core of our approach is calibration mod- ule that transforms logPa(t e) into calibrated confidence p(t e). We will show in section that directly applying thresholding using po e) is suboptimal as it models the distribution over target token sequences instead of the likelihood of e belonging to certain type f. Our approach builds on Platt scaling (Platt et al., 1999) with three proposed extensions specifi- cally tailored for the ultra-fine entity typing task: 1) incorporating model bias pe (t Ø). 2) frequency- based weight sharing across types, and 3) efficient parameter estimation with sparse approximation.
conditional pog-prbbbbility
k log (I yesse e)
1
log po e)
- 1:
(1)
where denotes the parameters the seq2seq model.
During inference, our model takes an entity men- tion e as input and generates small set of candi- date types autoregressively via constrained beam search by using relatively large beam size. We then employ a calibration module transform the raw conditional probabilities (Equation associ- ated with each candidate type into calibrated con- fidence scores p(t e) E [0, 1].² The candidate types whose scores surpass a global threshold are selected as the model's predictions.
Platt Scaling: We first consider calibration for each type separately, in which case the task re- duces binary classification problem. A stan- dard technique for calibrating binary classifiers is Platt scaling, which fits logistic regression model on the original outputs. A straightforward application of Platt scaling in our seq2seq set- ting computes the calibrated confidence score by (w, log (o e) + by), where a is the sigmoid function and calibration parameters wi and b are estimated the development set by minimizing the binary cross-entropy loss.
The parameters of the calibration module and the threshold are estimated on the development set before each inference either the end of each epoch when the training is complete). The detailed process of estimating calibration parameters discussed in $3.4.
(which takes place
Inspired by previous work (Zhao et al., 2021) which measures the bias of seq2seq models by feed- ing them with empty inputs, we propose to learn weighted combination of both the conditional probability pt(t e) and model bias po(t Ø). Specifically, we propose
3.3 Training
Our seq2seq model is trained to output a type t given an input entity mention e. In the training set, each annotated example (e, t) E Dusin with t| = ground truth types is considered as n sepa- rate input-output pairs for the seq2seq model. We initialize our model with pretrained seq2seq lan- guage model, T5 (Raffel al., 2020), and finetune it using standard maximum likelihood objective:
(1), log (t e) w( log 2) bt)
as the calibrated confidence score. We will show in section 4 that incorporating the model bias term improves task performance and reduces calibration error.
min
Σ Elog log (e e)
(2)
(e,t) EEt.
Multi-label Platt Scaling: We now discuss the extension of this equation in the multi-label setting where [T] 1. A naive extension that consid- each type independently would introduce 3/ parameters and involve training [T] logistic regres- sion models on |Daevl [T] data points. To mitigate this difficulty, we propose to share calibration pa- rameters across types based on their occurrence
Our seq2seq formulation greatly simplifies the training process by eliminating the need for nega-
slight abuse of notation by treating i as binary random variable that indicates whether belongs type
'Here,
"Note that although training example (e, t) is separated into n input-output pairs, the forward pass at the encoder only needs to be computed once.

---

Algorithm 1: Calihration parameters estimation
work (De Cao et al., 2020, 2022), pre-compute prefix tric based on and force the model se- lect valid tokens during each decoding step. Next, we compute the calibrated confidence scores us- ing Equation 3 and discard types whose scores fall below the threshold.
function GetCallibratinParrams
model,
n_groups)
D [C] for
(rggupps]
stores the data points for
estimating cal.bration parameters
for e, types in Dav
for t model. beam_seacch(()) x [log pa (t]()), logpo [t] if types then
In section 4, also conduct experiments single-label entity typing tasks. In such cases, we directly score each valid type using Equation and select the type with the highest confidence score.
L y
else
L y D[o(t)] append y))
Experiments
W w- np. zeros zeros (n_groups, 2))
4.1 Datasets
B + np. zewos for i in range(n_groups do
W[i, :], B[i]
We the UFET dataset (Choi al., 2018), stan- dard benchmark for ultra-fine entity typing. This dataset contains 10331 entity types and is curated by sampling sentences from GigaWord (Parker et al., 2011), OntoNotes (Hovy et al., 2006) and web articles (Singh et al., 2012).
Fitlogiistcceegrsssioo((tt)))
15
return W, B
frequency in the dataset:
To test the out-of-domain generalization abili- ties of our model, we construct five entity typing (3) datasets for three specialized domains. We derive these from existing NER datasets, WNUT2017 (Derczynski et al., 2017), JNLPBA (Collier and Kim, 2004), BC5CDR (Wei et al., 2016), MIT- restaurant and MIT-movie.? We treat each an- notated entity mention span an input to entity typing model. WNUT2017 contains user- generated text from platforms such Twitter and Reddit. JNIPBA and BC5CDR are both sourced from scientific papers from the biomedical field. MIT-restaurant and MIT-movie are customer re- view datasets from the restaurant and movie do- mains respectively. Table provides the statistics and an example from each dataset.
(1)
p(t c) = a w
(()) log (t
(2) log (t Ø) bo(e)
(())
where
(tt)
log, (Freq(t) 1]]
(4)
maps type t to its frequency category. Intuitively. rare types are more vulnerable to model bias thus should be handled differently compared frequent types.
Furthermore, instead of training logistic regres- sion models on all [Daev] [T] data points, we propose a sparse approximation strategy that only leverages candidate types generated by the seq2seq model via beam search. 6 This ensures that the entire calibration process retains the same time the devel-
4.2 Implementation
complexity regular evaluation opment set. The pseudo code for estimating cali- bration parameters is outlined in algorithm Once the calibration parameters have been estimated, we select the optimal threshold by running simple linear search.
We initialize the seq2seq model with pretrained T5-large (Raffel et al., 2020) and finetune on the UFET training set with a batch size of 8. We optimize the model using Adafactor (Shazeer and Stern, 2018) with a learning rate of le-5 and a con- stant learning rate schedule. The constrained beam search during calibration and inference uses beam size of 21. We mark the entity mention span with special token and format the input according to the template " "[CONTEXT) 
 (ENTITY) is 
". Input and the target entity type are tokenized using the standard T5 tokenizer.
3.5 Inference
At test time, given an entity mention e, we employ constrained beam search to generate a set of can- didate types autoregressively. Following previous
'On UFET dataset, this reduces number of calibra- tion parameters from 30993 27.
"This reduces the maximum number of calihration data points to [Dae Beamšize.
csail.m .im edu/s http///sddowwloddss

---

Dataset
Domain
Entity Types (T)
Example
UFET
News, web articles
10331 types
[The explosions Jevent, calamity attack, dissser occurred on the night of October 7, against the Hilton Taba und campsites used by Israclis in Ras al- al-Shi.a.
WNUT2017
Social media
[corporation, creative work, group, location, per- son, product
RT @ Doofs I did drawn the [Tiger
Marna jercative @ @BubiAArts
JNLPBA
Biomedical
DNA, RNA, call Jine, call copl,ype, procein]
In vivo control of [NP-kappa B Jprotcin activation by kappa B alpha.
BC5CDR
Biomedical
[disesse, chemical)
In previous phase II study with 3 weekly bolus
[5-FU Jehemicat, FA and mitomycin MMC) found toxicity and response regimens
ELE FAM FAMTX, promising median overall survival.
parable to
MIT-restaurant
Customer review
[riinng, amenity, location, restaurans, price, hours, dish, cuisine]
Can you make reservation at [pf changs restamment
for tonight?
MIT-movic
Customer review
(actor, plot, opinion, award, year, genre, origin,
[animated] movic about criminal master-
restor, relationship, character, quoce]
mind attempts steal
Table Dataset statistics examples. Only UFET multiple types for each entity mention.
4.3 Baselines
Method
P
R
F1
We compare our method to previous state-of- the-an art approaches, including multi-label busssffier-bssed methods such as BiLSTM (Choi et al., 2018), BERT, Box4Types (Onoe et al., 2021) and MI- MET (Dai et al., 2021). ). In addition, we include bi-encoder model, UniST (Huang et al., 2022) as well as the current state-of-the-art method, LITE (Li et al., 2022), which is based on a cross-encoder architecture.
Few-shot methods
ChatGPT (0-shot)
55.5
10.5
17.6
ChatGPT (8-shot)
46.7
34.9
40.0
ChatGPT (16-shot)
47.8
36.7
41.5
CharGPT (32-shot)
45.9
37.3
41.2
Supervised methods
BiLSTM (Choi et al., 2018)
47.1
24.2
32.0
BERT (Onoc and Durrett, 2019)
51.6
33.0
40.2
Box4Types (Onoe et al., 2021)
52.8
38.8
44.8
MLMET (Dai al., 2021)
53.6
45.3
49.1
UniST (Huang al., 2022)
50.2
49.6
49.9
We also compare with ChatGPT 8 and Flan-T5- XXL. (Chung et al., 2022), two large language mod- els that have demonstrated impressive few-shot and zero-shot performance across various tasks. For the UFET dataset, randomly select a small set of ex- amples from the training set as demonstrations for each test instance. Instruction is provided before the demonstration examples to facilitate zero-shot evaluation. Furthermore, for the five cross-domain entity typing datasets, we supply ChatGPT and Flan-T5-XXI with the complete list of valid types. Sample prompts are shown in Appendix A.
LITE (Li et al., 2022)
52.4
48.9
50.6
CASENT (Ours)
53.3
49.5
51.3
Table 2: Macro-averaged precision, recall and Fl score (%) the UFET test set. The model with highest F1 score is shown in bold and the second best is underlined.
the Fl score. Among the cross-encoder models demonstrate superior perfor- mance over both bi-encoder methods and multi- label classifier-based models.
models,
ChatGPT exhibits poor zero-shot performance with significantly low recall. However, it is able to achieve comparable performance to a BERT-based classifier with a mere few-shot examples. Despite this, its performance still lags behind recent fully supervised models.
5 Results
5.1
UFET
Table 2, we compare our approach with suite of baselines and state-of-the-an systems on the UFET dataset. Our approach outperforms LITE (Li et al., 2022), the current leading system based on cross- encoder architecture, with a 0.7% improvement in
5.2 Out-of-domain Generalization
We evaluate the out-of-domain generalization per- formance of different models on the five datasets discussed 1. The results are presented in Ta- ble It is important to note that we don't compare
"We use the gpt-3.. 3.555.5tu wwt.5-tttto033- 0301 model available via the OpenAl API.

---

Social Media
Riomedical
Customer Review
Method
WNUT 2017
JNLPBA
BC5CDR
MIT-restaurant
MIT-movie
Avg.
Zero-shot methods
Random
16.7
20.0
50.0
12.5
8.3
21.5
Flan-T5-XXL
62.9
71.8
63.0
39.6
45,4
56.5
ChatGPT
76.3
85.4
80.3
77.2
83.2
LITE (Li et 2022)
67.0
74.9
47.7
54.5
68.0
CASENT (no finetuning, no calibration)
63.5
79.2
98.2
52.9
51.2
69.4
Fhw-shot methods
RoBERTa-large (finetuned on 50 examples)
63.5
85.1
96.2
75.0
69.9
78.3
CASENT (no finetuning, calibration dev)
74.2
84.0
98.2
68.5
71.7
79.3
CASENT (finetuned examples)
77.3
92.2
98.8
81.8
86.2
87.2
set accuracy on five specialized domain entity typing datasets derived from existing NER datasets. The best score is shown in bold and the second best is underlined. The results of LITE are obtained by running inference using the model checkpoint provided by the authors.
Table 3:
with multi-label classifier models like Box4Types and MLMET that treat types as integer indices, as they are unable generalize unseen types.
Calibration on rare types
Calibration on frequent types
Perfect calibration CASENT
Acouncyy
0.6
In the zeoo-shot setting, LITE and CASENT are trained on the UFET dataset and directly evaluated the target test set. Flan-T5-XXL and ChatGPT are evaluated by formulating the task classifica- tion problem with all valid types as candidates. As shown in Table 3, ChatGPT demonstrates superior performance with large margin compared to other models. This highlights ChatGPT's capabilities on classification tasks with a small label space. Our approach achieves comparable results to LITE and significantly outperforms Flan-T5-XXL, despite having less than 10% of its parameters.
0.2
CASENT
0.8 0.0.0
Confidence
Confidence
Figure 3: Reliability diagrams of CASENT the UFET test set. The left diagram represents rare types with fewer than 10 occurrences while the right diagram rep- resents frequent types.
Analysis
We also conduct experiments in the few-shot setting, where either a small training set or develop- ment set is available. We first explore re-estimating the calibration parameters of CASENT on the tar- get development set by following the process dis- cussed in $3.4 without weight sharing and sparse approximation." 9 Remarkably, this re-calibration process, without any finetuning, results in an ab- solute improvement +9.9% and comparable performance with ChatGPT on three out of five datasets. When finetuned on 50 randomly sampled examples, our approach outperforms ChatGPT and finetuned RoBERTa model by significant mar- gin, highlighting the benefits of transfer learning from the ultra-fine entity typing task.
6.1 Calibration
Table 4 presents the calibration error of different approaches. We report Expected Calibration Error (ECE) and Total Calibration Error (TCE) which measures the deviation of predicted confidence scores from empirical accuracy. Interestingly, we observe that the entailment scores produced by LITE, the state-of-the-art cross-encoder model, are poorly calibrated. Our approach achieves slightly lower calibration error than Box4Types, which ap- plics temperature scaling (Guo et al., 2017) to the output of a BERT-based classifier. Figure dis- plays the reliability diagrams of CASENT for both rare types and frequent types. As illustrated by the curve in the left figure, nigh-confidence predictions for rare types are less well-calibrated.
"The number of calibration parameters is 3/7), which less than 10 on all five datuscts.

---

Method
Calibration Method
Test-Fl
Test-ECE (%)
Test-TCE (%)
Dev-TCE
Tyxes
Temperature scaling
44.8
11.19
LITE
50.6
52.36
52.36
52.56
CASENT
Eq.
1.23
9.75
Eq. without the model bias term pa Ø))
49,
3.87
20.34
14.76
Eq. with ((() = (independent weights)
48.8
7.37
$7.00
9.72
Eq. with o(t) to (all types share sume weights)
47.8
3,89
34.57
36.29
po(f c) calibration)
47.3
12.19
118.16
100.31
Table 4: Macro FI, ECE (Expected Calibration Error) and TCE (Total Calibration Error) the UFET dataset. ECE and TCE are computed using 10 bins. The best score is shown in bold. Onoe et al. (2021) only reported calibration results on dev set thus the results Box4Types on test set are not included.
Method
# params
F1
Method
Training Time
Inference Latency
GPU Mem.
T5-small
40.9
MLMET
180ht
0.02 + 0.05s
0.5Gb
80M
15-small + CASENT
47.2
LITE
40h
23.1 = 5. 73s
1.4Gb
CASENT
6h
0.39 0.04s
2.8Gb
T5-hase
250M
T5-base + CASENT
49.6
Table 6: Training time, inference latency and inference time GPU memory usage estimated single NVIDIA RTX A6000 GPU. Inference time statistics are estimated using 100 random UFET examples. Results marked by i are reported by Li et al. (2022).
Ta-large T5-large CASENT
47.3
780M
51.3
TS-3B
48.6
3B
T5-3B CASENT
51.4
Table Macro score (%) of CASENT on the UFET test set with different T5 variants.
0.52
3.2
Calibration error
6.2 Ablation Study
2.4
We also perform an ablation study to investigate the impacts of various design choices in our proposed calibration method. Table 4 displays the results of different variants of CASENT. vanilla seq2seq model without any calibration yields both low task performance and high calibration error, highlight- ing the importance of calibration. Notably, naive extension of Platt scaling that considers each type independently leads to significant overfitting, illus- trated by an absolute difference of 47. 78% TCE between the development and test sets. Removing the model bias term also has a negative impact on both task performance and calibration error.
ECE (9)
1.6
0.50
0.49
0.0
12
16
20
28
32
36
Beam size
Figure Test set Macro score and Expected Cali- bration Error (ECE) with respect the beam size the UFET dataset.
6.3 Choice of Seq2seq Model
single GPU, while previous methods require more than 40 hours. While CASENT achieves an infer- ence speedup of over 50 times over LITE, it is still considerably slower than MIMET, BERT-based classifier model. This can be attributed to the need for autoregressive decoding in CASENT.
In Table 5, we demonstrate the impact of calibra- tion on various T5 variants. Our proposed cali- bration method consistently brings improvement across models ranging from 80M parameters 3B parameters. The most substantial improvement is achieved with the smallest T5 model.
6.5 Impact of Beam Size
6.4 Training and Inference Efficiency
Given that the inference process of CASENT relies on constrained beam search, we also investigate the impact of beam size on task performance and calibration error. As shown in Figure 4, a beam
Table 6, we compare the efficiency of our method with previous state-of-the-art systems. Re- markably, CASENT only takes hours to train on

---

size of 4 results in a low calibration error but also low Fl scores, as it limits the maximum number of predictions. CASENT consistently maintains high Fl scores with minor fluctuations for beam sizes ranging from 8 to 40. On the other hand, a beam size between 8 and 12 leads to high calibration errors. This can be attributed to our calibration parameter estimation process in algorithm 1, which approximates the full [Paev] |T] calibration data points using model predictions generated by beam search. A smaller beam size leads to smaller number of calibration data points, resulting in a suboptimal estimation of calibration parameters.
copyright notation thereon. The views and con- clusions contained herein are those of the authors and should not be interpreted as necessarily repre- senting the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
7 Conclusion
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 87-96.
Engineering decisions often involve tradeoff be- tween efficiency and accuracy. CASENT simulta- neously improves upon the state-of-the-an in both dimensions while also being conceptually elegant. The heart of this innovation is constrained beam search with a novel probability calibration method designed for seq2seq models in the multi-labe clas- sification setting. Not only does this method outper- form previous methods-including including ChatGPT and the existing fully-supervised methods-or on ultra- fine entity typing, but also exhibits strong gener- alization capabilities to unseen domains.
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghami. Siddhartha Brahma, al. 2022. Scaling instruction-finetune language models. arXiv preprint arXiv: 2210.11476.
Nigel Collier and lin-Dong Kim. 2004. Introduction to the bio-entity recognition task at jnlpba. Proceed- ings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Appli- cations (NLPBA/BioNLP)), pages 73-78.
8 Limitations
Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from a masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1790-17999
While our proposed CASENT model shows promis- ing results on ultra-fine entity typing tasks, it does have certain limitations. Our experiments were conducted using English language data exclusively and it remains unclear how well our model would perform on data from other languages. In addition, our model is trained on the UFET dataset, which only includes entity mentions that are identified noun phrases by constituency parser. Conse- quently, certain types of entity mentions such as song titles are excluded. The performance and ap- plicability of our model might be affected when dealing with such types of entity mentions. Future work is needed to adapt and evaluate the proposed approach in other languages and broader scenarios.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive en- tity linking. Transactions of the Association for Com- putational Linguistics, 10:274-299
Leon Derezynski, Eric Nichols, Marieke Van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Acknowledgements
This material is based on research sponsored by the Air Force Research Laboratory under agreement number FA8750- 19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conf conference on machine learn- ing, pages 1321-1330, PMLR

---

Eduard Hovy, Mitch Mareus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In Proceedings of the human lan- guage technology conference of the NAACL, Com- panion Volume: Short Pupers, pages 57-60.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a umified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555
James Y Huang, Bangzheng Li, Jiashu Xu, and Muhao Chen. 2022. Unified semantic typing with mean- ingful label inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642-2654.
Noam Shazeer and Mitchell Stem. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. pages 4596-4604, PMLR.
Sameer Singh, Amaragg Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large- scale cross-document coreference corpus laheled via links to wikipedia. University of Massachusetts, Amherst, Tech. Rep. UM-CS-2012, 15.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguisties, 8:423-438.
Chih- Huth-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Pe- ter Davis, Carolyn J Mattingly Jiao Li, Thomas C Wiegers, and Zhiyong Lu. 2016. Assessing the state of the art biomedical relation extraction: overview of the biocreative v chemical-disease relation (cdr) task. Database, 2016.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer. Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Qiusi Zhan, Sha Li, Kathryn Conger, Martha Palmer, Heng Ji, and Jiawei Han. 2023. Glen: General- purpose event detection for thousands of types. arXiv preprint arXiv:2303.0909.
Bangzheng Li, Wenpeng Yin, and Muhao Chen. 2022. Ultra-fne entity typing with indirect supervision from natural language inference. Trunsactions of the Association for Computational Linguistics, 10:607- 622.
Yue Zhang. Hongliang Fei, and Ping Li. 2022. De- noising enhanced distantly supervised ultrafine entity typing. arXiv preprint arXiv:2210009553
Xiao Ling and Daniel Weld. 2012. Fine-grained entity recognition. In Proceedings of the AAAI Conference on Artificial Intellience, volume 26, pages 94-100.
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. 2021, Modeling fineggraieed entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 2051-26 1264..
Xinyu Zuo, Haijin Liang, Ning Jing, Shuang Zeng, Zhou Fang, and Yu Luo. 2022. Type-enrichen hi- erarchical contrastive strategy for fine-grained entity typing. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 2405- 2417.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly- data for entity typing. In Proceedings of NAACL-HLT, pages 2407-2417
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Macda. 2011. English gigaword fifth edition Idc201 1107 (tech. rep.). Technical report, Technical Report. Linguistic Data Consortium, Philadelphia.
John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances large margin classifiers, 10(3)661-74.

---

Labels: academician, scientist, person A ChatGPT Flan-T5 Prompts
Below is sample prompt for ChatGPT and Flan- T3-XXL for the five out-of-domain datasets:
Instruction: Identify the type of the entity men- tion tagged by <mark>. Output the type directly and do write any explanation.
Choices: DNA, RNA, cell_line, cell_type, protein
Entity: Number <mark>>glucocorticcod recep- tors</mark>> lymphocytes and their sensitivity to hormone action
Label:
For the UFET dataset, is not feasible to provide the model with the entire type vocabulary. Instead we provides demonstration examples sampled from the training set. Below is sample prompt with two demonstration examples:
fine-grained entity types for the entity mention tagged by <mark>. Separate the types with commas.
Instruction:
Predict
Entity: <markoHle</mark> get 's zero from Arafat said Benjamin Begin the science minister
Entity: President Obama 's surprise proposal to cancel the $ 188 billion moon program and the jobs that go with <mark>it</mark> triggered an uproar in Texas Florida and other states with space related industries Labels: work, job, bill
Entity: On <mark>late Monday night</marko 30th Nov 2009 Bangladesh Police arrested Rajkhowa somewhere near Dhaka Labels:


Reviews:
[{"review_id": "nwsYBiIFQi", "reviewer": "Reviewer_rRPt", "summary": "", "questions": "- How did you prompt ChatGPT? It would be helpful to include a discussion about its sensitivity to various prompts, etc.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments.\n\n> The evaluation focuses on quantitative results. Without qualitative analysis, it’s hard to see when/why calibration helps.\nSimilarly, no error analysis on baseline outputs is included. It’s difficult to understand what types of errors have been fixed by the proposed approach.\n\nThank you for making these excellent points. We have included an example in Fig. 1 but found it difficult to include discussion of our qualitative evaluation or error analysis due to space but acknowledge that it would greatly enhance the significance of our work. In the final version of the paper, we will use some of the additional space to present a richer analysis of the results errors and how they compare to the baseline outputs.\n\nHere are three examples: the first is from Fig. 1 with added baseline outputs, and the others are randomly sampled. The comparison between CASENT and T5 (CASENT w/o calibration) demonstrates that calibration reduces hallucination by reranking irrelevant concepts with high raw probabilities. Calibration also produces more consistent confidence scores (e.g. baseline T5 assigns higher confidence to “agency” than “organization” while “agency” is a subtype of “organization”).\n*****\nExample 1\n*****\nEntity: A court in Jerusalem sentenced <M>a Palestinian</M> to 16 life terms for forcing a bus off a cliff July 6, killing 16 people, Israeli radio reported.\n\nCASENT: person (0.98) criminal (0.65) male (0.55)\n\nBaseline T5: person (0.61), criminal (0.38), male (0.30), adolescent (0.25), convict (0.25)\n\nGold types: person, criminal\n*****\nExample 2\n*****\nEntity: She was made a doctor of music by the University of Cambridge in 1976 , and became <M>a Dame Commander of the Order of the British Empire -LRB- DBE -RRB-</M> in 1992 .\n\nCASENT: person (0.96), honor (0.45), aristocrat (0.36), leader (0.30), adolescent (0.29), honorary (0.28)\n\nBaseline T5: person (0.51), adolescent (0.41), aristocrat (0.38), honor (0.36), adversity (0.27)\n\nGold types: person, occupation, title\n*****\nExample 3\n*****\nEntity: NASA said the Galileo worm hadn't affected its computers or the computers of other government agencies because <M>they</M> had modified their systems to reject worms.\n\nCASENT: organization (0.67), agency (0.67), government (0.57), administration (0.48), authority (0.33), corporation (0.29)\n\nBaseline T5: agency (0.46), organization (0.40), bureaucracy (0.35), government (0.33), administration (0.28)\n\nGold types: organization, administration, group, working group\n\n> How did you prompt ChatGPT? It would be helpful to include a discussion about its sensitivity to various prompts, etc.\n\nFlan-T5-XXL/ChatGPT prompt for UFET:\n```\nInstruction: Predict the fine-grained entity types for the entity mention tagged by <mark>. Separate the types with commas.\n{randomly sampled demonstration examples}\nEntity: {entity_mention}\nLabels:\n```\nFlan-T5-XXL/ChatGPT prompt for the five specialized domain datasets:\n```\nInstruction: Identify the type of the entity mention tagged by <mark>. Output the type directly and do not write any explanation.\nChoices: {candidate_types}\nEntity: {entity_mention}\nLabel:\n```\nWe manually refine the instruction part of the prompt for both datasets until ChatGPT’s outputs do not have formatting issues. We also experiment with different formats of the entity mention (e.g. listing the context and the entity mention span separately, or tagging the entity mention within the context) and find that ChatGPT’s performance is quite robust (< 2% accuracy difference).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- CASENT includes architectural contributions for ultra-fine entity typing such as multi-label Platt scaling and an efficient training method. Each design choice is well-motivated and reasonable.\n- Experimental results show that the proposed approach is effective, outperforming prior work (the margin is 0.7 but it’s not small given that UFET is 10k multi-label classification). \n- I really appreciate that the authors compare their approaches with SOTA blackbox LLMs such as ChatGPT, confirming that supervised methods still exceed ChatGPT on ultra-fine entity typing.\n- This paper is well written and easy to read. All key concepts are clearly explained.\n", "reasons_to_reject": "- The evaluation focuses on quantitative results. Without qualitative analysis, it’s hard to see when/why calibration helps.\n- Similarly, no error analysis on baseline outputs is included. It’s difficult to understand what types of errors have been fixed by the proposed approach.\n", "questions_to_authors": "- How did you prompt ChatGPT? It would be helpful to include a discussion about its sensitivity to various prompts, etc.", "paper_topic_and_main_contributions": "This paper proposes a novel entity typing model (CASENT) that uses a calibration module to adjust the raw output. During inference, CASENT first employs a seq2seq model (e.g., T5) and autoregressively predicts entity types given an entity mention by using constrained beam search (i.e., forcing the model to select only predefined entity types). Then, the calibration module, which uses logistic regression with three parameters, recomputes the final probability based on the raw probability and the probability of an entity type given an empty input. To reduce the total number of parameters in the calibration module, the authors bucket the candidate entity types by frequency in the data and share the logistic regression parameters within each bucket. Additionally, this calibration module is trained with entity types existing in the beam, rather than with |T| types, to manage time complexity.\n\nThis approach is evaluated using entity typing benchmarks such as UFET, and is compared against SOTA approaches, including ChatGPT. The main result is observed on UFET, where CASENT outperforms all baselines, showing the best calibration errors. In the zero-shot setting, models are trained on UFET and evaluated on other entity typing benchmarks. In this setting, ChatGPT demonstrates strong results, indicating that the generalization abilities of smaller models are limited, while calibration enhances performance.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vwzp2wnUne", "reviewer": "Reviewer_hY4H", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments.\n> First of all, I think the authors should add the baseline seq2seq model (T5 in this case) to all experiments for us to understand better how well the calibration process is working.\n\nThe baseline seq2seq model (T5 w/o calibration) has been included in both Table 4 (last row) and Table 5 (all rows).\n- Results in Table 4 show that CASENT achieves significant improvements over baseline T5 in terms of both F1 score and calibration error (ECE, TCE)\n- Results in Table 5 show that CASENT achieves significant improvements over all T5 variants from T5-small to T5-3B.\n\n> The key ablation experiment in Table 4 seems flawed because the proposed calibration process uses more gold-labeled instances (the dev set), so the baseline T5 model should be trained on a joint set combining the training data and the dev data.\n\nThis is an interesting issue, and one which we also considered. Here is our reasoning for why we chose the experimental settings that we did:\n- Using dev sets for calibration and hyperparameter estimation is a standard practice in the ML literature. Calibration parameters function similarly as hyperparameters as they can only be estimated after the model has been trained.\n- Box4types (Onoe et al., 2021) follows the same setting. They trained BERT on the training set and did calibration on the dev set.\n- We followed the standard practice and used the dev set to estimate 27 calibration parameters and 1 threshold. We didn’t use the dev set to update any of the 770M T5 parameters. We ensure that all major results are based on the unseen test set.\n- All UFET models, including baseline T5, require the dev set for hyperparameter tuning and model selection (which involves at least tuning the threshold on the dev set). Therefore it is unreasonable to use dev set for training.\n\nTo address the concern raised by the reviewer that using dev set for calibration might account for the major improvement of CASENT over baseline T5, we would like to present the following two new results:\n- The dev set F1 score of CASENT is 51.5 (test F1: 51.3), which shows that CASENT does not overfit the dev set.\n- We trained baseline T5 on train+dev as suggested by the reviewer, showing that CASENT (without training on dev) still significantly outperforms it. Training CASENT on train+dev similarly yields (a marginal) improvement.\n| Method                      | Training data | UFET Test F1 |\n| --------------------------- | ------------- | ------------ |\n| Baseline T5 w/o calibration | train         | 47.3         |\n| Baseline T5 w/o calibration | train+dev     | 47.7         |\n| CASENT                      | train         | 51.3         |\n| CASENT                      | train+dev     | 51.5         |\n\n> In my view, the only contribution of this work is the calibration process that employs some tricks to address label prior biases from the occurring frequencies.\nWe would like to highlight the novelty of our model with the following points:\n- CASENT goes beyond heuristics for adjusting raw probabilities; rather, it is a principled calibration method that is generally applicable to any multi-label classification task (such as UFET). CASENT achieves not only SOTA task performance (measured by F1) but also SOTA calibration error (measured by ECE/TCE). This is also acknowledged by Reviwere rRPt.\n- Probability calibration is a core topic in ML studied by well-known papers (e.g., Guo et al., 2017) and requires non-trivial methods beyond post-hoc adjustments (see Sec. 2.2). To our knowledge, CASENT is the first calibration algorithm proposed for extreme multi-label classification tasks with thousands of classes like UFET. Results in Table 3 show that our proposed calibration algorithm reduces the expected calibration error (ECE) from 12.19 to 1.16.\n- CASENT is the first model that offers 1) generalization to unseen types, 2) efficient training & inference, 3) calibrated confidence scores (see lines 82-90, Sec.1). This fundamentally distinguishes CASENT from all previous approaches.\n\n|                      | Efficient training & inference | Generalization to unseen types | Calibrated confident scores |\n| -------------------- | ------------------------------ | ------------------------------ | --------------------------- |\n| Box4Types, MLMET, …  |Yes                              |                  No              |         No (except Box4types)                    |\n| LITE (previous SOTA) |  No                              | Yes                              |   No                          |\n| CASENT               | Yes                              | Yes                              |Yes                          |\n\n\n> ablation studies in Table 4 are unnecessary because they can be simply combined if T5 is used as a baseline in main experiments\n\nAblation studies in table 4 demonstrate the following two points (discussed in Sec. 6.1 - 6.2) that can be not be derived from the main results:\n- Various design choices of CASENT are reasonable and empirically useful.\n- CASENT archives state-of-the-art calibration error.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed system has good performances, and the calibration process seems effective on out-of-domain datasets.", "reasons_to_reject": "1) The experiment setting needs improvements. First of all, I think the authors should add the baseline seq2seq model (T5 in this case) to all experiments for us to understand better how well the calibration process is working. The key ablation experiment in Table 4 seems flawed because the proposed calibration process uses more gold-labeled instances (the dev set), so the baseline T5 model should be trained on a joint set combining the training data and the dev data. \n\n2) The contribution is limited as a long paper. In my view, the only contribution of this work is the calibration process that employs some tricks to address label prior biases from the occurring frequencies. Much content in the current version (e.g., Fig.1, Table 1, even the ablation studies in Table 4 because it can be simply combined if T5 is used as a baseline in main experiments) is unnecessary.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a calibration process on a seq2seq model trained for fine-grained entity typing. Specifically, after the model is trained to generate fine-grained types in the output sequence, the authors propose to calibrate the produced label confidences during inference time. The calibration process essentially learns a regression model using the annotated labels of the dev set that addresses some label prior biases. \n\nExperiments show that the proposed seq2seq entity typing model after the proposed calibration process outperforms baseline models, and the trained model can generalize to out-of-domain typing benchmarks with a few training examples.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JMv1Tk1NMg", "reviewer": "Reviewer_NPCS", "summary": "", "questions": "1. Is the \\mathcal{T} the types used in Choi’s work? \n2. In Figure 1, the input of calibration is person, criminal, adolescent, …, where is the adolescent after the calibration?\n3. Why after probability calibration, the sum of calibrated values are greater than 1?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  The definition of Ultra-fine Entity Typing in this paper is not exactly the same of the original definition. In the original definition of Ultra-Fine Entity Typing by Choi, et al. (2018), there is neither restriction about the number of types, nor restriction about the form. \n\n- We follow the standard evaluation protocol of UFET defined by Choi et al. 2018. \n- All UFET methods in Table 1, including Choi et. al 2018, use the same 10k type vocabulary (see Sec 2.1-2.2 of the Choi et al. paper). \n\nBelow we compare CASENT (our method) with prior methods  regarding the restriction on \\mathcal{T}:\n- Methods based on multi-label classifiers (e.g., Box4Types, MLMET, …) can only predict types in the dataset’s \\mathcal{T} \n- LITE and CASENT can generalize to types outside of the dataset’s \\mathcal{T}. This is illustrated in our out-of-domain experiments (Table 3).\n- CASENT can be easily extended to predict free-form types by replacing the constrained beam search with standard beam search. In contrast, LITE requires a predefined list of candidate types due to their formulation as an NLI task.  \n\n> Is the \\mathcal{T} the types used in Choi’s work?\n\nYes\n\n> In Figure 1, the input of calibration is person, criminal, adolescent, …, where is the adolescent after the calibration?\n\n“adolescent” still exists but is ranked lower (with a confidence score of ~0.1). Many types are omitted in Fig. 1 due to space limitations.\n\n>Why after probability calibration, the sum of calibrated values are greater than 1?\n\nUFET is a multi-label classification task where each entity can map to multiple types. Therefore, unlike the single-label multi-class classification, label probabilities don’t need to add up to 1. \n\n>The paper has not much novelty.\n\nWe would like to highlight the novelty of our model with the following points:\n- CASENT goes beyond heuristics for adjusting raw probabilities; rather, it is a principled calibration method that is generally applicable to any multi-label classification task (such as UFET). CASENT achieves not only SOTA task performance (measured by F1) but also SOTA calibration error (measured by ECE/TCE). This is also acknowledged by Reviwere rRPt.\n- Probability calibration is a core topic in ML studied by well-known papers (e.g., Guo et al., 2017) and requires non-trivial methods beyond post-hoc adjustments (see Sec. 2.2). To our knowledge, CASENT is the first calibration algorithm proposed for extreme multi-label classification tasks with thousands of classes like UFET. Results in Table 3 show that our proposed calibration algorithm reduces the expected calibration error (ECE) from 12.19 to 1.16.\n- CASENT is the first model that offers 1) generalization to unseen types, 2) efficient training & inference, 3) calibrated confidence scores (see lines 82-90, Sec.1). This fundamentally distinguishes CASENT from all previous approaches.\n\n|                      | Efficient training & inference | Generalization to unseen types | Calibrated confident scores |\n| -------------------- | ------------------------------ | ------------------------------ | --------------------------- |\n| Box4Types, MLMET, …  |Yes                              |                  No              |         No (except Box4types)                    |\n| LITE (previous SOTA) |  No                              | Yes                              |   No                          |\n| CASENT               | Yes                              | Yes                              |Yes                          |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment results are very good.", "reasons_to_reject": "The paper has not much novelty. The definition of Ultra-fine Entity Typing in this paper is not exactly the same of the original definition. In the original definition of Ultra-Fine Entity Typing by Choi, et al. (2018), there is neither restriction about the number of types, nor restriction about the form. In the task definition in Section 3.1, types are restricted to the set of \\mathcal{T}. Although it is the types used in Choi’s work, fixed type  structure is not what Ultra-Fine entity typing means.", "questions_to_authors": "1. Is the \\mathcal{T} the types used in Choi’s work? \n2. In Figure 1, the input of calibration is person, criminal, adolescent, …, where is the adolescent after the calibration?\n3. Why after probability calibration, the sum of calibrated values are greater than 1?", "paper_topic_and_main_contributions": "Authors claim a new seq2seq model, CASENT, for ultra-fine entitiy typing. The core contribution is a calibration method that transforms conditional log-probability into calibrated confidence value. Authors also show very positive experiment results.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line \t492: Often, … often… → one ‘often’ is enough."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a calibration method on a seq2seq model for fine-grained entity typing, which can give label confidence during inference time after the model is trained to generate fine-grained types. The main idea is to transform conditional log-probability into a calibrated confidence value, which shows promising results in experiments. Although some reviewers are concerned about the novelty and contribution of this paper, the paper is well-organized and clearly stated.
Decision: Accept-Findings