SpEL: Structured Prediction for Entity Linking
Authors: Hassan Shavarani
Anoop Sarkar
Venue: EMNLP 2023 Main (2023)
Keywords: Structured Prediction
Entity Linking
AIDA dataset
AIDA/testc
SpEL
TL;DR: SpEL is a structured prediction entity linking approach that uses new training and inference ideas obtaining a new state of the art on Wikipedia entity linking, with better compute efficiency and faster inference than previous methods.

Abstract: Entity linking is a prominent thread of research focused on structured data creation by linking spans of text to an ontology or knowledge source. We revisit the use of structured prediction for entity linking which classifies each individual input token as an entity, and aggregates the token predictions. Our system, called SpEL (Structured prediction for Entity Linking) is a state-of-the-art entity linking system that uses some new ideas to apply structured prediction to the task of entity linking including: two refined fine-tuning steps; a context sensitive prediction aggregation strategy; reduction of the size of the model's output vocabulary, and; we address a common problem in entity-linking systems where there is a training vs. inference tokenization mismatch. Our experiments show that we can outperform the state-of-the-art on the commonly used AIDA benchmark dataset for entity linking to Wikipedia. Our method is also very compute efficient in terms of number of parameters and speed of inference.

SPEL: Structured Prediction for Entity Linking
Hassan S. Shavarani
Anoop Sarkar
School of Computing Science Simon Fraser University
School of Computing Science
Simon Fraser University BC, Canada anoopesfu.ca ca
BC, Canada s sshavara@sfu.c ca
Abstract
prod:
pred:
pred:
pred:
Grice_ _elly_
O Mika (singer)
O
Entity linking is prominent thread of research focused on structured data creation by link- ontology or knowledge source. We revisit the use of structured predic- tion for entity linking which classifies each indi- vidual input token an entity, and aggregates the token predictions. Our system, called SPEL (Structured prediction for Entity Linking) is state-of-th the art entity linking system that uses some new ideas to apply structured prediction the task of entity linking including: two re- fined fine-tuning steps; context sensitive pre- diction aggregation strategy; reduction of the size of the model's oulput vocabulary, and; we address common problem in entity-linking systems where there a training vs. inference tokenization mismatch. Our experiments show that we can outperform the sate-off-the-ar on the commonly used AIDA benchmark dataset for entity linking to Wikipedia. Our method is also very compute efficient in terms of number of parameters and speed of inference.
ing spans of text
muslictiom
O 0.01 0.01
0.01
1.0
based
1.0
Grace_ Kelly_ (actress)
the highest
Grace (sonng(song)
0016 0.19
0.14
average
Malika Ayane
0.0
peobability
UK_ Singles_ Chart
Grace Kelly
0.0
0.0
(ika_[singer]
0.99 0.48
(actress) is not in the candidate
Mika Solo
0.01
0.52
FC Mika
0.0
0.0
United_ Kingdom
this phaase
[ingle_(music)
ignored"
classification head of fixed candidate set sizc
fine-tuned pre-trained language model
Gr
ace
Kelly
by
ika
reached
w1
w2
w3
w4
Figure 1: SPEL (Structured prediction for Entity Link- ing). In this example, we demonstrate top most proba- ble entities for each tokenized subword. In the first phrase, the most likely entity identified for Grace Kelly incorrect. We use candidate set for this mention to filter out irrelevant entity links with a high probability. In the second phrase, Mika is correctly narrowed down to the top 3 potential related entities. However, subword predictions do not consistently agree on the most probable prediction. In such cases, we cal- culate the average predicted probability for each entity across subwords and select the entity with the highest average probability.
C http://lttubb.oom//haaaaaaaatt
1 Introduction
Knowledge bases, such as Wikipedia and Yago (Pellissier Tanon et al., 2020), are valuable re- sources that facilitate structured information extrac- tion from textual data. Entity Linking (Shen et al., 2014) involves identifying text spans (mentions) and disambiguating the concept knowledge base entry to which the mention is linked.
Entity linking can be viewed as three interlinked tasks (Broscheit, 2019; Poerner al., 2020; van Hulst et al., 2020):
(3) Mention Disambiguation (Ratinov et al., 2011; Yamada et al., 2022) to select one of the poten- tial entity records for each detected mention.
(1) Mention Detection (Nadeau and Sekine, 2007) to scan the raw text and identify the potential spans that may contain entity links.
An end-to-end entity linking system does all three tasks and links text spans to concepts. The system can either have independently modelled compo- nents (Piccinno and Ferragina, 2014; van Hulst et al., 2020) or jointly modelled components (Kolit- sas ct al., 2018; De Cao et al., 2021a).
(2) Candidate Generation (e.g. Fang et al., 2020) to match each potential span with number of potential entity records in the knowledge base
Similar to almost all NLP tasks, recent en-

---

tity linking models use pre-trained representation learning methods that are based on Transformers (Vaswani al., 2017). These methods commonly utilize bidirectional language models like BERT (Devlin et al., 2019) or auto-regressive causal lan- guage models such as GPT (Brown et al., 2020) BART (Lewis et al., 2020), which are then fine- tuned specific entity linking training datasets. In number of such techniques, entity linking is framed as another well-studied problem, such as Aoquence-to-eqqunne translation (De Cao al., 2021b) or question answering (Zhang et al., 2022).
(5) Addressing the training/inference tokeniza- tion mismatch challenge in previous works which arises when differences in tokeniza- tion between training and testing phases lead to discrepancies. To address this challenge, we introduce an additional fine-tuning step where the model is fine-tuned on tokenized se- quences without explicit mention location in- formation. This encourages the model learn robust representations that are not dependent on specific tokenization patterns, improving its generalization to the mention -agnosii tok- enization at inference.
Entity linking can be viewed as sequence tagging using structured prediction' aiming to assign one of the finitely many classes to every input utterance. This approach involves using a pre-trained model to encode each input subword token into a multi- layer context-aware dense vector representation. classifier head calibrates each token representation to predict the entity for each subword token. Due to the large number of possible entities and issues with consistency of entity prediction across multi- ple subword tokens, structured prediction for entity linking (surprisingly) has not been studied in-depth. Our contributions this paper are as follows:
(6) New pre-training and inference ideas that can achieve new state of the with much better compute efficiency (fewer parameters) and much faster inference speed than previous methods.
(7) We have annotated and released AIDA/testo, a new entity linking test set for the AIDA dataset.
(8) Simplify and speed up the evaluation process of entity linking systems using the GERBIL. platform (Röder et al., 2018) by providing a Python equivalent of its required Java middle- ware.
(1) A new structured prediction framework for entity linking called SPEL (Figure 1). We demonstrate that SPEL establishes new state-of-the-an for Wikipedia entity linking on the commonly used AIDA (Hoffart et al., 201 I) dataset.
We introduce recent entity linking methods in Section 2; explain our approach structured pre- diction for entity linking and our context sensitive prediction aggregation strategy in Section 3 with comprehensive experiments Section 4.
(2) Two separate and refined fine-tuning steps. One for general knowledge about Wikipedia concepts and one for specifically tuning the AIDA dataset.
Related Work
Entity linking can be framed as another well- studied task and the best solution for that task is applied. Autoregressive encoder-decoder sequence- to-sequence translation is such approach. De Cao al. (2021b) consider the input text the source for translation and the text is annotated with Wikimedia markup containing the mention spans and the entity for each mention. Instead mapping the entity identifiers into a single id (this is the default in many techniques including this work because the approach can be easily ported to ontologies other than Wikipedia), their model generates the entity label in token-by-token ba- sis (it generates the Wikipedia URL one subword at time). The generation process follows a con- strained decoding schema that prevents the model from producing invalid entity URLs while limiting
(3) A context sensitive prediction aggregation strategy that enables subword-level token classification while enforcing word-level and span-level prediction coherence. This does not incur additional inference time but dras- tically enhances the quality of the predicted annotation spans.
(4) We use the in-domain mention vocabulary to create fixed candidate set. We this to improve the efficiency and accuracy of entity linking.
Structured prediction has shown successful enhancements in various NLP tasks, including Dependency Parsing (Zhou et al., 2015), Question Answering (Segal et al., 2020), and Machine Translation (Shavarani and Sarkar, 2021).

---

the generated output to the entities in a predefined candidate set (discussed in Section 3.1).
and (4) optimizing the implementation of the struc- tured prediction model. We obtain more than 6. 6.1% points Micro-F1 improvement as well as more than 10 times reduction in required disk space, and close to 4 times reduction in the required GPU mem- ory (in base case) compared to Broscheit (2019). Our approach is also faster at inference time than previous Transformer-basee methods.
De Cao al. (2021a) use a BERT-style bidirec- tional model fine-tuned to identify potential spans (mention detection) by learning spans using a be- gin probability and an end probability for each subword in the input text. For each potential span, they use a generative LSTM-based (Hochreiter and Schmidhuber, 1997) language model to generate the entity identifiers (token-by-token) and limit the generation process to pre-defined candidate sets.
A number of recent techniques focus on en- hancing the entity linking knowledge in BERT (or one of its variations) and utilize one or more such knowledge-en enhanced models to perform the task of entity linking. Peters et al. (2019) inject Wikipedia and Wordnet information into the last few layers of BERT, Poerner et al. (2020) inject pre-trained Wikipedia2Vec (Yamada et al., 2020a) entity em- beddings into the input layer of the language model while freezing the rest of its parameters, and Mar- tins et al. (2019) leverage Stack-LSTM (Dyer et al., 2015) Named Entity Recognition model enhance entity linking performance using multi- task learning to improve entity linking. These ap- proaches are Wikipedia- centrrc and while we also experiment on Wikipedia exclusively, our approach can be used on any entity-linking dataset that has entities from other ontologies such as the MedMen- tions dataset (Mohan and Li, 2019) which links to concepts in UMLS ontology (Bodenreider, 2004).
Mrini et al. (2022) frame entity linking as sequence-to-sqquenee translation task using BART (Lewis et al., 2020). They duplicate the BART de- coder three times to fine-tune the model multi- task setting. The two additional decoder modules are trained using auxiliary objectives of mention de- tection and re-ranking. While this training method increases the model size during training, they mit- igate increased model size and speed at inference time by excluding the auxiliary decoder modules and employing sampling and re-ranking techniques on the generated target sequences.
Zhang et al. (2022) use Question Answering as way frame the entity linking task. They suggest a two-step entity linking model in which they use a fine-tuned Transformer-based BLINK (Wu et al., 2020) model to find all the potential entity records that might exist in the text and then utilize a fine- tuned question answering ELECTRA (Clark et al., 2020) model to identify the matching occurrences of the potential entities discovered in the first step. This approach obtains high accuracy; however, it is very resource intensive and inference is slow.
Kolitsas et al. (2018) jointly model mention detection and mention disambiguation using an LSTM-based architecture while reusing the candi- date sets created by Ganea and Hofmann (2017) as a replacement for the candidate generation step. and Kannan Ravi et al. (2021) follow a similar framework while modeling each of mention detec- tion and mention disambiguation using separate BERT models. Feng et al. (2022) compute en- tity embeddings (instead of using pre-trained ones) using the average of the subword embeddings of the candidates and compare them to the average of the subword embeddings for the potential span (training a Siamese network; Bromley et al., 1993). Févry et al. (2020) investigate pre-training strate- gies specifically tailored for Transformer models to perform entity linking, diverging from the conven- tional use of pre-trained BERT models. And, van Hulst al. (2020) propose modular configuration that composes mention detection, candidate gen- eration, and mention disambiguation in pipeline approach, incorporating the most promising com- ponents from prior research.
Structured prediction (subword token multi-label classification) is the other well-studied problem. Broscheit (2019) proposes very simple entity link- ing model which places a classification head on top of BERT language model and directly classifies each subword representation using a softmax over all the entities known to the model.
Our work also uses structured prediction because it is one of the most lightweight techniques in terms of compute cost and inference speed. We extend the structured prediction model in this work by (1) utilizing context sensitive prediction aggrega- tion strategy to form meaningful span annotations (Section 3.2), (2) addressing a training/inference tokenization mismatch issue (Section 3.3), (3) ex- amining the role of different types of candidate sets (Section 3.1) in curating the predicted results,

---

In Section 4.4, we conduct experiments to com- pare SPEL the methods discussed in this section,
An alternative is use mention-specific candi- date sets (Kolitsas et al., 2018; Peters al., 2019; Kannan Ravi et al., 2021; De Cao et al., 2021b,a), Mention-specijpn candidate sets can be divided into two groups:
3 Entity Linking
Formally, entity linking receives a passage (p) con- taining words {w1.... .... wa and produces a list con- taining l span annotations. Each span annotation is expected to be triplet of the form (span start, span end, entity identifier). The span start and span end values are expected to be character posi- tions on the raw input p. and the entity identifier values are selected from a predefined vocabulary of entities (c.g. there would be approximately million entities to choose from when entity linking to Wikipedia). The massive entity vocabulary size increases the model's hardware requirements and in some cases renders the task intractable.
context-agnostic mention -prrtifo--peciiicc sets which are usually generated over large amounts of annotated text and try to model the prob- ability of each mention span all possi- ble entity identifiers without assuming spe- cific context in which the mention would appear. KB+Yago (Ganea and Hofmann. 2017) contains candidate lists for approxi- mately 200K mentions created over the entire English Wikipedia combined with the Yago dictionary of (Hoffart et 201
(2) context-aware mention-speciffe sets can be constructed if there is a method for identi- fying mentions and set of candidates for those mentions. For example, Pershina et al. (2015) have built such candidate sets, called PPRforNED*. Such lists have been primarily suggested for the task of entity disambigua- tion where the mention is provided in the in- put. As gold mentions are not available for real-world use cases of entity linking, this type of candidate sets have fallen out of favor.
3.1 Candidate Sets
To solve the entity vocabulary size problem, a com- mon approach is to limit to K most frequent enti- ties in the knowledge base?. This vocabulary can be simply considered as the fixed candidate set for linking each mention the knowledge base. Where no more information is available, the model will have to choose one entity from this fixed can- didate set.
The selected fixed candidate set may lack many of the expected entity annotations at inference. Consequently, even if the model is highly capa- ble, it may perform poorly during inference due to its inability to suggest the expected entities. Recog- nizing this challenge, there a consensus among existing literature (Kolitsas et al., 2018; Broscheit, 2019; Peters et al., 2019; Poerner et al., 2020) to augment the fixed candidate set by including the expected entities necessary for inference. We adopt a similar approach to create the fixed candidate set, following the same line of reasoning previous studies. Nonetheless, it's important underscore that our adherence to this approach is driven by the desire for consistency with prior research; our framework, however, does not necessitate this spe- cific method for effective functioning. In practical scenarios, one straightforward approach to con- struct the fixed candidate set is to base it on the anticipated entities (any subset of knowledge base entities that pertinent to the task hand) to be detected by SPEL.
Mention-specific candidate sets consist of many entity identifiers and the correct entity identifier is not guaranteed exist given the mention span.
3.2 Structured Prediction for Entity Linking
For a sequence of subwords $ = {81, (2, $2,,.,, sn}", } we employ RoBERTa (Liu et al., 2019), in both base and large sizes, as our underlying model M to encode S into H E Bnxd where d is the hidden representation dimension of M. Each repre- sentation h; C H, E 1,..., is then transformed into distribution over the fixed candidate set of size KB using a transformation matrix W E RR This results in P; = h;W, where P; represents the probability distribution for the 7th subword the input sequence.
When we set up fine-tuning for this task, we use hard negative mining (Gillick al., 2019) find the most probable incorrect predictions in the
httpps://gtthubb httpp://gituub..ooyiiiiinggggei wow/yifding/deeppp..cmmm ed. ed_PyTorch "hups://gituub. com/masha-pplee P/PRoooNDDP
Forr Wikipedia, we can define an entity frequeney as the number of times a title is hyperlinked in the other pages.
*When feeding long text in training and inference, we split the text into smaller overlapping chunks.

---

batch In each fine-tuning step. we update the network based on the subword classification proba- bilities of the hard negative examples as well as the expected prediction. To increase inference speed, the classification head does not normalize the pre- dicted scores.
in the phrase annotations, excluding the ones an- notated with (which means the phrase not an entity). As an additional post-processing cleanup step, we reject phrase annotations that span over a single punctuation subword (e.g. a single period or comma) or a single function (sub)word (c.g, and, by, ...). In such cases, we manually override the model's prediction to O.
We employ binary cross-entropy with logits, Equation (1), as our loss function, which is calcu- lated over many factors. Let N represent the total number of selected examples (1)) comprising the one positive example corresponding to the expected prediction and the negative examples. Additionally, oi takes a value of I when the j"h member of v correctly points to the entity identifier for subword Si. Pij denotes model's predicted score for link- ing the j"h member of the selected examples to the 7t subword, and a is the sigmoid function, which maps the scores to [0, 1].
This context sensitive prediction aggregation strategy leads to improved performance and en- hances prediction results in inference. Our strategy ensures that annotation spans do not begin or end inside a word", and the conflicts between the sub- word predictions within word are resolved by the average prediction probability for each entity identifier.
This simpler method to ensure label consistency does better than using a CRF layer (Lafferty et al., 2001). Although our experiments show that a CRF layer does not improve our model, our readers can think of the suggested strategy domain-tailored, non-parametric and rule-driven version of a CRF layer which guides the model unify the predicted subword-level entity predictions considering the local context. Based on our experiments (Table 2), although we do not explicitly model Mention Detection (as predicting the span start and span end probability scores or separate BIC tags) for each subword in inference time, we observe high in-domain accuracy in distinguishing O spans from non-O spans in predictions result of using the context sensitive prediction aggregation strategy.
N C. = N
ans log ((ppss)
(1)
1 ((oop))))
+ (1 -(g) ) log
During inference we collect the top predictions for each subword i based on the predicted prohabil- ities P.. We then collect subwords that belong to the same word into a single group, which we call the word annotation. For each word annotation, we generate an aggregated entity identifier predic- tion set by taking the union of the entity identifiers predicted for the subwords. We then compute the weighted average of the prediction probabilities for each entity identifier to obtain the word-level probability score over entities.
Our modelling framework, SPEL, stands for Structured Prediction for Entity Linking.
3.3 Fine-tuning Procedure
Consecutive word-level entity labels when they refer to the same concept are joined into single mention span over that phrase.
Heinzerling and Inui (2021) argue that pre-trained language models can produce better representations when they are first fine-tuned on much larger entity-linked training data (almost like further pre- training step) and then subsequently fine-tuned for the litity-linking task. We perform such multi- step fine-tuning procedure: first fine-tuning on a large dataset encompassing general knowledge on the set of linked concepts and then fine-tuning on an in-domain dataset specific to the target domain over which we aim to perform entity linking.
When mention-specifi candidate set is avail- able, and the mention surface form matches one of the mentions in the candidate set, we filter out any predictions from the phrase annotation that are not present in the candidate set, regardless of their probability?. 7, The final prediction for an entity span is generated based on the most probable prediction
"We add random negative examples in addition to hard neg- atives to make sure we get to 5K negative examples for each batch when fine-tuning on CoNLL/AIDA and 10K negatives for general fine-tuning.
*For instance, in the word U.S., if in the U. S part, the subwords have high likelihood for the concept The United
States and the ending refers to an O, the conflict is resolved so that the entire word U.S. is linked to The United States.
'The presence of a ccention-specigee candidate set is not a prerequisite for this technique to be effective.

---

General knowledge fine-tuning. In the first step, we fine-tune the pre-trained language model using text that includes links to the knowledge base (in our experiments, we use a large subset of En- glish Wikipedia"). As mentioned in (Broscheit, 2019). it helps if this data is aware of the mentions (using special space character subwords before and after each span that is linked to an entity). This helps the model learn the identification of the start- ing and ending subwords in entity mention spans, However, this imposes mismatch in the distri- butions of the data fine-tuning compared to in- ference, where the model does not have access to the entity mentions to perform the customized to- kenization. To address this issue, as subsequent fine-tuning step, we iterate again through the large matity-linked dataset which is re- te-tokenized without the knowledge of the mention spans.
20 subword overlap between consecutive chunks, After the split, our dataset contains 3,055,221 train- ing instances with 1000 instances for validation. We also create a mention-agnoste re-tokenized ver- sion of this dataset with the same exact mentions to perform the second step of general knowledge fine-tuning as explained above.
AIDA (Hoffart et al., 2011) contains manual Wikipedia annotations for the 1393 Reuters news stories originally published for the CoNLL-2003 Named Entity Recognition Shared Task (Tjong Kim Sang and De Meulder, 2003). Its train, testa, and testb splits contain 946, 216, and 231 documents, respectively. It has a fixed can- didate set size of 5600 (including O tag) and for evaluation the AIDA test sets, we shrink the classification head in the model these 5600 can- didates and disregard the rest of the 500K candi- dates
Domain specific fine-tuning. In the third and last fine-tuning step, we refocus the model's 's at- tention to the in-domain dataset annotated with a fixed candidate set which usually is a subset of all the knowledge base entities that the model has observed in the previous two fine-tuning steps. Sim- ilar to the second fine-tuning step, we tokenize the in-domain dataset without the knowledge of the mention spans.
4.2 Evaluation using GERBIL
The GERBIL platform (Röder et al., 2018) an evaluation toolkit (publicly available online) that eliminates any mistakes and allows for fair com- parison between methods. However, GERBIL is Java toolkit, while most of modern entity linking work is done in Python. GERBIL developers rec- ommend using ArpotrrapNiivo Test (a mid- dleware tool written in Java) to connect Python entity linkers to GERBIL. Because of the complex- ity of this setup, we have not been able to directly evaluate some of the carlier publications due to software version mismatches and communication errors between Python and Java. This is a drawback that discourages researchers from using GERBIL. To address this issue, we provide Python equiv- alent of SpotWrapNi FWS Test encourage entity linking researchers to use GERBIL for fair repeatable comparisons. We evaluate all SPEL models using GERBIL in the A2KB experiment type, and report InKB strong annotation matching scores for entity linking. Only five of the publica- tions to which we compare use GERBIL, however, all report InKB strong Micro-F1 scores allowing direct comparison our work.
4 Experiments
4.1 Data
For our experiments, we focus on Wikipedia as the knowledge base and we use the following datasets for the fine-tuning steps mentioned in Section 3.3. Wikipedia we use the 20230820 dump of Wikipedia (with approximately 238K documents), and we the script from (Broscheit, 2019) to handle incomplete annotations, perform mention- aware customized tokenization, and compute the average probability of linking to no entity (called the Nil probability) for the 1000 most frequent en- tities. The Nil probability is used to modify the Wikipedia training data annotations in such way that the chance of linking surface form referring to frequent entity O is almost O. We construct the Wikipedia fixed candidate set using the union of the 500K most frequent mentions in the Wikipedia dump and the fixed candidate set of AIDA and the test datasets. We split the content of Wikipedia pages into chunks consisting of 254 subwords with
4.3 Experimental Setup
For the first general knowledge fine-tuning step (Section 3.3), as a warm-up to full fine-tuning, we
Another implementation idea can revolve around mul- tiplying the predicted output vector into a mask vector that masks all the candidates not in the expected 5600 entities.
"I Limited to the articles that contain some presence of the entities in our selected fixed candidate set.

---

freeze the entire ROBERTA model and only mod- ify the classification head parameters on top of the encoder. We fine-tune with this configuration for epochs and subsequently continue with fine-tuning all model parameters. We stop the fine-tuning pro- cess in this phase when the subword-level entity linking Fl score on the validation set shows no improvement for 2 consecutive epochs. Following this, we proceed to the second phase of full fine- tuning, where we fine-tune all model parameters using the mention-agnostie re-tokenized Wikipedia fine-tuning data. Just like phase one, we stop this phase based on the same criteria. We implement SPEL using pyt orch, utilize Adam optimizer with a learning rate of 5e 5 to fine-tune the encoder parameters, and use SparseAdam optimizer with learning rate of 0.01 to fine-tune the classifica- tion head. We run fine-tuning phases one and two on the large subset of Wikipedia using two Nvidia Titan RTX GPUs.
rations: (1) utilizing the YB+Yago (Ganea and Hofmann, 2017) context-agnostic candidate set, (2) employing the PPRforNED (Pershina et al., 2015) context-aware candidate set, and (3) adapting PPRforNED to aggregate the candidate informa- tion for each mention surface form, resulting in a context-agnostic candidate set, excluding context- specific information.
Candidate sets help reject many over-generate spans. If mention's 's candidate set empty, the mention span is excluded from further consider- ation. While this approach typically leads to im- proved precision and subsequent enhancement in FI score, instances may arise where the model cor- rectly predicts mentions that are not encompassed within the candidate sets. This can lead to lower recall in the evaluation. The observed Micro-F1 score drop when employing KB+Yago candidate sets compared to the scenario where no mention- specific candidate set is utilized, can be attributed to these cases.
For the last phase of line-tuming on the AIDA dataset (Section 3.3), we freeze the first four layers of the encoder (including the embedding layer) as well as the shrunk classification head parameters, and we fine-tune the rest of the model parame- ters for 30 epochs (over the train part of AIDA dataset). We run this step using one Nvidia 1060 with 6 GBs of GPU memory, and accumulate gradi- ents (Ott et al., 2018) for 4 batches before updating model parameters. We perform redirect normaliza- tion!! 11 on the final predicted spans.
SPEL- using context-aware candidate sets achieves the highest boost, reporting 2.1 and 2.3 Micro-F1 scores improvement over testa and testb sets of AIDA, respectively, and establishes a new state-of-the-art for AIDA dataset. It is note- worthy to consider that the proposed model by Zhang et al. (2022) demands significant compu- tational resources, including tens of gigabytes of RAM and over 7 and 2.7 times the number of parameters on GPU compared to SpEL-base and SPEL-large, respectively. Despite these resource- intensive requirements, SPEL outperforms Zhang et al. (2022). The comparison between our results and that of De Cao et al. (202 demonstrates that generating entity descriptions (which can share lexical information with the mention text) is not necessary even for high accuracy Wikipedia entity linking. Our approach can be easily extended to on- tologies without textual concept deseriptions, while methods that generate entity descriptions cannot.
4.4 Experiments and Results
In this section, we conduct experiments to eval- uate the performance of both SpEL-base and SPEL-large (referring to the size of the underlying ROBERTA model) in different configurations con- cerning the use of candidate sets (Section 3.1) and report our experimental results over the AIDA test datasets in Table 1.
In the first configuration, we examine our model without any mention-specific candidate sets. Our results show a minimum of 5.3 Micro-Fl score improvement in AIDA test sets compared to (Broscheit, 2019) while significantly reducing the required parameter size on GPU by fourfold, re- sulting in a 7.2 times increase in inference speed in base case.
Lastly, in Table 2, we compare SpEL-base, which utilizes the context sensitive prediction ag- gregation strategy to convert subword- pre- dicted entity identifiers into span-level predictions, to the model proposed by De Cao al. (2021a),
The latter model explicitly models the start and end positions of the spans for mention detection. We employ the evaluation script released by De Cao et al. (2021a) for our assessment. The results confirm that, despite not using BIO tags or ex-
Next, we run SPEL in three other configu-
1l See Appendix A for more on this standard normalization technique.

---

EL Micro-Fl
#params on GPU
speed sec/doc
Approach
testa
testb
Hoffart et al. (2011)
(Linear)
72.4
72.8
Kolitsas (2018)
(LSTM)
89.4
82.4
330.7M
0.097
Broscheit (2019)
(BERT)
86.0
79.3
495.1M
0.613
Peters et al. (2019)
(BERT)
82.1
73.7
Martins et al. (2019)
(Stack-LSTM)
van Hulst et al. (2020)
(LSTM)
83.3
82.4
19.0M
0.337
Févry et al. (2020)
(Transformer)
79.7
76.7
Poerner et al. (2020)
(BERT)
90.8
85.0
131.1M
Kannan Ravi
(2021)
(BERT)
83.1
De Cao et al. (2021b)
(BART)
83.7
406.3M
40.969
De Cao et al. (2021a)
(RoBERTa+LLSTM)
(no mention-speeneeeeeeee candidate set) (using PPRforNED candidate set)
61.9
49.4
124.8M
0.268
90.1
85.5
124.8M
0.194
(train) 811.5M
Mrini et al. (2022)
(BART)
85.7
(test)
406.2M
Zhang (2022)
(BLINK+FLECCRA
85.8
1004.3M
Feng et al. (2022)
(BERT)
86.3
157.3M
SpEL-basc (no mention-specific candidate set)
91.3
85.5
128.91
0.084
SPEL-base (aB+Yago candidate set)
90.6
85.7
128.9M
0.158
SPEL-base (PPRRorNEDD candidate set)
context-agnostic
91.7
86.8
128.98 9M
0.156
context-aware
92.7
88.1
128.9M
0.156
SPEL-large (no mention mention-speccifc candidate set)
91.6
85.8
361.1M
0,273
SPEL-large (KB+Yago candidate set)
90.8
85.7
361.1M
0.267
SPEL-large (PPRforNED candidate set)
context-agnostic context-aware
92.0
87.3
361.1M
0.268
92.9
88.6
361.1M
0.267
Table 1: Entity Linking evaluation results SPEL compared to that of the literature over AIDA test sets. #params GPU only considers the total number of parameters that will directly effect the cost of GPU acquisition and does not reflect upon the total amount of data loaded into/from main memory and disk.
plicitly modeling span boundaries, SPEL demon- strates strong performance in mention detection, with high level of accuracy. Its near-perfect pre- cision scores indicate its ability minimize generated predictions, contributing its state-of- the-art entity linking performance.
into few-shot in-context learning on GPT-4 is likely to improve these results since LLMs have extensive knowledge about entities but cannot directly reason about specific Wikipedia URL
4.6 Practicality of the Fixed Candidate Sets A valid concern regarding SPEL pertains the construction of the fixed candidate set and its prac- ticality in real-world scenarios, where the testing data may not be predetermined, making chal- lenging when attempting to assemble a subset of knowledge base entries for this purpose. As men- tioned in Section 3.1, it is possible to construct this set based the expected entities that SPEL should detect. In this section, we take more flexible ap- proach, and consider the entire set of 500K general fine-tuning entities as the fixed candidate set.
4.5 Comparison OpenAl GPT
Large generative language models are effective zero shot and few shot learners (Brown al., 2020) at many NLP tasks. We evaluate GPT-3.5 and GPT-4 for the task of entity linking using various prompts. We frame the problem for the generative LM as in De Cao et al. (2021b), where it produces markup around the mentions (described in detail in Appendix B). Table 3 compares the GPT eval- uation results that of SPEL. For fair compar- ison, we consider the evaluation results without any mentionnspepciie candidate sets. Currently the results are much worse than the state-of tftte-of-theann art and using GPT-4 is more expensive. Further research
Furthermore, taking inspiration from Liu and Rit- ter (2023) regarding the extended existence of the CoNLL-2003 dataset, and consequently the AIDA
12eec (Cho et al., 2022) for some ways to address this issue.

---

MD Micro Scores
Approach
testa
testb
P
R
F1
P
R
F1
De Cao et al. (2021a)
93.9
96.7
95.2
92.2
94.8
93.5
(using PPRforNED c. set)
SPEL-base (no mention-specific set)
94.6
94.4
94.5
92.5
90.1
91.2
SpEL-hase (using PPRforNEED c. set comlext-agnostic)
98.3
91.6
94.8
98.3
86.4
92.0
SPEL-base (using PPRforNTED c. set context-aware)
99.4
90.9
95.0
99.4
84.9
91.6
Table 2: Mention Detection evaluation results of SPEL in comparison the work of De Cao al. (2021a) using
their released evaluation code (from utils.py). As De Cao et al. (2021a) use PPRforNED candidate sets, we only compare the SPEL results calculated using these candidate sets.
EL Micro-F1
US$ for
EL Micro-F1
Approach
Approach
testa
testb
1000 docs
testa
testb testc
GPT-3.5 (zero-shot)
47.3
52.9
4.22
no mention-specific c. set
89.6
82.3
73.7
SPEL -base
GPT-4.0
KB+Yago set PPRforNED set
89.5
83.2
57.2
(zero-shot)
40.4
54.1
42.17
shot w/ CoT)
62.4
66.2
59.37
context-agnostic context-aware
90.8
84.7
45,9
SPEL-base
91.3
85.5
2.28
91.8
86.
SpEL-large
91.6
85,8
2.64
SPEL- Jarge
no mention-specific c. set
89.7
82.2
77.5
KB+Yago set PPRforNED c. set
89.8
82.8
59.4
Table 3: Comparison of the performance of SPEL (in no mention- mention-speciffi candidate set setting) to zero and few shot GPT-3.5-turbo- (accessed on June 16, 2023) and GPT-4-0613 (for the best performing prompts we attempted; accessed August 24, 2023). For few-shot experiments we constructed the prompt using the chain- of-thought (CoT) method of Wei al. (2022).
context-agnostic
91.5
85.2
46.9
context-aware
92.0
86.3
Table 4: Entity Linking evaluation results of SPEL with fixed candidate set size of 500K over AIDA test sets. Since the context-aware candidate sets require a mecha- nism for generating/looking up the candidate set during inference, we do not evaluate testi in this setting.
dataset, for over two decades acknowledge the potential concern of adaptive overfitting. In re- sponse this concern, we used their newly anno- tated NER test set of 131 Reuters news articles pub- lished between December 5th and 7th, 2020. We meticulously linked the named entity mentions in this test set to their corresponding Wikipedia pages, using the same linking procedure employed in the original AIDA dataset. Our new entity linking test set, A A testc, has 1,160 unique Wikipedia identifiers, spanning over 3,777 mentions and en- compassing total of 46,456 words.
Conclusion
We introduced several improvements to structured prediction approach for entity linking. Our experi- mental results on the AIDA dataset show that our proposed improvements the structured predic- tion model for entity linking can achieve state-of- the-art results using a commonly used evaluation toolkit providing head to head numbers for com- peting methods on the same dataset. We show that our approach has the best Fl-score on this task compared to the state-of-the-art on this dataset. We more compute efficient with many fewer pa- rameters and our approach is also much faster at inference time, providing faster throughput, com- pared to previous methods.
We re-evaluate SPEL across all four settings outlined in Section 4.4 using the 500K entity out- put vocabulary and over three AIDA test sets: testa, test,, and testc. We report our find- ings in Table 4. Examining the results shows that our newly created testc presents new challenge for entity linking because the currently available candidate sets prove unhelpful and, in fact, detri- mental entity linking. The SPEL-large results for testa and testb show that SPEL with an unconstrained fixed candidate set size still matches the performance of the best model published before SPEL (with fixed candidate sets).
In future work, we plan to extend our work to other entity linking datasets in other domains such as biomedical research. We also plan to research multilingual applications of structured prediction for entity linking including benefits to projection of entity linking concepts from one language another and using multilingual representation learn- ing for our base model.

---

Limitations and Ethical Considerations
Advances neural information processing systems, 6.
Like other deep learning-based entity linking sys- tems, SPEL relies on predefined fixed candi- date set. This implies that the user needs prior knowledge about the entities which they seek (c.g.. Barack_Obane and must include those entities in the model's output mention vocabulary. It is important to note that SPEL is unable to detect entities that are not included in its defined fixed candidate ser. Expanding SPEL to support zero- shot entity linking is area that we leave for fu- ture exploration and development. Our research is on English only, and we acknowledge that en- tity linking for other languages also relevant and important. We hope to extend our work to cover multiple languages in the future. We inherit the biases that exist in our training data and we do not explicitly de-bias the data. It is possible that certain types of entities are under-represented in this dataset, so care must be taken before using our model for general purpose entity-linking that it is re-trained on a suitably biased training dataset that compensates for the fact that some under repre- sented entities might be infrequent or missing from our training data. We are providing our models and code to the research community and we trust that those who use the model will do so ethically and responsibly.
Samuel Broscheit. 2019. Investigating entity knowl- edge in BERT with simple neural end-to-end entity linking. In Proceedings of the 23rd Conference Computational Natural Language Learning (CoNLL). pages 677-685, Hong Kong, China, Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffreey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020, Language models few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Young Min Cho, Li Zhang, and Chris -allch.n-Burch. 2022. Unsupervised entity linking with guided sum- marization and multiple-choice selection. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9394- 9401, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kevin Clark, Minh-Thame Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders discriminators rather than generators. In International Conference on Learning Representa- tions.
Acknowledgements
Silviu Cucerzan. 2007. Large-scale named entity disam- biguation based Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-Come CoNLL., pages 708-716, Prague, Czech Republic, Association for Computational Linguistics.
We would like to thank the anonymous review- ers for their helpful comments. The research was partially supported by the Natural Sciences and Engineering Research Council of Canada grants NSERC RGPIN-2018-06437 and RGPAS-2018- 522574.
Nicola De Cao, Wilker Aziz, and Tvan Titov. 202 la Highly parallel autoregressive entity linking with dis- criminative correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7662-7669, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
References
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.
Dbpedia: A nucleus for a web of open data, In international semantic web conference, pages 722-
Nicola De Cao, Gautier Izacard. Sebastian Riedel, and Fabio Petroni. 2021b. Autoregressive entity retrieval. In International Conference on Learning Representa- rions.
735. Springer.
Olivier Bodenreider. 2004. The unified medical lan- guage system (umls): integrating biomedical termi- nology. Nucleic acids research, D):D277- D270.
Leon Derezynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Management. 51(2)322-49
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature verifi- cation using a" siamese" time delay neural network.

---

Jacob Devlin, Ming-Wo Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012. Kore: keyphrase overlap relatedness for entity disambigua- tion. In Proceedings of the s1 ACM international conference on Information and knowledge manage- ment, pages 545-554.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. Proceedings of the 2011 Conference on Em- pirical Methods in Natural Language Processing, pages 292-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguistics.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition- based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 334-343, Beijing, China. Association for Com- putational Linguistics.
Manoj Prabhakar Kannan Ravi, Kuldeep Singh, Isa- iah Onando Mulang', Sacedeh Shekarpour, Johannes Hoffart, and Jens Lehmann. 2021. CHOLAN: A modular approach for neural entity linking on Wikipedia and Wikidata. In Proceedings of the 16th Conference of the European Chapter of the Associ- ationfor Computational Linguistics: Main Volume, pages 504-514, Online. Association for Computa- tional Linguistics.
Zheng Fang, Yanan Cao, Ren Li, Zhenyu Zhang, Yan- bing Liu, and Shi Wang. 2020. High quality candi- date generation and sequential graph attention net- work for entity linking. In P Proceedings of The Web Conference 2020. pages 640-650.
Yukun Feng, Amir Fayazi, Abhinav Rastogi, and Man- abu Okumura. 2022. Efficient entity embedding con- struction from type knowledge for BERT In Find- ings of the Association for Computational Linguis- tics: AACL-IJCNLP 2022, pages 1-10, Online only. Association for Computational Linguistics.
Nikolaos Kolitsas, Octavian- -uggnn Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking, In Proceedings of the 22nd Conference on Computa- tional Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares, and Tom Kwiatkowski. 2020. Empirical eval- uation of pretraining strategies for supervised entity linking. In Automated Knowledge Base Construc- tion.
John D. Lafferty, Andrew McCallum, and Fermando Pereira. 2001. Conditional random fields: Probabilis- tic models for segmenting and labeling sequence data. In International Conference on Machine Learning.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-scqueee for natural language generation, translation. and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 Online. Association for Computa- tional Linguistics.
Octavian-Eugee Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619-2629, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
pre-training
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessan- dro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia- Olanoo 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Con- ference Computational Natural Language Learn- ing (CoNLL), pages 528-537, 537, Hong Kong, China. Association for Computational Linguistics.
Shuheng Liu and Alan Ritter. 2023, Do CONLL-2003 named entity taggers still work well in 2023? In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 8254-82211 Toronto, Canada. Association for Computational Linguisties.
Benjamin Heinzerling and Kentaro Inui. 2021. Lan- guage models knowledge bases: On entity repre- sentations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1772-1791, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/ 11692.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2019. Joint learning of named entity recog- nition and entity linking. In Proceedings of the 57th Annual Meeting of the Association for Computational
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 18): 1780.

---

Linguistics: Student Research Workshop, pages 190- 196, Florence, Italy, Association for Computational Linguistics.
Nina Poerner. Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient- eeee----ee--eeeeee effective entity embeddings for BERT. In Findings of the Association for Compu- rational Linguistics: EMNLP 2020, pages 803-818, Online. Association for Computational Linguistics.
Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls con- cepts. In Automated Knowledge Base Construction (AKBC), Amherst, Massachusetts, USA.
Lev Ratinov, Dan Roth, Doug Downey, and Mike An- derson. 2011. Local and global algorithms for disam- biguation to Wikipedia. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1375-1384, Portland, Oregon, USA. Association for Computational Linguistics.
Khalil Mrini, Shaoliang Nie, Jiatao Gu, Sinong Wang.
Maziar Sanjabi, and Hamed Firooz. 2022. Detec- tion, disambiguation, re-ranking: Autoregressive en- tity linking as a multi-task problem. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1972-1983, Dublin, Ireland. Association for Computational Linguistics.
Michael Röder, Ricardo Usbeck, Sebastian Hellmann, Daniel Gerber, and Andreas Both. 2014. N col- lection of datasets for named entity recognition and disambiguation in the nlp interchange format. In LREC, pages 3529-3533.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvis- ticae Invextigationes, 30(1):3-26.
Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. 2018. Gerbill-enccharkkngg named entity recognition and linking consistently. Semantic Web, 95)x605-625.
Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, Aldo Gangemi, Darío Garigliotti, and Roberto Navigli, 2015. Open knowledge extrac- tion challenge. In Semantic Web Evaluation Chal- lenges: Second Sem WebEval Challenge at ESWC 2015, Portoroř, Slovenia, May 31-June 4, 2015, Re- vised Selected Papers, pages 3-15. Springer.
Elad Segal, Avia Efrat, Mor Shoham, Amir Glober- son, and Jonathan Berant. 2020. A simple and effec- tive model for answering multi-span questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3074-3080, Online. Association for Computa- tional Linguistics.
Myle Oit, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Hassan S. Shavarani and Anoop Sarkar. 2021. Better neural machine translation by extracting linguistic information from BERT. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation
Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. 2020. Yago 4: A reason-able knowledge base. In The Semantic Web: 17th International Con- ference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings 17, pages 583-596. Springer.
for for Computational Linguistics: Main Volume, pages 2772-2783, Online. Association for Computa- tional Linguistics.
Wei Shen, Jianyong Wang, and Jiawei Han. 2014. Entity linking with a knowledge base: Issues, techniques, and solutions. TEEF. Transactions on Knowledge and Data Engineering, (7(2):443466)
Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized page rank for named entity disambigua- tion. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 238-243, Denver, Colorado. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Longuage Learning at H.L.-NAACI. 200. pages 142- 147.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Johannes van Hulst, Faegheh Hasibi, Koen Dercksen,
Krisztian Balog. and Arjen P de Vries. 2020. Rel: An entity linker standing on the shoulders of giants. In Proceedings of the 43rd International ACM SI- GIR Conference on Research and Development in Information Retrieval, pages 2197-2200
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances neural information pro- cessing systems, pages 5998-6008.
Francesco Piccinno and Paolo Ferragina. 2014. From tagme wat: a new entity annotator. In Proceedings of the first international workshop on Entity recogni- tion & disambiguation, pages 55-62.

---

Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
and some of the older identifiers used in annotating test datasets may now function as redirect links, To tackle this issue, researchers such as Broscheit (2019) and Yamada et al. (2020b, 2022) have con- sidered redirect link normalization. We follow the same approach and use the collection of Wikipedia redirect links to find all the redirect pairs (u, v) where u is not in our fixed candidate set and is in the set. In inference, whenever SPEL predicts u, we automatically replace with v.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computa- tional Linguistics.
B More on Comparison to OpenAl GPT Models
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2020a. Wikipedia2Vec: An ef- ficient toolkit for learning and visualizing the beddings of words and entities from Wikipedia. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 23-30. Association for Com- putational Linguistics.
In this section, we provide more information on our experimental procedure which can help replicating our results.
When utiliziig pre-trained GPT language model, it is common practice to structure the task description as prompt, which is then passed to the model. The model leverages its linguistic un- derstanding to generate a solution in the form of a completion, based on the given prompt. However, a crucial limitation arises in the process of identify- ing the appropriate prompt, as its selection greatly influences the successful completion of the task.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020b. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLPP), pages 6442-6454, On- line. Association for Computational Linguistics.
Tkuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2022. Global entity disambiguation with BERT. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3264-3271 Seattle, United States. Association for Computational Linguisties.
B.1 Zero-shot experiments
Our best performing prompt was as follows: You are a Wikipedia annotator. Annotate the Wikipedia entities in the following paragraph, and produce the output in markup using the <mark> element and the data-entity attribute: In each query'4, in the line after the prompt. add the AIDA document received from GERBIL and pass it GPT using openai ChatCoopletion create API'S In cases where the documents exceed the maxi- mum subword limit of GPT, we employ spacy python library, to divide the document, and create
Wenzheng Zhang, Wenyue Hua, and Karl Stratos. 2022. Entqa: Entity linking as question answering. Interna- tional Conference Learning Representations.
Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A neural probabilistic structured- structured-rrediction preddiciro model for Pansition-basen dependency parsing, In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 1213 1222, Beijing, China. Association for Computational Linguistics.
http:://dwwloodds wwtssssdppdddor/bba..e 6-10/core- 118n/en/rdirretss en.ttl.b22
A Wikipedia Redirect Normalization
'Choo al. (2022) employ GPT for entity linking by imple- menting a process that involves a sequence of summarization and multiple-choice queries to GPT. However, we have found this approach to be rather costly. Funthermore, necessitates prior knowledge of the target mention to condition the sum- mary accordingly. Additionally, it relies on a set of candidates generated through heuristies which undermines the feasibility of utilizing GPT for end-to-end entity linking.
van Hulst et al. (2020) report better results using an older Wikipedia dump from 2014 compared to the dump from 2019. One possible explanation for this finding is that the 2014 dump contains Wikipedia entries with page identifiers that are more closely aligned with the annotated data. Over time, page identifiers in Wikipedia have undergone changes,
(We used python's 's openai package version "0.27.6" and we set remperaturent tcp_p=1, trequency_p enal l ty- presence ciresence_penaltty_e py-0.

---

query prompts consisting of approximately 1000 tokens cach. Subsequently, we concatenate the received responses to these queries to form a comprehensive result. Following this, we parse the generated markup and associate each annotation with its corresponding segment in the original text. We proceed by forwarding the predicted annotations that match entries in the Wikipedia fixed candidate set, along with the extracted spans, back GERBIL.
amples. Given the promising outcomes associated with the chain-of-thought (CoT: Wei et al., 2022) prompting technique, we chose to conduct our few-shot experiments using this particular method. To construct our best performing few-shot CoT prompt, we retained our zero-shot prompt and added the following lines as an extension:
Document: "EU rejects German call to boycott British lamb."
Answer: <p> <chain-of-thought> Considering EU, German, and British are shown in the text together with the word boycott, this is political document, should annotate EU with "European Union", German with the country "Germany", and British with the country "United Kingdon". make sure do not mistake Wikipedia identifiers with entity type identifiers, for example choose "United Kingdom" instead of the incorrect general entity type "country". make sure to annotate all entities even if there is large number of entities. </chain-of-thought>> <result> <mark data aata- Union"> EU </mark> rejects <mark date- </mark> call to boycott <mark date-entity="United Kingdom"> British</mark> 1amb < </ssult></p>>> Adding more examples in the same format did not significantly improve performance, but it substan- tially increased the prompting cost to GPT-4.
During the experiments, we analyzed the vali- dation set results and observed consistent patterns that shed light on the challenges posed by gener- ative language models in entity linking. One no- table observation from mixture of knowledge bases and domains, indicating that the model possesses an excessive amount of knowledge, leading to distructions the annotation process focused on entity linking over Wikipedia. With this regards, we observed a lack of stability in the model's output even when setting the temperaa raterraturr parameter to O. Despite using the same prompt, the model occasionally confused entity linking (EL) with named entity recognition (NER) and reported mentions annotated with NER tags such as Person or Location, iments, we removed all predicted spans with such tags and did not relay them back to GERBIL. Furthermore, due to the nature of generative models, there were instances where the model failed generate the complete entity, resulting incomplete predictions (for example gen- erated Leicestershire instead of the full entity identifier Lei cestershire re County Cricket Club or Pohang instcad of Pohang Steelers). In these instances, if an exact match to an entity in the knowledge base was not found, collected all entities in the fixed candidate set that included the full prediction from the genera- tive language model. From this collection 16, we randomly selected one of those mentions and re- ported it back to GERBIL instead the original prediction generated by the model.
the presence of annotations
our exper-
German
We maintained the rest of the configurations and setups for the few-shot experiments the same as we had the zero-shot experiments.
C Out-of-domain Experiments and Results
B.2 Few-shot experiments
Few of the publications listed in Table I recom- mend assessing entity linking models on out-of- domain testing datasets. These datasets typically lack associated training sets and often anno- tated with entity links to variations or subsets of the DBped a (Auer et al., 2007) knowledge base. Oft-of-domain annotation typically operates under the assumption that the knowledge base entry iden-
In light of the growing popularity of few-shot prompting techniques with GPT language models, we conducted a survey some of the leading approaches for experimenting with few-shot ex-
In the majority of cases, the candidate set contains only
one clement if it is not empty.

---

Approach
MSNBC
Derezynski
KORE
Reuters
N RSS
OKE2015
OKE2016
Hoffart et al. (2011)
65.1
32.6
55.4
46.4
42.4
63.1
0.0
Kolitsas et al. (2018)
72.4
34.1
35.2
50.3
38.2
61.9
52.7
van Hulst et al. (2020)
74.4
41.2
61.6
49.7
34.3
64.8
58.8
De Cao al. (2021b)
73.7
54.1
60.7
46.7
40.3
56.1
50.0
Zhang et al. (2022)
72.1
$2.9
64.5
54.1
41.9
61.1
51.3
SpEL-base
64,5
50.7
48.7
47.9
41.9
55.9
57.4
SpEI-larre
63.1
59.1
53.7
47.1
44.4
59.5
56.6
Oracle
93.2
91.4
99.6
99.7
98.0
88.2
91.4
Table 5: Comparison SPEL (with a fixed candidate set size 500k) evaluation results with the literature on out-of-domain datasets. The best score is shown bold and the second best is shown as underlined. fResults from (Kolitsas et al., 2018 Table 2).
#The "Oracle" results are calculated through feeding the gold annotations of each dataset to GERBIL, and depicts the In-KB annotation quality of each dataset.
tifiers remain consistent between in-domain and out-of-domain scenarios. While this assumption may hold true to a certain extent, as DBpedia's primary focus has been on information extraction from Wikipedia, it's important to note that the tem- poral evolution of both knowledge bases has in- troduced discrepancies. These datasets, which are between 16 years old at the time of writing this paper, have been affected by temporal changes, and the knowledge bases are always perfectly aligned. The following offers a concise overview of some of the most commonly utilized out-of-domain datasets for evaluation:
phies of scholars), respectively, annotated using a mixture of annotations from DBpedia and the OKE entity identifiers.
We provided the out-of-domain data sets to SPEL, using a fixed candidate set of 500K enti- ties, and compared its performance against other methods that have reported results on these datasets. The comparative results be found in Table 5.
Please note that SPEL's tokenization procedure does not allow the generation of annotations that start or end within a single word (separated by space characters). For instance, in SPEL., the to- ken washington gle word, whereas out-of-domain datasets con- tain several annotations that commence con- clude within single word. Additionally, each dataset necessitates specific redirect normaliza- tion schema; for example, China is annotated People' s S__Repuulliccc sttpuullllooo_oeeeeeeeeeeeeeee _f___hi__aa__a_ in KORE, but in N RSS, it is annotated as China.
is considered a sin-
MSNBC (Cucerzan, 2007) contains 20 MSNBC news stories (annotated with Wikipedia) from dif- ferent categories including Health, Technology. Sports, etc.
KORE (Hoffart et al., 2012) contains 50 sen- tences annotated with DBpedia and chosen from five domains: celebrities, music, business, sports, and politics. It was created to examine the disam- biguation functionality in the older entity disam- biguation models.
Nevertheless, SPEL-large delivers the best re- sults on two out of seven and the second-best result on one out of seven test sets. It doesn't 't signifi- cantly underperform the other models in terms of performance the remaining four test sets.
3 Reuters and N RSS (Röder et al., 2014) contain mentions referring to persons, places and organizations (DBpedia annotations). The Reuters dataset contains 128 news stories from Reuters news agency and the RSS dataset contains 500 RSS feed messages from worldwide newspapers (in English).
Derezynski (Derezynski et al., 2015) contains 182 tweets annotated with DBpedia knowledge base entities.
OKE challenge 2015 and 2016 evaluation sets (Nuzzolese et al., 2015) contain 101 and 55 sen- tences from Wikipedia articles (reporting biogra-


Reviews:
[{"review_id": "aq0j4elBTT", "reviewer": "Reviewer_8Rzo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Experiment part needs to be enhanced. First, only one dataset is used in the experiments; more datasets should be tested.\n\n\nWhile setting up the experiments for this paper, we also struggled with finding more than the AIDA-CoNLL dataset for Wikipedia entity linking. Unfortunately, the other datasets we could find for Wikipedia entity linking were too small or had problematic out-dated annotations w.r.t. Wikipedia entities. In response to this situation, we have manually annotated a new dataset comprising 131 Reuters news articles basing on the NER dataset of (Liu and Ritter 2023; https://aclanthology.org/2023.acl-long.459) which was published after our submission. Our new dataset contains 1,145 unique new entity identifiers and spans over 4,028 mentions, encompassing a total of 46,456 words. We are in the process of the final annotation review and collecting results on current SoTA methods including ours. We plan to incorporate the evaluation results from this new dataset, along with the currently reported test sets from AIDA, if we are accpted in the camera-ready version of the paper. We believe this addition will enhance the comprehensiveness of our evaluation and address the reviewer's concern adequately.\n\n\n>Second, there is no analysis on the influence of underlying language model to the entity linking results; for example, what are the results of SPEL if RoBERTa is replaced with BERT?\n\nWhile we acknowledge the importance of exploring different underlying language models, our resources have been constrained by budget limitations (see Section 3.3 for our fine-tuning procedure). Despite this, we did conduct a secondary set of experiments with RoBERTa-large as the underlying model to study the effect of base model parameter size on the results.\n\n\n| Approach                                                                                        | EL Micro-F1 test-a          | EL Micro-F1 test-b  |           #params on GPU         | speed sec/doc  |\n|---------------------------------------------------------------------------------------|:--------------------------------:|:-------------------------:|:--------------------------------------:|:---------------------------:|\n| **SpEL-large** (no mention-specific candidate set)                       |          91.4                       |            85.8             |                 361.1M                  |       0.112        |\n| **SpEL-large** (KB+Yago candidate set)                                       |          90.6                       |            85.8             |                 361.1M                  |       0.157        |\n| **SpEL-large** (PPRforNED candidate set) (context-agnostic)     |          92.5                       |            87.4             |                 361.1M                  |       0.156        |\n| **SpEL-large** (PPRforNED candidate set) (context-aware)        |          **93.2**                        |            **88.4**             |                 361.1M                  |       0.156        |\n\n\nPlease note that the SpEL-large parameter size on GPU is due to the size of the underlying RoBERTa-large model (355.4M parameters) and the classification layer (5.7M parameters). The model speed remains in the range of of 0.1 to 0.16 seconds on average to process a news document even for the larger model. The new results push our previously reported state-of-the-art results by 0.8 and 0.9 Micro-F1 scores higher for AIDA testa and testb respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.A new entity linking system is proposed, which outperforms compared approaches on AIDA dataset.\n\n2.Several new methods are designed to enhance the performance of entity linking, including a context sensitive predication aggregation strategy, in-domain mention vocabulary for creating fixed candidate set, a fine-tune method without using explicit mention location information.\n", "reasons_to_reject": "Experiment part needs to be enhanced. First, only one dataset is used in the experiments; more datasets should be tested. Second, there is no analysis on the influence of underlying language model to the entity linking results; for example, what are the results of SPEL if RoBERTa is replaced with BERT?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new entity linking system called SPEL. The new system employs three new ideas to improve the accuracy and efficiency of entity linking. Experiments on AIDA dataset is conducted, and SPEL is compared with several baselines. The results show that SPEL outperforms the compared approaches.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0SXTFjgrT0", "reviewer": "Reviewer_8QvA", "summary": "", "questions": "Q1: Why were other LLMs left out of the evaluation? GPT-3.5 may be the best, but Claude 2 sometimes outperforms it.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The introduction does not clearly contain a plain English explanation of what is structured prediction...\n\nThank you for highlighting this. We will add additional context to the introduction to improve the paper's accessibility to a wider audience.\n\n> A minor issue from my point of view is represented by the comparison with GPT-3.5-turbo-16k, mainly due to the framing used (a setup from a 2021 work). Currently SoA has advanced significantly when it comes to few-shot learning for LLMs - and the paper does not contain any references to Chain of Thought, Tree of Thought and other similar problem-solving techniques...., the comparison with LLMs is rather thin - I see no reason to include it, but if it's included it needs to be done properly, Why were other LLMs left out of the evaluation? GPT-3.5 may be the best, but Claude 2 sometimes outperforms it.\n\nThe field is moving very fast and at the time of submission, GPT-3.5 was the best GPT language model that was available to us and prompt engineering was still quite exploratory. We did try many different prompts since then with no improvements. Since our submission, OpenAI opened up GPT-4 for API access and there are new heuristics in writing prompts and as a result we were able to extend our experiments. The table below reports our results using GPT-4 and better prompting.\n\n| Approach                                         | EL Micro-F1 tetsa | EL Micro-F1 testb | US$ for 1K documents |\n|--------------------------------------------------|-------------------|-------------------|----------------------|\n| GPT3.5 (zero-shot)                               | 47.3              | 52.9              | 4.22                 |\n| GPT4.0 (zero-shot)                               | 40.4              | 54.1              | 42.17                |\n| GPT4.0 (few-shot with chain-of-thought prompts)  | 62.4              | 66.2              | 59.37                |\n| SpEL-base (no c. set)                            | 90.9              | 84.2              | 2.28                 |\n| SpEL-large (no c. set)                           | 91.4              | 85.8              | 2.64                 |\n\nAdditionally, we thank the reviewer for bringing Claude 2 to our attention. We have initiated the process of obtaining API access from Anthropic to incorporate comparison results between Claude 2 and GPT-4 into our evaluation. We plan to add Claude-2 results in the camera-ready version, if accepted, provided that we receive the necessary API access from Anthropic.\n\nIt is thought-provoking that despite any published results on improved entity linking using the instruction-tuned few-shot setting of large language models like GPT4 we had to pay to compare against such models and provide a comparison to the SoTA. This is a new source of expense for academic groups conducting research into undeniably useful NLP applications like entity linking that do not nicely fit into the chat and question-answering framework of instruction-tuned few-shot LLMs. We noticed that even GPT4 will give up on longer outputs and had to be coerced to produce output on the entire input sentence.\n\n> The paper is too focused on the NER/NEL problems and no consideration is given to the larger issue of slot filling.\n\nWe appreciate the reviewer's feedback and their observation about the focus of our paper. While it is true that both Entity Linking (EL) and Slot Filling (SF) are part of the broader domain of information extraction from unstructured text, however, we see EL as a method of __alignment__ of mentions to an existing ontology such as Wikipedia. This can be helpful in addressing issues with hallucinations and truthfulness because facts about mentions in text can be linked to an ontology such as Wikipedia and the facts can be compared against a curated human expert on the entity/topic. In SF the objective is to extract specific information about an entity to populate predefined slots (as illustrated in Figure 1 of the KILT paper, https://aclanthology.org/2021.naacl-main.200, for example). It would be interesting to explore how EL can be combined with SF for a more comprehensive look at information extraction but unfortunately that was out of scope for our current paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- good structured prediction technique\n- implemented with GERBIL, therefore reproducibility is not an issue\n- results are better than expected", "reasons_to_reject": "- the comparison with LLMs is rather thin - I see no reason to include it, but if it's included it needs to be done properly\n- too focused on the NER/NEL problems and no consideration is given to the larger issue of slot filling", "questions_to_authors": "Q1: Why were other LLMs left out of the evaluation? GPT-3.5 may be the best, but Claude 2 sometimes outperforms it.", "paper_topic_and_main_contributions": "The paper is generally well-written and well illustrated. The authors tackle the issue of structured prediction in the context of named entity linking.\n\nThe introduction does not clearly contain a plain English explanation of what is structured prediction. There is a definition in Section 3.2, but that one is more formal. This makes it less accessible to casual readers. \n\nOverall, the formalization and definitions of NEL, spans, candidate sets (3.1), structured prediction (3.2), and fine-tuning procedure (3.3) are good and easy to understand.\n\nThe experimental section is rather good. The experiments are conducted with GERBIL and details about how to add new annotators (e.g., via a Java middleware called SpotWrapNifWS4Test). A minor issue from my point of view is represented by the comparison with GPT-3.5-turbo-16k, mainly due to the framing used (a setup from a 2021 work). Currently SoA has advanced significantly when it comes to few-shot learning for LLMs - and the paper does not contain any references to Chain of Thought, Tree of Thought and other similar problem-solving techniques. Granted, their technique is better than the straightforward output of GPT-3.5 in this particular instance, but we will not know for sure if that's really the case without testing at least several few-shot problem-solving techniques with GPT-3.5. The prompt used for this task, on the other hand is rather good.\n\nThe limitations section acknowledges some problems with zero-shot entity linking and multilingualism, but otherwise the method is generally sound and can be applied in various settings.\n\nI have read the rebuttal and the authors have done an impressive job of addressing most comments.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Could not identify missing references, but I am working on this particular topic (structured prediction) at the moment.", "Typos_Grammar_Style_and_Presentation_Improvements": "Overall the paper seems to be well-written and couldn't really identify major typos or grammar issues."}, "extra_text": {}}, {"review_id": "Baqpow5Ff8", "reviewer": "Reviewer_FVrb", "summary": "", "questions": "A) Could you please clarify whether the context sensitive prediction aggregation strategy can be utilized in a setting where the right answer is not added to the candidate set?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> One of the main contributions of this is a context sensitive prediction aggregation strategy; however, it only appears to be used in the setting where the candidate set always includes the correct answer. While this artificial setting is commonly used in the literature, it is important to also report results when the right answer is only present because the model included it the candidate set. This would be help determine the importance of this contribution. As the paper is currently written one cannot determine if this is a useful technique for a real-world setting. Could you please clarify whether the context sensitive prediction aggregation strategy can be utilized in a setting where the right answer is not added to the candidate set?\n\nWe acknowledge the reviewer's thoughtful observation regarding the real-world applicability of our context-sensitive prediction aggregation strategy. Ensuring the relevance of our contributions to practical scenarios has been a paramount concern for us throughout this research. Our paper addresses this concern by conducting experiments in two distinct settings: one with semi-automatically curated candidate sets and the other without them. In the latter setting, the softmax in the final prediction layer is over a set of entities which we refer to as the \"fixed candidate set\". Furthermore, we also use a compute-expensive fine-tuning step in Section 3.3 that is done on the entire Wikipedia, not just on the AIDA dataset to allow our model to be trained and tested on different Wikipedia-based EL datasets with and without candidate sets.\n\nOur experimental settings (both with and without hand-selected candidate sets) do not presume the inclusion of the correct answer in the candidate set (if used). In fact, as we explain in lines 558-570 of the submitted paper, using candidate sets can sometimes hinder performance because they may not encompass all the correct answers. As an example, consider the KB+Yago dataset, generated by examining the co-occurrence of mentions and Wikipedia links across the entire Wikipedia. In a real-world scenario, when linking \"american composer\" in a news story about \"John Luther Adams,\" who is another American composer, using a candidate set like [[\"John_Adams_(composer)\", 0.875], [\"United_States\", 0.125]] leads to error. Our \"SpEL (no mention-specific candidate set)\" results do not make any assumptions about availability of candidate sets and choose from the entirety of the in-domain entity identifiers, which is certainly a realistic setting for real-world applications.\n\nWe can obtain higher results sometimes without a candidate set, as we can see from the results using KB+Yago candidate set on AIDA test-a in Table 1 and also the RoBERTa-large results in the same setting in our response to reviewer `8Rzo`.\n\nWe include the results using candidate sets mainly because previous papers have reported evaluation scores only using candidate sets (different candidate sets in different papers). We report our results with and without candidate sets for a fair comparison to previous work. \n\n> Another contribution is the in-domain fixed vocabulary, which reduces the model size to prevent the model from generating entities that are not part of the in-domain data. This also seems problematic. While this scenario clearly occurs in the AIDA dataset where only a fraction of Wikipedia entries occur the text, having such knowledge in a real world setting needs to motivated. Otherwise, you are just making the problem easier in the artificial environment of the evaluation.\n\nWe recognize the reviewer's concern regarding the in-domain fixed vocabulary and its relevance to real-world settings. We'd like to clarify that the concept of a fixed candidate set is not inherently tied to specific datasets or data dependencies. While it can be derived from entities linked in the in-domain data, it can also be constructed from any subset of knowledge base entities that are pertinent to the task at hand. Furthermore, all previous research into entity linking makes the same assumptions that we do. In addition, unlike many previous papers, we also report our results without a constrained candidate set.\n\nTherefore, we include experiments with hand-crafted candidate sets as a way to have a fair comparison with prior published work in this area. However, we think research groups (e.g. from Facebook) who have published in this area do have some real-world examples in mind. To illustrate this with a simplified real-world example, imagine a company's interest lies solely in recognizing the names of football teams. In this scenario, the company can easily compile an exhaustive list of all football teams from Wikipedia (e.g., https://en.wikipedia.org/wiki/Coastal_Carolina_Chanticleers_football, https://en.wikipedia.org/wiki/Kansas_State_Wildcats_football, and so on) and employ this set as their fixed candidate set within our SpEL framework. The use of the in-domain set of mentions in our experiments was intended to represent this scenario, not as a rigid requirement to generate the fixed candidate set exclusively from in-domain data. The flexibility to create such sets tailored to specific real-world applications is a key feature of our approach. Our work, and prior related work in this area extends beyond artificially simplifying the problem; it offers adaptability to diverse real-world contexts where different subsets of entities may be of interest.\n\nWe also emphasize that we also provide results without candidate sets which to the best of our knowledge are better than prior results without candidate sets that use the same dataset (compare with De Cao et al. 2021a without candidate sets in Table 1). SoTA results like EntQA (Zhang et al, 2022) and (Feng et al, 2022) use methods to compute/use a candidate set. EntQA does a document level search for candidate entities, and (Feng et al, 2022) says that they \"use KnowBert’s candidate generator to first find all spans that might be potential entities in a sentence. These spans are matched in a precomputed spanentity co-occurrence table (Hoffart et al., 2011) and each span is annotated with linked entity candidate IDs associated with prior probabilities based on frequency.\" The main result in our paper is that structured prediction is a more effective and less compute-intensive method to get SoTA results for entity linking. Our method is a lot simpler and easier to implement from scratch than many of the existing approaches because we can effectively exploit the bidirectional context more efficiently than prior work.\n\n> A final weakness concerns reproducibility. It would be very difficult for another researcher to reproduce this work from the description in the paper. Unless it was missed, no mention was made to releasing the code and there was no supplementary data indicating the intent to release the code.\n\nWhile we appreciate the reviewer's concern, we already state in the footnote on page 2 that, when accepted, we will release the training data, pre-trained models and our source code for training and inference (including the settings with and without candidate sets). In addition, we also plan to release our new test set for entity linking as explained in further detail in our response to reviewer `8Rzo`.\n\nThank you for you feedback on grammar, style, and presentation. If accepted, we are committed to implementing the recommended improvements in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Entity linking is an important and challenging problem, and the approach proposed in this paper  performs better then prior approaches. In addition, while the community tends to evaluate in the unnatural setting of ensuring that the right answer is among the candidate set, the proposed approach also performs quite well when no candidate set is used. In addition the improvement to the GERBIL platform to make it easier to integrate with a python based approach is an important contribution of the paper. Finally, the paper makes a nice discovery that fine-tuning on a large entity-linking dataset which is re-tokenized without the knowledge of mention spans after fine-tuning on the dataset with linking to the knowledge base improves performance because the final fine-tuning step matches the way the model will encounter text when it is performing the task.", "reasons_to_reject": "One of the main contributions of this is a context sensitive prediction aggregation strategy; however, it only appears to be used in the setting where the candidate set always includes the correct answer. While this artificial setting is commonly used in the literature, it is important to also report results when the right answer is only present because the model included it the candidate set. This would be help determine the importance of this contribution. As the paper is currently written one cannot determine if this is a useful technique for a real-world setting.\n\nAnother contribution is the in-domain fixed vocabulary, which reduces the model size to prevent the model from generating entities that are not part of the in-domain data. This also seems problematic. While this scenario clearly occurs in the AIDA dataset where only a fraction of Wikipedia entries occur the text, having such knowledge in a real world setting needs to motivated. Otherwise, you are just making the problem easier in the artificial environment of the evaluation.\n\nA final weakness concerns reproducibility. It would be very difficult for another researcher to reproduce this work from the description in the paper. Unless it was missed, no mention was made to releasing the code and there was no supplementary data indicating the intent to release the code. \n\nThe authors can address these concerns using the explanations provided in the response to reviewers.", "questions_to_authors": "A) Could you please clarify whether the context sensitive prediction aggregation strategy can be utilized in a setting where the right answer is not added to the candidate set?", "paper_topic_and_main_contributions": "This paper proposes a structured prediction approach to entity linking where the goal is to link appropriate text to the entitles from the surface forms that represent those entities. The paper claims several contributions including a new structured prediction framework, a context sensitive prediction aggregation strategy, in-domain mention vocabulary, a model fine-tuning on the tokenized sequence to make it more capable of recognizing mentions, improved model parameter and speed \tefficiency, and an improvement to the GERBIL platform to run reproducible experiments.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It appears that some of the approaches (yours and others) use a candidate set and others do not. In addition, it appears that many only report results on the augmented fixed candidate set that always includes the right answer. It would be useful if Table 1 would identify which techniques use an augmented fixed candidate set so help understand which comparisons are fair comparisons. \n\nFigure 1 would be greatly improved by using real words and entities rather than word1..word5, and c1, c2, etc. In addition saying that c1 is not part of the candidate set rather than that c2 is the only one in the candidate set would make it easier to follow the example. In particular, a candidate set of size 1 is problematic because there is no point to generate a ranking for one candidate. It should just be the answer.  \n\nL176 ; however,\nL331 remove the\nL390-394 Awkward: rewrite. A sentence in the form of “Because X, our methods does not need to A or B” would be more clear. Be sure that X is more specific than “this strategy”. What part of the strategy means that A and B are unnecessary.\nL632 much less -> many fewer\nL633 must -> much"}}]

Author rebuttal:

Meta-review: The paper proposes to solve entity linking from a new perspective of structure prediction. To to this, the authors design a fixed vocabulary strategy to reduce the model prediction space, and address the training/inference mismatch issue by 2-stage finetuning. After serious discussion and consideration, the reviewers' main concerns are about the datasets (GERBIL is a platform easily scalable to new datasets), baselines (comparison with LLMs), as well as the above two contributions. The authors actively provide more evidence which we think has mostly solved the concerns. 

Overall, we appreciate both the efforts of reviewers and authors. This work is ready to publish. We hope the authors can continue to improve the paper according to the comments and your response.
Decision: Accept-Main