Instructed Language Models with Retrievers Are Powerful Entity Linkers
Authors: Zilin Xiao
MING GONG
Jie Wu
Xingyao Zhang
Linjun Shou
Daxin Jiang
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge Grounding
Entity Linking
Generative Model
Large Language Model
TL;DR: The first to enable causal language models to perform retrieval-augmented generative entity linking with state-of-the-art evaluation results.

Abstract: Generative approaches powered by large language models (LLMs) have demonstrated emergent abilities in tasks that require complex reasoning abilities. Yet the generative nature still makes the generated content suffer from hallucinations, thus unsuitable for entity-centric tasks like entity linking (EL) requiring precise entity predictions over a large knowledge base. 
We present Instructed Generative Entity Linker (INSGENEL), the first approach that enables casual language models to perform entity linking over knowledge bases. 
Several methods of equipping language models with EL ability were proposed in this work, including (i) a sequence-to-sequence training EL objective with instruction-tuning, (ii) a novel generative EL framework based on a light-weight potential mention retriever that frees the model from heavy and non-parallelizable decoding, achieving 4$\times$ speedup without compromise on linking metrics.
INSGENEL outperforms previous generative alternatives with +6.8 F1 points gain on average, also with a huge advantage in training data efficiency and training compute consumption. In addition, our skillfully-engineered in-context learning (ICL) framework for EL still lags behind INSGENEL significantly, reaffirming that the EL task remains a persistent hurdle for general LLMs.

Instructed Language Models with Retrievers Are Powerful Entity Linkers
Zilin Xiao Xingyao Zhang* Rice University
Ming Gong Linjun Shou Microsoft STCA*
Jie Wu*
Daxin Jiang*
zilin@rice. edu
(migon, jiewul, xingyaozhang, lisho, (gjiann)@micoosooe
Abstract
mention context and entity candidates into repre- sentations. Then a simple maximum inner product search (MIPS) utilized to capture mention-entiy correspondence, resulting in final entity predictions over the entire knowledge base (KB).
Generative approaches powered by large lan- guage models (LLMs) have demonstrated emer- gent abilities in tasks that require complex rea- soning abilities. Yet generative nature still makes the generated content suffer from hal- lucinations, thus unsuitable for entity-centric tasks like entity linking (EL) requiring precise entity predictions over large knowledge base. We present Instructed Generative Entity Linker (INsGENEL), the first approach that enables casual language models to perform entity link- ing over knowledge bases. Several methods equip language models with EL capabil- ity were proposed in this work. including (i) a sequence-to-sequenee with instruction-tuniin -tnniig (ii) novel generative EL framework based on light-weight poten- tial mention retriever that frees the model from heavy and non-parallelizable decoding. achiev- ing 4x speedup without compromise on linking metrics. INSGENEL outperforms previous gen- erative alternatives with +6.8 Fl points gain on average, also with a huge advantage in training data efficiency and training compute consump- tion. In addition, our skillfully engineered in- context learning (ICL. framework EL still lags behind INSGEN significantly, reaffirm- ing that the EL task remains a persistent hurdle for general LLMs.
A more recent trend in EL research focuses on building an end-to-end pipeline that intertwines MD and ED and formulates them into different tasks, such as question-anne answering (Zhang et al., 2022b), multi-task learning (Ayoola et al., 2022) and language generation (Cao et al., 2021b).
While generative large language models (LLMs) have shown dominant abilities in a multitude of NLP tasks (Wang et al., 2022; Ouyang et al., 2022; Xu et al., 2023), their abilities under-explored in the realm of entity-centrie NLP tasks, especially EL. Unlike numerous knowledge language ground- ing tasks that can be readily unified to text-to-text frameworks (Xie et al., 2022), thanks to their close adherence to a unified surface form (i.e., text), EL presents distinctive challenges. The foremost dif- ficulty lies in the fact that unconstrained genera- tion frequently fails to yield precise entity identi- fiers, because of the notorious hallucination issuc of LLMs. Even though the pre-defined prefix trie proposed in Cao et (2021b) can ensure a legal generation sequence, the non-parallel beam search process makes it unsuitable for real-time usage, not to mention its performance lags behind later discriminative alternative. In this work, we revisit generative EL by proposing three variants: INS- GENEL, INSGENEL-R and INSGENEL-ICL. INSGENEL solves EL by leveraging a method- ology which constrains the next possible tokens, and eliminates invalid options during generation, thus ensuring the generated text can be success- fully parsed into EL results. In contrast Cao al. (2021b), the baseline model in INS- GENEL diverges by incorporating the input doc- ument as part of the prompt and optimizes the casual language modeling (CLM) objective on
training objective
1 Introduction
Entity linking (EL) has emerged as a critical re- search problem in the intersection of Natural Lan- guage Processing (NLP) and Information Retrieval (IR), and serves as a fundamental task in vari- ous NLP applications, including text understand- ing (Févry et al., 2020), question answering (Asai et al., 2020), to name few.
Previous EL approaches typically divided EL into two steps: Mention Detection (MD) and En- tity Disambiguation (ED). Once the MD model produces mention span proposals over the input document, dense ED models attempt to encode

---

decoder-only transformer models through instruc- tion fine-tining approach, as opposed employ- ing a sequence-to-sequenee objective based on an encoder-decoder neural architecture. We ob- serve that by instruction fine-tuning decoder-only LMs such as OPT-series (Zhang ct al., 2022a) and LLaMA-series (Touvron et al., 2023). INS- GENEL yields superior EL performance compared previous work that finetunes generative alter- native on BART (Lewis et al., 2020a) with +6.8 Fl points on average. This suggests that instruc- tion fine-tuning may unlock specific entity-related knowledge within pretrained language models. Ad- ditionally, this approach exhibits significant im- provements in both training compute efficiency and data efficiency. indicating that foundation language models effectively reduce the learning diffi- culty of task-specific objectives.
manner on LLMs such as GPT-3 (Brown al., 2020). GPT-3.5 and Codex, named INSGENEL- ICL. Side-by-side comparative results indicate that while generic LLMs can correctly adhere to the format of exemplars by learning in context, they are unable to match the same accuracy exhibited by INSGENEL-R.
In summary, this paper pushes the generative EL paradigm new limits, both in terms of eval- uation metrics and runtime performance. The novel retrieval-augmentee generative EL emulates an agent interacting with the dynamic environ- ment, and sheds light on real-time generative EL without incurring substantial metric drop. The new in-context learning, paradigm for EL also indicates that current LLMs can not support an optimal ICL solution for EL. We release our code and checkpoints at https: //gi thub. com/ wr.iiinkkko/msneenntttttttnnnn
However, directly generating sequences incurs significant computational overhead during infer- ence, as both memory footprint and computation increase with sequence length, not to mention the non-parallelizable nature of regressgressive decod- ing. To address these challenges, we propose of- floading the Mention Detection (MD) responsibil- ity to an external retriever. For each document, the external retriever selects top-K K entities that possi- bly exist in the document and constructs possi- ble mention set. Then the surface form matching process dynamically determines the range where decisions are needed during the generation. Finally, greedy decoding is employed only when a choice is necessary (within a decision range), i.e., whether to start mention, end an entity identifier, or choose among dynamic candidates.
Related Works
Entity Linking (EL) is a task of locating men- tions and disambiguating these surface forms into entities in some knowledge base. Each EL mention-entig prediction should be in the format of (ms Me, ent), where Ms, Me, end indicate the start, end position of a mention and the entity iden- tifier in the knowledge base respectively.
While early EL methods (Hoffimann et al., 2011; Daiber et al., 2013) treat EL as decomposed sub- tasks, such as Mention Detection (MD) and En- tity Disambiguation (ED), a more recent trend is considering EL an end-to-end task. Kolitsas et al. (2018) first propose to use neural-based model to conduct MD and ED jointly. Broscheit (2019) transform the EL into a BIO tagging problem by training token-clasiffictioon model with an exter- nal entity classification head. Zhang et al. (2022b) formulate EL into a question-answering (QA) prob- lem and borrowed the popular eetrieve-then-reat pipeline in QA. Ayoola al. (2022) leverage the type and description of entities and employs aggre- gation of discriminative scores to obtain the final result.
Named INSGENEL-R, this novel EL generation framework offers several key advantages: a) Com- pared to constrained beam search, INSGENEL-R significantly reduces the number of heavy genera- tion forwards at the cost of a simple vector retrieval, b) It does not fall into beam failure, i.e., getting stuck with improbable mentions during the process of generating mention spans, thereby wasting valu- able inference compute. c) It is less likely to miss obvious mentions, while traditional generative EI. tends to make mistakes when generating mention boundaries. INSGENEL-R R achieves 4x reduction in terms of the number of LM calls, reduces run- time by a similar ratio at the minor cost of perfor- mance decline. Moreover, we extend the usage of the same retriever in an co-tontext learning (ICL)
For the generative EL paradigm, Cao et al. (2021b) first turn EL into sequence-to-sequeen constrained generation with an encoder-decoder transformer model, fully exploiting the flexibility of sequence generation. Cao al. (2022) extend the same paradigm into multilingual EL. De Cao et al. (2021) propose an efficient generative EL

---

model that relies on a shallow LSTM-based de- coder, at the cost of generalization over general EL benchmarks. Mrini et al. (2022a) add an entity- matching prediction module over generated se- quence and train EL using the MD objective, au- toregressive objective and matihiy objec- tive jointly.
els with the ability to retrieve relevant information from a large corpus of external knowledge. Most works augment the input context with retrieved ex- ternal documents, thus the encoded representation or generated sequence will be conditioned on ex- ternal knowledge. Guu et al. (2020) firstly train an end-to-end language encoder with dense retrieval system. Lewis et al. (2020b) finetune general sequence-to-sequence model with external triever.
Our methods fall within the generative EL cat- egory well, but they stand out in several dis- tinctive features. First, putting the input docu- ment in the instruction enables EL generation on decoder-only casual language models rather than using an encoder to capture the global context. This approach enables us to benefit from the advance- ments in recent foundation models. Moreover, the mention detection offloading alleviates the burden of non-parallel generation. The practice of invok- ing the generative model only within the decision range drastically enhances computation efficiency, also presenting an intriguing parallel with interac- tive agents in knowledge base question answering (KBQA) (Gu et al., 2022) and the principle of In- teractive NLP (Wang et al., 2023).
Our INSGENEL-R utilizes a lightweight en- coder for each document to retrieve the most rele- vant entities, then uses generative agent to elim- inate impossible solutions dynamically. For sim- ilar entity-relater retrieval scenarios, Févry et al. (2020) is the first to integrate entity supervision on language understanding tasks. Zhang et (2022c) constructs an entity memory bank and allows dy- namic aggregation of entity representations, boost- ing the performance of entity-intensive question- answering and generation tasks. However, none of them use retrieved entities to explicitly guide text generation.
Instruction-tuning (Wei et al., 2022a) usually refers to finetuning language models with collec- tion of tasks that are formulated as plain-text in- structions. Recent LLM pre-training (Brown et al., 2020; Chowdhery al., 2022) show that emergent abilities (Wei et al., 2022b) exhibit when model size, data and training compute scale. Specifically, one of such abilities that large models can lever- age natural language instructions to solve language problems in a zero-shot fashion. When instructions alone can not guide satisfactory generation, In- Context Learning (ICL) can guide LLM to learn from in-context exemplars perform complex soning, such as solving mathematical problems (Wei al., 2022c).
Methodology
3.1 Vanilla Generative EL
Vanilla generative EL addresses entity linking as an autoregressive sequence generation task, that is, given the document, the generated sequence should both indicate the mentions and their associated KB entities. Special boundary tokens are used to mark up mentioned plain string and entity identifiers. The training setup generally follows standard pequuence-to-sqqennce neural machine translation task (Wu et al., 2016). where Cao et al. (2021b) maximize pe (y x) with respect to the model's parameters 6. We refer to Appendix for inherent problems of vanilla generative EL.
Although eestruction-tuning was originally pro- posed for zero-shot language tasks, we find that tun- ing models using document instructions also leads to improved prediction and disambiguation of un- common entities in generative EL. Taking inspira- tion from ICL, we have extended our approaches to encapsulate EL with the ICL paradigm, facilitating equitable comparison between INSGENEL-R and general-purposs LLMs.
3.2 Instruction-tumed INSGENEL Baseline
Our baseline focuses on instruction-tuning a decoder-only casual language model. The prompt part consists of an optional natural language task instruction plus the document to be linked. The target sequence comprises the linked document in its plain-text form, but with special boundary sym- bols indicating mentions and corresponding entity identifiers!
Retrieval-auggnenteed Language Models are an emerging class of models in the field of NL.. They offer innovative solutions to knowledge-related tasks by combining the power of language mod-
Following Can et al. (202 we use parentheses as men- tion boundarics and brackets as entity identifier houndaries.

---

Document
Retrieved Entities
Possible Mentions
After death of Steve, the former CEO of Apple...
Apple (O89)
Apple (from Apple)
2. Apple Inc. (Q312)
(from Appie)
Doc Encoder
3. Steve Jobs (Q198377
Apple (from Apple Inc.)
K
Stotes (Q30)
(from Steve Jobs)
Retrieaal
5. Dell (C330873)
Jobs (from Jobs)
6. Google (QB5)
CEO (from Steve Jobs)
Dynamic Montion- Candidate Mapping
Apple
Steve Jobs)
Stave: ('Steve Jobs';; CEO: ('Steve Jobs'];
Dell (from Dell)
Apple: ['Apple', 'Apple Inc.]
Google (from Google)
After the death of Steve, the former CEO Apple...
Surface Form Matching
Decision required Spans
Guided Entity Linking Generation
After the death of Steee Steve Jobs
the former
Apple
CEO of
45 tevé loreddy decocing, thus do not continue)
Apple Apple
call genorative model
Inc.
only when decision needed
Generated
After the death [Seeee] [5leve lobs] the former CEO of [Apple] (Apple Inc.]
Number of Forward Pastes: 4
Figure 1: Overview of INSGENEL-R with greedy decoding strategy. Each box with grey background indicates generative agent decision, and the dotted arrow denotes an abandoned decoding path. Best viewed in color and be in comparison with Appendix and Figure
We optimize the following cross-entropy loss, which is known the next token prediction loss in casual language modeling:
weights, [CLS] and [SEP] are BERT special to- kens. (e) and (desc(e) are text identifier and
text deseription of an entity e, respectively.
Specifically, we use multi-label variant of noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010) objective to train an entity re- triever conditioned on document input, following the setup in Zhang et al. (2022b).
N
CEL.
log P (y: .1...... v1-1),
where y [y1..... Un..... UN] is the concatena- tion of the prompt and target sequence, and de- notes the length of the prompt sequence. Note that we do not backward the next token prediction loss on the prompt sequence. The fine-tuned model will serve as the generative backbone all experiments in this paper.
During training, we prepare document chunk x and a set of oracle entities E(x) e & that are men- tioned in z. We train the retriever with maximizing the following objective:
exp (S(e))
Σ log (E((x)
exp exp(S(e)) + EvENIE,.) exp (S(c"))
3.3 INSGENEL-R: Retrieval-augmenter
Generative EL
where S(e) XT stands for the matching score between document chunk x and entity e, N(E, x)) is a set negative entities that do not overlap with gold entity set E(x). This objective constructs NCE instances on the fly, treating one gold entity as the only correct answer in each train- ing sample, while excluding other gold entities out of negative examples. 90% negative samples are selected randomly and 10% chosen by hard negative mining, i.e., selecting the highest-scoring incorrect entity.
Given a document chunk x C X', we would like to build dual encoder that retrieves top-K can- didate entities {01, C22 ..,...,.. } which might be mentioned in The retriever computes document representations XP and entity representations E follows,
X = PERT <[CLS];:; [SEP]),
E BERT ([[CLS]: (iiata chasse (e); [SEP]).
where BERTp and BERT g are two BERT (De- vlin et al., 2019) text encoders that do not share
During inference, entity representations E are cached into Faiss (Johnson et al., 2021) index to al-

---

low fast top-K retrieval. With retrieved entities, we construct a set of possible mentions by looking up an entity-to-mention dictionary². The top-right cor- ner of Figure| illustrates an example set of possible mentions. Each tuple within the possible mention set comprises one of the l entities retrieved and its associated mention string. Be aware that sev- eral different entities can correspond to the same mention string.
4. Within the entity part of decision-required span: Once span has completed the decod- ing of the mention, the agent will continue to decode the entity identifier part. Similar to the decoding of the mention part, if there is only one entity associated with the decoded mention, the agent will directly copy this can- didate entity (such as "Steve Jobs" with italic font in Figure 1). Otherwise, the agent will dynamically construct prefix tree containing associated entities to constrain the generation of the entity identifier (such as "Apple" and "Apple Inc." " in italics in Figure l).
Then, we run surface form matching between a possible mention set and document text. Any parts of document text that match possible mentions are marked as decision requuired... Each decision- required span comprises the start and end indices, and possible mentions that may be within the span.
Incurring only the cost of one dense vector re- trieval, INSGENEL-R reduces the calls to the gen- erative model by 90% in this sample document and no longer relies massive, pre-defined pre- fix tree. Since the retrieval procedure takes into account the entity description, mitigates the chal- lenge inherent to the generative EL paradigm, that is to distinguish between entities with similar iden- tifiers.
In the Guided Entity Linking Generation stage, the generative agent will determine the next action based on its current state:
1. Out of a Vanilla Generative EL in 3.1, which needs to decide whether initiate mention prediction at each document token, INSGENEL-R only needs to directly copy the next token when out of a decision-required span.
required span: Unlike
3.4 INSGENEL-ICLA In-Context Learning Entity Linking Paradigm
2. At the beginning of decision decision-required required span: INSGENEL-R has decide when to start mention within a decision-required span. This is achieved by comparing the log probabil- ity next document token and mention start boundary token. A constant score offset is added to the mention start token due to the elevated probability of mention appearing within decision-required span. Note that it is also valid choice for decision-required span not generate any mention at all, like the "CEO" span in Figure 1.
In-context learning (ICL) with large language mod- els (LLMs) demonstrates strong zero-shot and few- shot performance in many NLP tasks. However, the direct application of ICL to entity linking (EL) is difficult, primarily due to the limitations on the context window size which prevent the gen- erative model from directly accessing the over- whelming number of candidate entity identifiers. Nonetheless, equipped with a well-trained retriever in INSGENEL-R, we condense the EL task into an advanced machine reading comprehension (MRC) problem: given potential entities and document, the LLM is required to choose the mention span and the respective entity from document.
3. Within the mention part of a decision-require span: Once mention has been initiated, if there is only one possible mention with this span, the agent will directly copy this mention (as in the case of "Steve" in sky-blue font in Figure 1). If not, a decision made on which mention to choose within this span, which is constrained by a dynamically generated pre- fix tree that covers all mention choices in the span.
The INSGENEL-ICL paradigm begins with a fixed exemplar and task instruction, both of which are fed to the LLM as an in-context demonstration. The task instruction prompt words have been itera- tively refined, integrating well-known engineering techniques for prompting such as bad case demon- strations, and have leveraged automatic prompt op- timization tricks. We encourage readers to Figure 5 in the Appendix for the in-context prompt template. Notably, each prediction will have its final result matched by a regular expression; to prevent failed parsing due to multiple identical surface forms
We refer readers to Appendix C.2 for details building such dictionary.
'Note that the marked segments may overlap, and will require merging mentions into unified decision span through the algorithm provided in Appendix C.4.

---

In-domain
Out-of-domain
Category
Method
AIDA
MSNBC
Der
K50
R128
R500
OKE15
OKE16
Avg
Hoffart al. (2011)
72.8"
65.1
32.6
55.4
46.4
42.4
63.1
0.0
47.2
Kolitsas et al. (2018)
82.4"
72.4
34.1
35.2
50.3
38.2
61.9
52.7
53.4
Discriminative
Hulst
(2020)
80.5"
72.4
41.1
50.7
49.9
35.0
63.1
58.3
56.4
Zhang et al. (2022b)
85.8
72.1
52.9
64.5
54.1
41.9
61.1
51.3
60.5
Ayoola et al. (2022)
84.0"
71.8
50.7
64.7
58.1
42.0
64.4
59.1
Cao (2021b)
83.7"
73.7
54.1
60.7
46.7
40.3
56.1
50.0
58.2
Generative
Cao (2021a)
85.5"
Mrini (2022b)
85.7"
INSGENEL
81.5
69.5
60.9
73.8
58.6
46.8
65.7
62.1
64.9
Ours
INSGENEL-RR R
80.6
74.2
59.8
71.9
56.8
45.5
64.1
63.3
64.5
Table InKB Micro F, on the eight popular test sets. For each dataset, bold indicates the best model and underline indicates the second best. Metric with denotes that this model trains the AIDA-CoNLL train split, while our methods do not utilize any in-domain train set. indicates the authors neither report the metric on certain test sets nor release their code and checkpoints.
appearing in the same document, we require the model to output not only the mention text in the ex- emplar, but also the surrounding context for precise span matching.
labels with a string matching heuristic to solve co- reference following Cao et al. (2021a). because when entity is mentioned multiple times in a Wikipedia article, often only the first mention is correctly linked. Additionally, construct weak entity labels to increase Wikipedia data quality ac- cording to Ayoola al. (2022).
Considering that the inputs for INSGENEL- R and INSGENEL-ICL are entirely identical, INSGENEL-ICL can serve as a fair comparison point for in-context learning LLM, which helps examine the distinctions between a generic LLM and fine-tuned generative model when perform- ing generative EL.
Training and Evaluation. We utilize two se- ries of decoder-only models our base models: LLaMA (Touvron al., 2023) and OPT (Zhang et al., 2022a). The OPT series provide pre-trained models of varying sizes, enabling us to examine the correlation between model size and generative EL performance. The best result is reported on LLaMA 7B version. We do not conduct hyperpa- rameter search; the hyperparameters used during training are detailed in Appendix A. All pieces of training were performed SXM2-32GB GPUs, utilizing DeepSpeed (Rasley et al., 2020) for distributed training management. We train all models for one epoch. We report train- ing time, size of training data and training compute in Section 4.3. The evaluation was conducted on ELEVANT (Bast et al., 2022) platform with a sin- gle V100 GPU, Best-performing INSGENEL-R is with k = 100, and we report the impact of k in Section 4.3.
4 Experiment
4.1 Setting
Datasets. We follow the established standard and report InKB Micro FI score on eight entity link- eight out-of-
ing datasets. Specifically, domain test sets: the AIDA-CoNLL (Hoffart et al., 2011) test split, MSNBC (Cucerzan, 2007), Der- ezynski (Der) (Derczynski al., 2015), KORE 50 (K50) (Hoffart et al., 2012a), N3-Reuters- 128 (R (R128), N3-RSS-500 (R500) (Röder et al., 2014), and OKE challenge 2015 and 2016 (OKE15 and OKE16) (Nuzzolese et al., 2015). Training datasets were built from all article abstracts from English Wikipedia 2023-02-20 dump. Notably, do not fine-tune our models domain-specific datasets, but rely solely on Wikipedia, examine the gen- eralization capability of our method. This means we do not use the train split the AIDA-CoNLL dataset. We use hyperlinks in Wikipedia as entity
node with 8 V100-
4.2 Main Result
We report the model evaluation results in Table 1. Our model exhibits consistent performance ad- vantages across all test sets, excluding AIDA. This achievement is noteworthy given that, unlike all preceding works, we did not apply domain-
'Following common practice in previous works (Ayoola et al., 2022; Zhang et al., 2022b; Kolitsas et al., 2018), only mentions with valid KB entities are used for evaluation.

---

specific fine-tuning on AIDA. Overall, INSGENEL achieves the state-of-the-art micro Fl score across eight evaluation datasets with +3.0 relative gain compared with the previous best of discriminative peers, with +6.8 compared with the previous best of generative peers.
hours used for training. INSGENEL, the generative state-of-the-an peer GENRE (Cao al., 2021b), and the discriminative best model ReFinED (Ay- oola et al., 2022) were all trained using V100 GPUs, thus, their training GPU hours are comparable.
We set the training of GENRE using all Wikipedia abstracts as a data size reference point (i.e., training data ratio of 1) and sequentially downsample all Wikipedia abstract data using coef- ficients of 0.01, 0.05, 0.1, 0.2, and 0.5 as our com- parative data splits. Meanwhile, ReFinED trained on the full volume of Wikipedia, approximately ten times the volume of Wikipedia abstracts. Our best-perforee perforrrrrmin model, trained for around 2.5 days on an 8-V100 node using half of the Wikipedia abstracts, corresponds to 480 GPU hours in the legend. For comparison, GENRE was trained for 30 hours on 64 V100s, and ReFinED for 24 hours on 8 VV00s, corresponding 1,920 and 192 GPU hours in the legend, respectively.
The performance of INSGENEL-R is marginally affected since the top-K retrieved entities may not always cover the gold entity. The influence of k on INSGENEL-R's performance is discussed compre- hensively in Section 4.3.
Owing to the API quota budget, we only present the INSGENEL-ICL performance of selected four test sets under two OpenAl endpoints in Ta- ble 2. The evaluation on cide-ddvvnci--00 and text-davinci-00 are similar on average, despite varying metrics across different datasets. While our In-Context Learning approach for EL has un- dergone considcrable prompt optimization, still falls significantly short when compared to our INSGENEL-R which also takes the document and top-K entities as inputs. This may suggest that In-Context Learning for EL needs further investiga- tion and we leave it as future work and list possible solutions in the Limitation Section.
Compared to the previous Generative EL method GENRE, our method exceeded the evaluation per- formance of GENRE (60.16 vs. 58.2) using just tenth of the data and twenticth of the training compute (96 GPU hours vs. 1920). This gap fur- ther increased to +6.8 Fl points with the increase of training data and computation.
Method
AIDA MSNBC K50 R500|
Avg
INSGENEL-ICL
Likewise, against the earlier Discriminative EL method ReFinED, our method accomplished supe- rior performance (63.72 vs. 61.90) using the same training compute but only 2% of the data volume. Similarly, this lead widened to +3.0 Fl points training resources increased.
text-davvnci-008
50.0
53.3
39.2
34.9
44.4
coddeddavinc--002
60.7
47.4
39.0
25.4
43.1
INSGENEL-R 80.6
74.2
71.9
45.5
|68.1
Table 2: InKB Micro Fj reported on selected four test sets Metrics for INSGENEL-RR are listed in the last row for direct comparison.
Model Size. We seek to explore the potential correlation between model size, type and EL per- formance by training on different scales of decoder- only generative models. As shown in Figure 3, the five data points correspond to the models of the OPT series 350m, 1.3b, 2.7b, 6.7b, and LLaMA 7b, and their evaluation results after training on the same split of data. We observed a certain emergent ability in the models, with opt-2. 7b surpassing the previous Generative EL method. Also, despite similar number of parameters, opt-6. 7b and 1lama-7b exhibit noticeable performance gap. This further highlights the ability of our instruction- tuning method to exploit the excellent pre-training quality of LLaMA, as well as to stimulate the latent knowledge transfer capability.
4.3 Ablation Study
We diseuss ablation studies on INSGENEL, mainly focusing on the training data efficiency and model size. Additionally, we will evaluate INSGENEL- R retriever top-k gold entity coverage and its influence on the performance of our retrieval- augmented generative EL system.
Data Efficiency. Our ablations commence with data efficiency to highlight the superiority of our approach in terms of training data utilization. As depicted in Figure 2, we illustrate the correlation among training data relative size, training compute and EL evaluation performance. The legends indi- cate that colors of data points represent different EL methods, while the size of data points denotes GPU
Retriever Coverage. Although INSGENEL-R delivers exceptional results both runtime perfor-

---

Method
R (%)
Micro Fl Score (%)
INSGENEL Performance per Model Type
5
42.57
37.70
Model Parformance
GENRE F1 Score
64.9
10
54.05
49.33
INSGENEL-R
20
62.16
60.58
60.0
50
75.00
67.71
Averuge 11 Seore
58.2
100
89.20
71.90
49.3
50.0
Table Retriever coverage and performance impact of INSGENELL-R'S K50 test set.
45.0
36.8
Training Data Relative Size va. Performance
35.0
Type LNSGENEL (Duss)
opc-350m
opc-1.b
opt-2.7b Modal Type
opt-6.7b
Ilama-7b
65
RaFinED (D SOTA)
Figure 3: INSGENEL, performance with different base models.
Annerage Score
63.72
64.90
60
61 90
60.16
20
Method
of Forwards Runtime (s) K50
55-
Trainiog GPU Bours
Ayoola al. (2022)
50
2.97
64.7
61:34
9.6
192.0
Hulst al. (2020)
7.32
50.7
47.04
48.0
480.0
Zhang et al. (2022h)
131.32f
64.5
15
96.0
1920.0
Cao al. (2021b)
196.30
60.7
0.2
0.6
INSGENEL
2221
160. 160.861...8
73.8
Training Ratio
INSGENEL-R
594
44.92
71.9
-w/ FA
594
23.76
71.9
Figure 2: Comparison among training data relative size, training compute and EL performance. Selected works are all trained with V100, thus their training GPU hours are comparable. Letters "G" and "D" in the legend represent generative and discriminative respectively.
w/ KV
594
16.32
71.9
Table 4: Runtime performance benchmark on K50 test set. denotes that the runtime is estimated based the maximum throughput of the base model, and the actual runtime should be higher. denotes the runtime estimated based the typical speedup ratio reported here, and the real runtime may vary. "FA" and "KV" mean FlashAttentio and KV caching, respectively.
mance and linking metrics compared with previous generative EL works, one might be curious about how the coverage of entity retriever might impact the EL evaluation results of INSGENEL-R, After all, if the gold entity is not retrieved, INSGENEL-R would be impossible to link any mention to the cor- rect entity. Table 3 reveals the relationship among the number of top retrieved entities kt the corre- sponding gold entity recall@k within the docu- ment chunk of length L = 32, and the Miero Fl score evaluated on the K50 dataset when complet- ing -ariivall-augeented generative EL using the corresponding à entities.
4.4 Runtime Performance Benchmark
One major barrier the application of generative EL its autoregressive nature, which hinders real- time use. In Table 4 we report the runtime of lead- ing and competitive EL. systems on the K50 test set covering 50 documents. Among these, our INS- GENEL and INSGENEL-R were run 10 times us- ing different random seeds and reported the mean and standard deviation of runtime. Evidently, our model substantially curtails nearly three-quarters of the generative model calls, albeit at a minor sac- rifice accuracy.
We notice that INSGENEL-R performance gen- erally improves as k increases, which aligns with our intuition since as k increases, candidates cover more gold entities the chance for INSGENEL-R link the correct entity also increases. Unfortu- nately, as the EL checkpoint in Cao al. (2021b) is not publicly available, we are unable to test whether our Aarriverr-augmente EL scheme would work in other -orenceeet--seeeneee EL frameworks.
Admittedly, there is still nearly 15x the runtime difference compared to the efficienncy-entrrc peer, but recognize an abundant scope for runtime im- provement. For instance, by simply hot-patching attention layers with FlashAttention (Dao et al., 2022), we gain a doubling of inference speed. Also, the decoder-only property of our model enables the

---

convenience of caching previously generated hid- den states, known as KV caching. Furthermore, our retrieval-augmented framework can benefit from parallel decoding with reference (Yang et al., 2023) since many tokens are copied rather than generated a convenience other discriminative models can not avail. We leave further inference optimization of generative EL as future work.
ful, offensive, or biased content. Even though our work involves controlled generation, we can not guarantee that the fine-tuned model will strictly ad- here to ethical requirements in an unconstrained generation setting. As such, we do not recommend direct conversion from our work to downstream applications without further investigation and miti- gation of risks.
5 Conclusion
References
We present three variations of generative entity linking (EL) solutions, where INSGENEL realizes an improvement of +3.0 Fl points on popular EL test sets (+6.8 increase compared to previous gen- erative methods). Built upon this model, pro- pose novel strieval-auggeentrr generative EL framework INSGENEL-R that determines when to invoke the large language decoder on the fly. More- over, our pioneering INSGENEL-ICL marks the inception of in-context learning in EL., despite ne- cessitating additional follow-up research to achieve competitive results.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learn- ing to retrieve reasoning paths over wikipedia graph for question answering. In 8th International Confer- ence Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Tom Ayoola, Shubhi Tyagi. Joseph Fisher. Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re- FinED: An efficient zero-shot-capable approach to end-to-end entity linking, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 209- 220, Hybrid: Seattle, Washington Online. Associa- tion for Computational Linguistics.
Limitations
Although our work pushes the generative EL paradigm to its limit and uses fewer computational resources and training time than most peers, its runtime performance still lags behind that of dis- criminative EL models, even with novel retrieval- augmented EL framework in place. This may ren- der our approach suitable for scenarios prioritizing higher linking accuracy over real-time performance. Also, we do not investigate numerous works that improve LM training efficiency, such as low-rank adaption (Hu et al., 2022). These possibilities re- main as future work as they could potentially accel- erate the training further. In addition, due to budget limitations and inaccessibility to the gpt-4-32k endpoint, our INSGENEL-ICL paradigm has not been tested on the GPT-4 series. We may ob- serve a significant performance improvement on the gpt-4 or gpt-4-32k, especially with the lat- ter one's s expanded context window that will allow more diverse in-context demonstrations. Last, how to properly organize, select and format exemplars for EL could be an interesting future work of our INSGENEL-ICL paradigm.
Hannah Bast, Matthias Hertel, and Natalic Prange. 2022. ELEVANT: A fully automatic fine-grained entity linking evaluation and analysis tool. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- rions, pages 72-79, Abu Dhabi, UAE Association for Computational Linguistics.
Samuel Broscheit, 2019. Investigating entity knowl- edge in BERT with simple neural end-to-end entity linking. In Proceedings of the 23rd Conference Computational Natural Language Learning (CoNILL), pages 677-685, Hong Kong, China. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neurul Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.
Ethics Statement
Nicola De Cao, Wilker Aziz, and Ivan Titov. 202 Highly parallel autoregressive entity linking with dis- criminative correction. In Proceedings of the 2021
Large foundation models carry inherent risks and potential harms, such as the generation of harm-

---

Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7662-7669. Association for Computa- tional Linguistics.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Highly parallel autoregressive entity linking with dis- criminative correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7662-7669, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021b. Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke van Erp, Niraj Aswani, Raphaël Troney, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Infor- mation Processing and Management, Volume 5I, 2 March 2015, Elsevier. ? Elsevier. Per- sonal use of this material is permitted. The defini- tive version of this paper was published in Infor- mation Processing and Management, Volume 51, N March 2015, Elsevier and is available at : http://dx.doi.or/100..000 htt.://.....21.........
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive entity linking. Trans. Assoc. Comput. Linguistics, 10:274-290.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaram, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai. Marie Pellat, Aitor Lewkowycz, Erica Moreira. Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.02311.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4 11864 Minneapolis, Minnesota. Association for Computational Linguistics.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. 2020. En- tities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP)), pages 4937-4951, Online. Association for Computational Linguistics.
Yu Gu, Xiang Deng, and Yu Su. 2022. Don't gen- erate, discriminate: A proposal for grounding lan- guage models real-world environments. CORR. abs/2212.09736.
Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artifi- cial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sandinia, Italy. May 13-15, 2010, volume 9 of JMLR Proceedings, pages 297-304. JML.Rorg.
Silviu Cucerzan. 2007. Large-scale named entity disam- biguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLA) pages 708 -16, Prague, Czech Republic. Association for Computational Linguistics.
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In I- SEMANTICS 2013 9th International Conference on Semantic Systems, I-SEMANTICS '1, Graz, Aus- tria, September 4-6, 2013, pages 121-124. ACM.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Ming- Wei Chang. 2020. Retrieval augmented language model pre-training, In Proceedings of the 37th International Conference
Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficien exact attention with IO-awareness. In Advances in Neural Information Processing Sys- lems.
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012a. Kore: Keyphrase overlap relatedness for entity disambigua- tion. In Proceedings of the 21 ACM International

---

Conference on Information and Knowledge Manage- ment, CIKM '12. page 543-554, New York, NY, USA. Association for Computing Machinery.
knowledge-intee NLP tasks. Advances in Neu- ral Information Processing Systems 33: Anmual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012b. KORE: keyphrase overlap relatedness for entity dis- ambiguation. In 21st ACM International Confer- ence on Information and Knowledge Management, CIKM 12, Maui, HI, USA, October 29 November 02, 2012, pages 545-554, ACM.
Khalil Mrini, Shaoliang Nie, Jiatao Gu, Sinong Wang, Maziar Sanjabi, and Hamed Firooz. 2022a. Detec- tion, disambiguation, re-ranking: Autonegressive tity linking as multi-task problem. In Findings of the Association for Computational Linguistics: ACL 2022. pages 1972-1983, Dublin, Ireland. Association for Computational Linguistics.
Johannes Hoffart, Mohamed Amir Yosef, larii Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Em- pirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguistics.
Khalil Mrini, Shaoliang Nie, Jiatao Gu, Sinong Wang. Maziar Sanjabi, and Hamed Firooz. 2022b. Detec- tion, disambiguation, re- re-ranking: Autoregressive en- tity linking as a multi task problem. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022. pages 1972- 1983. Association for Computational Linguistics.
Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, Aldo Gangemi. Darío Garigliotti and Roberto Navigli. 2015. Open knowledge extrac- tion challenge. In Semantic Web Evaluation Chal- lenges, pages 3-15, Cham. Springer International Publishing.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zeulemoyer, and Daniel S. Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 29, 2022. OpenReview.net.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery &amp; Data Mining, KDD '20, page 3505-3506, New York, NY, USA. Association for Computing Machinery.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. JEEE Trans. Big Data, 733):535-547.
Nikolaos Kolitsas, Octavian- Ganea, and Thomas Hofmann. 2018 End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computa- tional Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Michael Röder, Ricardo Usbeck, Sebastian Hellmann, Daniel Gerber, and Andreas Both. 2014. No a col- lection of datasets for named entity recognition and disambiguation in the NLP interchange format. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC" 14), pages 3529-3533, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. pre-training
BART: Denoising sequence-to-sequeee for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971..
Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for
Johannes M. van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog, and Arjen P. de Vries. 2020. REL: an entity linker standing on the shoulders of giants. In Proceedings of the 43rd International ACM SIGIR

---

conference on research and development Infor- mation Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 2197-2200. ACM.
Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao. Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young. Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado. Macduff Hughes, and Jeffree Dean, 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. CoRR, abs/1609.. 0099.88144. 144.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phami Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Generaliza-
Shen. 2022. Huper-Naturallnsttrette tion via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing. pages 5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 602-631. Association for Computational Linguistics.
Zekun Wang. Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao. Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, Qingqing Zhu, Zhenzhu Yang, Adam Nik, Qi Liu, Chenghua Lin, Shi Wang. Ruibo Liu, Wenhu Chen, Ke Xu, Dayiheng Liu, Yike Guo, and Jie Fu. 2023. Interactive natural language processing. CoRR, 0bs/2305.13246..
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large lan- guage models to follow complex instructions. CoRR, 1et/2304.12244.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Nan Yang. Tao Ge. Liang Wang. Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless accelcration of large language models. CoRR, abs/2304.04487.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto. Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe. Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang. and Luke Zettlemoyer. 2022a. OPT: open pre-trained transformer language mod- els. CoRR, ths/2205.01066.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022c. Chain-of-thought prompt- ing clicits reasoning in large language models. In NeurIPS.
Wenzheng Zhang. Wenyue Hua, and Karl Stratos. 2022b. Entqa: Entity linking as question answer- ing. In The Tenth International Conference on Learn- ing Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite. Julien Plu, Canwen Xu. Teven Le Seao. Sylvain Gugger, Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System
Zhihan Zhang, Wenhao Yu, Chenguang 7hu, and Meng Jiang. 2022c. A unified encoder-decoder framework with entity memory. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 689-705, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

A Experimental Details
are with 2,048 tokens of context window, which can hardly satisfy our requirements since typical length of our input prompt is around 3,000 tokens.
Hyperparameter
OPT
LLaMA
learning rate weight_ _eccy batch size per device effective batch size learning rate strategy optimizer
9.63e-6
2e-5
B Inherent Problems of Vanilla Generative EL
4
3
128
96
Decar
To ensure the legality of generated sequence, i.e., the generated entities are within the KB, vanilla generative EL employs a constrained beam search strategy for inference. At each generation time step, the vanilla approach either chooses to gen- erate the input document verbatim, or start a new mention. Note that this mention start decision is mandatory for each possible token in the docu- ment, thus resulting in massive inference overhead as a considerable number of document tokens are unlikely mentioned. See case (a) with grey back- ground in Figure 4, where each arrow or group of arrows represents a forward pass of the generative model.
A
dropout
0.1
gradient clipping
1.0
Disabled
Table 5: Hyperparameter settings for OPT and LLaMA training INSGENEL.
We implemented all of our neural models using the transformers (Wolf et al., 2020) library. As decision spans in INSGENEL-R are not lengthy, we do not notice significant performance improve- ment in beam search, thus we employ greedy de- coding in all INSGENEL-R R experiments. We use in INSGENEL experiments. If out- of-memory is triggered during training for some model types, optimizer partitioning, gradient state partitioning, and parameter partitioning will be se- quentially enabled to ensure successful training completion.
Once it chooses to start a mention, the vanilla approach seeks advice from a pre-generated prefix tree (a.k.a. trie) constrain the tokens that are allowed in the next time step and eliminate other options. The same strategy is also used to guide entity identifier generation to remove impossible entities from candidate sequences. Vanilla gener- ative EL relies solely on unique entity title identi- fiers to distinguish between different entities. This might lead to potential confusion among entities with closely related names when finer-grained in- formation, such as entity descriptions, is not taken into account.
Due to the license attached to LLaMA (Touvron et al., 2023) models, we are not able to directly dis- tribute LLaMA-based weights. Instead, we provide delta weights of our pre-trained checkpoint, and interested readers should fill this form to get base model from Meta AI, then apply delta weights on it to get functional generative EL model. Hyper- parameters for training neural models are listed in Table 5.
Considering the local optimality of greedy de- coding and the potential for it to get stuck in infeasi- ble options during generation, the vanilla approach uses beam search to maintain top-k sequences, falls back previous states when walking into unrea- sonable paths, and ultimately parses the top-1 se- quence into EL result. As the generative model decides only ONE next step in each beam, some generated sequence in a beam may be illegal to form valid mention, resulting in wasted inference compute. See case (b) in Figure 4 for details.
Experiments with online OpenA models were conducted earlier in March 2023 with openai-python library. We set temperature=0 in OpenAl generation to ensure maximum re- producibility, but generation results may dif- fer OpenAl backend models keep evolv- ing. gpt-3.5-turbo endpoint which sup- ports ChatGPT can not correctly adhere to our instruction. For code-davinci-002 and text-davinci-003, we apply generation con- figuration of max_token=300, top_p=1, fre- quency_penalty=000 and pessence_penalty=O.. We do not run experiments on text-ada-001, text-babbage-001 and text-curie-001 they
Last, the large generative model pre-trained on web-scale text data sometimes suffers from missing important mentions in the generative EL setting even after fine-tuning, which leads to a low recall score during evaluation. See case (c) in Figure 4.
'https://plattes www.m.ppppaacccmmmmm com/

---

Document
After the death of Steve, the former CEO of Apple...
case (a)
ConcTuB
metiion
ftter
the
continue
('fter not valld mention prefix)
sart
meniion
case (b)
mention ComPinU
Inc.
0,62 Q312
0.77 S teve
Apple
entiy
Apple
(band)
0.01
Q532100
close
start
same
same
0.35 Q89
the
App
Store
0.01 Q368215
0.23
X
(Beam
due low beam score)
Sam
Apple
0.01 Q7407109
case
Generated
After the death of Steve, the former CEO of [Apple] (Appee Inc.)...
Number Forward Passes: 64 with beam size 6
Figure 4: Overview of vanilla generative EL proposed in Cao et al. (2021b) with constrained beam search. Each path a possible beam during beam search, and decimal number at the end each path the normalized beam score. The decimal numbers in case (c) indicate generative models sometimes make mistakes when initiating mention boundaries. The color scheme for text tokens is adopted from the OpenAI Tokenization webpage and for idea depiction only, and the real tokenization depends on base model. Best viewed in color.
Entity Linking Setting
tionary that is utilized by peer works, without the addition of other knowledge or data, provides a fair point of comparison.
C.1 Candidate Set Construction
Previous neural entity linking systems pre-selects reasonable number of candidate entities for each mention based on empirical probabilistic p(c|m) scores. We build such mention candidate dictio- nary with the combination of Kolitsas et al. (2018) and Hoffart et al. (2011) for INSGENEL, following these systems. Given the generative character of our work, as opposed to a discriminative one, the size of the candidate set does not affect the infer- ence speed. Thus in line with Cao et al. (2021a), we place no restrictions on the candidate set size during inference.
C.3 Evaluation Dataset Statitics
Following Ayoola et al. (2022), we present the topic, number of documents and number of men- tions for each dataset used for evaluation. The datasets cover a variety of sources including Wikipedia text, news articles, web text, and tweets. Note that the performance of the model outside these domains may be significantly different.
Topic
Num docs
Num Mentions
C.2 Candidate-to-Meentoon Mapping Construction
AIDA
news
231
4464
MSNBC
news
20
656
DER
tweets
182
Differing from all previous entity linking works, we need an empirical candidate-tee to-mention mapping for determining the decision range of INSGENEL- R. We reverse the key-value pairs in the men- tion candidate dictionary in Appendix C.I, remove duplicates of mentions under the same entity en- try, and climinate stop words. The candidate-to- mention mapping, developed from a known dic-
K50
mixed
50
145
R128
news
128
638
R500
500
530
OKE15
wikipedia
199
1017
OKE16
wikipedia
254
1402
Table 6: Dataset statistics for EL datasets

---

C.4 Algorithm for Merging Overlapping Decision Spans for Different Mentions
Algorithm 1 Merging Overlapping Decision Span
1: procedure MERGL(decision, span__saa 1st)
2: Sort decision Lepccssionssan_____ saaa__sa______________ _st in ascending order
by start
3:
mentions <-
4: for each tuple (start, end, mention) in deci-
sion_span_lss do if merged_mentte mentions is not empty and m mentions[-1][0] I start I perged__mettionss-]]]] then
5:
[
max(merged_menttne [ end)
7: append
werged_mentions// mention
to
8: else
9: append (start, end, [mention]) to
merged_menttn
10: return merged_ merged_mentions

---

AA Your task is to read the example document and identify the mentions for each candidate entity. Then you need link these mentions entities given candidate entity list. output and their corresponding mention text. when you performing this task, renenber:
answers
Rules
rule 8: Do not include any predictions that do not appear the document. duplicated predictions texts in the same surface form.
rule
same nention span,
indeed multiple mention
rule 2: To help disambiguate nentions in the same surface forn, you must produce a few surrounding words of the Predictions without
nention
document, using brackets [] indicate the brackets will be considered invalid.
rule 3: You can not 1ink multiple mentions in single predicted line. For example, prediction of Linked Yosemite National Park; corresponding nention
[reservoin] [Yasenite National Park] nat allowed, because it predicted two nention in single prediction; instead, you should predict Linked answer 8: Yosenite National Park] corresponding mention text: reservoir near [Yosemite National Park]"
0: United Kingdon; carresponding mention text: [british] government warned friday" would be an acceptable prediction as opposed to "Linked answer friday".
rule 4: You should locate the mention accurately. For example, LLinked
Kingdon; corresponding
british [governnent]
#AR Examples
## Example Document 0
began transmitting from Mumich, Germany, in 1951, spreading uncensored news to Saviet contralled countries behind Iron Curtain during the Cold
Example Document Candidates
candidate for example document Soviet Union
candidate for exanple document 0: East Germany
candidate for example document 8: west Germany
candidate for example docunent 0: Iron Curtain
candidate for example document 0: Europe
candidate 94 for example document 8 Cold (1962-1979)
candidate 95 for example document 0: Communist party
example candidate $7 for example document 8 1972 Summer Olympics
candidate
candidate 98 for example document 0 Kazakhstan
candidate
example document Cold war (general tern)
Example Answers 0
Linked answer Munich; corresponding nention text: transnitting [Munich, Gernany], 1951, spreading
Linked answer 1: Soviet Union; corresponding mention text: uncensored news to [Soviet] controlled countries behind
Linked
Curtain; corresponding mention text: countries behind the [Iron curtain] during the cold
War
Linked answer 3: Cold War; corresponding nention text: Curtain during the [Cold Kar]
AA Test Document
After times.
Steve,
CEO Apple, comnencement speech at Stanford was thousands
Document Candidates
candidate for test document: Stanford University
candidate far test document: Apple Inc.
candidate for
Allen Stanford
candidate far test dacument: Stanford, California candidate far test document: Apple
candidate 94 for test document: white House
candidate 95 for test document: Stanford, New York
candidate 96 for test document: Jesus
candidate 97 for test docunent: 20th century
candidate 98 for test document: 2808
candidate 99 for test document: Hollywaod
Answers
.......response.
Linked answer 8: Steve Jobs; corresponding mention text: death of [Steve], the forner CEO of Apple Linked answer
University; corresponding mention text: conmencement speech [Stanford] was
thousands of times
Figure 5: Example prompt input and text-davinci-003" response for an example document from kore50 dataset (Hoffart al., 2012b). The ellipsis omits the majority of top-100 potential entities for clear depiction. Markdown format highlight is enabled. INSGENEL-ICL with noxt-davinci-003 made two predictions in this document, and both were correct. However, missed an obvious mention "Apple". Best viewed color.


Reviews:
[{"review_id": "Wm8dYS6v3w", "reviewer": "Reviewer_hkkb", "summary": "", "questions": "Can you explain a bit about the weakness in the ``Reason to Reject’’? Also the in-context learning setting can be explored in the same text2data way.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we appreciate the time and effort you invested in reviewing our work, and are especially grateful for your positive recognition. As we delve into the feedback, we would like to provide responses for your comments.\n\n1. Instruction Design:\n\n   We recognize the potential of `text2data` paradigm in information extraction tasks, especially that `text2data` can output structured JSON format string with bulletproof generation strategy such as jsonformer [1]. However, one primary reason we chose not to employ this approach is the observed tendency of generative models to be less sensitive to numerical indices during our preliminary experiments. For instance, the output JSON of test sentence \"After the death of *Steve*, the former CEO of *Apple*\" in Figure 1 could be:\n\n   ```\n    [\n      {\"mention\": \"Steve\", \"entity\": \"Steve Jobs\", \"span\": [19, 23]},\n      {\"mention\": \"Apple\", \"entity\": \"Apple Inc.\", \"span\": [44, 48]}\n    ]\n   ```\n\n   And we haven't been successful in training language decoder end-to-end using these document-JSON pairs, as the model often captured wrong mention span when being evaluated. Outputting a mention span is crucial for a mention-entity prediction as there are usually multiple matches for the same surface form that resolve to different entities.\n\n   It is worth mentioning that such insensitivity could be mitigated when data, compute and model size scale, but it might be more like a general-purpose knowledge grounding agent [2] than a generative entity linker.\n\n   We expect future works on general generative information retrieval tasks, both via fine-tuning a LM or via constructing in-context templates for LLMs.\n\n2. Necessity for Generating the Whole Document:\n\n   Like the comment said, generating the whole document is not the only option. Apart from generating character index spans as we do in document-JSON training, we can also parse a unique entity linking prediction through generating context on both sides of a mention (see `rule 2` and the `Answers` part of Figure 5).\n\n   In `INSGENEL` setting, the decoding indeed takes some time since at each generation step, the model has to make decision.\n\n   However, with the proposed mention detection offloading strategy, `INSGENEL-R` only needs to focus on potential mention spans. Although it appears that the entire document has been generated, many tokens in Guided Entity Linking Generation process (Figure 1) are actually **copied** from the document, with almost no computational overhead. Reviewer 7xon raised similar issues and we refer interested reviewers to Answer H of response to Reviewer 7xon.\n\n   Therefore, in the `INSGENEL-R` setting, \"generating\" the entire document might be a more elegant choice. And to maintain **consistency** throughout our work, we have opted for the approach of \"generating\" the entire document in this manuscript.\n\nHope these would address all your concerns of our work! And thanks again for the constructive comments.\n\nReference:\n\n[1]: Jsonformer: a Github repository, URL redacted due to rebuttal policy\n\n[2]: Xie et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. Strong motivation and effective novel retriever for generative entity linking. \n\n2. Ablation study and analysis are comprehensive and detailed, which well supports the claim and explains the experimental results. \n\n3. Point out some interesting directions and improvements of future entity linking work. \n", "reasons_to_reject": "The only weakness is lack of careful instruction design. Actually information extraction tasks like entity linking can be converted to text2data problems, i.e., the input is the sentence and the output is the json object or dict of extracted elements. Many works explore instruction tuning on information extractions. It might not be necessary to decode the whole sentences in the autoregressive way. \n", "questions_to_authors": "Can you explain a bit about the weakness in the ``Reason to Reject’’? Also the in-context learning setting can be explored in the same text2data way.\n", "paper_topic_and_main_contributions": "This paper explores the instruction-tuned autoregressive language models for the task of entity linking. Besides framing the problem as instruction tuning, this work also equips with an effective retriever for entity mention generation, which significantly reduces the inference time of previous unified methods and achieve 4 times speedup. Experimental results on eight out-of-domain datasets demonstrate the superiority to previous generative methods. Finally the authors mentioned the potential of in-context learning settings for the generative entity linking. Overall, this paper has conducted comprehensive studies for tailing recent generative models to the specific entity linking tasks.  \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kqNnocdRWS", "reviewer": "Reviewer_7xon", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the reviewers' time and effort in providing positive feedback on our manuscript. And we would like to respond to the concerns raised, especially on experiment settings.\n\n1. Lack of Experiment Clarifications: We apologize for not clearly specifying the type of language model in the experiment part Table 1. We would like to clarify the underlying model for baselines here and in the future manuscript. Note that early non-neural methods are not listed, and `N/A` denotes there are no avg score for reference.\n\n   | Method                 | Model Type (and Size if available)  | Avg  |\n   | ---------------------- | ----------------------------------- | ---- |\n   | Discriminative         | -                                   | -    |\n   | Kolitsas et al. (2018) | BiLSTM                              | 53.4 |\n   | Zhang et al. (2022b)   | 3 * BERT-large (1.02B)              | 60.5 |\n   | Ayoola et al. (2022)   | RoBERTa-base + Task Heads (154M)    | 61.9 |\n   | Generative             | -                                   | -    |\n   | Cao et al. (2021b)     | BART-large (406M)                   | 58.2 |\n   | Cao et al. (2021a)     | RoBERTa-base + BiLSTM (202M)        | N/A  |\n   | Mrini et al. (2022b)   | BART-large + Task Heads (>406M)     | N/A  |\n   | Ours                   | -                                   | -    |\n   | INSGENEL               | LLaMA-7B (7B)                       | 64.9 |\n   | INSGENEL-R             | LLaMA-7B + BERT-large (7.3B)        | 64.5 |\n   | INSGENEL-ICL           | text-davinci-003 / code-davinci-002 | N/A  |\n\n   It's worth noting that discriminative EL methods and generative EL methods are typically **not comparable** in terms of their model sizes, as they operate in different ways during the inference process. Even within the same category of generative EL methods, their efficiency cannot be determined solely based on model size due to variations in decoding strategies (beam search or greedy, constrained or not, etc.). **What might be more informative** is the runtime presented in Table 4, where we compared the inference times of various representative methods.\n\n   We are grateful that the comment raises the baseline model size clarification issue. While we have made every effort to optimize the runtime of `INSGENEL`, it's hard to deny that there is still an order-of-magnitude difference in runtime compared to discriminative methods.\n\n2. Insufficient Ablation Study on Retriever:\n\n   Due to the length limitations of the initial submission, we were unable to comprehensively discuss the effectiveness of different retrievers and instead focused on retriever coverage through ablation studies in Lines 556-579. We thank for this comment and will append sparse retriever's ablations in Table 3.\n\n   In preliminary experiments, we used BM25 as a baseline for the retriever, and the validation results showed a Recall @ Top 100 of only **35%**. We speculate that this may be due to the fact that entity identifiers are typically short and have surface form overlaps with many documents, totally disregarding semantic similarity.\n\n   Please note that the inefficiency of sparse retrievers in the top-k recall will propagate errors to the generation stage, as `INSGENEL-R` is unlikely to produce reasonable results from incorrectly retrieved candidate entities. Such an error accumulation makes it difficult for sparse retrievers like BM25 to be effectively integrated into `INSGENEL-R`.\n\nThank you again for your valuable feedback and we hope that our responses clarify any ambiguities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel approach by combining a retriever with a language model to enhance entity linking.\n1. The proposed method significantly enhances performance and expedites inference, contributing to practical applicability.\n1. The paper stands out through its extensive, meticulously documented experimental results that robustly validate the author's assertions.", "reasons_to_reject": "1. Lack of Experiment Clarifications: The paper falls short in providing sufficient details regarding the experiment settings. It remains unclear whether the baseline methods in Table 1 employ the same language model as the proposed methods. Clarification in this area is essential for proper evaluation.\n\n2. Insufficient Ablation Study on Retriever: While Table 1 reveals that INSGENEL outperforms INSGENEL-R, the effectiveness of the retriever itself requires more comprehensive exploration. To bolster the study, an ablation analysis on the retriever's impact alone should be included. Additionally, considering other retriever types like BM25 or TF-IDF could offer valuable insights into the method's versatility and generalization.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a significant advancement in enhancing the efficiency and effectiveness of generative entity link models. This advancement holds particular significance for large language models, as it substantially diminishes the search space during constraint decoding through the utilization of a prefix tree.\n\nIn the context of a given document, the proposed method operates by initially identifying a set of pertinent entities. It dynamically regulates the subsequent token generation process, resulting in a substantial reduction of the search space within the generative entity link model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dEMnUg5Myk", "reviewer": "Reviewer_zVxi", "summary": "", "questions": "Question A: (Lines 091:094) Could this improvement be solely due to the model's size? Have you tried fine-tuning BART with your approach? Or have you tried using their fine-tuning on your models?\n\nQuestion B: The paper mentions multiple times instruction-tuning. How does it differ in this case from just fine tuning? The instruction stays the same, it's just a different document that is put in the prompt/prefix/input?\n\nQuestion C: (Section 3.1) An example of input/target should be provided. The paper should be understood without appendices.\n\nQuestion D: Instruction: Can you please explain the difference between Vanilla Generative EL and Instruction-tuned InsGenEL? The description in Section 3.1 and 3.2 is not very clear. Is it that Vanilla Generative EL only generates linked entities, while InsGenEL generates the whole document?\n\nQuestion E: (Lines 275): What's the text of this instruction? Why is it optional? When is it used and when it's not?\n\nQuestion F: 338:343 How do you deal with not perfect overlap of mentions (due to hyphens, capitalization, inflection)?\n\nQuestion G: 358:361 - What's the value of this score? How was it chosen? Why is it a reward (is there any RL used)?\n\nQuestion H: How is token copying performed in the InsGenEl-R? You still have to run the forward pass for those tokens to have key/values for the attention for next tokens, which wouldn't give you any speedup.\n\nQuestion I: (Section 3.4) It's not possible to understand this part without going to the Appendix (\"As a general rule, the paper should be readable as a standalone document, and any details important for understanding the key aspects of the work should be in the paper rather than in appendices\").\n\nQuestion J: (Lines 426:428) - How lack of the overlap between train (Wikipedia 2023-02-20 dump) and benchmarks is ensured?\n\nQuestion K: (Lines 465:467) - Have you tried to finetune your model on AIDA?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and valuable suggestions. We address each of the concerns below, with particular emphasis on clarifying questions related to the **motivation** of our work.\n\n### Motivation of our work\n\nWe aim to clarify the motivation of our work from two aspects ('why generative' and 'why our generative') to address the motivation concerns about using generative models rather than discriminative ones. We will emphasize this part in the introduction of our future manuscript.\n\nA. Why generative:\n\nCao et al. [1] were the pioneers in applying prefix-tree constraints for accomplishing generative entity linking. In contrast to discriminative EL methods, which usually treat candidate entities as discrete labels, the autoregressive feature of generative EL methods enrich the interactions between document context and entities. Additionally, prefix-tree constraints in the generative process help to ensure the legitimacy of the entities being predicted.\n\nB. Why our generative:\n\nWhile generative Entity Linking (EL) methods have many stunning advantages, they are still plagued by several hard-to-ignore drawbacks, as listed below: a) The non-parallelizable beam search decoding makes generative EL extremely slow; b) The performance no longer holds a breakthrough advantage and is even surpassed by subsequent score-based discriminative methods like ReFinED [2] that incorporate knowledge priors, both in terms of metrics and speed; c) It struggles with short and information-scarce entity identifiers since the entity identifiers are the only source a vanilla generative entity linker can see.\n\nIn response to these issues, **`INSGENEL-R`** offers a compact solution: By employing a simple vector retrieval, it takes into account the entity description, thereby alleviating the problem of information-scarce entity identifiers. Moreover, the retrieved entities can be utilized in Guided Entity Linking Generation (Figure 1), thereby increasing computational efficiency (fewer model forward passes) and storage efficiency (no longer relying on a large prefix tree, but dynamically generating it during inference). We would like to note that these design features distinguish our method from previous generative EL solutions, which is appreciated by Reviewer hkkb and 7xon. Additionally, our approach can also be viewed as an agent that interacts with the environment (Line 200-206), where documents and retrieved entities are considered as the environment.\n\n\n\n### Clarity of our work\n\nWe apologize for not describing all the important details in the main text, especially regarding the implementation of **`INSGENEL-ICL`**. In light of your comments, we plan to make the following changes in the future manuscript to enhance readability:\n\nA. We will integrate the content of Appendix C into Section 3.4 *INSGENEL-ICL: In-Context Learning Entity Linking Paradigm*, to demonstrate that accomplishing EL tasks using our ICL strategy on a general LLM is still challenging. Not mentioning the specific ICL design in the main text was a significant oversight in our submission.\n\nB. A pair of example input/output will be added to Section 3.1 *Vanilla Generative EL*, to more intuitively reflect the sequence-to-sequence training objective.\n\nC. We will horizontally compare the vanilla generative EL in Appendix Figure 4 with Figure 1, to intuitively show the differences in the implementation methods between the two.\n\nD. The entire Section 3 *Methodology* will be reorganized, and details that significantly contribute to readability from the appendix will be incorporated wherever possible.\n\n\n\n### Responses for all questions\n\nQuestion A: (Lines 091:094) Could this improvement be solely due to the model's size? Have you tried fine-tuning BART with your approach? Or have you tried using their fine-tuning on your models?\n\nAnswer A:\n\nWe do not think that the improvement solely comes from the model size.\n\n1. LLaMA-7b and OPT-6.7b, which have similar model sizes, also display a noticeable difference (Line 549-555). This suggests that the quality of pre-training, as well as inherent attributes of the model itself (e.g., tokenizer choice, pre-training corpus and domain, model structure, etc.), can also impact the performance of generative EL.\n\n2. BART has an encoder-decoder structure, whereas LLaMA/OPT are decoder-only. We have not provided a direct comparison, but we did attempt preliminary experiments to perform instruction-tuning on the BART model using the same training data pairs and settings but did not achieve reasonable results. A point of reference might be the average F1 score of 58.2 achieved by Cao et al. [1] using BART-large in a machine translation setting (not able to reproduce it due to possible wrong checkpoint the authors provided).\n\n   You can also refer to our response for Reviewer 7xon for baseline model sizes and corresponding performance metrics of various methods.\n\n------\n\nQuestion B: The paper mentions multiple times instruction-tuning. How does it differ in this case from just fine tuning? The instruction stays the same, it's just a different document that is put in the prompt/prefix/input?\n\nAnswer B: You are correct; as can be seen from Line 222-230, Entity Linking is a very specific task rather than a zero-shot language task. During the training process, if we consider the input document as part of the instruction, then the entire fine-tuning process can be viewed as instruction fine-tuning. Otherwise, our training process may be seen as CLM.\n\nFurthermore, unlike direct CLM fine-tuning, since the input document is considered a part of the prompt, the contextual relationships within the document itself are not modeled as part of the loss (Line 287-289), as we did not intent to train a general language model.\n\n------\n\nQuestion C: (Section 3.1) An example of input/target should be provided. The paper should be understood without appendices.\n\nAnswer C: Although we have elaborated on the changes we prepare to make in the part B of 'Clarity of our work' section, we hope to resolve your queries here by providing a toy example:\n\n> For each source provided, write an output that includes the mentions that linked to the appropriate entity identifier. Use brace brackets { } to denote a mention, and box brackets [ ] to denote a linked entity: `[SEP]` After the death of Steve, the former CEO of Apple, his commencement speech at Stanford was watched thousands of times. `[SEP]` After the death of { Steve } [ Steve Jobs ] , the former CEO of { Apple } , his commencement speech at { Stanford } [ Stanford University ] was watched thousands of times. `[SEP]`\n\n------\n\nQuestion D: Instruction: Can you please explain the difference between Vanilla Generative EL and Instruction-tuned InsGenEL? The description in Section 3.1 and 3.2 is not very clear. Is it that Vanilla Generative EL only generates linked entities, while InsGenEL generates the whole document?\n\nAnswer D: Both vanilla generative EL and InsGenEL generate the whole document during their EL inference.\n\nAs Cao et al. [1] stands on the shoulders of encoder-decoder BART, sequence-to-sequence training is quite straightforward by putting document on encoder side and autoregressively generate tokens on decoder side in a machine translation setting.\n\nAs for InsGENEL, the entire decoder-only model runs autoregressively. And the training has slight difference (Answer B) as we do not want the model to focus on language modeling inside documents, but on contextualized information that could contribute to accurate entity grounding and disambiguation.\n\n------\n\nQuestion E: (Lines 275): What's the text of this instruction? Why is it optional? When is it used and when it's not?\n\nAnswer E: The text of instruction is as follows, which went through automatic prompt optimization. The same instruction is shown in the toy example in Answer C.\n\n```python\nDEFAULT_INSTRUCTION = \"For each source provided, write an output that includes the mentions that linked to the appropriate entity identifier. Use brace brackets { } to denote a mention, and box brackets [ ] to denote a linked entity: \"\n```\n\nWe apologize for the issues left in our early writing. In all of our experiments, we have appended instructions before input documents. They are not optional.\n\n------\n\nQuestion F: 338:343 How do you deal with not perfect overlap of mentions (due to hyphens, capitalization, inflection)?\n\nAnswer F: We are wondering you’re discussing a case where the document contains the text \"After the death of Steve...\" and potential mentions are ['steve', ...]. Appendix D2 explains the details of how we build possible mention sets. We explore every combination of capitalization and hyphenation for mentions so that the possible mentions will cover most cases.\n\nWhile we don't account for inflection changes, `INSGENEL-R` can still catch the word \"apple\" in \"These fruits are apples,\" thanks to a surface form match with `apple`. On the other hand, it might fail to identify \"Leaves\" in \"Leaves are falling\" if the set of mentions only features \"Leaf\", which isn't a surface form match.\n\n------\n\nQuestion G: 358:361 - What's the value of this score? How was it chosen? Why is it a reward (is there any RL used)?\n\nWe used $ \\gamma= $1e-4 as the static elevated score. Though the score was chosen empirically, running cross-validation on held-out wiki validation sets could potentially improve evaluation metrics further.\n\nIt is worth noting that a similar threshold design also appears in Zhang et al. [3], where $ \\gamma $ is used for filtering spans in the document.\n\nAdditionally, this score controls trade-off between precision and recall, where the higher the score is, the more likely mentions are predicted in a decision-required span.\n\nAnd we apologize for the confusion that the word reward brings, and will use the term elevated score instead. There is no RL involved in our work.\n\n------\n\nQuestion H: How is token copying performed in the InsGenEl-R? You still have to run the forward pass for those tokens to have key/values for the attention for next tokens, which wouldn't give you any speedup.\n\nAnswer H: Line 347-352 and boxes with grey background in Figure 1 show how the token copy works.\n\nTo elaborate, we begin with token-by-token generation in vanilla generative EL, where a forward pass is mandatory for deciding whether to start a mention at each generation step, not to mention that some forwards are wasted due to impossible decoding paths during beam search.\n\nIn INSGENEL-R, the agent does not have to forward the model to decide anything when out of a decision-required span, but just copy the corresponding token from the document. It is **TRUE** that all copied tokens still have to produce intermediate activations for later tokens to attend on. However, we significantly reduce decoding time by minimizing the number of forward passes. This is particularly evident when no KV Caching optimization is employed, as decoding time correlates almost linearly with the number of forwards in practice. \n\nIn addition, we refer interested reviewers to LLMA [6], a preprint concurrent with our work, which elaborates in detail on how to accelerate decoding by utilizing tokens already generated in the LLM context. Our designs share similarities on an intuitive level.\n\n------\n\nQuestion I: (Section 3.4) It's not possible to understand this part without going to the Appendix.\n\nAnswer I: We fully recognize the oversight of not detailing INSGENEL-ICL in the main part, and we've laid out steps for improvement in the section A of 'Clarity of our work'.\n\n------\n\nQuestion J: (Lines 426:428) - How lack of the overlap between train (Wikipedia 2023-02-20 dump) and benchmarks is ensured?\n\nAnswer J: The training dataset setup of our work follows established best practices in the EL field, similar to prior works [1,2,3,4,5] that has also used Wikipedia dumps for training and well-known benchmarks for evaluation. After running text similarity checks between abstract part of Wikipedia 2023-02-20 dump and benchmarks, we did not notice overlaps. This aligns well with data collection descriptions of these benchmarks.\n\n------\n\nQuestion K: (Lines 465:467) - Have you tried to finetune your model on AIDA?\n\nAnswer K: Recent EL works typically follows a recipe of pre-training on Wikipedia then fine-tuning on AIDA. We have not found a recipe on `INSGENEL` that maintains model performance after fine-tuning on AIDA. Training on AIDA trainset improved performance on AIDA but deteriorated performance on all out-of-domain benchmarks. We are still diagnosing this phenomenon and believe that it may be related to larger models overfitting small training sets due to the absence of hyperparameter optimization.\n\n### Grammar, Style and Presentations\n\nLine 042: Yes, but in this context, what we wish to emphasize is that the encoder in recent EL work is usually neural-based, so we use the term \"dense\" to indicate that most EL encoders are based on neural networks. Indeed, sparse bag-of-words models can also be used to retrieve entities for entity linking, but this is beyond the scope of this manuscript.\n\nLine 081: We kindly refer you to our response of Question D for the difference.\n\n### Reproducibility\n\nWe notice that you gave an unusual rating on reproducibility but did not raise any related concerns in the question section. Therefore, we would like to clarify the following matters:\n\na) All training data comes from the publicly available Wikipedia dump, and the data post-processing methods are detailed in Line 433-440.\n\nb) The training code and model weights will be open-sourced after the review process.\n\nc) For experiments conducted on the OpenAI API, as `code-davinci-002` has been deprecated and `text-davinci-003` is also evolving over time, we decide that the generated text and corresponding entity linking parser code we obtained on `INSGENEL-ICL` in March 2023 will also be made public.\n\n\n\nThank you for your valuable feedback and we hope these could address most of your concerns!\n\n### References\n\n[1]: De Cao et al. Autoregressive entity retrieval. ICLR 2021.\n\n[2]: Ayoola et al. ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking. NAACL 2022.\n\n[3]: Zhang et al. ENTQA: Entity Linking as Question Answering. ICLR 2022.\n\n[4]: Hoffart et al. Robust disambiguation of named entities in text. EMNLP 2011.\n\n[5]: Kolitsas et al. End-to-end neural entity linking. CoNLL 2018.\n\n[6]: Yang et al. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Introduces novel methods that use pretrained generative large language models.\n - Proposes an approach (InsGenEl-R) that is 4x faster without sacrificing quality.\n - Provides comprehensive evaluation and comparison against discriminative and generative baselines.", "reasons_to_reject": "- Paper's clarity. Many details and examples are either omitted or put in the Appendix, making it difficult to understand (see questions).\n - The motivation for using generative models rather than discriminative models is not clearly stated.", "questions_to_authors": "Question A: (Lines 091:094) Could this improvement be solely due to the model's size? Have you tried fine-tuning BART with your approach? Or have you tried using their fine-tuning on your models?\n\nQuestion B: The paper mentions multiple times instruction-tuning. How does it differ in this case from just fine tuning? The instruction stays the same, it's just a different document that is put in the prompt/prefix/input?\n\nQuestion C: (Section 3.1) An example of input/target should be provided. The paper should be understood without appendices.\n\nQuestion D: Instruction: Can you please explain the difference between Vanilla Generative EL and Instruction-tuned InsGenEL? The description in Section 3.1 and 3.2 is not very clear. Is it that Vanilla Generative EL only generates linked entities, while InsGenEL generates the whole document?\n\nQuestion E: (Lines 275): What's the text of this instruction? Why is it optional? When is it used and when it's not?\n\nQuestion F: 338:343 How do you deal with not perfect overlap of mentions (due to hyphens, capitalization, inflection)?\n\nQuestion G: 358:361 - What's the value of this score? How was it chosen? Why is it a reward (is there any RL used)?\n\nQuestion H: How is token copying performed in the InsGenEl-R? You still have to run the forward pass for those tokens to have key/values for the attention for next tokens, which wouldn't give you any speedup.\n\nQuestion I: (Section 3.4) It's not possible to understand this part without going to the Appendix (\"As a general rule, the paper should be readable as a standalone document, and any details important for understanding the key aspects of the work should be in the paper rather than in appendices\").\n\nQuestion J: (Lines 426:428) - How lack of the overlap between train (Wikipedia 2023-02-20 dump) and benchmarks is ensured?\n\nQuestion K: (Lines 465:467) - Have you tried to finetune your model on AIDA?", "paper_topic_and_main_contributions": "The paper introduces InsGenEl, a new entity linking approach that uses generative large language models (LLMs) instead of discriminative methods. The authors describe three variations of their method: InsGenEl, InsGenEl-R (a generative approach that also uses entity candidate neural retrieval), and InsGenEl-ICL (which uses in-context learning for entity linking with LLMs).\n\nThey also propose a new sequence-to-sequence training entity linking objective with instruction tuning, as well as a new approach for the decoding algorithm of the generative model that uses retrieved information.\n\nThe evaluation was performed on multiple entity linking benchmarks, and the results showed that InsGenEl achieved an average improvement of over +3.0 F1 points against discriminative baselines and +6.8 F1 points against generative baselines. Additionally, InsGenEl-R was shown to be 4x faster than the baseline without any loss in quality.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 042: why dense? it could also be used by a sparse network.\n\nline 081: why diverges?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to entity linking using generative models, by combining instruction tuning with a retrieval model. In doing so, they achieve improvements on both speed and linking metrics.

Reviewers were impressed by the experimental design and results ("extensive, meticulously documented experimental results"). The major concerns on soundness were with respect to lack of details regarding the effect of model size, and insufficient ablations. The second of these was resolved in the rebuttal. The first seems to still be somewhat of an issue, but the authors are limited somewhat by what models are available.

Although one reviewer was only ambivalent in terms of excitement, the others were more positive, including the most confident reviewer. The reviewer that currently has an ambivalent rating raised many useful points, and believes that the resulting changes will greatly benefit the paper. No reviewers seemed to think this work would be especially transformative, but all seemed to think it worthy of publication.
Decision: Accept-Main