Real World Conversational Entity Linking Requires More Than Zero-Shots (2024)

Abstract: Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to the scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, longtail entities. We designed targeted evaluation scenarios to measure the efficacy of EL models under resource constraints. Our evaluation employs two KBs: Fandom, exemplifying realworld EL complexities, and the widely used Wikipedia. First, we assess EL models’ ability to generalize to a new unfamiliar KB using Fandom and a novel zero-shot conversational entity linking dataset that we curated based on Reddit discussions on Fandom entities. We then evaluate the adaptability of EL models to conversational settings without prior training. Our results indicate that current zero-shot EL models falter when introduced to new, domainspecific KBs without prior training, significantly dropping in performance. Our findings reveal that previous evaluation approaches fall short of capturing real-world complexities for zero-shot EL, highlighting the necessity for new approaches to design and assess conversational EL models to adapt to limited resources. The evaluation setup and the dataset proposed in this research are made publicly available.1

heading: 1 Introduction
text: Entity Linking (EL) is the process of detecting and resolving ambiguous mentions of entities in a given text by accurately associating them with their corresponding entries in a knowledge base (Kolitsas et al., 2018; Sevgili et al., 2022). This is a pivotal step in many downstream tasks such as semantic search (Gerritse et al., 2022; Chatterjee and Dietz, 2022; Hasibi et al., 2016), question answering (Liu et al., 2023), and conversational search (Zamani et al., 2023).
1https://github.com/informagi/reddit_ConEL
The significance of EL particularly comes to the fore in the realm of conversational systems as it helps to enhance the accuracy and relevance of the information provided to users during a dialogue session. As these systems are becoming increasingly prevalent in various applications, their ability to ground discussions in real-world knowledge is indispensable for maintaining the integrity and usefulness of the system (Ahmadvand et al., 2019; Fan et al., 2023; Kandpal et al., 2023).
Conversations possess characteristics that render common EL models suboptimal; e.g., noisy text, informal language, and entity-related information spreading through turns (Joko et al., 2021; Joko and Hasibi, 2022). However, conversational EL has been less explored in prior research and is predominantly focused on techniques and benchmarks for long documents (Logeswaran et al., 2019) or stand-alone queries (Hasibi et al., 2015, 2017). El approaches also often assume the existence of ample training data (Cao et al., 2021; Van Hulst et al., 2020; Piccinno and Ferragina, 2014), a similar distribution of entities in KB during training and at inference time, and a structurally/textually rich KB for training. These assumptions, however, do not usually hold in real-world EL scenarios, especially in a conversational context, making EL in practice more challenging.
Train and deployment of EL systems in general poses several other challenges as well. Creating an entity-annotated training dataset can be prohibitively exhaustive, or the data might be unavailable due to privacy concerns (Sui et al., 2023). In addition, the distribution of train and test entities might differ as knowledge bases may expand with time, and new entities can be added to the KB which results in an incomplete KB at training time (Aydin et al., 2022; Zhang et al., 2018). Lastly, real-world KBs do not often come with dense structural/textual entity information.
As a result, zero-shot entity linking (Logeswaran
1 13938
et al., 2019; Bhargav et al., 2022) was introduced to address some of these challenges. This setup is aimed to allow disambiguating mentions of previously unseen entities by relying on pre-trained models.
In this study, we design an evaluation framework and a dataset, addressing the gap between real-world conversational EL and the existing zeroshot EL studies, showing that current zero-shot models do not adequately address practical challenges. We pose our research questions as RQ1) Are zero-shot EL models able to generalize effectively when introduced to a whole new KB, not included in their initial training? RQ2) How much can zero-shot EL models adapt to conversational settings without prior training?
The contributions of this paper are:
• Introducing evaluation scenarios to highlight gaps in zero-shot EL research and evaluation inadequacies specifically in conversational settings. • Creating a conversational dataset to demonstrate real-world EL challenges empirically and to facilitate research in addressing practical EL challenges. • Demostrating that current zero-shot EL models significantly underperform when applied to new, domain-specific KBs without prior exposure to their entities, emphasizing that zero-shot EL is yet to be effective in solving real EL tasks.

heading: 2 Analysis Scenarios
text: To assess models based on practical constraints we perform the following groups of analysis.

heading: Generalization to Unfamiliar KB and EL task
text: This set of experiments is aimed to assess how well EL models are capable of generalizing to a new KB at inference time. Given G and G′ as KBs, models are previously trained on G and encounter G′ only at the evaluation step. Particularly selecting G′ to ensure the frequency of domain-specific and longtail entities, makes the task more challenging.
Our definition of generalizability diverges from the definition adopted by (Logeswaran et al., 2019) and (Wu et al., 2020). In our approach, we strictly enforce that there is no overlap between the knowledge bases used for training and evaluation. Specifically, we use models that are exclusively trained on Wikipedia (G) and are only exposed to the Fandom knowledge base (G′) during evaluation. This
contrasts with the methodologies in the cited studies, where models receive training on some segments of the Fandom knowledge base before evaluation, even though these are distinct from the test segments.
Along unfamiliar KB scenario, we also assess the generalization of the EL systems to a new setting which is conversational EL, since these models have not been previously trained on this setting, we intend to assess their generalizability to this setting as well.

heading: Adaptability to Conversational EL Task
text: In the second set of evaluation experiments, we examine how well EL models perform in a conversational setting. We formulate this as a zero-shot EL task since it tests the model’s adaptability to a new setting (i.e., conversational), given that zeroshot EL models are typically trained for documents, and queries and not conversations.

heading: 3 Reddit Conversational Dataset for Zero-shot EL
text: We introduce the Reddit Conversational EL dataset, specifically curated for evaluating zero-shot EL methods in conversational setup and with unseen KBs, used for our analysis scenarios. To curate this dataset we used the Convokit’s Reddit corpus2 (Chang et al., 2020), which includes subreddit posts and comments until October 2018, sourced from the broader Pushshift Reddit dataset3 (Baumgartner et al., 2020). Convokit offers 948,169 subreddits, among which, we only opt for the discussions around each of the 16 domains used in ZESHEL (Logeswaran et al., 2019) (introduced in 4.2). We extract subreddits that contain a ZESHEL domain title in their name. From each Reddit conversation, we extract its unique threads.
2https://convokit.cornell.edu/documentation/ subreddit.html
3https://pushshift.io/
2
In this context, a thread is a distinct path in a hierarchical structure of user utterances, beginning with an original post (the root) and encompassing all subsequent replies until the last reply (the leaf) (Zhang et al., 2020; Henderson et al., 2019). To create gold mention spans along with their gold Fandom entities, we rely on instances where users include hyperlinks to the Fandom website as a way of disambiguating their mention of an entity in their utterance. Next, several preprocessing, pruning, and augmentation steps were performed:
1. Removed URLs, special symbols, non-English characters, and repetitive nonsensical tokens. 2. Pruned utterances including profanity keywords (based on a publicly available profanity list (Harel et al., 2022)) and utterances with less than 5 or more than 70 tokens 3. Excluded annotations with nonsensical mentions (e.g; "here", "this link", "link" etc.) 4. Augmented user annotations in cases where the exact mention text is annotated by the user in some occurrences but not others 5. Excluded threads with less than 5 utterances and threads with no annotations
We checked the extracted annotations for instances where the gold mention and entity were exact matches. To avoid trivial disambiguation tasks, following (Logeswaran et al., 2019), we ensured no more than 5% of our threads have such annotations. Splitting the final data to train and test sets, we relied on conversation timestamps and annotation density (details in Appendix C). Dataset statistics can be found in Table 1. Samples of the dataset are included in Table 6.

heading: 4 Experimental Setup
text: We detail the entity linking models, datasets and knowledge bases used in our experiments, as well as experimental details of our analysis setups.

heading: 4.1 Entity Linking Models
text: We focus on assessing two of the very few models purported to facilitate zero-shot entity linking; ELQ (Li et al., 2020) and BLINK (Wu et al., 2020), both BERT-based models that are pre-trained on Wikipedia for EL. ELQ is based on a biencoder model and performs mention detection and entity disambiguation simultaneously in a single pass, showing promise in zero-shot QA contexts. Our analysis, however evaluates its ability to adapt to conversations. BLINK, on the other hand, specializes in entity disambiguation, requiring either predefined mention spans or an external mention detection module. It uses a BERT-based biencoder for initial entity ranking followed by a cross-encoder for candidate re-ranking. The crossencoder’s processing, while thorough, is slower compared to the biencoder in ELQ, which can be a disadvantage for applications requiring real-time response, such as conversational systems. Additionally, BLINK’s segmented approach to entity linking, which involves separate processes for mention detection and entity disambiguation, further reduces its suitability for conversational scenarios.
It is crucial to note that the BLINK model we employ was trained using the Wikipedia KB and has not been exposed to the Fandom KB, ensuring no overlap with the knowledge base used in our evaluations.

heading: 4.2 Knowledge Bases
text: We have selected Fandom,4 as the KB for our generalizability analysis. Fandom acts as a hub for ‘fan-created wikis’, covering a range of entertainment topics. We use a specific extraction of Fandom for zero-shot EL research called ZESHEL (Logeswaran et al., 2019) consisting of 16 Fandom domains and comprising approximately 500,000
4https://www.fandom.com/
3
entities. For our standard setup, we employ the Wikipedia dump from 2019-08-01,5 encompassing more than 5 million entities. This version of Wikipedia serves as the standard KB against which ELQ and BLINK are benchmarked.

heading: 4.3 Datasets
text: Along with the test set of the zero-shot conversational Reddit dataset introduced in Section 3, we perform experiments using ConEL datasets (Joko et al., 2021; Joko and Hasibi, 2022) and Wikia6 documents. The ConEL datasets, comprising ConEL 1 and 2, are derived from various sources and have been specifically curated and human-annotated for the entity linking task against Wikipedia. On the other hand, the Wikia dataset comprises documents featuring mentions of entities from Fandom, with entity annotations contributed by users of the Fandom website. Employing both these datasets allows us to effectively delineate the distinctions between conversational and traditional entity linking which mainly focuses on document-level entity linking.

heading: 4.4 Analysis Scenarios Setups
text: Generalizability. ELQ and BLINK share the same entity encoder which is trained on Wikipedia (for language understanding and also for EL) but not on Fandom. To assess their generalizability, the mentioned encoder is used to encode Fandom entities using the first 128 tokens of each entity description. Our assessment leverages two distinct data sources; our conversational Reddit data and Wikia validation set. As BLINK does not support mention detection, we evaluated BLINK’s performance in two ways. Once we detected potential mentions using Flair (Akbik et al., 2018) and provided these mentions to BLINK for entity disambiguation. Next, to assess BLINK’s zero-shot entity disambiguation capabilities, we supply it with gold mention spans of the Wikia validation and Reddit test sets and compare it to a naive baseline (Levenshtein distance). Conversational Context Adapatability. This scenario aims to evaluate the EL models’ adaptability in a new setting: conversational EL. We evaluate the performance of EL methods in a standard conversational setting using ConEL datasets and
5https://github.com/facebookresearch/BLINK/ tree/main/elq
6https://github.com/lajanugen/zeshel
Wikipedia as the KB. We then asses the adapatability of ELQ method by fine tuning it on the conversational data (ConEL-2 dataset) and compare its performance with the original mode.

heading: 5 Results and Discussion

heading: 5.1 Are Zero-Shot EL Models Generalizable?
text: We employed Flair+BLINK and ELQ as end-toend zero-shot entity linking systems evaluating their generalizability on Reddit coversations and Wikia documents. Results in Table 2 reveal a significantly low performance when these systems are tested against Fandom without any pre-training on this specific KB, in both documents and conversations. This stark underperformance raises questions regarding the practicality and reliability of these systems as zero-shot EL solutions when confronted with novel, domain-specific knowledge bases in the real-world. The results depict substantial scope for improvement in the mention detection capabilities of both Flair and ELQ. By inspecting the predictions, we realized that numerous
4
text spans are considered as possible correct mentions by Flair/ELQ, many of which do not align with the gold mentions in the Wikia and Reddit datasets. Given that annotations in both datasets is done by users, this raises the question of whether these methods can model entity saliency so that predictions are relevant and align with the user expectations. Considering table 3 we observe that even given the gold mention spans, correctly linking entities in conversations is more challenging for BLINK than in documents, highlighting the complexity of this environment. This highlights the need for better entity disambiguation techniques that consider and leverage conversational characteristics for improved disambiguation.

heading: 5.2 Are Zero-Shot EL Models Adaptable to Conversational EL Task?
text: We analyzed adaptability of end-to-end EL systems, specifically Flair+BLINK and ELQ, for disambiguating entity mentions in conversations without prior training in this context—a zero-shot setup. Findings are summarized in Table 4, where the top rows show common EL systems evaluated by Joko and Hasibi, 2022, with only CREL (Joko and Hasibi, 2022) being optimized for conversations and the rest (GENRE, TagMe, WAT and REL) are general-purpose EL systems. Results for Flair+BLINK and ELQ can be found in the second part of table. Flair underperforms in conversation mention detection, while fine tuned ELQ (adapted to conversational setup) excels in both mention detection and entity disambiguation, outdoing most models except CREL which is optimized for conversations. The adaptability of ELQ is likely due to the end-to-end MD and ED training, as well as similarity to the domain it is initially trained on.

heading: 6 Conclusions and Future Work
text: This study re-examined the efficacy of current EL models in conversational scenarios with limited data and KB resources. Motivated by the realworld challenges frequent when integrating EL components into conversational assistants, we recognized overlooked practical limitations in zeroshot EL research. We showed that current zeroshot EL models critically underperform when introduced to a new KB at inference time, due to shortcomings in both mention detection and entity disambiguation functions. These results highlight the need for designing better end-to-end zero-shot
EL systems that are reliable in various tasks and KB constraint scenarios. We conclude that the evaluation approaches being used so far in EL literature to evaluate zero-shot EL models are quite naive and not representative of the user’s perspective on entity saliency, a crucial point when in interactive systems. For future work, we will leverage our curated dataset to advance model capabilities.

heading: 7 Limitations
text: Our experiment setup involves the use of a new KB, however, the number of EL systems allowing such a use case is very limited. On the other hand, end-to-end EL systems capable of integrating mention detection and entity disambiguation is also limited. These made our choice of models to evaluate quite restricted. Additionally, to test the capabilities of models in zero-shot conversational setup, we needed a conversational dataset that is annotated by entities in a specific-domain KB with long-tail entities. Such data is usually proprietary and not open-access, thereby we had to simulate such a scenario. It would be interesting to assess whether our results hold for other domains.

heading: 8 Acknowledgement
text: This publication is part of the project LESSEN with project number NWA.1389.20.183 of the research program NWA ORC 2020/21 which is (partly) financed by the Dutch Research Council (NWO). We also would like to express our gratitude towards Emma Gerritse and Hideaki Joko for their insightful advice and inspiring discussions, which influenced the direction of the paper.

heading: A Replicating BLINK Results on Fandom
text: To ensure our results are comparable to those reported in (Wu et al., 2020), we used their Wikipedia-trained bi-encoder and cross-encoder model (the only trained models they released) and evaluated it on Wikia’s validation set using the evaluation approaches and metrics employed by BLINK’s authors. We included the results in Table 5. As this model is only trained on Wikipedia and the scores in BLINK paper are based on a Fandomtrained model, the performance is close but still lower than the ones reported by the authors.

heading: Dataset R@64 Bi Cross All

heading: B Evaluation Metrics
text: We evaluate the performance of the EL systems across three aspects; mention detection (MD), entity disambiguation (ED) (Cornolti et al., 2013), and entity linking (EL). To assess mention detection (MD) we employ a strict matching criterion, where a predicted span is deemed accurate only if it has complete overlap with the corresponding gold standard mention span. Given the entity catalogue E, let T and T̂ be the set of gold and predicted mention and entity pairs respectively. Consequently, with our matching criterion, the set of final true positives for entity linking will be defined as;
C = { e ∈ E | [ms,me] = [m̂s, m̂e], (e, [ms,me]) ∈ T, (e, [m̂s, m̂e]) ∈ T̂}
We report precision (p), recall (r) and F1-score (F1) for the three aspects whenever it is relevant. For generalizability experiments, both micro and macro averaging are used to report the scores across multiple Fandom domains.

heading: C Zero-Shot Conversational EL Reddit Data
text: Our final threads timeline spans from April 27, 2010, to October 31, 2018. Threads dated up to January 1, 2015, were allocated to the training set. For the test set, we selected the densest thread from conversations post-January 1, 2015, as the test thread, incorporating the rest into the training set. We include samples of the dataset in Table 6.
7

heading: Mentions and Entities

heading: Utterances

heading: Mentions and Entities

heading: Utterances

heading: Mentions and Entities

heading: Utterances

heading: Utterances
text: User #1: Secret New Hero : Jabba ? Jabba as a hero anyone ? User #2: Was there actually ever a Hutt Jedi ? What about Tusken Raider ? Imagine Jabba being a bullet sponge with ATAT health . User #3: Beldorian the Hutt was a Jedi , but fell to the darkside . Sharad Hett was a Jedi who left the Order and joined the Tuskens . His son A’Sharad Hett eventually became Darth Krayt . User #4: Beldorian the Hutt was killed in a light saber duel by Leia Organa Solo . What . User #5: Leia becomes a jedi in the old EU

heading: Mentions and Entities
text: Mention #1: Beldorian the Hutt Entity #1: https://starwars.fandom.com/wiki/Beldorion Mention #2: Sharad Hett Entity #2: https://starwars.fandom.com/wiki/Sharad_Hett Mention #3: A’Sharad Hett Entity #3: https://starwars.fandom.com/wiki/Darth_Krayt Mention #4: Beldorian the Hutt Entity #4: https://starwars.fandom.com/wiki/Beldorion
Conversation Sample #5

heading: Utterances
text: User #1: [Question ?] Borreload Dragon and Eater of Millions Whats the interaction between these two cause bouth trigger at the begin of the damage step User #2: Both cards trigger at the same time . Turn player’s trigger is added to the chain first according to SEGOC . CL1: Borreload CL2: Eater Resolve the chain backwards : Eater banishes Borreload Borreload resolves without effect , as it no longer points at any zones User #3: What about the other way around ? I . e . if the turn player controls Eater . CL1: Eater CL2: Borreload Resolve backwards : Borreload takes control of Eater Eater banishes Borreload ? User #4: Borreload can only trigger when it’s attacking , so it’s controlled by the turn player . This means Borreload will be CL1 under SEGOC , the turn player’s optional effects Borreload come before the non turn player’s optional effects Eater . User #5: Ahhhh I didn’t realise it only triggered when attacking . Thanks !

heading: Mentions and Entities
text: Mention #1: SEGOC Entity #1: https://yugioh.fandom.com/wiki/Simultaneous_Effects_Go_On_Chain
9

heading: Future Work
text: For future work, we will leverage our curated dataset to advance model capabilities