Improving Few-Shot Domain Transfer for Named Entity Disambiguation with Pattern Exploitation

Named entity disambiguation (NED) is a critical subtask of entity linking, which seeks to connect knowledge base entities with textual mentions of those entities.Naturally, the performance of a model depends on the domain it was trained on; thus, reducing the amount of data required to train models is advantageous.In this work, we leverage recent research on pattern exploitation for NED and explore whether it can reduce the amount of data required for domain adaptation by reformulating the disambiguation task as a masked language modeling problem.Using ADAPET (Tam et al., 2021), which implements a new approach for few-shot learning using fine-tuned transformerbased language models, we produce an NED model which yields, without any sacrifice of in-domain accuracy, a 7% improvement in zeroshot cross-domain performance as evaluated on NEDMed, a new NED dataset of mental health news which we release with this work. Related WorkNamed entity disambiguation has a storied history, stemming from the work in Bunescu and Paşca (2006), which utilized a support vector machine (SVM) (Cortes and Vapnik, 1995) kernel based on a similarity measure between input documents and the Wikipedia articles for each candidate.Cucerzan (2007) and Kulkarni et al. (2009) were seminal works in incorporating the KB topology into this decision-making process (e.g.looking at links between candidate entities across the document), but the computational cost of these techniques was a major limitation.Numerous additional authors later provided their own approximations for this problem; a recent success in this area is known as dynamic context augmentation, or DCA (Yang et al., 2019).This technique opts to sequentially process the mentions in the document, using context related to previous extractions (the linked entity itself along with entities related to that entity) to inform subsequent extractions.While pretrained language models have a long history (Devlin et al., 2019) and are traditionally fine-tuned using masked-language modeling in order to improve their modeling ability on new domains, pattern exploitation training (PET) (Schick and Schütze, 2021) and a densely supervised approach to pattern exploitation training (ADAPET) (Tam et al., 2021) are relatively recent applications of this fine-tuning approach.These techniques utilize the linguistic information contained in language models to solve natural language processing tasks by formulating them as cloze-style phrases.Tam et al. (2021) demonstrate its efficacy on a range of SuperGLUE (Wang et al., 2019) tasks, showing state-of-the-art or competitive performance on few-shot textual entailment and the BoolQ (Clark et al., 2019) question-answering dataset.To our knowledge, we are the first to apply pattern exploitation to NED.The gold standard dataset for NED was defined in Hoffart et al. (2011b).This work extends the CoNLL 2003 shared task's NER dataset (Tjong Kim Sang and De Meulder, 2003) to contain links to entities from the YAGO KB (Hoffart et al., 2011a).This dataset is known in literature as the AIDA CoNLL-YAGO dataset, and is discussed further in Section 4. Additional datasets include