SlowGuess/NAACL_2022_OCR huggingface dataset:
G P Shrivatsa Bhargav,\* Dinesh Khandelwal,\* Saswati Dana,\* Dinesh Garg, Pavan Kapanipathi, Salim Roukos, Alexander Gray, L Venkata Subramaniam IBM Research AI
{gpshri27,dikhand1, sdana027,garg.dinesh}@in.ibm.com {kapanipa@us, roukos@us, alexandergray, lvsubram@in}.ibm.com
Entity Linking (EL) maps an entity mention in a natural language sentence to an entity in a knowledge base (KB). The Zero-shot Entity Linking (ZEL) extends the scope of EL to unseen entities at the test time without requiring new labeled data. BLINK (Wu et al., 2020) (BERT-based) is one of the SOTA models for ZEL. Interestingly, we discovered that BLINK exhibits diminishing returns, i.e., it reaches  $98\%$  of its performance with just  $1\%$  of the training data and the remaining  $99\%$  of the data yields only a marginal increase of  $2\%$  in the performance. While this extra  $2\%$  gain makes a huge difference for downstream tasks, training BLINK on large amounts of data is very resource-intensive and impractical. In this paper, we propose a neuro-symbolic, multi-task learning approach to bridge this gap. Our approach boosts the BLINK's performance with much less data by exploiting an auxiliary information about entity types. Specifically, we train our model on two tasks simultaneously - entity linking (primary task) and hierarchical entity type prediction (auxiliary task). The auxiliary task exploits the hierarchical structure of entity types. Our approach achieves superior performance on ZEL task with significantly less training data. On four different benchmark datasets, we show that our approach achieves significantly higher performance than SOTA models when they are trained with just  $0.01\%$ ,  $0.1\%$ , or  $1\%$  of the original training data. Our code is available at https://github.com/IBM/NeSLET.
Entity linking is a fundamental task in the field of Natural Language Processing (NLP) and plays an important role in numerous applications including Knowledge Base (KB) question answering, document understanding, dialogue systems, etc. Consider the sentence - "She noticed a Jaguar speeding on the highway." In this sentence, the phrase
Jaguar is called as entity mention and the task of mapping this mention to a real world entity from a given KB, e.g. Wikipedia, Wikidata, DBpedia, etc., is called as entity linking  $(EL)^{1}$ . For Wikipedia KB, above mention of Jaguar should be mapped to Jaguar car and not to Jaguar cat.
Majority of the prior works on EL focus only on in-domain linking (Jiang et al., 2021; Orr et al., 2021; Yamada et al., 2016; Chisholm and Hachey, 2015) where, gold entities (ground truth entities) of the test examples are seen during training. In other words, they assume both KB and entity set are static. However, KBs evolve over time with addition of new entities and relations (Morsey et al., 2012) because they are inherently incomplete and facts change over time. This brings in a more useful and challenging scenario of zero-shot linking where the gold entity of test examples is unseen during training. This task is known as zero-shot entity linking  $(ZEL)^{2}$  and has gained attention in recent times (Wu et al., 2020; Logeswaran et al., 2019; Vyas and Ballesteros, 2021; Onoe and Durrett, 2020; Gupta et al., 2017). A BERT-based model, called BLINK (Wu et al., 2020), is a state-of-the-art (SOTA) solution for the ZEL task. Recently, a BART-based model, called GENRE (Cao et al., 2021), was proposed and is claimed to outperform BLINK. While experimenting with BLINK, we observed an interesting phenomenon: BLINK exhibits diminishing returns. As shown in Figure 1, BLINK attains  $98\%$  of its performance with just  $1\%$  of training data and further increase in the training data from  $1\%$  to  $100\%$  yields an additional gain of just  $2\%$  in the performance. But note that
<sup>1</sup>In literature, this task is also called as Entity Disambiguation (ED) and the combined task of mention detection plus ED is called as EL. In this paper, we assume entity mentions are given to us.
In practice, a small percentage of test entities do get seen during training time while working with large benchmark datasets. We call this phenomenon as leakage. In the computer vision field, the very same phenomenon of leakage is called as Generalized Zero-Shot Learning (GZSL) (Jiang et al., 2019).
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1681 - 1697
Figure 1: Figure on the right is a zoomed version of the figure on the left. The solid curve denotes diminishing returns for linking accuracy of BLINK's bi-encoder on the target domain as we increase the source domain's training data size. The dotted curve denotes the  $\%$  of test entities leaking into training. A 10-fold increase in the training data (from 900 to  $9K$  examples) yields a performance gain of less than  $10\%$ . A further 10-fold increase (from  $9K$  to  $90K$ ) merely yields a boost of  $2.07\%$ . As source domain training data increases, the leakage increases and problem drifts from ZEL to EL setting.
even a slight increase in entity linking performance has been shown to improve downstream tasks such as question answering significantly (Kapanipathi et al., 2021). However, training BLINK on a such a large volume of data just to offer a marginal gain of  $2\%$  is quite impractical due to heavy investment required in terms of data, time, and hardware. As per a tweet<sup>3</sup> from an author of the BLINK, its training took about a week's time on 8-GPUs for  $9M$  examples. Based on the above observation, this paper aims to beat BLINK performance<sup>4</sup> while utilizing only  $1\%$  of training data.
To achieve above goal, we design a novel neurosymbolic approach that combines an easily available symbolic information, called entity types hierarchy, with the BLINK model to reduce training data requirement without compromising ZEL performance. For example, most of the KBs would tag the entity Mercedes-Benz as Company or Organization. Type information can help disambiguate Mercedes-Benz even if it was unseen during training because mentions of similar-typed entities, say Jaguar, would have been seen. Our experiments empirically verify the above hypothesis. Previous works (Gupta et al., 2017; Raiman and Raiman, 2018; Onoe and Durrett, 2020) have also demonstrated performance gains by using entity types. However, their primary focus was on improving the performance, not on reducing the training data requirement, as they were not based
on large language models. Even after utilizing type information, their performance is substantially lower than systems based on large language models such as BLINK and GENRE.
Furthermore, to the best of our knowledge, all of the existing systems that exploit entity types for entity linking task ignore the hierarchical structure of types. We hypothesize that exploiting the hierarchical structure of types can aid in the task of entity linking, especially when training with less data. A common strategy while training models with less training data is to apply a strong prior (arising from domain knowledge) on the parameters. The type hierarchy can be used as a prior and encoded into the model directly so that the model will not have to learn it from the very limited training data. We encode such a prior via ensuring the logical consistency (imposed by the hierarchy) of predicted entity types – if a type is predicted, then its parent must also be predicted. For this, we have developed a novel neuro-symbolic technique inspired from Gödel and Łukasiewicz norms (Klement et al., 2000).
The following are the contributions of this paper:
- We show BLINK exhibits diminishing returns (Fig. 1).
- By utilizing diminishing returns behavior of BLINK, we propose a solution, called Neurosymbolic entity Linking using Entities Type (NeSLET), for the zero-shot entity linking problem in low training data regimes. NeSLET is a novel combination of techniques from a diverse set of fields namely multi-task learning, fuzzy
3https://twitter.com/riedelcastro/status/1256283045855969286
4In our experiments BLINK consistently outperforms GENRE by a significant margin in low data regimes.
logic, and hierarchical multi-label classification. NeSLET is trained on two tasks simultaneously - entity linking (primary task) and hierarchical entity type prediction (auxiliary task).
- We are the first ones to show that accounting for hierarchical structure of entity types improves entity linking as compared to treating it flat.
- For the scenarios where a ZEL model is given to us in the form of just black-box, we have proposed a neuro-symbolic inference algorithm that explicitly uses types just at inference time to improve the performance of given black-box.
- Our experiments on four benchmark datasets show that NeSLET beats both SOTA baselines (BLINK and GENRE) in multiple low-data regimes.
Entity Linking (EL): Recently, the models (Wu et al., 2020; Cao et al., 2021) based on large language models such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) have achieved SOTA results for the EL task. Table 1 slices the EL literature based on the flavor of underlying neural model (BERT or non-BERT) and variants of the EL task. Here, end-to-end EL means solving both mention detection and entity disambiguation tasks at the same time. BERT-based models require large training data and hence leads to poor generalizability in the case of cross-domain EL (aka ZEL) when training data is less. This motivated us to use auxiliary information about entity types. As shown in Table 1, previous works have used entity types to help the EL task. However, to the best of our knowledge, none of these approaches are designed to work in a low-data regime. Moreover, these approaches do not leverage intrinsic hierarchical structure of types and instead work with a flat hierarchy. Logeswaran et al. (2019) proposed an interesting dataset for ZEL based on the community portal (https://www.fandom.com). We, however, do not work with this dataset and related approaches (Tang et al., 2021) as it lacks entity type information.
Fine-grained Entity Typing (FET): FET (Ling and Weld, 2012; Gillick et al., 2014) is the task of assigning types from a semantic hierarchy to the entity mentions in text. FET is a hierarchical multi-label classification (HMLC) task and our auxiliary task is also an FET task. The popular datasets for FET, such as AIDA, BBN (Weischedel
and Brunstein, 2005), OntoNotes (Gillick et al., 2014), FIGER (Ling and Weld, 2012), etc., come with shallow hierarchies (2 to 3 levels deep) compared to our DBpedia type hierarchy (7 levels deep). Ultra-Fine entity typing dataset (Choi et al., 2018) comes with around  $10k$  types, but the types are not arranged in a hierarchy. There are also independent studies on the abstract HMLC problem (Giunchiglia and Lukasiewicz, 2020; Srivastava et al., 2020) without tying it to any application. The latest work on HMLC (Giunchiglia and Lukasiewicz, 2020) uses a loss function that is similar to our Gödel and Lukasiewicz  $t$ -norms (Klement et al., 2000).
Multi-task Learning (MTL): An MTL framework (Caruana, 1995) is often used to improve the performance on the primary task by learning a shared representation between the primary and one or more closely related auxiliary tasks. Learning a joint representation between related tasks helps in preventing over-fitting (Maurer, 2006), even when the amount of training data is less for each task. In our case, the primary task is entity linking and the auxiliary task is entity type prediction.
The ZEL task is akin to a cross-domain classification task where entities play the role of classes. It has two distinct characteristics - (i) number of classes (in both source and target domain) could be in the order of millions, e.g., Wikipedia has more than 5 million entities, (ii) classes are not merely labels but have rich features in the form of short textual descriptions. Formally, in a ZEL task, we are given an entity set  $\mathcal{E}_S$ , called seen (aka train) entities. Each entity  $e \in \mathcal{E}_S$  is seen during training in the form of a linked gold entity for some training mention  $m$ . We are also given another entity set  $\mathcal{E}_U$ , called unseen (aka test) entities, where  $\mathcal{E}_S \cap \mathcal{E}_U = \emptyset^5$ . The element-level structure of these entity sets is as follows:  $\mathcal{E}_S = \{(e_i, d_i)\}_{i=1}^K$  and  $\mathcal{E}_U = \{(e_i, d_i)\}_{i=(K+1)}^L$ , where  $e_i$  is the unique title of the entity and  $d_i$  is a short textual description of the entity. The train, validation, and test sets look as follows.  $\mathcal{D}_{train}^{zel} = \{(m_i, e_i) | e_i \in \mathcal{E}_S\}_{i=1}^N$ ;  $\mathcal{D}_{val}^{zel} = \{(m_j, e_j) | e_j \in \mathcal{E}_S\}_{j=1}^V$ ;  $\mathcal{D}_{test}^{zel} = \{(m_k, e_k) | e_k \in \mathcal{E}_U\}_{k=1}^M$ . In these datasets, the
In an ideal ZEL task, for each of the test example, the corresponding linked gold entity comes from the set  $\mathcal{E}_U$ . However, in practice, for a small fraction of test examples, we have their linked gold entities coming from the set  $\mathcal{E}_S$ .
EL Technique	Variants of EL Task
In-domain	Cross-domain	End-to-end
Neural (non-BERT)	w/o types	Chisholm and Hachey Yamada et al.	Logeswaran et al.,Le and Titov, Ganea and Hofmann	Banerjee et al.
with types	Raiman and Raiman	Gupta et al.,Onoe and Durrett
Neural (BERT)	w/o types	Orr et al.	Wu et al.,Vyas and Ballesteros, Chen et al.	Li et al.
with types	Jiang et al.	This Paper
Table 1: Slicing the space of prior art on EL.
first part  $m_{i}$  (and  $m_{j}, m_{k}$ ) corresponds to the input text string along with the entity mention substring marked. The second part  $e_{i}$  (and  $e_{j}, e_{k}$ ) corresponds to the gold entity that must be linked to this mention. A standard practice is to represent  $m_{i}$  as the following tuple - (left context, mention, right context). For the sentence "She noticed a Jaguar speeding on the highway", we have  $m_{i} =$  (She noticed a, Jaguar, speeding on the highway) and  $e_{i} =$  Jaguar_Cars.
A typical model for ZEL task is a scoring function  $f_{zel} : \mathcal{M} \times \mathcal{E} \mapsto \mathbb{R}$ , here  $\mathcal{E} = \mathcal{E}_S \cup \mathcal{E}_U$ . For any given mention  $m \in \mathcal{M}$ , it induces a score for each entity  $e \in \mathcal{E}$ . The ZEL model uses these induced scores  $f_{zel}(m, e)$  to rank all the entities and highest-scoring entity is predicted as the final answer. The performance of a ZEL model is measured via Hits@k for  $k \geq 1$ . Hits@k measures if the gold entity appears within the top-k elements. For  $k = 1$ , it is called accuracy.
3.1 Entity Type Prediction - Auxiliary Task
We use entity type prediction (ETP) as an auxiliary task. The goal behind ETP is to link an entity mention to one or more type classes from a given entity type set  $\mathcal{T} = \{t_j\}_{j=1}^{|\mathcal{T}|}$ . For e.g., in sentence "She noticed a Jaguar speeding on the highway", the mention Jaguar is classified into Organization and Company classes as per DBpedia's (Lehmann et al., 2015) type classes hierarchy. Training data for such a task is given in the form of  $\mathcal{D}_{train}^{type} = \{(m_i, \mathbf{t}_i)\}_{i=1}^N$  where  $\mathbf{t}_i = [t_{ij}]_{j=1}^{|\mathcal{T}|}$  is a binary vector of size  $|\mathcal{T}|$  where  $t_{ij}$  equals 1 if  $t_j \in \mathcal{T}$  is a valid type for the corresponding gold entity  $e_i$ , and 0 otherwise. A typical model for the entity type prediction is given by  $f_{type}: \mathcal{M} \times \mathcal{T} \mapsto [0,1]$ . For any given mention  $m \in \mathcal{M}$ , it induces a probability score for each type class  $t_j \in \mathcal{T}$ . That is,  $p(t_j | m) = f_{type}(m, t_j)$ ,  $\forall t_j \in \mathcal{T}$ . The performance of the ETP task is measured via  $F_1$  score
computed over predicted type set and gold type set. We make a simplifying assumption that size of gold type set is known to us; this helps us avoid hassle of setting a threshold.
4 NeSLET
As mentioned earlier, NeSLET is an MTL-based neuro-symbolic approach where we use ETP as an auxiliary task and jointly learn it with the primary task of EL. Both the primary and auxiliary tasks are classification tasks for a given mention  $m$  except that: (i) the corresponding classes are different - entities for the primary task and types for the auxiliary task, and (ii) auxiliary task is a HMLC task. In our modeling, we make a simplified assumption that the probability of entity  $e$  being the gold entity of a mention  $m$  is conditionally independent of the probability of any type  $t_j$  being the valid type of the gold entity. In other words,  $p(e, \mathbf{t} \mid m) = p(e \mid m) \cdot \prod_{j=1}^{|\mathcal{T}|} p(t_j \mid m), \forall e, \mathbf{t}, m$ . Neural Model for  $p(e \mid m)$ : We use following models for  $p(e \mid m)$  where,  $V_m$  and  $V_e$  are the vector representations of mention  $m$  and entity  $e$ , respectively.
$$
p (e \mid m) = f _ {z e l} (m, e) = \frac {\exp \left(V _ {m} ^ {\top} \cdot V _ {e}\right)}{\sum_ {e ^ {\prime} \in \mathcal {E}} \exp \left(V _ {m} ^ {\top} \cdot V _ {e ^ {\prime}}\right)} \tag {1}
$$
As shown in Figure 3, we use BERT to obtain vector representation for both mention  $m$  and the entity  $e$  (as described in Wu et al. (2020)). Parameters of these BERT models are denoted by  $\theta_{m}$  and  $\theta_{e}$ , respectively. Thus, we can say  $V_{m} = \text{mention-bert}(m, \theta_{m})$  and  $V_{e} = \text{entity-bert}(e, \theta_{e})$ .
Neuro-symbolic Model for  $p(t_j \mid m)$ : In practice, KB's organize entity types in the form of a logical hierarchy as shown in Figure 2. In such a hierarchy, if we traverse along a path from leaf node to the root, e.g. cat → mammal → animal, the corresponding types become coarse grained. To ensure this logical consistency property of the type
hierarchy, we require that our proposed model satisfies the following path monotonicity property.
[Path Monotonicity Property]: Let  $t_k$  be the parent node of  $t_j$  in the given type hierarchy. For any mention  $m$ , we must ensure that type probabilities predicted by our model satisfy the following monotonicity condition:  $0 \leq p(t_j \mid m) \leq p(t_k \mid m) \leq 1$ . We ensure this by exploiting Gödel or Łukasiewicz  $t$ -norm (Klement et al., 2000) used in the field of fuzzy logic (Klir and Yuan, 1995). As per these  $t$ -norms, if  $t_j$  is an internal node in the hierarchy then we compute its probability purely in a symbolic fashion by using the probabilities assigned to its children nodes. We use logical OR formula (given below) of the Gödel (equation 2a) or Łukasiewicz (equation 2b)  $t$ -norm for this purpose.  $C_{t_j}$  in these formulas denotes the set of children nodes for  $t_j$ .
$$
p (t _ {j} | m) = \left\{ \begin{array}{l l} \max  _ {t _ {k} \in \mathcal {C} _ {t _ {j}}} p (t _ {k} | m) & (2 \mathrm {a}) \\ \min  \left\{1, \sum_ {t _ {k} \in \mathcal {C} _ {t _ {j}}} p (t _ {k} | m) \right\} & (2 \mathrm {b}) \end{array} \right.
$$
Figure 2: An illustration of type hierarchy and issue of path consistency during training.
For above model to work, we still need to address the issue of computing type probability for each leaf node  $t_j$  in the hierarchy. We achieve this via a neural model where we obtain vector representations  $V_{t_j}$  for each leaf node by using a type network. The type network comprises an initial embedding vector for each type class and an optional linear layer followed by non-linearity. The parameters of this type network are denoted by  $\theta_t$  and hence, we have  $V_{t_j} = \text{type-network}(t_j, \theta_t)$ . Type probability of each leaf node is computed using:
$$
p (t _ {j} \mid m) = f _ {t y p e} (m, t _ {j}) = \frac {1}{1 + \exp (- V _ {m} ^ {\top} \cdot V _ {t _ {j}})} (3)
$$
Because of equations (4) and (3), we call this model as neuro-symbolic. While training the above model, we need to prepare our data so as to ensure a property called upward closure of entity types.
[Upward Closure of Entity Types]: If  $t_j$  is given to be a type of an entity  $e_i$  in a training example then all the nodes on the path from  $t_j$  till root must also be considered as its valid types. For e.g., suppose  $\{t_9\}$  is given as a valid type for entity  $e_i$  as shown in Figure 2. Then, we must augment its type set by adding all the ancestor nodes of  $\{t_9\}$ . This results in  $\{t_4, t_9\}$  as the upward closed type set for  $e_i$ .
Model Training: For training our model parameters  $\theta_{m},\theta_{e}$  and  $\theta_t$  we define loss  $\ell_{zel}$  for the primary task and  $\ell_{type}$  for the auxiliary task.
$$
\ell_ {z e l} (\theta_ {m}, \theta_ {e}) = - \sum_ {(m _ {i}, e _ {i}) \in \mathcal {D} _ {t r a i n} ^ {z e l}} \log p (e _ {i} | m _ {i})
$$
$$
\begin{array}{l} \ell_ {t y p e} (\theta_ {m}, \theta_ {t}) = - \sum_ {(m _ {i}, e _ {i}) \in \mathcal {D} _ {t r a i n} ^ {z e l}} \sum_ {t _ {j} \in \mathcal {T}} \mathbb {1} _ {e _ {i}} (t _ {j}) \log p (t _ {j} | m _ {i}) \\ + \left(1 - \mathbb {1} _ {e _ {i}} (t _ {j})\right) \log \left(1 - p (t _ {j} | m _ {i})\right) \\ \end{array}
$$
where,  $\mathbb{1}_{e_i}(t_j)$  is an indicator variable capturing whether  $t_j$  is a valid type of entity  $e_i$  or not. The loss function  $\ell_{zel}$  is similar to the bi-encoder loss function used in BLINK (Wu et al., 2020). The combined loss for the two tasks is given by
$$
\ell_ {m t l} (\theta_ {m}, \theta_ {e}, \theta_ {t}) = \ell_ {z e l} (\theta_ {m}, \theta_ {e}) + \alpha \ell_ {t y p e} (\theta_ {m}, \theta_ {t})
$$
where  $\alpha$  is a hyperparameter.
Thus, learning of NeSLET involves solving the following optimization problem.
$$
\begin{array}{l} \theta_ {m} ^ {*}, \theta_ {e} ^ {*}, \theta_ {t} ^ {*} = \arg \min  \ell_ {m t l} \left(\theta_ {m}, \theta_ {e}, \theta_ {t}\right) \tag {4} \\ (\theta_ {m}, \theta_ {e}, \theta_ {t}) \\ \end{array}
$$
Observe, the parameter  $\theta_{m}$  is common across both the tasks' loss terms. Due to this, both these tasks get tied together during training and the ETP task induces a bias in the hypothesis selection for the EL task. If there are multiple equally good EL hypotheses (i.e. model parameters  $\theta_{m}$  and  $\theta_{e}$ ), the inductive bias forces model to pick an hypothesis that does well on the ETP task. Such inductive bias helps in better generalization for the EL task across new domain even when trained with the less data. We train EL and ETP tasks jointly using hard parameter sharing (Sun et al., 2020; Ruder, 2017) strategy. In this strategy, we train  $\theta_{m},\theta_{e},\theta_{t}$  simultaneously. The backpropagation scheme for this scenario is depicted in Figure 3.
Inference: Given a mention  $m$ , the trained EL model is used to predict entities in a ranked order.
Figure 3: Training strategy with hard parameter sharing.
Datasets and Type Hierarchy: Table 2 summarizes various benchmark datasets used in our experiments. We use Wiki BLINK (Wu et al., 2020) and Wiki FGET (Onoe and Durrett, 2020) as two different datasets for the training domain. Each of these datasets are based on Wikipedia. For the test domain, we use two benchmark datasets - WikilinksNED UM (NED for short) (Onoe, 2020; Onoe and Durrett, 2020) and CoNLL-YAGO (CoNLL for short) (Hoffart et al., 2011; Max Planck Institute for Informatics, 2013). For the Wiki FGET dataset, we have two variants available - one for each of the test domains. These two variants were prepared by the authors so as to ensure a good amount of entity types being covered between train and test domains where entity types were taken from Wikipedia categories. For each entity across all datasets, we use DBpedia (Lehmann et al., 2015) to get the entity types. DBpedia ontology (dbp, 2020) contains 769 types including root type owl:Thing. These types are arranged in a tree structure having 7 levels and 611 leaves.
Domain	Dataset	# Examples	# Unique Entities
Train	Val.	Test	Train	Val.	Test
Train	Wiki BLINK	9M	10K	-	1.49M	8.7K	-
Wiki FGET (NED)	5.6M	3K	-	1.3M	2.8K	-
Wiki FGET (CoNLL)	6M	5K	-	1.14M	4.5K	-
Test	NED	-	10K	10K	-	2.3K	2.5K
CoNLL	-	4791	4485	-	1.6K	1.5K