Polar Ducks and Where to Find Them: Enhancing Entity Linking with Duck Typing and Polar Box Embeddings
Authors: Mattia Atzeni
Mikhail Plekhanov
Frederic A Dreyer
Nora Kassner
Simone Merello
Louis Martin
Nicola Cancedda
Venue: EMNLP 2023 Main (2023)
Keywords: Entity linking
Entity disambiguation
Box embeddings

Abstract: Entity linking methods based on dense retrieval are widely adopted in large-scale applications for their efficiency, but they can fall short of generative models, as they are sensitive to the structure of the embedding space. To address this issue, this paper introduces DUCK, an approach to infusing structural information in the space of entity representations, using prior knowledge of entity types. Inspired by duck typing in programming languages, we define the type of an entity based on its relations with other entities in a knowledge graph. Then, porting the concept of box embeddings to spherical polar coordinates, we represent relations as boxes on the hypersphere. We optimize the model to place entities inside the boxes corresponding to their relations, thereby clustering together entities of similar type. Our experiments show that our method sets new state-of-the-art results on standard entity-disambiguation benchmarks. It improves the performance of the model by up to 7.9 F1 points, outperforms other type-aware approaches, and matches the results of generative models with 18 times more parameters.

Polar Ducks and Where to Find Them: Enhancing Entity Linking with Duck Typing and Polar Box Embeddings
Mattia Atzeni Mikhail Plekhanov', Frédéric A. Dreyer', Nora Kassner', Simone Merello', Louis Martin' Nicola Cancedda"
Meta AI, 2EPFL
Abstract
Movies
Entity linking methods based on dense re- trieval are widely adopted in large-scale ap- plications for their efficiency, but they can fall short of generative models, as they are sensi- tive to the structure of the embedding space. To address this issue, this paper introduces DUCK, an approach infusing structural in- formation in the space of entity representa- tions, using prior knowledge of entity types. Inspired by duck typing in programming lan- guages, we define the type of an entity based on relations with other entities in knowl- edge graph. Then, porting the concept of box embeddings to spherical polar coordinates, we represent relations as boxes on the hypersphere. We optimize the model to place entities in- side the boxes corresponding to their relations, therehy clustering together entities of similar type. Our experiments show that our method sets new -feth--rrt results on standard qaiitydiiambiiguation benchmarks. It improves the performance of the model by up to 7.9 Fi points, outperforms other type-aware ap- proaches, and matches the results of generative models with 18 times more parameters.
cast acreeneeter
Screenswiter
Casst member
Fantastic beasts
Jaws
where to find them
Countries
Capptal
popusetion
cumency
osguisation
Italy
Germany
Figure 1: Examples from Wikidata showing how, fol- lowing the concept of duck ryping, relations in knowl- edge graph can help identifying entities of different types (e.g., movies and countries).
on dense retrieval (Plekhanov et al., 2023; Botha et al., 2020; Ayoola et al., 2022), as they are easier to train and can be more than one order of magni- tude faster (Ayoola et al., 2022). These approaches learn to represent entities and mentions separately in the same embedding space, that, inference time, the method only requires encoding the men- tion and retrieving the most similar entity. Methods based on dense retrieval have the drawback of be- ing very sensitive to the structure of the embedding space, thereby reaching lower accuracy compared to generative models (Aghajanyan et al.. 2022).
1 Introduction
State-of-the-art approaches to entity linking. namely the task of linking mentions of entities in text the corresponding entries in knowledge base (KB) (Ferragina and Scaiella, 2010; Ganea and Hofmann, 2017), are nowadays large genera- tive models (Aghajanyan et al., 2022; De Cao et al., 2021) which perform entity retrieval in a autore- gressive way. This category of methods achieves the best results, as it allows effectively capturing relations between the context of a mention and en- tity descriptions. However, the preferred choice in large-scale applications are often methods based
In this paper, aim to close the gap with gener- ative approaches by infusing structural information in the latent space of retrieval-based methods. Re- cent work (Mulang et al., 2020; Ayoola et al., 2022) has shown the benefit of infusing prior factual knowledge in the models. In particular, Raiman and Raiman (2018) reported that prior knowledge of the type of mention would result in nearly per- feet disambiguation performance. Prior methods used type labels extracted from knowledge graphs (KGs) (Ayoola al., 2022; Chen et al., 2020; Orr et al., 2021), but as KGs can be highly incomplete, we aim to define type information in a fuzzy and more -rained manner.
"Corresoondence to mattia.. aeni eeeen@@epf...mm ch and ncan @metta. com. Work done while the first author was an intern at Mcta Al.

---

Mention
The first time a World Cup finsl was settled penally shootout was 1994, when Italy loet Brazil.
Mention Encoder
Madrid
Knowledge Graph
Head coach
Sport
Entities
Hoad
Population
The Italy national football team has represented Italy in football since 1910.
Italy national footbalf toam
Sport national football team
Population
Relation Encoder
football team
Italy national
Italy
Entity Encoder
Sport
Flag
Itafy...
Italy, officially the Republic or the
Republic of Italy, is country in Europe.
Italy
Flag...
Germany
Coach
Head
Population
Real Madrid
Germany
CF
Figure 2: Entity disambiguation flow in DUCK. A mention encoder and entity encoder learn to represent mentions and entity descriptions respectively. Following the concept of duck ryping, relations a knowledge graph are used determine entity types. Relations are represented as box embeddings in spherical polar coordinates and the model is optimized to place entities inside the boxes corresponding to their relations.
We achieve this goal by drawing inspiration from the concept of duck ryping in programming lan- guages, which relies on the idea of defining the type of an object based on its properties. Extend- ing this idea to the realm of KGs, we define the type of an entity based on the relations that it has with other entities in the graph. Figure/ 1 shows examples demonstrating how relational infor- mation from KG like Wikidata (Vrandečié and Krötzsch, 2014) can help identifying entities of different types without any need for type labels.
matches the overall results of more computation- ally intensive generative models, with 18 times more paramcters than DUCK. Our ablation studies show that incorporating type information using box embeddings in polar coordinates improves the per- formance by up to 7.9 micro-Fi points with respect to the same model trained without duck typing. Fi- nally, qualitative analyses support the intuition that our method results in clear clustering of types and that DUCK is able predict the relations of an entity despite the incompleteness of the KG.
Motivated by this intuition, we propose DUCK (Disambiguating Using Categories extracted from Knowledge), an approach to infusing prior type in- formation in the latent space of methods based on dense retrieval. Building on recent work on region- based representations (Dasgupta et al., 2020; Ab- boud et al., 2020), we introduce box embeddings in spherical polar coordinates and employ them model relational information, as shown in Fig- ure 2. We define this representation as it naturally aligns with the use of the dot product (or the cosine similarity) as the similarity function for entity and mention embeddings (see Section 3), which is the prevalent choice in dense-retrieval methods. Then, we optimize the model to structure the latent space in such a way that entities fall within the boxes corresponding to their relations, that entities that share many relations (which are assumed be of the same type) will be clustered together.
2 Preliminaries
We start by formalizing the entityddisaniiguation problem, then we outline the main intuitions under- lying methods based dense-retrieval.
Problem statement. The goal of entity disam- biguation (ED) is to link entity mentions in piece of text to the entity they refer to in a reference KB. For each entity we assume we have an entity description expressed kens S. = (st" s...l,, Similarly, each men- tion m is associated with a sequence of tokens (1) ( representing the mention it- Sm = (sm .m.... self and its context. We denote the entity a mention m refers to as e" Further, we assume that the reference KB is a knowledge graph G = (E,R), where & is set of entities and R. is a set of rela- tions, namely boolean functions r & x E {0, 1) denoting whether a relation exists between two tities. Then, given set of entity-mention pairs D {Im) e ..... (mp)) eimp: } we aim learn a model : E, such that the entity predicted by the model for a given mention én = f(m is the correct entity e
sequence of to-
We use our approach to train a bi-encoder model with the same architecture Wu et al. (2020). Our experiments show that DUCK sets new state-of- the-art results on standard iisttyiiiannbiguation datasets, exceeds the performance of other type- aware models (trained on 10 times more data), and

---

Dense-retrieval methods. Methods based on dense retrieval (Wu et al., 2020) learn repre- sent mentions and entities in the same latent space, often optimizing a eross-entropy loss of the form:
this similarity to rank entities for a given mention. the norm of the mention embedding irrelevant, whereas the entity norms encode a "prior" over en- tities. Therefore, we choose to represent relations as boxes spherical polar coordinates, as shown in Figure 2. This representation allows guaraniceing that the cosine of the angle between two embed- dings falling in the same region is constrained by the boundaries of the box. At the same time, it keeps boxes open on the radial coordinate, so as leave the training free use entity norms to encode prior probabilities without interference from type information. Concretely, we parameterize the box corresponding to relation as pair of vectors:
Len(m) = -s(m, (m) + log Σ Eexp(s(m,ee)),,
where a similarity function between entities and mentions. This objective encourages the represen- tation of mention m to be close to the representation of the correct entity en and far from other entities ej. according to the similarity This similarity function s(m,e) is usually chosen be the dot product between learned representations m, e IRd of the mention and entity respectively. At inference time, a mention encoded in the dense space of entity embeddings and the entity with the highest similarity is returndd.
Box(r) (,)))
where 0,, E Rd-1 R are vector of angles denot- ing respectively the bottom-left and top-right cor- ners of the box in spherical coordinates. For an en- tity E &, say that E Box(r), if the expression in polar coordinates Q. of the entity representation between or and Ø across all dimensions. Then, our goal is to structure the latent space such a way that e E Box(r') for every r E R(e) and e & Box(r") for every r (R(e).).
3 DUCK: enhancing entity disambiguation with duck typing
Our approach builds on dense-retrieval methods and aims to enhance their performance using fine- grained type information.
3.1 Modeling fine-grained type information from knowledge graphs
3.2 Duck typing as an optimization problem In order to achieve the goal mentioned above, we need to turn the intuition of Section 3.1 into an optimization problem. To this end, it helps to define distance function between an entity and a box.
Duck typing on knowledge graphs. Duck typ- ing is a well-known concept in dynamically typed programming languages and is based the overall idea of weakly defining the type of an object based on its properties. Extending this concept KGs, without any need for type labels, we can describe the type of an entity e C terms of the set of re- lations labeling the edges originating from e in the KG. With slight abuse of notation, we will denote this set as R(e) = {r R 3e' E : An example of how the set of relations of an entity can be used to determine its type is shown in Fig- ure 2. For qualitative analysis showing how duck typing works in real-world knowledge graphs, we refer the reader to Appendix A.
Entity-box distance. Following Abboud et al. (2020), who defined similar function for box em- beddings in cartesian coordinates, we define the distance between an entity and a box as:
[I( 0,)/ 1)ll2
ifee Box(r) otherwise,
l}.
disi(e, r) [1. 0,10 x)lla
where ,- = (0, Φ )/2 is the center of the box corresponding to relation r, 8, = --- Φ, is a vector containing the width of the box along each dimension, is the Hadamard product, / is element- wise division, and K is vector of width hidth-dependent scaling coefficients defined as:
Relations as polar box embeddings. Inspired by region- based representations (Vendrov et al., 2016;
8, 0 8 K 2
Lai and Hockenmaier, 2017), and particularly by box embeddings (Vilnis et al., 2018; Li et al., 2019; Dasgupta et al., 2020), regions of the space. Our similarity function s, the dot product, is the product of the two norms of the entity and mention embeddings and the cosine of the angle between them. Since we are using
+1 1).
8
represent relations as
Intuitively, this function heavily penalizes entities outside the box, with higher distance values and gradients, whereas mildly pushes entities lying already inside the box towards the center. We refer the reader to Appendix B for more details.

---

Loss function for typing. To encourage an en- tity E & lie inside all boxes representing the relations Re) and outside the other boxes, we use a negative negative-sampling loss similar to the one of Sun et al. (2019). Our loss function is defined as: (e) = - E.- [log (iyddist(err"))) E,- [log o (dist(e,r 7
4.1 Bi-encoder
Bi-encoders, introduced in this context by Wu et al. (2020), are an efficient architecture for approaches based on dense retrieval. These methods rely on two different encoders fentity and fmention to repre- sent entities and mentions respectively.
Entity encoder. Given a textual description of an entity E EE expressed as a sequence of tokens S,- = learn entity representation
Above, Y IR a margin parameter, is the sig- moid function, r+ is a relation of entity e, drawn uniformly from the set of relations R(e), whereas I is a relation drawn from the set of relations R (e) according to the probability distribution:
(se (1) .....Se !!!l,)!%.l) E Rd as:
f (Se).
Concretely, following prior work (Wu et al., 2020), we extract entity descriptions S from Wikipedia, and structure each description Se using the ti- tle of the Wikipedia page associated with entity followed by the initial sentences of the body of the page, separated by a reserved token. We trun- cate entity descriptions se to a maximum sequence length of ng- For the entity encoder femitty we used a pre-trained RoBERTa model (Liu et al., 2019), resorting to the encoding of the [CLS] token for the final entity representation
exp(-a disi(e,r, ))
p(r (e) E, ER a dis((e,r)))
where a E [0, is a temperature parameter. The lower a, the closer the distribution is to a uniform distribution, whereas higher values of a result in more weight given to boxes that are close to the entity. Notice that this objective forces the distance between an entity and relations r+ R(e) to be small, while keeping the entity far from boxes cor- responding to the negative relations r-. Hence, optimizing the objective Duck will result in clus- tering together entities that share many relations.
Mention encoder. We model a mention as a se- quence of tokens Sn. - (5m sm. ((mml)) denoting both the mention itself and the context surrounding it, up to maximum mention length nm- Following Wu et al. (2020), we used reserved tokens to denote the start and the end of mention and separate it from the left and right context. We then calculate mention representations as:
Overall optimization objective. We train the model to optimize jointly the amiit-diismbiguattio loss of Section 2 and the duck-typing loss D Although we defined the loss Liuck for entities, we calculate it for mentions well, defining the set of relations of a mention based on the ground-truth entity R(m) = R(e,,) order to prevent boxes from growing too large during training, we further introduce an L2 regularization term l on the size of the boxes:
m mention (5m):
where smention is a mention encoder based on a pre-trained RoBERTa model and the final mention representation m obtained using the encoding of the [CLS] token. Overall, our bi-encoder is the same as the one used by Wu et al. (2020), with the only difference that we rely on RoBERTa instead of BERT (Devlin et al., 2019) the underlying language model.
E, 8. 8,].
d
Then, final optimization objective is:
C(m) (ZD(m) + ADuck (Duck(ett)) (Dme(()) 2412).
where ADuck 242 E [0, I] are hyperparameters defin- ing the weight of each component of the loss.
4.2 Relation encoder
4 A bi-encoder model with duck typing
Relation modeling. We model a relation r E as sequence of tokens s, (s.") .....,/!o.l).. These sequences are extracted from Wikidata (Vrandečić and Krötzsch, 2014), using the English label of the property and its description, separated by a reserved token. We used the same mapping from Wikipedia titles to Wikidata identifiers of De Cao
Building prior work, we used the method de- scribed in Section 3 to train bi-encoder model with the same architecture of Wu et al. (2020). Compared to Wu et al. (2020), DUCK adds just a relation encoder which is only used at training time to represent relations as boxes.

---

al. (2021). Based on Sy, we then compute relation embedding for each relation r re R. as: freiation (5r),
Inference. At inference time, our approach is not different from the method of Wu al. (2020). We simply match mention m to the entity that maximizes the similarity function s:
where Fretation is a relation encoder similar to fentiry and Smention. which computes the relation repre- sentation as the embedding of the [CLS] ] token produced by pre-trained RoBERTa model.
pm arg sag eCEn
where &m C & is set of candidate entities for men- tion m. In practice, we can precompute all entity embeddings, so that inference only requires one forward pass through the mention encoder and se- lecting the entity with the highest similarity.
Learning boxes in polar coordinates. Given relation representation calculated as described above, we parametrize box as a pair of vectors Box(r) = (P7, 97), where:
P (FFN (r))
5 Experiments
P P, + mmin + OFFN ( p, Smin).
This section provides a thorough evaluation of our approach. First, we show that DUCK achieves new statc-of-the teef-thc-atttt results popular datasets for entity disambiguation, closing the gap between retrieval- based methods and more expensive generative mod- els. Then, we discuss several ablation studies, showing that incorporating type information us- ing box embeddings in polar coordinates improves the performance of the model. Finally, we dig into qualitative analyses, showing that our model is able to place entities in the correct boxes despite the in- completeness of the information in the KG.
Above, FFN and FAN are 2-layer feed-forward networks, O is the sigmoid function, and ômin is margin parameter denoting the minimum width of a box across any dimension. Calculating the cor- ners of a box in this manner allows us to achieve two main objectives: (i) all components of o, and o, range from 0 to π, hence they assume valid val- ues in the spherical coordinate system, and (ii) o is greater than 9, across all dimensions, so that boxes are never empty and the model does not have to learn how to produce non-degenerate regions. Notice that, in spherical coordinate system, only one of the coordinates is allowed to range from to 2n, while all remaining coordinates will range from to π. For simplicity, we constrain all coor- dinates in the interval [0, π]. thereby reducing all representations to half of the hypersphere.
5.1 Experimental setup
We reproduce the same experimental setup of prior work (De Cao et al., 2021; Le and Titov, 2019): us- ing the same datasets, the same candidate sets, and comparing the models based on the InKB micro-Fi Fi score. Following De Cao al. (2021); Wu et al. (2020), we train the model on the BLINK data (Wu et al., 2020), consisting of 9M mention-entity pairs extracted from Wikipedia. Entity descriptions are taken from the Wikipedia snapshot of Petroni et al. (2021). Then, we measure in-domain and out-of- domain generalization by fine-tuning the model on the training set of the AIDA-CoNLL dataset and evaluating six test sets: AIDA (Hoffart et al., 2011). MSNBC (Cucerzan, 2007), AQUAINT (Milne and Witten, 2008), ACE2004 (Ratinov al., 10, CWEB (Gabrilovich et al., 2013) and WIKI (Guo and Barbosa, 2018).
4.3 Training and inference
Training. We train DUCK by optimizing the over- all objective defined in Section 3. In order to com- pute the loss Duckk we calculate the representa- tions Pe. Pm E d by converting to spherical co- ordinates the entity and mention representations and m produced by the entity and mention encoders respectively. To make training more efficient, the relation representations r are pre-computed and kept fixed at training time. We use the dot prod- uet between entity and mention representations to evaluate the entity disambiguation loss LED:
s(e.m) = m.
5.2 Entity disambiguation results
The expectations in the loss Lruck are estimated across all relations yt E R(e) and by sampling k relations r E R R(e) according to p(r e). The L2 regularization on the width of the boxes is performed across all relations a batch.
We compared DUCK against three main categories of approaches: (a) methods based on dense re- trieval, (b) generative models, and (c) type-aware models, namely other approaches to adding type

---

Method
MSNBC
AQUAINT
ACE2004 CWEB
WIKI
Avg.
Dense retrieval
Ganea and Hofmann, 2017
92.2
93.7
88.5
88.5
77.9
77.5
86.4
Yang al., 2018
95.9
92.6
89.9
88.5
81.8
79.2
88.0
Shahbazi ct al., 2019
93.5
92.3
90.1
88.7
78.4
79.8
87.1
Yang al., 2019
93.7
93.8
88.2
75.6
78.8
86.7
Le and Titov, 2019
89.6
92.2
90.7
88.1
78.2
81.7
86.8
Fang al., 2019
94.3
92.8
87.5
91.2
78.5
82.8
87.9
Wu et al., 2020"
79.6
80.0
80.3
82.5
64.2
75.5
77.0
Generative models
De Cao al., 2021
93.3
94.3
89.9
90.1
77.3
87.4
88.8
Aghajanyan al., 2022 (CM3-Medium))
94.2
90.1
90.4
76.5
86.9
88.6
Aghajanyan al., 2022 (CM3-Large)
94.8
94.8
91.1
91.4
78.4
88.7
89.8
Type-aware models
Chen et al., 2020
94.5
90.8
78.2
81.0
86.7
Orr et al., 2021
80.9
80.5
74.2
83.6
70.2
76.2
77.6
Ayoola et al., 2022 (Wikipedia)
87.5
94.4
91.8
91.6
77.8
88.7
88.6
Ayoola et al., 2022 (fine-tueed)
93.9
94.1
90.8
90.8
79.4
87.4
89.4
DUCK (Wikipedia)
91.0
95.1
91.3
95.4
76.9
86.1
89.3
DUCK (fin--uued))
93.7
94.6
91.3
95.0
78.2
85.9
89.8
Table I: Micro-Fi (InKB) results on six sedit-ddisambiguation datasets. Bold indicates the best model, underline indicates second best results. Our results are highlighted gray. 'Model without candidate set, results from De et al. (2021). Ressult from Ayoola et (2022).
information to retrieval-baan methods (DUCK per- tains to this category). We report the results both for the model trained only on the BLINK data and for the model fine-tuned AIDA, referring the former as "DUCK (Wikipedia)" and to the latter as "DUCK (fne-tuned)).
Knowledge-aware methods. DUCK uses knowledge graph (Wikidata) infuse additional in- formation in the model. While some methods listed indeed type information extracted from Wikidata (Ayoola al., 2022; Orr al., 2021; Chen al., 2020), other existing knowledge-awan methods for entity disambiguation have reported results in different experimental settings, evaluat- ing on AIDA, with the candidate set of Pershina et al. (2015). In order to compare with these meth- ods, evaluated DUCK the candidate set of Pershina al. (2015), and report the results in Table 2. Interestingly, our model outperforms both DeepType (Raiman and Raiman, 2018) and the methods of (Mulang et al., 2020) and Onoe and Durrett (2020). State-of- results in this
in Table
Main results. Table shows the performance of DUCK in comparison other methods. First, notice that DUCK obtains state-of-the-an results on MSNBC and ACE2004, second best performance on AQUAINT, and state-of-the-ar results on aver- age across all datasets. We also observe that DUCK outperforms all the other type-aware models, show- ing the effectiveness of our approach to define type information and infuse it in the model. In addi- tion, it is worth noticing that DUCK exceeds the results of generative models like GENRE (De Cao et al.. 2021) and CM3-Medium. This impressive considering that generative models are notoriously more expensive than bi-encoder models and require one order of magnitude more time per mention at inference (Ayoola et al., 2022). Finally, we see that DUCK meets the performance of CM3-Large (Aghajanyan et al., 2022), a generative model that, with its 13 billion parameters, is almost 5 times larger than -M3-Medium (2.7 billion parameters) and more than 18 times larger than DUCK (717 million parameters).
Method
AIDA
Onoe and Durrett (2020)
85,9
Raiman and Raiman (2018)
94.9
Mulang et al. (2020)
94.9
Ayoola (2022) (Wikipedia)
89.1
Ayoola et al. (2022) (fine-tuned)
97.1
DUCK (Wikipedia)
94.3
DUCK (fine-tuned)
96.4
Table 2: Miero-F, (InKB) results of knowledge-aware methods on the candidate set of Pershina et al. (2015).

---

Method
AIDA
MSNBC
AQUAINT
ACE2004
CWEB
WIKI
Avg.
As before, 8'in is a margin parameter that defines the minimum width of a box, FFN and FFN+ are feed-forward networks, and ReLU((x = max (0,x) is the ReLU activation function. Finally, we report the results obtained by DUCK when no candidate set is provided (DUCK w/o candidate set). In this case, we score each mention against the whole set of entities (which amounts almost 6M entities). DUCK w/o types (Wikipedia)
85.0
93.1
87.5
87.5
73.6
84.5
85.2
DUCK cartesian coord. (Wikipedia)
90.6
94.9
91.3
95.0
76.5
85.1
88.9
DUCK w/o candidate set (Wikipedia)
87.4
89.9
85.2
88.8
69.1
82.0
83.7
DUCK (Wikipedia)
91.0
95.1
91.3
95.4
76.9
86.1
89.3
Table 3: Micro-Fi results achieved by several ablations of DUCK
setting are obtained by Ayoola et al. (2022), con- firming the overall good performance obtained by this method on AIDA in Table 1. However, no- tice that (a) Ayoola al. (2022) trained the model on a custom Wikipedia dump, consisting of 100M mention-entity pairs (more than one order of mag- nitude larger than our dataset) and (b) DUCK ob- tains excellent results even in an out-of-domain scenario (without fine-tuning AIDA), reaching 94.3 micro-Fi points (an improvement of 5.1 points with respect to Ayoola et al., 2022).
results further show the benefit of using spherical coordinates and that the model achieves good per- formance even without candidate set.
5.4 Qualitative analyses
This section complements the quantitative results discussed so far with some qualitative analyses.
Analysis of the boxes. Table 4 shows qualita- tive analysis of the relative placement of entities and boxes in the latent space. In the left side of the table, we looked into three boxes correspond- ing to the relations flag, sport, and director, and we reported the top 10 entities that are closer to the center of the box. The examples show a clear clustering of types, as all entities closer to the box flag countries, entities inside the box sport are sport teams, and entities inside the box director are movies. For the latter box, we observe that the model predicts two movics that, in Wikidata, are missing the relation director, showing the ability of the model to robustly deal with incomplete in- formation. In the right side of Table 4, we show which boxes are closer to three entities, namely a country, football team and movie, according to the distance function defined in Section 3. The examples show that the model is able to correctly place entities of different types in different boxes.
5.3 Ablation studies
In order to provide more insights into the perfor- mance of the model, we performed several ablation studies. First, we performed an ablation where we removed the contribution of the L terms and the L2 regularization /2 from the loss function (DUCK w/o types). In this case, we only train the model using the miittydisammbiuuatoon loss LED. without infusing any type information. In addition, we assessed the benefit of using box embeddings in spherical polar coordinates by experimenting with version of the model where boxes are ex- pressed in cartesian coordinates (DUCK cartesian coord). In this case, we parametrize box as a pair of vectors Box(r) = (r r where
= FFN (r), r= = + ReLU (r)) &
Examples. Figure shows examples of the en- tities predicted by our method, for inputs where the prediction of DUCK differs from the ablation that does not use type information. The first two examples (left and center), clearly show how type information can help the disambiguation in cases where some keywords in the context of the men- tion misleads the model to making wrong predic- tion. The third example (right) shows a case where DUCK predicts correctly the type of the mention, but fails to leverage some contextual information and links a wrong entity. This likely due to the two entities sharing most boxes and being very close in the embedding space.
Table 3 shows the results achieved by the ab- lations described above. Including entity types boosts the performance by approximately micro- Fi points and up to 7.9 points on ACE2004. The

---

Italy national
Fantastic Beasts and Where Find Them
Flag
Sport
Director
Italy
football team
Czech Republic
The Invinolbles (en. lootblll)
A Secret Life (flm)
country
aport
orig. lang. of film or TV show ganre
France
New England Tea Men
The Dream (1959 film
GeoNames
head coach
Austria
EMKA Racing
The Prodigal (1983 film)
flag image
country
publication date form creative work
Poland
Angeles Heroes
Time Sing (flm)
logislative body
inoeption
Sweden
Hamlinn Pipers football
Modern Romance (film)
cat. for ppl. born here
cat. for memb. of team
language of work or name main subject
Maxico
La Máquina
Sinful Life
locator map image
Facebook
Germany
A. J. Foyt Enterprises
The Devil's Arithmatic (fim)
Commons gallary
owned by country for sport league
distributed by director
D. of Congo
Atéérica Minaro
Unchained (film
shares border with continent named after
Italy
Artlach (cycling team)
As is (fil)
cast member
Argentina
PS Berito Putera U-20
Identity Criels (fimm
social media followers
spoken taxt audio
Table 4: Qualitative analysis of the boxes predicted by DUCK. The left side of the table shows the closest entities a box, ranked according to the distance function defined in Section 3. The right part of the table shows the closest boxes to given entity. Correct predictions are highlighted in green, whereas predictions that do not match relations in Wikidata are highlighted in red. Best viewed in color.
the new England captain. [..] Shearer takes the captaincy trial basis, but eoach Gienn Hoddle said he saw no reseon why the foreer Blackburn and Southampton shipper should not make the post his own.
SOCCER [..] Alan Shearar was
BOXING PANAMA'S ROBERTO DURAN FIGHTS THE SANDS OF TIME:
SOCCER ROMANIA LITHUANIA
UNDER 21 MATCH.
Panamanian bexing legend Roberio
[... Romania beat Lithuania (halflime
Hands Stone Duran climbs into the ring
1) in their European under 21 soccer match on Fridayy [--}
on Saturday another defying attempl
to sustain his long career [..].
DUcK:
Foctbell
DUCK: Blackbum Rovers F.C.
Footbail
DUCK: Panama v/
Country
Lithuenia national football team X
leam
Baxing trainor
DUCK w/o types:
Footbal
DUCK types: Eddie Blackbum
Footballer
DUcK w/o types: Panams Lewis
Lithusnie net. under-21 footbell leom V
Team
Figure 3: Examples of the predictions of DUCK and DUCK w/o types, showing cases where DUCK predicts the
correct entity (left and center) and where it predicts a wrong one (right). Mentions are highlighted bold green.
6 Related work
to infuse prior knowledge in neural models (Lake et al., 2017: Atzeni et al.. 2023). Several methods have been proposed to achieve this goal, like infus- ing commonsense knowledge extracted from KGs in attention-basse models (Bosselut et al., 2019; Murugesan et al., 202 b,),,,a), constraining attention weights in transformers using graph-structured data (Sartran et al., 2022), and improving reasoning abil- ities of language models with graph neural net- works (Yasunaga et al., 2021; Atzeni et al., 2021).
Our work builds on top of the bi-encoder architec- ture of Wu et al. (2020) and was partially motivated by the work of Raiman and Raiman (2018), who showed the benefit of using type information for entity disambiguation. Previous research has em- ployed variety of methods to model mentions and entities using neural networks (He al., 2013; Sun et al., 2015; Yamada et al., 2016; Kolitsas et al., 2018). Our method falls within a recent line of work that has proposed approaches to use type in- formation in the disambiguation process (Raiman and Raiman, 2018; Khalife and Vazirgiannis, 2019; Onoe and Durrett, 2020; Chen et al., 2020; Orr al., 2021: Ayoola et al., 2022). The closest method to DUCK is the one of Ayoola al. (2022), who incorporated type knowledge in bi-encoder model similar to the one of Wu et al. (2020). The main difference between DUCK and the model of Ayoola et al. (2022) is that they used type labels extracted from Wikidata instead of our duck-typing approach, they represented types as points in the latent space, and they improved the performance of the model by using global entity priors (i.e.. prior probabilities of an entity given a mention) ex- tracted from count statistics. Broadly speaking, our method falls within the scope of recent research
7 Conclusion
This paper introduced DUCK, a method to improve the performance of entity disambiguation models using prior type knowledge. The overall idea un- derlying our method was inspired by the concept of duck typing, as we defined types in fuzzy manner, without any need for type labels. We introduced box embeddings in spherical polar coordinates and we demonstrated that using this form of represen- tation allows effectively clustering entities of the same type. Crucially, we showed that infusing structural information in the latent space is suffi- cient close the gap between efficient methods based on dense retrieval and generative models. As a future line of research, it might be interesting to explore methods to infuse prior knowledge of entity types in generative models as well.

---

Limitations
and Li et al., 2019) in order to get rid of the margin parameter and optimize the model using cross- entropy loss, but we obtained better results with the method described in the paper.
Our method assumes that have access to both entity descriptions in natural language and knowl- edge graph providing relations between pairs of entities. Methods based on dense retrieval (without type information) usually rely only on the first as- sumption. In our experiments, entity descriptions are obtained from Wikipedia (more precisely, from the KILT dump of Petroni et al., 2021) and we rely on Wikidata (Vrandečić and Krötzsch, 2014) the underlying KG. In domain-specific applications. one of the two sources of information (typically the KG) might not be available. However, notice that all existing type-aware methods have a similar limitation, as they require type labels at training time. We believe that other forms of structured knowledge might be used in some cases to obtain the attributes needed to represent the type of tity. Also, we point out that training on Wikipedia is a very common choice and many real-world ap- plications rely on the same setup employed this paper (Plekhanov et al., 2023; Ayoola et al., 2022). Compared to other type-aware methods, DUCK has the disadvantage that we cannot predict the type of mention in the form of label. This is a design choice that allows modeling type infor- mation in more fine- meeigiaieed manner. As shown in the paper, this choice results in better overall satity-diammbiuuttion performance compared to other type-aware methods. In applications where it would be interesting to obtain the type of men- tion in the form of label, we believe that simple heuristic correlating the relations of an entity to its type in Wikidata would be very effective. We refer the reader to Appendix A for insights on the type information carried by relations in a KG.
Finally, our loss function does not optimize only for entity disambiguation. Hence, DUCK might occasionally loose contextual information, in favor of placing the mention in the correct boxes. This is shown the rightmost examples of Figure 3 and Figure 5. In both cases, the correct entity and the prediction have lexically similar descriptions and share many relations. Therefore, their embeddings are close, making the disambiguation task more difficult. We noticed empirically that, when this happens, the model might be biased towards the more common entity. For instance, in the example of Figure 3, DUCK predicts the main national team over the under-21 team, whereas in the example of Figure 5, the model favors the football team over the basketball one.
Ethical considerations
Entity disambiguation is well-known task in nat- ural language processing, with several real-world applications in different domains, including con- tent understanding, recommendation systems, and many others. As such, it is of utmost importance to consider ethical implications and evaluate the potential bias that ED models could exhibit. DUCK is trained on Wikipedia and Wikidata (Vrandečić and Krötzsch, 2014), which can carry bias (Sun and Peng, 2021). These biases may be related to geographical location (Kaffce et al., 2018; Beytía, 2020), gender (Hinnosaar, 2019; Schmahl et al., 2020), and marginalized groups (Worku et al.. 2020). Since at inference time DUCK is essentially bi-encoder architecture based on dense retrieval, the index of entity embeddings can be updated with- out retraining the model. This would allow incorpo- rating efforts to reduce bias in Wikipedia efficiently. Another source of bias in DUCK (and in many downstream tasks natural language processing) is the underlying pre-trained language model used to initialize the entity and mention encoders. In our experiments, we used the RoBERTa model of Liu et al. (2019). Steed et al. (2022) show that bias mit- igation needs be performed on the downstream task directly, rather than on the language model. We refer the reader to Rudinger et al. (2018) and Zhao et al. (2018) for methods to mitigate bias in downstream tasks.
Additionally, we emphasize that the choice of spherical polar coordinates for modeling relational information is dependent on the use of the dot product or the cosine similarity as the function for ranking the closest entities given mention. In case a different function is used (c.g., the L2 distance), then box embeddings in cartesian coor- dinates might be better suited. We used the dot product because it is the most popular choice, al- lowing us to build on the model of Wu et al. (2020).
One more caveat is that our method is sensitive to the margin parameter y. In case DUCK is trained on different domains, might be beneficial tune this parameter carefully. We tried using probabilistic box embeddings (similar to Dasgupta et al., 2020

---

References
AAAI 2020, The Thirty-Second Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7529-7537. AAAI Press.
Ralph Abboud, Ismail Ilkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. Boxe: box embedding model for knowledge hase completion. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Silviu Cucerzan, 2007. Large-scale named entity dis- ambiguation based on wikipedia data. In EMNLP- CoNLL 2007, Proceedings of the 2007 Joint Con- ference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, June 28-30, 2007, Prague, Czech Republic, pages 708-716. ACL.
Armen Aghajanyan, Bernie Huang. Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis. and Luke Zettlemoyer. 2022. CM3: A causal masked multimodal model of the internet. CoRR, abs/2201.07520..
Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang. Luke Vilnis, Xiang Li, and Andrew McCal- lum. 2020. Improving local identifiability in prob- abilistic box embeddings. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020. December 6- 2020, virtual.
Mattia Atzeni, Jasmina Bogojeska, and Andreas Loukas. 2021. SQALER: scaling question answering by de- coupling multi-hop and logical reasoning. In Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, 1, NeurlPS 2021, December 6-14, 2021, virtual, pages 12587-12599.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Eyent, Austria, May 3-7, 2021. OpenReview.net.
Mattia Atzeni, Mrinmaya Sachan, and Andreas Loukas. 2023. Infusing lattice symmetry priors in attention mechanisms for sample-efficient abstract geometric reasoning. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1200-1217. PMLR.
Jacob Devlin, Ming-Wäs Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re- fined: An efficient zero-shot-capable approach to end-to-end entity linking. In NAACL 2022.
Pablo Beytía. 2020. The positioning matters: Estimat- ing geographical bias in the multilingual record of biographies on wikinedia. In Companion of The 2020 Web Conference 2020, Taipei. Taiwan, April 20-24, 2020, pages 806-810. ACM/ IW3C2.
Zheng Fang, Yanan Cao, Qian Li, Dongjie Zhang, Zhenyu Zhang, and Yanbing Liu. 2019. Joint en- tity linking with deep reinforcement learning. In The World Wide Web Conference, WWW 2019, San Fran- cisco, CA, USA, May 13-17, 2019, pages 438 447. ACM.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4762-4779. Association for Computational Linguis- tics.
Paolo Ferragina and Ugo Scaiclla. 2010. TAGME:
on-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM Conference on Information and Knowledge Manage- ment, CIKM 2010, Toronto, Ontario, Canada, Octo- ber 26-30, 2010, pages 1625-1628. ACM.
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. En- tity linking in 100 languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proeesing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 7833-7845. Association for Computational Linguistics.
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. FACC1: Freebase annotation of clueweb corpora.
Octavian-Eugee Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619 2629, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Shuang Chen, Jinpeng Wang, Feng Jiang, and Chin- Yew Lin. 2020. Improving entity linking by mod- eling latent entity type information. The Thirty- Fourth AAAI Conference on Artificial Intelligence,

---

Zhaochen Guo and Denilson Barbosa. 2018. Robust named entity disambiguation with random walks. Se- mantic Web, 9(4):599479,
Brenden M. Lake, Tomer Ullman, Joshua B. Tenen- baum, and Samuel J. Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and Brain Sciences, 40re253.
Zhengyan Hc, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning entity representation for entity disambiguation. In Proceed- ings of the S1st Annual Meeting of the Association for Computational Linguisties ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 2: Short Papers, pages 50-34.15 The Association for Computer Linguistics.
Phong Le and Ivan Titov, 2019. Boosting entity linking performance by leveraging unlabeled documents. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1935- 1945, Florence, Italy, Association for Computational Linguistics.
Marit Hinnosaar. 2019. Gender inequality in new dia: Evidence from wikipedia. Journal of economic behavior & organization, 163:262-276.
Xiang Li, Luke Vilnis, Dongxu Zhang. Michael Bo- ratko, and Andrew McCallum. 2019. Smoothing the geometry of probabilistic box embeddings. In 7th In- terntiinnll Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Johannes Hoffart, Mohamed Amir Yosef, llaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Process- ing. EMNLP 2011, 27-31 July 2011, John MeIntyre Conference Centre, Edinburgh, UK, A meeting of SIGDAY, Special Interest Group of the ACL. pages 782-792. ACI..
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ 1692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scall similarity search with gpus. JEEE Trans. Big Data, 733):535-547.
David N. Milne and Ian H. Witten 2008 Learning link with wikipedia. In Proceedings of the 17th ACM Conference on Information and Knowledge Manage- ment, CIKM 2008, Napa Valley, California, USA, October 26-30, 2008 pages 509-518, ACM.
Lucie-Aimée Kaffee, Hady ElSahar, Pavlos Vougiouklis, Christophe Gravier. Frédérique Laforest, Jonathon S. Hare, and Elena Simperl. 2018. Learning generate wikipedia summaries for underserved languages from wikidata. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Short Pupers), pages 640-645. Association for Computational Linguistics.
Isaiah Onando Mulang, Kuldeep Singh, Chaitali Prabhu, Abhishck Nadgeri, Johannes Hoffart, and Jens Lehmann. 2020. Evaluating the impact knowl- edge graph context on entity disambiguation models. In Proceedings of the 29th ACM International Con- ference on Information and Knowledge Management, CIKM '20, page 2157-2160, New York, NY, USA. Association for Computing Machinery.
Sammy Khalife and Michalis Vazirgiannis. 2019. Scal- able graph-based method for individual named en- tity identification. In Proceedings of the Thirteenth Workshop on Graph-Bac Based Methods for Natural Lan- guage Processing, Tex:Graphs @ EMNLP 2019, Hong Kong, November 4, 2019, pages 17-25. Association for Computational Linguistics.
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapani- pathi, Pushkar Shukla, Sadhana Kumaravel, Gerald Tesauro, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. 2021a. Text-based RL agents with commonsense knowledger New challenges, en- vironments and baselines. In Thirty-Fifth AAAI Con- ference on Artificial Intelligence, AAAT 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelli- gence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 9018-9027. AAAI Press.
Nikolaos Kolitsas, Octavian-Fuge Fugen Ganea, and Thomas Hofmann. 2018 End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computa- tional Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Alice Lai and Julia Hockenmaier. 2017. Learning to predict denotational probabilities for modeling entail- ment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics, EACL 2017, Valnncia, Spain, April 3-7, 2017, Volume I: Long Pupers. pages 721-730. Association for Computational Linguistics.
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapani- pathi, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. 202 Efficient text-based rein- forcement learning by jointly leveraging state and commonsense graph representations. In Proceedings of the 59th Annual Meeting of the Association for

---

Computational Linguisties and the 11th International Joins Conference on Natural Language Processing. ACI/JUCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 719-725. Association for Computational Linguistics.
Oregon, USA, pages 1375-1384, The Association for Computer Linguistics.
Rachel Rudinger. Jason Naradowsky, Brian Leonard. and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-III.T, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 8 14. Association for Computational Linguisties.
Yasumasa Onoe and Greg Durrett. 2020. Fine-grained entity typing for domain independent entity linking. In The Shirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8576-8583. AAAI Press.
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Milos Stanojevic. Phil Blunsom. and Chris Dyer. 2022. Transformer grammars: Augmenting trans- former language models with syntactic inductive bi- ases at scale. Trans. Assoc. Comput. Linguisties, 10:1423-1439.
Laurel J. Orr, Megan Leszezynski, Neel Guha, Sen Wu, Simran Arora, Xiao Ling, and Christopher Ré. 2021. Bootleg: Chasing the tail with self--uperrised named entity disambiguation. In 1 1ith Conference on In- novative Data Systems Research, CIDR 2021, Vir- tual Event, January 1 11-15, 2021, Online Proceedings. www.cidrdbborgg.
Katja Geertruida Schmahl, Tom Julian Viering, Stavros Makrodimitris, Arman Naseri Jahfari, David Tax, and Marco Loog. 2020. Is Wikipedia succeeding in ducing gender bias? assessing changes gender bias in Wikipedia using word embeddings. In Proceed- ings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 94-103, Online, Association for Computational Lin- guistics.
Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized page rank for named entity disambigua- tion. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 238 243, Denver, Colorado. Association for Computational Linguistics.
Hamed Shahbazi. Xiaoli Z. Fern, Reza Ghacini, Rasha Obeidat, and Prasad Tadepalli. 2019. aatity- elmo: Learning contextual entity representation for entity disambiguation. CoRR, 1td//990..5762..
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. Proceedings gs of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2523-2544. Associa- tion for Computational Linguistics.
Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael L. Wick. 2022. Upstream mitigation is not all you need: Testing the bias transfer hy hypothssis in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Compu- rational Linguistics (Volume 1: Long Papers). ACL 2022, Dublin, Ireland, May 22-27, 2022. pages 3524- 3542. Association for Computational Linguistics.
Mikhail Plekhanov, Nora Kassner, Kashyap Popat, Louis Martin, Simone Merello, Borislav Kozlovskii, Frédéric A. Dreyer, and Nicola Cancedda. 2023. Multilingual end to end entity linking. CoRR, abs/2306.08896.
Jiao Sun and Nanyun Peng. 2021. Men are elected, women are married: Events gender bias on wikipedia. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing. ACL/JCCLLP 2021, (Volume 2: Short Papers). Virtual Event, August 1-6, 2021, pages 350-360. Association for Computational Linguistics.
Jonathan Raiman and Olivier Raiman. 2018. Deeptype: Multilingual entity linking by neural type system evolution. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educa- tional Advances in Artificial Intelligence (EAAI-18). New Orleans, Louisiana, USA, February 2-7, 2018, pages
Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity dis- ambiguation. In Proceedings of the Twenty-Fond International Joint Conference on Artificial Intelli- gence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 1333-1339. AAAI Press.
AAAI Press.
Lev-Arie Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In The 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies, Proceed- ings of the Conference, 19-24 June, 2011, Portland,
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019, Rotate: Knowledge graph embedding by relational rotation in complex space. 7th In- ternational Conference on Learning Representations,

---

ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, NAACI-IILT 2021, Online, June 6-11, 2021. pages 533-546. Association for Computational Linguistics.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-cmbeddings of images and lan- guage. In 4th International Conference on Learning Representations, ICIR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. 2018. Probabilistic embedding knowl- edge graphs with box lattice measures. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, pages 263- 272. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Or- donez, and Kai- Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and dehiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Technologies. NAACL-HLL, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 15-20. Association for Computational Linguistics.
Denny Vrandečić and Markus Krötzsch, 2014. Wiki- data: A free collaborative knowledgebase. Commum. ACM, 57(10):78-85 85.
Zena Worku, Taryn Bipat, David W. McDonald, and Mark Zachry. 2020. Exploring systematic bias through article deletions on wikipedia from behav- ioral perspective. In Proceedings of the 16th Interna- tional Symposium on Open Collaboration, OpenSym '20. New York, NY, USA. Association for Computing Machinery.
Ledell Wu, Fabio Petroni, Martin Josifoski. Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 6397- 6407. Association for Computational Linguistics.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the em- bedding of words and entities for named entity disam- biguation. In Proceedings of the 20th SIGNLL Con- ference Computational Natural Language Lear- ing, CONLL 2016, Berlin, Germany, August 11-12, 2016, pages 250-259. ACL.
Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, and Xiang Ren. 2019. Learning dynamic context augmentation for global entity linking. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 271-281, Hong Kong. China. Association for Computational Lin- guistics.
Yi Yang. Ozan Irsoy, and Kazi Shefaet Rahman. 2018. Collective entity disambiguation with structured gra- dient tree boosting. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-IHI.T 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pages 777-786. Association for Com- putational Linguistics.

---

A Duck typing on knowledge graphs
3.5
6, 1.0
To get more insights into our definition of duck typing on knowledge graphs, we performed qual- itative analysis of entities that share large number of relations in Wikidata (Vrandečić and Krötzsch, 2014). Precisely, we used the cardinality of the symmetric difference between the sets of relations two entities e1.e2 E &, defined as:
6, 1.5
3.0
6, = 2.0
6, 2.5
2.5
2.0
distte,
1.5
di)) = =R(ee)
1.0
= (((ee))\R(ee))U((RRR ((ei)) (((()) (R(e)))R()) (RG)))))
0.5
as a measure of the distance between the types of two entities é and e2- Notice that the dis- tance defined above can be expressed as the Ham- ming distance between binary encodings of the sets of relations, hence we can efficiently retrieve the neighbors of given entity GPU, following the method of Johnson et al. (2021). If our definition of duck typing works well, we expect entities with low distance to be likely of the same type. Table 5 shows the top-10 neighbors that minimize the distance function defined above for several entities. We emphasize that these lists of neighbors are not produced by our model, rather they are examples of the prior knowledge that aimed infuse in DUCK. This analysis shows that notion of duck typing carries fine-grained type information, as allows detecting countries, cities, highly influential computer scientists and mathematicians, football players, singers, politicians, animals, companies, scientific awards and more.
0.0
Figure 4: Plot of the entity-box distance in the uni- dimensional case, for boxes centered at x/2 with differ- scalar widths 8,
correlates with box size. This allows maintaining low distance values inside large boxes while pro- viding gradient to keep points inside. For entities outside their target boxes, the distance linearly cor- relates with the width of the box, to penalize points outside larger boxcs more severely.
C Details the model
In order to train DUCK, need to convert the entity representations into spherical polar coordi- nates (the same applies the mention representa- tions m). This can be done as follows:
B Entity-box distance
e1
This section provides more details on the entity- box distance function defined in Section 3.2. A plot of the distance function in the uni-dimensional case, for scalar entity representation and sev- eral boxes centered n/2 with different scalar widths is shown Figure 4. The plot shows that the distance function has different slopes for entities inside and outside the boxes. This meant to strongly penalize entities that lie outside boxes corresponding to their relations, it ensures that outside points receive high gradient through which they can more easily reach their target box. Addi- tionally, recalling the expression for re) given in Section 3.2, notice that the distance depends on the width of the box. More precisely, whenever an entity is inside its target box, the distance inversely
Pe,1 arccos
e
+
e
+
+ep
e2
2..2 arccos
2
eg
+
cd-2
Pe.f arccos
[ + e3
+ e2
arccos
if 2
Vej 1
Pc.d-1
2.T arecos
if <0

---

Italy
London
Rome
Alan Turing
Ada Lovelace
Portugal
Istanbul Madrid
Milan
Bernhard Riemann
Lady Byron
Spain
Florence
Kurt Gödel
Rosalind Franklin
Norway
Province
Naples
John Neumann
Elzzbett Fry
Greece
Stockholm
Venice
Herbert A, Simon
Catherine Dickens
Poland
Turin
John Forbes Nash
Lytton
Denmark
Buenos Aires
Palermo
Claude Shannon
Lady Emmeline Stuart-Worlle
Belgium
Manchester
Rio Janeiro
Nikolai Lobachevsky
Elsenor Mark
Hungery
Amsterdam
Genoa
Benolt bandeibrot
Winnemiiaa Powlett, Duchess of Cleveland
Finland
Bologna
Willard Van Quine
Rachel Russell, Lady Russell
Republic of Ireland
Ankara
Lisbon
Nels Henrik Abel
Martha Jefferson
Cristiano Ronaldo
Justin Bieber
Donald Trump
Lion
Jaguar
Lionel Messi
Harry Slyles
Joe Biden
Tiger
Cougar
Luis Sušrez
Chris Brown
Mrs. Bill Clinton
Leopard
Ocelot
Gerard Piqué
Ed Sheeran
Barack Obama's
Cheetah
Giant anleater
Neymar
Eminem
George W. Bush
Jaguar
Giant armadillo
Manuel Neuer
Camila Cabello
Gore
Red panda
Chimpanzee
Paul Pogba
Nick Jonus
Kamala Harris
Giant panda
Bonobo
Ronaldinho
Jordin Sparks
Elizabeth Warren
Cougar
Indian rhinpoeros
Luka Modrič
Richard Marx
Bill Clinton
Okapi
Giant otter
Antoine Griezmann
Niall Horan
Michael Bloomberg
Hapoootaamus
Black rhinoceros
Gareth Bale
Shawn Mandes
Benjamin Netanyahu
Fernec fox
Pronghorn
Jaguar Cars
Maserati
Vellko Tarnovo
Gracilinanus
Fields Medal
Land Rover
Lancia
Kluczbork
Scolomys
IEEE Medal of Honor
Stayr-Daimlr-Puue
VinFast
Yambol
Aethalops
Kavli Prize
MG Cars
McLaren Automotive
Kyustendi
Oligoryzomys
Rosenstlel Award
Gulf
Massimo Dutti
Targovishte
Raphicerus
Paul Ehrlich and Ludwig Dermetaedter Prize
Brilish Motor Corporation
Paper Mate
Pančavo
Vesper mouse
Albert Einstein World Aiarr of Science
Saleway (UK)
Intinili
Kragujevac
Rhabdomys
Canada Gairdner International Award
Rover Group
Peroni Brewery
Ruse, Bulgaria
Balantiopierys
Earle K. Plyler Prize Molecular Spectroscopy
Rover Company
Lotus Cars
Sombor
Arielulus
Dannie Heineman Prize for Matheratical Physics
MIPS Technologies
Colruyt (supermarket)
Vratse
Bassariacus
Böcher Memorial Prize
W. Woolworth Company
Overkill Software
Pazardzhik
Microsclurus
NAS Award Chemical Sciences
Table 5: Top 10 entities with the most similar set of relations given entity in Wikidata
where e; is the i-th entry of the entity representa- tion e E Rd and Pe, is the i-th component of the representation in spherical coordinates Φ. R2 Looking at the equations above, notice that in spherical coordinate systems, all angles range from 0 to π, with the only exception of the last coordi- nate Pe.d-1. which ranges from 0 π if ea posi- tive and from æ to 2% otherwise. In order to make the definition of the boxes and of the entity-box dis- tance simpler, we decided to constrain the last coor- dinate in the range [0, π as well. We achieved this objective by constraining the last coordinate of the entity and mention representations to be positive, applying absolute value to the last dimension of the output of the entity and mention encoders. This essentially restricts all representations and boxes to be on half of the hypersphere, more precisely on the portion where ed > 0. We apply this transfor- mation before computing the overall optimization objective of Section 3.2.
First, we trained the model using, as negative entities for each mention, all entities in the same batch. In order to provide more meaningful infor- mation, we further added entities that maximize prior probability p(e)m), extracted from count statistics derived from large text corpora. In de- tails, we used the prior probabilities of Ayoola al. (2022). Notice that, differently from Ayoola et al. (2022), we do not use these prior probabilities at inference time, but we only use them to provide better negative entities to the model in this first training stage. For each mention, we included in batch 3 negative entities that maximize the prior probability, limiting the total number of entities in batch to 32 (per GPU). In this stage, we use a batch size of 16 mentions (per GPU), which means that, for some mentions, we do not have the entities that maximize the prior probability. This is not an issue, as we still use all entities in the same batch as negatives for every mention. To compute the loss Duck, we used a sampling temperature of a 0. Furthermore, in order to provide better training signal and counteract missing information in the knowledge graph, we only trained the model using entities that have at least relations.
D Training details
In order to train DUCK, we need to select negative entities e for the cnitt-disambiguation loss LED of Section Having high-quality negative entities is crucial to achieve high performance, thus we trained DUCK in several stages.
We trained the model for epoch on GPUs, validating on the BLINK validation set every 5000

---

Member of political party
Member of sports team
Cast member
Richard Nixon
Justin Moore (soccer)
Time to Sing (film)
Édouard Philippe
Scott Jones (Puerto Rican footballer)
A Secret Life (film)
Kaname Tajima
Blake Camp
The Dream (1989 film)
Laurent Fabius
Miles Robinson (soccer)
Can You Hear the Laughter? The Story of Freddie Prinze
Albert II, Prince of Monaco
Simon Thomas (soccer)
I Was a Teenage TV Terrorist
Joe Biden
Scott Wilson (footballer, born 1993)
A Sinful Life
François Hollande
Scott Jenkins (soccer)
The Morning After (1986 film)
Jean-Marc Ayrault
Ali Mohamed (footballer)
Cries Unheard: The Donna Yaklich Story
Ursula von der Layen
Scott Fraser (fontballer, born 1995)
My Sex Life... How Got into an Argument Enemies, Love Story (film)
Yasutomo Suzuki
Justin Willis (soccer)
Table 6: Closest Wikipedia entities to different boxes according to the entity-box distance function. Correct predictions are highlighted in green, whereas predictions that do not match relations in Wikidata are highlighted in red. Best viewed in color.
Member of political party
Member of sports team
Cast member
Ronald Reagan
Tayfun Korkut
The Crying Game
Jacques Chirac
Lilian Thuram
Ice Cold in Alex
George H. W. Bush
Scott Sanders (baseball)
Michael Collins (film)
Madeleine Albright
Roberto Carlos (footballer)
Viva Zapatal On the Waterfront
Deng Xiaoping
Tony Adams (footballer)
Bob Dole
Stuart McCall
Lawrence of Arabia (fili)
Masoud Barzani
Joakim Persson
My Turn (memoir)
Yasser Arafat
Cosmin Contra
Aidan Quinn
Bill Clinton
Todd Martin
Der Spiegel
Liam Neeson
Geoff Aunger
Julia Roberts
Table 7: Closest entities (extracted from the validation set of the AIDA dataset) to different boxes according to the entity-box distance function. Correct predictions are highlighted in green, whereas predictions that do not match relations in Wikidata are highlighted in red. Only 6 entities in AIDA have the relation Cast member and the model is able to correctly retrieve all of them, has shown above. Best viewed in color.
gradient steps. Then, we used the model that max- imizes the validation performance to produce a representation for every entity, and we mined the closest representations for each entity in Wikipedia. This step is usually referred to as hard-negative mining. We used these entities as negative exam- ples for the LED loss and trained the model again, starting from the same checkpoint employed for the aegative-mining stage. We used batch size of 16, with negative examples for each mention and up to 32 entities in batch. We increased the sampling temperature for the boxes to a = 0.5, keeping a threshold of at least 5 relations for each entity. We trained the model for one more epoch, validating every 5000 gradient steps as before.
gradient accumulation for steps. Furthermore, we increased the maximum length of a mention from 128 tokens to 512, and set the sampling tempera- ture to a = 1.0. In this final stage, we assumed the model had already learned to place entities in their target boxes, hence we used all entities in the dataset, regardless of the number of relations they have in Wikidata.
E Additional results
Table provides additional results obtained by the ablations of DUCK described in Section 5.3, af- ter fine-tuning on AIDA. For reference, we report the results of the main model as well. Overall, the experiments confirm what we observed in Sec- tion 5.3. The main model performs consistently better than all ablations across all datasets except ACE2004, where the model in cartesian coordi- nates obtains the same results achieved by DUCK (Wikipedia) in Table 1. Overall, we notice that in- fusing type information using duck typing improves downstream performance and using polar coordi- nates is beneficial over boxes cartesian coordi-
Finally, we repeated the hard-negative mining process and kept training the model for 10000 ad- ditional gradient steps, using batch size of 4, 5 hard negatives for each mention and up to 3 enti- ties that maximize the prior probability p(e)m m) (if distinct from the negatives). As we increased the number of negative entities, the batch size is sig- nificantly smaller than before. Therefore, we did

---

Method
AIDA
MSNBC
AQUAINT
ACE20M
CWEB
WIKI
Avg-
DUCK w/o lypes (ine-tned)
89.1
92.5
87.4
87.1
74.9
83,8
85.8
DUCK curtesian courd. (fine-tuned)
92.1
94.0
90.6
95.4
77.5
85,5
89.2
DUCK w/o candidate set (fine-tumed)
90.9
90.5
86.3
89.2
71.1
81,9
85.0
DLCK (fne-tured)
93.7
94.6
91.3
95.0
78.2
85.9
89.8
Table 8: Micro-Fi results achieved by several ablations of DUCK after fine-tuning AIDA
CRICKET: [.-] Australia will defend the Ashes in a six test series against England [..] in London [..]
Watched by an array former Olympic sprint champions at the Berlin grand prix meeting. Mitchell made brilliant
BASKETBALL [..]
Result in international basketbail
foumnmmett on Friday: Red
start in the 100 metres [..] and held off
Star (Yugoslavia) beat Dinamo
Balley's strong finish win in 10.08
(Russia) 92-90 [..].
seconds [..]]
DUCK: City
DUCK:
Football
London
DUCK: Berlin v
City
Red Star Belgrade x
team
DUCK w/o types:
Cricket Club
Sport
DUCK w/o types:
Basketball
London Cricket Club X
DUCK w/o types: Berlin Marathon
Eveni
KK Crvena zvezda
team
Figure 5: Further examples of the predictions of DUCK and DUCK w/o types. Mentions highlighted in bold green.
nates. Interestingly, the model is able to achieve a Fi score of 85.0 even without the candidate set, confirming the intuition that using type information structure the latent space is advantageous for the entit-disambiguation task.
one error in the top 10 predictions for the relation member of political party). On the other hand, the dataset includes only 6 entities that are movies (more precisely, entities with the relation cast mem- ber). Interestingly, the top-10 entities closer to the center of the box corresponding to the relation cast member are all the movies mentioned in AIDA. The remaining 4 entities listed in Table 7 include two actors (Aidan Quinn and Julia Roberts), sug- gesting that the embedding space carries semantic information and that actors are closer to movies than other entities.
F Additional qualitative results
Following the qualitative analyses Section 5.4, in this section we provide additional results and further examples.
Analysis of the boxes. Table 6 shows additional examples of the top-10 entities lying closer to the center of box. This analysis is performed all entities in Wikipedia (approximately 6M entities for the English language) and complements the examples reported on the left side of Table 4. In this case, we analyzed three more relations, namely member of political party, member of sports team, and cast member. The model correctly reports politicians for the first box, athletes for the second, and movies for the latter, confirming the cluster- ing of entity types that we noticed in Section 5.4. Additionally, the model appears robust to missing information in the knowledge graph, being able to predict the relation cast member for movies that are missing it in the KG.
Examples. Figure shows further examples of the predictions of DUCK and the ablation DUCK w/o types. Confirming the insights of Figure 3, the first two examples (left and center), show that DUCK is usually able to predict entities of the cor- rect type and how this can help the model in making the correct prediction. The third example (right) shows case where the model prediets wrong entity, it links the mention though the context clearly suggests that the correct entity should be basketball team instead. This suggests that, in some rare cases, DUCK might give too much weight to the prior knowledge about the relations of candidate entities, loosing knowledge coming from the description of the entity and from contextual information about the mention.
football team,
We performed the same analysis, using the same set of relation, on the entities appearing in the val- idation set of the AIDA dataset. The results are reported in Table 7. Since AIDA contains news articles, the dataset includes several mentions of politicians and athletes, and the model is able to correctly cluster the two types of entities (with only
G Hyperparameters and reproducibility
We trained DUCK using the AdamW optimizer (Loshchilov and Hutter, 2019) on 8 NVIDIA A100 GPUs, each with 40 GB of memory. Following

---

Ayoola et al. (2022), we initialized the learning rate to 0 and linearly increased up 1.00 10-3 over the first 5000 steps. To avoid catastrophic forgetting, we set a maximum learning rate of 1.00 x 10 when fine-tuning on AIDA. We lim- ited the number of entities in batch (which are used to compute the loss term LED) to 32 per GPU, but we shared entity representations across all de- vices when computing the loss, reaching effec- tive maximum number of entities of 32 256 per batch. We increased the maximum length of a mention to 512 tokens at inference time. Table 9 reports the values of all hyperparameters of the model for reproducibility of our results.
Hyperparameter
Value
Learning rate (max)
1.00 x 10-5
Learning rate warm-up steps
5000
2
Apack
0.1
As
0.1
Number of negative boxes &
512
8min
0.1
8
0.1
1024
a
See Appendix
Max entity length
128
Max mention length ne
See Appendix
Max relation length
256
Batch size
Appendix
Max num. entities batch (per GPU)
32
Table 9: Hyperparameter values of DUCK


Reviews:
[{"review_id": "H2EVrzzrMa", "reviewer": "Reviewer_VEWh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent on our paper and the overall positive feedback.\n\nWe agree that our main contribution is an addition of a loss term to model the type of an entity, based on a method inspired by duck typing and relying on polar box embeddings. We believe this is a strength of our approach, as it considerably boosts downstream performance while being independent of the model used for the mention and entity encoder. Due to its simplicity, our approach can be, in principle, combined with a wide range of methods based on dense retrieval. We chose to build on the work of Wu et al. (2020) as it is computationally efficient and widely used in real-world applications (Plekhanov et al. 2023; Ayoola et al. 2022).\n\nWe thank the reviewer for the question on the qualitative analysis. During our experiments, we noticed that boxes with a large fraction of incorrect entities often correspond to relations that, in Wikidata, are used to provide references to other knowledge bases (e.g., _“BabelNet ID”_). These relations do not carry type information, therefore it is difficult for the model to predict them based on the label and description of an entity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The applied methodology and the formulation of the task is clear and the limitations and the reason of the choice is very well described. The authors show multiple models under comparison, thus the setting of experiment is adequate. Ablation study and qualitative analysis are also a good contribution to the work.\n\nEven though the idea of using entity typing for ED I snot novel, the approach presented in the paper is interesting and the authors evaluate and show improvement (SotA and to the base approach) on multiple datasets. Moreover, it is less computationally costly, as it does not use generative models, but achieves similar results.", "reasons_to_reject": "This paper might be considered as an incremental study as it just adds another loss function to the main computation. However, duck typing along with polar box embeddings significantly improve the results, therefore, I find the amount of contribution sufficient for publication.\n\nAs a recommendation for quantitative analysis / error analysis, it would be interesting to look into relation boxes with mostly incorrect entities and try to understand what confused the model most?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors solve the Entity Disambiguation task by applying Duck typing and polar box embeddings. The authors represent entity types via relations, so that entities that share many relations are clustered together as the same type. Then the authors apply a bi-encoder model to a pair of a mention and a candidates entity and combine the loss function with the loss function of duck typing.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t6TPioVivE", "reviewer": "Reviewer_89ts", "summary": "", "questions": "- Given an entity, could you provide more details about relation modelling and relation encoder? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and the useful feedback. We address your main questions below.\n\n**Relation modeling.** We model relational information as follows. For each entity in Wikipedia, we look for the node corresponding to that entity in the Wikidata knowledge graph. Then, for each edge originating from that entity, we consider the relation labeling that edge and we model it using its label and description in English.\n\nAs an example, the entity “_Italy_” in Wikipedia corresponds to node _Q38_ in Wikidata. We used the mapping from Wikipedia titles to Wikidata identifiers in De Cao et al. (2021) (https://github.com/facebookresearch/GENRE). Node _Q38_ has several relations (see https://www.wikidata.org/wiki/Q38) each with a label and a description. For instance, one of these relations is property _P1082_ (https://www.wikidata.org/wiki/Property:P1082), whose English label is “_population_” and its description is “_number of people inhabiting the place; number of people of subject_”. We encode the label and description of a relation using a pre-trained RoBERTa model and we keep these embeddings frozen at training time. The relation encoder then is a feed-forward network that produces the corners of the boxes based on the pre-trained relation embeddings with the approach mentioned in Section 4.2. We will clarify these details in the final version of the paper.\n\n**Reproducibility.** As a note on reproducibility, we strived to include in the paper all details on the infrastructure used for our experiments and all hyperparameters in Appendix G. We further reported details on how the model was trained in Appendix D. All datasets used to train the model and evaluate it are publicly available and consistent with prior work. We downloaded the datasets from the [repository](https://github.com/facebookresearch/GENRE/blob/main/scripts_genre/download_all_datasets.sh) of De Cao et al. (2021). **We will open source the code of our method upon acceptance of the paper**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a critical problem and develops an intuitive and sensible in-depth analysis. \n- The paper is very well written, including nice clear examples, and appropriately discusses the state of the art.", "reasons_to_reject": "- I have an overall positive impression, except for the fact that the paper may still miss some discussions about how relations are  modelled and extracted from Wikipedia. ", "questions_to_authors": "- Given an entity, could you provide more details about relation modelling and relation encoder? ", "paper_topic_and_main_contributions": "In this proposal, a novel approach for entity linking is presented, called Ducks. This method utilizes a combination of relations from a knowledge graph and context to determine entity types. The entity's relations are represented using box embeddings in spherical polar coordinates. The mention and entity are optimized to position entities with similar types within the corresponding boxes of their relations.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ob7J6Ov7jL", "reviewer": "Reviewer_F4hB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the accurate review. We provide answers to your main comments below.\n\n**Reproducibility**. **We will publish the source code upon acceptance of the paper**, together with a clear documentation explaining how to reproduce our results. We strived to include in the paper all details on the infrastructure used for our experiments and all hyperparameters in Appendix G. We further reported details on how the model was trained in Appendix D. All datasets used to train the model and evaluate it are publicly available and consistent with prior work. We downloaded the datasets from the [repository](https://github.com/facebookresearch/GENRE/blob/main/scripts_genre/download_all_datasets.sh) of De Cao et al. (2021). \n\n**Requirement for relational information.** We agree that the requirement for both textual descriptions and a knowledge graph is a limitation of our method, as mentioned in the limitations section on page 9. Notice that all existing type-aware methods we are aware of, including Ayoola et al. (2022), Orr et al. (2021) and Chen et al. (2020), have a similar limitation (even stricter, as they require type labels). However, we point out that, in real-world applications, training on Wikipedia is a very popular choice due to the large coverage of this dataset. Several industrial applications rely on the same training setup described in this paper and can take advantage of the availability of both textual descriptions and a knowledge graph (Plekhanov et al. 2023, Botha et al., 2020; Ayoola et al., 2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. New type of embeddings --  box embeddings in polar coordinates is proposed.\n2. The proposed embeddings are  used for representing similarities of entities according to sets of their relations.\n3. The approach allows improving the results of entity linking over strong state-of-the art results. \n", "reasons_to_reject": "1. The approach is quite complex and difficult for reproduction. In the paper there are no promices to publish the code,\n2.  The method requires much information about an entity:  entity description from Wikipedia and the entity relation set from Wikidata. Such information can be not accessible in real tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is devoted to improving entity linking using polar box embeddings and relation encoding. The basic approach is based on dense retrieval. To improve entity matching, special encoding of relation sets is aded. The approach is tested on several datasets and shows average improvement in performance.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper well-written, easy to follow, and addressing an important problem. The reasons to reject are few and seem easy to address before camera ready. For example, publishing the source code could help readers grapple with the proposed method's complexity. During the author response period, the authors confirmed that they will publish the source code together with the paper.
Decision: Accept-Main