Learning Cross-Context Entity Representations from Text
Authors: Jeffrey Ling
Nicholas FitzGerald
Zifei Shan
Livio Baldini Soares
Thibault Févry
David Weiss
Tom Kwiatkowski
Venue:  (2020)
Keywords: entities
entity representations
knowledge representation
entity linking
entity typing

Abstract: Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we achieve a score of 87.3% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as "Scottish footballers", and can answer trivia questions such as "Who was the last inmate of Spandau jail in Berlin?".

Under review as a conference paper at ICLR 2020
LEARNING CROSS-CONTEXT ENTITY REPRESENTA- TIONS FROM TEXT
Anonymous authors Paper under double-blind review
ABSTRACT
Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases human readable encyclopedias tend to be entity-centric we investigate the use a all-in-the-blant task learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typ- ing benchmarks, including 64% error reduction over previous work TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) exist- ing entity linking benchmarks, where we match the state-of-the-ar on CoNLL- Aida without inking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain train- ing data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as Who was the last inmute of Spandau jail in Berlin?.
INTRODUCTION
A long term goal of artificial intelligence has been the development and population of entity- centric representation of human knowledge. Efforts have been made to create the knowledge repre- sentation with knowledge engineers (Lenat al., 1986) crowdsourcers (Bollacker et al. 20088. However, these methods have relied heavily on human definitions of their ontologies, which are both limited in scope and brittle in nature. Conversely, due recent advances in deep learning, we can now learn robust general purpose representations of words (Mikolov et al. (013)) and contex- tualized phrases (Peters et al., 2018) directly from large textual corpora. In particular, we observe that existing methods of building contextualized phrase representations capture a significant amount of local semantic context (Devlin et al.) 2019). We hypothesize that by learning an entity encoder which aggregates all of the textual contexts in which an entity is seen, we should be able to extract and condense general purpose knowledge about that entity.
Consider the following contexts in which entity mention has been replaced [MASK]:
the second woman in space, 19 years after [MASK].
[MASK], a Russian factory worker, was the first woman in space
[MASK], the first woman in space, entered politics....
As readers, we understand that first woman in space a unique identifier, and we are able to fill in the blank unambiguously. The central hypothesis of this paper is that, by matching entities to the contexts in which they are mentioned, we should be able to build a representation for Valentina Tereshkova that encodes the fact that she was the first woman in space, that she etc. and that we should be able to use these representations across a wide variety of downstream entity-centric tasks.
politician,

---

Under review as a conference paper at ICLR 2020
We present RELIC (Representations of Entities Learned in Context), table of independent entity embeddings that have been trained to match fixed length vector representations of the textual context in which those entities have been seen. We apply RELIC to entity typing (mapping each entity to its properties in an external, curated, ontology); entity linking (identifying which entity is referred to by textual context), and trivia question answering (retrieving the entity that best answers question). Through these experiments, we show that:
RELIC accurately captures categorical information encoded by human experts in the Free- base and Wikipedia category hicrarchies. We demonstrate significant improvements over previous work on established benchmarks, including a 64% error reduction in the TypeNet low data setting. We also show that given just few exemplar entities of given category such as Scotrish footballers we can use RELIC to recover the remaining entities of that category with good precision.
Using RELIC for entity linking can match state-of-the-art approaches that make use of non-local and non-linguistic information about entities. On the CoNLL-Aida benchmark, RELIC achieves 94.9% accuracy, matching the state-of-the-ar of Raiman & Raiman (2018), despite not using any entity linking-specific features. On the TAC-KBP 2010 benchmark RELIC achieves 89.8% accuracy, just behind the top ranked system (Raiman & Raiman 2018), which makes use of external knowledge bases, alias tables, and task- specific hand-engineered features.
RELIC learns better representations of entity properties if it is trained to match just the contexts in which entities are mentioned, and not the surface form of the mention itself. For entity linking, the opposite is true.
We can treat the RELIC embedding matrix as a store of knowledge, and retrieve answers to questions through nearest neighbor search. We show that this approach correctly answers 51% of the questions in the TriviaQA reading comprehension task (Joshi et al.) 2017) de- spite not using the task's evidence text at inference time. The questions answered correctly by RELIC are surprisingly complex, such as Who was the last inmate of Spandau jail in Berlin?
RELATED WORK
Entity linking The most widely studied entity-level task is entity linking mmapping each entity mention onto a unique entity identifier. The Wikification task (Ratinov et al. [011 [[ennnChnngg & Roth, 2013), in particular, is similar the work presented in this paper, as it requires systems map mentions to the Wikipedia pages describing the entities mentioned. There is significant previous work that makes use of neural context and entity encoders in downstream entity linking systems (Sun et al. 20155] Yamada et al.) 2016 2017 Gupta et al. 2017 Murty et al. [2018] Kolitsas et al., 2018), but that previous work focuses solely on discriminating between entities that match a given mention according to an external alias table. Here we go further in investigating the degree to which RELIC can capture world knowledge about entities.
Mention-level entity typing Another well studied task is mention-level entity typing (e.g. Ling & Weld, [0122] Choi et al., 2018). In this task, entities are labeled with types that are supported by the immediate textual context. For example, given the sentence 'Michelle Obama aftended her book signing', : Michelle Obama should be assigned the type author but not lawyer. Subsequently, mention-level entity typing systems make use of contextualized representations of the entity men- tion, rather than the global entity representations that we focus on here.
Entity-level typing An alternative notion of entity typing is entity-level typing, where each entity should be associated with all of the types supported by corpus. Yaghoobzadeh & Schütze (2015) and Murty et al.) (2018) introduce entity-level typing tasks, which we describe more in Section 5.2 Entity-level typing is important task in information extraction, since most common ontologies make use of entity type systems. Such tasks provide a strong method evaluating learned global representations of entities.
2

---

Under review as a conference paper at ICLR 2020
Using knowledge bases There has been strong line of work in learning representations of enti- ties by building knowledge base embeddings (Bordes et al. 2011 Socher et al.] 2013] Yang et al. 2014 Toutanova et al. 2016: Vilnis et al.] 2018), and by jointly embedding knowledge bases and information from textual mentions (Riedel et al. 2013 Toutanova et al. 2015: Hu et al. 2015). Das al. (2017) extended this work to the SPADES fill-n-thheblank task (Bisk al. 2016), which is a close counterpart to RELIC's training setup. However, we note that all examples in SPADES correspond to a fully connected sub-graph in Freebase Bollacker et al. (2008). Subsequently, the contents are very limited in domain and Das et al. (2017) show that it is essential to use the contents of Freebase to do well on this task. We consider the unconstrained TriviaQA task (Joshi et 2017). introduced in Section 5.5 to be better evaluation for open domain knowledge representations.
Fill-i--the-blank tasks There has been significant previous work in using fill-in-the the-blank losses to learn context independent word representations (Mikolov et al.) 2013). and context-dependee
word and phrase representations (Dai & Le 2013: Peters et al., 2018, Radford al. 2018 Devlin ct al. 2019). Cloze-style tasks, in which a system must choose which of few entities best fill blanked out span, have also been proposed as a method of evaluating reading comprehension (Her- mann et al. 2015 Long et al., 2016 Onishi et al. 2016). For entities, Long et al. (201)) consider a similar fil-in-the-blan
task as ours, which they frame as rare entity prediction. Yamada et al. (2016) and Yamada et al. (2017) train entity representations using fill-in-the- blank style loss and bag-of-words representation of mention contexts. Yamada et al. (2016) 2017) in particular take an approach that is very similar in motivation RELIC, but which focuses on learning entity repre- sentations for use as features in downstream classifiers that model non-linear interactions between small number of candidate entities. In Section [5.] we show that Yamada al. (2017)'s entity em- beddings are good at capturing broad entity types such as Tennis Player but less good at capturing more complex compound types such as Scortish Footballers. In Section 5.1 we also show that by performing nearest neighbor search over the 818k entities in the TAC knowledge base, RELIC can surpass Yamada et al. 20.720 performance on the TAC-KBP 2010 entity linking benchmark (Ji et al.) 2010). This is despite the fact that Yamada et al.) massively restrict the linking search space with an externally defined alias table, and incorporate task-specific hand- hand-engineered features. On the CoNLL-Aida benchmark, show that RELIC surpasses Yamada et al. 2017 and matches Raiman
& Raiman 2018 without using any entity linking-specific features.
LEARNING FROM CONTEXT
3.1 RELIC TRAINING INPUT
Let E = (co...eN) be predefined set of entities, and let == [[MASK], [Fx], [Fe]]. W1... ... ..M be a vocabulary of words. A context x [mo ... ..] a sequence of words 2; E V. Each context contains exactly one entity start marker zg = [E,] and one entity end marker æ; = [Fg] where j k >>>>>> 1. The sequence of words between these markers, 117411....--1/ is the entity mention.
Our training data is a corpus of (context, entity) pairs D [[xo, yo) ... (xN, WN)]. Each y; E E identifies an entity that corresponds to the single entity mention in x,- We train RELIC to cor- rectly match match the entities in D to their mentions. We will experiment with settings where the mentions are unchanged from the original corpus, as well as settings where with some probabil- ity (the mask rate) all of the words in the mention have been replaced with the uninformative [MASK] symbol. We hypothesize that this parameter will play a role in the effectiveness of learned representations in downstream tasks.
For clean training data, we extract our corpus from English Wikipedia F See Section 4 for details.
3.2 CONTEXT ENCODER
We embed each context in D into a fixed length vector using a Transformer text encoder (Vaswani ct al. 2017), initialized with parameters from the BERT-base model released by Devlin et al. 2019 All parameters are then trained further using the objective presented below in Section 3.4
htts://enn wttp://iiw.iiikiiia....oo
3

---

Under review as a conference paper at ICLR 2020
We take the output of the Transformer corresponding to the initial [CLS] token in BERT's sequence representation as our context encoding. and we linearly project this into R' using a learned weight matrix W E Rx 768 to get context embedding in the same space as our entity embeddings.
3.3 ENTITY EMBEDDINGS
Each entity e E & has a unique and abstract Wikidata QID RELIC maps these unique IDs directly onto dedicated vector in Rd via [&] x d dimensional embedding matrix. In our experiments, have a distinct embedding for every concept that has an English Wikipedia page, resulting in 5m entity embeddings overall.
3.4 RELIC TRAINING LOSS
RELIC optimizes the parameters of the context encoder and entity embedding table to maximize the compatibility between observed (context, entity) pairs. Let 9x) - IR be context encoder, and let f(e) + Id be an embedding function that maps each entity to its d dimensional representation via lookup operation. We define a compatibility score between the entity e and the context x as the scaled cosine similarity
sxx e) = a g(x) T T((e)
(1)
where the scaling factor is learned parameter, following Wang et al. (2018a). Now, given context x, the conditional probability that was the entity seen with x defined as
exp(s(x, e))
p(e[x) Eveg exp(s(x, e!))
and we train RELIC by maximizing the average log probability
logp(u|x).
D
(x.y)ED
In practice, the definition of probability in Equation is prohibitively expensive for large E (we
use [8] 5M). Therefore, we use a noise contrastive loss (Gutmann & Hyvärinen 2012 Mnih & Kavukcuoglu 2013). We sample K negative entities from noise distribution Proise ee::
é .,. .e.....,,. dx ~ (n)
(4)
Denoting 6 e, then compute per-example loss using cross entropy:
exp(s(x,e))
x(s = - llg exp(sxx((
(5)
In practice, we train our model with minibatch gradient descent and use all other entries in the batch negatives. That is, in batch of size 4, entities for rows 1, 2, 3 will be used negatives for row This roughly equivalent to Pnoise (c) being proportional to entity frequency.
EXPERIMENTAL SETUP
To train RELIC, obtain data from the 2018-10-22 dump of English Wikipedia. We take & to be the set of all entities in Wikipedia (of which there are over million). For each occurrence of hyperlink, we take the context as the surrounding sentence, replace all tokens in the anchor text with a single [MASK] symbol with probability m (see Section 5.3 for a discussion of different masking rates) and set the ground truth to be the linked entity. We limit each context sentence 128 tokens. In this way, we collect high-quality corpus of over 1 112M (context, entity) pairs. Note in particular that an entity never co-occurs with text on own Wikipedia page, since page will not hyperlink to itself. We set the entity embedding size to d 300.
We train the model using TensorFlow Abadi et al. 2016) with batch size of 8,192 for 1M steps on Google Cloud TPUs.
hhttps://www. hhtps://wwwwwkkddaaa.oom In experiments, found cosine similarity he more effective than dot product.
4

---

Under review as a conference paper at ICLR 2020
System
CoNLL-Aida
TAC- 2010
Sil al. 2018
94.0
87.4
Yamada et al. 2016
91.5
85.5
entity linking features
81.1
80.1
Yamada et al. 2017
94.3
87.7
Kadhakrishnan et 2018
93.0
89.6
Ramman & Rauman 2018
94.9
90.9
RELIC
81.9
87.5
RELIC + CoNLL-Aic tuning
94.99
89.8
Table I: RELIC achieves comparable results to best performing dedicated entity-linking systems de-
spite using no external resources or task specific features. When given a standard CoNLL-Aida alias table and tuned on the CoNLL-Aide training RELIC's learned representations match the state- of-the-art Type system which relies on the large hand engineered Wikidata knowledge base.
EVALUATION
We evaluate RELIC's ability to: (1) solve the entity linking task without access to any task spe- eific alias tables or features; (2) accurately capture entity properties that have been hand-coded into TypeNet and Wikipedia categories; (3) capture trivia knowledge specific to individual entities.
First we present results on established entity linking and entity typing tasks, compare RELIC's performance to established baselines and we show that the choice of masking strategy (Section 3) has significant and opposite impact on performance on these tasks. We hypothesize that RELIC approaching upper bound on established entity-level typing tasks, and we introduce much harder category completion task that uses RELIC to populate complex Wikipedia categories. We also apply RELIC's context encoder and entity embeddings to the task of end-to-end trivia question answering, and we show that this approach can capture more than half the answers identified by the best existing reading comprehension systems.
5.1 ENTITY LINKING
RELIC can be used to directly solve the entity linking problem. We just need to find the single entity that maximizes the cosine similarity in Equation for a given context. For the entity linking task, we create a context from the document's 's first 64 tokens as well as the 64 tokens around the mention to linked. This choice context is well suited the documents in the CoNLL-Aida and TAC-KBP 2010 datasets, since those documents tend to be news articles in which the introduction particularly information dense. In Table T we show performance for RELIC two settings. First, we report the accuracy for the pure RELIC model with no in in-domain tuning. Then, we report the accuracy for a RELIC model that has been tuned on CoNLI training set. On the CoNLL- Aida benchmark, we also adopt standard alias table (Pershina et al., 2015) for this tuned model, as is commonly done in previous entity linking work.
It is clear that for the CoNLL-Aida benchmark in-domain tuning is essential. We hypothesize that this is because of the dataset's bias towards certain types of news content that is very unlike our Wikipedia pre-training data specifically sports reports. However, when do adopt the standard CoNLL-Aida training set and alias table, RELIC matches the state of the art on this benchmark, despite using far fewer hand engineered resources (Raiman & Raiman (2018) use the large Wikidata knowledge base to create entity representations). We do not make use of the TAC-KBP 2010 training set alias table, and observe that RELIC is already competitive without these enhancement
is significant that RELIC matches the performance of Raiman & Raiman (018)), which uses the large hand engineered Wikidata knowledge base represent entities. This supports our central hypothesis that it is possible to capture the knowledge that has previously been manually encoded in knowledge bases, using entity embeddings learned from textual contexts alone. In the rest of
' *Our finetuned CoNLI result uses the alias table of Pershina al. 2015) at inference time.
'We do reduce the candidate set from the 5m entities covered by RELIC to the 818k entities in the TAC-KBP 2010 knowledge hase avoid ontological misalignment.
5

---

Under review as a conference paper at ICLR 2020
System
P@
Acc
Yaghoobzadeh et al. 2018
82.3
91.0
56.5
RELIC
87.9
94.8
68.3
RELIC with 5% of FIGMENT training data
83.3
90.9
59.3
Table 2: Performance FIGMENT. We report P@ (proportion of entities whose top ranked types are correct), Micro Fl aggregated over all (entity, type) compatibility decisions, and overall accuracy of entity labeling decisions. RELIC outperforms prior work, with only 5% of the training data.
System
TypeNet 78.6
TypeNet Low Data (5%)
Murty et al. 2018
58.8
RELIC
90.1
85.3
Table 3: Mean Average Precision TypeNet tasks. RELIC's gains are particularly striking in the low data setting from Murty al. (2018).
this section, we will show further support for our hypothesis by recreating parts of the Freebase and Wikipedia ontologies, and by using RELIC to answer trivia questions.
Finally, we believe that RELIC's entity linking performance could be boosted even higher through the adoption of commonly used entity linking features. As shown in Table Yamada et al. (2016) use a small set of well chosen discrete features to increase the performance of their embedding based approach by 10 points. These features could be simply integrated into RELIC's model, but we consider them to be orthogonal to this paper's investigation of purely learned representations.
5.2 ENTITY-LEVEL FINE TYPING
We evaluate RELIC's ability to capture entity properties on the FIGMENT (Yaghoobzadeh & Schütze 2015) and TypeNet (Murty et al. 2018) entity-level fine typing tasks which contain 102 and 1,077 types drawn from the Freebase ontology (Bollacker et al. 20088. The task in both datasets predict the set of fine- fine-grained types that apply given entity. We train a simple 2-layer feed- forward network that takes as input RELIC's embedding fee of the entity e and outputs binary vector indicating which types apply to that entity.
Tables klow that RELIC significantly outperforms prior results on both datasets. For FIGMENT, Yaghoobzadeh et al. (2018) is an ensemble of several standard representation learning techniques: word2vec -iip-gram contexts (Mikolov et al., 2013), structured skip-gram contexts (Ling et al.) 2013), and F Text representations of the entity names (Bojanowski et al. 2017). For TypeNet, Murty et al. (2018) aggregate mention-level types and train with a structured loss based on the TypeNet hierarchy, but is still outperformed by our flat classifier of binary labels. We expect that including hierarchical loss is orthogonal to approach and could improve our results further.
The most striking results in Tables 2 and are in the low data settings. On the low-data TypeNet setting of Murty et al. (0188) RELIC achives 63% error reduction over previous work, while RELIC also matches Yaghoobzadeh et al. 2018's results on FIGMENT with 5% of the training data.
5.3 EFFECT OF MASKING
In Section we introduced the concept of masking entity mentions, and predicting on the basis of the context in which they are discussed, not the manner in which they are named. Figures [ and 2 show the effect of training RELIC with different mask rates. It clear that masking mentions during training is beneficial for entity typing tasks, but detrimental for entity linking. This is in accordance with our intuitions. Modeling mention surface forms is essential for linking, since these mentions are given at test time and names are extremely discriminative. However, once the mention is known the model only needs to distinguish between different entities with the same name (e.g. President Washington, University of Washington, Washington State) and this distinction rarely requires deep knowledge each entity's 's properties. Subsequently, our best typing models are those that are forced to capture more of the context in which each entity is mentioned, because they not allowed to

---

Under review as a conference paper at ICLR 2020
100
Typelitt of
CONLL
95
Typeliet 5% of data TypeNct
0.8
0.775
0.830
0.809
0.781
90
88.6
89.0
89.7
0.734
86.1
83.2
83.8
84,9
0.6
0.612
79.2
0.4
75
73.5
74.6
0.2
0.170
70
69.2
0.685
0%
10%
50%
100%
0.0
Mask
Mask rate
Figure 1: TypeNet entity-level typing mAP on the development set for RELIC mod- els trained with different masking rates, A higher mask rate leads to better performance, both in low and high-data situations.
Figure 2: Entity linking accuracy for RELIC models trained with different masking rates. No alias table nor in-domain fine-tuning is used. Higher mask rates lead to worse down- stream performance in entity-linkiin tasks.
Yamada Subset
All Entities
TypeNet
Wikipedia
TypeNet
Wikipedia
# Entities
291,663
707,588
323,347
3,667,933
Random
2.7
0.1
2.5
0.1
Yamada et al. 2017
25.9
8.0
RELIC
27.8
21.0
29.3
13.8
Table 4: Mean average precision on exemplar-based category completion (Section 5.4 The Yamada subset is filtered to only contain entities that are covered by Yamada et al. 2017 and categories are filtered to those which contain at least 300 entities (131 categories). For the "All Entities" setting, we use all Wikipedia entities covered by RELIC, and filter to categories which contain at least 1000 entities (1083 categories). The embeddings learned by Yamada et al. 2017 are competitive with RELIC on the task of populating TypeNet categories, but they are much worse at capturing the complex, and compound, typing information present in Wikipedia categories.
rely on the mention itself. The divergence hetween the trends in Figures and and suggests that there of entity embeddings that are optimum for all tasks. However, would like
may not be to point out that that a mask rate of 10%, RELIC nears optimum performance on most tasks. The optimum mask rate is open research question, that will likely depend on entity frequency as well as other data statistics.
5.4 FEW-SHOT CATEGORY COMPLETION
The entity-level typing tasks discussed above involve in-domain training step. Furthermore, due to the incompleteness of the the FIGMENT and TypeNet type systems, we also believe that RELIC's performance is approaching the upper bound both of these supervised tasks. Therefore, to properly measure RELIC's ability to capture complex types from tll-in-the-blan training alone, we propose:
1. new category completion task that does not involve any task specific optimization,
2. Wikipedia category based evaluation set that contains much more complex com- pound types, such as Scottish footballers,
We this task compare RELIC the embeddings learned by Yamada al. 2017
In the new category completion task, we represent each category by randomly sampling three exem- plar entities. and calculating the centroid of their RELIC embeddings. We then rank all other entities according to their dot- product with this centroid, and report the mean average precision (MAP) of the resultant ranking.

---

Under review as a conference paper at ICLR 2020
Open-domain Uniel
Verified Web
Classifier baseline Joshi et al. 2017)
30.2
SLQA (Wang et al 2018D)
82.4
RELIC
35.7
51.2
ORQA (Lee et al. 2019)
45.0
Table 5: Answer exact match TriviaQA. RELIC's fast nearest neighbor search over entities achieves 80% of the performance of ORQA, which runs a BERT-based reading comprehesion model over multiple retrieved evidence passages. Unlike ORQA and RELIC, the classifier baseline and SLQA have access to a single evidence document that is known to contain the answer. As result they are solving much easier task.
First, apply this evaluation to the TypeNet type system introduced in (Murty et al. 2018). These types are well-curated, but tend to represent high-level categories. To measure the degree to which entity embeddings capture finer grained type information, we construct aditional dataset based on Wikipedia categorie These tend to be compound types, such as Actresses from London, which capture many aspects of entity-in this case gender, profession, and place of birth.
From Table 4 we can see that the embeddings introduced by Yamada et al. 2017 approach RELIC's performance the TypeNet completion task, but they significantly underperform RELIC in com- pleting the more complex Wikipedia categories. Figure Ba shows example reconstructions for ran- domly sampled Wikipedia categories, two from TypeNet and three from Wikipedia. Both models achieve high precision on TypeNet categories, but on the finer-grained Wikipeida categories, the Yamada et al. (2017) model tends to produce more broadly-related entites, whereas the RELIC em- beddings capture entities which are much closer to the exemplars. In fact, we identify several false negatives these examples.
5.5 TRIVIA QUESTION ANSWERING
Our final experiment tests RELIC's ability to answer trivia questions which can be considered high precision categories that only apply to single entity using retrieval of encoded entities. TriviaQA (Joshi et al. 2017) is question-answering dataset containing questions sourced from trivia websites, and the answers are usually entities with Wikipedia pages. The standard TriviaQA setup is reading comprehension task, where answers are extracted from evidence documents. Here, we answer questions in TriviaQA without access to any evidence documents at test time.
Model and training Given a question, we apply the context encoder g from Section 3.4 and re- trieve out of 5M entities using cosine similarity. For training. initialize both g and f from RELIC training. We tune only g's parameters by optimizing the loss in Equation 5Japplied to (ques- tion, answer entity) pairs, rather than the (context, entity) pairs seen during RELIC's training.
Results TriviaQA results are shown in Table 5 and randomly sampled RELIC predictions are illustrated in Figure [3b] All systems other than RELIC in Table [5] have access evidence text at inference time. In the open domain unfiltered setting, ORQA (Lee et al. 2019) retrieves this text from cache of Wikipedia. In the more standard verified-weh reading comprehension task, the classifier baseline and SLQA are provided with single document that is known to contain the answer.
We consider ORQA to be the most relevant point of comparison for RELIC. We observe that the retr-eve-then-read approach taken by ORQA outperforms the direct answer retrieval approach taken by RELIC. However, ORQA runs BERT based reading comprehension model over multiple idence passages at inference time and we are encouraged to see that RELIC's much faster nearest neighbor lookup captures 80% of ORQA's performance.
also significant that RELIC outperforms Joshi et al. (2017)* 's reading comprehension baseline by 20 points, despite the fact that the baseline has access to single document that is known to contain the answer. However, RELIC is still far behind the reading comprehension upper bound set
We use the Yago 3.1 Mahdisoltani et al. 2013 dump of extracted categories that cover least 1,000 entities, resulting in 1 1,083 categories.
8

---

Under review as a conference paper at ICLR 2020
Calegury and Exemplars
Yamada 2017
RELIC
tennia a player
Ekaterina Makarova
1. Prakush Amritraj
Goffin
Vera Zvonareva
2. Marco Chiudinelli
Yves Allegro
3. Flavia Pennetta
3. Mare Glequel
Hlvii Pennatta
Max Mirnyi
4. Marius Copil
5. Lisa Raymond AP=71.87
5. Benjamin Balleret
AP=56.87
exhibit L Lon.orr conpprrdueer
Smithsonian American Art Museum
1. Cleveland Museum Art
Toledo Museum of Art
2. Honolufu Museum of Art
2. Smithsonian American Art Museum
Egyptian Museum
3. Brooklyn Museum
3. Indiunapolis Museum of Art
San Jose Museum of Art
Whitney Museum American Art
4. Cincinnati Art Museum
5. Hirshhorn Museum and Sculpture Garden
5. Museum Fine Arts, Boston
AP=38.41
AP=52.52
Bcottish
1. Ayr United FCC. Clyde
1. Tommy Callaghan Gordon Wallace
footballers
Pat Crerand
3. Scottish League Cup
3. Dovid White
Gerry Britton
4. Stranraer FCC
4. Davie Dodds
Jim McLeam
Arbrouth F.C.
5. John Coughlin
APu4.57
AP 10
Number Singles
Billbourd
Try
song)
in Germony
2. Grammy Award for Best Female Pop Vical Performance
2. Whatayo foom from Me*
Lady Marmalade
3. Fuckin' Perfect
Just Give Me Reason
3. Dance Club Songs
4. Beautiful (Christina Aguilra song)
F'd Do Anything for Love
4. Pop 100
5. Raise Your Glass AP=4.59
(But Won't Don'Do Do That)
Hot Latin Songs
AP=4.14
2010 Albums
FAA airport categories
1. All Wont Is You"*
This Is the Warning
2. Rugby league county cups
2. Don't Mess with the Dragon
Tin Can Trust
3. Digital Songs
3. Believe (Orianthi album) Sci-Fi Crimes
Blonic (Christina Aguilera albumn)
Country Airplay
5. Swiss feteral election, 2007
5. Interpol
AP=6.78
(a) Top 5 predictions for of randomly selected categories, given exemplars. The first categories come from TypeNet, and the second two from our Wikipedia categorization datasct. Correct predictions are bolded. Predictions which are judged by the authors to be false negatives (predictions which properly belong to the target category) are indicated with asterisks**.
Q: Who was the last inmate of Spandau jail in Berlin?
A: Rudolf Hess 2. Adolf Hitler Hermann Gring Heinrich Himmler Emst Rhm
Q: Which fashionable London thoroughfare, about three quarters of a mile (1.2 km) long, runs from
Hyde Park Corner Marble Arch, along the length of the eastern side of Hyde Park?
A: Park Lane 2. Piccadilly Knightshridge 4. Leicester Square 5. Tottenham Court Road
Q: In which Lake District town would you find the Cumberland Pencil Museum?
A: 1. Keswick 2. Hawkshead 3. Grasmere 4. Cockermouth 5. Ambleside
Q: The Wimbledon tennis tournament held at which tennis club in London?
A: Queen's Club All England Lawn Tennis and Croquet Club 3. Wimbledon Championships 4.
Stade Roland-Garros Wentworth Club
(b) TriviaQA predictions from retrieval. Questions randomly sampled, and top ranking answers are shown. Correct answer in bold. Note that even when the model is wrong, the predictions are all of the correct type.
Figure 3: Random example predictions drawn from category completion, and TriviaQA tasks.
by Wang et al. (2018b) and there is a long way to go before RELIC's embeddings can capture all of the facts that can identified by crestion-dependeee inference time reading.
CONCLUSION
In this paper, demonstrated that the RELIC fill-in-hhe-llann task allows to learn context inde- pendent representations of entities with their own latent ontology. We show successful entity-level typing results FIGMENT (Yaghoobzadeh & Schütze 2015) and TypeNet (Murty et al. 2018)

---

Under review as a conference paper at ICLR 2020
when only training small fraction of the task-specific training data. We then introduce novel few-shot category reconstruction task and when comparing to Yamada et al. (2017). we found that RELIC is better able to capture complex compound types. Our method also proves successful for entity linking, where we match the state the art on CoNLL-Aida despite not using linking- specific features and fare similarly to the best system TAC-KBP 2010 despite not using an alias table, any external knowledge bases, iinkin--peccific features or even in-domain training data. Fi- nally, we show that our RELIC embeddings can be used answer trivia questions directly, without access to any evidence documents. We encourage researchers to further explore the properties of our entity representations and BERT context encoder, which we will release publicly.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- 2016.
mentation (OSDI 2016), pp.
Yonatan Bisk, Siva Reddy, John Blitzer, Julia Hockenmaier, and Mark Steedman. Evaluating in- duced ocg parsers on grounded semantic parsing. In EMNLP, 2016.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 2017.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collabora- tively created graph database for structuring human knowledge. In ACM SIGMOD international conference on Management of data, 2008
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. Learning structured embed-
dings of knowledge bases. In AAAI, 2011.
Xiao Cheng and Dan Roth. Relational inference for wikification. EMNLP, 2013.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. ACL, 2018.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In NIPS, 2015.
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. Question answering knowl- edge bases and text using universal schema and memory networks. In ACL, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Nitish Gupta, Sameer Singh, and Dan Roth. Entity linking via joint encoding of types, descriptions, and context. In EMNLP, 2017.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. JMLR, 13(Feb).307-361, 2012. 2012.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, 2015.
Zhiting Hu, Poyao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing. Entity hierarchy embedding. In ACL, 2015.
Heng Ji. Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. Overview of the tac 2010 knowledge base population track. In In Third Text Analysis Conference (TAC), 2010.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
Nikolaos Kolitsas, Octavian-Euger Ganea, and Thomas Hofmann. End-to-end neural entity linking.
CoRR, 2018
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1612. URL hwwptttp///// www.w..www.bb.mmm...w.ww. org/
anthology/P19- 1612
Douglas Lenat, Mayank Prakash, and Mary Shepherd. Cyc: Using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks. Al Magazine, 6:65-85, 12 1986.
Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. Two/too simple adaptations of word2vec for syntax problems. In NAACI,

---

Under review as a conference paper at ICLR 2020
Xiao Ling and Daniel S Weld. Fine Fine-grained entity recognition. In AAAI, 2012.
Teng Long, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. Leveraging lexical resources for learning entity embeddings in multi-relational data. In ACL, 2016.
Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. World knowledge for reading comprehension: Rare entity prediction with hierarchical Istms using ex- ternal descriptions. EMNLP, 2017.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. Yago3: A knowledge base from multilingual wikipedias. CIDR, 2013.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. NIPS, 2013.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS, 2013.
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. Hierarchical losses and new resources for fine-grained entity typing and linking. ACL, 2018.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. Who did what: A large-scale person-centered cloze dataset. EMNLP, 2016.
Maria Pershina, Yifan He,, and Ralph Grishman. Personalized page rank for named entity disam- biguation. In Proceedings of the 2015 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 238-243, 2015.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, Technical report, OpenAI, 2018.
Priya Radhakrishnan, Partha Talukdar, and Vasudeva Varma. ELDEN: Improved entity linking using densified knowledge graphs. In NAACL, 2018.
Jonathan Raiman and Olivier Raiman. DeepType: Multilingual entity linking by neural type system evolution. arXiv:1802.01022 2018
Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. Local and global algorithms for dis- ambiguation to wikipedia. In ACL, 2011.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In NAACL, 2013.
Avirup Sil, Gourab Kundu, Radu Florian, and Wael Hamza. Neural Cross-Lingual entity linking. In AAAI, 2018
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. NIPS, 2013.
Yaming Sun, Lei Lin, Duyu Tang, Nan Yang. Zhenzhou Ji, and Xiaolong Wang. Modeling mention, context and entity with neural networks for entity disambiguation. In IJCAI, 2015.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. EMNLP, 2015.
Kristina Toutanova, Victoria Lin, Wen-tau Yih, Hoifung Poon, and Chris Quirk. Compositional learning of embeddings for relation paths in knowledge base and text. In ACI,, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
12

---

Under review as a conference paper at ICLR 2020
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowl- edge graphs with box lattice measures. In ACL, 2018.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926-330. 2018a.
Wei Wang, Ming Yan, and Chen Wu. Multi gulti-granullritt hierarchical attention fusion networks for reading comprehension and question answering. In ACL, 2018b.
Yadollah Yaghoobzadch and Hinrich Schütze. Corpus-level fine- fine-grained entity typing using contex- tual information. In EMNLP, 2015.
Yadollah Yaghoobzadeh, Heike Adel, and Hinrich Schüitze Corpus-level fine-grained entity typing. Journal of Artificial Intelligence Research, 2018.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. Joint learning of the embedding of words and entities for named entity disambiguation. In CoNLL, 2016.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. Learning distributed representations texts and entities from knowledge base. TACL, 2017.
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, 2014.
13


Reviews:
[{"review_id": "rkgHVddg5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to learn entity representations by matching entities to the context it occurs in. It also shows that using these representations is very effective for a wide variety of down-stream entity-centric tasks such as entity typing, linking, and answering entity centric trivia questions. They train the model using a corpus of entity linked Wikipedia contexts (sentences unto length 128 tokens). The context is encoded with a BERT model and the CLS representation is used as the representation of context. After obtaining the representation, they train the entity embedding (present in the sentence) to be similar to the context embedding. They test their embeddings on few down-stream entity-centric tasks — linking, typing and trivia question answering.\n\nStrengths:\n1. They try the entity representations on a wide variety of entity-centric tasks and get reasonable results.\n\nWeaknesses:\n1. The biggest weakness of the paper is wrt novelty. Masking out entities and training to context is not a new idea. As pointed by the paper, Yamada et al., 2017 have a very similar objective and it is not very clear from the paper what is the additional contribution that this paper makes. Is using pretrained LMs the major difference? If not, it would have been nice to see Yamada et al’s results with BERT. Over all, this paper needs to make its own contribution clear compared to Yamada et al., 2017.\n2. The paper needs to be written more clearly at several places. Few examples are, even though in entity linking results (Table 1) the model achieves 83.0 with other papers achieving 90.9. I didnt see a discussion on how to close the gap. Even in the coNLL benchmark, the initial results of the paper is significantly behind. Claims like “CoNLL -Aida is known to be restricted and idiotic-synctatic domain” should be backed by detailed analysis or atleast a citation. Even after finetuning on the CoNLL benchmark, the result is 2.2 points behind state of the art and no discussions have been provided. As a result, I think the entity linking section needs major re-writing and explanation of the results.\n3. The paper makes an interesting observation that masking of entities is better for typing tasks and it affects linking performance, because spelling features are really important for linking. It would be interesting to see a discussion on what could be done to remedy this. Because if we have to retrain entity embeddings for different tasks, then it goes against the hypothesis of the paper which is to use entity representations for a wide variety of down-stream tasks.\n4. I found it confusing to read the setup in sec 5.4. especially where it says we represent each category with three random exemplars. Initially I thought 3 randomly sampled entities formed a category, which didnt make sense, but from figure 3, I think I understood that you first pick a category from Typenet and Wikipedia and then 3 entities are sampled from there. Is that correct? Regarding the results, can the poor results of Yamada et al., can be understood by the fact that it was trained using smaller number of categories? Also, why are the numbers wrt All entities left blank in Table 4. Given that your model is similar, I am assuming its easy to retrain Yamada et al and test it on the all entities benchmark?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rke_FcY0YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper aims to learn entity representations by aggregating all the contexts that an entity appears in based on English Wikipedia. \n\nThe idea is very simple, basically it represents each entity as a vector, and also represents each context as a vector, and maximizes the cosine similarity between the two vectors using a negative-sampling training objective. The training process is similar as word2vec, and they leveraged all the hyperlinks in Wikipedia, and used BERT to encode the context (instead of learning a context vector for each word). \n\nAs a result, the paper demonstrates that this set of entity embeddings are highly useful, and they were evaluated on 1) entity-level typing 2) entity linking 3) few-shot category reconstruction 4) answering trivia questions (TriviaQA).\n\nI think this is a nice empirical paper and the experiments are thorough. If they are going to release the entity embeddings, that would also benefit the community a lot and also encourage more research in this direction.\n\nI am a bit concerned about the novelty of the approach. It is a bit surprising that nobody has experimented with this before. It seems that Yamada et al took a very similar approach but used simple bag-of-words approaches to encode the context (instead of BERT).  To me, this paper may be better for the NLP community but it should be fine to the ICLR community too.\n\nI am also not completely sure how strong the evaluation results are indeed, esp. related to the comparisons with Yamada et al, 2017, given the approaches are similar. \n\n- It seems to be on par with or slightly worse than Yamada et al 2017 on entity linking.\n\n- For the category completion task, RELIC is doing much better than Yamada et al 2017 on more complex Wikipedia categories but I am not sure if it is a completely fair comparison. It’d be great if the authors can discuss all the key differences between the two approaches, from the model design to all the experimental details, that would help clear out all these confusions. I have read the related work section but can't figure out all the details.\n\n- The entity typing results are very strong. I also like the TriviaQA experiments but the numbers are way behind the standard reading comprehension results. \n\n\nMinor comment:\n- Why sec 5.3 (effect of masking) is listed together with other evaluation tasks? Isn’t better to move it to analyses/ablation studies?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJl1Q0wptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a new way of learning context dependent entity representations that are capable of encoding fine-grained entity types. This is realised by matching entities to all their contexts and thereby encoding all their properties.\nThe contribution is RELIC, a table of entity representations that are learned given the above objective. This table, RELIC, can be used in various tasks like entity typing, entity linking and question answering. Given a corpus of entities and their respective contexts, the probability of an entity occurring under its context is maximised (additionally, the probability is minimised for negative samples). The data is taken from a Wikipedia dump. Without adding further information, the RELIC table can directly be used for entity linking.\nIn the entity linking task, RELIC achieves a lower precision at two benchmarks (CoNLL-Aida, TAC-KBP 2010) than other approaches. This can be tackled by fine-tuning the RELIC table, which is done with the training set from the CoNLL-Aida task. With fine-tuning, comparable results are achieved.\nFor entity typing, RELIC embeddings of entities are used as input for a 2-layer FF network, which then outputs which types belong to the entity. The FIGMENT and TypeNet datasets are used as benchmark here and it is shown that the RELIC based approach outperforms the other approaches.\nRELIC is also used for a category completion task, where a category is represented by a centroid of three randomly sampled entities belonging to it. The entities from RELIC are then ranked with the dot-product with the centroid. Two tasks are used to measure the performance here: the TypeNet completion task and a Wikipedia-based task. Compared to embeddings of a 2017 approach, the RELIC embeddings perform better. \nFinally, a QA task is performed using RELIC. Here, the questions are modelled as contexts and the entity which is closest in terms of cosine similarity is taken as answer. The performance is below the upper bound from 2018, but better than a classifier system from 2017.\n\nOverall, the paper is well structured and manages to show that RELIC is a versatile tool on which various tasks can be performed. The technical details are well covered and the evaluation is done in a detailed way.\nThe contribution of the paper is mostly in the definition of context-entity pairs as input to a transformer model and in empirical evaluations. I specifically found the findings in 5.3 interesting. The technological contribution is rather limited.\n\nMinor comments: At page 3, chapter 3.1 there is a word duplication […]to correctly match match[…] that should be corrected. Additionally, on page 5 it should be mentioned that the values in table 1 are precision values.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an approach for learning context dependent entity representations that encodes fine-grained entity types. The paper includes some good empirical results and observations, but the proposed approach is very simple but lacks technical novelty needed to top ML conference; the clarify of the presentation can also be improved. 
Decision: Reject