Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Authors: Wenhan Xiong
Jingfei Du
William Yang Wang
Veselin Stoyanov
Venue:  (2020)

Abstract:  Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

Published as a conference paper at ICLR 2020
PRETRAINED ENCYCLOPEDIA: WEAKLY SUPERVISED KNOWLEDGE-PRETTRANEE LANGUAGE MODEL
Jingfei Duu, William Yang Wang', Veselin Stoyanov f University of California, Santa Barbara
Wenhan Xiong', "iong',Jinggee
8 Facebook AI {xwhan, william} @cs.ucsb.edu, (jingfeidu, ves) @ fb.com
ABSTRACT
Recent breakthroughs of pretrained language models have shown the effective- ness of self-supervised learning for wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge. suggesting that large-scale language modeling could be an implicit method to cap- ture knowledge, In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objec- tive, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, model consis- tently outperforms BERT on four entity-related question answering datasets (iee. WebQuestions, TriviaQA, SearchQA and Quasar-T) with average 2.7 F1 im- provements and standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.
INTRODUCTION
Language models pretrained on a large amount of text such as ELMo (Peters et al. 2018a| BERT (Devlin al. 2019) and XLNet (Yang et al.) 2019c) have established new state of the art on wide variety of NLP tasks. Researchers ascertain that pretraining allows models to learn syn- tactic and semantic information of language that is then transferred on other tasks (Peters et al., 2018b, Clark et al. 2019). Interestingly, pretrained models also perform well on tasks that require grounding language and reasoning about the real world. For instance, the new state-of-the-ar for WNLI (Wang et al., 2019a), ReCoRD (Zhang et al.) 2018) and SWAG (Zellers et al.) 2018) is achieved by pretrained models. These tasks are carefully designed so that the text input alone does not convey the complete information for accurate predictions external knowledge is required to fill the gap. These results suggest that large-scale pretrained models implicitly capture real-world knowledge. [ogaan al. (019) and Petroni et al. (2019) further validate this hypothesis through a zero-shot fact completion task that involves single-token entities, showing that pretrained models achieve much better performance than random guessing and can be on par with specifically-traine relation extraction models.
As unstructured text encodes great deal of information about the world, large-scale pretraining over text data holds the promise of simultaneously learning syntax, semantics and connecting them with knowledge about the real world within single model. However, existing pretraining objectives are usually defined at the token level and do not explicitly model entity-centric knowledge. In this work, we investigate whether we can further enforce pretrained models to focus on encyclopedic knowledge about real-world entities, so that they can better capture entity information from natural language and be applied to improving entity-related NLP tasks. We evaluate the extent to which a pretrained model represents such knowledge by extending existing fact completion evaluation to a cloze ranking setting that allows us to deal with a large number of multi-token entity names with- out manual judgments. Our experiments relations reveal that existing pretrained models encode entity-level knowledge only to a limited de- gree. Thus, we propose a weakly supervised knowledge learning objective that requires the
10 common Wikidata (Vrandečić & Krötzsch 2014)

---

Published as a conference paper at ICLR 2020
Entity Boundary Representations
Entity Boundary Representations
Bryan Johnson
Steve Ditko
Steve Ditio
Marvel Comics
DC Comics
Transformer Encoders
Transformer Encoders
Original Article:
Replaced Article:
Spider-Man is fictional superhero created by writer- editor Stan Lee writer-artist Steve Ditko.
Spider-Man is a fictional superhero created by writer- editor Bryan Johnson and writer-artist Steve Ditko. He first appeared in the anthology comic book American comic books published by DC Comics
He first appeared in the anthology comic book American
comic books published by Marvel Comies
Entity Replacement Procedure
entity linking
type lookup
clustered by type Q1320047
Marvel Comics
Q173496
Q1320047
publishing company
DC Camics
Dark Horse Camics
random sample
DC Comics
Image Comics
WIKIPEDIA Diyilpota
WIKIDATA
Figure 1: Type- Constrandd Entity Replacements for Knowledge Learning.
model to distinguish between true and false knowledge expressed in natural language. Specifically, we replace entity mentions the original documents with names of other entities of the same type and train the models to distinguish the correct entity mention from randomly chosen ones. Models trained with this objective demonstrates much stronger fact completion performance for most rela- tions test Compared with previous work (Zhang et al. 2019 Peters et al. (019) that utilizes an external knowledge base to incorporate entity knowledge, our method is able to directly derive real-world knowledge from unstructured text. Moreover, processing, memory or modifications to the BERT model when fine-tuning for downstream tasks.
method requires additional data
We test model two practical NLP problems that require entity knowledge: Question Answer- ing (QA) and fine-grained Entity Typing. We use four previously published datasets for open-domain QA and observe that questions in these datasets often concern entities. The Entity Typing task quires the model to recognize fine-grained types of specified entity mentions given short contexts. On three of the QA datasets, our pretrained model outperforms all previous methods that do inter-passage normalizationg On the FIGER entity-typing dataset, our
rely on semoryy-consuiii model sets a new state of the art. Through ablation analysis, we show that the new entity-centric training objective is instrumental for achieving state-of-the-ar results.
In summary, this paper makes the following contributions: 1) We extend existing fact completion evaluation settings to test pretrained models' ability encoding knowledge of common real-world entities; 2) We propose new weakly supervised pretraining method which results in models that better capture knowledge about real-world entities from natural language text; 3) The model trained with our knowledge learning objective establishes new state of the art on three entity-related QA datasets and standard fine- fineggrained entity typing dataset.
We begin by introducing our weakly supervised method for knowledge learning (:2 and then dis- cuss experiment settings and evaluation protocols, compare model to previously published work and perform ablation analysis. Finally, we review related work in 44 and conclude in $
ENTITY REPLACEMENT TRAINING
We design an entity-centric training objective that utilizes weakly supervised training signals to explicitly encourage knowledge learning during pretraining, Given an input document, we first
Wang et al. 2019b) propose to use BERT to encode multiple retrieved paragraphs within the same forward- backward pass. This operation requires around 12G GPU memory even for encoding single example which includes 512-token paragraphs.
2

---

Published as a conference paper at ICLR 2020
recognize the entity mentions and link them to Wikipedia entities We consider the original texts as positive knowledge statements and create negative statements by randomly replacing the entity mentions (E with the names of other random entities (E > that have the same entity type as the mentioned entity. This setup is similar in spirit to the type-constrained negative sampling technique used to train knowledge base representations (Bordes et al.) (013). The latter technique creates negative triples by replacing the subject or object entity with random entities of the same type. Instead of knowledge base triples, we treat unstructured texts factual statements. For a certain entity mentioned in a context C, we train the model to make binary prediction indicating whether the entity has been replaced:
Je.c = LeES+ log P(e|C) + Le-=+ log(1 - P(e(c)).
Compared to the language modeling objective, entity replacement is defined the entity level and introduces stronger negative signals. When we enforce entities to be of the same type, we pre- serve the linguistic correctness of the original sentence while the system needs to learn to perform judgment based on the factual aspect of the sentence.
We describe the implementation in more detail in the following paragraphs.
Data Preparation We use the whole English Wikipedia dump as training data and rely on all Wikipedia entitie Entities in documents entity alias from Wikidata. That is, we first retrieve the entities annotated by anchor links and then find other mentions of these entities by string matching their Wikidata alias. We split each document into multiple text chunks with the same size (512 tokens). Although our experiments rely on the Wikipedia corpus, this setup can be easily extended to larger corpora with off the-shelf entity linking tools. We leave the larger scope of the experiments to future work.
recognized based on Wikipedia anchor links and
Replacement Strategy When replacing entities, we first lookup type information from Wikidata and then randomly select other entities with the same type. We do not replace adjacent entities. In other words, there must be at least one unreplaced entity between any two replaced ones. This reduces cases where we replace all entities in the same sentence and the resulting sentences happen to introduce correct entities by chance. For replacement, we randomly sample a string from the entities' alias set. For each text chunk, replicate it 10 times with different negative entities for each replacement location. We show an illustration of the entity replacement method Figure
Model Architecture We use the Transformer (Vaswani et al. 2017) model used by BERT (Devlin ct al. 2019). We use the same architecture as BERT base: 12 Transformer layers, each with hidden dimension 768. We initialize the transformer with model pretrained based implementations]] For each entity, we use the final representations of its boundary words (words before and after the entity mention) to make predictions. We simply concatenate the boundary words' representations and add a linear layer for prediction. During training, we use 0.05 dropout at the final layer.
own BERT
Training Objectives Masked language model pretraining has been proven to be effective for downstream tasks. While training for entity replacement we also train with the masked language model objective in multi-task set-up. When masking tokens, we restrict the masks to be outside the entity spans. We use a masking ratio of 5% instead of 15% in the original BERT to avoid mask- ing out too much of the context. We train the model for approximately I million updates using batch size of 128.
"The entity links only required for pretraining.
'Fach Wikipedia entity (title) corresponds to a unique entity node in Wikidata. BBased on the "instance of" relation in Wikidata. If there multiple correct types, we randomly choose one.
'We use the masked language model implementation in Fairseq (Ot et al. 2019 to pre-train model for 2M updates the combination of BooksCorpus (7hu al. 2015) and English Wikipedia.
3

---

Published as a conference paper at ICLR 2020
3 EXPERIMENTS
We first test our model on fact completion task. This task resembles traditional knowledge base completion: requires the model complete missing entities factual triples. We further test on two real-world downstream tasks that require entity-level knowledge question answering and fine-grained entity typing. We describe the hyperparameter and training settings all experiments in the appendix.
3.1 ZERO-SHI SHOT FACT COMPLETION
In traditional knowledge base completion tasks models have access set of training triples. In- stead, we utilize a zero-shot test to examine the model's ability to automatically derive relational knowledge from natural language.
Dataset We rely factual triples from Wikidata. Each triple describes the relationship between two certain entities, e.g., {Paris, CapitalOf, France). Following recent practices (Bosselut et al.) 2019 Logan et al. 2019) that decode structured knowledge from language models, we first manually create templates to convert triples of 10 common relations into natural language expressions ( (Paris, CapitalOf, France) > the capital of France is Paris). We then create queries by removing the object entity in the expression and use pre-trained models to predict the missing entities, e.g., the capital of France is ?. We create 1000 cloze examples for each of the 10 relations.
Evaluation Metries Previous work (Logan et al. 2019 Petroni al. 2019) either relies human evaluation or only considers single-token entities for fact completion. In contrast, we consider an entity-ranking setup and create set of candidate entities for each relation. This setting allows us to automatically evaluate a large number of queries that usually involve multi-token entities. We test pretrained models on their ability to recover the correct object entity from the candidate set. To create the negative choices, we select from the set of all object entities in the particular relation, which generally have the same type as the groundtruth and are more challenging to distinguish than entities with different types. Our evaluation strategy is similar to previous work on knowledge base completion (Nikee et al., [011]: Bordes et al. [0133] Xiong et al. 2017). We follow these studies and use Hits @ 10 as the evaluation metric.
Baselines We compare our model with two pretrained language models BERT (Devlin et al. 2019) (both base and large) and GPT-2 [Rafford al. 2019). We make use of their output token proba- bilities to rank candidate entities. For BERT, feed in the masked queries (e.g., Qmakked the capital of France is [MASK] ). For multi-token candidates, we use the same number of [MASK] tokens in the query inputs. We use the average log probability of masked tokens for rank-
ing. Given a multi-token entity E, = [e], 2 e..,..,, e !B.1) the ranking score from BERT calculated
SE. =
1 E.
Σ
log P(c IQ masked).
For GPT-2, we feed in the original query without the answer entity and use the first-token probability of candidate entities for ranking, which performs better than using average log probabilities. As our model learns to predict a plausible probability (P(elC)) for each entity mention during entity replacement training, we can directly use these predicted probabilities to rank the candidates.
Results T shows the fact completion results for all relations. We denote our method WKLM for (Weakly Supervised Mhowledge-Pretraine Language Model). Overall, WKLM achieves the best results on 8 of the 10 relations. We also observe that GPT-2 outperforms BERT on average. We think this because the fact completion task requires models to predict the missing entities using only a short context on the left, while BERT pretraining incorporates context from both di- rections. Interestingly, BERT achieves good performance on several geographical relations such as PlaceOfBirg hth LocatedIn and P PlaceOfDeath. We conjecture that this is because loca- tion entities usually appear at sentence ends in Wikipedia articles, e.g., Obama was born in
For each relation, use the top triples that connecting most common entities.
4

---

Published as a conference paper at ICLR 2020
Honolulu, Hawaii.. This sentence pattern is similar templates and BERT may learn to rely mostly on the left context to make predictions. For most relations that include answers that are person names, BERT lags behind both GPT-2 and model.
Comparing the top and bottom five relations, we observe that BERT's performance is correlated with the size of the candidate set, while WKLM and GPT-2 are less sensitive this number. A similar pattern exists between models' performance and the cardinality of groundtruth answers, i.e., our model achieves similar performance both single-answer and multiple-answer queries while BERT is usually better at single-answer queries. WKLM both outperforms BERT and GPT-2 and achieves robust performance across relations with different properties. Visualization of correlations between relation properties and model performance can be found in the appendix.
Table Zero-Shot Fact Completion Results.
Relation
# of Candidates
of Answers
Model
Name
BERT-base
BERT-large
GPT-2
Ours
HASCHILD (P40)
906
3.8
9.00
6.00
20.5
63.5
NOTABLEWORK (P800)
901
5.2
1.88
2.56
2.39
4.10
CAPITALOF (P36)
820
1.87
1.55
15.8
49.1
FOUNDEDBY Y (P112)
798
3.7
2.44
1.93
8.65
24.2
CREATOR (P170)
536
3.6
4.57
4.57
7.27
9.84
PLACEOFBIRTH (P19)
497
1.8
19.2
30.9
8.95
23.2
LOCATEDIN (P131))
382
13.2
52.5
21.0
61.1
EDUCATEDAT (P69)
374
4.1
9.10
7.93
11.0
16.9
PLACEOFDEATH (P20)
313
1,7
43.0
42.6
8.83
26.5
OCCUPATION (P106)
190
1.4
8.58
10.7
9.17
10.7
Average Hits @ 10
11.3
16.1
16.3
28.9
3.2 DOWNSTREAM TASKS
Background knowledge is important for language understanding. We expect our pretraining ap- proach to be beneficial to NLP applications where entity-level knowledge essential. We consider two such applications: question answering and entity-typing We find that large portion of the questions in existing QA datasets are about entities and involve entity relations. training objective is analogous question answering in multiple-choice setting (Hermann al.) 2015). The entity-typing task requires the model to predict a set of correct types of entity mentions short context. The context itself can be insufficient and the training data for this task is small and noisy. We believe a model that encodes background entity knowledge can help in both cases.
way, our pre-
3.2.1 QUESTION ANSWERING
Datasets
consider four question answering datasets:
WebQuestions (Berant et al., 2013) is originally dataset for knowledge base question answering. The questions are collected using Google Suggest API and are all asking about simple relational facts of Freebase entities.
TriviaQA Joshi et al. 2017) includes questions from trivia and quiz-league websites. Apart from a small portion of questions to which the answers are numbers and free texts, 92.85% of the answers are Wikipedia entities.
Quasar-T DDhingra et al. 2017) is another dataset that includes trivia questions. Most of the answers this dataset are none phrases. According to our manual analysis on random samples, 88% of the answers are real-world entitie
SearchQA (Dunn et al. 2017) uses questions from the television quiz show Jeopardy! and we also find that almost all of the answers real-world entities.
the splits used by Lin al. 2018). Different methods also use different retrieval systems for this dataset, iee, ORQA BeRTserimi retrieve documents from Wikipedia while the other methods use retrieved Web documents.
The splits of TriviaQA might be different in previous work,
We consider as entities long as they correspond Wikidata entities Wikipedia titles.
5

---

Published as a conference paper at ICLR 2020
Questions in all three datasets created without the context of paragraph, which resembles the scenario of practical question answering applications. All the questions except WebQuestions are written by humans. This indicates that humans generally interested to ask questions to seek information about entities. We show the statistics and example questions in Table 2 We split the training data (created by distant supervision) of WehQuestions with ratio (9:1) for training and development. Since our model is based on our own BERT implementations, in addition to the afforementioned entity-related datasets, we first use the standard SQuAD (Rajpurkar al. 2016) benchmark to validate our model's s answer extraction performance.
Table 2: Properties of the QA Datasets.
Dataset
Train
Valid
Test
Example Questions
WebQuestions
3778
2032
Who plays Stewie Griffin on Family Guy?
TriviaQA 87291
11274
10790
What is Japanese share index called?
ScarchQA
99811
13893
27247
Hero several books 11 discover's wizard? Which vegetable is a Welsh emblem?
Quasar-T 37012
3000
3000
Settings We adopt the fine-tuning approach to extract answer spans with pretrained models. We add linear layers over the last hidden states of the pretrained models to predict the start and end posi- tions of the answer. Unlike SQuAD, questions in the datasets we use are not paired with paragraphs that contain the answer. We follow previous work (Chen et al. 2017 Wang et al. 20) and re- trieve context paragraphs with information retrieval systems. Details of the context retrieval process for each dataset can be found in the appendix. Reader models are trained with distantly supervised data, i.e., we treat any text span in any retrieved paragraph as ground truth as long as it matches the original answers. Since the reader model needs read multiple paragraphs predict a single answer at inference time, we also train a BERT based paragraph ranker with distant-supervised data assign each paragraph a relevance score. The paragraph ranker takes question and paragraph pairs and predicts a score in the range [0, 1] for each pair. During inference, for each question and its evidence paragraph set, we first use the paragraph reader to extract the best answer from each paragraph. These answers are then ranked based on a linear combination of the answer extraction score log sum of the answer start and end scores) and the paragraph relevance score. We also evaluate model performance without using the relevance scores.
Open-Domale QA Baselines
We compare our QA model with the following systems:
DrQA Chen al. 2017 7) an open-domain QA system which uses TF-IDF with bigram
features for ranking and simple attentive reader for answer extraction.
R (Wag et al. 2018a) is a reinforcement learning based system which jointly trains a paragraph ranker and a document reader.
DSQA (Lin et al. 2018) uses RNN- based paragraph ranker and jointly trains the paragraph ranker and attentive paragraph ranker with a multi-tast task loss.
Evidence Aggregation (Wang et al., 2018b) uses a hybrid answer reranking module to aggregate answer information from multiple paragraphs and rerank the answers extracted from multiple paragraphs.
BERTserini Yang et al., 2019a)
BERT-based open-domain QA system, which uses BM25-based retriever to retrieve 100 paragraphs and a BERT-based reader to extract an- swers. The paragraph reader is either trained with SQuAD (Rajpurkar et al. 2016) data or distant-supervision data (Yang et al. 2019b)
ORQA (Lee et al. 2019) replaces the traditional BM25 ranking with BERT-based ranker. The ranker model pretrained on the whole Wikipedia corpus with an inverse cloze task which simulates the matching between questions and paragraphs. All text blocks in Wikipedia are pre-encoded as vectors and retrieved with Locality Sensitive Hashing.
Table 3 shows the SQuAD results and Table 4 shows the open-domain results the four datasets that are highly entity-related. From the SQuAD results, we observe that our BERT reimplementation performs better than the original model this is due to the fact that it is trained
Results

---

Published as a conference paper at ICLR 2020
for twice many updates: 2 million 1 million for the original BERT. Although lots of the answers in SQuAD are non-entity spans, the WKLM model we propose achieves better performance than BERT. We believe the improvement is due to both the masked language model and entity replacement objectives. Ablation experiments on the training objectives will be discussed 3.2.3
Having established that BERT re-implementation performs better than the original model, we com- pare with only our own BERT for the following ex- periments. From Table we see that our model produces consistent improvements across different datasets. Compared to the 0.8 F1 improvements over BERT on SQuAD, we achieve an average of 2.7 FI improvements over BERT on entity-related datasets when the ranking scores are not used. On TriviaQA and Quasar-T, WKLM outperforms our BERT even when uses ranking scores. Improvements in natural language question datasets (WebQuestions, TriviaQA, and Quasar-T) are queries. When we utilize ranking scores from simple BERT based ranker, we are able to achieve the state-of-the-an three of the four datasets.
Table SQuAD Dev Results.
Model
EM
F1
Google's BERT-base
80.8
88.5
Google's BERT-large
84.1
90,9
Our BERT-base
83.4
90.5
WKLM (base)
84.3
91.3
significant than SearchQA where the questions are informal
Table 4: Open-domain QA Results.
WebQuestions F1
TriviaQA EM
Quasar-T F1
SearchQA EM
Model
EM
F1
EM
F1
DrQA (Chen et al.
2017
20.7
R (Wang
2018a)
50.6
57.3
42.3
49.6
57.0
63.2
DSQA (Lin ct al.
2018
18.5
25.6
48.7
56.3
42.2
49.3
49.0
55.3
Evidence Agg. [Wang el al.
2018b
50.6
57.3
42.3
49.6
57.0
63.2
BERTserini Yann et al. 2019a
51.0
56.3
BeRTseinni+DS (Yng
2019b
54.4
60.2
ORQA (Lee et al. 2019)
36.4
45.0
Our BERT
29.2
35.5
48.7
53.2
40.4
46.1
57.1
61.9
Our BERT + Ranking score
32.2
38.9
52.1
56.5
43.2
49.2
60.6
65.9
WKLM
30.8
37.9
52.2
56.7
43.7
49.9
58.7
63.3
WKLM Ranking score
34.6
41.8
58.1
63.1
45.8
52.2
61.7
66.7
3.2.2 ENTITY TYPING
To compare with an existing study (Zhang et al. 2019) that also attempts incorporate entity knowledge into language models, we consider an additional entity typing task using the large FIGER dataset (Ling& & Weld 2012). The task is to assign fine mme-graieed type to entity mentions. We do that by adding two special tokens before and after the entity span to mark the entity position. We use the final representation of the start token ([CLS)) to predict the entity types. The model is fine-tuned weakly-supervised training data with binary cross-entropy loss. We evaluate the models using strict accuracy, loose micro, and macro Fl scores.
We show the results in Table 5 We compare our model with two non-BERT neural baselines (Inui al. 2017) that integrate set of hand-crafted features: LSTM + Hand-crafted and Attentive
Table Fine-grained Entity Typing Results on the FIGER dataset.
Model
Acc
Ma-Fi
Mi-F1
LSTM + Hand-crafted (Inui et al. 2017)
$7.02
76.98
73.94
Attentive Hand-crafted (Inu all 2017)
59.68
78.97
75.36
BERT baseline (Zhang 2019
52.04
75.16
71.63
ERNIE [Zhang et al. 2019
57.19
75.61
73.39
Our BERT
54.53
79.57
74.74
WKLM
60.21
81.99
77.00

---

Published as a conference paper at ICLR 2020
Hand-crafted; vanilla BERT baseline and the ERNIE model (Zhang al. 2019) that enhances BERT with knowledge base embeddings.
First, that naively applying BERT is less effective than simple models combined with sparse hand-crafted features. Although the ERNIE model can improve over BERT by 5.15 points, its performance still lags behind models that make good of hand-crafted features. In contrast, although based on stronger BERT model, our model achieves larger absolute improvements (5.68 points) and sets new state-of-the-art for this task. Given the larger improvement margin, we believe our model that directly learn knowledge from text is more effective than the ERNIE method.
3.2.3 ABLATION STUDY: THE EFFECT OF MASKED LANGUAGE MODEL Loss
In view of a recent study (Liu et al. 2019b) showing simply extending the training time of BERT leads stronger performance various downstream tasks, conduct further analysis to differ- entiate the effects of entity replacement training and masked language modeling. We compare our model with three variants: model pretrained only with the knowledge learning objective (WKLM without MLM), a model trained with both knowledge learning and masked language modeling with more masked words (WKLM with 15% MLM) and BERT model trained with additional million updates on English Wikipedia (BERT + 1M MLL updates) and no knowledge learning.
The ablation results are shown Table 6 the language model objective is essential for downstream performance. We also find that masking out too many words (i.e., 15% masking ratio in the original BERT) leads worse results. We conjecture that too many masked words outside entity mentions break parts of the context informa- tion and introduce noisy signals to knowledge learning. Results of continued BERT training show that more MLM updates are often beneficial, especially for SQuAD. However, tasks that are more untity-centries continued MLM training is less effective than our WKLM method. This suggests that W WKLM method could serve effective complementary recipe masked language modeling when applied to entity-related NLP tasks.
The results of WKLM without MLM validate that adding
Table Ablation Studies on Masked Language Model and Masking Ratios.
SQuAD EM
TriviaQA EM
Quasar-T EM
FIGER
Model
F1
F1
Acc
Our BERT
83.4
90.5
48.7
53.2
40.4
46.1
54.53
WKLM
84.3
91.3
52.2
56.7
43.7
49.9
60.21
WKLM without MLM
80.5
87.6
48.2
52.5
42.2
48.1
58.44
WKLM with 15% masking
84.1
91.0
51.0
55.3
42.9
49.0
59.68
Our BERT + 1M MLM updates
84.4
91.1
52.0
56.3
42.3
48.2
54.17
4 RELATED WORK
Pretrained Language Representations Early research on language representations focused on static unsupervised word representations (Mikolov et al.) 2013: Pennington et al. 2014). Word embeddings leverage co-occurrences to learn latent word vectors that approximately reflect word semantics. Given that words have different meanings in different contexts, ies (McCann et al.) 2017 Peters et al. 2018a) show that contextual language representations can be more powerful than static word embeddings in downstream tasks. This direction has been fur- ther explored at a larger scale with efficient Transformer architectures (Radford et al.) 2019 Devlin al. 2019 Yang al.) 2019c). Our WKLM method is based on these techniques and we focus on improving the knowledge ability of pretrained models.
recent stud-
Knowledge-Enhanee NLP Models Background knowledge has been considered an indispens- able part of language understanding (Fillmore et al. 1976; Minsky 1988), As standard language en- coders usually do not explicitly model knowledge, recent studies (Ahn et al. 2016 Yang & Mitchell 2017 Logan et al. 2019; Liu et al.) 2019a) have explored methods to incorporate external knowl- edge into NLP models. Most of these methods rely on additional inputs such as entity representa- tions from structured knowledge bases. With the breakthrough of large-scale pretrained language
8

---

Published as a conference paper at ICLR 2020
encoders (Devlin et al., 2019). Zhang al. 2019) and Peters al. (2019) adopt similar ideas and propose entity-level knowledge enhancement training objectives to incorporate knowledge into pre- trained models. Other recent studies (Mihaylov & Frank 2018: Xiong et al. 2019) leverage external knowledge bases to enhance text-hased question answering models. In contrast to these methods, method utilizes minimal external entity information and does not require additional memory or architectural changes when applied to downstream tasks.
5
CONCLUSION
We introduce a weakly supervised method to encourage pretrained language models to learn entity- level knowledge. Our method uses minimal entity information during pretraining and does not in- troduce additional computation, memory or architectural overhead for downstream task fine-tuning. The trained model demonstrates strong performance on a probing fact completion task and two entity-related NL. tasks. Together, our results show the potential of directly learning entity-level knowledge from unstructured natural language and the benefits of large-scale knowledge-awar pre- training for downstream NLP tasks.
REFERENCES
Sungjin Ahn, Heeyoul Choi, Tanel Pämamaa, and Yoshua Bengio. neural knowledge language model. arXiv preprint wwXiv:160..003/11 2016.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, pp. 1544. ACL, 2013.
Antoine Bordes, Nicolas Usumier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pp. 2787-2795, 2013.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin
Choi. COMET: commonsense transformers for automatic knowledge graph construction. In ACL (1), pp. 4762-4779. Association for Computational Linguistics, 2019.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. In ACL (I), pp. 1870-1879. ACL, 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.0434/ 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In (IA pp. 4171-4186. ACL, 2019.
Bhuwan Dhingra, Kathryn Mazaitis, and William Cohen. Quasar: Datasets for question answer- ing by search and reading, arXiv preprint ar 1707v:1707.03904, 2017.
Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqu: A new q&a dataset augmented with context from search engine. arXiv preprint 17X4v:170..05179, 2017.
Charles J Fillmore et al. Frame semantics and the nature of language. In Annals of the New York Academy of Sciences: Conference on the origin and development of language and speech, volume 280. pp. 20-32, 1976.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pp. 1693- 1701, 2015.
Kentaro Inui, Sebastian Riedel, Pontus Stenetorp, and Sonse Shimaoka. Neural architectures for fine-grained entity type classification. In EACL (1), pp. 1271-1280. Association for Computa- tional Linguistics, 2017.

---

Published as a conference paper at ICLR 2020
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL pp. 1601-1611. Association for Computational Linguisties, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 2014.
Kenton Lee, Ming-Wei Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In ACL (1), pp. 4086-6096 Association for Computational Linguisties, 2019.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. Denoising distantly supervised open- domain question answering. In ACL (1), pp. 1745. Association for Computational Linguis- tics, 2018.
Xiao Ling and Daniel S. Weld. Fine-grained entity recognition. In AAAI. AAAI Press, 2012.
Angli Liu, Jingfei Du, and Veselin Stoyanov. Knowledge-augmeented language model and its appli- cation to unsupervised named-entity recognition. In (LACL-HLT()),, pp. 1111-1141111 1150. Association for Computational Linguistics, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini 1907. 1692,, 2019b.
Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. ACL, pp. 5962-5971.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In NIPS, pp. 6294-6305, 2017.
Todor Mihaylov and Anette Frank. Knowledgeable reader: Enhancing cloze-style reading compre- hension with external commonsense knowledge. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 821 8318, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 25.18863/v1BB 1076.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Marvin Minsky. Society of mind. Simon and Schuster, 1988.
Maximilian Nickel, Volker Tresp. and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In ICML, volume 11, pp. 809-816, 2011.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543 1543, ACL, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.00 100::000000055 201 8a.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In EMNLP, pp. 1499-1509. Association for Computational Linguistics, 2018b.
Matthew E Peters, Mark Neumann, IV Logan, Robert, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations, EMNLP, 2019.
10

---

Published as a conference paper at ICLR 2020
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? EMNLP, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy I. Liang. Squad: 100, 000+ questions for machine comprchension of text. In EMNLP, pp. 2383-2392. The Association for Computational Linguistics, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp.
2017.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledge base. 2014.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019a.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R3: Reinforced ranker-reader for open-domain question answering. In AAAI, pp. 5981-5988. AAAI Press, 2018a.
Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR, 2018b.
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv::908. 08167, 2019b.
Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 364-573. Copenhagen, Denmark, September 2017.
Association for Computational Linguistics. doi: (0.18653/11/D17-100))
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Improving question answering over incomplete KBs with knowledge reader. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguisties, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.19655/v//119-1417.
Bishan Yang and Tom M. Mitchell. Leveraging knowledge bases in lstms for improving machine reading. In ACL (1), pp. 1436 1336-1446. 1446. ACI,, 2017.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv: 1902.01718, 2019a.
Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. Data augmentation for bert fine-tuning in open-domain question answering. arXiv preprint arXhh:190..06552 2019b.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint trXiv:/906.082.7, 2019c.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP, pp. 93-104. ACL, 2018.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 1810. /285, 2018.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: enhanced language representation with informative entities. In ACI( pp. 1441-1451. ACL,, 2019.
11

---

Published as a conference paper at ICLR 2020
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19-27. IEEE Computer Society, 2015.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
Implementation Details and Hyperparameters We implement our method using Fairseq Ont al. (2019) and the fact completion baselines are implemented with Huggingface's Pytorch- Transformery We pretrain the models with 32 V100 GPUs for 3 days. We use at most GPUs for fine-tuning the paragraph reader, use 8 GPUs for fine-tuning the paragraph ranker. The entity-typing experiments require larger batch sizes and take 8 GPUs for training.
For the knowledge learning pretraining phase, we use the Adam optimizer (Kingma & Ba 2014) with learning rate le-5, batch size 128 and weight decay 0.01. The model is pretrained on 32 V100 GPUs for days. To train the paragraph reader for open-domain QA, we select the best learning rate from (1e-6, 5e-6, le-5, 2e-5] and last layer dropout ratio from {0.1, 0.2]. We set the maximum training epoch to be 10 and batch size to be 32. The maximal input sequence length is 512 for WebQuestions and 128 for the other three datasets that use sentence-level paragraphs. For the paragraph ranker, we choose learning rate from {1e-5, 2e-5, 5e-6). use dropout 0.1 and batch size 256. The maximal sequence length for each dataset is consistent with the one used for training the paragraph reader. The linear combination ranking and extraction scores is selected based on validation performance. For SQuAD experiments, 3e-5), learning rate from {8. 16). last layer dropout ratio from {0.1, 0.2}. We set the maximal sequence length 512 and the maximal training epoch as For entity typing, we select learning rate from (le-5, 2e-5, 3e-5, 5e-5} and batch size from {128, 256}. We set the maximal sequence length be 256, the last layer dropout ratio to be 0.1. The model fine-tuned for at most epochs to prevent overfitting. The threshold for type prediction is selected on the validation set.
select learning rate from (1e-5, 5e-6, 2e-5,
Context Collection for QA Datasets For WebQuestions, we collect evidence context using the document retriever of DrQA (Chen et al.] 2017). which uses TF-IDF based metric to retrieve the top Wikipedia articles. For Q T, we use Lucene ranked paragraphs. For SearchQA and TriviaQA,
paragraphs ranked by search engines. Following existing research (Wang al. 2018b Lin et al. 20188) we use sentence-level paragraphs for SearchQA (50 sentences), TriviaQA (100 sentences) and SearchQA (100 sentences).
Correlation between Fact Completion Results and Properties of Relations Figure 2 shows the fact completion results of BERT are unstable on different relations with different properties, i.e., BERT's performance is strongly correlated with the size of candidate entity set and the number of groundtruth answers. Compared to BERT, WKLM often less sensitive to these two factors.
50
40
Hits@10
Hits@10 10
WKLM BERT-large
-20
WKLM BERT-lange
-10
200
300
400
500
900 500
700
800
2
Number candidate entities
Number of Groundtruth Answers
Figure 2: Left: Correlation between candidate set size and hits@ 10; Right: Correlation between number of groundtruth answers and hits @ 10.
13


Reviews:
[{"review_id": "r1llQTd0FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposed to improve pre-training of language models (e.g. BERT) by incorporating information around entities based on English Wikipedia. The idea is very simple and straightforward: it takes all the anchor links from Wikipedia and replaces some entities by randomly sampling negative ones of the same entity type (according to Wikidata) and adds an extra binary prediction task which predicts if the entity has been replaced or not. \n\nThe model was initialized by BERT (or the authors’ BERT reimplementation) and trained for another 1M steps with the new training objective and reduced % of masking tokens.\n\nThe model was evaluated on a fact completion task (created by the authors on the 10 sampled Wikidata relations) and several open-domain QA datasets and an entity typing dataset FIGER, and achieved significant improvements on the BERT baselines.\n\nOverall, I think this is a strong paper. The idea is simple but effective, the experiments are thorough and improvements over the BERT baselines are significant. \n\nBelow are some concerns I had when I read the paper and also some suggestions on how to improve this paper: \n\n1) I am slightly concerned about the evaluation of the fact completion task and its baselines. \n\n- Why are there only 190-906 candidates for these relations? How were the candidates chosen? Why not use the full set of possible candidates of that entity type?\n\n- I am not sure why you picked the most common entities for predictions. Fact completion for rare entities would be more challenging and practical. Also, the models might favor choosing more common entities as well. \n\n- I am also not sure if the BERT baseline (by using k [MASK] tokens when the candidate answer has k tokens and taking the average of the k probabilities) is a strong one or not in this setting, as BERT was not trained in this way and it is unclear if this would make BERT favor shorter entities or not.\n\n2) OpenQA results (Table 4): there is a very strong baseline coming out recently (an EMNLP’19 paper): \n\nMulti-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. \n\nEven for their BERT-base model, TriviaQA F1 was 67.5, SearchQA F1 was 70.6 and Quasar-T F1 was 59.0. It is okay to not directly compare to their results (the focus is different), but the authors should be aware of their results and perhaps remove the state-of-the-art claim.\n\n3) I’d be interested in seeing more ablation studies on the importance of masking/replacement choices. What is the percentage of entities that have been replaced? 50%? The only thing I can find is that no adjacent entities have been replaced at the same time. How important is that?  I imagine that the percentage of entities that have been replaced should also matter the performance significantly. \n\n4) If I understand correctly, the model was first trained (as BERT) on English Wikipedia + BooksCorpus and then later trained only on English Wikipedia. I wonder how important the first stage would still be. Could add an experiment that trains on Wikipedia only?\n\nMinor suggestions:\n1) Please use “English Wikipedia” instead of “Wikipedia” (#BenderRule)\n2) Table 1: don’t put “572” next to “Average Hits @10”. It is confusing. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxt_nQFtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to incorporate world knowledge for the pretraining approach so that (1) pretrained models contain useful information about the world, and (2) benefit downstream NLP tasks. The paper does so by introducing the objective which distinguishes the groundtruth entity and the false entity in the Wikipedia text. This was carefully done by detecting entities in the text, find the corresponding entity in Wikidata, randomly choose another entity which has the same type as the original entity, and make sure this doesn’t happen for neighboring entities or too much in order to avoid context change. Adding this objective to the original masked LM objective, this pretrained model is shown to be effective and outperform baselines significantly in many tasks such as zero-shot fact completion, question answering, and fine-grained entity typing.\n \nParticularly, I appreciate (1) the fact that they compare with other knowledge-aware pretrained models such as ERNIE, and (2) their ablation in Table 6 which compares only knowledge learning objective, more masking or finetuning with knowledge learning objective starting from MLM instead of multi-task learning (actually, is BERT+1M MLM updates a correct term? Since MLM is masked LM, shouldn’t it be BERT+1M knowledge learning?): it clearly shows that it is important to do both MLM and knowledge learning, appropriate ratio for masking, and multi-task learning between MLM and knowledge learning instead of some kind of pretrain-finetune approach.\n\nSome marginal concerns I have is that some settings for downstream tasks are not clear. For example, for WebQuestions the number of validation examples is not specified, but I believe they should have split the train set into train/valid for development (such as early stopping or hyperparameter tuning), and conventionally people have split the train set into 90/10 for train/valid. In addition, as far as I know, TriviaQA has a bunch of settings such as Wiki setting, Web setting, unfiltered setting and open setting. What exactly is this setting? The paper mentions they follow Lin et al. 2018, but the statistics shown in Table 2 are different from their statistics. In addition, the authors compare results in different settings in Table 4 (for example, ORQA is in open setting, while I think the setting in this paper is not).\n\nSome clarification questions:\n\n1. I understand Wikipedia anchor links means hyperlinks. How do you know that the mention with a hyperlink is always an entity? Also, there are many cases that the mention with the hyperlink does not match in meaning with the linked article. For example, in “https://en.wikipedia.org/wiki/Barack_Obama”, there is a sentence “He was elected over Republican John McCain and was inaugurated on January 20, 2009” where “elected” is linked to “https://en.wikipedia.org/wiki/2008_United_States_presidential_election” which is probably not what we want.\n\n2. If there are same mentions in the text chunk, are all of them replaced? Probably with the same negative entity?\n\n3. How often were entities to replace chosen? E.g. is one entity in the text chunk replaced at a time? Or multiple ones?\n\n4. How do you look up the type of the entity? For instance, in the example of “Marvel Comics” illustrated in Figure 1, I see there are two triples, <Marvel Comics, instance of, business> and <Marvel Comics, instance of, book publishing company>. Is \"instance of\" used? How did you choose “book publishing company” over “business”? Or is it randomly chosen? I think ideally more fine-grained type should be chosen, but wonder if there is a way to find which one is more fine-grained.\n\n5. Perhaps not necessary, but I wonder what happens if an entity is detected, and there is another masked LM objective which only predicts this entity, without the process of looking up its type and randomly choose the negative entity.\n- I’m curious because I’m not sure how much effect do negative entities have. Although they are the same type of entities, it’s hard that the chosen entity is a strong negative entity. For instance, Barack Obama (Q76) is an instance of ‘human’, and I don’t think another entity of human that is randomly chosen is helpful. A similar observation was found in WikiHop dataset, which is a multi-choice QA dataset that all negative candidates are guaranteed to be the same type, but still question-candidates-only baseline (without context paragraphs) outperforms state-of-the-art models (https://openreview.net/forum?id=B1lf43A5Y7). It indicates that entities with the same type are not strong negative candidates.\n- For this reason, I think it is possible that negative entities do not help, but the fact that the training objective is more entity-centric helps the performance on various different downstream tasks. That’s why I think the ablation with entry-centric masked LM can be good to see.\n- Of course, this is just one of my hypotheses, and I believe the results in this paper are significant regardless, but I just wonder what authors think about this.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkeu-1ZTKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to trained better entity centric text embeddings by switching entities mentioned in the text to some other entities with the same type. The target is modeled as a binary classification task, which is trained jointly with the MLM loss. The authors do experiments on multiple tasks, and the model shows strong performance on all tasks. And the ablation study justifies that \"knowledge pre-training\" is crucial. The idea is novel and the experiment results suggest that the additional \"adversarial\" target helps. The writing is clear in general, but misses some implementation details. \n\nA few questions:\n1. In section 3.1, how do you rank the candidate answers with your model? Do you compute the logits in the same way as the baseline models?\n2. In section 3.2.1, why do you use a different split for TriviaQA? Do you rerun the baseline models on this new split?\n3. In section 3.2.1, the author claims that most answers in TriviaQA, SearchQA and Quasar-T datasets are entities. A interesting metric to evaluate would be how much improvements WKLM obtain on those questions, versus those whose answers are text spans.\n3. In section 3.2.2, the authors mention that they use the CLS token to predict the entity types. How do you train the embedding your CLS token?\n4. For the entity typing task in section 3.2.2, do you fine tune your model or it's evaluated in a zero-shot setting?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a secondary objective when learning language models like BERT that improves the ability of such models to learn entity-centric information. This additional objective involves predicting whether an entity has been replaced. Replacement entities are mined using wikidata.

Strengths:
-The proposed method is simple and shows significant performance improvements for various tasks including fact completion and question answering.

Weaknesses:
-The experimental settings and data splits were not always clear. This was sufficiently addressed in a revised version.
-The paper could have probed performance on tasks involving less common entities.

The reviewer consensus was to accept this submission.

Decision: Accept (Poster)