Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
This paper proposed to improve pre-training of language models (e.g. BERT) by incorporating information around entities based on English Wikipedia. The idea is very simple and straightforward: it takes all the anchor links from Wikipedia and replaces some entities by randomly sampling negative ones of the same entity type (according to Wikidata) and adds an extra binary prediction task which predicts if the entity has been replaced or not. 

The model was initialized by BERT (or the authors’ BERT reimplementation) and trained for another 1M steps with the new training objective and reduced % of masking tokens.

The model was evaluated on a fact completion task (created by the authors on the 10 sampled Wikidata relations) and several open-domain QA datasets and an entity typing dataset FIGER, and achieved significant improvements on the BERT baselines.

Overall, I think this is a strong paper. The idea is simple but effective, the experiments are thorough and improvements over the BERT baselines are significant. 

Below are some concerns I had when I read the paper and also some suggestions on how to improve this paper: 

1) I am slightly concerned about the evaluation of the fact completion task and its baselines. 

- Why are there only 190-906 candidates for these relations? How were the candidates chosen? Why not use the full set of possible candidates of that entity type?

- I am not sure why you picked the most common entities for predictions. Fact completion for rare entities would be more challenging and practical. Also, the models might favor choosing more common entities as well. 

- I am also not sure if the BERT baseline (by using k [MASK] tokens when the candidate answer has k tokens and taking the average of the k probabilities) is a strong one or not in this setting, as BERT was not trained in this way and it is unclear if this would make BERT favor shorter entities or not.

2) OpenQA results (Table 4): there is a very strong baseline coming out recently (an EMNLP’19 paper): 

Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. 

Even for their BERT-base model, TriviaQA F1 was 67.5, SearchQA F1 was 70.6 and Quasar-T F1 was 59.0. It is okay to not directly compare to their results (the focus is different), but the authors should be aware of their results and perhaps remove the state-of-the-art claim.

3) I’d be interested in seeing more ablation studies on the importance of masking/replacement choices. What is the percentage of entities that have been replaced? 50%? The only thing I can find is that no adjacent entities have been replaced at the same time. How important is that?  I imagine that the percentage of entities that have been replaced should also matter the performance significantly. 

4) If I understand correctly, the model was first trained (as BERT) on English Wikipedia + BooksCorpus and then later trained only on English Wikipedia. I wonder how important the first stage would still be. Could add an experiment that trains on Wikipedia only?

Minor suggestions:
1) Please use “English Wikipedia” instead of “Wikipedia” (#BenderRule)
2) Table 1: don’t put “572” next to “Average Hits @10”. It is confusing. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes to trained better entity centric text embeddings by switching entities mentioned in the text to some other entities with the same type. The target is modeled as a binary classification task, which is trained jointly with the MLM loss. The authors do experiments on multiple tasks, and the model shows strong performance on all tasks. And the ablation study justifies that "knowledge pre-training" is crucial. The idea is novel and the experiment results suggest that the additional "adversarial" target helps. The writing is clear in general, but misses some implementation details. 

A few questions:
1. In section 3.1, how do you rank the candidate answers with your model? Do you compute the logits in the same way as the baseline models?
2. In section 3.2.1, why do you use a different split for TriviaQA? Do you rerun the baseline models on this new split?
3. In section 3.2.1, the author claims that most answers in TriviaQA, SearchQA and Quasar-T datasets are entities. A interesting metric to evaluate would be how much improvements WKLM obtain on those questions, versus those whose answers are text spans.
3. In section 3.2.2, the authors mention that they use the CLS token to predict the entity types. How do you train the embedding your CLS token?
4. For the entity typing task in section 3.2.2, do you fine tune your model or it's evaluated in a zero-shot setting?


review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper aims to incorporate world knowledge for the pretraining approach so that (1) pretrained models contain useful information about the world, and (2) benefit downstream NLP tasks. The paper does so by introducing the objective which distinguishes the groundtruth entity and the false entity in the Wikipedia text. This was carefully done by detecting entities in the text, find the corresponding entity in Wikidata, randomly choose another entity which has the same type as the original entity, and make sure this doesn’t happen for neighboring entities or too much in order to avoid context change. Adding this objective to the original masked LM objective, this pretrained model is shown to be effective and outperform baselines significantly in many tasks such as zero-shot fact completion, question answering, and fine-grained entity typing.
 
Particularly, I appreciate (1) the fact that they compare with other knowledge-aware pretrained models such as ERNIE, and (2) their ablation in Table 6 which compares only knowledge learning objective, more masking or finetuning with knowledge learning objective starting from MLM instead of multi-task learning (actually, is BERT+1M MLM updates a correct term? Since MLM is masked LM, shouldn’t it be BERT+1M knowledge learning?): it clearly shows that it is important to do both MLM and knowledge learning, appropriate ratio for masking, and multi-task learning between MLM and knowledge learning instead of some kind of pretrain-finetune approach.

Some marginal concerns I have is that some settings for downstream tasks are not clear. For example, for WebQuestions the number of validation examples is not specified, but I believe they should have split the train set into train/valid for development (such as early stopping or hyperparameter tuning), and conventionally people have split the train set into 90/10 for train/valid. In addition, as far as I know, TriviaQA has a bunch of settings such as Wiki setting, Web setting, unfiltered setting and open setting. What exactly is this setting? The paper mentions they follow Lin et al. 2018, but the statistics shown in Table 2 are different from their statistics. In addition, the authors compare results in different settings in Table 4 (for example, ORQA is in open setting, while I think the setting in this paper is not).

Some clarification questions:

1. I understand Wikipedia anchor links means hyperlinks. How do you know that the mention with a hyperlink is always an entity? Also, there are many cases that the mention with the hyperlink does not match in meaning with the linked article. For example, in “https://en.wikipedia.org/wiki/Barack_Obama”, there is a sentence “He was elected over Republican John McCain and was inaugurated on January 20, 2009” where “elected” is linked to “https://en.wikipedia.org/wiki/2008_United_States_presidential_election” which is probably not what we want.

2. If there are same mentions in the text chunk, are all of them replaced? Probably with the same negative entity?

3. How often were entities to replace chosen? E.g. is one entity in the text chunk replaced at a time? Or multiple ones?

4. How do you look up the type of the entity? For instance, in the example of “Marvel Comics” illustrated in Figure 1, I see there are two triples, <Marvel Comics, instance of, business> and <Marvel Comics, instance of, book publishing company>. Is "instance of" used? How did you choose “book publishing company” over “business”? Or is it randomly chosen? I think ideally more fine-grained type should be chosen, but wonder if there is a way to find which one is more fine-grained.

5. Perhaps not necessary, but I wonder what happens if an entity is detected, and there is another masked LM objective which only predicts this entity, without the process of looking up its type and randomly choose the negative entity.
- I’m curious because I’m not sure how much effect do negative entities have. Although they are the same type of entities, it’s hard that the chosen entity is a strong negative entity. For instance, Barack Obama (Q76) is an instance of ‘human’, and I don’t think another entity of human that is randomly chosen is helpful. A similar observation was found in WikiHop dataset, which is a multi-choice QA dataset that all negative candidates are guaranteed to be the same type, but still question-candidates-only baseline (without context paragraphs) outperforms state-of-the-art models (https://openreview.net/forum?id=B1lf43A5Y7). It indicates that entities with the same type are not strong negative candidates.
- For this reason, I think it is possible that negative entities do not help, but the fact that the training objective is more entity-centric helps the performance on various different downstream tasks. That’s why I think the ablation with entry-centric masked LM can be good to see.
- Of course, this is just one of my hypotheses, and I believe the results in this paper are significant regardless, but I just wonder what authors think about this.
review_title: Official Blind Review #2

Meta-review: This submission proposes a secondary objective when learning language models like BERT that improves the ability of such models to learn entity-centric information. This additional objective involves predicting whether an entity has been replaced. Replacement entities are mined using wikidata.

Strengths:
-The proposed method is simple and shows significant performance improvements for various tasks including fact completion and question answering.

Weaknesses:
-The experimental settings and data splits were not always clear. This was sufficiently addressed in a revised version.
-The paper could have probed performance on tasks involving less common entities.

The reviewer consensus was to accept this submission.

Decision: Accept (Poster)