Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: (This review is a collaboration between a senior and a junior reviewer. The junior wrote most of the review, and the senior modified the review. Both reviewers read the paper in detail.)

Summary:

This paper presents an alternative pretraining technique for language models in order to make them more "knowledge aware". In addition to pre-training a model with the traditional language modeling objective, this work proposes an entity prediction task as pre-training. Furthermore, this work also incorporates entity tokens at the input level of the model by summing the word embedding and its corresponding entity embedding.

Experimental results show that such pretraining method yields models with greater factual knowledge according to the LAMA knowledge probing task compared to vanilla GPT2 models. In addition, it is also reported that such models perform better than a GPT2 model of the same size on TriviaQA, Natural Questions and Web Questions, in a zero-shot setting; and achieves competitive results when compared to larger models.

This paper is well written and easy to follow. This work is timely because it shows that increasing the model size and pre-training data size is not the only way to achieve strong performance on language related tasks (which is the current trend). Inductive biases like knowledge about entities can improve the performance of models to the point of achieving competitive results with bigger models.

Some limitations of this work include the lack of experiments on more diverse architectures like encoder-decoders to make sure that the proposed technique not only works on autoregressive GPT-type models. Moreover, only one approach for entity integration is explored.

Other than that, I have the following questions:

Interesting choice of the margin loss for entity prediction. More motivation should be added into why this loss instead of the cross-entropy loss. Optionally, an experiment comparing the performance of the model when trained with the cross entropy -vs- the margin loss would be interesting to have.

At inference time, you are forced to put entity labels on the input text as a preprocessing step. How much does this impact the generalization capacity of the model? How do you manage cases when an unseen entity is seen as input?

Are the experiments on QA really zero-shot? The examples presented in Appendix are not dummy as they reflect true questions and answers?

Minor:

Have you tried your pre-training technique on an encoder-decoder architecture like T5? It would be interesting to see that this technique not only works with auto-regressive models.

Have you tried to fine-tune your resulting pre-trained models on various tasks? It would be interesting to see if your pretraining method also helps finetuning more efficiently on some tasks. In addition, I would be curious to see if, while fine-tuning, the entity prediction accuracy and the LAMA probing performance of the model drops or remains more or less the same.

[1] Sun, Yu, et al. "Ernie: Enhanced representation through knowledge integration." arXiv preprint arXiv:1904.09223 (2019). [2] Ji, Yangfeng, et al. "Dynamic Entity Representations in Neural Language Models." Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.


After author response:

Thanks for the response! After reading other reviews, I feel the novelty of this work is somewhat limited and it would be useful to highlight the differences with respect to previous work. Also a discussion on when your method is preferred over existing proposals. I am also decreasing my confidence score after reading other reviews.
review_title: Simple method to improve a language modeling knowledge about entities

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes KALM - a knowledge-aware language model that incorporates entity information. Specifically, the method involves using a dictionary lookup to match n-grams to entities from a database. Then, each token is embedded into two vectors — one using the surface form, and one using the entity ID it is matched to (if there is a match). These embeddings are then added and passed into a Transformer model to predict the next word given the context. Experiments compare the model to GPT-2 and T5 (two state-of-the art Transformer-based LMs) and demonstrate the benefit of adding entity information. 

Strengths: The idea of using entity information is certainly interesting and new. Most details of the paper are clear from the writing.

Weaknesses: The paper leaves open several questions that should really be explored and discussed. Given that the modeling contribution is simply adding an extra embedding layer, it would be good to have detailed analyses that provide more insights into the contribution itself. (see points 5,6,7 below)

Detailed comments/questions:
1. The details of the dictionary lookup are not entirely clear. Do you lowercase all tokens? Do you handle small misspellings in the tokens? Do you perform fuzzy matches?
2. Thanks for providing several implementation details. However, the values of hyperparameter $\alpha$ are not discussed. Does this require a lot of tweaking or is the method stable to this choice?
3. (minor) The ‘zero-shot’ experiments really seem to be ‘few-shot’ tests since you provide the model with 8 or so examples. 
4. The metrics used for the LAMA knowledge probing should be explained in the paper once (at least expand the abbreviations so readers can understand). 
5. Table 3: The parameter sizes indicated are misleading. Large KALM has close to 700M params and not 300M as indicated. Similarly base KALM has ~550M params not 100M. 
6. Related to the above point, it seems like KALM base has more parameters than GPT-2 base (and even GPT-2 Large) and yet this aspect is not accounted for/investigated in the paper. In fact, the paper says “(KALM) more efficiently packs practical knowledge into its parameters … 12L KALM is very close … to 24L GPT-2 Large”. This seems to be very misleading information. Since Transformers are essentially a form of graph networks, I’m not sure entity embeddings can be treated separately from the model parameters since one can view those as edges part of the graph. 
Concrete suggestion: Can you compare KALM vs GPT-2 (in all tables and Fig 1) while controlling for the #params? To me, the only closest comparison seems like KALM base vs GPT-2 Large where GPT-2 Large seems to be the better model?
7. I appreciate the authors efforts to test on a variety of benchmarks and I understand that training these huge models is not easy, but to me, the essence of the paper (i.e. addition of entity information) seems under-explored. For instance, could you perform ablations studies that only uses the entity embeddings in the input to see how much knowledge it provides? Another ablation can be on the matching done in the dictionary lookup (in terms of frequency cutoffs, n=1,2,3,4 in the n-gram matching, etc. ). One could also analyze the prediction accuracy of the model to see which types of entities (e.g. persons, organizations, etc.) work better vs those that don’t help as much. 

-------
Update: Thanks to the authors for their response, which helped clarify several of my minor questions and I believe those can be revised with a writing pass. However, I still think this paper has two significant deficiencies:

1. The parameter size comparison still seems flawed to me. The authors say that one can discount the entity embeddings, but can we really? Aren’t they part of the model’s ‘representation’ even if the inference does not use all of them in a single forward pass? Several neural net architectures exist (including large-scale LMs) that do not use all inference paths but still count them in the total params. The BERT vs GPT-2 example provided in the rebuttal is only a difference of 26k tokens (still significant!) but here we're talking about a few hundred million parameters!     
At the very least, I think the true sizes of the models must be acknowledged and one can add the point on entity embeddings vs 'brain' parameters as a caveat, but it seems scientifically inaccurate to me to claim otherwise. To be clear, I don't think the size issue detracts much from the main contribution of adding entity embeddings here (i.e. this work may still be of interest even if the size of model is larger), but the current version of the paper has several claims about size savings that seem incorrect.

2. The analysis of the proposed method (where it helps, where it fails, which hyperparams matter) is still lacking. The authors did mention one ablation in the supplementary that I missed, but I don’t think that is sufficient for a reader to understand how to build on this method in this future without re-running all the experiments, doing an extensive hyperparam search, etc.

review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces to use additional knowledge to pretrain language models, which adds entity information to input and output. And the method improves the performance compared to the corresponding models without the knowledge. This is an interesting piece of work and worth publishing. However, I have some comments and questions about the article. 

Since the main point of the paper is adding entity information to the models, the paper would be better to include more explanations about the entity information. For example, with the dictionary look-up, what is e_i exactly? Why did the authors use such knowledge rather than other knowledge like POS or so? Why do we need negative sampling? How many classes are there for entity? 

The models enjoy more information provided by entity. Then what about more data to provide more information? How much can we improve the performance by adding entity information rather than additional data samples? 

The authors said that the models were trained with entity information from scratch. Can we use pretrained models, since we can simply add entity information to the input and output layers? It seems like that the additional information can fine-tune the pretrained models.

What is the role of the dummy QAs? And how did the authors come up with the 8 dummy QAs?  


review_title: entity information needs further explanation

final_score: rating: 5: Marginally below acceptance threshold
review_content: - Summary

This paper presents a knowledge-aware language model pretraining method without changing model architecture. Specifically, they add entity prediction task along with language modeling task to make the model aware of knowledge. Experiments show improved results on the LAMA knowledge probing task compared to GPT-2 models. They also show comparable results on zero-shot question answering task, even with only 2% transformer parameters compared to GPT-2 17B.

- Strengths

1. The authors propose a simple (i.e. without changing model architecture) method to give knowledge-aware signal during pretraining.

2. Proposed method consistently outperforms GPT-2 on LAMA and zero-shot QA.

3. Interesting to see that entities from fuzzy frequency-based matching can make LMs significantly better.

- Weaknesses & Questions

1. Need extra parameters to save entity embedding (~471M). However, as the authors mentioned, this can be viewed as an external memory module and embedding lookup doesn't need much computation.

2. I would like to see how much KALM can transfer to downstream tasks. The authors already show zero-shot QA results, but it would be better to see fine-tuning results on knowledge-intensive tasks. For instance, will KALM still outperform GPT-2 after finetuned on KILT tasks (https://arxiv.org/abs/2009.02252)?
(I know KILT released right before the due date of ICLR, but tasks in KILT are all already published)

3. Even though KALM outperforms GPT-2, it is still unclear for me the advantages of KALM compared to Entities as Experts (or EaE, https://arxiv.org/abs/2004.07202). For example, EaE outperforms KALM on the LAMA probing tasks, and EaE also outperforms T5-11B when finetuned.

4. The authors mentioned fuzzy frequency-based entity matching is better than precise entity linking (e.g. EaE) on page 3, but EaE outperforms KALM on LAMA probing tasks. Based on the experimental results, precise entity linking seems more effective to me. It would be helpful if the authors can provide a more thorough comparison with EaE.

- Typo?

Page 3, "United States" at $w_{i:i+2}$ --> "United Staes" at $w_{i:i+1}$ ?

=======================================================================

Post AR:

Thank you for preparing detailed responses, which helped me to clarify several questions. However, one of my major questions is still unclear.

Although 4 out of the 11 KILT tasks are already included in the main paper, most of them are LAMA knowledge probing tasks or zero-shot QA tasks. It is still unclear how much and how robustly KALM can transfer to other downstream tasks with fine-tuning (e.g. Wizard-of-Wikipedia, FEVER, QA with fine-tuning). For instance, CorefBERT paper (which uses a similar idea but as you mentioned it use bidirecitonal attention) shows its transferability on QUOREF, six extractive QA benchmarks, DocRED, FEVER, five coreference resolution benchmarks and GLUE, which can convince me to choose CorefBERT over BERT.

Experiments on the paper are promising but not diverse enough to make me choose KALM over GPT2. Thus, I would like to stick to my original score.
review_title: Official Blind Review #1

Meta-review: The authors propose to improve the LMs ability on modelling entities by signalling the existence of entities and also allowing the model to represent entities also as units. The embeddings of the surface form and then entity unit are then added and passed through a layer to predict the next word.
The paper evaluates on QA and conducts probing tasks and shows that such an entity modelling results in better performance.

All reviewers have found the idea conceptually simple and novel. At the same time a number of concerns are raised, with the most important being the lack of understanding around which and how hyper-parameters matter for this model and, most importantly, the confounder introduced to the model by the much larger size of parameters introduced by the embedding layers. While the authors comment that not all the parameters are used all the time, the size of the embeddings still count at the total size of parameters a model has. Thus, without properly controlling for this (e.g., have an another model where the extra embedding params are given to another part of the model), it is difficult to determine whether adding more parameters was the solution or adding more parameters for modelling the entities. 
Decision: Reject