Pretrain Knowledge-Aware Language Models
Authors: Corbin L Rosset
Chenyan Xiong
Minh Phan
Xia Song
Paul N. Bennett
saurabh tiwary
Venue:  (2021)
Keywords: Pretraining
Natural Language Generation
GPT-2
QA
Knowledge Graph

Abstract: How much knowledge do pretrained language models hold? Recent research observed that pretrained transformers are adept at modeling semantics but it is unclear to what degree they grasp human knowledge, or how to ensure they do so. In this paper we incorporate knowledge-awareness in language model pretraining without changing the transformer architecture, inserting explicit knowledge layers, or adding external storage of semantic information. Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task. Our experiments show that solely by adding these entity signals in pretraining, significantly more knowledge is packed into the transformer parameters: we observe improved language modeling accuracy, factual correctness in LAMA knowledge probing tasks, and semantics in the hidden representations through edge probing. We also show that our knowledge-aware language model (\kalm{}) can serve as a drop-in replacement for GPT-2 models, significantly improving downstream tasks like zero-shot question-answering with no task-related training.  

Under review as a conference paper at ICLR 2021
PRETRAIN KNOWLEDGE-AWAN LANGUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
How much knowledge do pretrained language models hold? Recent research ob- served that pretrained transformers adept at modeling semantics but it is unclear to what degree they grasp human knowledge, or how to ensure they do so. In in language model pretraining without changing the transformer architecture, inserting explicit knowledge layers, adding external storage of semantic information. Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with entity- extended tokenizer; and at the output, with an additional entity prediction task. Our experiments show that solely by adding these entity signals in pretraining, signif- icantly more knowledge is packed into the transformer parameters: we observe improved language modeling accuracy, factual correctness in LAMA knowledge probing tasks, and semantics the hidden representations through edge probing. We also show that our knowledge-awee language model (KALM) can serve as drop-in replacement for GPT-2 models, significantly improving downstream tasks like zero-shot question-answent with task-related training.
this paper we incorporate showledge-awarente
INTRODUCTION
The strong effectiveness and rich generalization ability of pretrained language models (PLMs) 3 5) have raised many questions about what is captured in transformer networks and why. Recent explorations found the pretrained language models may "rediscover" the linguistic pipeline at various transformer layers (6), can serve as implicit knowledge bases for relation extraction 7 perform soft reasoning tasks 8 9 and conduct some language tasks reasonably in fully unsupervised, zero-shot, fashion 3 10). With sufficiently large amount of parameters, i.e. several billions, and enough task-specific supervision, the pretrained language models even directly generate answers for natural language questions, at the same accuracy with statc-of-the-art reading comprehension systems, without using any context documents knowledge graphs (11).
Impressive as they are, language models are still far from ready to serve as an "unsupervised multi-task learner that learns knowledge directly from human language and generalizes to downstream language
tasks 3 There are notable gaps of language models' performances on downstream tasks between
models with (11 12) and without (3) large amounts of task-specific fine-tuning. The language models still (de- geenrrate dull. fictually-incorree
or dream-like text when used in natural language generation [4: 16). These challenges often necessitate over-parameterization (17), grounding on external structural semantics (14) I6] 18 19), or large amount of task-specifie fine-tuning (II) which are costly, complicated, and not always feasible for every language task.
One potential limitation of these language models is their style of pretraining, e.g, auto-regressive language modeling (3) or masked language modeling (I) wherein transformer networks process sequence of words and are asked to predict the next/masked words. There is no explicit guidance to the transformers that humans prefer them to capture correct, real-world information. As result, all the knowledge captured in these pretrained language models is only signaled by patterns of co-occuring words the input sequence that is learned implicitly during pretraining.
In this paper, instead of creating bigger models or adding knowledge-ppecific architectures, propose to more efficiently leverage the existing parameters in the standard transformer language model, by simply making them aware of the various forms entity manifest itself as, and its role in the surrounding text. More specifically, this knowledge-awaid awwrnness is communicated via the input fed to PLMM and in the output expected from them during pretraining. For input-awareness,

---

Under review as a conference paper at ICLR 2021
entity-namm (surface form) dictionary that tokenizes word spans to their most popularly referred-to entity, e.g., as fuzzy frequency- based entity annotations (20). and serve these entity tokens parallel imput channel along with the word tokens. For output-awarrness, in addition to the language modeling objective, we add an entity prediction task that guides the model to distinguish the correct entity from various negative distractions. The two objectives together explicitly guide the language model to predict not only the correct words, but also the correct entity behind those words during pretraining, without changing the network architecture.
By adding knowledge awareness to GPT-2 style auto-regressive language models, our pretrained language model, "Knowledge-Awir Language Model" (KALM), shows significantly improved han- dling of knowledge-sensitive tasks. In the LAMA knowledge probing tasks (7), KALM outperforms its entity-unawan baseline, GPT-2, by about 25% across all tasks at both base and large transformer sizes. Our 24 layer KALM (Large) is even comparable with the 17 Billion parameter GPT-2 on some tasks. It more accurately captures commonsense knowledge, factual semantics, and also relation semantics in these LAMA tests. The knowledge signals also aid generic language understanding: we have observed better language modeling perplexity and word prediction accuracy with KALM too.
The advantages in language modeling also transfer to downstream tasks. In zero-shot question answering, the exact match accuracy of the answers generated by KALM are 20%- 100% better than those of an equivalent GPT-2 model. We did not use any task-specific supervision or additional gradient updates, relying solely on the unsupervised knowledge learned in KALM. We only feed in few example question-answer pairs templates format how generated answers should look. Injecting rich knowledge signals leads to improvements approximately equal to those gained by doubling the transformer layers, indicating that PLMs can be trained more efficiently growing the parameters exponentially is not the only way to improve language understanding.
To better understand pretraining and the advantage of knowledge awareness, we leverage the edge probe technique 21) and dissect what is Iearned in the representations at various gradient step numbers throughout pretraining. We observe that the auto-regressive transformers start to learn the basis of language in the beginning of the pretraining, and gradually learns more complex semantics in the process; adding nooledge-awarrene
greatly accelerates learning of higher level semantics, e.g.. coreferences and entity types, and helps the model perform better in those more complicated tasks.
PRETRAINING SNOWLEDGE-AWAR LANGUAGE MODELS
In this section we first present preliminaries in language modeling and then how we add knowledge- awareness in their pretraining.
2.1 PRELIMINARY
In this paper, without loss of generality, we mainly focus on the auto-regressive language modeling. Considering the text X as a sequence of tokens (words or sub-words): X (Wi.... Wy...., Wn the classical unidirectional factorization of language probabilities (22) (23) describes:
x(X)= = II p(wxtwe))
(1)
where Wei refers to all the tokens appear before i. This conditional probability can be parameterized in various ways. An effective choice use the uni mii-directional transformer, as done in GPT-2 (3): p(us|we)) trans r (w:; (w::/Wes).
The language modeling task provides a large amount of data to pretrain very deep transformer networks 24)4 Scaling up the transformer parameter sizes will lead to significant improvements in the language model capability: with wider and deeper transformer layers, it is observed that transformer language models start to output more complicated semantics beyond lexical and syntactic patterns 6 g (7). On the other hand, a roughly log-linear relationship between transformer size and output quality has been established, e.g. doubling the quality requires ten times more parameters and training data 33) I7 I0). Even in industry, the marginal gain of increasing parameters will eventually be outweighed by the cost to train and serve such models.
2

---

Under review as a conference paper at ICLR 2021
2.2 KNOWLEDGE-AWARK PRETRAINING
As shown in Eqn. [ the pretraining is solely at the token level. All the semantics in PLMs are captured by the transformer indirectly; there is explicit requirement in pretraining to better capture knowledge
we expect them capture knowledge beneath the raw word sequences implicitly, e.g.. to generate factually correct statements.
In this work we mitigate this discrepancy by making transformer networks aware of knowledge in language model pretraining. Instead of stacking more layers or adding external knowledge storage, we present into the same amount of transformer parameters. The first step to introduce knowledge awareness is an entity tokenizer that forms an additional entity token sequence (26) to signal the existence of entities in the input and outpul of the pretraining process.
showledge-award language modeling (KALM) framework that packs more information
Entity Tokenizer. An entity tokenizer segments the text sequence into entity ids using a surface form dictionary, which maps word-ngrams to entities: Waith diet look up Ct. where is the most popular entity referred by the word k-gram Waitks and 6, = null if we not part of any known entity surface names.
This simple dictionary look-up can be conducted efficiently, similar to the (sub)word tokenizer. Simultaneously, the text is tokenized into two channels a word-entity duet token sequence (26):
{uei,... Wis wr} (ei,..., Cs. er] Entity Sequence.
Word Sequence;
Xdwet
The two sequences are aligned position by position. If multiple (sub)words together form an entity name, the corresponding entity id is duplicated in each position corresponding these words. For example, the name "United States" at Wii++2 is mapped to entity "USA" in e, and 6;+1-
Instead of enlisting a more precise entity linker supervisions from entity labels (16) 18) fuzzy frequency-based dictionary look places higher expectations on the model to use clues in the text to jointly build token and entity representations that better reflect how language conveys knowledge. Using a highly tuned entity linker would propagate its own biases into the transformer.
Knowledge-Awel Aware Input. Just as there is input embedding for every word token, allow the model to learn an entity embedding for each entity:
E Embedding, (c.) E Rd
(3)
Wi Embeddingg,, Rd
(4)
The two embeddings are combined to form the knowledge aware input:
= J L, (e)) Li Linear, E Rdd.
(5)
All the embeddings are randomly initialized and learned in pretraining.
Knowledge-Aware Output. The knowledge-aware input is fed into standard transformer layers, the same the word-only input. Then in pretraining. besides the next-word prediction task, also employ next-entity prediction task to further incorporate knowledge-avareness
Specifically, we use one output head for the word probability, one for the entity, and share all transformer layers between words and entities. If there are L transformer layers and h is the output of the final layer's i-th token, the loss for position is computed as
le(elttes) max(0, shh," ,e) s(h, ,e=) + -h
(6)
s(hi G.)) = cos cos ,, ),))
(7)
L
h trans
(8)
The transformers in Eqn. 3 are stacked multiple times similar to GPT-2. The scoring function in Eqn. proojecss the hidden state into the embedding space with I. near layer and takes the cosine
3

---

Under review as a conference paper at ICLR 2021
similarity with an arbitrary entity ej. Assuming that position i is linked by our entity linker to es. we carefully choose a corresponding negative entity e to contrast with e, using margin loss with margin À in Eqn. [
Pretraining. The knowledge- aware input and output are incorporated in the standard multi-task set up for our KALM knowledge-aware language model pretraining:
IKALM (Xduet
Σ la (p(wr(()))) + als(es)ter)
combines the language modeling loss 1e - cross-entropy as in standard PLMs with the entity prediction loss (), where a is a
to balance the two.
Inference. At inference time- whenever generating output text - KALM use the word prediction head p(wy)(-e)) which is consistent with GPT-2. The entity prediction task is an auxiliary task that guides the model to attend to the entity semanties during pretraining; in inference only the shared transformer representations is used upon the input word and entity tokens. Compared with the standard GPT-2 style PLMs. the architecture of KALM only differs with an enlarged tokenization vocabulary with additional entity tokens, and their entity embeddings before the input to the transformer network. The transformer architecture and its layer configurations are kept consistent.
PROBING LANGUAGE MODELS
This section describes the techniques we use probe the knowledge capability incorporated in the weights of pretrained language models, including LAMA Knowledge Probing (7), Edge Probing (6) 2D,, and the zero-shot performance on downstream tasks (3).
Knowledge Probe. Petroni et al. 7 developed the LAMA knowledge probing test which evaluates whether the language model can predict the factually correct token in "filltteeblankk cloze state- ments. For example, the model is considered to include the corresponding commonsense knowledge "isCapbleOf" if it can predict the token "fly" or "cat" for the cloze test "birds can
The LAMA cloze statements were semi-manually constructed from four knowledge sources: Google- RE (Wikipedia Relations). T-REx (Wikidata Relations), ConceptNet (Commonsense Relations), and SQuAD (Questions on Wikipedia). The knowledge graph triples were transformed to statements using manually defined templates. The SQuAD questions are manually rewritten to statements. LAMA only keeps cloze statements where the missing word is single token w.r.t the tokenizer of the model in question, to avoid the influence of decoding. Sometimes the missing word can appear in the middle of a sentence, but for auto-regressive language models such as GPT-2 and Transformer- XL (27).
LAMA only evaluates on those at the end. We refer to their paper for more details 7
Edge Probe. Besides looking at the token-level predictions, Tenney et al. (6 21 develop the edge probing" tasks to study the information (e.g., syntactic, semantic, or long range structures) in the learned hidden representations. Similar to the evaluation protocol in representation learning (28), the edge probing technique uses the PLM's hidden representations on one or multiple text spans (e.g.. has+h) fixed feature representations to linear classifiers on various linguistic/eemanii tasks A better performance indicates stronger knowledge capability of the PLM in the corresponding task.
In total eight core NLP tasks are included in edge probing 22:: part-of-speech tagging (POS), constituent labeling (Consts.), dependency parsing (Deps.), named entity typing (Entities), semantic role labeling (SRL), coreference (Coref.), semantic probe-role (SPR), and relation classification (Relations). The first four tasks are considered more local, syntactical, while the later four involve "righer-order" and/or long-range phenomena (6).
Zero-Shot Evaluation. Radford et al. (3) demonstrate that their GPT-2 language models can be viewed as "unsupervised multitask learners" that perform downstream tasks without using any task specific pretraining or fine-tuning 3. Their deep GPT-2 models, though far from fully-supervised PLM's performance, also provide meaningful zero-shot results on several downstream tasks, such as question answering, summarization, and machine translation. These zero-shot results show promise that one centralized deep model can conduct many real world tasks without requiring much human annotation task-specific finetuning. The new GPT-3 is another step in this direction, but at the cost of 175 Billion parameters (10). In the short-term, this zero-shot evaluation is a good strategy to directly evaluate the knowledge captured in language model parameters.
4

---

Under review as a conference paper at ICLR 2021
Table Size of evaluation sets, The number of relation types in LAMA are in brackets. All evaluations are zero- shot on their official testing data.
Table 2: Specifications of language models: pa- rameters in the network layers (Net.P.#), in embed- dings (E.P.#), number of layers (L.#), and hidden Dimension. GPT-2 from OpenAl is marked by (OAT), Note that embeddings are looked up in constant time; the network capacity are mostly defined by
Dataset
Items
Language Modeling
Net.P.# (17).
WikiText- 103 (tokens)
270k
Lambada
5.1k
Model
Net.P.#
E.P.#
L.#
Dim
LAMA
Base
Google-Re
4.6k
GPT-2
90M
38M
12
768
T-Rex 1-1 (2)
937
KALM
90M
458M
12
768
T-Rex N-1 (23)
20k
Large
T-Rex N-M (16)
13k
GPT-2 (OAI)
304M
51M
1024
ConceptNet (16)
11k
GPT-2
304M
5IM
24
1024
SQuAD (Statements)
305
KALM
304M
471M
24
1024
Zero-Shot QA
tr" Large
Trivia QA
11k
GPT-2 XL (OAI)
1.46B
80M
1600
Natural Questions (Short)
3.7k
GPT-2 1.5B
1.46B
80M
48
1600
WebQuestions
2.0k
GPT-2 17B
16.9B
214M
78
4256
We focus on the zero-shot QA setting in Radford et al. (3) and include all the three datasets used by Roberts et (II): Trivia QA (29), Natural Questions (30) and Web Questions (3I). In zero- shot setting, the PLMS are directly asked to generate the answer for the question; no task related information is provided for training, nor any context documents or knowledge graph of these questions
used. All the knowledge has come from the model parameters optimized the pretraining corpus alone.
The only additional information available the PLMs is the task format context during inference. Several example question-anae answer pairs were concatenated to the front of the question, to notify the model to generate answers (3). In total appendix), e.g. "Question: how many wheels does a semi truck have? Answer: 18", and prepend them to each question of the testing dataset. All our models take the input in the format of
eight manually written dummy QA pairs (listed in
Question: dummy Q'n Answer: dummy A A)n)n Question: testing 2\n Answer:
and generates an answer without any additional training besides knowledge-aware pretraining.
4 EXPERIMENTAL METHODOLOOGIES
Pretraining with Knowledge Awareness. All models in this study are pretrained on a dataset similar to OpenWebTexx[ it contains about 30 billion tokens after filtering and deduplication.
The entity tokenizer in KALM uses the entity dictionary from the CMNS Linker (20). which was harvested from the FACCI and FAKBA entity annotations (32). We keep surface forms with frequency at least 1k, and entities with at least 100 links in total about 1.4M entities. The tokenizer is forward -ereuddgreedyyyyoo by mapping the longest (at most 4-grams) surface form to its most popular entity (20). We choose this frequency-bart based dictionary lookup to favor higher entity coverage, linking speed, and recognition of entity polymorphism, while relying on the language model pretraining to handle any inaccuracy in the linking (26).
Evaluations. The language modeling evaluations include perplexity on WikiText- 103 (33) and accuracy in LAMBADA last-word prediction (34), implemented the same as Megatron-LMM 24
The LAMA knowledge probes use their official setting evaluate auto-regressive LMs such as GPT (7). Our results are directly comparable with GPT-2 but not BERT as the two use different tokenizers. In zero-shot QA evaluations, the official evaluation splits of Trivia QA (29). Natural Questions (Short Answer Setting) (30), and Web Questions (31)
Particularly we use their set-up to
5

---

Under review as a conference paper at ICLR 2021
Table 3: Results on language modeling tasks and LAMA knowledge probing tasks. The LAMA
numbers are average Precision@1
Language Modeling Lambada last word
LAMA Knowledge Probing
Wiki-103 Perplex.
G-Re Total
T-REx
C-Net Total
Squad
Model
1-1
N-1
N-M
Total
Total
Base (~1100M)
GPT-2
20.85
33.73
3.99
24.17
14.19
16.72
15.66
7.67
4.55
KALM
22.51
40.26
3.27
44.70
24.95
25.07
25.96
8.61
6.64
Large (~300M)
GPT-2 (OAI)
22.50
42.98
3.71
56.03
19.77
19.93
21.6
10.86
8.04
GPT-2
20.46
42.63
4.90
45.95
19.28
18.59
20.31
9.72
5.94
KALM
17.05
49.14
5.41
63.18
25.74
27.15
28.12
10.70
11.89
eX"tra Large (>1BB
GPT-2 1.5B (OAI)
17.37
51.23
4.30
62.31
21.53
19.61
22.77
12.28
11.54
GPT-2 1.5B
14.68
56.72
6.48
65.04
24.04
21.51
25,06
12.79
11.54
GPTT2 17B
10.21
67.98
8.77
76.82
29.60)
27.14
30.95
14.39
22.38
used. Besides exact match (EM), we also show whether the generated answers contains the ground truth answer (cover-EM). The edge probe tasks use their official implementation (21 We probe the last layer of PLMS at every 1k pretraining gradient steps from 5k to 10k, and again at convergence.
Compared Methods. We mainly compare with the autoregressive GPT-2 architecture, and pretrain our own version GPT-2 in-house, following the implementation of Megatron-LM (24). All else remaining equal, we pretrain KALM exactly as in-house GPT-2 networks, except with knowledge awareness. The statics of all the evaluation tasks are listed in Table The architecture specification of these models are listed in Table [ We measured that the KALM models had forward- -aaard-passs runtime that was a constant added to that of an equivalent GPT-2; doubling the number of layers does not change this constant look up cost.
Implementation Details. We trained our models on either 32 or 64 Nvidia v100 GPUs in DGX-2 pods for 3-6 days using the DeepSpeed ZeRO optimizer (35). In all cases, we used the same linear learning rate warm-up over 3.2k steps to maximum of 1.5e-4 with batches of 512 sequences each of length 1024, with cosine learning rate decay to a minimum of le-5 over at most 300k steps. Models trained until convergence around 200-250k steps (4-5 epochs). Dropout of 0.1 applied on the input tokens and the attention layers; the model had 8-2 regularization of 0.01. We initialized all matrices with uniform normal.
For KALM we added an additional dropout on the linked entities, replacing them with the nu entity 10% of the time. We also sampled negatives for the output entity prediction task according to the following regimen: 1 % of the time the negative was the null entity, 49% it was a random entity chosen from among all 1.4M, and the remaining 50% was "difficult" negative chosen from the 100 nearest Trans-E E neighbors of the positive entity. All our models are pretrained from scratch with entity embedding dimension de 300. More details can be found in Appendix.
5 EVALUATION RESULTS
This section present evaluations on language modeling tasks and the probe tasks.
5.1 LANGUAGE MODELING AND KNOWLEDGE PROBING RESULTS
In 3 3 show both the WikiText (perplexity) and Lambada (last word accuracy), adjacent to the LAMA knowledge probe (precision at 1). In general, the in-house GPT-2 results, with more training data, are slightly better than their Open AI implementations of equivalent architecture. We also consistently see that KALM has around 15%-20% improvement on LAMBADA accuracy. More importantly, on LAMA which directly tests factual correctness, KALM consistently improves over GPT-2 by margins of 40-80% even approaching performance of a GPT-2 model with 5-fold more transformer parameters.

---

Under review as a conference paper at ICLR 2021
Table Zero-shot question answering performance of different models for three different question answering benchmarks. EM is exact match percent, and cover-En Eeer-EMM is the percent counting whether the correct answer a substring the generated answer. All generated answers were generated by greedy decoding of at most 20 tokens.
Trivia QA
Natural Questions cover-EM
Web Questions
Model
EM
cover-EM
EM
EM
cover-EM
Fully Supervised
Base (11)
n.a.
27.0
n.a.
29.1
n.a.
T5 Large (1
35.9
n.a.
29.8
n.a.
32.2
n.a.
T5 11B (II)
50.1
n.a.
34.5
n.a.
37.4
n.a.
Zero-Shot
GPT-2 Base
3.44
4.77
0.81
2.08
2.92
KAIMM Base
5,87
7.16
1.75
2.13
3.53
4.79
GPT-2 Large
7.32
9.05
3,48
4.26
4.79
6.20
KALM Large
11.68
13.34
4.34
5.07
6.56
9.48
GPT-2 1.5B
17.78
21.59
6.08
7.95
6,20
12.65
GPT-2 17B
42.32
47.56
14.34
17.30
12.15
21.68
The improvements from KALM are most noticeable on T-Rex, which tests the model's skill on hundreds of knowledge base relations like "X is owned by The gain is even more sigmificant on more complex relations (N-1 and N-M), where GPT-2 is confused the most. KALM Large even matches the accuracy of GPT-2 17B on the most difficult N-M relations, using only 2% transformer parameters. Simply by being aware the existence of entities in pretraining, KALM enjoys significantly better effectiveness and parameter efficiency in learning how to capture factual knowledge standard transformer layers.
5.2 ZERO-SHOT TASK PERFORMANCES
In Table we compare the zero-shot question answering performance of GPT-2 and KALM, as well T5 fine-tuned in supervised fashion. Remarkably, even though the largest GPT-2 model was only trained on unsupervised text prediction tasks, it still performs nearly as well as the fully-supervised T5 11B model on Trivia QA, confirming that indeed, larger language models do capture some knowledge in an unsupervised way. Yet, the captured knowledge seems more on the "trivial side" as GPT-2 performs much worse on NQ and WQ, which are more similar to what real users ask in search engines. This observation still holds even in the selly-announced GPT-3 with 175 Billion parameters (IO). which has 30 EM on NQ, versus 44.5 in the RAG model which uses "only" hundreds millions parameters (12).
Though far from perfect, KALM significantly improves the zero-shot accuracy on three QA datasets. It more efficiently packs practical knowledge into its parameters that can answer human questions: the 12L KALM Base is very close to the zero-shot accuracy of the 24L GPT-2 Large. This indicates that one day pretrained language models may be able to capture complex semantics in real world tasks without needing orders magnitude more parameters.
KALM has greater capabilities without sacrificing speed. Its extra entity embeddings do not increase run time much they only introduce constant time matrix lookup operations after a linear time entity tokenization this is of our motivations of first adding the knowledge "awareness" standard transformers before proceeding to specialized knowledge layers or external "memory" modules.
5.3 PROBING KNOWLEDGE AWARE PRETRAINING
In Figure I we show the edge probing results during the language model pretraining in terms of F1 on the eight NLP tasks (2l). Theses tasks are ordered following the observed "NLP Pipeline" order in BERT layers (6). Tasks thought to require more sophistication long range semantics are shown further left. Intuitively, the "casier" tasks that focus on local syntax, e.g., POS, are learned earlier in the pretraining process, while the "harder" ones, especially those require long range dependencies, such as relation extraction and coreference, are captured later in the pretraining. Transformers seems to learn what words before learning how they interact to form higher levels of meaning.
7

---

Under review as a conference paper at ICLR 2021
F1
Relations
F1
SPR
F1
Coret.
F1
SRL
78.0
86.0
74.0
93.0
84.0
70.0
80.0
92.0
10k Entities
200k
5k
10k
200k
10k Consts.
200k
5k
10k
200k
F1
F1
Deps.
F1
POS
92.0
78.0
94.0
76.0
GPT-2 Base
92.0
91.0
74.0
94.0
KALM
5k
10k
200k
5k
10k
200k
5k
10k
200k
5k
10k
200k
Figure Edge probing results on eight NLP tasks. The base versions of GPT-2 and KALM are probed, along different training steps (x-axies), using micro- F1 scores (y-axes).
The hidden representations learned by KALM are more informative in all eight categories at con- vergence. KALM starts slower in the beginning of pretraining (at 5k steps), as it needs to learn the additional entity embeddings from scratch; then the extra knowledge awareness quickly help KALM surpass GPT-2 after 5k steps. The benefits KALM are more apparent on knowledge-related tasks, such as Entity typing, Relations, and Semantic Proto-Roles.
6 RELATED WORK
Are deep transformers, just memorizing the corpus, e.g., using as many as one parameter per two training tokens (I0), or do they abstract concepts and knowledge from the data B 3 36)? Many observed that the pretrained transformer weights do include certain forms of logic and semantic knowledge 6 in zero-shot fashion (3 10): T5 with task-specific fine-tuning can accurately answer some questions without using the background document or knowledge graph (5) D..
37 GPT-2 and GPT-3 can perform downstream tasks they never saw training
Yet it is also observed these enormous pretrained transformers often "dream" and produce language that might look right at first glance, but actually include biased (38), repetitive (5), useless (39), even worse, misleading information (14) To address this, previous research mainly grounds the language model to some forms explicit semantic structures, for example, additional entity-oriented transformer layers (16) (8). or explicitly stored knowledge graph triples (19 40; 4I). Grounding provides a "safety net" for language models to look up a semantic facts in their external storage, instead of relying their transformer parameters incorporate correctness.
CONCLUSION AND FUTURE DIRECTIONS
KALM improves the knowledge learning capability of language models by signaling the language model the existence of entities during pretraining. This simple knowledge-awareness significantly improves the parameter efficiency of transformers pretraining. In our experiments on various tasks, KALM often matches the effectiveness accuracy of the Sinoweedg--nnwrr GPT-2 that uses twice as many transformer layers KALM.
The fact that, adding knowledge awareness leads to improvements almost equal to those from doubling the transformer layers, indicates language models can be pretrained with better parameter efficiency. Brute-force blowing-up the model parameter numbers exponentially is not the only way to achieve a better language model better intelligence is only created by network sizes but can also by proper optimization, better understanding of pretraining, model architecture design, and the inductive biases within
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4171-4188 2019.
[2] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907 11692, 2019.
[3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
[4] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, pages 5754 5764, 2019.
[5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:I910. 10683, 2019.
[6] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint 1et..0905500000 2019.
[7] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLPP pages 2463 2473, 2019.
[8] Peter Clark. Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.00 05867, 2020.
[9] Peter Clark, Oren Etzioni, Tushar Khot. Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon. Sumithra Bhakthavatsalam. et al. From' on the ny regents science exams: An overview of the aristo project. arXiv preprint 1rXivv1909.01955 2019.
[10] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
[11] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint ar Xrxv:2002.009910 2020.
[12] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-imtensive intensive nlp tasks, arXiv preprint arXiv:2005.1 2020.
[13] Margaret Li, Stephen Roller, Ili Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho. and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv: 191 1.03860, 2019.
[14] Robert Logan, Nelson F. Liu, Matthew Peters, Matt Gardner, and Sameer Singh. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of ACL, pages 5962-5971, 2019.
[15] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of neural text degencration. arXiv preprint arXiv:1904.0973: 2019.
[16] Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (FMNLP-IJCNIPP. pages 43-54, 2019.

---

Under review as a conference paper at ICLR 2021
[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeftrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.088 000301 2020.
[18] Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Funsol Choi, and Tom Kwiatkowski. Entities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:200400220: 2020.
[19] Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, and Graham Neubig. Latent relation language models. In Proceedings of AAAI 2020, 2020.
[20] Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsherg. Entity Linking in Queries: Efficiency vf.Ffffectiveness In European Conference on Information Retrieval, pages 40-53, 2017.
[21] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure contextualized word representations. arXiv preprint arXiv:1905.06316 2019.
[22] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Fcb): 137-1155, 2003.
[23] Frederick Jelinek. Interpolated estimation markov source parameters from sparse data, 1980.
[24] Mohammad Shoeybi, Mostofa Patwary. Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint 4rXi::1909.0005: 2019.
[25] Corby Rosset.
Turing-nle
A
17-billoon language model by
microsoft.
https: /obs::/wwwee www.oof....oooooomomoo wwtp:///ssccooeeeeeeeeeeeee reweeaachceeeeee/ebbbbeb
turi turing-nlg-an
oag.:papaa...at..eaaaaaaaaaooeaa cow.oooooooooooooooooooooooooo// 2020.
[26] Chenyan Xiong, Jamie Callan, and Tie-Yas Yan Liu. Word-entity duet representations for document ranking. In Pruceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 763-772, 2017.
[27] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Ie,, and Ruslan Salakhutdinev. Transformer- XL: Attentive Language Models beyond a Fixed-L.ength Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
2019.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint 1rXiv:1911.05722, 2019, 2019.
[29] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: arXiv:/705.000 770500351, 2017.
[30] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[31] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1533-1544, 2013.
[32] Exgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. Facel: Freebase annotation of clueweb corpora, version 1. Note: http://emurprojeet. org/cluewedV/FFCCC/CCcc by. 5, 2013.
[33] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609::6609007888 2016.
[34] Luong Hoang, Sam Wiseman, and Alexander Rush. Entity tracking improves cloze-style reading compre- hension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
[35] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2019.
[36] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Schastian Riedel. Language models knowledge hases? arXiv preprint arXiv: 190.999990000000... 2019.
10

---

Under review as a conference paper at ICLR 2021
[37] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, August 2019.
[38] Moin Nadeem, Anna Bethke, Siva Reddy. Stereoset: Measuring stereotypical bias pretrained language models. arXiv preprint arXiv:2004.094533 2020.
[39] Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. Leading conversational search by suggesting useful questions. In Proceedings of The Web Conference 2020, pages 1 1160 11:00-177 1170, 2020.
[40] Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. Grounded conversation generation as guided traverses in commonsense knowledge graphs. In Proceedings of ACL, 2020.
[41] Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language model. arXiv preprint arXiv 00318, 2016.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 MORE IMPLEMENTATION DETAILS
CMNS Entity Tokenization We performed entity linking in linear time with a forward-greede -rreddy approach. This means that the linker looks at all consecutive groups of to four space-delimited words and repeatedly querying variants of that window in the dictionary (variants such lowercase, uppercase, capital case, and without punctuation). We first query down to word, before incrementing the window position and resetting it to 4. Whenever there is hit in the dictionary, we link all the word tokens in the candidate mention span to the entity with the highest frequency, and then move the window up past the words that were hit. Note that the entity linking is done in the "word space" meaning it looks at space-delimited words, but take great "sub-word" space comprise matched entity string are labeled as well. BPE subword vocabulary, but this technique will work for any standard word or subword vocabulary.
variants of the 4-word window before moving down words, and so on
ensure that all the tokens in the our experiments, we used a
Zero-shot QA Context Template. We normalize both gold and predicted answers by lowercasing them. We filter Natural Questions to those with short answers as in the T5 paper (II): we ignore questions whose answers are longer than 5 tokens. We use the unfiltered version of the Trivia QA dataset. For both Natural Questions and Trivia QA datasets, we report the performance on the validation sets hecause the test sets with lahels are not publicly available.
For each question in any QA dataset, we prefix the test question with a template of eight example questions. The exact template use is:
question question: where is the lincoln Memorial located? \n answer: Washi nggtn, DC, USA \n \n
question: How long is the Nile river \o answer: 6250 miles |a |n
question: who was elected president of the united states in 1928? \n
answer: Herbert Hoover \n \n
question: what year did the September 11th attack occur? \n answer: 2001 \n (a
question: what elements of the periodic table are liquid at room temperature? |n answer: bromine, mercury \n \n
question: which pigment helps plant absorb energy frem light? In answer: chlorophyll (n \n
question: who was the commander of the japanese navy for the majority of World war IT? \n answer: Tsoroku Yamamoto \n \n
question: name of a famous highway without speed limits? Aa answer: Autobahn \a \n
question: how many wheels does semi truck have? \n answer: 18 In In
KALM with Input entities only. We found that KALM model that only had knowledge awareness at the imputs (no entity contrasting loss at the output) had quality that was between that knowledge-unawaal GPT-2 and our full KALM. For 12 layer input-only model, the wikitext perplexity was 26.45 and the lambada accuracy was 36.03. On the zero-shot question answering tasks, the triviaQA exact match was 4,5%, the natural questions FM was 1.5%, and lastly the webquestions EM was 2.7%.
12


Reviews:
[{"review_id": "a8jGXC5BNc4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces to use additional knowledge to pretrain language models, which adds entity information to input and output. And the method improves the performance compared to the corresponding models without the knowledge. This is an interesting piece of work and worth publishing. However, I have some comments and questions about the article. \n\nSince the main point of the paper is adding entity information to the models, the paper would be better to include more explanations about the entity information. For example, with the dictionary look-up, what is e_i exactly? Why did the authors use such knowledge rather than other knowledge like POS or so? Why do we need negative sampling? How many classes are there for entity? \n\nThe models enjoy more information provided by entity. Then what about more data to provide more information? How much can we improve the performance by adding entity information rather than additional data samples? \n\nThe authors said that the models were trained with entity information from scratch. Can we use pretrained models, since we can simply add entity information to the input and output layers? It seems like that the additional information can fine-tune the pretrained models.\n\nWhat is the role of the dummy QAs? And how did the authors come up with the 8 dummy QAs?  \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_jkKFgW2pa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(This review is a collaboration between a senior and a junior reviewer. The junior wrote most of the review, and the senior modified the review. Both reviewers read the paper in detail.)\n\nSummary:\n\nThis paper presents an alternative pretraining technique for language models in order to make them more \"knowledge aware\". In addition to pre-training a model with the traditional language modeling objective, this work proposes an entity prediction task as pre-training. Furthermore, this work also incorporates entity tokens at the input level of the model by summing the word embedding and its corresponding entity embedding.\n\nExperimental results show that such pretraining method yields models with greater factual knowledge according to the LAMA knowledge probing task compared to vanilla GPT2 models. In addition, it is also reported that such models perform better than a GPT2 model of the same size on TriviaQA, Natural Questions and Web Questions, in a zero-shot setting; and achieves competitive results when compared to larger models.\n\nThis paper is well written and easy to follow. This work is timely because it shows that increasing the model size and pre-training data size is not the only way to achieve strong performance on language related tasks (which is the current trend). Inductive biases like knowledge about entities can improve the performance of models to the point of achieving competitive results with bigger models.\n\nSome limitations of this work include the lack of experiments on more diverse architectures like encoder-decoders to make sure that the proposed technique not only works on autoregressive GPT-type models. Moreover, only one approach for entity integration is explored.\n\nOther than that, I have the following questions:\n\nInteresting choice of the margin loss for entity prediction. More motivation should be added into why this loss instead of the cross-entropy loss. Optionally, an experiment comparing the performance of the model when trained with the cross entropy -vs- the margin loss would be interesting to have.\n\nAt inference time, you are forced to put entity labels on the input text as a preprocessing step. How much does this impact the generalization capacity of the model? How do you manage cases when an unseen entity is seen as input?\n\nAre the experiments on QA really zero-shot? The examples presented in Appendix are not dummy as they reflect true questions and answers?\n\nMinor:\n\nHave you tried your pre-training technique on an encoder-decoder architecture like T5? It would be interesting to see that this technique not only works with auto-regressive models.\n\nHave you tried to fine-tune your resulting pre-trained models on various tasks? It would be interesting to see if your pretraining method also helps finetuning more efficiently on some tasks. In addition, I would be curious to see if, while fine-tuning, the entity prediction accuracy and the LAMA probing performance of the model drops or remains more or less the same.\n\n[1] Sun, Yu, et al. \"Ernie: Enhanced representation through knowledge integration.\" arXiv preprint arXiv:1904.09223 (2019). [2] Ji, Yangfeng, et al. \"Dynamic Entity Representations in Neural Language Models.\" Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.\n\n\nAfter author response:\n\nThanks for the response! After reading other reviews, I feel the novelty of this work is somewhat limited and it would be useful to highlight the differences with respect to previous work. Also a discussion on when your method is preferred over existing proposals. I am also decreasing my confidence score after reading other reviews.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RbnRkTxq2Rh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes KALM - a knowledge-aware language model that incorporates entity information. Specifically, the method involves using a dictionary lookup to match n-grams to entities from a database. Then, each token is embedded into two vectors — one using the surface form, and one using the entity ID it is matched to (if there is a match). These embeddings are then added and passed into a Transformer model to predict the next word given the context. Experiments compare the model to GPT-2 and T5 (two state-of-the art Transformer-based LMs) and demonstrate the benefit of adding entity information. \n\nStrengths: The idea of using entity information is certainly interesting and new. Most details of the paper are clear from the writing.\n\nWeaknesses: The paper leaves open several questions that should really be explored and discussed. Given that the modeling contribution is simply adding an extra embedding layer, it would be good to have detailed analyses that provide more insights into the contribution itself. (see points 5,6,7 below)\n\nDetailed comments/questions:\n1. The details of the dictionary lookup are not entirely clear. Do you lowercase all tokens? Do you handle small misspellings in the tokens? Do you perform fuzzy matches?\n2. Thanks for providing several implementation details. However, the values of hyperparameter $\\alpha$ are not discussed. Does this require a lot of tweaking or is the method stable to this choice?\n3. (minor) The ‘zero-shot’ experiments really seem to be ‘few-shot’ tests since you provide the model with 8 or so examples. \n4. The metrics used for the LAMA knowledge probing should be explained in the paper once (at least expand the abbreviations so readers can understand). \n5. Table 3: The parameter sizes indicated are misleading. Large KALM has close to 700M params and not 300M as indicated. Similarly base KALM has ~550M params not 100M. \n6. Related to the above point, it seems like KALM base has more parameters than GPT-2 base (and even GPT-2 Large) and yet this aspect is not accounted for/investigated in the paper. In fact, the paper says “(KALM) more efficiently packs practical knowledge into its parameters … 12L KALM is very close … to 24L GPT-2 Large”. This seems to be very misleading information. Since Transformers are essentially a form of graph networks, I’m not sure entity embeddings can be treated separately from the model parameters since one can view those as edges part of the graph. \nConcrete suggestion: Can you compare KALM vs GPT-2 (in all tables and Fig 1) while controlling for the #params? To me, the only closest comparison seems like KALM base vs GPT-2 Large where GPT-2 Large seems to be the better model?\n7. I appreciate the authors efforts to test on a variety of benchmarks and I understand that training these huge models is not easy, but to me, the essence of the paper (i.e. addition of entity information) seems under-explored. For instance, could you perform ablations studies that only uses the entity embeddings in the input to see how much knowledge it provides? Another ablation can be on the matching done in the dictionary lookup (in terms of frequency cutoffs, n=1,2,3,4 in the n-gram matching, etc. ). One could also analyze the prediction accuracy of the model to see which types of entities (e.g. persons, organizations, etc.) work better vs those that don’t help as much. \n\n-------\nUpdate: Thanks to the authors for their response, which helped clarify several of my minor questions and I believe those can be revised with a writing pass. However, I still think this paper has two significant deficiencies:\n\n1. The parameter size comparison still seems flawed to me. The authors say that one can discount the entity embeddings, but can we really? Aren’t they part of the model’s ‘representation’ even if the inference does not use all of them in a single forward pass? Several neural net architectures exist (including large-scale LMs) that do not use all inference paths but still count them in the total params. The BERT vs GPT-2 example provided in the rebuttal is only a difference of 26k tokens (still significant!) but here we're talking about a few hundred million parameters!     \nAt the very least, I think the true sizes of the models must be acknowledged and one can add the point on entity embeddings vs 'brain' parameters as a caveat, but it seems scientifically inaccurate to me to claim otherwise. To be clear, I don't think the size issue detracts much from the main contribution of adding entity embeddings here (i.e. this work may still be of interest even if the size of model is larger), but the current version of the paper has several claims about size savings that seem incorrect.\n\n2. The analysis of the proposed method (where it helps, where it fails, which hyperparams matter) is still lacking. The authors did mention one ablation in the supplementary that I missed, but I don’t think that is sufficient for a reader to understand how to build on this method in this future without re-running all the experiments, doing an extensive hyperparam search, etc.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IrN81COH3lw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\n\nThis paper presents a knowledge-aware language model pretraining method without changing model architecture. Specifically, they add entity prediction task along with language modeling task to make the model aware of knowledge. Experiments show improved results on the LAMA knowledge probing task compared to GPT-2 models. They also show comparable results on zero-shot question answering task, even with only 2% transformer parameters compared to GPT-2 17B.\n\n- Strengths\n\n1. The authors propose a simple (i.e. without changing model architecture) method to give knowledge-aware signal during pretraining.\n\n2. Proposed method consistently outperforms GPT-2 on LAMA and zero-shot QA.\n\n3. Interesting to see that entities from fuzzy frequency-based matching can make LMs significantly better.\n\n- Weaknesses & Questions\n\n1. Need extra parameters to save entity embedding (~471M). However, as the authors mentioned, this can be viewed as an external memory module and embedding lookup doesn't need much computation.\n\n2. I would like to see how much KALM can transfer to downstream tasks. The authors already show zero-shot QA results, but it would be better to see fine-tuning results on knowledge-intensive tasks. For instance, will KALM still outperform GPT-2 after finetuned on KILT tasks (https://arxiv.org/abs/2009.02252)?\n(I know KILT released right before the due date of ICLR, but tasks in KILT are all already published)\n\n3. Even though KALM outperforms GPT-2, it is still unclear for me the advantages of KALM compared to Entities as Experts (or EaE, https://arxiv.org/abs/2004.07202). For example, EaE outperforms KALM on the LAMA probing tasks, and EaE also outperforms T5-11B when finetuned.\n\n4. The authors mentioned fuzzy frequency-based entity matching is better than precise entity linking (e.g. EaE) on page 3, but EaE outperforms KALM on LAMA probing tasks. Based on the experimental results, precise entity linking seems more effective to me. It would be helpful if the authors can provide a more thorough comparison with EaE.\n\n- Typo?\n\nPage 3, \"United States\" at $w_{i:i+2}$ --> \"United Staes\" at $w_{i:i+1}$ ?\n\n=======================================================================\n\nPost AR:\n\nThank you for preparing detailed responses, which helped me to clarify several questions. However, one of my major questions is still unclear.\n\nAlthough 4 out of the 11 KILT tasks are already included in the main paper, most of them are LAMA knowledge probing tasks or zero-shot QA tasks. It is still unclear how much and how robustly KALM can transfer to other downstream tasks with fine-tuning (e.g. Wizard-of-Wikipedia, FEVER, QA with fine-tuning). For instance, CorefBERT paper (which uses a similar idea but as you mentioned it use bidirecitonal attention) shows its transferability on QUOREF, six extractive QA benchmarks, DocRED, FEVER, five coreference resolution benchmarks and GLUE, which can convince me to choose CorefBERT over BERT.\n\nExperiments on the paper are promising but not diverse enough to make me choose KALM over GPT2. Thus, I would like to stick to my original score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to improve the LMs ability on modelling entities by signalling the existence of entities and also allowing the model to represent entities also as units. The embeddings of the surface form and then entity unit are then added and passed through a layer to predict the next word.
The paper evaluates on QA and conducts probing tasks and shows that such an entity modelling results in better performance.

All reviewers have found the idea conceptually simple and novel. At the same time a number of concerns are raised, with the most important being the lack of understanding around which and how hyper-parameters matter for this model and, most importantly, the confounder introduced to the model by the much larger size of parameters introduced by the embedding layers. While the authors comment that not all the parameters are used all the time, the size of the embeddings still count at the total size of parameters a model has. Thus, without properly controlling for this (e.g., have an another model where the extra embedding params are given to another part of the model), it is difficult to determine whether adding more parameters was the solution or adding more parameters for modelling the entities. 
Decision: Reject