Language models as knowledge bases?
Huaisheng Zhu
04/05/2023

Petroni, Fabio, et al. "Language Models as Knowledge Bases?." EMNLP(2019)Knowledge Bases• A knowledge base allows for rapid search,
retrieval, and reuse• Stores information as answers to questions
or solutions to problems• Can be fed into a language model2

Examples of Knowledge Bases

Visualization as a graph3

Knowledge Bases• Concepts like classes and individuals aremodeled as nodes• Relations as edges of graphs• Classes – concepts like documents,
events, or subjects• Individuals – instances of a class or anobject• Relations – capture relationships between
classes and individuals• is-type-of, is-instance-of, and hasattribute4How knowledge bases are used in NLP models:• Entity extraction – replace or augment entity occurrences in text5How knowledge bases are used in NLP models:• Coreference resolution• Coreference resolution is the task of finding all expressions that referto the same entity in a text.

• Entity Linking

Querying knowledge bases from these methods needsupervised data. Moreover, errors can easily propagate andaccumulate throughout the pipeline.

6

Proposed solution for the knowledge bases• Language Models• Ask the model to fill in masked tokens• “Alex was born in [MASK]”• Pre-trained high-capacity models such as ELMo and BERT store vast
amounts of linguistic knowledge useful for downstream tasks

The Pros:- Requires no schema engineering- No need for human annotations- Supports a more diverse/open set of inquiries

7

Research Questions• How much relational knowledge do they store?
• How does this differ for different types of knowledge such as facts aboutentities, common sense, and general question answering?
• How does their performance without fine-tuning compare to symbolic
knowledge bases automatically extracted from text?

8

LAMA (Language Model Analysis)• Consisting of a set of knowledge sources, each comprised of a set offacts (subject, relation, object)• Success depends on predicting masked objects such as “Dante was born
in ___”• Tested for a variety of types of knowledge: relations between entities
stored in Wikidata, common sense relations between concepts from
ConceptNet, and so on.

9LAMA (Language Model Analysis)• Steps for LAMA
• Query each model for a missing token• Evaluate each model based on how highly they rank the ground truth
token against every word in a fixed candidate vocabulary10Knowledge Sources and datasets used
• Google-RE – contains ~60K facts manually extracted from Wikipedia• Only utilized 3 relations: “place of birth”, “date of birth” and “place ofdeath”• manually defined a template for each considered relation, e.g.,“[Adam] was born in [Illinois]” for “place of birth”• T-Rex – is a subset of Wikidata triples• consider 41 Wikidata relations and subsample at most 1000 facts
per relation.
• Facts were automatically aligned to Wikipedia (can be noisy)• e.g. ‘Adolphe Adam died in _.’ for ‘Paris’11Knowledge Sources and datasets used
• SQuAD• Question-answering dataset• a subset of 305 context-insensitive questions with single token
answers• Example: rewriting “Who developed the theory of relativity?” as “Thetheory of relativity was developed by __”.
• ConceptNet• Multilingual knowledge base, initially built on top of Open Mind
Common Sense sentences• English parts that have single-token objects covering 16 relations• Example: Time is __”.
12

Language Models used
• Fairseq-fconv• Multiple layers of gated convolutions• Trained on the WikiText-103 corpus• Transformer-XL• Large-scale LM based on the Transformer• Takes into account a longer history• Used relative instead of absolute positional encoding• Trained on the WikiText-103 corpus

13

Language Models used
• ELMO:• ELMo: Forward and backward LSTM, resulting in ℎ𝑖 and ℎ𝑖• Trained on the Google Billion Word Dataset• ELMo 5.5B:• Trained on English Wikipedia and monolingual news crawl from WMT
2008-2012

Trained through averaged forward and backward probabilities from the corresponding softmax layers

14

Language Models used
• BERT:• Transformer architecture• Trained on the BookCorpus and English Wikipedia• language modelling (15% of tokens were masked and BERT was trained topredict them from context) and next sentence prediction (if a chosen nextsentence was probable or not given the first sentence)• BERT-base (12 encoders with 12 bidirectional self-attention heads)• BERT-large (24 encoders with 16 bidirectional self-attention heads)
Masked the token at position t, fed output to vector corresponding to masked token (ht) into softmax layer

15Baselines• Freq:• subject and relation pair, this baseline ranks words based on how
frequently they appear as objects for the given relation in the test data
• Relation Extraction (RE):• extracts relation triples from a given sentence using an LSTM-based
encoder and an attention mechanism• constructs a knowledge graph of triples• At test time, they queried this graph by finding the subject entity and thenrank all objects in in the correct relation based on the confidence scores bythe RE

16

Baselines• DrQA• a popular system for open-domain question answering• Two-step pipeline:• First, a TF/IDF information retrieval step is used to find relevant articles
from a large store of documents (e.g. Wikipedia)• Secondly, on the retrieved top k articles, a neural reading
comprehension model then extracts answers

17Metrics• Rank-based metrics• For multiple valid objects for Subject-Relation pair, removed all other valid
objects from the candidates when ranking at test time other than the ones theywere testing• Mean precision at k (P@k)• For a given fact, this value is 1 if the object is ranked among the top kresults, 0 otherwise

18

Results

19

Results

REn – naïve entity linking, i.e. exact string matching

REo – uses an oracle for entity-linking, i.e. any given (s, r, o) in sentence x, if any other (s’, r, o’)has been extracted in the same sentence, s will be linked to s’, and o to o’
20Results on Google-RE• From earlier example, “Adam was born in [MASK]”• BERT-Large (last column) outperformed all models by a substantial margin21

Results on Google-RE••••

More facts and relations than Google-RE
BERT-Large performed better on 1-to-1 relations, i.e. “capital-of”
N-1: Multiple valid subjects-relations-> 1 correct object
N-M relations: multiple objects for a subject-relation pair. i.e. “Brian owns [car,laptop, iPhone,etc]

22

Results on ConceptNet and SQuAD

ConceptNet• BERT-Large achieved best performance for ConceptNet• Able to retrieve commonsense knowledge at a similar level to factual knowledge
SQuAD• Open domain cloze-style (fill in the blanks)• Huge performance gap between BERT-Large and supervised DrQA
• Note: BERT and ELMo were both unsupervised and not fine-tuned for this task
• In terms of P@10 (Top-10 best answers), gap is remarkably small (57.1 for Bl and 63.5 forDrQA)

23

Results on ConceptNet and SQuAD

ConceptNet• BERT-Large achieved best performance for ConceptNet• Able to retrieve commonsense knowledge at a similar level to factual knowledgeSQuAD• Open domain cloze-style (fill in the blanks)• Huge performance gap between BERT-Large and supervised DrQA
• Note: BERT and ELMo were both unsupervised and not fine-tuned for this task
• In terms of P@10 (Top-10 best answers), gap is remarkably small (57.1 for Bl and 63.5 forDrQA)

24

Other Results

25Conclusion• For an unsupervised, not fine-tuned, pre-trained model BERT-Large, it is possible to recall
knowledge better than its competitors.
• The factual knowledge can be recovered surprisingly well from pretrained language models,
however, for some relations (particularly N-to-M relations) performance is very poor.
• Language models trained on ever-growing corpora might become a viable alternative to
traditional knowledge bases extracted from text in the future.

26

Limitations• Only used Single-Token objects as prediction targets• Chose only query objects in triples• Still spent time manually defining templates for each relation

27

Thank you!

28