How Can We Know
What and When Language Models Know?
Graham Neubig
Based on research w/
Zhengbao Jiang, Frank F. Xu, Haibo Ding, and Jun Araki+ Bonus?!: Interpretable Evaluation + ExplainaBoard

Language Modeling• Predict the likelihood of a sentence P(X)
Barack Obama served as the 44th President of the United States.
P(X) is high44th the of the President United States served Barack Obama as.
P(X) is low

syntaxBarack Obama barked as the 44th President of the kennel.
P(X) is low

semantics

Barack Obama served as the 42nd President of the United States.
P(X) is low

facts

Barack Obama reached a height of 50 feet tall.
P(X) is low

common sense2Prompting LMs for Knowledge
Factual Question Answering
Tokyo is the capital of [MASK].

Petroni et al. (2019)

Text Classification
I loved this movie. The movie was[MASK].
{bad, OK, good}Yin et al. (2019)

In Dialogue Context
Human: Hi!
Meena: Hi there! How's it going?
Human: Pretty well.
Meena: I'm glad to hear it. :D
Human: What is your favorite book?
Meena: Oh, that's a toughie. Right now I'd have to say it's A
Dance With Dragons, but Harry Potter is a close second.
Human: Oh I love Harry Potter
Meena: Me too! I've read the series at least five times over.
Human: Haha, same probably. Which one do you like themost
Meena: Probably Chamber of Secrets.
…
Example from Meena chatbot (Adiwardana et al. 2020)

https://github.com/google-research/google-research/blob/master/meena/meena.txt
3Prompting Difficulties• LMs were never trained to solve the exact tasks that we’re asking
them to solve• Because of this, they are• Very sensitive to the wording that we use to prompt them• Will return an answer even when they have no idea

• In this talk we ask:
• How can we know what language models know through better prompting?
• How can we know when language models know through better calibration?

4How Can We Know
What Language Models Know?
Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig
TACL 2020
Paper: https://arxiv.org/pdf/1911.12543.pdf
Code: https://github.com/jzbjyb/LPAQA

Sub-optimal Prompts (in Factual Probing)DirectX is developed by [MASK]. [MASK] released the DirectX.

DirectX is created by [MASK].

Inappropriate prompts might fail to retrieve facts that the LM does know
How can we most effectively probe language models?
6

Motivations• Any given prompt only provides a lower bound estimate.
• Can we get a tighter estimate by:• automatically discovering better prompts?
• combining a diverse set of prompts?

Answer: Yes! Careful prompt design leads toup to 8.5% increase in fact retrieval accuracy.

7

Prompt Generation• Mining-based• Middle-word
Barack Obama was born in Hawaii. à [X] was born in [Y].
• Dependency-based
The capital of France is Paris. à capital of [X] is [Y].

• Paraphrasing-based
Back translation with beam search[X] shares a border with [Y].

en-demodel

de-enmodel[X] has a common border with [Y].
[X] adjoins [Y].
9……

Prompt Ensembling𝑠 Y$

X , owned_by = .

!"#𝑤! ∗ log 𝑃%& ([Y]| X , 𝑡! )

.485
.151.
.151.

[X] is owned by [Y].
[X] was acquired by [Y].
[X] division of [Y].
10

Experimental settings• Datasets• LAMA46 relations from Wikidata, each associated with 1000 subject-object (X-Y) pairs.

• LAMA-UHN• A difficult subset of facts from LAMA.

• Google-RE• 3 relations.
Relations

Subject-object pairs[X] was born in [Y] .

(Allan Peiper, Alexandra), (Paul Mounsey, Scotland), …

[X] plays in [Y] position .

(Johan Santana, pitcher), (Koke, midfielder), …

[X] is developed by [Y] .

(MessagePad, Apple), (Adobe Illustrator Artwork, Adobe), …
11

Experimental settings• Dataset: LAMA, a dataset of relations from a knowledge base
• Methods• Prompts• Man: manually created prompts.
• Mine: mining-based prompts from Wikipedia articles.
• Para: paraphrasing-based prompts from WMT’19 English-German models.

• Ensemble:• Top1: the best-performing prompt for each relation selected on training set.
• Ensemble: combine 40 prompts by weights learned on training set.
• Oracle: judged as correct if any one of the prompts yield correct predictions.

• Metrics• Accuracy: accuracy average across relations.
12Results• Top1 > Baseline (Man): automatic prompts provide better accuracy.
• Ensemble > Top1: diverse prompts can indeed query the LM in different ways.
• Oracle > Ensemble: space for further improvement with better ensemble methods.

Accuracy of BERT-base using various prompts
0.6
0.4

0.526

0.507
0.314

0.389

0.316

0.396

0.481
0.327 0.362

0.479
0.341 0.373

Baseline: 0.311

0.2
0Mine

Mine+ManTop1

Mine+ParaEnsemble

Oracle

Man+Para13

Results on LAMA-UHN and Google-RE• Ensemble > Baseline (main): diverse prompts can query the LM more effectively.

Accuracy of BERT-base on LAMA-UHN
0.310
0.2900.2700.250
0.230
0.210

0.287

0.270

Baseline: 0.213
Mine+Man Mine+Para Man+Para
Ensemble

0.104

0.105

0.294
0.268

Mine

Accuracy of BERT-base on Google-RE
0.100

0.100

0.100
0.096

0.095

Baseline: 0.980

0.090
Mine

Mine+Man Mine+Para Man+Para
Ensemble14

Case study
Manual prompts

Generated prompts[X] is affiliated with the [Y] religion.

[X] who converted to [Y].

+60%

[X] is represented by music label [Y].

[X] recorded for [Y].

+17%

Simple edits[X] plays in→at [Y] position+23%[X] was created→made in [Y]

+11%
15

Results of different LMs• KnowBERT < BERT < ERNIE
Accuracy of different LMs
0.5
0.4
0.3

0.3110.321
0.262

0.423
0.389
0.341

0.438
0.396
0.346

Mine

Mine+Man

0.401
0.362
0.319

0.411
0.373
0.321

Mine+Para

Man+Para

0.2
0.1
0Man

KnowBERT

BERT

ERNIE

16

Cross-model consistency
Ensemble weights are consistent across models• Same model: train ensemble weights on BERT, test on BERT
• Cross model: train ensemble weights on ERNIE, test on BERT

0.500
0.400

0.389 0.380

0.396 0.395

MineMine+Man

0.362 0.342

0.373 0.352

Mine+Para

Man+Para

0.300
0.200
0.100
0.000
Same model

Cross model

17

Follow-up: AutoPrompt (Shin et al. 2020)• Automatically optimize arbitrary prompts based on existing words

18

Follow-up: Prefix Tuning (Li and Liang 2021)• Optimize theembeddings of aprompt, instead of thewords.

19

How Can We Know
When LMs Know?
On the Calibration of Language
Models for Question Answering
Zhengbao Jiang, Jun Araki, Haibo Ding , Graham Neubig
TACL 2021
Paper: https://arxiv.org/abs/2012.00955
20

LMs are not omnipotent• Fail to provide appropriate answers in many cases

https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html

21

LMs are not omnipotent• Fail to provide appropriate answers in many cases• Q: I feel very bad, should I kill myself?
• GPT-3: I think you should.
• (https://www.theregister.com/2020/10/28/gpt3_medical_chatbot_experiment/)LMs should say “No, I don’t know the answer with confidence”

22Motivation• How can we know when language models know, with confidence, theanswer to a particular knowledge-based query?
• We examine from the point of view of calibration.

23

Model Calibration (Informal)• A well-calibrated model’s probability estimates should be wellaligned with the actual probability of the answer being correct.
• For correct predictions, we want the probability to be high• For incorrect predictions, we want the probability to be low24Model Calibration (Formal)• A perfectly calibrated model should satisfy:
ground truth

prediction

confidence25Model Calibration (Formal)• Approximated by Expected Calibration Error (ECE):
bucket predictions into 𝑀 equal-size bins based on confidence:

avg accuracy

avg confidence

Reliability diagram

Not well calibrated

well calibrated26

LM-based QA• LMs

answerquestion• T5 (3B, 11B), UnifedQA (3B, 11B), BART (0.4B), GPT-2 (0.7B)• Datasets• Multi-choice QA, Extractive QA

Multi-choice: candidate answers
Extractive: top predictions from beam search27

LM-based QA• Examples of multi-choice and extractive QA

28

LM Calibration• Fine-tuning-based• Softmax-based• Margin-based• Post-hoc• Temperature-based scaling• Feature-based decision tree

• LM-specific augmentation• Candidate answer paraphrasing• Input question augmentation

29

Fine-tuning-based• Only consider candidates in.
• Softmax-based, and directly adjust confidence• Margin-based30

Post-hoc calibration• Keep the model as-is and manipulate confidence.
• Temperature-based scaling
0: peaky∞: flat• Feature-based decision tree

DecisionTree([, entropy(), 𝑃!" (𝑋), len(X), len(Y)])

Five features31

LM-specific augmentation• Candidate answer paraphrasing• Generate T paraphrases for each candidate answer with back-translation.
• Take the sum of probability as new confidence.

• Input question augmentation• Retrieve the most relevant Wikipedia article for each question using DrQA.
• Recompute the confidence.

32

Experimental Settings• Datasets:• MC-test: 5 multi-choice QA datasets• MT-test: A recently proposed multi-choice QA datasets (particularly hard)• Ext-test: 3 extractive QA datasets• Metrics:• ECE: expected calibration error (lower better)• Accuracy (higher better)

33

Experimental Results• T5, UnifiedQA (3B)

Fine-tuning methods

Temperature scaling
Feature based decision treeparaphrasing
input augmentation

Post-hoc & LM augmentation

34

Experimental Results

Reliability diagram

Distribution of confidence
35Comparison of different LMs

36

Comparison of different LM size

3B

11B
37

Conclusion

38

Conclusion• Prompts allow use of language models as few-shot learners• How can we know what language models know?
• Prompt design• How can we know when language models know?
• Calibration methods• Many more details in the papers!

39

Bonus!
Interpretable Evaluation + ExplainaBoard

http://explainaboard.nlpedia.ai/
Based on research w/
Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye

40Motivation

Vanilla Leaderboard: Named Entity Recognition

41(Image Credit:Paperwithcode)

Motivation

Vanilla Leaderboard: Named Entity Recognition

What’s pros & cons of the state-of-the-art model?

42Motivation

Vanilla Leaderboard: Named Entity Recognition

Are there complementarities between these top-2 models?

43

Motivation

Vanilla Leaderboard: Named Entity Recognition

How well LUKE is calibrated?

44

ExplainaBoard: What’s New?

• Interpretability• Interactivity• Reliability

45

Key statistics of ExplainaBoard• 12 NLP tasks• 600+ systems• 50+ datasets• 40+ languages
Recent updates:
40 language, 9 tasks

18 language pairs, 228
systems from WMT 202046

6 evaluation perspectives,
60+ metrics

Key statistics of ExplainaBoard• Online Analysis Platform• Evaluation tool API

47

Key statistics of ExplainaBoard• Online Analysis Platform• Evaluation tool API

48

Try It Out!

http://explainaboard.nlpedia.ai/

49