Language Models (Mostly) Know What They KnowYusen Zhang

yfz5488@psu.edu

Introduction● Is AI system honest?
○ accurately and faithfully evaluate their level of confidence?

The answer given by ChatGPT is not correct,
but it is very confident about its answer.

2

Larger Models are Calibrated on Diverse Multiple Choice Questions● calibration:- A model makes calibrated predictions if the probability it assigns tooutcomes coincides with the frequency with which these outcomes
actually occur.
- Perfect Calibration[1]:Where Y hat is a class prediction and P hat its associated confidence, i.e.
probability of correctness.
- Expected Calibration Error (ECE)[1]:[1] Guo C, Pleiss G, Sun Y, et al. On calibration of modern neural networks[C]//International conference on machine learning. PMLR, 2017:3

Larger Models are Calibrated on Diverse Multiple Choice Questions● calibration: Let the model answer the multiple choice questions!

And the model prediction should be (B).

4

Larger Models are Calibrated on Diverse Multiple Choice Questions● calibration: Let the model answer the multiple choice questions!
○ Conclusion1: models are well-calibrated!

5

Larger Models are Calibrated on Diverse Multiple Choice Questions● calibration: Let the model answer the multiple choice questions!
○ Conclusion2: no noticeable correlation between accuracy and calibration○ Conclusion3: few-shot improves calibration, no use: None of the Above

6

From Calibration to Knowing What You Know

● Replacing an Option with ‘None of the Above’ Harms Performance andCalibration

7

From Calibration to Knowing What You Know

● Replacing an Option with ‘None of the Above’ Harms Performance andCalibration○ Conclusion1: it harms the performance○ Conclusion2: it makes calibration much worse

8

From Calibration to Knowing What You Know

● Models are Well-Calibrated on True/False Tasks

we expect either ‘ (A)’ or ‘ (B)’ as an answer
9From Calibration to Knowing What You Know

● Models are Well-Calibrated on True/False Tasks○ Conclusion1: The 52B model is very well-calibrated○ Conclusion2: except near the tails, where it is overconfident.

10

From Calibration to Knowing What You Know

● RLHF Policy Miscalibration Can Be Remediated with a Temperature Tuning○ Proximal Policy Optimization○ Reinforment Learning from Human Feedbacks

Advantage is positive: Suppose the advantage for that state-action pair is positive, in which case its
contribution to the objective increases
Advantage is negative: Suppose the advantage for that state-action pair is negative, in which caseits contribution to the objective reduces

11Ask the AI: Is your proposed answer True or False?

● RLHF Policy Miscalibration Can Be Remediated with a Temperature Tuning○ Proximal Policy Optimization○ Reinforment Learning from Human Feedbacks○ Temperature: higher temperature encourage the model to generate more
new content that is not in the source text.

12

From Calibration to Knowing What You Know

● RLHF Policy Miscalibration Can Be Remediated with a Temperature Tuning○ these policies naively appear very miscalibrated○ a simple temperature adjustment helps a lot!

13

Ask the AI: Is your proposed answer True or False?

● Basic Self-Evaluation○ First, ask the question:-Then, ask if it is correct:

14

Ask the AI: Is your proposed answer True or False?

● Showing Many T = 1 Samples Improves Self-Evaluation○ improve performance further by showing the model other T = 1 samples,for comparison (Thomas, Johyn etc. 5 shots)*Note: Brier Score for Multiple Classes:- f: prediction probability- o: label (0 or 1)- R: # classes- N: # samples

15

Ask the AI: Is your proposed answer True or False?

● Showing Many T = 1 Samples Improves Self-Evaluation○ Conclusion2: Overall, if given a few examples from a given distribution,models can generate samples and then self-evaluate them to productively
differentiate correct and incorrect samples, with reasonably calibrated
confidence.

16

Ask the AI: Is your proposed answer True or False?

● Showing Many T = 1 Samples Improves Self-Evaluation○ Conclusion2: conditional accuracies are substantially higher than theoverall accuracy

17

Training Models to Predict Whether They Can Answer Questions Correctly● Showing Many T = 1 Samples Improves Self-Evaluation○ Value Head: We train P(IK) as the logit from an additional value ‘head’
added to the model (independent of the logits for language modeling). An
advantage of this approach is that we can easily probe P(IK) at generaltoken positions.
○ Natural Language: We train P(IK) by asking the model to literally address"With what confidence could you answer this question?", and output ananswer like 0%, 10%, 20%, · · · 100%.

18

Training Models to Predict Whether They Can Answer Questions Correctly● Example: harder quesitons, lower P(IK) at the last token

19Training Models to Predict Whether They Can Answer Questions Correctly● Evaluating P(IK) Training and Model Size Trends○ P(IK) is very well calibrated on TriviaQA

20

Training Models to Predict Whether They Can Answer Questions Correctly● Out of Distribution Generalization of P(IK)○ the generalization of P(IK) when training only on TriviaQA and thenevaluating on other datasets (Lmababa etc.)○ Conclusion1: increasing AUROC as model size increases for all three outof-distribution evals21

Training Models to Predict Whether They Can Answer Questions Correctly● Out of Distribution Generalization of P(IK)○ Conclusion2: nontrivial generalization from TriviaQA to the other tasks,
but training on the other tasks improves performance greatly

22Other Discussions● P(IK) Generalizes to Account for Source Materials○ then P(IK) appropriately predicts a low value, specifically 18% for a 52Bmodel. However, if we prepend a Wikipedia article, then the P(IK) scorerises to 78%.

23

Other Discussions● P(IK) Generalizes to Account for Source Materials○ then P(IK) appropriately predicts a low value, specifically 18% for a 52Bmodel. However, if we prepend a Wikipedia article, then the P(IK) scorerises to 78%.

24

Other Discussions● P(IK) Generalizes to Account for Hints Towards GSM8k Solutions○ Conclusion1: showing more of the hint generally leads to higher P(IK),○ Conclusion2: good hints that lead to the correct answer result in higherP(IK) scores than bad hints,

25

Other Discussions● Comparing Models Trained with Distinct Pretraining Distributions○ Is P(IK) really showing self-knowledge or just the difficulty of the task?
○ 2 same models: A was trained with four repetitions of a high quality
dataset, while B uses a single copy lowerquality distribution of webdata.
○ Table shows: there is some signal that P(IK) is encoding model-specificinformation (because the samples are the same)○ However, the difference is not obvious 6%26

Other Discussions● Comparing Models Trained with Distinct Pretraining Distributions○ training a P(IK) classifier to predict whether or not model A knows theanswer to a question would work better when starting from model A
itself as the initial checkpoint, as compared to if we start from model B○ starting from pretrained model X should do better than starting frommodel Y when training P(IK) using data from model X

27

Conclusions and Comments●●●●Quesiton: Language Models Know What They Know ?
Calibration: Multiple Choice, None of the Above, and True/False
Self-Evaluation of Model-Generated Samples, without Finetuning
Finetuning to Identify the Questions Models Can Correctly Answer

This paper investigate an important issue. ChatGPT sometimes cannot confess it
limitations. What if we ask the P(IK) to ChatGPT?
I think the analysis is thorough, I can see their motivation and storyline, however,needs more proof reading.

28