Prompting GPT-3 To Be Reliable
Authors: Chenglei Si
Zhe Gan
Zhengyuan Yang
Shuohang Wang
Jianfeng Wang
Jordan Lee Boyd-Graber
Lijuan Wang
Venue: ICLR 2023 poster (2023)
Keywords: prompting
GPT-3
large language models
reliability
robustness
biases
calibration
knowledge updating

Abstract: Large language models (LLMs) show impressive abilities via few-shot prompting. Commercialized APIs such as OpenAI GPT-3 further increase their use in real-world language applications. However, the crucial problem of how to improve the reliability of GPT-3 is still under-explored. While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality. Our core contribution is to establish simple and effective prompts that improve GPT-3’s reliability as it: 1) generalizes out-of-distribution, 2) balances demographic distribution and uses natural language instructions to reduce social biases, 3) calibrates output probabilities, and 4) updates the LLM’s factual knowledge and reasoning chains. With appropriate prompts, GPT-3 is more reliable than smaller-scale supervised models on all these facets. We release all processed datasets, evaluation scripts, and model predictions. Our systematic empirical study not only sheds new insights on the reliability of prompting LLMs, but more importantly, our prompting strategies can help practitioners more reliably use LLMs like GPT-3.

Published as a conference paper at ICLR 2023
PROMPTING GPT-3 To BE RELIABLE
Chenglei Si Zhe Gan', Zhengyuan Yang", Shuohang Wang?
Jianfeng Wang", Jordan Boyd-Graber', Lijuan Wang" University of Maryland Microsoft clsi@ (eml@uddeel edu pkuganzhe@gmain com 1ijuawemicrosoft. com
ABSTRACT
Large language models (LLMs) show impressive abilities via few-shot prompt- ing. Commercialized APIs such as OpenAl GPT-3 further increase their use real-world language applications. However, the crucial problem of how to im- prove the reliability of GPT-3 still under-explored. While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality. Our core contribution is to establish simple and effective prompts that improve GPT-3's reliability it: 1) generalizes aut-of-distribution, bution and uses natural language instructions to reduce social biases, 3) calibrates output probabilities, and 4) updates the LLM's factual knowledge and reasoning chains. With appropriate prompts, GPT-3 is more reliable than smaller-scale su- pervised models on all these facets. We release all processed datasets, evaluation scripts, and model predictions[] Our systematic empirical study not only sheds new insights on the reliability of prompting L.Ms,, but more importantly, our prompting strategies can help practitioners more reliably use LLMs like GPT-3.
balances demographic distri-
INTRODUCTION
NLP is dominated by large language models (LLMs)) pretrained on large, unlabeled text data
that are then used for downstream tasks (Devlin et al. 2019a Brown et al. 2020 Scaling the model and data size often brings gains on downstream tasks (Kaplan et al. al. 2020 BIG-Bench
allowing what some call emergent abilities (Wei et al. (022a)). These emergent behaviors are accom- plished through prompting-4 crafted, natural language text to shape predictions or offer relevant information without expensive supervised data. Among all the existing LLMs, GPT-3 (Brown al. 2020 is particularly popular due to its flcxibility and ease of use from the OpenAI
Existing empirical studies investigate GPT-3 specific tasks such as mathematical reason- ing (Hendrycks et al.] (021a),, multi-hop reasoning (Wei et al.] [022b] Kojima et al. 2022), and code generation (Chen et al. 2021a). However, rising numbers these evaluations do not ensure
LLM reliability. For example, LLMs (including GPT-3) produce biased (Lucy & Bamman 2021
generations, false statements (Lin et 2022b), and outdated information (Chen et al. 2021h Kasai
ct al. 2022). Deploying such models in the real world could result in catastrophic harm.
In the context of prompting LLMs, several previous works have explored their reliability. For ample, in the release reports of GPT-3 (Brown et al., 2020), OPT (Zhang et al. 20022). Gopher (Rac) al. (022), there are dedicated experiments evaluating these LLMs representational bias and toxicity. Another line of work has evaluated calibration (Lin et al.)
al.
202T) and PaLMM (Chowdhery
2022a Kadavath et al.) 2022) of prompting based LLMs on math questions or multiple-choice ques- tions. We differ from these prior works in two key aspects: (i) We perform a more comprehensive study of four core facets of reliability, serving as a meta-an.lysis. (ii) We focus particularly on find-
Work done during intersship at Microsoft.
https: / /i thub. com/ Novi Sc1 /GP1 Reliabilii lity
By default, we use the CODE- NODE-DAVINCI-OU) model (also known Codex or GPT 3.5) in our experiments unless otherwise specified, hecause our preliminary results show that this is the most accurate model most NL.P datasets tried.

---

Published as a conference paper at ICLR 2023
Prompt Generatizes Qut-of--istributiio
Balanced Prompt Minimizes Social Biases
OOD Prompt:
In Domainn Prompt: What type of enzyme peroxrreoxin 2 (PRDX2jP
Stereotype lawyer the assisiant because he needed help. "he" refers to: "the lawver"
Kagoshima
Aco Gender Gep: $1.0
international is
country?
Ado Gender Gep: 8.1
Test Domain: Biology
What nerve involved caroal tunnel syndrome?
Anti-Stereotype Demo: The laweatt fired the ansidlari because alse needed help. "she" refers "the Inwyer"
MRGA Acc = 77.2
MRQA Acc = 77.2
3. Prediction Confidence Reflects Correctness
Prompt Can Updato Memorized Knowledge
Original:
CPS-23 I+p-: DIFS BERT
does Chucky's ' Cult of Churky?
Brad Dounif (Correct)
Higher
Counterfactual Update: Chusky 2017 American supernaurra psychological slasher fim wrillen and directed Don Mancini Vinci Chucky.
Higher Confedence
Leanardo Vind
Figure I: Four main reliability factors we examined and the core findings.
ing prompting strategies that are effective under these reliability facets, rather than just evaluating intrinsic model characteristics (Figure IT.
Our reliability testing framework takes inspiration from the survey unsolved problems in ML safety (Hendrycks et al. 20211b: withstanding hazards (generalizability), identifying hazards (cal- ibration), steering ML systems and reducing deployment hazards (reducing social biases and im- proving factuality). These facets also aim to address the risks of ML systems identified in existing conceptual frameworks (Tan et al. 2022 2021). We have a more extensive discussion of related works in Appendix Section
As summarized in Figure our simple prompting strategies beat smaller-scale supervised models on all reliability metrics consider: 1) prompting with randomly sampled examples from the source domain allows GPT-3 to generalize robustly on unseen domains and challenge examples; 2) exam- ples sampled from balanced demographic distribution and natural language intervention reduce social biases; 3) language model probabilities are calibrated to reflect accuracy; and 4) appending up-to-date knowledge can supplant GPT-3's memorized knowledge or reasoning chains.
FACET 1: GENERALIZABILITY
LLMs are often criticized for missing the forest for trees. They overfit training data from a particular domain (domain shift), are not robust to minor changes in a text (perturbations), use shortcuts to make predictions (spurious correlations. These pathologies make models unreliable since these distribution shifts happen all the time in real-world data and could incur significant performance drops. In this section, we study whether GPT-3 can stay robust when the test data come from different distributions than the demo examples in the prompt, and how their generalization compares to supervised models.
We study all three types of distribution shifts mentioned above. For each of them, researchers have created datasets that target modern language models' weaknesses which we adopt for evaluation. For domain shift, MRQA [Fisch et al.) 2019) trains on six machine read- ing datasets from the source domain and tests on six different target domains; for perturbations, AdvGLUE /Wang et al., (021) craft adversarial versions of GLUE (Wang et al.] 2018) based on automatic adversarial perturbations and human filtering, and Contrast Sets (Gardner al. 2020) expert-authored minimal edits that change the label: for spurious correlation, HANS (McCoy et al. 2019) and PAWS (Zhang al. 2019) challenge sets designed for models trained MNLI and
Experiment Setup
2

---

Published as a conference paper at ICLR 2023
MRQA
AdvGLUE Perturbed-
Contrast Set
Souree;
Target,
Gap.
Original,
Gap,
Original,
Perturbed-
Gap.
RoBERTa GPT-3
81.6
62.1 77.2 (S) 77.2 (T)
19.5
91.7
51.7
40.0
86.1
71.1
15.0
79.8
2.6
84.2
69.3
14.9
85.5
80.0
5.5
Table 1: For GPT-3 on MRQA target domain test sets, we report results for using both demos from the source (S) and target (T) domains, which surprisingly achieve the same Fl the target domains (77.2). For AdvGLUE and Contrast Set, we use accuracy as the metric and we use demos from the clean data the prompt. In all cases, GPT-3 few-shot prompting incurs much smaller performance gaps on the OOD or challenge test sets than the supervised RoBERTa (123M) baseline. We include the comparison with many other supervised baselines in Appendix B and GPT-3 exhibits better generalization than all of them.
QQP where the lexical overlap feature in the training data does not hold during testing. For each of these settings, we evaluate a simple prompting strategy by sampling examples from the source domains (for MRQA,
fixed prompt consisting of eight randomly sampled examples from the source domain on all target datasets; for perturbations and spurious correlation, we randomly sample 16 demos from the original clean training data from GLUE, MNLI, and QQP respectively). In addition, for domain shift, we also consider a prompt where we sample eight examples from the training set of each target domain ablate the impact of the distribution of the demo examples.
Results Table and Table 2 compare supervised RoBERTa (Liu al., 2019) and BERT (Devlin et al. 2019b) models trained on the source domain datasets or the clean training data with GPT-3 that uses exam- ples sampled from the same training data as in the su- pervised models. 3 GPT-3 achieves higher accuracy on the OOD tests even when it is slightly worse on the in-domain test sets than the supervised baselines, lead- ing smaller generalization gaps. This shows that prompting GPT-3 can be more robust than supervised finetuning of smaller-scale language models. Surpris- ingly, we compare using demo examples sampled from the source domains versus target domains on MRQA, and both prompting methods give the same OOD gen- eralization results, indicating that GPT-3 prompts can directly generalize to OOD test sets where the test ex- amples are from a different distribution than the prompt demo distribution, possibly because the role of demon- stration examples is more specifying the task rather than informing the input distribution (Min al. 2022
BERT
RoBERTa (354M)
GPT-3
(340M)
MNLI
HANS
MNLI,
86.2
89.1
77.6
HANS
71.4
77.1
75.3
Gap,
14.8
12.0
2.3
QQP
PAWS
QQP+ PAWS,
91.3
89.0
83.5
40.1
39.5
73.7
Gapı
51.2
49.5
9.8
Table 2: When using demos sampled from MNLI and QQP, GPT-3 few-shot prompting achieves much better gener- alization than smaller supervised models (BERT and RoBERTa) on the OOD test sets HANS and PAWS.
Takeaway (i) Few-shot prompting of GPT-3 more robust than supervised models such as fine- tuned BERT and RoBERTa, under all three settings (domain shift, perturbations, spurious correla- tion). (ii) Using randomly sampled demos from the source datasets a simple but strong baseline, in fact, it performs the same using demos sampled from the target distributions.
FACET 2: SOCIAL BIAS AND FAIRNESS
Apart from high performance on in-domain and OOD datasets, the second key facet of reliability is that we expect models to be fair to different demographic groups. Biased models cause severe harm when deployed in real-world applications, especially to the minority groups being discriminated against (Cao et al. 2022). In this section, we examine whether GPT-3 produces biased predictions in two downstream tasks coreference resolution and question answering.
Forr Contrast Set, show results the BoolQ subset in the main paper and present results for other subsets in Table 14 which gives the same conclusion that GPT-3 is more robust than supervised models.
3

---

Published as a conference paper at ICLR 2023
Prompt
Type Pro,
Type Anti-
Gapiul
Type II Pro,
Type II Anti,
Gap
Supervised Baseline
E2E
Lee et al.
2017
74.9
47.4
27.2
88.6
77.3
11.3
GPT-3 SootShot: Bias Distribution in the Prompt (16 shots)
Balanced
89.2
81.1
8.1
99.2
95.5
3.7
Type I- Pro
93.4
42,4
51.0
91.1
78.9
12.2
Type 11 Pro
87.6
59.5
28.1
100.0
98.7
1.3
Type Anti
50.8
80.8
-30.0
57.4
51.1
6,3
Type Anti
85.5
68.2
17.3
100.0
99.5
0.5
GPT-3 Few-Shot: Prompp Ordering (16 shots, Balanced)
Randomly Shuffled
89.2
81.1
8.1
99.2
95.5
3.7
Pro the end Anti in the end
89.5
76.3
93.7
81.8
11.9
94.2
73.2
21.0
95.5
87.1
8.4
Table 3: GPT-3 results on WinoBias. The bias gap between Pro-Bias subsets and Anti-Bias subsets indicates the extent gender biases exhibited by the model (smaller-scale gap is better). Results of the baseline ECE model (Lee et al.) (017) are taken from the WinoBias paper (Zhao et al., (018). Prompt with balanced pro-bias and anti-bias answers best shrinks the bias gap, and randomly shuf- fling the demos is better than putting one group the end.
3.1 THE CASE OF GENDER BIAS: WINOBIAS
Dataset We start with the WinoBias dataset İZhao et al. 2018) which uses templates check whether models are more likely to assign gender pronouns to stereotypical occupations. WinoBias has two types of examples: Type are ambiguous, challenging examples that require world knowl- edge; Type II can be resolved using only syntactic information. For each type, examples either confirm (pro-bias) or challenge (anti-bias) societal biases. Ideally, coreference accuracy should be similar on the pro-bias and anti-bias subsets (small gaps).
Prompt Design For case of evaluation, we re-format the WinoBias dataset into a question- answering format where provide the original sentence and then add question "What does the pronoun refer to in the above sentence?" ("the pronoun" is replaced with the actual pronoun in the sentence) and examples from the training set as the prompt and then evaluate on the Pro and Anti test sets.
the answer exact match as the evaluation metric. We randomly sample
Which Examples Should be in the Prompt We compare: sampling four demo examples from each of the Type I-Pro. Type I-Anti, Type II-Pro, and Type II-Anti subsets (Balanced), which results total of 16 demos; 2) sampling 16 demo examples from single subset. The balanced prompt induces the least biased predictions (Table second block). In particular, if we only keep Pro-Bic examples, the model will favor Pro-Bias predictions (especially on Type test examples because they are more ambiguous while Type II examples have clear syntax cues).
How Should Examples be Ordered We compare: 1) randomly shuffling the demo examples; and 2) putting all Pro-Bias Anti-Bias examples at the end of the prompt. Random shuffling reduces bias gaps most (Table third block). Interestingly, putting either Pro-Bias or Anti-Bias examples at the end increases bias gaps.
3.2 BROADER SOCIAL DIMENSIONS: BBQ
Dataset We now explore additional social dimensions using BBQ (Parrish et al.) 2022). which tests social biases against people from nine protected classes (age, disability status, gender identity, nationality, physical appearance, race, religion, socio-economic status, sexual orientation). BBQ examples are sets of four multiple-choice questions. Two questions are ambiguous the context lacks evidence to point to answer. Two other questions in each set have context that points to an unambiguous answer: the model should choose the correct answer rather than abstaining. Each question has three options: pro-bias answer that supports the stereotype, an anti-bias answer
4

---

Published as a conference paper at ICLR 2023
Prompt
Ambig Acet
DisAmbig Ace;
Ambig Bias Score
DisAmbig Bias Score
Supervised Baselines
RoBERTa- (123M)
61.2
52.7
4.9
4.7
RoBERTa-Large (354M)
49.4
87.3
10.4
1.2
DeBERTa-Base (184M)
47.6
90.4
12.8
2.9
DeBERTa-Large (435M)
30.1
95.5
24.7
-1.0
GPT-3 Few-Shot Prompting
0-shot
60.5
43.2
3.7
4,4
BBQ Balanced Neutral 99.9
96.8
76.0
2,4
1.5
BBQ Ambig-Neutral
13.2
0.0
-3.5
BBQ Ambig-Pro-Bie
2.6
97.3
24.7
3.2
BBQ Ambig-Anti-Bal
2.5
97.0
23.6
Table 4: Results on the BBQ dataset. For GPT-3 prompting, apart from the zero-shot result, others use 8-shots. For accuracy (Acc), higher value is better; for bias score, lower magnitude better. The balanced prompt best trades-off accuracy and bias for GPT-3.
Prompt
Ambig Acc;
DisAmbig Ace;
Ambig Bias Score,
DisAmbig Bias Score,
Before Intervention
2.6
97.3
24.7
3.2
After Intervention
96.6
51.5
1.9
3.8
Table 5: The impact of adding natural language intervention biased prompt on GPT-3. Adding an instruction leads the model to make more neutral predictions and reduce bias scores.
that counters the stereotype, and a neutral answer (e.g., "Cannot be determined."), When reporting results, we report: 1) accuracy on ambiguous and unambiguous questions (higher is better); 2) bias scores on ambiguous and disambiguated questions (smaller scale better). Intuitively, the bias score measures the frequency of the model predicting pro-bias answer when it makes a non-unknown prediction, where 0% means bias and 100% means always following the bias.
Answer Distribution Affects Biases We sample the number of demo examples with differ- ent answer distributions (Table D. With zero-shot inference, GPT-3's low bias is balanced by low accuracy. balanced prompt-where where sample both ambiguous and disambiguated questions with pro-bias and anti-bias answers-increant
accuracy while maintaining low bias scores. Next, when only present ambiguous questions with neutral answers in the prompt, GPT-3 easily pick up this pattern to always predict the neutral answer, resulting in extremely low bias scores and high ac- curacy on ambiguous questions, but sacrificing accuracy on the disambiguated questions. Lastly, we experiment with using biased prompts where we only sample ambiguous questions in the prompt, and we set their answers either all bias- supportinn rows in Table both cases lead to biased model that gets high bias scores ambiguous questions. This shows that when the demo examples are biased towards certain demographic groups, GPT-3 will pick up such biases, and thus should prefer balanced prompt downstream applications.
bias-countering. As shown in the last two
Lastly, we explore the possibility of using nat- ural language instructions guide the predictions of GPT-3 towards better faimess. We use the Ambig-Pro-Bil prompt and prepend the additional instruction: "We should treat people from dif- ferent socioeconomic statuses, sexual orientations, religions, races, physical appearances, nation- alities, gender identities, disabilities, and ages equally. When we do not have sufficient information, we should choose the unknown option, rather than making assumptions based on our stereotypes." to every test question. In Table 5 after adding this intervention, GPT-3 prefers the neutral answer much more often, leading to a much higher accuracy on the ambiguous questions, and at the same time significantly reducing the bias scores. This shows that GPT-3 is sensitive such natural lan- guage intervention. This is in contrast with smaller language models such as RoBERTa (Zhao et al.) 202 which is more rigid. This finding offers way for effectively reducing social biases.
Natural Language Intervention Mitigates Biases
5

---

Published as a conference paper at ICLR 2023
Takeaway (i) Demographic distribution of answers has huge impact models' biases, sampling balanced prompt best reduces biases. (ii) Randomly shuffling the demos leads to smaller biases than putting all pro-bias anti-bias examples in the end. (žiz) Specifying intended model behaviors such as being fair via instructions in the prompt can effectively guide model predictions.
4 FACET 3: UNCERTAINTY CALIBRATION
No language model can ever be perfect, and to safely use these imperfect models, users must de- cide when to trust model predictions to avoid mis- trusting wrong predictions, especially high-stake settings. This requires another facet of reliabil- ity uncertainty calibration: providing confidence scores for each model prediction that accurately re- flects the likelihood of the predicted answer being correct.
Acct
ECE,
Brier
NQ
DPR-BERS (110M)
36.1
29.4
33.5
GPT-3 LM Prob
40.5
18.9
23.3
GPT-3 Self-Con
40.2
14.3
20.1
TriviaQA (TQA)
GPT-3 LM Prob GPT-3 Self-Con
73.8
3.8
15.9
73.2
11.9
16.5
4.1 EVALUATION SETUP
HotpotQA (HQA)
Experiment Setup We study the setting of free- form answer generation: given a test question, we prompt the model to generate an answer string and obtain its confidence score (more below), and we evaluate the correctness of the generated answer based on exact match with the gold answer. We ex- periment with three QA datasets: NQ, TriviaQA, and HotpotQA. In all cases, we adopt the closed- book setting (i.e., additional evidence passages). We focus on intrinsic calibration results: using raw confidence scores rather than post-hoc calibration, which requires an additional dev set for parameter- tuning. We report the standard calibration metric expected calibration егror (ECE), the reliability di- agram and selective prediction results where we rank all predictions by their confidence and see if the accuracy of the most confident predictions significantly higher than the average accuracy. Be- cause of ECE's known flaws due to its bucketing mechanism (Si et al.] [2022), so we also report the Brier score BBiie 1950). Our baseline is a super- vised QA model BERT (Si et al., 2022) with a dense passage retriever (DPR; Karpukhin al. 2020) feed the top passages into BERT reader model for answer extraction. We follow their joint calibration setup for scoring predictions of DPR-BERT.
GPT-3 LM Prob GPT-3 Self-Con
29.8
25.0
23.5
28.5
20.7
19.9
Different Prompts on NQ w/ LM-Prob
GPT-3 2-shot
37.0
11.7
20.8
GPT-3 4-shot
38.3
13.4
21.0
GPT-3 8-shot
38.8
24.4
25.5
GPT-3 16-shot
40.5
18.9
23.3
GPT-3 64-shot
42.8
13.4
22.1
OOD Prompts w/
TQA i.i.d. Prompt
73.8
3.8
15.9
NQ Prompt on TQA
73.0
1.6
15.2
NQ - TQA
33.1
33.1
35.2
HQA i.i.d. Prompt
29.8
25.0
23.5
NQ Prompt on HQA
27.7
24.1
25.2
DPR-BERT NQ - HQA
23.6
45.7
42.4
Table 6: Accuracy, ECE, and Brier scores of GPT-3 and the DPR-BERT baseline. GPT-3 is better calibrated than supervised DPR-BERT on both in-domain and OOD settings.
Confidence Scoring We compare two ways of estimating confidence for GPT-3 predictions. LM- Prob: the (normalized) language model probability, also equivalent the reciprocal of perplexity, Conf P(w w2www.... ... w,) ) where wi Wiw.... W. are the generated tokens in the answer. Self- Con: We also explore using self-consistency (Wang et al. 2023) to obtain confidence measures. Following Wang et al. (2023), during decoding we set a high temperature value (0.7) and sample 10 times for set of different predictions. Among all the generated answers, we take the most frequent answer as the final prediction and its frequency as the confidence score.
'In Appendix Figure

---

Published as a conference paper at ICLR 2023
DPR-BERT NQ
LM-Prob NQ
Self-Con NQ
LM-Prob TriviaQA
LM-Prob HotporQA
100%
36.1
40.5
40.2
73.8
29.8
50%
41.9
58.8
62.0
88.5
47.6 6
10%
60.1
83.1
77.0
95.4
68.1
Table 7: Selective prediction results. All numbers represent the accuracy (EM) at the corresponding coverage thresholds. For example, 100% means performance on the entire test set while 10% means the performance on the most confident 10% predictions. Both LM-Prob and Self-Con allow effective selective prediction with high accuracy on the most confident subsets. More results in Appendix D
4.2 RESULTS
While still imperfect, GPT-3 (with either LM-Prob or Self-Con) is better calibrated than supervised DPR-BERT (Table (). Most calibration errors come from overconfidence where the predictions' confidence is higher than expected accuracy. Interestingly, while increasing the number of examples the prompt improves accuracy. the calibration does not improve. For example, the 2-shot accuracy is 5.8 points worse than 64-shot but better calibrated. Moreover, while OOD transfer is challenge for supervised models' calibration (tends to be overconfident on OOD test sets), GPT-3 has similar calibration regardless of the source of examples.
The selective prediction results show confidence scores can rank model predictions (Table 7 the most confident predictions have much higher accuracy. Moreover, GPT-3's confidence scores more discriminative. For example, while the average accuracy on NQ is similar between GPT-3 and DPR-BERT, the top 10% predictions get accuracy of 83.1% while for DPR-BERT it is only 60.1% 10. Such selective prediction can be very useful in practical settings, for example, we only trust the most confident predictions from the model and ask humans verify the rest, making the use of GPT-3 more reliable.
Takeaway (i) Language model probability and self-consistency frequency can produce better cali- bration on GPT-3 than supervised DPR-BERT model, especially on OOD test sets. (it) Increasing the number of demos in the prompt improves accuracy but not necessarily calibration. (iii) We can perform effective selective prediction based on GPT-3 confidence scores.
FACET 4: FACTUALITY VIA KNOWLEDGE UPDATING
Although large language models store vast knowledge in their parameters (Petroni al. 2019 the model is sometimes wrong or out of date, rendering them unreliable for knowledge-intensiee tasks. In this section, we improve this factuality aspect of reliability by improving the prompting methods.
5.1 MEMORIZATION VS UPDATING
The larger a model, the more can memorize (Carlini al. 2023). this raises the concern of whether large models like GPT-3 can forget memorized knowledge when needed and update its knowledge.
Experiment Setup Our evaluation setup is inspired by Longpre et al. (2021). who reason about counterfactual scenarios. Specifically, we sample 36K and 18K questions from NQ and SQuAD's training splits (respectively, using the splits provided by MRQA). We use 16 demo examples from each dataset as the prompt for closed-bood book QA first. We assume that if GPT-3 gets the answer the question right in the closed-bood setting, then it has already memorized that piece of knowledge. We keep the set questions where GPT-3 got right in the closed-book setting (for NQ, 21 1888 questions; for SQuAD, 7035 questions), and for these questions, we append a counterfactual passage support- ing an alternative answer. We construct these counterfactual using the entity-swap from Longpre et al. (2021): for each question, take its gold passage and replace the gold answer entity with an- other entity with the same type sampled from the same QA corpus. After such entity substitution, the counterfactual passages support the substituted answer instead of the original answer. Our pectation is that the model should generate this updated answer given this counterfactual passage, instead of its original memorized answer. We randomly sample 16 demo examples the prompt

---

Published as a conference paper at ICLR 2023
NQ
TriviaQA
SQuAD
DPR-BERT (supervised)
41.5
56.8
24.1
AtBas-I1B (64-shot)
42.4
74.5
GPT-3 Closed-Book
40.6
73.6
20.2
Contriever top-5
43.3 (61.8%)
75.6 (69.6%)
31.7 (48.8%)
+ Contriever top-10
44.2 (70.5%)
76.0 (75.1%)
34.0 (57.7%)
Table 9: GPT-3 16-shot prompting results open-domain QA datasets. We use Exact Match as the metric. For Contriever results, we additionally show the retriever's recall in brackets. Adding retrieval to GPT-3 consistently improves QA accuracy.
and we use triples of the answer-substituted passage, the question, and the substitution answers ((P), ()) in the prompt to specify the task of performing reading comprehension based on the passage.
Measuring How Well can GPT-3 Up- date its Knowledge There are three possible outcomes: 1) the model retains the memorized answer; 2) the model pre- dicts the updated answer (i.e., the sub- stitution entity in the counterfactual pas- sage); 3) the model predicts some other answer. We measure the proportion of those outcomes and hope models to up- date answers more often. For a baseline, include results from Longpre et al. (2021): a fine-tuned T5 reader- trained NQ and NewsQA-moot model with DPR retriever.
Retaing Update, Othery
NQ with Code-Davinci-002
T5 (770M, supervised)
20%
33%
47%
GPT-3
4.5%
85.4%
10.2%
SQuAD with Code-Davinci-001.
GPT-3
7.1%
84.8%
8.1%
NQ with different GPT-3 models
Text--Davnci-001 (175B)
7.2%
57.9%
34.9%
Text-Curie-001 (6.7B)
14.8%
40.0%
45.2%
Table 8: In-context knowledge updating results for mem- orized answers NQ and SQuAD. When giving coun- terfactual examples in the prompt, GPT-3 updates its an- swers around 85% of the time, much higher compared to the supervised model. Moreover, larger models are better at in-context knowledge updating.
Results As shown in Table 8 we find that when prompting with counterfactual triples ((P', Q. A")) GPT-3 can update about 85% of the time, much higher than the supervised baseline (Table R). Com- paring Text Dex--Davnnc-000 and Text-Curie- 001, the larger model also updates better to new answers in counterfactual passages.
5.2 RETRIEVAL-AUGMENNE
MEEN-DOMINN QA
Large language models can answer closed-book QA from the model's stored knowledge (Roberts) al. 2020). However, prompt can judiciously add more relevant information especially given our findings from the previous section that GPT-3 can update its knowledge with information in the prompt. We thus explore improving factual QA via augmenellaugmmented prompts.
Approach We use the unsupervised Contriever model (Izacard et al.] (022a): for test question, retrieve the top passages from the Wikipedia dump, concatenate them, and prepend them to test question. Since the context is length-limited, we only prepend retrieved passages to the test question, not the demo examples, so the demo examples are only in the form of question-answee pairs. We compare this retriever-augmenter approach with closed-book baseline where do not add the retrieved passages in the prompt. The demo examples used for both the retrieval-augmented prompting and closed-book prompting exactly the same.
Adding retrieved passages into the prompt consistently boosts GPT-3 performance on all three open- open-domain QA datasets (Table with particularly large gains on SQuAD (possibly because answers in SQuAD are spans from Wikipedia passages rather than free-form answers). Moreover, having better recall for retrieval gives better performance.
Results
8

---

Published as a conference paper at ICLR 2023
Overall
Sub-Q1
Sub-Q2
Standard Prompting
18.0 / 28.1
40.1 49.66
4
CoT
25.2 35.2
30.3 37.4
CoT + Human Sub-Q1
30.0 / 42.3
44.2/ 54.1
CoT + Human Sub-Q1 + Gold Sub-A1
/4.3/59.00
Table 10: Results on HotporQA well the decomposed sub-questions (we report EM FI). Incorporating decomposed sub-questions in the prompt makes the model adjust its follow-up step predictions and significantly improves the overall answer accuracy.
5.3 REASONING-AUGMENTED MULTI-HOP QA
The above experiments demonstrate the effectiveness of ensuring GPT-3's factuality via in-context knowledge updating; however, it is mostly constrained on simple single-hop factual questions. In real-world applications, many user queries are multi-hop- they require multiple steps of reasoning over factual knowledge. Ensuring factuality in multi-hop questions involves additional challenges: models may fail because they derive the reasoning steps wrongly. To tackle this more challeng- ing multi-hop setting, study whether it is possible improve GPT-3's multi-hop reasoning by incorporating human-written question decomposition in the prompt.
HotpotQA and Decomposed Sub-Questions We use the HotpoiQA dataset (Yang et al. 2018) for our experiments, which consists of multi-hop questions that require at least two steps of reason- ing. We use the question decomposition from Tang et al. (202I). where HotpoiQA questions annotated as decomposed (single-hop) sub-questions with corresponding intermediate answers.
Chain-of-Thoughh Prompting Chain-of- Thhuught (CoT) prompting (Wei al. 2022b) a new prompting method tailored to multi-step questions, which we adopt in our experiments as a baseline, where provide human-written reasoning steps for all demo examples to induce similar reasoning on test examples. We measure accuracy of GPT-3's final answer predictions on HotpotQA (Overall) well as on the decomposed single-hop sub-questions. From the first row of Table [O] we see that standard prompting achieves higher accuracy on the single-hop sub-questions than the entire multi-hop questions expected. CoT generates the entire reasoning chain along with its decomposed sub-questions and the intermediate answers to sub-questions, where the accuracy on the multi-hop questions is higher than standard prompting (second row of YableeTO)
Baseline:
Incorporating Human Decomposition Instead of relying on GPT-3 itself to generate reasoning chains, we add the numan-written question decomposition into the prompt. When adding the human-written sub-questions for the first step of reasoning (second last row of Table TO), we see a clear improvement both the overall multi-hop QA accuracy as well as the sub-question accu- racy. Moreover, when we further add the human-
QA pair of the first decomposed question in the reasoning chain (last row of Table 10). there is an even larger performance gain on the multi-hop QA performance. This shows that GPT-3 is able to adapt to the question decomposition information from humans and deduce the subsequent reasoning steps to eventually obtain the correct answers, offering better control and reliability.
Takeaway (i) Adding retrieved evidence passages can improve GPT-3 performance on factual QA. (ii) GPT-3 can update its knowledge when provided passages conflicting with its memorized knowl- edge. (iii) Incorporating human-written question decomposition corrects the reasoning chains of GPT-3 and improves performance on multi-hop QA.
CONCLUSION
Our work systematically studies the reliability of GPT-3 from four key facets: generalizability. fair- ness, calibration, and factuality. We develop effective prompting strategies to make GPT-3 outper- form supervised models by large margins on these facets. Our work reveals new insights of LLMs and provides practical recommendations for users of GPT-3, We hope our work can inspire more future work to: (1) examine more facets reliability, such as avoiding harmful generations; (2) real-world applications, such incorporating human feedback for collaborative multi-step planning; (3) further explore more effective prompting strategies improve reliability, such as post-hoc calibration language model probabilities.
apply the prompting methods in this paper to

---

Published as a conference paper at ICLR 2023
ETHICAL STATEMENT
Ethical Use of GPT-3 The goal of this project is to avoid the potential harm of GPT-3 and all of our GPT-3 experiments are motivated to better study and improve reliability. We believe our experiments and findings can improve the reliability and allow safer use of the model. In particular, our section on social biases and fairness is a key aspect of the ethical use of GPT-3. We presented evidence that the model exhibits biased predictions, especially when the demo examples in the prompt have skewed demographic distribution. Although we explored ways of mitigating these biases, the model is still far from perfect, and there is much more work needed to further improve its fairness. We take our work as an initial step towards more ethical use of GPT-3.
Limitations of This Work We note several limitations of this work and suggest list of open questions for future work.
Other reliability facets: In this work, we covered four key facets of reliability, but there are surely other facets that we may have missed. For example, combatting adversarial examples identified via human or Al red-teaming (Ganguli ct al. 2022 Branch et al. 2022
Perez et al., 2022), detecting and handling malicious prompts such as prompt injection
and avoiding toxic and hallucinated generations (Gehman et al. 2020 Gao et al. (022))
Methods for improving reliability: Although we have taken initial steps and discovered some effective prompting strategies for these reliability facets, readers should not take this work as evidence that GPT-3 is already reliable and ready for deployment. In fact, our experiments indicate ample room for further improvement, for example in reducing so- cial biases and improving calibration. We hope this work inspires more future work that develops more effective strategies to make L.MMs reliable.
Analysis to understand model behaviors: While we have found interesting properties of GPT-3, it remains unclear what exactly caused these behaviors. For example, if the small generalization gap due the use of prompting, or the training data, or the training objectives or model architecture? When GPT-3 is sensitive to the prompt in debiasing, is it triggered by certain keywords or phrases? Why ordering anti-bias examples the end of the prompt does not lead to the recency bias (Zhao et al., (222 but rather still incurs strong biases against minority groups? Can we attribute model behaviors to the pretraining data or interpret model attention patterns? These analysis can potentially help better understand how and why prompting works and therefore allow us to better leverage LLMs.
ACKNOWLEDGMENT
We thank Jason Phang, Ziyi Yang, Dan Friedman, Sewon Min, Jieyu Zhao, He He, Alicia Parrish, Chen Zhao, Shi Feng, Han Guo, Weijia Shi, Jungo Kasai, Xi Ye, Su Lin Blodgett, Trista Cao, Ekin Akyürek, Leo Boytsov, Aishwarya Kamath, Weijia Xu, Yankai Lin, Xiaozhi Wang, Zhengyan Zhang, and many other friends from UMD CLIP and the Azure Al team at Microsoft for their helpful discussion and feedback.
'tttps:/sii isn.. net /00// /etp//22///p////22/ http://///epp//////o/ottt------/ injptttt-njiiiioio on/
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Udit Arora, William Huang, and He He. Types of out-of-distributtin texts and how to detect them. In EMNLP, 2021.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
BIG-Bench. Beyond the imitation game: Quantifying and extrapolatingg the capabilities of language models. ArXiv, 2022. httttttts//aaattt httttp/s///axxi..oorr//a//////// 04615
Ben Bogin, Shivanshu Gupta, and Jonathan Berant. Unobserved local structures make compositional generalization hard. In EMNLP, 2022.
Hezekiah J. Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility of pre- trained language models via handcrafted adversarial examples. ArXiv, 2022. URL https:
//a //arxiv.co //axxiv.og/aabs 2209.02223
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1-3, 1950.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, 2021.
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, J. Dhamala, and Aram Galstyan. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations. In ACL, 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri- son Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Flizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa. Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ArXiv, 2021a. URL https: wlaxxv.orr/abs//200
Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time-sensitive ques- tions. In NeurlPS, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim,
11

---

Published as a conference paper at ICLR 2023
Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omer- nick, Andrew M. Dai. Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, ArXiv, 2022. URL htLLhttt:///// /ttp:://aaxvv.oo/ org/ wwtp///gg//////////////ee///e//e abs/2204 02311
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In EMNLP, 2020.
Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019a.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019b.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Workshop on Machine Read- ing for Question Answering, 2019.
Dan Friedman, Ben Dodge, and Danqi Chen. Single-dataset experts for multi-dataset question answering. In EMNLP, 2021.
Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yushi Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk. Stanislav Fort, Zachary Dodds, T. Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom B. Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv, 2022.
URL
org abs/ 2209. 07858
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, N. Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu, Attributed text generation via post-hoc research and revision. ArXiv, 2022. URL pattps://arxixe atrg/bbs/2211 08726
Matt Gardner, Yoav Artzi, Jonathan Berant, Ben Bogin, Sihao Chen, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala,, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Eric Wallace, Ally Zhang. and Ben Zhou. Evaluating models' local decision boundaries via contrast sets. In Findings of EMNLP, 2020.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxic- ityprompts: Evaluating neural toxic degeneration language models. In Findings of EMNLP, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL, 2018.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: Retrieval- augmented language model pre-training. In /CML, 2020.
Camille Harris, Matan Halevy, Ayanna M. Howard, Amy Bruckman, and Diyi Yang. Exploring the role of grammar and word choice bias toward african american english (aae) in hate speech classification. 2022 ACM Conference on Fairness, Accountability, and Transpurency, 2022.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Xi- aodong Song. Pretrained transformers improve out-of-distribution robustness. In ACI., 2020.
12

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurlPS, 2021a.
Dan Hendryeks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. ArXiv, 2021b. /tt.tttt////x...org/////////
13916
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Muchine Learning Research, 2022a. URL hnttps://peenreview nett/
pdf?id= Ddf?id=jKNlpxi70 JKN1pi17b
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval aug- mented language models. ArXiv, 2022b. URL / wttp://///aab/////// 03299
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, 2017.
Zhengbao Jiang, J. Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In AAAI, 2020.
Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yushi Bai, Sam Bowman, Stanislav Fort. Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. Language models (mostly) know what they know. ArXiv, 2022. URL hotphttss:///xiiit org/ abs/22013 05221
Jared Kaplan, Sam McCandlish, T. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, 2020. URL https://s/ wwwp:xwwwwwog///////eooooa 20011. 08361
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentarou Inui. RealTime QA: What's the answer right now? ArXiv, 2022. URL StLhhttp:: /ttp///aivvv///////////ea./aaaia org/ abs 13332
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. Mcasuring compositional generalization: A compre- hensive method on realistic data. In ICLR. 2020.
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In EMNLP, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In ICML, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurlPS, 2022.
13

---

Published as a conference paper at ICLR 2023
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference reso- lution. EMNLP, 2017.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In CONLL,, 2017.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurlPS, 2020.
Hongyu Li, Xiyuan Zhang, Y. Liu, Yiming Zhang, Xiangyang Zhou, and Jing Liu. D-net: A pre- training and fine tunntt framework for improving the generalization machine reading compre- hension. In EMNLP, 2019.
Linyang Li, Ruotian Ma, Qipeng Guo, X. Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. EMNLP, 2020.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. ArXiv, 2022a.
14334
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. ACI. 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, 2019. URL https:
Shayne Longpre, Yi Lu, Zhucheng Tu, and Christopher DuBois. An exploration data augmenta- tion and sampling techniques for
question answering. In EMNLP, 2019.
Shayne Longpre, Kartik Kumar Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In EMNLP, 2021.
Li Lucy and David Bamman. Gender and representation bias in gpt-3 generated stories. In NUSE, 2021.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL, 2019.
Sabrina J. Mielke, Arthur D. Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Com- putational Linguistics, 00:857-872, 2022.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In ICLR, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing at scale. In ICML, 2022.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Mcasuring stereotypical bias in pretrained language models. ACL, 2021.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilitics using bayesian binning. Proceedings of the AAAI Conference Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901-2900 2015.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In EMNLP, 2020.
14

---

Published as a conference paper at ICLR 2023
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Sam Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of ACL, 2022.
Ethan Perez, Saffron Huang. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nathan McAleese, and Geoffrey Irving. Red teaming language models with language models. ArXiv, 2022. URLhttps: htttps://aatt http//aaiivvr...aaaaa/aa
org/ abs///s///// /2022 03286
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models knowledge bases? In EMNLP, 2019.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jemnite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks. In NAACL, 2021.
John Platt. Probabilistic outputs for support vector machines and comparisons regularized likeli- hood methods. Advances in Large Margin Classifiers, 1999.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In SemEval, 2018.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Suther- land, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhi- guna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Mas- son d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hecht- man, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hass- abis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & in- sights from training gopher. ArXiv, 2021. URL httpp::t////........ //arxii www. hatgg/bb/////a htt///ab///1.... 11112.11144 11446
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In ACL, 2018
Adam Roberts, Colin Raffel, and Noam M. Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In NAACI, 2018
Chenglei Si, Shuohang Wang, Min- Yen Kan, and Jing Jiang. What does bert learn from multiple- choice reading comprehension datasets? ArXiv, abs/1910. 12391, 2019. URL https://
Doxiv.org/abs/11e 1910. 1910..139111 12391
Chenglei Si, Ziqing Yang, Yiming Cui, Wentao Ma, Ting Liu, and Shijin Wang. Benchmarking robustness of machine reading comprehension models. In Findings of ACL, 2021a.
Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. Better robustness by more coverage: Adversarial training with mixup augmentation for robust fine-tuning. In Findings of ACL, 2021b.
Chenglei Si, Chen Zhao, Sewon Min, and Jordan I.. Boyd-Graber. Revisiting calibration for question answering. In Findings of EMNLP, 2022.
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. In NeurlPS, 2021.
15

---

Published as a conference paper at ICLR 2023
Alon Talmor and Jonathan Berant. MultiQA: An empirical investigation of generalization and trans- fer in reading comprehension. In ACL, 2019.
Samson Tan, Shafiq R. Joty, Min-Yen Kan, and Richard Socher. It's morphin' time! combating linguistic discrimination with inflectional perturbations. In ACL, 2020.
Samson Tan, Shafiq R. Joty, K. Baxter, Araz Taeihagh, G. Bennett, and Min- Yen Kan. Reliability testing for natural language processing systems. In ACL, 2021.
Samson Tan, Araz Taeihagh, and Kathy Baxter. The risks of machine learning systems. ArXiv, 2022. URL
org/ abs/ 2204. 09852
Yixuan Tang, Hwee Tou Ng, and Anthony K. H. Tung. Do multi-hop question answering systems know how to answer the single-hop sub-questions? In EACL, 2021.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal, FEVER: a large- scale dataset for fact extraction and verification. In NAACL, 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@ : 2018.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: multi-task benchmark for robustness evaluation of language models. In NeurlPS, 2021.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In ICLR, 2023.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022b.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi nulti-hop question answering. In EMNLP, 2018.
Xi Ye and Greg Durrett. Can explanations useful for calibrating black box models? In ACL, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: Open pre-trained transformer language models. ArXiv, 2022. URL https: //axxiv.org/ab//222 01068
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word seram- bling. In NAACI, 2019.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL, 2018.
Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Kai-Wei Chang. Ethical-advice taker: Do language models understand natural language interventions? In Findings of ACL, 2021a.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In JCML, 2021b.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Brooke Anderson, and Diyi Yang. Value: Under- standing dialect disparity in nlu. In ACI., 2022.
16

---

Published as a conference paper at ICLR 2023
APPENDIX
A MORE RELATED WORK
Robustness to Distribution Shifts. Machine learning models are known to overfit their training distribution and often suffer performance degradation when the test distribution differs from the training distribution. In the case of language models, various forms of distribution shifts have been studied. For example, domain shifts pose great challenges for LLMs on question answering (Talmor) & Berant 2019: Fisch et al., 2019) and text classification (Hendryeks et al. 2020 Arora al. 2021); );
various forms of adversarial attacks can break LLMs even by just strategic synonym substitution.
paraphrase, or distractor insertion (Jin et al. 2020 Li et al. 2020 Ribeiro et al. 2018 Si et al., 2019
202 Jia & Liang 2017 Si et al. [b2 LLMs have been shown to exploit shortcuts or spurious correlations the training data which fail on counter test examples (McCoy al. 2019 Zhang et al, 2019 Poliak et al., 2018 Gururangan et (0018): LLMs also fail on new compositional structures that are not observed during traing (Kim & Linzen 2020 Keysers et al., 2020, [ogiin et al.) 2022). In real-world settings, various forms of distribution shifts can happen and reliable models should perform well even when encountering such ditto--diitribution (OOD) examples. Intuitively. in- context few-shot prompting should suffer less OOD degradation since the pretrained parameters preserved, unlike the case of supervised finetuning. We perform a series of empirical evaluations on domain shift, curated challenge sets, and spurious correlation to validate this hypothesis.
Bias and Fairness. Language models producing toxic biased content can cause severe harm especially to the groups being biased against (Bender et al. 202T). A series of benchmarks have been developed to show that LLMs can generate toxic outputs (Gehman et al., 2020), contain gen- der biases (Rudinger et al.] 2018 Zhao et al. 2018) and other categories of social biases (Nangia jet al., 2020; Nadeem et al.] [202 Parrish et al. 2022), ) perform poorly against minority demographic groups (Koh et al.) 2021 Harris et al. 2022) or dialectical variations (Ziems et al. [022, Tan et al.) 2020). Ideally. LLMs should not exhibit biased behaviors and not discriminate against any group. While many of these evaluations focus on evaluating the internal representation of LLMs in zero- shot setting or evaluating the biases on specific downstream applications in a supervised setting, remains unclear how these biases change under different prompting schemes in the few-shot setting, which will be the focus of our analysis. A closely related work is Lucy & Bamman] (202T) which study representation biases in GPT-3 generated stories. We instead evaluate on the downstream tasks of coreferece resolution and question answering. Apart from few-shot prompting, |Solaiman & Dennison (2021) proposed a general method align language models with human values, but involves expensive iterative training.
Uncertainty Calibration. No model can ever be perfect, and so it is crucial for users to be able to identify model mistakes, especially in high-stage settings where trusting wrong model predic- tions can cause severe harm. One important way to help identify wrong model predictions is by obtaining well-calibrated confidence scores for model predictions. By definition, a calibrated confi- dence (probability) score should match the expected accuracy of the prediction (Plat] 1999 Nacini et al. 2015) Guo et al.) 2017). In this way, users can put more trust in highly-contidence predic- tions and discard low-confidence predictions. While various methods have been proposed to obtain confidence scores and perform post-hoc calibration for language models (Jiang et al./ [222]] Desai & Durreut 2020 Ye & Durret 2022), they are mostly focused on classification settings rather than free-form generation, which is more common for the use of GPT-3. In this work, we explore two simple (but surprisingly effective) ways of obtaining confidence scores for GPT-3's generated an- swers and we analyse the impact of scaling as well as prompt design. For studying calibration of GPT-3 style L.MMs, [in et al. (2022a) explore the idea of expressing uncertainty in verbal words but is restricted to math questions. Mielke et al. (2022) study linguistic calibration on conversational models. Kadavath et al. (022) study adopting a multiple-choice setting in which case obtaining a confidence score is much easier (since the model only needs to predict one token to indicate which option to choose rather than generating the entire answer string). We differ from them in: 1) we focus on obtaining probabilistic confidence scores rather than verbal uncertainty expressions; 2) study the more general and realistic free-form answer generation setting: and 3) we do not involve finetuning any additional training of the language model.
17

---

Published as a conference paper at ICLR 2023
SQuAD
HotporQA
TriviaQA
NewsQA
SearchQA
NQ
Average
D-Net
84.1
Delphi
82.3
MultiFT
91.8
81.0
80.1
72.3
84.7
79.5
81.6
MADE
91.9
80.7
80.1
71.8
84.5
79.5
81.4
T5- F-netunn
94.9
1S-PromppTune
94.8
GPT-3 Source-P
87.8
78.9
88.6
60.1
87.3
76.2
79.8
Table 11: Results on MRQA in-domain datasets. We use the Fl metric for all datasets, D-Net and Delphi results are taken from the MRQA system report (Fisch et al. 2019) which only reported the average performance. MultiFT and MADE results are taken from Friedman al. (2021). T5- Finetune and TS- PS-Prrmpeeune results are from the prompt tuning paper (Lester et al.] 2021] which only trained on the SQuAD datsaet. GPT-3 few-shot performance slightly lags behind these super- vised baselines in the in-domain setting.
Knowledge Updating, Despite the fact that LLMs like GPT-3 are pretrained on very large corpora. they are still far from perfect terms of factual knowledge. On one hand, they still make factual mistakes even on domains that have seen before during pretraining (e.g., Wikipedia); on the other hand, they are pretrained on static corpora and hence their knowledge can become outdated. In order for LLMs to serve reliable knowledge bases (Petroni et al. 2019) or power knowledge- intensive downstream applications (Petroni et al., 2021). is important to keep LLMs' knowledge factually correct and up-to-update. A recent line of work attempts to edit factual knowledge in LLMs by making targeted modifications of the model's neurons (Cao et al. 202T; Mitchell et al.j 2021 2022). However, these methods are hard to be applied GPT-3 since it is much larger in size and often treated as a black box without access to internal parameters. To address this issue, in this paper we explore the feasibility of performing in-context knowledge updating by directly appending relevant knowledge pieces in the prompt to guide model predictions. Since it has been shown that larger models are better at memorization (Carlini et al., (203). we analyze whether is possible to make larger models forget their memorized knowledge and adapt to the new information presented in the prompt, especially when these two are in conflict. The idea of adding retrieved passages is conceptually similar to the line of work on retrieval-augie -ugmented methods for knowledge- intensive NLP (Lewis al., [2020] [zacard et al. 2022b) Guu et al. 2020). However, these methods still require supervised training while we focus the setting of few-shot prompting with all the language model's parameters being frozen.
B ADDITIONAL RESULTS: GENERALIZABILITY
We provide full experimental results and comparisons with more baselines.
MRQA Table 11 and Table 12 present detailed results MRQA. For baselines, include sults from the top-performing systems of the MRQA competition: D-Net (Li et (019) and Del- phi (Longpre et al. 2019). recent adapter-based robust tuning method MADE (Friedman al.] 2021) as well as their multi-dataset finetuning baseline. We also report the finetuning and prompt tuning Lester et al.) (202T) result of using T5, which achieves state-of-the-ee OOD transfer results on MRQA. Note that this T5 baseline only uses SQuAD as the in in-domain training data.
AdvGLUE and Contrast Sets Table 13 and Table 14 present detailed results on AdvGLUE and Contrast Sets, where GPT-3 shows better generalization than supervised baselines.
Spurious Correlation Table 16 shows full the performance breakdown on HANS based on the three spurious features. We can see that for the subsequence and constituent features in HANS, GPT-3 still suffers significant performance gaps between the bias-supp and bias-countering subsets. This leaves curious questions like why such gaps only occur for certain bias features but not others, and how such spurious biases arise (most likely due to pretraining), and we leave a more thorough analysis of these questions to future work.
18

---

Published as a conference paper at ICLR 2023
BioASQ
DROP
DuoRC
RACE
RE
TextbookQA
Average
D-Net
69.7
Delphi
68.5
MultiFT
64.1
51.5
63.0
47.6
87.3
59.0
62.1
MADE
66.5
50.9
67.2
47.8
86.7
58.5
62.9
T5- TS-Finetune
77.9
68.9
68.9
59.8
88.4
54.3
69.7
T5- 15-PromptTTnee
79.1
67.1
67.7
60.7
88.8
66.8
71.7
GPT-3 Source-P
86.2
67.7
70.5
69.0
89.3
84.8
77.2
GPT-3 Target-P
85.9
68.9
69.7
65.4
91.0
82.1
77.2
Table 12: Results on MRQA OOD datasets. ID-P (second last row) uses a fixed set of sampled demo examples from the in-domain datasets and evaluates these OOD datasets, while OOD-P (last row) uses sampled demo examples from each of these OOD datasets for evaluation. GPT-3 significantly outperforms all other supervised baselines these OOD datasets, moreover, using the in-domain prompt successfully transfers to OOD test data, achieving the same average performance as using demos examples drawn from these OOD datasets as the prompt.
SST-2
MNLI
QNLI
QQP
Average
Clean Test Sets
RoBERTa
96.0
89.8
86.6
94.1
92.0
91.7
ALBERT
95.2
89.6
88.4
95.3
92.3
92.2
DeBERTa
96.3
90.9
90.2
94.9
92.3
92.9
GPT-3
96.1
78.1
83.4
79.8
83.5
84.2
Adversarial Test Sers
RoBERTa 58.5
45.4
52.5
51.7
ALBERT 66.8
48.0
73.0
63.8
56.4
61.6
DeBERTa 57.9
55.5
78.9
57.9
60.4
62.1
GPT-3
78.4
56.4
87.7
57.4
66.7
69.3
Table 13: Results on the clean and adversarial test sets AdvGLUE, we use accuracy as the metric for all datasets. For MNLI, report the average performance on the matched and mismatched dev sets. The supervised baselines are trained on the clean training data, and GPT-3 uses few-shot prompts sampled from the clean training data. While GPT-3 few-shot prompting lags behind super- vised models on clean test sets, it significantly outperforms the supervised models on the adversarial sets. That being said, still note performance drop of GPT-3 the adversarial test sets when using clean demo examples.
In Table 15 perform additional ablation the impact of the number of demos and the different GPT-3 variants. With fewer demo examples from QQP, despite a slight drop on the QQP test set, GPT-3 actually remains robust (even higher accuracy on PAWS than the 16-shot results). On the other hand, using the Dext-Davinci-00 (175B) and the smaller Text-Curie-00f (6.7B) performs far worse on both the QQP test and the PAWS challenge test set.
C ADDITIONAL RESULTS: SOCIAL BIASES
We also break down the accuracy and bias scores of using different prompts in Table 17 by the different bias categories. We observe that there can be large differences across different categories. Moreover, underlined the categories from which the demo examples come, and we observe that having same-category demos in the prompt does not correlate with the bias scores. For instance, have bias supportig examples from the Nationality category in the Ambig Pro case but the bias score remains low, while the bias score for the Physical Appearance and Disability categories becomes much higher when the biased examples not from these categories.
19

---

Published as a conference paper at ICLR 2023
IMDB Original
IMDB Contrast
Gap
BERT GPT-3
93.8
84.2
9.6
94.1
93.6
0.5
QuoREF Original
QuoREF Contrast
Gapı
XLNet GPT-3
70.5
55.4
15.1
86.1
77.0
9.1
BoolQ Original
BoolQ Contrast
Gapı
RoBERTa GPT-3
86.1
71.1
15.0
85.5
80.0
5.5
Table 14: Results on Contrast Sets. For IMDB and BoolQ. we report accuracy; for QuoREF, we report F1, Apart from the performance the original and contrast sets of the three datasets, also note the gap between performance on the original and contrast sets. We see clear trend that GPT-3 incurs smaller gap than the supervised models.
QQP (ID)
PAWS (OOD)
Gap
BERT (supervised)
91.3
40.1 39.5
51.2
RoBERTa (supervised)
89.0
49.5
Mode-Davinci-000 (4-shots)
78.2
80.5
-2.3
Code-Davinci-009 (16-shots)
83.5
73.7
9.8
Texl-Davinci-001 (16-shots)
42.4
30.0
Text-Curie-001 (16-shots)
40.1
32.1
8.0
Table 15: Ablation for the impact of the number of demos and different GPT-3 model variants MNLI-HANS.
D ADDITIONAL RESULTS: CALIBRATION
The full selective prediction results in Table 18 show that the confidence scores can be used to rank model predictions. We see clear trend that the most confident predictions have much higher accuracy.
The reliability diagrams in Figure 2 show that in most cases the calibration errors come from over- confidence where the predictions' confidence is higher than the expected accuracy. It is also worth noting while OOD transfer a big challenge for the calibration of supervised models where there tends to be overconfidence the OOD test sets, GPT-3 exhibits similar calibration results when using in-domain or OOD demo examples as the prompt (bottom-left plot in Figure 2
To further disentangle the impact of better accuracy and better calibration, perform controlled evaluation of selective prediction in Table 19 where we sub-sample the NQ test set so that the three calibration methods achieve the same accuracy the test set. We clear trend that despite DPR-BERT and GPT-3 get same accuracy on this sub-sampled test set, DPR-BERT gets much higher accuracy the most confident predictions indicating the usefulness of better calibration.
ADDITIONAL. RESULTS: KNOWLEDGE UPDATING
IMPACT OF PROMPTS FOR MEMORIZATION UPDATING
For knowledge updating, compare several different prompt designs detailed below, for all cases, we randomly sample 16 demo examples as the prompt.
(C. A) We use the original question-ant question-answer answer pairs in the prompt. (P. Q. A) We use the original saasuge-questionnanswew answer triples in the prompt (i.e., the answer in the passage remains the original gold answer).
20

---

Published as a conference paper at ICLR 2023
HANS Category
GPT-3 Acc.
Lexical Overlap Entailment
87.9
Lexical Overlap Eot--nntilment
96.9
Subsequence Entailment
84.0
Subsequence Non-Entalnn Entailment
53.7
Constituent Entailment
87.2
Constituent Con-Entailment
42.2
Table 16: Breakdown of GPT-3 results on HANS by categories. All demo examples in the prompt from MNLI. GPT-3 still suffers significant performance gaps between the entailment (bias- supporting) and non-entailment (bias-countering) subsets for the subsequence and constituent fea- tures in HANS.
Balanced
Ambig Pro
Ambig-Ane Anti
Acc₃
Bias
Acc+
Bias,
Ace;
Bias
SES
96.4 / 74.2
3.2 0.0
14.2 / 199.2
-12.6
8.4 99.6
/.00
Sexual orientation
97.4 76.3
1.6/1.2
1.9/
19.1/-0.9
2,8 /97.2
17.2 -0.9
Religion
97.0 75.8
1.8/0.5
0.4/98.0
35.2/1.2
1.0997.2
24.6
Race
9 85.1
-0.1
1.2/99.0
4.0/0.1
2.7/
3.7
Physical Appearance
56.0
2.6/
1.48
75.0/14.8
0.62
77.0 14.8
Nationality
98.2/80.8 80.8
1.4/-11.6
1.0/
-0.2/0.0
1.0/98.6
0.6/0.0
Gender identity
99.0 66.8
0.6/-3.9
6.0/ 98.6
5.6/0.4
4.6
3.8
Disability
94.2/
2.2
0.0 96.6
85.2/6.0
0.2 197.2
82.6 4.8
Age
82.2/ 76.6
1..0/8.1
0.0 95.4
52.0/ 12.4
0.4
48.4/
Table 17: Breakdown of accuracy and bias score results on BBQ. For accuracy and bias scores, the first number represents the ambiguous set and the second number represents the disambiguated set. Underlined numbers indicate that there are demo examples in the prompt from the same bias category.
(Q.A"): We use the question guestion-ansser answer pairs, but with the substitution entities gold answers in the prompt.
. (P', Q. A") tution answers in the prompt.
: We use triples of the answer-substituted passage, the question, and the substi-
As shown in Table 20 find that the prompt design has a big impact on the knowledge updating behavior. In particular, showing only the original cassage-question answer triples ((P, Q. A)) still causes high memorization ratios, however, when prompting with counterfactual triples ((P", Q, A A"), GPT-3 can update 85% of the time with much lower memorization ratios than a supervised model.
E.2 TARGETED IN-CONTEXT KNOWLEDGE UPDATING
The experiments in the previous section showed promise that GPT-3 adapt to new knowledge given in the prompt when there is a confiitt with its memorized knowledge. One missing aspect from the above analysis whether we can perform targeted knowledge update: when given a piece of knowledge update, we expect the model to predict the updated answer for all questions related to that knowledge, but not change its answer for other unrelated questions. To assess model behavior on this front, we adopt evaluation setup closer to recent knowledge updating literature (Cao et al., 2021 Mitchell al. 2021).
Experiment Setup We use two evaluation datasets from Mitchell et al. (202I): 1) We first use the fact checking dataset FEVER (Thorne et al.) 2018); each claim requires a binary true / false judgement. We create the edited label which is opposite to the originally predicted label from GPT- 3. For example, for test example, if the original GPT-3 prediction is true, then the new label for editing would be false. We present the knowledge update in the form of a natural sentence that supports the target label for editing. We then test whether GPT-3 predicts the target label for
21

---

Published as a conference paper at ICLR 2023
DPR-BERT NQ
LM-Prob NQ
Self-Con NQ
LM-Prob TriviaQA
LM-Prob HotpotQA
100%
36.1
40.5
40.2
73.8
29.8
90%
38.0
43.7
44.3
78.3
32.7
80%
39.5
46.8
48.7
81.7
36.0
70%
40.6
50.2
53.1
84.1
39.7
60%
41.2
53.7
57.8
86.5
43.5
50%
41.9
58.8
62.0
88.5
47.6
40%
66.0
90.5
52.1
30%
46.1
70.2
71.2
92.5
56.5
20%
49.2
77.4
74.7
93.7
61.6
10%
60.1
83.1
77.0
95.4
68.1
Table 18: Selective prediction results. All numbers represent the accuracy (EM) at the corresponding coverage thresholds. For example, 100% means performance on the entire test set while 10% means the performance on the most confident 10% predictions.
DPR-BERT NQ
LM-Prob NQ
Self-Con NQ
100%
35.0
35.0
35.0
90%
37.6
37.9
38.2
80%
39.0
40.8
42.8
70%
40.9
44.1
46.1
60%
41.6
47.8
51.6
50%
43.1
52.9
55.1
40%
44.1
58.0
60.6
30%
45.4
65.3
63.5
20%
75.3
71.5
10%
59.2
82.5
70.5
Table 19:
Selective prediction results controlled set sampled from the original test set that the accuracy on the sub-sampled test set is the same.
paraphrase of the original test claim. We measure accuracy these paraphrases the editing success rate. We sample a same-sized set from the training data that do not overlap with the test as the set of unrelated questions. The intended behavior that adding knowledge updates in the prompt does not impact performance on these unrelated questions. We measure performance drop after and before adding knowledge updates as the accuracy drawdown. 2) We
on this unrelated also use the zsRE question-answeringg dataset (Levy et al. (017)). For each test question, the target label for editing randomly sampled from predictions by smaller QA model which different from the original GPT-3 prediction. Similarly, measure accuracy on paraphrased test questions as success rate, and accuracy drop on a set of randomly sampled non-overlapping training questions as the accuracy drawdown.
Prompt Design We compare several prompt designs (in particular what types of demo examples to use). For all cases, we sample 16 demos to use in the prompt.
Original Examples Only: We only sample the original QA pairs (without editing informa- tion).
Original Edited Relevant Examples: We include demos examples for both original QA pairs well as for questions with edited answers.
Original Edited Relevant Edited Irrelevant Examples: We include demo examples cring all possible cases: the original QA pairs, QA pairs with knowledge update and up- dated answer, as well QA pairs with knowledge update but original answer (where the question is unrelated to the knowledge update).
we see that different prompts give vastly different results. Specifically, using only the original examples in the prompt leads relatively poor success rate (especially
Results From Table 21
22

---

Published as a conference paper at ICLR 2023
GPT-3 and DPR-BERT
Different Scales of GPT-3 on TrivieQA
Ideal
100
Ideal
90
DPR-BERT
90-
Code-Davicci-022
LM-Prob
Text Davinci-001
50
GPT-3 Selt-Con
Text Curie-001
70
60
eccurccy554
40
30
10
10-
0.1
0.2
0.3
0.4
0.7
0.5
0.9
1.0
011
0.2
0.3
0.4
0.3
0.6
0.7
0.0
0.9
1.0
confidence
confidence
In-domain
TriviaQA
100
ideal
100
ideal
GPT-3 TQA Prompt
2-shot
GPT-3 NO Frompt
Bishot
80
DPR-BERT Trained
80-
50-
50
30
20
0-
0-
0.3 confidence
Figure 2: Reliability diagrams for different calibration setups. x-axis: confidence of each bucket;
y-axis: accuracy of each bucket.
Retain
Update
Other
Memorization Ratio
NQ with Code Sade-Davinci-002
T5 (supervised)
20%
33%
47%
30%
GPT-3 Prompt (Q. A)
60.8%
25.8%
13.4%
70.2%
GPT-3 Prompt Q. A)
59.1%
25.4%
15.5%
70.0%
GPT-3 Prompt (Q. A')
10.4%
56.6%
32.9%
15.6%
GPT-3 Prompt (P', Q. A')
4.5%
85.4%
10.2%
5.0%
SQuAD with Code-Davinci
GPT-3 Prompt (2, A)
58.0%
29.1%
12.9%
66.6%
GPT-3 Prompt (P. Q, A)
32.8%
52.9%
14.3%
38.2%
GPT-3 Prompt (Q, A')
15.4%
48.1 1%
36.5%
24.3%
GPT-3 Prompt (P", A')
7.1%
84.8%
8.1%
7.8%
NQ with different versions of GPT-3
Text- Lest-Davinci-O0e (175B)
7.2% 14.8%
57.9% 40.0%
34.9%
11.0%
Text-Carie-001 (6.7B)
45.2%
26.9%
Table 20: In-context knowledge updating results for memorized answers in NQ and SQuAD. When giving counter-factual demo examples in the prompt, GPT-3 can update its answers around 85% of the time with low memorization ratio (as compared to supervised models). Moreover, we find that larger models better at in-context knowledge updating.
FEVER), while adding edited relevant examples in the prompt leads to better success rate, it leads the model to over-rely the knowledge updates irrelevant questions. However, when
23

---

Published as a conference paper at ICLR 2023
Success Rate (Relevant Paraphrases)
Ace. Drawdown (Irrelevant Questions)
Prompt: Original Examples Only
FEVER
44,2
85.1 84.9 = 0.2
zsRE QA
92.9
40.0-39.7 0.3
Prompt: Original Examples + Edited Relevant Examples
FEVER
100.0
83.9 48,6 = 35.3
zsRE QA
99.9
39.7 11.6 28.1
Prompt: Original Examples Edited Relevant Examples Edited Irrelevant Examples
FEVER
99.9
84.0 - 83.5 = 0.5
7sRE QA
98.8
40.6 40.1 = 0.5
Targeted in-context knowledge updating results for FEVER and zsRE. From the last block of results, we see that when using a mixture of all three types of demo examples, GPT-3 is able to achieve very high knowledge edit success rate (99.9% and 98 88) while incurring minimal drawdown (0.5% on irrelevant questions.
Table 21:
incorporating all cases of original examples, edited relevant and irrelevant examples in the prompt, GPT-3 is able to achieve high editing success rate and low drawdown on irrelevant questions.
24


Reviews:
[{"review_id": "l4bvDLJSsJw", "reviewer": "Reviewer_S8uY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper systematically studies the reliability of GPT-3 from four key facets: generalizability, fairness, calibration, and factuality. With proper prompting strategies, GPT-3 can outperform some supervised baselines like RoBERTa and BERT on these four facets. Reliability is one important aspect of deep learning models and experiment results are comprehensive for these facets. This work shows the capabilities of large language models and sheds new insights into the reliability of prompting large language models.\n", "summary_of_the_review": "This paper studies the reliability of large language models - GPT-3 empirically and provides some interesting conclusions. Reliability is one important topic in the deep learning field and the experiment is comprehensive in this paper. Even though there exists some unfairness for further verification, their conclusions are meaningful to the deep learning community. ", "strength_and_weaknesses": "Strength: \n(i) Reliability is an important perspective for the deep learning community. This paper investigates the reliability of the famous model GPT-3. Comprehensive experiments are performed from four different aspects: (1) generalizability to distribution shifts; (2) social biases and fairness; (3) uncertainty calibration; and (4) factual correctness.\n(ii) The experiment is comprehensive in general. For instance, for generalizability, the authors consider domain shift, perturbations and spurious correlation, adversarial attacks completely. These perspectives are common facts regarding the robustness of the model.\n(iii) The conclusion is interesting for large language models, which can inspire the community further and is helpful for model deployment. For example, prompts can update memorized knowledge, which is a meaningful characteristic.\n\nWeaknesses: \n(i) In figure 1, the authors claim that \"prompt generalizes out-of-distribution\" and \"prompt can update memorized knowledge\". However, I think other facts are influencing the final results like the scale of the model and the number of data. Can you perform an ablation study to verify that?\n(ii) Since the whole paper uses empirical methods to study the reliability of the model, the number of baselines is somehow limited. For example, in Table 1, the authors only compare GPT-3 with one supervised model RoBERTa. Generalizability is a hot topic in the deep learning area and can the authors show more supervised baselines here?\n(iii) I also doubt the fairness of the experiment. Since the training strategies and the number of parameters will also influence the final experiment results, it should be better to list the experiment details and the number of parameters in each experiment. \n(iv) There are mainly two differences between LLMs and other baselines. One is the scale of the model, and another is the training setting (number of data, training strategies). However, I think the architecture of different models is also important for reliability. For example, can you provide experimental results for other types of architectures like CNN?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MbUuNWmDF3i", "reviewer": "Reviewer_UbF3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore reliability in GPT3, specifically in four areas: generalizability, fairness/bias, uncertainty calibration, and knowledge updates. They develop prompting methods for improving accuracy in these areas.", "summary_of_the_review": "If this paper's contributions were phrased as \"Others have studied reliability in LLMs. We propose a range of prompting strategies to make this work even better\", I would be more supportive. As it is, it seems more like the authors are presenting the main analysis of reliability in LLMs, which feels like an overstatement.", "strength_and_weaknesses": "Strengths:\n- In general, it's great that these ares are getting more attention, especially in LLMs.\n- The deeper analysis on prompt engineering for these specific bias use cases seems very useful (e.g., prompting with balanced demographic groups leading to more fair outputs).\n- Section 5.2 (Memorization vs Updating) experimental setup was really interesting and well-designed. This felt like it could be expanded into a larger paper, with more exploration on why types of information the model memorizes (and what it has trouble forgetting)\n\nWeaknesses:\n- It feels like this paper over-claimed its novelty. In the abstract, they say that \"existing research focus on models’ accuracy on standard benchmarks and largely ignore their reliability\", however the GPT-3 paper has multiple pages on bias analysis, as do the Gopher paper, PaLM paper, etc. \n- Similarly, there is a lot of very relevant related work in Appendix A which should be in the actual paper. For example, in Section 3 (Social Bias and Fairness) it should be made more clear that similar analyses (e.g., WinoGender) were indeed run in the original GPT3 paper, and that the main contribution in this paper is \"how these biases change under different prompting schemes in the few-shot setting\" (quoted from the appendix, but not clear in the paper itself, at least to me). \n- Typos:\n   - Page 7, \"the 2-shot accuracy is 5.8 points worse than the 64-shot is better calibrated\"\n   - Page 3, \"coreference resolution question answering\" (needs an \"and\"?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CzptoP5gvI", "reviewer": "Reviewer_r9LV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effects of GPT3 on generalization, bias, uncertainty, and knowledge discovery.", "summary_of_the_review": "This is an interesting paper that looks into how reliable GPT3 is in various settings. The takeaways will be well-received by our community. ", "strength_and_weaknesses": "Strengths\n- The four facets considered are important and timely given the popularity and impact of GPT3. \n- The paper is coherent and thorough.\n\nWeaknesses\n- It is hard to follow the conclusions. It would be a useful endeavor to have a summary of the main takeaways from each facet.\n- It would be nice to have a list of open questions that result from this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0q6lzEVLwNx", "reviewer": "Reviewer_d2Hh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the \"reliability\" of large language models (LLM) with prompt engineering or in-context learning.  Authors consider four \"facets\" of reliability: (1) robustness to domain shift, (2) neutrality w.r.t. social biases (3) uncertainty calibration and (4) ability to update the learned knowledge. For each facet, authors study how LLM prompt-tuning / in-context learning compares against traditional fine-tuning of models such as RoBERTa, T5, DeBERTa, etc. For some facets, authors compare several prompting strategies in terms of their reliability. Aside from this general line of thought, the paper also contains a plethora of other results in each of the four respective \"facets\".", "summary_of_the_review": "\n\nThis paper combines 4 related (but separate) short research papers and an overview. To the best of my knowledge, all four research directions could be upgraded to a standalone short/full paper, through the addition of baselines, missing related work, fixing some task-specific issues and providing further justification of the already claimed results. At least 2 of 4 papers (and likely all 4) would stand a good chance to be accepted here or on *ACL / EMNLP-type conferences. The overview part has its own merit: should authors modify it to add relevant prior work and focus on analyzing existing results, it would be a valuable resource to beginners.\n\nHowever, in the current form, each of the four research directions is only given a couple of pages, lacking background and task-specific nuances - and adding this information would be difficult because of the size limit, and deferring it to the appendix would make the paper difficult to follow. I would consider one of two strategies:\n\n* splitting the paper: giving each research case at least a short standalone paper -- where you will have space to better support your main claims.\n* reformatting as a meta-analysis: focus on summarizing existing work and pinpoint open problems in model reliability; possibly consider more facets, or provide a better justification for the four aspects currently in the paper.\n\nTo clarify: that is what I would have done. I will be glad if authors prove me wrong by finding a more elegant solution to these issues.", "strength_and_weaknesses": "### Strengths\n\n1. The paper studies multiple highly impactful issues; As LLM prompting is gaining new applications, it is important to understand whether or not we can \"trust\" it - and how to make it more trustworthy.\n2. Authors report a metric ton of evaluations. If the paper's worth was measured by the \"total weight\" of all contributions, this one would be among the best.\n3. Authors make considerable effort to make the paper easier to follow. The main sections (\"facets\") follow a consistent structure. Each has a short summary at the end.\n4. The paper is well-written and reasonably \"tidy\" -- especially if we take into account the density of information.\n5. Authors clearly state the limitations of their work, both in the main paper and in the dedicated limitations section.\n\n### Weaknesses\n\nIn general, it feels like the paper attempts to squeeze too much information into a limited space. Each of the four \"facets\" is an established research area with its own complexity, caveats, and prior work. It feels like the paper tries to be both an overview of these facets *and* an independent analysis of each one. In each specific section (facet), the paper has area-specific flaws: inconsistent evaluation, incomplete ablation, missing some task-specific caveats\n\n\n1. missing comparisons and related work in individual facets, e.g. GPT-3 social / ethical issues [1,2,3], GPT-3 uncertainty[4], GPT-3 domain adaptation[5], knowledge editing[6], retrieval-augmented question answering [7, 8].\n   * for instance, in Section 5 (knowledge updating), it is unclear how the retrieval-based editing compares with [6]  -- and which of the results are novel compared to [7, 8], and how does the proposed calibration strategy compare with [4].\n   * please do not consider [1-8] as an exhaustive list of related work. I am not an expert in most of these facets. My point is: since a non-expert (me) can name multiple missing baselines, authors are likely to find more relevant work if they deliberately search for it.\n\n2. As an overview of reliability: the choice of four specific facets feels arbitrary: e.g. it is unclear why we focus on generalizability and not, say, robustness to adversarial [9,10,11], or prioritize mitigating social bias over avoiding toxicity[2,12].\n\n4. (Most) results are presented as-is, with no analysis. For instance, in Section 2, GPT-3 has a smaller generalization gap than a fine-tuned MLM model -- but what exactly causes this behavior?\n  * Example A: Is this because of prompting v.s. fine-tuning? -> What if we use prompting for the MLM model?\n  * Example B: Is this because of language model's training objective?  -> what if we compare similarly-sized CLM and MLM, e.g. T5-11B vs BLOOM/BLOOM-13B -- and try prompt tuning in the baseline model\n  * Example C: Is this because of the training data? -> What if we compare CLMs and MLMs trained on similar *public* datasets?\n  * Note: it is not a weakness that authors did not perform every possible analysis - but it would help the paper\n\n\n5. (Some) claims are overly broad, e.g. \"Language model probability and self-consistency frequency can produce better calibration than a supervised DPR-BERT model, specially on OOD test sets\" -- this claim requires testing multiple language models in different setups.\n\n6. It is not always clear how authors selected the baseline models for each task, e.g. S2 only has roberta, S3 adds deberta, S4 only bert, S5 is only T5\n\n7. In T7, authors evaluate uncertainty calibration through selective prediction (e.g. accuracy on top-10% most confident). However, it is difficult to assess: what portion of the selective accuracy gain is due to better uncertainty calibration, and what portion is simply a more accurate base model. Prior art typically uses additional uncertainty metrics to decouple these two effects.\n\n\n\n[1] https://aclanthology.org/2021.nuse-1.5/\n\n[2] https://cdn.openai.com/palms.pdf\n\n[3] https://aclanthology.org/2021.acl-long.416/\n\n[4] https://arxiv.org/abs/2205.14334\n\n[5] https://arxiv.org/abs/2112.08786\n\n[6] https://arxiv.org/abs/2110.11309\n\n[7] https://arxiv.org/abs/2005.11401\n\n[8] https://arxiv.org/abs/2206.06520\n\n[9] https://simonwillison.net/2022/Sep/12/prompt-injection/\n\n[10] https://arxiv.org/abs/1908.07125\n\n[11] https://arxiv.org/abs/2209.02128\n\n[12] https://arxiv.org/abs/2112.04359\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster