The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Authors: Xi Ye
Greg Durrett
Venue: NeurIPS 2022 Accept (2022)

Abstract: Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially.

We further show that explanations generated by the LLMs may not entail the models’ predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs’ predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good—logically consistent with the input and the prediction—more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.

The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Greg Durreft Department of Computer Science The University of Texas Austin (xiye, gdurrett) @cs. utexas edu
Xi Ye
Abstract
Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (rext-davinci-00 only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially.
We further show that explanations generated by the LLMs may not entail the models' predictions nor factually grounded the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs predictions post-hoc. Through analysis in our three settings, show that explanations judged by humans to be good good-logically consistent with the input and the prediction -more likely cooccur with accurate predictions. Following these observations, extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoe across all of our datasets
train calibrators using automatically
Train
Missing You is South Korean television series starring Park You-chun and Yoo Seung-ha ho. Yoo Seung ho (bom 17 August 1993) South Korean Pork Yoa-chun (born 23 1990) is South Korean Q Missing actor was born Augast 17 1993?
Example
Prompt
Explanation Labbel
A: First, Missing You stars You Seung-hu ho. Second, You Seung- ho is born 17 August 1993, The unswer is You Seung-ho.
Test
Cresdallen is a track on The Smashing Pumpkins" album. Adore. The singk's artwork by Yelena Yemehuk, Johnny McDaid is Croatian professional photographer. Yelena Yemchuk is Ulrminian professional photographer. by photographer which nationalit??
Example
Crestfallen's
GPT-3
A: First, Crestfallen's artwork is done by Yelena Yemehuk. Second, Yelena Yemchuk is a Croatian professional photographer. The answer Croation.
Output
Calibrator
The prediction is incorrect. The explanitiio not factual with respect to the context.
Figure 1: Prompting GPT-3 with explanations. By including explanations in the in-context examples, cause GPT-3 generate explanation for the test example well. In this case, the generated explanation nonfactual, despite the simple reasoning involved here. However, we show this nonfactuality actually provides signal that can help calibrate the model.
Data and available at http::////gtttaaco////////ttttta InContet
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Recent scaling of pre-training has empowered large language models (LLMs) to learn NLP tasks from just a few training examples "in context," " without updating the model's parameters (Brown) et al. 2020). However, this learning process is still poorly understood: models are biased by the order of cn-ctneext examples (Zhao et al., 2021) and may not leverage the instructions or even the labels of the examples in the ways one expects (Min et al. 2022 Webson and Pavlick, 2022). Existing tools for interpreting model predictions have high computational cost (Ribeiro et al., 2016) or require access to gradients (Simonyan et al. 2014 [Sundararajan et al., 2017), making them unsuitable for investigating in-context learning or explaining the predictions of prompted models.
One appealing way to gain more insight into predictions obtained through in-context learning is to let the language model "explain itself" (Nye et al.) 2021 Wei al. 2022; Chowdhery al. 2022
Marasović et al. 2022: Lampinen et al. 2022). In addition to input-label training pairs in context, one can prompt the language model with an explanation for each pair and trigger the model generate an explanation for its prediction (Figure D. Prompting with explanations introduces much richer information compared to using labels alone, which might guide the inference process and allow the model to learn more information from the examples.
In this work, investigate the nature of the explanations that LLMs generate and whether they can improve few-shot in-context learning for textual reasoning tasks, specifically QA and NLI. Recent prior work that finds success with this approach largely targets symbolic reasoning tasks with very different structure, such as math word problem solving (Nye et al. [021] Wei et al., (0022). We experiment on three different datasets spanning QA and NLI with four LLMs: OPT, GPT-3 (davinci), InstrucIGPT fextt-dvinnci-011, and tex--davinci-002. The results suggest that explanations only substantially improve accuracy for text-davinci-O00 but give a smaller improvement or even hurt the performance with the other LLMs.
Surprisingly. we find that the explanations generated by LLMs can be unreliable, even for a very simple synthetic dataset. We evaluate the explanations along two axes: factuality, whether the explanation is correctly grounded in the input, and consistency, whether the explanation entails the final prediction. LLMs tend to generate consistent explanations that account for the predictions, but the explanations may not be factual, as as shown in Figure [ Furthermore, our analysis suggests an unreliable explanation more likely indicates a wrong prediction compared to a reliable explanation.
Despite LLMs' failures here, we can still benefit from for calibration. If we are able to automatically assess the reliability of an explanation, we can allow an LLM to return a null answer when its explanation is unreliable, since the prediction in this case is less likely to be correct. Unfortunately, there is no automated way to perfectly assess the reliability, but we can extract features that approximately reflect it. We use these features to calibrate InstructGPT O predictions, and successfully improve the in-context learning performance across all the datasets.
explanations by using them
In summary, our main findings are: (1) Simply plugging explanations into the prompt does not always substantially boost the in-context learning performance for textual reasoning. (2) LLMs generate explanations consistent with their predictions, but these explanations might not be factually grounded in the inputs. (3) The factuality of an explanation can serve as an indicator for the correctness of the corresponding prediction. (4) Using features that can approximate the factuality of explanations, successfully use explanations to improve the in-context learning performance across all tasks.
2 Does Prompting with Explanations Improve In-Context Learning?
In this paper, we specifically focus on tasks involving reasoning over natural language. These are tasks where explanations have been traditionally studied (Camburu et al. 2018 Rajani et al. 2019), but which are more complex than tasks like sentiment analysis which are well explained by extractive rationales (Zidan et al., 2007 Young et al. 2020). We experiment on two tasks,
TThoughout our paper, we primarily test InstructGPT for two reasons. First, it was the most capable model available the time we were conducting the majority of our experiments. Second, it still has significant room to improve on the datasets we explore in this work. This setting is a representative testbed for the situation where an LLM-based system does not yet give satisfactory performance on a target task, causing the system designer to turn explanations in prompts to improve things.

---

Context:
Christopher agrees with Kevin. Tiffiayy agrees with Matthew. Mary lungs out with Danielle. James hungs out Thomas. Kevin student. Matthew is a plumher. Danielle student, Thomas plumber.
B-SNLI SYNTH
Question:
Who hangs with student!
Answer:
Mary
Explanation: Danielle a student and Mary hangs out with Danielle.
Premise: A toddler in green jersey being followed by wheelchair bound wornn in a red sweater past wooden bench.
Hypothesis: todileri walking near his wheekchuir bound grandmother.
Label:
Neitber Explanation: the woman may not be his grandmother.
Figure 2: A SYNTH example and an E-SNLI example. See Figure
3 for ADVHOTPOT examples.
reading comprehension question answering (QA) and natural language inference (NLI), three Inglish-language datasets. For each dataset, we create a test set with 250 examples.
2.1 Datasets
Synthetic Multi-hop QA (SYNTH) In order to have a controlled setting where we can easily understand whether explanations are factual and consistent with the answer, we create a synthetic multi-hop QA dataset. Shown Figure 2 each example in this dataset asks bridge question (using the terminology of Yang et al. (2018) over context consisting of supporting facts paired with controlled distractors. This dataset is carefully designed to avoid spurious correlations, giving us full understanding over the correct reasoning process and the explanation for every example, which naturally consists of the two supporting sentences. See Appendix B for full details this dataset
Adversarial HotpotQA (ADYHOTPOT) We also test on the hogllsh-languaage Adversarial Hot- PotQA dataset Yang et al. 2018 Jiang and Bansal 2019). We use the adversarially augmented version since InstructGPT achieves high performance the distractor setting of the original dataset. We make a challenging set of examples by balancing sets of questions on which InstructGPT makes correct and incorrect predictions. The context of each question includes two ground truth supporting paragraphs and two adversarial paragraphs. Full details of preprocessing the ADVHOTPOT dataset can be found in Appendix C
For ADVHOTPOT, we manually annotated explanations for the training examples. Figure I shows an example of such explanation, highlighted in orange. We could use the supporting sentences the explanations, but we found they are usually too verbose and not sufficient, c.g., with anaphors that resolve outside of the supporting sentences. Therefore, manually annotate set of explanations which clearly describe the reasoning path for each question.
classification dataset commonly used to study explanations, released under the MIT license. Shown in Figure 2 each example consists of premise and hypothesis, and the task is to classify the hypothesis entailed by, contradicted by, or neutral with respect to the premise. As notable contrast to the other datasets, the explanations here abstrac/ natural language written by human annotators, opposed to mostly constructed from extracted snippets of context.
E-SNLI E-SNLI (Camburu al. 2018) is an
2.2 Baselines
We study the effectiveness of plugging in explanations by comparing the in-context learning perfor- mance of prompting with or without explanations. Prompting without explanations resembles the standard few-shot in-context learning approach (Few-Shot). To incorporate explanations into the prompt, consider the following two most commonly used paradigms:
Explain-then-PPe trppain-then-Predicct (F-P) prepends an explanation before the label (Figure Tp. The language model is expected to generate explanation first followed by the prediction. The prompting style of past work involving computational traces can be categorized into this paradigm, including Nyee al. I202I) and Wei et al. (2022). This approach is also called pipeline model in other literature on training models using explanations (Jacovi and Goldberg 2021 Wiegreffe et al. 202T).
'This dataset is inspired by task 15 of the bAbl dataset [Weston et al. 2016). In our preliminary experiments with some of the other bAbl tasks, we found poor performance from InstructGPT similar to our results on SYNTH, both with and without explanations.
3

---

Table Results of prompting with explanations on four large language models. Using explanations leads to small to moderate improves performance on OPT, GPT-3, and InstructGPT, and has more prominent effects on text-davinci-002.
SYNTU
ADVHOTPOT
E-SNLI
FEW-SHOT
40.52s
49.726
44.0aa
OPT (175B)
E-P
29.60.5
52.6,s
39.37
P-E
40.2gg
43.345
43.416
FEW-SHOT
49.5us
49.162
43.357
GPT-3
E.P
47.
54.141
40.445
P-E
51.3ss
48.746
48.724
FEW-SHOT
54.8s1
53.2as
56.820
InstrccGPPT
E-P
58.521
58.241
41.82s
P-E
53.6
51.52a
59.4.
FEW SHOT
72.0,
77.7a2
69.120
text devi-dvvinciiiio 002
E-P
86.93
82.451
75.626
P-E
81.128
77.248
69.450
tredict-then-Explain (P-E) generates the explanation after the prediction. Unlike E-P, the predicted explanation does not influence the predicted label, since we use greedy inference and the explanation comes afterwards. However, the explanations in the prompt still impact the predictions.
2.3 Setup
For few-shot learning, we use roughly the maximum allowed shots in the prompt that can fit the length limit of OPT (Zhang et al. 2022) and GPT-3 (Brown al. 2020), which 16 for SYNTH. 6 for ADVHOTPOT, and 32 for E- SNLI, respectively We experiment with four LLMs, including OPT (175B), GPT-3 (davinci), InstructGPT (1ext-davinci-001), and text-davinci-002. OPT and GPT-3 are trained using the standard causal language modeling objective, whereas InstructGPT and text-davinci-002 are trained with special instruction data and human annotations. We generate outputs with greedy decoding (temperature set to be O). Our prompt formats follow those in Brown et al. (2020). The explanations are inserted before/after the prediction with conjunction words like because. Please refer to Appendix for full prompts. Because the results of in-context learning vary with the examples presented in the input prompt, for each dataset, we randomly sample multiple groups of training shots, and report the mean and standard deviation of the results (subscript). We use 5 groups for InstructGPT, the primary LM we are using throughout our paper, and 3 groups for the rest.
2.4 Results
As shown in Table OPT, GPT-3, and InstructGPT show small to moderate improvements from using explanations for textual reasoning tasks. On the two QA tasks, SYNTH and ADV HotFor, E-P improves the performance of InstructGPT, the best among these three LMs, from 54.8 to 58.5 and 56.8 to 59.4, respectively 5 On E-SNLI, P-E outperforms FEW-SHOT by 2.6, whereas E-P substantially lags FEW-SHOT. Comparing E-P against P-E on SYNTH and E- SNLL, E-P typically degrades performance (except on SYNTH for InstructGPT) and P-E is inconsistent across the different models, whereas E-P consistently leads to performance improvements on ADV HOTPOT. There is no single winner between the two paradigms of using explanations; choosing the most effective way is task-spe Overall, vanilla LLMs (OPT and GPT-3) see limited benefit from producing explanations, and even the Instruct-series InstructGPT does not see substantial improvements.
The only exception text-davinci-002. text-davinci-002 greatly benefits from explanations in the prompt across all three tasks, and E-P is consistently more effective than P-E. However, is unclear what contributes to this difference. As far as we are aware, the differences between text-davinci-002
'This contrasts with recent work like Zhao al. 2021 that focuses improving performance in the 1 1-o-shot setting: by using more we achieve much stronger results on our tasks. For SYNTH, we also tried using an alternative style of explanations (reversing the order of the two sentences in the explanations), which leads to mild performance degradation.

---

Pedro Rubens! The individual chapters were published into 64 "lanköbon" by Kodamsha.
Yako Shäji (hom June 1950, Mohara, Chiba) Japanese manga artist.
known for writing "Seito Shokun!
Nonfactun
Mulder Scully! The individual chupters were published into 14 "lanköbon" by Kodansha.
Seito Shokun! The individual chapters were published into "tankihon" by Kodannhu between.
How many chapters does Yoko Shoji's
famous manga have?
A: First, Yoko Shōji's 's most famous manga is "Scito Shokun!". Socond, "Scito Shokun!" has 64 chaptors. The answer 64.
Tim Minchin (December 29, 1808 July 31. 1875) the President of the United States.
Inconsistent
Andrew Johnson (December 29, 1808 July 31, 1875) was the President of the United States,
George Andrew Atzerodt (June 12, 1835 July 7. 1865)
conspirator, with John Wilkes Rooth.
Josse Andrew Williams (June 12, 1835 - July 7, 1865) was a conspirator, with John Wilkes Booth.
Q: Who was older, George Atzerodi Androw Johnson?
A: First, Geurge Atzerodi was burn on June 12, 1835. Second, Andrew Johnson was born on December 29, 1808. The
answer George Atzerods.
Figure 3: Explanations generated for ADVHOTPOT. InstructGPT may generate nonfactual explana- tions containing hallucination (red) or inconsistent explanations contradicting the answer (red).
Table Left: factuality (Fac) and consisteney (Con) of the generated explanations. Right: the % of the examples whose explanation bactuality/consistene is congruent with the prediction accuracy. In general, L.LMs tend to generate consistent but less likely factual ex explanations.
Acc
Fac
Con
Fac
Acn
reliability of explanations generated by InstraciGPP
SYNTH (E-P)
58.4
72.8
64.8
66.5
68.8
SYNTH (P-B)
54,8
51.6
95.2
89.6
57.2
InstructGPP
ADVHP
62.0
79.6
91.2
80.0
68.4
ADVHP (P-E)
54,0
69.2
82.0
77.6
67.2
E-SNLI (P-E)
62.0
98.8
62.0
reliability explamations generated by other LLMs SYNTH
SYNTH (E-P)
30,0
71.2
47.2
45.6
58.8
OPT (175B)
SYNTH (P-B)
39.6
64.0
81.2
69.2
49.6
SYNTH (E-P)
46.8
59.2
64.8
66.8
61.2
GPT-3
SYNTH (P-B)
52.4
52.4
83.2
78.4
58.0
text-davinci-002
SYNTH (E-P)
86.0
91.6
85.2
91.2
84.8
SYNTH (P-E)
81.6
83.2
95.8
82.8
and InstructGPT are not described in any publication or blog post 7 Comparing GPT-3 and Instruct- GPT, we see the move to Instruct series models is not sufficient to explain the difference. Given the lack of transparency with this model, we hesitate to make scientific claims about the results it yields.
Our results do not suggest immediate strong improvements from incorporating explanations across all LLMs, even for our synthetic dataset, contradicting recent prior work. This can be attributed to the difference between the tasks we study. The tasks that receive significant benefits from using explanations in Nye et a 2 and Wei et al. (2022) are all program-like (e.g., integer addition and program execution), whereas the tasks this work emphasize textual reasoning grounded in provided
inputs. In fact, in Wei et al. (022) and Chowddeery et al. (2022). explanations only show mild benefit
on open-domain QA tasks like StrategyQA (Geva et al. 2021 that are closer to our setting.
3 Can LLMs Generate Factual and Consistent Explanations?
Prompting LLMs with explanations and having models generate them may not guarantee higher performance our tasks. But what about the quality of the model-generated explanations themselves? We assess the reliability of the explanations for the three datasets, measured in terms of two aspects.
Factuality refers to whether a generated explanation is faithfully grounded in the corresponding input context (context for QA and premise/hypoohhesis pair for NLI). factual explanation should not contain hallucinations that contradict the context. See Figure for nonfactual explanation.
Consistency measures if the explanation entails the prediction. Our concept of consistency resembles plausibility as described in Jacovi and Goldberg (202T), in that we assess whether the prediction follows from the explanation as perceived by a human. See Figure 3 for an inconsistent explanation.
One publicly-described difference is the addition of editing and insertion, discussed at /ttttttss///pp/// com/ wwwog/gppt-oooo 3-edit- wwweediittttrr///eeeeee but this does explain the performance differences we observe.
5

---

For SYNTH, we use rules to automatically judge whether an explanation is factual and consistent on all four LLMs, For HOYHOTPOT and E-SNLI, the authors manually in- spected the explanations generated by InstructGPT and annotated them for these two characteristics (more details in Appendix D). Note for each setting, the results are based on the explanations and predictions obtained with a single set of training shots. We only show the results P-E on E-SNLI, E-P is substantially worse here.
of Cerrereddderree Predictions Cactualigy/onssstaacc
Systheac 1P-8)
60-
40
Results We summarize the results in Table We only report consistency on E-SNLI, as the explanations for E- SNLI often require some external commonsense knowl- edge which cannot be easily grounded in the inputs judged as true or false (examples Appendix IF). The results suggest disconnect between the model predic- tions and the "reasoning" in explanations. On InstructGPT, though using explanations improves its performance across three tasks, the generated explanations are unreliable (upper section), even for the straightforward synthetic setting. Comparing the factuality of explanations for SYNTH generated by GPT-3, InstructGPT, and lext-davinci-002, we see that instruction tuning improves the factuality, but even the most powerful text-davinci-002 still fails to generate explanations that are perfectly grounded in the input context. Overall, LLMs tend generate consistent explanations (>80% for all three datasets with the right prompt structure), but the explanations are less likely to be factual, which is concerning as they can deceive user of the system into believing the model's answer.
Nochat Far Iniens Cors
Nonffer Part
Insos Cors
Figure 4: Explanations to be nonfactual than to be inconsistent, and nonfactual explanation usually in- dicates an incorrect prediction.
more likely
3.1 Reliability of Explanations and Prediction Accuracy
LLMs may hallucinate problematic explanations, but this could actually be advantageous if it gives us way of spotting when the model's "reasoning" has failed. We investigate the connection between the reliability of an explanation and the accuracy of prediction and ask whether reliable explanation indicates an accurate prediction. (This resembles the linguistic calibration of Mielke al. (2022). but using a different signal for calibration.)
As shown in Table (right), accuracy and factuality/consisteenyy are typically correlated, especially factuality. By knowing whether an explanation factual, we can guess the model's accuracy a high fraction of the time (Accuracy Factuality). A nonfactual explanation very likely means incorrect prediction on the SYNTH dataset across all four LLMs. On ADVHOTPOT, factuality and Instruct- GPT's prediction correspond 80.0% of the time, substantially surpassing the prediction accuracy itself. We show fractions of correct and incorrect predictions when the explanations are factual/nonfactual and consistent/inconsiitenn in Figure for two of our settings. Factual explanations are much more likely paired with correct predictions compared to nonfactual explanations. Consistency is also connected accuracy but an inferior indicator compared to factuality in general (Table 2
Calibrating In- In-Context Learning using Explanations
we that human oracle assessment of the factuality of an explanation could be of substantial use for calibrating the corresponding prediction. Can we automate this process?
From Section 3.1
We first show how to achieve this goal on the perfectly controlled SYNTH dataset (Section 4.1 On our other two datasets, we use surface lexical matching approximate semantic matching and give real-valued scores approximately reflecting factuality. Following past work on supervised calibration (Kamath et al. 2020; Chen al.) 202TT] Ye and Durrett (022), we can learn a calibrator that tunes the probabilities of prediction based on the score of its explanation (Section 4.2). We show such calibrator can be trained with a handful of examples beyond those used for in-context learning and
successfully improve the in-context learning performance on realistic datasets! We note that, as mentioned before, the experiments in this section are conducted on InstrucIGPT.
"This procedure does require extra data. However, it provides a natural avenue for using a small number of additional examples that otherwise would be impossible to incorporate into this procedure, when the size of the context actually limits the amount of data for in-context learning.

---

4.1 Motivating Example: Improving SYNTH Dataset
We first show how post-hoc calibration functions in the controlled SYNTH setting, where we can simply check the factuality of an explanation. Since the generated explanation always follows the format "B is [profession] and A .verb] B. (example in Figure 2) we can split the explanation into two sentences. The explanation factual if and only if each of the two sentences exactly matches one of the sentences in the context.
We use the assessment improve the performance of P-E for SYNTH, where nonfactual explanation typically indicates an incorrect prediction. This gives us a way to reject presumably incorrect answers. Specifically, we iterate through the top 5 candidate answers (restricted by the API) given by InstructGPT and reject any answer-explanation pair if the explanation is nonfactual until we find factual one. This procedure dramatically improves the accuracy from 52.4% to 74.8%. Note that this SYNTH dataset is a challenging task given its lack of reasoning shortcuts: for reference, neither ROBERTA (Liu et al. 2019) nor DEBERTA (He et al. (021) finetuned with 16 examples can achieve an accuracy surpassing 50%. With the help of the explanations and the checking procedure, we can use InstructGPT achieve strong results using few-shot learning.
4.2 Learning-based Calibration Framework
Framework We now introduce the framework that can leverage the factuality assessment of an explanation to calibrate a prediction. Let p be the vector of predicted probabilities associated with each class label in NLI (or the probability score of predicted answer QA). Let o be a scalar value extracted from the explanation to describe the factuality. Then, we can adjust the probabilities accordingly using a linear model: p softmax(W (p; v| 6), where P is the tuned probabilities.
Our calibration framework extended from classical calibration methods (Platt 1999, Guo et al. 2017 Zhao et al., 2021), which apply affine transformation the probabilities alone: softmax p + b). In contrast, we use an additional factor e in calibration to incorporate the factuality assessment of the explanation.
There are a small number of parameters (W and b) that need to be trained such calibration framework. We will rely on few more examples in addition to the shots we use in the prompt to train the calibrator. Specifically, we use the prompt examples to generate the predictions and explanations for these extra examples, and extract predicted probabilities, factors, and target probabilities triples to construct training data points used to train the calibrator. Note this procedure requires no explanation annotations for the extra examples.
Approximating Factuality We approximate the factuality using lexical overlap between the explanations and the inputs, which we found to work fairly well for our tasks.
HDyHOTPOT: We use an explanation consisting of two sentences (examples in Figure 3 as an illustration. Let E = (E E2) be the generated explanation, where g() and (2) are the two sentences, and the E()) (2,,62.-.. contain tokens 61, -...... Similarly, let P (P(1), p(2),
be the context paragraphs, and p(t) (p1, P2. ...) be the tokens. The pP
factuality estimation of explanation sentence E(0 is defined as: V(E(0) = max [0O1
Intuitively, the factuality score for a sentence E is defined as the maximum number of overlapping tokens over all paragraphs P. normalized by the number of tokens in E. We then define the factuality score for the whole explanation as V(E)= min geg (EE), as it requires all sentences to be factual in order to make the entire explanation factual
The explanations of E-SNLI do not really involve concept of factuality. Nevertheless, we use an analogous score following the same principle by viewing the premise as the context. Let E (e1, 62, ..) be the explanation and P (P1, P2. be the premise. We simply score the explanation by (EE)= = Ei factual we judge it to be.
E-SNLI:
The more an explanation overlaps with the premise, the more [E]
Altermatively, one might use fine-tuned NLI model as a proxy Chen et al. 2021 However, our focus is on the pure black-box setting, and avoid models that require substantial amounts of data make work.
7

---

4.3 Calibrating E-SNLI
Setup For E-SNLI, we use calibration meth- ods to postprocess the final probabilities. Un- like classical temperature scaling (Platt 1999). note that the methods we use here can actually change the prediction; we will therefore evaluate on accuracy of the calibrated model.
Table 3: Accuracy (mean dev) of various methods on E-SNLI under different data conditions. L de- notes number of labels (as well the total number of examples); E denotes the number of explana- tions. Calibrating using explanations successfully improves the performance of in-context learning.
We study the effectiveness of our explanation- based calibrator under different training data sizes varying from 32 to 128. Recall that we only require explanation annotations for 32 data points, and only need the labels for the rest to train the calibrator. For E-SNLI, we calibrate P-E, which is shown to be more effective than E-P in this setting (Section 2.4
w/o Explanation
32L
64L
96L
128L
RoBERTa
40.
43.0g.1
49.0g;
54.948
FEW-SHOT SHOT(NN)
56.820
58.9
FEW PRO-HHC
61.938
62.426
63.229
63.9,2
w/ Explanation
32L+32E
64I.+32E
961.422
128L+32E
P-E
59.420
P-E-PRONCAL P-E-EXPLCAL
64.4ss
65,41,2
65.41.
65.4
64.22n
65,8,3
67.61s
68.51,
Baselines We provide the performance of fine- tuned RoBERTA (Liu et al., (219)) model as reference, finding this to work better than De- BERTa (He et al. 202T). To isolate the effec-
P-ENZHANG
63.0g2
66.22,
65 A1.
65.925
tiveness of using explanations for calibration, we introduce three additional baselines using non- explanation-based calibrators. We apply the probability-basee calibrator as described in dection[4.2jo the results obtained on few-shot learning (FEW-SH SHEW-SHOT+PROBC.. (P-E+PROBCAL) We note that the parameters of these calibrators are trained using the additional data points, as opposed to being heuristically determined as /hao a (202T). Furthermore, we experiment with a recently proposed supervised calibrator from ZZhng et al. (202T), which uses the CLS representations from an additional language model as features in the calibrator. The probabilities tuned using
and predict-then-explee then-explain pipeline
softmax W [p: h) + b). where h is the CLS representation. Since we do not have access to the embeddings obtained by GPT-3, we use ROBERTA to extract the vectors instead. We use such calibrator on top of our best-performing base model, P-E, resulting P-E+ ZIIANG ET AL. (021).
Limited by the maximum prompt length, in-context learning is not able to take as input the additional data used for training the calibrator. For a fair comparison, we can allow the in-context model to use this data by varying the prompts across test examples, dynamically choosing the prompt examples to maximize performance. Choosing closer data points for prompting is common and effective way of scaling up the training data size for in-context learning (Shin et al. [222]] Liu et al. 202T). Following Liu et al. (202I), we test the performance of choosing nearest neighbors for the prompt based on CLS embedding produced by ROBERTA model (Liu et al. 2019), referred as (NW-SHOTT(NN). It worth clarifying that the FEW-SHOT and NEw-SHoOT+PROBCA. approaches use the same set of 32 training shots in the prompt for every test example, whereas the shot sets vary from example to example in FEW- SHW-SHOTTNN).
Results We show the results in Table We use 5 different groups of training examples and report the mean and standard deviation across the groups. For FEW-SHOT(NN), we only report the results obtained using 128 examples; results using a smaller number of examples will be worse than this.
Under 128 training examples, applying trained calibrator on top of prompting with explanation (i.e., P-E+EXPLCA. achieves the best accuracy of 68. 3%,, which is 12% higher than the performance of the vanilla uncalibrated few-shot in-context learning (FEW-SHOT). P-E+EXPLCAL also outperforms SEw-SHOT+PROBCAL and P-E+PROBCAL by 5% and 3%, respectively. Using explanations is more effective than using probabilities alone. In addition, P-E+EXPLCAL also outperforms P- P-E-ZHHANG ET AL. 20021). whose performance is on par with P-E+PROBCAL. This suggests the additional CLS information not very helpful in this setting.
As the data size increases from 32 to 128, the performance of the explanation-baat calibrator keeps calibrators nearly saturates at
improving notably, whereas the performance of probability- data size of 96. The performance of FEW-SHOT(NNN) with 128 training instances only improves the performance by 3.3%, compared to FEW-SHOT with 32 training instances. Choosing nearest
8

---

Table 4: AUC scores (mean dev) on ADVHOT- POT under different data conditions. L and E denotes the number label annotations and ex- planation annotations, respectively. Explanation- based calibration successfully improves the per- formance top of prompting with explanations.
CoveragerAc: on Advitolpot
73-
60-
w/o Explanation
6L
32L
64L
FEW-SHOT
59.624
45-
POW-SHOT(NN)
61.3u9
0.2
0.5
0.1
0.5
0.7
6.8
0.6
1.6
w/ Explanation
6L+6E
32L+6E
64L+6E
Coverage
E-P
64.429
Figure 5: Coverage- curves various methods on ADVHOTPOT. E-P+EXPLCAL is better calibrated compared to uncalbrated E-P well as the other approaches.
E-P+EXPLCAL
66.0.s
68.83.0
E-P+ZHANG
65.61.
66. 66.1sa
neighbors the shots, while being effective when having access to large amount of data, is helpful in the extreme data-scarce regime. Calibrating using explanations is an effective way of using few extra data points that cannot fit in the prompt, which a pitfall of standard in-context learning.
Finally, ROBERTA finetuned using 128 shots only achieves an accuracy of 54.9%, lagging the performance of GPT-3 based models. The limited training data size is insufficient for finetuning smaller language models like ROBERTA, but is sufficient for P-E+EXPI.CAL to be effective.
4.4 Calibrating ADVHOTPOT
Setup For the ADVHOTPOT dataset, our calibration takes the form of tuning the confidence scores of the predicted answers to better align them with the correctness of predictions. These confidence scores can be used in a "selective QA" setting (Kamath et al. 2020). where the model can abstain on certain fraction of questions where assigns low confidence its answers. We use the area under coverage-accuraey curve (AUC) to evaluate how well a model is calibrated as in past literature
(Kamath al. 2020 Chen al.) 2021 Zhang et al. 2021 Garg and Moschitti] 202T] Ye and Durrett 2022). The curve plots the average accuracy with varying fractions (coverage) of questions being answered (examples in Figure [5. For any given coverage, better calibrated model should be able to
identify questions that it performs best on, hence resulting a higher AUC.
We experiment with training data set sizes of 6, 32, and 64. We report the results averaged from trials using different training sets. For ADVHOTPOT, we calibrate E-P, which is shown to be more effective than P-E in this setting (Section 2.4). Our approach also effective for calibrating P-E; please refer to Appendix E for details.
We show the AUC scores in Table 4 By leveraging explanations. E-P+EXPLCAL success- fully achieves an AUC of 68.8, surpassing both FEW-SHOT by 7 points and E-P by 4 points. We note this is substantial improvement, given that the upperbound of AUC is constrained by the accuracy of answers and cannot reach 100. Figure Shoows the coverage-accuracy curves of various methods averaged across the 5 training runs. E-P+EXPI..C. uncalibrated counterpart, E-P, under a certain coverage, and the gap is especially large in the most confident intervals (coverage 50%). E- E-P+ZHANG ET AL. 12021) is able to calibrate the predictions on this dataset, but still lags our explanation- based calibrator, E-P+EXPLCAL.
Results
always achieves a higher accuracy than its
In addition, the explanation-bast based calibrator can be effective with as few as 32 examples. This is because there are only two parameters (the probability of predicted answer and the explanation- based factor) in the calibrator, which can be easily learned in this few-shot setting. Comparing E-P+EXPL.CC CAL against FNW-SHOT(NN), using nearest neighbors
the prompt is also able to improve the performance compared to using a fixed set of shots (FEW-SHOT) yet our lightweight calibrator can better utilize such a small amount of data, and learn to distinguish more accurate predictions based on the explanations.

---

5
Related Work
Our investigation is centered around in-context learning (Brown al. 2020), which has garnered increasing interest since the breakthrough of various large pretrained language models. Recent work has been devoted to studying different aspects of in-context learning, including its wayward behaviors (Min et al., 2022; Webson and Pavlick 2022) and approaches to overcome them (Zhao et al. (021). whereas our exploration focuses on using explanations.
The utility of explanations for few-shot in-context learning has also been discussed concurrently (ye et al. 2021] Wei et al. 2022 Marasović et al. 2022; Chowdhery et al.) [022] Lampinen et al.) 2022; Wiegreffe et al. 20022, especially in symbolic reasoning tasks. We differ in that we study more free-form explanations in tasks (QA and NLI, specifically) focusing on textual reasoning over provided contexts. Furthermore, our work focuses on the nature of the explanations generated by LLMs, which are found to be unreliable. Regarding our use of calibration, similar ideas of explanation-base
performance estimation have been applied to other tasks (Rajani and Mooney 2018; Ye al., 202T] Ye and Durrett 2022). but we rely on the free-text explanations generated by the model instead of interpretations obtained through post-hoc interpretation techniques.
More broadly, how use explanations in various forms (textual explanation, highlights, etc.) train better models is a longstanding problem (Zaidan et al. 2007). Past work has built a series of pipeline models that first generate the explanations and then make predictions purely based on the generated explanations (Wiegreffe et al., | Zhou and Tan 2021 Chen et al. 2022). Prior research has also explored using explanations as additional supervision to train joint models (Hancock et al. 2018 Dua et al. 2020; Lamm et al. [02T]] (Stacyy et al., 2022). Another line of work seeks to align the reasoning process of a trained model with the explanations, which typically done by interpreting a prediction post-hoc through explanation techniques and optimizing the distance between the obtained explanation and ground truth explanation (Liu and Avei 2019; Rieger et al. 2020] Plumb et al., 2020) Erion et al.] [021 Yao et al. (202T) These aforementioned methods all update the model parameters and typically require a considerable amount of explanation annotations to be effective. By contrast, our setting treats language models pure black boxes and only requires few-shot explanations.
6 Discussion & Conclusion
Caveats and Risks of Explanations from Large Language Models Our analysis suggests that LLMs' internal "reasoning" does not always align with explanations that it generates, as shown by our consistency results. More concerning, the explanations might not be factually grounded in the provided prompt. This shortcoming should caution against any deployment of this technology in practice: because the explanations are grammatical English and look very convincing, they may deceive users into believing the system's 's responses even when those responses are incorrect. Section 6 of Bender et al. (2021) discusses these risks in additional detail. The fact that language models can hallucinate explanations is also found in other work (Zhou and Tan [20T). This result is unsurprising in some sense: without sufficient supervision or grounding, language models do not learn meaning as distinct from form (Bender and Koller 2020), so we should not expect their explanations to be strongly grounded.
We have shown that even explanations which don't lead to accuracy gains can still be useful for calibration. However, the lexical overlap feature we use here is a weak signal of explanation correctness (see the example in Figure D). Strong enough entailment models should theoretically be able to perform this task and work across a range of tasks without fine-tuning. This explanation assessment model can even be language model itself trained for this particular propose to approach the verification tasks for a given domain by in in-context learning.
Conclusion We have explored the capabilities of LLMs in using explanations in in-context learning for textual reasoning. Through our experiments with four LLMs and on two QA datasets and an NLI dataset, we find that simply including explanations in the prompt does not always improve the performance of in-context learning. Our manual analysis demonstrates that LLMs tend to generate nonfactual explanations when making wrong predictions, which can be a useful leverage to assess the correctness of the predictions. Lastly, we showcase how to use explanations to build lightweight calibrators, which successfully improve InstructGPT's in-context learning performance across all three datasets.
10

---

Acknowledgments
We would like to thank Eunsol Choi, Ruiqi Zhong, Jocelyn Chen, Zayne Sprague, and Jiacheng Xu for their helpful feedback drafts of this work, as well as the anonymous reviewers for their thoughtful reviews. This work was partially supported by NSF Grant IIS-1814522, NSF CAREER Award IIS-2145280, a grant from Open Philanthropy, gift from Salesforce Inc., and gift from Adobe.
References
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2027 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Emily M. Bender and Alexander Koller. 2020, Climbing towards NLU: On meaning, form, and understanding in the age data. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Mlerbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the Conference Advances in Neural Information Processing Systems (NeurIPS).
Jannis Bulian, Christian Buck, Wojeiech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. arXiv preprint
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In Proceedings of the Conference Advances in Neural Information Processing Systems (NeurIPS).
Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. 2022. Can rationalization improve robustness? In Proceedings of the Annual Conference of the Association for Computational
Linguistics (ACL).
Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI models verify QA systems' predictions? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-IILT).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Baindoor Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope. James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaIM:: Scaling Language
Modeling with Pathways. ArXiv, abs/2204.02311.

---

Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: benchmark to evaluate rationalized NLP models. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACI.).
Dheeru Dua, Sameer Singh, and Matt Gardner. 2020. Benefits of intermediate annotations in reading comprehension. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee, 2021. Improv- ing performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7):620-63 631.
Siddhant Garg and Alessandro Moschitti. 2021. Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies Transactions of the Association for Computational Linguistics, 9:346-361.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML).
Braden Hancock, Paroma Varma, Stephanie Wang. Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the Annual Conference of the Association for Computationaal Linguistics (ACL).
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced enhanced BERT with Disentangled Attention. In Proceedings of the International Conference on Learning Representations (ICLR).
Alon Jacovi and Yoav Goldberg. 2021. Aligning faithful interpretations with their social attribution Transactions of the Association for Computational Linguistics (TACL), 9:294-310.
Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shorteuts: Adversarial evaluation, training. and model development for multi-hop QA. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Amita Kamath, Robin Jia, and Perey Liang. 2020. Selective question answering under domain shift. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.
Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. QED: A framework and dataset for explanations in question answering Transactions of the Association for Computational Linguistics (TACL), 9:790-806.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James I. McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2004.02329.
Frederick Liu and Besim Avei. 2019. Incorporating priors with feature attribution on text classification. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL.).
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What Makes Good In-Context Examples for GPT-3? ArXiv, abs/2101.06804
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692.
Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E. Peters. 2022. Few-shot self-rationalizzatin with natural language prompts. In Findings of the North American Chapter of the Association for Computational Linguistics (NAACL Findings).
12

---

Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversa tional agents' overconfidence through linguistic calibration Transactions of the Association for Computational Linguistics, 07:25-8772.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 20:::22722./.77.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. ArXiv, abs/2112.0011.
John Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74.
Gregory Plumb, Maruan Al-Shedivat, Ângel Alexander Cabrera, Adam Perer, Eric Xing, and Ameet Talwalkar. 2020. Regularizing black-box models for improved interpretability. In Proceedings of the Conference Advances in Neural Information Processing Systems (NeurIPS).
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Nazneen Fatema Rajani and Raymond Mooney. 2018. Stacking with auxiliary features for visual question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), New Orleans, Louisiana.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I TTrust You?": Explaining the Predictions of Any Classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. 2020. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In Proceedings of the International Conference on Machine Learning (ICML).
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks; Visualising image classification models and saliency maps] In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.
Joe Stacey, Yonatan Belinkov, and Marek Rei. 2022. Supervising model attention with human explanations for robust natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML).
Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT),
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
13

---

Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards Al-Complete Question Answering: A Set of Prerequisite Toy Tasks. In Proceedings of the International Conference on Learning Representations (ICIR).
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing Human-Al Collaboration for Generating Free-Text Explanations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning, 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)..
Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. 2021. Refining language models with compositional explanations. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
Xi Ye and Greg Durrett. 2022. Can Explanations Useful for Calibrating Black Box Models? In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Xi Ye, Rohan Nair, and Greg Durrett. 2021. Connecting attributions and QA model behavior on realistic counterfactuals. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator rationales" to improve ma- chine learning for text categorization. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. Findings of the Annual Conference of the Association for Compulational Linguistics (ACL Findings).
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. ArXiv, abs/2205.01068.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Im- proving few-shot performance of language models. In Proceedings of the International Conference on Learning Representations (ICIR).
Yangqiaoyu Zhou and Chenhao Tan. 2021. Investigating the effect of natural language explanations on distrr--iittibttoon generalization in few-shot NLI. In Proceedings of the Workshop on Insights from Negative Results in NLP.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
14

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We use the GPT-3 Instruct-series API (text-davinci-001)
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See reference (Jiang and Bansal 2019) and (Camburu et al. 2018).
(b) Did you mention the license of the assets? [Yes] See Section 2.1
(c) Did you include any assets either in the supplemental material URL? [Yes] We included the Synthetic dataset in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "rC5W4rhbJLz", "reviewer": "Reviewer_67Ay", "summary": "This paper aims to investigate different aspects of explanation-enhanced few-shot in-context learning, especially under textual reasoning scenarios with the task of question answering and natural language inference. The paper mainly discusses: 1) whether the explanation fed into the prompt with P-E or E-P format can help the model improve its accuracy on downstream tasks, 2) whether the explanation is consistent with its prediction, 3) whether the explanation itself is factual enough to be trusted, 4) whether we can use the explanation to help the model calibrate its confidence in prediction. The paper performs experiments on three datasets, one synthetic dataset plus two existing annotated datasets. Through the paper, the authors claim that: 1) the explanation can mildly help the accuracy, 2) the explanation is mostly consistent with the prediction, 3) explanations are quite unfaithful, causing significant hallucination, 4) using the explanation to calibrate the model could further help the model.", "questions": "Did you also test on the different math-based datasets like MultiArith, GSM8K to evaluate their factuality and consistency, etc?", "limitations": "The limitation of this paper is mainly in its deficiency of experiments to support its claims. First of all, the paper only test two tasks, namely question answering and natural language inference. Furthermore, for each of these two tasks, the authors only pick one human-annotated dataset. I think this might not be sufficient to draw a more general conclusion about LLM/GPT-3. I would hope to see more tasks, even many synthesized tasks would help in better understanding LLM's explanations. \n\nThe other limitation is that the paper doesn't provide enough detail about its prompt engineering. According to the \"Large Language Models are Zero-Shot Reasoners\", picking a correct prompt could totally change the landscape. I'm afraid that the lack of comprehensive prompt engineering will make the claims weaker. For the experiment result in Table-1 E-P E-SNLI, I'm not sure whether better prompt engineering or Davinci-02 will make the number totally different. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. the paper builds on top of the recent breakthrough of explanation-based in-context few-shot learning (Chain-of-thoughts) to investigate its reliability issue.\n2. the paper investigates a highly important problem in large language model, which could contribute a lot to the community to influence the follow-up research.\n3. the paper dives deeper to exploit the issue of explanation and propose explanation-based calibration methods.\n\nWeakness:\n(See the Limitation Section)\nWithout sufficient experiments (synthetic + human-dataset across different tasks), it's hard to draw a more convincing conclusion. On the other hand, re-running the experiments on Davinci-002 will help tease out the confounding factor of prompt/model. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K8GZ4Ryirl8", "reviewer": "Reviewer_NkfS", "summary": "The authors suggest that conditioning on and generating explanations of a task or problem yields only minimal gains over raw in-context learning, but that the validity of such explanations may be correlated with the accuracy of answers the model produces. They use this hypothesis to motivate approximating reliability automatically as a method of calibrating in-context learning.\n\nThree datasets are used. One is an automatically generated synthetic dataset that is engineered so that automatically testing the reliability of explanations is possible, at least in theory. One is an adversarial subset of a previous QA dataset (HotPotQA) that is balanced so that GPT-3’s default  performance is split 50/50 between correct and incorrect answers. The third is a Natural Language Inference (NLI) dataset with human-annotated explanations.\n\nThe authors consider two methods for using explanations: explain-then-predict and predict-then-explain. In the former, the model first provides reasoning and then makes a prediction, with past work making the claim that this explanation helps the model predict more accurately. In the latter, the model predicts then explains, so that its explanation does not directly impact the prediction, but the explanations in the few-shot demonstrations still impact the final prediction. Varying number of few-shot demonstrations are used, as can fit in GPT-3’s context window.\n\nThe results show only mild gains from using explanations on these benchmarks (which were specifically intended to probe the use of these explanations and thus may not be representative of the average few-shot learning case). The comparison to previous work is narrativized on lines 123-129, but evidence is not given for whether this narrative holds, which is makes it difficult to assess.\n\nFactuality and consistency are inspectd. For the synthetic dataset, rules are used to judge consistency, but this rule system is not described, even in the appendices. Again, while it is possible these rules are sound, it seems there are many potential pitfalls and not describing these rules makes their validity impossible to assess. For the other datasets, the authors manually annotate 100 of the same examples to report an impressively high annotator agreement. The results suggest that non-factual explanations indicate incorrect prediction, as a rule of thumb. Reporting of results is somewhat inconsistent: on E-SNLI E-P is not reported  and factuality is not judged, though the authors give some reasoning for both of these. Not reporting the E-P strategy because results are so poor still seems strange.\n\nNext the authors attempt to calibrate the model, using a handful of extra data and human judgements. This is especially interesting, as the authors are more-or-less using the full context window available to GPT-3 so these examples could not have been used as in-context demonstrations. A method for calibrating factuality on each dataset is proposed. While sensible, these methods are dataset specific and somewhat adhoc, with no evidence given that they actually correlate with the factuality annotations reported in Table 2.\n\nDespite suggesting that E-SNLI can’t be assessed as factual/non-factual the authors attempt to calibrate E-SNLI using factuality, without discussion of how this works other than using “an analogous score following the same principle” where they consider the premise of the NLI inference as the context. If this is how calibration happens, why couldn’t this have been done when annotating E-SNLI for factuality?\n\nAll the calibration methods are shown to improve the accuracy of the results.\n\nRelated work is reviewed. A clear discussion of the potential risks of explanations given by models, especially trusting such explanations too much, is presented.", "questions": "- Why wasn’t the pseudo-factuality setup used for calibrating E-SNLI factuality also used to annotate E-SNLI?\n- What automatic rules were used for analyzing the factuality and consistency of the synthetic dataset?\n- Why weren’t any standard few-shot learning datasets considered, especially ones where previous work found explanation generation helpful?", "limitations": "~~Authors have adequately discussed the limitations of their work.~~ Edit: Given the new framing, it will be very important for authors to revise the paper to make it clear how the available datasets limit the claims that can be drawn from this work, as the types of textual reasoning studied are rather limited.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The research questions are well-motivated: despite impressive performance on selected datasets it is unclear how much or why explanations help models.\n- Evidence given that factuality is correlated with consistency and with non-factuality with incorrect prediction is an interesting result.\n- Exploiting extra data that can’t be put into the context window of a model for calibration is potentially a useful technique for many tasks.\n- The calibration shows seemingly meaningful performance improvements (though on highly specific and sometimes custom datasets)\n\nWeaknesses\n\n- ~~Only one version of GPT-3 is considered—GPT-3 Instruct, which is finetuned on human feedback, making the results somewhat less comparable to other work.~~ Edit: Authors have added significant new results.\n- Result reporting is selective—on E-SNLI factuality is not reported, the E-P method is not assessed,  and Table 3 is missing many values. Edit: Authors have provided reasoning as to why some numbers were not included though in some cases (e.g. the expense of calculating those number), this is still a weakness, albeit a less intense one.\n- While the bespoke datasets for looking at this hypothesis are useful tools, it is unclear what these results mean for tasks where few-shot learning is commonly used. Edit: the authors have revises their framing in a way that partially alleviates this, focusing on textual reasoning. However, the available datasets still comprise a tiny fraction of textual reasoning datasets.\n- ~~The comparison to previous work is described, but evidence is not directly given. (Lines 123-129)~~ Edit: the authors have specified a new framing that makes this comparison significantly more valid.\n- ~~The automatic rules for evaluating factuality and consistency are not described, even in the appendices.~~ Edit: Authors have added a description of their synthetic task.\n- ~~Despite suggesting that E-SNLI can’t be assessed as factual/non-factual the authors attempt to calibrate E-SNLI using factuality, without discussion of how this works other than using “an analogous score following the same principle”.~~ Edit: Authors have described why they can heuristically calibrate factuality without being able to assess it.\n- ~~The calibration methods are methods are dataset specific and somewhat adhoc, with no evidence given that they actually correlate with the factuality annotations reported in Table 2~~ Edit: Authors have provided reasoning why their calibration methods should correlate with factuality annotations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9vzRj9H6mTo", "reviewer": "Reviewer_5YVG", "summary": "This paper explored the capabilities of GPT-3 in using explanations in in-context learning. Although using in-context explanation has been demonstrated quite helpful for symbolic reasoning tasks, this paper finds that it's not the case for textual reasoning tasks such as NLI. Upon investigation, they find that simply including an explanation as in-context example does not always yield big improvements. Their analysis shows that this is because the explanations generated by GPT3 could be nonfactual. Given those observations, they showcase how to use in-context explanation to calibrate GPT3's prediction. The proposed method achieves significant improvements over three datasets.", "questions": "Why there are so many missing cells in Table3&4? Will those results affect the conclusion?", "limitations": "The authors did not address the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The observation in this paper is interesting and could be a good complement to current in-context explanation research. \nThe proposed explanation-based calibration method is simple yet effective. \nThe paper is clear and easy to follow.\n\nHowever, I also have the following concerns. The claim that incorporating explanation as in-context examples only marginally improves performance is a strong claim that contradicts prior works. To back up the claim, I suggest the author include a few more datasets in table 1. Besides, for experiments in S2, only 250 testing examples are considered. Can authors justify their choice, as compared to using the official test set? Since on such a small test set, the conclusion is not that convincing. \nBesides, for results in Table 1, I do not agree that such improvements (10% relative improvements) are \"mild\". They are just not that surprising compared to some other tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept