Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors suggest that conditioning on and generating explanations of a task or problem yields only minimal gains over raw in-context learning, but that the validity of such explanations may be correlated with the accuracy of answers the model produces. They use this hypothesis to motivate approximating reliability automatically as a method of calibrating in-context learning.

Three datasets are used. One is an automatically generated synthetic dataset that is engineered so that automatically testing the reliability of explanations is possible, at least in theory. One is an adversarial subset of a previous QA dataset (HotPotQA) that is balanced so that GPT-3’s default  performance is split 50/50 between correct and incorrect answers. The third is a Natural Language Inference (NLI) dataset with human-annotated explanations.

The authors consider two methods for using explanations: explain-then-predict and predict-then-explain. In the former, the model first provides reasoning and then makes a prediction, with past work making the claim that this explanation helps the model predict more accurately. In the latter, the model predicts then explains, so that its explanation does not directly impact the prediction, but the explanations in the few-shot demonstrations still impact the final prediction. Varying number of few-shot demonstrations are used, as can fit in GPT-3’s context window.

The results show only mild gains from using explanations on these benchmarks (which were specifically intended to probe the use of these explanations and thus may not be representative of the average few-shot learning case). The comparison to previous work is narrativized on lines 123-129, but evidence is not given for whether this narrative holds, which is makes it difficult to assess.

Factuality and consistency are inspectd. For the synthetic dataset, rules are used to judge consistency, but this rule system is not described, even in the appendices. Again, while it is possible these rules are sound, it seems there are many potential pitfalls and not describing these rules makes their validity impossible to assess. For the other datasets, the authors manually annotate 100 of the same examples to report an impressively high annotator agreement. The results suggest that non-factual explanations indicate incorrect prediction, as a rule of thumb. Reporting of results is somewhat inconsistent: on E-SNLI E-P is not reported  and factuality is not judged, though the authors give some reasoning for both of these. Not reporting the E-P strategy because results are so poor still seems strange.

Next the authors attempt to calibrate the model, using a handful of extra data and human judgements. This is especially interesting, as the authors are more-or-less using the full context window available to GPT-3 so these examples could not have been used as in-context demonstrations. A method for calibrating factuality on each dataset is proposed. While sensible, these methods are dataset specific and somewhat adhoc, with no evidence given that they actually correlate with the factuality annotations reported in Table 2.

Despite suggesting that E-SNLI can’t be assessed as factual/non-factual the authors attempt to calibrate E-SNLI using factuality, without discussion of how this works other than using “an analogous score following the same principle” where they consider the premise of the NLI inference as the context. If this is how calibration happens, why couldn’t this have been done when annotating E-SNLI for factuality?

All the calibration methods are shown to improve the accuracy of the results.

Related work is reviewed. A clear discussion of the potential risks of explanations given by models, especially trusting such explanations too much, is presented.
strengths_and_weaknesses: Strengths

- The research questions are well-motivated: despite impressive performance on selected datasets it is unclear how much or why explanations help models.
- Evidence given that factuality is correlated with consistency and with non-factuality with incorrect prediction is an interesting result.
- Exploiting extra data that can’t be put into the context window of a model for calibration is potentially a useful technique for many tasks.
- The calibration shows seemingly meaningful performance improvements (though on highly specific and sometimes custom datasets)

Weaknesses

- ~~Only one version of GPT-3 is considered—GPT-3 Instruct, which is finetuned on human feedback, making the results somewhat less comparable to other work.~~ Edit: Authors have added significant new results.
- Result reporting is selective—on E-SNLI factuality is not reported, the E-P method is not assessed,  and Table 3 is missing many values. Edit: Authors have provided reasoning as to why some numbers were not included though in some cases (e.g. the expense of calculating those number), this is still a weakness, albeit a less intense one.
- While the bespoke datasets for looking at this hypothesis are useful tools, it is unclear what these results mean for tasks where few-shot learning is commonly used. Edit: the authors have revises their framing in a way that partially alleviates this, focusing on textual reasoning. However, the available datasets still comprise a tiny fraction of textual reasoning datasets.
- ~~The comparison to previous work is described, but evidence is not directly given. (Lines 123-129)~~ Edit: the authors have specified a new framing that makes this comparison significantly more valid.
- ~~The automatic rules for evaluating factuality and consistency are not described, even in the appendices.~~ Edit: Authors have added a description of their synthetic task.
- ~~Despite suggesting that E-SNLI can’t be assessed as factual/non-factual the authors attempt to calibrate E-SNLI using factuality, without discussion of how this works other than using “an analogous score following the same principle”.~~ Edit: Authors have described why they can heuristically calibrate factuality without being able to assess it.
- ~~The calibration methods are methods are dataset specific and somewhat adhoc, with no evidence given that they actually correlate with the factuality annotations reported in Table 2~~ Edit: Authors have provided reasoning why their calibration methods should correlate with factuality annotations.
questions: - Why wasn’t the pseudo-factuality setup used for calibrating E-SNLI factuality also used to annotate E-SNLI?
- What automatic rules were used for analyzing the factuality and consistency of the synthetic dataset?
- Why weren’t any standard few-shot learning datasets considered, especially ones where previous work found explanation generation helpful?
limitations: ~~Authors have adequately discussed the limitations of their work.~~ Edit: Given the new framing, it will be very important for authors to revise the paper to make it clear how the available datasets limit the claims that can be drawn from this work, as the types of textual reasoning studied are rather limited.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to investigate different aspects of explanation-enhanced few-shot in-context learning, especially under textual reasoning scenarios with the task of question answering and natural language inference. The paper mainly discusses: 1) whether the explanation fed into the prompt with P-E or E-P format can help the model improve its accuracy on downstream tasks, 2) whether the explanation is consistent with its prediction, 3) whether the explanation itself is factual enough to be trusted, 4) whether we can use the explanation to help the model calibrate its confidence in prediction. The paper performs experiments on three datasets, one synthetic dataset plus two existing annotated datasets. Through the paper, the authors claim that: 1) the explanation can mildly help the accuracy, 2) the explanation is mostly consistent with the prediction, 3) explanations are quite unfaithful, causing significant hallucination, 4) using the explanation to calibrate the model could further help the model.
strengths_and_weaknesses: Strength:
1. the paper builds on top of the recent breakthrough of explanation-based in-context few-shot learning (Chain-of-thoughts) to investigate its reliability issue.
2. the paper investigates a highly important problem in large language model, which could contribute a lot to the community to influence the follow-up research.
3. the paper dives deeper to exploit the issue of explanation and propose explanation-based calibration methods.

Weakness:
(See the Limitation Section)
Without sufficient experiments (synthetic + human-dataset across different tasks), it's hard to draw a more convincing conclusion. On the other hand, re-running the experiments on Davinci-002 will help tease out the confounding factor of prompt/model. 


questions: Did you also test on the different math-based datasets like MultiArith, GSM8K to evaluate their factuality and consistency, etc?
limitations: The limitation of this paper is mainly in its deficiency of experiments to support its claims. First of all, the paper only test two tasks, namely question answering and natural language inference. Furthermore, for each of these two tasks, the authors only pick one human-annotated dataset. I think this might not be sufficient to draw a more general conclusion about LLM/GPT-3. I would hope to see more tasks, even many synthesized tasks would help in better understanding LLM's explanations. 

The other limitation is that the paper doesn't provide enough detail about its prompt engineering. According to the "Large Language Models are Zero-Shot Reasoners", picking a correct prompt could totally change the landscape. I'm afraid that the lack of comprehensive prompt engineering will make the claims weaker. For the experiment result in Table-1 E-P E-SNLI, I'm not sure whether better prompt engineering or Davinci-02 will make the number totally different.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explored the capabilities of GPT-3 in using explanations in in-context learning. Although using in-context explanation has been demonstrated quite helpful for symbolic reasoning tasks, this paper finds that it's not the case for textual reasoning tasks such as NLI. Upon investigation, they find that simply including an explanation as in-context example does not always yield big improvements. Their analysis shows that this is because the explanations generated by GPT3 could be nonfactual. Given those observations, they showcase how to use in-context explanation to calibrate GPT3's prediction. The proposed method achieves significant improvements over three datasets.
strengths_and_weaknesses: The observation in this paper is interesting and could be a good complement to current in-context explanation research. 
The proposed explanation-based calibration method is simple yet effective. 
The paper is clear and easy to follow.

However, I also have the following concerns. The claim that incorporating explanation as in-context examples only marginally improves performance is a strong claim that contradicts prior works. To back up the claim, I suggest the author include a few more datasets in table 1. Besides, for experiments in S2, only 250 testing examples are considered. Can authors justify their choice, as compared to using the official test set? Since on such a small test set, the conclusion is not that convincing. 
Besides, for results in Table 1, I do not agree that such improvements (10% relative improvements) are "mild". They are just not that surprising compared to some other tasks. 
questions: Why there are so many missing cells in Table3&4? Will those results affect the conclusion?
limitations: The authors did not address the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The authors perform an analysis that suggests that explanations may not provide reliable signal in few-shot in-context learning, showing that adding explanations yields only minimal gains over raw in-context learning. They then develop an approach to approximate the reliability of predictions automatically using these explanations.

In the initial reviews, the reviewers pointed out issues with the empirical rigor of the study and its framing. However, they seem to have addressed these concerns by narrowing the scope of their contributions and providing additional experiments supporting their claims.
confidence: Less certain
Decision: Accept