Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces Chain of Thought (CoT) prompting, which augments few-shot training samples for in-context learning with explicit reasoning steps. They evaluate the effectiveness of CoT on math word problems, commonsense reasoning and symbolic reasoning and shows performance improvements from the standard prompting as well as fine-tuned GPT-3 with smaller numner of parameters. In addition, a 540B parameter language model prompted with CoT outperforms fine-tuned 175B GPT-3. 
strengths_and_weaknesses: ## Strengths
- The proposed idea is easy, simple, (potentially) widely-applicable and well-motivated.
- They show that CoT gives large performance improvements when the LMs are large enough, although performance improvements are somewheat limited when the base LMs are smaller than 100B. 
- Their through analysis shows that (i) in many cases the models' generated explanations are correct, (ii) removing sub part of CoT or changing the answer / CoT order largely degrades models' performance, indicating the necessity of using the reasoning process writte in natural languages, and (iii) CoT is robust to the different expalanation written by different annotations (Author A, B and C). 

## Weaknesses
- Although the performance improvements given by CoT is impressive when the LMs are big enough, with the models smaller than 175B, CoT prompting does not provide sigifincat improvements from the standard prompting. Many groups cannot afford to use 175B or larger models, which may make the aplicability and usefulness of CoT limited. Moreover, the evaluations are done in mostly relatively limited domains (e.g., mathword problem). Recent work (Ye and Durret, 2022)* shows that explanations generated by GPT-3 may not even be factually grounded in the input, even on simple tasks with straightforward extractive explanations. 
- All of the models used in this papers are only particially publically available (GPT-3 APIs) or completley publically unavailable (LM1 and LM2), which makes it difficult to verify the experimental results and findings in this paper.   

======= 
[2022/08/08] I updated my score given the new results and dicussions with the authors. 

*Ye, Xi, and Greg Durrett. "The Unreliability of Explanations in Few-Shot In-Context Learning." arXiv preprint arXiv:2205.03401 (2022).
questions: - Is it possible to test CoT with the publiclly available models with the same scale (e.g., Meta OPT)? 
limitations: The authors mention several limitations of CoT (e.g., writing CoT prompts).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes chain-of-thought prompts for in-context few-shot learning. The authors found that with properly designed prompts, which are decomposed reasoning sub-procedures, large language models can achieve substantially higher performances on the arithmetic reasoning, commonsense reasoning, and symbolic reasoning tasks that are tested in the experiments. For many tasks studied in this paper, the performance exceeds that of the supervised finetuning methods.  Analyses show that the chain-of-thought prompts significantly improve few-shot performance compared to the standard prompts. The performance of chain-of-thought prompts has been shown to be robust in different perspectives (e.g., different annotators and various language models). 
strengths_and_weaknesses: Strength:

-	The proposed model that incorporates chain of thought into prompts achieved substantial and consistent improvements on different tasks with different language models. Although the originality of the paper is a bit of concern, I do think these results are valuable to the community.
-	The paper performs comprehensive and detailed analysis to help establish the claims and draw the conclusions with solid evidence and justification. The detailed analyses provide insights and interesting discussions on the property of the model, e.g., the property with regard to language model sizes and complexities of the target problems, which may invite more studies along the line (although unfortunately, many of such studies may only be conducted in a limited number of organizations).
-	The paper is clearly written and I like the detailed discussions in the FAQ session, which further addressed my questions.

Weakness:

-	The originality and novelty, particularly from the methodology perspective, is limited. (However, due the above strengths, I would think the paper’s overall contributions outweighs this limitation.)
-	In terms of generalizability, the paper claims that the proposed approach is applicable in principle to any tasks that humans can solve via language. This is not well supported in the paper and may need further justification. (The contributions of the paper, though, are not dependent on that claim.). For example, the paper mentioned that spurious paths (wrong reasoning steps leading to correct answers) seem to be less of a concern in some of the datasets used in this paper, but that problem could be more serious in other complicated reasoning tasks. One may think that the reasoning problems that need more expressive/powerful model for commonsense may put challenges to the proposed approach.

questions: -	When annotator A, B, and C wrote chain of thought, what’re their strategies? 

-	The paper mentioned that one contribution of the proposed model is facilitating interpretation and debugging of the few-shot learning models. Can the authors add some discussion on how to perform the debug?


limitations: please refer to the weakness comments.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed chain of thought prompting, which can significantly improve the ability of large language models to perform complex reasoning. The proposed method is simple and effective, and also can generalize to various arithmetic, commonsense, and symbolic reasoning tasks. 
strengths_and_weaknesses: The paper is well motivated -- scaling up model size alone has not proved sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning.

The proposed method – the chain of thought prompting – is novel, simple, and effective. The prompts help large language models to generate intermediate natural language reasoning steps that lead to the final output. 

The experiments demonstrated the chain of thought prompting could generalize to various arithmetic, commonsense, and symbolic reasoning tasks.

Overall, there is no apparent weaknesses found in the paper. I champion for accepting this paper.

questions: n/a
limitations: n/a
review_title: null

Meta-review: metareview: All reviewers have voted to accept the paper, and this is a solid work.
confidence: Certain
Decision: Accept