Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper compares different chain-of-thought (CoT) prompting methods proposed recently and shows their limitations via case studies. The paper further proposes a new automatic chain-of-thought prompting method, Auto-CoT. Auto-CoT samples questions from representative question clusters and generates reasoning chains to construct demonstrations. Results on a wide range of datasets show that it performs competitively compared to Manual-CoT, which requires manual designs.
strength_and_weaknesses: **Strengths**

**1. The tackled problems and proposed solutions are attractive to the community.**

The paper systematically analyzes and compares the strengths and weaknesses of different chain-of-thought prompting methods: including zero-shot-CoT, few-shot-CoT, manual-CoT, and Auto-CoT. The paper conducts case study experiments to look into the limitations of existing methods and proposes improvement directions. Finally, the paper proposes an improved method for Auto-CoT that could achieve a competitive advantage over Manual-CoT. 

**2. The experiment part is very detailed and comprehensive.**

**3. The paper is well-organized.**

The writing is good and most of the content is very clear to me.

**Weaknesses/Feedback**

**1. The writing could be improved.**

It would be helpful to draw a table to compare different CoT prompting methods across different dimensions.

How and why shall we make an assumption that “questions of all the wrong demonstrations fall into the same frequent-error cluster”?

Is the selection criteria in section 4.2 reasonable? Namely, why do we not choose questions with more than 60 tokens and rationales with more than 5 reasoning steps?


**2. Some experimental details are missing.** 

The experimental details for Table 1 are not very clear and lack some details. For example, how the manual-CoT examples are built. What is the number of demonstration examples?

The experimental details for the Codex baseline are missing. I am curious about the instructions you used to prompt the Codex model.


**3. It would be better to discuss more recent related work.** 

I understand some work has been released very recently. Since this work is closely related to the paper, it would be nice to include it in the revised paper. Recent work includes but is not limited to:
[1] Calibrate Before Use: Improving Few-Shot Performance of Language Models, 2021
[2] Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning, 2022
[3] Complexity-Based Prompting for Multi-Step Reasoning, 2022
[4] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, 2022
[5] What Makes Pre-trained Language Models Better Zero/Few-shot Learners?, 2022

**4. Is it possible to apply Auto-CoT to sample the examples with manually designed CoT?** 

clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and easy to follow. The paper proposes a new automatic method to select demonstration examples. The main concern, from my point of view, is that I am not sure if there is enough modeling novelty.

summary_of_the_review: The paper is overall high quality. The proposed method is beneficial to the community in developing more performant few-shot GPT-3 models. The experiments are comprehensive and the proposed model achieves competitive results on various datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for automatically constructing demonstrations for prompting large language models to perform complex reasoning. The proposed method consists of two main steps. First, partition questions of a given dataset into a few clusters. Second, select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics. The proposed method is evaluated on ten benchmark reasoning tasks from arithmetic reasoning, commonsense reasoning, and symbolic reasoning. Experimental results show that the proposed method performs competitively compared to Manual-CoT which requires manual designs.

strength_and_weaknesses: Strengths

- The proposed method is simple yet effective. It is intuitive to implement the proposed method on different tasks.
- The analysis of failure cases of Retrieval-Q-CoT is helpful in understanding the proposed method.

Weaknesses

- Would it be possible to get the best of both worlds by combining Retrieval-Q-CoT and Random-Q-CoT? One can sample tasks similar to the target task, while also sampling questions from other clusters.
- During question sampling the algorithm also has access to answers, and the chain of thoughts. It would be interesting to see if clustering with CoT / Q-A pairs improves the results. 

clarity,_quality,_novelty_and_reproducibility: Clarity
- The paper is well written and easy to follow. The main paper and the appendix are well-organized and things are written in the right granularity of details.

Novelty
- The proposed method is novel, and combines the best of Zero-shot CoT(kojima et al) and manual CoT(wei et al). It is a great first step to automatically construct CoT prompts based on simple objectives (diversity). 


Reproducibility

- All the details required to implement this paper is clearly presented. The authors are also committed to release the code.
- The proposed method is only evaluated on GPT-3, and it would be interesting to see if the proposed method also works on smaller language models.


summary_of_the_review: This paper proposes a method for automatically constructing demonstrations for prompting large language models to perform complex reasoning. The proposed method is simple yet effective. It is intuitive to implement the proposed method on different tasks. I would recommend that the authors try their method on smaller models and see if the results still hold. The analysis of failure cases of Retrieval-Q-CoT is helpful in understanding the proposed method. I would recommend that the authors try to combine Retrieval-Q-CoT and Random-Q-CoT to get the best of both worlds, or maybe the proposed demo sampling scheme is strictly better than retrieval? Open to increasing my score if all the raised questions get answered in author's response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for automatically constructing chain-of-thought prompts for multistep reasoning in large language models by using zero-shot chain-of-thought prompting (i.e. prepending "let's think step by step" to the generation) to generate rationales for selected prompt questions, and various heuristics for ensuring diversity and simplicity of the generated questions and rationales, including: only keeping fairly short rationales, selecting diverse prompting questions by partitioning the space of questions via kmeans, etc.

The authors first conduct a detailed analysis of what questions might serve as useful prompts for a specific test question. They compare a retrieval-based method (where similar questions are retrieved from a training dataset) with a random method (where random questions), with rationales generated. They observe that the retrieval-based method actually performs worse, and suggest various reasons why this might be the case (e.g., retrieval encourages selecting hard questions, where errors in demonstrations/answers are propagated to the test question). This motivates them to present their final method, Auto-CoT, which takes the lessons learned from the previous section (namely, that diversity matters) and selects diverse questions with fairly straightforward reasoning steps. This method matches and/or outperforms manually constructed chains of thought across ten datasets from arithmetic to commonsense reasoning and symbolic manipulation tasks. Finally, the authors explore a streaming setting, where questions arrive iteratively. These are interesting results though very preliminary.
strength_and_weaknesses: # Strengths

- This is a fairly straightforward paper that attempts to (broadly) combine strengths of both manual chain-of-thought prompting with zero-shot chain of thought prompting. The idea is (mostly) simple and is likely to be of wide interest to the community.
- Fairly diverse set of results across arithmetic/commonsense/symbolic reasoning, with two LLMs are explored, GPT-3 and Codex, both via the OpenAI API.
-  I like the didactic/exploratory nature of section 3. I like that the paper reads like an investigation, and begins with a failure case, as well as diagnosing reasons for the failure, before then presenting the real method. This feels more organic and the lessons learned along the way are insightful (though I think the analysis is not as straightforward as it could be - see weaknesses)

# Weaknesses

I have one primary hesitation with the paper that prevents me from assigning it a higher score, as well as some secondary (less major) concerns.

## Primary: Lack of understanding of why Auto CoT works

As I mentioned, I like the section 3 analysis which analyzes exactly why fancy retrieval (vs random retrieval) works. But it seems like for Auto CoT, a lot of this nuance is lost, and we don't have a clear understanding of why this method is doing better, especially given that there are some ad-hoc decisions being made here.

If I understand correctly, Auto CoT is similar to Random-Q-CoT except (1) we enforce sampling from diverse clusters of questions and (2) heuristics to filter for simple rationales. Both of these modifications are presented at once in Table 3, without ablation studies, and compared only to Manual CoT. Strangely the Section 3 methods are omitted from this table (Random-Q-CoT, Retrieval-Q-CoT), but I think their inclusion would add a ton of insight into what exactly is causing gains for Auto CoT.

Again I think it is quite important to have some controls/ablations that explain why Auto CoT improves. The baselines I imagine are:

1. How well does the simple random retrieval + zero-shot CoT, i.e. Random-Q-CoT, do in Table 3? Again I think it's strange that this baseline is omitted here.
2. How much are the simple heuristics buying you? I am concerned that the simple heuristics are entirely responsible for the improved performance across tasks (i.e. it's just easier to learn from simpler demonstrations). Appendix C.2 is useful for understanding how heuristics improve Auto CoT but it's done while keeping the kmeans clustering fixed. It also does not measure the ultimate metric we care about, which is downstream task performance (instead focusing on "number of mistakes" in sampled demonstrations) I would love Auto CoT run without kmeans, but with the heuristic filtering turned on.
3. Less important is understanding how crucial/brittle this method is to the kmeans clustering (if the kmeans clustering does indeed end up being important for improved performance), e.g. what happens if you have a larger k, smaller k (perhaps sampling clusters w/ replacement), etc.

Note if heuristic filtering is entirely responsible for the gians here, I think the paper is still interesting (and even more effective because we don't need this somewhat brittle kmeans step) - but it'd be important to disentangle the performance here.

## Secondary: Lack of numerical rigor in some table results

In general, there are some naked numbers presented in tables that could really use some error bars/additional contextualization, especially since the differences between methods are marginal at best, yet the authors make fairly broad conclusions about the relative efficacy of methods. For example, the numbers in Table 1, especially for Random-Q-CoT, are screaming for error bars. For MultiArith, is 86.2 and 82.8 a meaningful difference? How much variance in performance do you get as you sample different random questions for Random-Q-CoT? It seems like the retrieved demonstration questions, and generated rationales, for Retrieval-Q-CoT are deterministic, if I'm not mistaken. But I'm still curious in general about how much variance there is in these results when you sample different. E.g. if you set temperature != 1, how much does performance vary? For GSM8K in Table 1, a difference of 1 percentage point over Manual-CoT seems rather small and is very close to Random Q-CoT (which again presumably has plenty of variance in the number achievable). So is retrieval not doing better than random in this setting? AQuA results seem more convincing.

## Secondary: 3.1 and 3.2 analysis slightly strange

I wrote some additional thoughts about Section 3, but note this is not as essential to my final recommendation:

I think the essential (interesting!) question that this section aims to answer is: "For hard questions, does Q-Retrieval-CoT underperform because it retrieves similarly hard questions that the model can't do well zero-shot?"

I find the analysis in 3.1 and 3.2 interesting but I feel like they don't quite answer this question as directly as they could.

First, in 3.1, the conclusion that *misleading by similarity* is the blanket reason why Retrieval-Q-CoT underperforms across the board feels premature. The example in Table 2 is convincing. But does this mean the authors are claiming that *every* unresolved example in Figure 2 is because all similar retrieved questions have lexical overlap, with phrases (e.g. *the rest*) that the model is encouraged to consistently misunderstand? This is a very narrow failure case. A more general reason of why Retrieval-Q-CoT might underperform in Figure 2 is just that, for hard questions like those in Figure 2, retrieval might just retrieve similarly hard questions that the model gets wrong. I'm not sure if this is the more specific *misleading by similarity* phenomenon authors describe---if it is, it would be nice to clarify the wording in the paper here.

3.2 is also interesting but strikes me as a rather weird way to answer the  question above. Why was k = 8 decided here? Authors have shown that there exists a cluster with a high zero-shot error rate, but does retrieval indeed tend to retrieve questions within this cluster? More straightforwardly, does Q-Retrieval-CoT tend to retrieve questions that the model gets wrong? In other words, the more straightforward analysis in my opinion is to simply look at the percentage of times the model gets the retrieved demonstration questions correct for any given test question. For harder test questions, perhaps retrieved question accuracy is lower for Retrieval-Q-CoT than Random-Q-CoT, because the retrieved questions are harder. And again, unlike 3.1, this finding would make no claim about any kinds of specific (lexical?) errors propagated from the prompt questions to the test question. 

# Minor/questions

- Figure 1 is very helpful for understanding zero shot/manual CoT, but I think the the methods would be much clearer if there were similar visual depictions of how ranadom-q-cot and retrieval-q-cot operate in this figure. The process is not trivial (my understanding: for a test question, retrieve similar questions, generate a rationale via zero-shot CoT, add rationales to prompt, then do standard CoT with the test question)
- Figure 5 would be far more informative if less plots were shown (I don't think you need to visualize the clustering for every single dataset), and instead auhtors focused on a few examples as well as gave concrete examples of what different clusters look like (e.g. in MultiArith, maybe one cluster refers to addition problems, one refers to multiplication, etc.)

# Missing References

This is a matter of principle, but I believe that [Nye et al., 2021](https://arxiv.org/abs/2112.00114) should be cited with any chain of thought prompting work, as it introduced the basic idea prior to it being rebranded "chain of thought"
clarity,_quality,_novelty_and_reproducibility: Clear enough to me.
summary_of_the_review: Overall I think this is an interesting paper with decent results, but I think the strongest possible version of the paper would give us a clear sense of what specifically in the final Auto-CoT model causes increased performance over the simpler variants (Retrieval/Random-Q-CoT). I'm looking forward to the author response and the other reviews and am open to increasing my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Prompting has been an important strategy lately to query large language models (LLMs). Chain of thought (CoT) prompting has demonstrated state-of-the-art on many datasets by forcing the models to generate intermediate steps while solving a problem. Some other manual designs of prompts (for example, appending “let's solve this step by step” to the solution steps) have improved the reasoning capabilities of LLMs on various tasks leading to superior results than normal prompting. However, designing prompts is a manual task often requiring lots of brute force to find a suitable prompt. In this work, the authors propose a strategy to automatically design a chain of prompting prompts known as Auto-CoT. On public benchmarks, Auto-CoT performs competitively to the manual CoT prompts.  
strength_and_weaknesses: ### Strengths: 
- The paper proposes to counter the problem of manually designing prompts for various tasks by first clustering the samples (test) into different clusters using sentenceBERT, finding the most representative from each cluster, and using the Zero-shot-CoT approach on them to design prompts. This method is better than randomly sampling n-samples from the dataset and using Zero-shot-CoT prompts over them. 
- The results of ten tasks demonstrate the effectiveness of the method with very competitive results on each one of them.

### Weaknesses:
- Firstly, when mentioning that the method automatically finds prompts using chain of thoughts (Auto_CoT) approach, this does not imply finding the best samples from the dataset and using the same method as before. In other words, the proposed method does not find the best chain of thought prompts as claimed but rather believes that “let's do this step by step” is the best zero-shot-CoT prompting strategy and the method merely uses that with different prompt samples. The title and the abstract are highly misleading in that way. 
- Secondly, this is very obvious to anyone that the prompts must be representative of the test samples (as much as possible). This work takes the dataset, clusters them, finds the most representative sample from each cluster, and uses that as prompts. There is nothing extra-ordinary about this approach and is merely a small improvement over the randomly sampling strategy. This can also be seen from a marginal improvement in the results over a random sampling strategy. 
- The paper suggests that the manual prompt search will be reduced with this approach but rather introduces a clustering strategy, and a fixed hyper-parameter might or might not be apt for each dataset and requires manual intervention. If the number of clusters go beyond the number of prompts a large LM can handle, I am not sure what can be done in those situations. 
- Finally, if clustering was the way to go, the authors could have tried to cluster samples into n-clusters, see where the test sample is fitting, and could have tried to use just one prompt instead of k different ones at each time. This might have reduced the compute cost and would be interesting to see the performance vs resource trade-off. 

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written and easy to follow. All the results can be reproduced well. 
summary_of_the_review: The paper discusses a way to automatically create prompts which is a very promising direction. However, the work does not target how to create the prompts, rather uses different samples with the previously proposed prompting strategy. The approach used by the authors to cluster the samples together and construct prompts using the most representative samples improves the results marginally, but the bottleneck is not the samples for prompts but how the prompts are constructed. The work does not target that and is just an obvious extension of the other prompting strategies of the past. I would recommend then authors to extend their approach to find suitable prompts and not just suitable samples which is not that effective.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an approach for automatic prompting to elicit chain-of-thought reasoning in large language models without needing manually-designed demonstrations.
All reviewers agreed that the paper addresses an important problem and appreciated the depth of the experiments and analysis. However, some reviewers raised concerns about the lack of support for the claim of automatic chain of thought prompting and the limited improvement provided by the proposed method over manual prompts.
After the discussion among the reviewers, some of these concerns were unresolved. However, after carefully considering the reviews, the meeting between AC and reviewers, and the author response, it does seem like that addressing the second concern is not really the focus of this paper, as manual prompts can be considered as a kind of an upperbound for performance. The first major concern can be addressed by asking the authors to rephrase the "automatic CoT prompting" claim to make things more clear. It does seem like the proposed approach is an automatic "example selection" for zero-shot prompting (Kojima et al., 2022). 

justification_for_why_not_higher_score: There are some issues regarding clarity of the writing and the main claim to be too strong.
justification_for_why_not_lower_score: Please see the detailed AC-Reviewer meeting notes.
Decision: Accept: poster