Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper proposes a new way to improve the reasoning power of language models by decoding several sequences of CoT/Scratchpad reasoning and then using a voting-based scheme to choose an answer. At the cost of having to sample multiple times from the model, the authors show a consistent improvement across tasks and models. Through thorough analysis, the authors show the utility of their method, and its advantages over other sampling techniques and highlight the best practices that might be useful in practical applications.
strength_and_weaknesses: - **Clarity:** The paper is very clearly written and is easy to understand. The takeaways from each section are clear and flow from one to the other.
- The tasks, baseline sampling techniques, models and various tradeoffs while using different hyperparameters have been explored well! They give a clear picture of what is useful in a practical setting.
- **********************************Reproducibility:********************************** The method is easy to implement, but not all the models are publicly available or are accessible only through a paid API.
- The method itself is quite general and can be used across several task types and models.
- Further Discussion: CoT/ Scratchpads are useful in understanding the output of a model; how would the authors suggest choosing a reasoning path to provide an explanation for the answer given by the model?
- ******************************Novelty:****************************** The paper is novel in its approach.
- ******************Quality:****************** The paper, analysis and writing are of a very high quality
clarity,_quality,_novelty_and_reproducibility: Refer to main review.
summary_of_the_review: The paper is extremely strong in almost exhaustively testing different settings and tasks. It has very limited weaknesses and will probably have a large impact on various practical applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper investigates an improved version of Chain-of-thought prompting by leveraging self consistency and shows empirical evidence that their self consistency prompting indeed improve reasoning tasks performance compared with the original chain-of-thoguht prompting.

strength_and_weaknesses: Strength: 
1.	The self-consistency method is very well motivated, and authors did a great job presenting this motivation. The method is very principled.
2.	The empirical evidence of the superiority of the method is extensive and solid.

Weakness: authors’ notation of r_i is a bit unclear to me, my understanding is that they are just the generated text before the final answer (what authors call reasoning path). It is not clear to me what is the difference between the generated text before the final answer and what authors call the reasoning paths. Did authors do something specific to exclude some potentially irrelevant text that were generated but did not contribute to reasoning? If not, perhaps calling them reasoning paths is a bit confusing? It seems r_i and a_i always appear together, which gives me the impression that the only distinction is that a_i is the final answer and r_i is everything before that. 

Another confusion is the presentation of the weighted sum, is it correct to understand the probability weight is just the logsumexp of the logprobs of the generation? That makes sense to be, equations 1 is a bit misleading to me because it seems to place an emphasis on r_i as being the reasoning path when it’s just the probability of a specific generation. I think it is okay to mention the generation is supposed to be reasoning paths, but it might be more clear to just note them as generated text in those formal expressions.

clarity,_quality,_novelty_and_reproducibility: For clarity and quality see the previous section.
In terms of novelty, I think though authors self consistency method is indeed novel when applied on chain of thought prompting, the idea itself is somewhat an obvious next step and something that should work well. So even though the empirical results are extensive and solid, they are not as surprising. So I think the novelty of this paper is only marginal. However the empirical results are nevertheless significant. 

summary_of_the_review: I’d recommend marginal acceptance, as I think the self consistency method is very principled and well motivated and empirical evidence is extensive and solid. However the notation and the presentation could be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a decoding strategy to improve chain-of-thought (CoT) prompting in pre-trained language models. The proposed strategy uses nucleus sampling to sample diverse outputs as opposed to a single one in CoT (based on greedy decoding) and selects the answer that is most consistent through marginalization. Evaluation performed on arithmetic and commonsense reasoning shows large improvements compared to CoT. Results are consistent across different language models and sizes.
strength_and_weaknesses: **Strengths**

Proposes a simple and effective decoding strategy to improve LMs with CoT prompting. 

Contains extensive experiments with different LM types and sizes that demonstrate large gains compared to vanilla CoT without requiring additional training. 

Self-consistency strategy surfaces diverse outputs that correspond to rationales in CoT-prompted LMs and improves robustness to imperfect inputs. Also, it can provide uncertainty estimates for calibration.

**Weaknesses** 
The decoding strategy itself has limited novelty since sampling strategies (including nucleus sampling) and ranking according to model scores have been previously used for pre-trained LMs albeit not with CoT prompting. 

Even though it generally leads to improvements, the proposed decoding strategy creates a significant overhead proportional to the number of outputs sampled e.g 5x latency for 5 samples assuming that samples have equal lengths.

The marginalization process of self-consistency assumes that answers are coming from a fixed answer set, which decreases the scope of applicability. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written overall and its motivation is quite clear i.e. improving recently proposed CoT. The claims are supported by extensive experiments with a variety of model configurations. I also do not have any major concerns about reproducibility since the authors provide sufficient descriptions and artifacts (code, prompts, model outputs, etc). 

In terms of originality, sampling different model outputs and ranking them according to a metric has been largely explored in the past (e.g. MT, text generation). In the context of pre-trained LMs, the GPT3 paper used beam search results for free-form completion, Adiwardana et al., (2020) used a very similar sample-and-rank method based on regular sampling, and more recently Jung et al. (2022) used nucleus sampling to sample diverse generated outputs to improve reasoning [1]. The last one is arguably parallel work but I am surprised it was not cited.

Having said that, the approach itself is not that novel but it is a straightforward direction to explore for CoT LMs and the community can benefit from such a simple method and large-scale empirical findings. 

Questions:
- It looks like that self-consistency is an instance of the sample-and-rank method that uses nucleus sampling instead of temperature sampling and majority vote (simple sum) instead of log probability. Do the authors share this view? I'd suggest making the connection clear to avoid misinterpretations regarding novelty.
- In table 1, could the authors specify on which set the accuracy scores are computed? It's likely not very crucial since the majority vote was eventually selected, but it would be useful to know. 

[1] https://arxiv.org/pdf/2205.11822.pdf
summary_of_the_review: Overall, this paper conducts a large empirical study of a simple decoding strategy for pre-trained LMs that use CoT prompting. The decoding strategy itself has limited novelty but the empirical findings are quite interesting/useful. The experiments show strong improvements over vanilla CoT prompting, which should be of interest to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new decoding and answer generation method for LMs to achieve multi-step math problems in a “step-by-step” way. Different from the standard “chain-of-thought” prompting which uses greedy decoding, this method samples several reasoning paths generated by LM, and then generates the final answer by finding the most frequent answer. Author conducted a series of experiments varying in LM type and LM size to assess the self-consistency method’s performance.
strength_and_weaknesses: Strength:
1 Motivation is straightforward.
2 The method is simple and effective.
3 The author conducted a variety of experiments to assess the method’s performance.

Weak:
Self-consistency is an interesting point. However, my main consideration is that this paper does not go deep enough into the study of self-consistency. This lack of depth is mainly reflected in two perspectives (1) only the majority vote of multiple randomly generated answers is considered. There is a lack of in-depth observation on how to combine different answers; (2) different answers are obtained by controlling the temperature and k. For text generation, the effect of these two hyperparameters is unclear. In particular, how these two hyperparameters affect answer consistency is unclear.

Other weak points:
1 Experiment results suggest that LM cannot distinguish well between correct and incorrect answers (at least from the aspect of token probability), but why taking majority vote over answers generated from different reasoning paths works? More in-depth studies should be done.
2 Since “diversity of reasoning paths is the key to better performance”, why other ensemble-based approaches (e.g. the multiple prompts setting) didn’t get larger gain for their diversity?
3 Figure 2 is confusing: why greedy decode has different number of sampled paths?
4 It would be better to show full sampled paths in table 4 and other examples if the purpose is to claim self-consistency can repair errored reasoning process.


clarity,_quality,_novelty_and_reproducibility: Please refer to Strength And Weaknesses.
summary_of_the_review: Overall, I suggest that the authors do a deeper study of the problem, e.g., for LMs, how does the problem of consistency arise and what is its fundamental nature? What exactly does improving self-consistency mean? Why does majority vote improve the model effect?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors introduce an expansion of the chain of thought prompting framework. In this change, they generate multiple reasoning chains for each input example (using sampling methods) and explore different methods for aggregating the confidence in these reasoning chains to make a final prediction. In the end, using a majority vote among the predicted final answer options of the reasoning chains seems to work best. 

Strengths:

The authors all highlight the rigour of the evaluations

Weaknesses:

Most reviewers complained about the novelty. While I typically avoid critiquing novelty, this case is challenging. The paper tests a rigorous set of hypotheses and achieves a strong empirical improvement with the final approach, but the methodological question seems to simplify down to greedy decoding vs. sample + aggregate. The novelty of sample + aggregate is specific to CoT because there happens to be a correct / incorrect answer at the end of the reasoning chain. In a typical text generation paper, though, (and CoT is a form of text generation), this could potentially be explored as a hyperparameter in the initial task paper. As a result, the contribution seems a bit incremental. That being said, "conceptual novelty vs. impact of results" is a question of opinion, and the empirical improvements are indeed substantial, so I'll vote Accept.
justification_for_why_not_higher_score: See meta-review
justification_for_why_not_lower_score: If the outcome of a rejection is just that the paper is re-submitted, and four more reviewers + 1 AC weigh in (all of whom sit somewhere else on the "novelty vs. results" importance spectrum), I'm not sure it's productive to reject this paper.
Decision: Accept: poster