Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a simple complexity-based prompting method for chain-of-thought prompting. The idea is to prompt with more complex questions: The authors use example questions with more reasoning steps (9 per question). When combining the predictions from multiple prompts, they choose the top K predictions with the most reasoning steps. Results show consistent improvement on 3 maths reasoning datasets, across various complexity of questions and formats of the prompts, and perform better than other prompt example selection methods.
strength_and_weaknesses: The proposed method is very simple and it shows consistent improvement over the previous methods. It substantially improves the GPT-3 performance on chain-of-thoughts reasoning. The ablation study shows the method is very robust across many different angles.

Weakness: The experiments are only carried out on GPT-3, and the performance is generally inferior to those of PaLM and Minerva. The authors probably cannot get the performance on that two models, it would be great if they can illustrate on other large models, like OPT and Codex.

clarity,_quality,_novelty_and_reproducibility: The paper is nicely written and easy to understand. Experiments are complete (although lacking a bit on different LM backbones). 

I think the method is novel for prompting, and the findings are interesting.

The code and data are not shared.

summary_of_the_review: Questions:
1. How does changing the complexity in input change the complexity in output? I did not find an answer in the paper. Does it often result in more reasoning steps in output?
2. For the "retrieval"-based method in Table 3, how did you get all the annotations for the whole dataset? The CoT annotation is probably different from the original rationales in the dataset.
Typo: Page 2 second paragraph: "n. umber".

Overall, I think the paper is nicely written and good for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the task of prompting large-scale language models to perform multi-step reasoning, and proposes a simple and effective example selection scheme for multi-step reasoning based on reasoning complexity. The proposed method is to select prompts with higher reasoning complexity, according to the author's definition of complexity. The proposed method is evaluated on math word reasoning tasks, and is shown to outperform baseline CoT construction methods. The method is also robust to format perturbation and distribution shift.

strength_and_weaknesses: Strength: 
- The proposed method is intuitive and easy to implement, as claimed by the authors. 
- The paper is well-written and easy to follow. The authors provide detailed experimental results and analysis.
- The proposed method is a simple and effective example selection scheme for multi-step reasoning, which is an improvement over existing methods that require hand-engineering.

Weaknesses: 
- The proposed method is only evaluated on GPT-3, and it would be interesting to see if the proposed method also works on smaller language models. 
- The proposed method is only evaluated on math word reasoning tasks. It would be interesting to see if the proposed method also works on other tasks, such as the Big-Bench tasks.
clarity,_quality,_novelty_and_reproducibility: 
Concerns on clarity: 

- The definition of complexity is not clear, and it is not detailed how the complexity is calculated. What is considered a “reasoning step”.
- Adding "Let's think step by step" is a confounding factor that may confound the main results. It would be better to report results without this factor. (table 2)

Concerns on novelty:

- The method attempts at finding a way to design a good chain-of-thought, which is a valid task.
- The proposed method is based on the intuition that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, would achieve better performance. This can be seen as a special case of heuristic-based methods. It is not clear what is fundamentally novel about this work
- It is not clear what would be the cause of this effect shown in the paper. Longer reasoning chains might be breaking down the problems into smaller bite-sized problems that LLMs can handle. It would be interesting to see more analysis on this. Perhaps the reasoning chain length is the wrong notion, but the “granularity” of the reasoning steps is the right metric.


Concerns on reproducibility: 

- The notion of complexity is not consistent across datasets and tasks, and needs hand annotation. This limits the applicability of the proposed method.
- Some steps makes this paper a bit difficult to reproduce, like how to annotate the reasoning chain for MultiArith. 
- It is not clear if this effect can be reproduced on other datasets.
summary_of_the_review: In this work, the authors study the task of prompting large-scale language models to perform multi-step reasoning, and propose a simple and effective example selection scheme for multi-step reasoning based on reasoning complexity. The proposed method is evaluated on math word reasoning tasks, and is shown to outperform baseline CoT construction methods.  There are some concerns about the novelty of this work. In addition, the proposed method is only evaluated on GPT-3 and a few reasoning tasks, and it would be interesting to see if the proposed method also works on smaller language models and more tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed complexity-based prompting for multi-step reasoning and it showed prompts with more reasoning steps, achieving substantially better performance on math word reasoning tasks over strong baselines. Their experimental results show that such sample selection and voting over top-k complex output samples improve the accuracy.
strength_and_weaknesses: Strength:
1- The paper considered example selection for a multi-step reasoning task and it picked examples with complex reasoning chains, i.e., chains with more reasoning steps, as the prompt.

2- In comparison to existing example selection schemes like manual tuning or retrieval-based selection, complexity-based prompting sounds intuitive, easy to implement.

3- The paper also proposed complexity-based consistency, where instead of taking a majority vote over all generated chains, we vote over the top K complex chains. Their experimental results also show that voting among more complex reasoning chains generalizes better than voting among all.


Weakness:
1- The paper mentioned that majority voting over top K complex generated samples.performs better than the top-k simple generated samples. However the results in Figure 5 for the MathQA are different. Is there any explanation for that?

2- The generalization results are not clear.  According to Figure 3 complex prompts improve the performance on simple cases, however the accuracy of the complex prompts is very low. I think some statistics on the dataset where how many examples are there in each bucket may make it more clear. For example in GSM8k, for how many samples the number of reasonings are 8, etc.

3- I understand the paper focused on generalizability based on the number of steps. However, I am wondering if the accuracy error is because of the ask complexity or the complexity of the numbers (length of the digits)

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. The proposed in the paper is simple but it seems it improved the existing results.

summary_of_the_review: The paper proposed a simple example selection that improves the existing results. The paper also mentioned that rather than majority voting over the output sample voting over top-k complex outputcsamples (those with more number of steps) contributes to the accuracy improvement. The paper still needs to clarify/justify the results in Figure 5 since it conflicts with these statements.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the example selection problem which is selecting reasoning examples that make the most effective prompts when prompting large language models with a chain of thoughts (CoT) prompt. The authors apply their approach at both prompting and decoding time. Evaluating on multi-step reasoning task, they observe improvements over the baselines.
strength_and_weaknesses: - The idea is nice, focusing on the instance selection problem by looking at the complexity is interesting. 

- The authors define a step as a line, separated by the linebreak “\n”. They do look at the length of the input at some point but I am wondering if there are some side effects from defining a step like this. Do the authors know whether processing steps together would provide additional context or is it too difficult to learn from? 

- From the results (e.g., Table 1) it looks like the number of parameters is an important factor in the overall performance. Have the authors looked into the effectiveness of their proposed approach with language models of different sizes?

- Typo: section 3.1: the n. umber of
clarity,_quality,_novelty_and_reproducibility: This work is clearly written and the message is evident to the reader without any struggle. The experiments are sound and the quality of the work is well done. 
They propose a new complexity-based instance selection scheme which is shown to be effective in multistep reasoning. This is a valid and useful idea and it contributes to the field of research. 
summary_of_the_review: This work is well-motivated, clearly explained, and sufficiently supported by analysis. There are some questions for the authors (see section Strength And Weaknesses), however overall it's a valuable contribution to the field.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new approach to prompting for language models, called complexity-based prompting, which achieves state-of-the-art results on several math reasoning tasks. The approach uses a measure of complexity to select the most complex chain of thoughts from a set of generated chains, which are then used to prompt the language model. The authors show that these prompts result in a significant increase in performance compared to standard CoT prompting. Although the reviewers have initially raised concerns about the novelty of the method, and the fact that it was evaluated on one language model and only mathematical datasets, the authors were able to address these concerns in the discussion period, where they provided new results on the Codex language model and on two hard subtasks from the BigBench dataset, showing that their approach outperforms existing methods. They have also addressed concerns about the reproducibility, applicability, and effectiveness of their approach. Overall, the paper shows that complexity-based prompting is a promising approach for improving the performance of language models on complex reasoning tasks.
justification_for_why_not_higher_score: While this paper shows good results with the proposed method, the method itself is quite straightforward and the paper is largely empirical.
justification_for_why_not_lower_score: The paper proposes a method that does work well and will be of interest to the increasingly growing community of researchers and practitioners working with large language models.
Decision: Accept: poster