Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel prompting algorithm based on a decomposition or divide-and-conquer idea. The idea is instead of feeding long CoT into the LLM, it decomposes the difficult task into easier subtask. The advantage is the smaller task can either be solved better with LLM or can be handled by external knowledges. Several experiments have been conducted to prove the effectiveness of the proposed method.
strength_and_weaknesses: - Following some predecessors, solving a difficult task by devide-and-conquer, decomposing them into simpler one, is quite novel to me. 

- The potential of the proposed is on solving challenging tasks, such as arithmetics.  I would encourage the authors try on challenging GSM8K dataset. It's not only because it's a dataset that CoT haven't solved it well, but I believe it could be a great fit to demonstrate the strength of the proposed algorithm. Many complex  arithmetics problems can be decomposed into simple sub-problems. I would expect each subproblem can be solved well by LLM, so as the whole problem under the decomposition scheme. 

- I have a few questions.  In practice, how do we decide which decomposition scheme to use? also, how do  we decide the question of each subproblem? is it required human design to tailor for each application like CoT? if so, how sensitive the results is to the design of the questions and decomposition? 

- How does this model work under a smaller LLM? In CoT, an interesting finding is it works better with larger LLM. How about the proposed one?
clarity,_quality,_novelty_and_reproducibility: The idea of decomposition is novel.  Reproducing the results should be easy based on the provided prompts.
summary_of_the_review: The paper provides a novel idea, and several experiments to support its claim. It would be even better to experiment on challenging task, such as GSM8K as suggested.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use decomposed prompting to solve complex NLP tasks. The idea behind decomposed prompting is very simple: divided and conquer. For a given complex task, they first decompose it into simpler sub-tasks via task prompting and then solve each sub-task by prompting-based LLMs dedicated to the sub-task. They conduct experiments on four different tasks including $k^{th}$ LETTER CONCATENATION, LIST REVERSAL, LONG-CONTEXT QUESTION ANSWERING, and OPEN-DOMAIN QUESTION ANSWERING. The empirical results not only show that the proposed framework improves the few-shot performance over CoT prompting but also demonstrate flexibility: it is easy to incorporate symbolic operations and information retrieval systems into the framework.

strength_and_weaknesses: Strengths:
- The proposed approach is well-motivated.
- Strong empirical results across different datasets.
- Comprehensive experiments to show the flexibility of the proposed framework.

Weaknesses:
- To me, the main weakness of this paper is the lack of novelty. It is more of an extension of the least-to-most prompting (Zhou et al., 2022). The least-to-most prompting can also generate the decompositions iteratively based on the outputs from previous steps. What's really new in this paper is that instead of solving the decompositions all at once by a pretrained language model, the authors show that solving them by dedicated LLMs or other systems (e.g., external API calls) is even better. 
- The scope of the experiments is not broad enough given the flexibility of the proposed framework. It would be better if the authors could test it in more challenging settings. There are two possible settings I think might be interesting: 
    - Answering the "why" questions in long-form questions answering (Fan et al., 2019). For example, to answer "why were the main predators in New Zealand large birds, compared to other places in the world?", we could possibly decompose it into several sub-questions to gather relevant information and synthesize them:
        1.  What are some of the main predators in the world?
        2.  Why New Zealand doesn't have *results from #1* as the main predators?
    - Fact-checking real-world complex claims through decomposition (Chen et al., 2022). For example, to check a claim like "Joe Biden stated on August 31, 2020, in a speech: "When I was vice president, violent crime fell 15% in this country. ... The murder rate now is up 26% across the nation this year under Donald Trump." We may need to know the answers to the following sub-questions and synthesize them to produce final veracity for the claim:
        1. Did the crime rate fall by 15% during Joe Biden's presidency?
        2. Did the murder rate in 2020 increase by 26% from 2019?
        3. Is Biden comparing crime rates from the same time interval in his statement?
        4. Is the violent crime rate and murder rate directly comparable?

    It would be better to show to what extent the decomposed prompts work in such real-world scenarios. Can you generate the decompositions iteratively? It is possible to synthesize the final answer by the decompositions?

References:
 - Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
 - Chen, Jifan, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. "Generating Literal and Implied Subquestions to Fact-check Complex Claims." arXiv preprint arXiv:2205.06938 (2022).

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and organized.

Quality: The experiments are solid but the scope is limited. See weaknesses for details.  

Novelty: To me, the novelty of this paper is limited. It is a solid extension of the Least-to-most prompting (Zhou et al., 2022).

Reproducibility: With all the details provided, it won't be too hard to reproduce the results.

summary_of_the_review: Although the idea is not brand new, the experiments are well-designed and demonstrate how the proposed approach could be used in different scenarios. I think the general community would find this useful and many researchers would benefit from the design choices for different tasks. Personally, I don't feel quite excited after reading this paper. It is a solid extension of the previous literature, but I expect more from an ICLR publication. Therefore, I lean toward a weak rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A framework called DecomP for solving multi-step reasoning problems with large language models is proposed. A task is "decomposed" into a sequence of smaller subtasks, where each subtask calls a reusable submodule (itself a LLM call or an external knowledge retrieval); a simple syntax for control flow allowing substitution of past subtask results and iteration over lists is introduced. Experiments on two string and list operation tasks and two question-answering tasks show that DecomP compares favourably with chain-of-thought approaches.
strength_and_weaknesses: Strengths:
- (++) I like the approach of separating controller and subtask modules. This work contributes to an increasing realization that linear attention -- where the entire reasoning sequence is a single chain of tokens -- could be insufficient for hard reasoning problems; the chosen comparisons with CoT illustrate this well. The ability, even if only hypothetical, to teach a controller to query external structured retrieval systems, execute programs, etc., is an exciting prospect.
- (+) Interesting combination of few-shot prompting (to teach the decomposer/controller) with question decomposition along the lines of [Perez et al., 2020; Wolfson et al., 2020].

Weaknesses / questions:
- (--) The breadth of applications is a little unsatisfying given how general the proposed approach can be. The examples currently given are either ones that can be turned into exact programs (list and string operations) or ones that use a chain of questions for knowledge retrieval (QA). But where can decomposed prompting be applied beyond these cases? The most interesting tasks may be the ones that require a mix of formal steps  (that a simple program could do) / querying of symbolic systems and knowledge generation / retrieval.
- (-) The following would be important to illustrate:
  - How does direct prompting -- no CoT -- performs on these tasks?
  - How sensitive is the system to the model size and prompt format? Can GPT-2 XL or the smallest variant of GPT-3 be used for (a) the decomposer and (b) the submodules, and if so, which ones? (From my own experience playing with CoT-like approaches, I would expect large differences even among davinci, text-davinci-001, and text-davinci-002 variants of GPT-3 for some problems.)
- When the submodules are expected to produce structured output, such as a list of strings, how reliably do they generate text in the correct format? (What would happen if a submodule outputs something not of the form "[ item1, item2, ... ]", but the following call contains "foreach"?)
clarity,_quality,_novelty_and_reproducibility: Quality, clarity: Good; see above.

Originality: This is hard to assess given that there are so many preprints on similar topics appearing over the last several months (as the authors surely know, judging by the related work section!). However, I think this particular idea is new and placed well in the context of other works.

Reproducibility: Although code is not provided, I would likely have no trouble reproducing the results on these tasks and extending them to new ones.
summary_of_the_review: The paper is quite well written and adds some interesting ideas to our understanding of multi-step reasoning with LLMs. On the other hand, given all the recent findings that go in a similar direction, the strong results with this particular method did not surprise me very much. This isn't a reason not to publish the paper, of course, but I would hope to see something that would make it stand out more, such as results on a broader (more diverse) set of tasks that illustrate more variety of subtasks and interesting ways to combine them. ~Therefore, I am giving a cautious "marginally below", but am willing to be convinced to raise the score.~ See post-rebuttal comment below.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Decomposed Prompting, a new prompting method to decompose a task into subtasks and handle them by modular prompts. This allows hierarchical and recursive decomposition, as well as calling external APIs. Across symbolic manipulation (k-th letter concatenation, list reversal) and question answering (CommaQA-E, HotpotQA) tasks, Decomposed Prompting is shown to outperform Chain-of-thought prompting.
strength_and_weaknesses: - Strength: the method seems a natural step in the direction of prompting, and empirical results across different tasks are good. 

- Weakness: experiments seem to only focus on establishing better results than baselines (e.g. chain-of-thought), but I feel we need better analysis about the method itself. A key thing missing is failure modes and limitations -- I'd argue such insights are more valuable for practitioners interested in adopting Decomposed Prompting. 

Also, there is some recent and related work around modular use of LLMs, such as Socratic Models [1]. Although motivation and tasks are not the same, some discussion is needed.

[1] Zeng et al., Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.

clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is mostly clear and easy to follow. 

- Novelty: The idea seems very intuitive and a prompting version of previous ideas such as modular network, so I don't know how to comment on novelty.

- Reproducibility: appendix provides prompts used in experiments.
summary_of_the_review: I believe Decomposed Prompting is a natural next step for prompting, and results here look good. We probably need more analysis and insights so that people can design decomposed prompts for their tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This is an interesting paper with a new prompting strategy that decomposes a task into smaller tasks that can also be solved by prompting. It is closely related to but potentially more powerful than chain-of-thought prompting. Experimental results show that this approach improves performance across a variety of tasks.

While reviewers originally felt that the experiments were limited, some further experiments were added. Also, concerns about novelty were addressed in the rebuttal. As a result of the changes made to the paper, reviewers were satisfied with the resulting paper. Some reviewers felt that more analysis and insights would be helpful to make it easier for people to use this prompt decomposition approach.
justification_for_why_not_higher_score: There were largely mixed feelings about this paper, especially considering its similarity to chain-of-thought prompting and least-to-most prompting. 
justification_for_why_not_lower_score: Note: Reviewer WUGg wrote that they plan to change their score to a 6 but hasn't done it despite prodding. So the scores should be 6 6 6 8 with an average of 6.5
Decision: Accept: poster