Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes whether large language models can answer multi-hop, compositional questions via an automatically constructed dataset, Compositional Celebrities, composed of 2-hop questions. The authors find that although the model gets better with scale at answering compositional questions, this is due to the fact that the model acquires knowledge about their sub-questions, rather than an improved ability of composing knowledge from the sub-questions. To address this fact, the authors propose an elicitive prompting technique, which basically primes the model to decompose the question into the relevant sub-questions. The proposed elicitive prompt technique improves the performance w.r.t to direct prompting and also over chain-of-thought on a novel manually sourced dataset of ~100 questions, Bamboogle.
strength_and_weaknesses: Strengths:
- *Proposed approach is simple and effective*: I think the most interesting results are: 1) the integration with the search engine; and 2) the fact that self-ask can generalize to Bamboogle while being prompted to solve 2WikiMultiHopQA, and can do so better than Chain-of-Thoughts. 

- *Paper is well-written and clear*: I found the paper very clear and enjoyable to read.

Weaknesses:
- *Experimental setting seems simplistic*: One thing that bugs me a bit is that the experimental setting right now seems simplistic and I am left to wonder whether this method can actually be useful for some settings. I imagine this method being useful in web search, when users can ask complex questions that the search engine can't answer. The authors tried to mimic this setting by manually creating Bamboogle,  which is composed of questions that the search engine can't answer, and on which Self-Ask is shown succeed to some extent. In a more realistic setting, I imagine that the distribution of web queries / questions might be pretty "wild", much more diverse than the kind of questions present in Bamboogle. I appreciated the fact that the improvement on Bamboogle has been obtained by using prompts tuned on 2WikiMultiHopQA (thus putting the model in a situation of distribution shift), but I feel that there is a very small distribution shift between Bamboogle and 2WikiMultiHopQA, both being mined from Wikipedia and mostly composed of 2-hops reasoning. Therefore, first of all, I wonder if the authors could think about coming up with a more convincing setting to show the potential of the approach in a realistic scenario. Some questions that could shed light on the results could be:

1) What happens if the authors try their method for answering realistic web queries contained in Natural Questions ?
3) What is the performance gap when using prompts from a different category on Compositional Celebrities ? Right now the model uses a prompt tailored to each category.

- *Novelty with respect to previous works*: Decomposing a multi-hop question in a series of single-hop questions has received considerable amount of attention in the literature. The proposed method can be seen like an "in-context" formulation of the sub-question decomposition problem. Nevertheless, I think the authors do a good job in acknowledging related works. A reference the authors might also want to include is the following recent work: https://aclanthology.org/2022.coling-1.142.pdf. Due to considerable similarity with these approaches, the authors might consider being a bit more explicit in framing their method with respect to relevant previous work directly in the main text. Moreover, I wonder:

3) How does the method compare to *least-to-most* prompting? Why this baseline is not included in the paper?

- *Baselines for 2WikiMultiHopQA*: I am wondering if it is possible to have some supervised baselines for the datasets tested here, especially for 2WikiMultiHopQA. I think the reference I joined above contains some recent methods one could include for comparison. 

clarity,_quality,_novelty_and_reproducibility: Questions and Concerns:

- I am not sure to understand whether in the Self-Ask prompts there are examples where "Are follow-up questions required here:" is sometimes followed by "No"? What happens if you prompt the model with such examples? Can the model solve both compositional and single-hop questions?
- What happens if a similar baseline to least-to-most is used in CC, where the prompt is composed of both the sub-questions and the compositional question?

summary_of_the_review: This paper was a pleasure to read. It is clear and self-contained. The Compositional Celebrities dataset, provided it will be released, can benefit the community. Overall, the novelty of the approach consists in performing complex question decomposition with a LM using in-context learning. The integration ability with a search engine is interesting. However, the experimental setting seems a bit simplistic, and therefore I feel that more evidence is needed to understand whether the approach can perform well in a more realistic scenario. I hope the rebuttal phase can elucidate this and my other questions :-)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Big language models have shown impressive abilities to solve NLP problems. It is natural to assume they can solve single-step problems well and decompose multi-step problems into sub-problems. This paper studies the ability of language models to solve multi-step reasoning tasks from correctly composing the answers to sub-problems. It defines a metric called compositionality gap to measure how often models can correctly answer all sub-problems  but not generate the overall solution. It observes that in the GPT-3 family of models, as model size increases, the single-hop question answering performance improves faster than the multi-hop performance does. The compositionality gap does not decrease. The paper interprets the results to mean that big models show no corresponding improvement to perform compositional reasoning. The paper presents an elicitive prompting, e.g. chain of thought, called self-ask to narrow the compositionality gap. Self-ask explicitly asks itself (and then answers) follow-up questions before answering the initial question. The paper also shows that self-ask's structured prompting allows for easy plug in a search engine to answer the follow-up questions, which can further improve accuracy.
strength_and_weaknesses: Strength
1. The paper proposes a metric called compositionality gap to measure big language models ability to solve multi-step problems.

2. The paper presents self-ask, an elicitive prompting similar to chain of thought to improve big LMs ability to solve compositional reasoning tasks. 

3. The method is evaluated on four datasets, Compositional Celebrities (CC), Musique, 2WikiMultiHopQA, and a new dataset, Bamboogle (this new dataset is designed by manually creating compositional questions that are simple enough for a popular internet search engine to propose an answer to but hard enough so the answer it proposes is incorrect).

Weaknesses
1. Compared with Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (https://arxiv.org/abs/2205.10625), the paper does not seem to have much novelty. Least-to-most prompting essentially decomposes into sub-problems by querying LMs. 

2. Self-ask performs marginally better than chain-of-thought on 2WikiMultiHopQA and Musique. It only performs much better than chain-of-thought on Bamboogle. I wonder least-to-most prompting which improves on chain-of-thought would perform better than self-ask.

3. There are even more principled methods, 
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
https://arxiv.org/abs/2205.11822

4. The paper should evaluate on more reasoning datasets and tasks, e.g. StrategyQA, TimeDial, high school reading comprehension and those in Big-Bench (beyond the Imitation Game benchmark https://arxiv.org/pdf/2206.04615.pdf)
clarity,_quality,_novelty_and_reproducibility: Compositionality gap is a bit confusing. Compositional generalization is well defined for NLP problems. Instead of compositional reasoning, it is more clear to just use multi-step reasoning. Otherwise, the paper is mostly written clearly. 

However, I do not find much novelty when compared with least-to-most prompting and maieutic prompting. 
summary_of_the_review: The paper considers an important problem which is to improve multi-step reasoning with answering intermediate questions. However, the paper is not particularly novel given least-to-most prompting and maieutic prompting. Furthermore, it does not have deep insights on whether self-ask represents genuine improvements across a wide range of reasoning tasks. Evaluation on more datasets will help,

== post authors' response

The comparison results with Least-to-most (LtM) on Musique and 2Wiki are not conclusive. Like LtM, self-ask is very much a heuristic method. Therefore, extensive evaluation results on several datasets are required to have enough empirical evidence. I do not find adding a search engine is a contribution. It introduces more problems that the paper does not address, e.g. search engine often returns whole pages and you have to extract the relevant answers. As a result, I keep my existing rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies LLMs’ performance in answering a composition of two questions vs. answering each question separately and how this performance gap (compositionality gap) evolves with scale. They find this gap does not shrink using vanilla in-context learning with scale, but decreases using more sophisticated methods of prompting LLMs, such as chain of thought or their proposed method called self-ask (similar to CoT with more structure). They also observe that as models get more confident in their answers to each single question it is more probable for the LM to compose two facts and answer the compositional question. 

Key contributions are:
* Studying the compositional gap using 1-hop and 2-hop questions
* Two new dataset, Compositional Celebrities (CC) and Bamboogle (125 examples) with 2-hop questions
* Self-ask prompting

strength_and_weaknesses: ### Strength
The proposed method of using 2-hop questions as a way of evaluating the compositional abilities of large language models.

The automatically generated 2-hop QA Compositional Celebrities dataset with the constraint that the two facts/questions in each example, have been seen during pretraining separately with a high probability, but not together.

The proposed self-ask prompting method is more structured than Chain of Thought prompting which enables the authors to use it in combination with a search engine to boost the performance. 

### Weaknesses
The author's study of this compositional gap is limited to k-hop questions with k=2. Studying k>2 instances or other types of tasks (e.g. [Qui et al., 2022](https://arxiv.org/abs/2205.12253)) would shed more light on the compositional abilities of LLMs. 

The term “multi-hop” has been used a couple of times in the paper. It would be better to change them to 2-hop since that is the only setting experimented with.

Studying k>2 instances or other types of tasks could also be useful in evaluating the proposed prompting method (self-ask) with longer reasoning chains and decomposition of questions. I think a more thorough evaluation of this method is missing from the paper. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and mostly clear. Only a few comments:

* It would be nice to keep the colors the same in Figure 1 and 4.
* What is the reason for the self-ask method answering more compositional questions than separate 1-hop questions in Figure 4? It is mentioned that “This might be because elicitive prompts contain more information than direct ones”. This is a bit unclear. Could you elaborate?

summary_of_the_review: The paper studies an important topic of compositional abilities of large language models in a clever way using k-hop QA datasets where k is limited to 2. The authors propose a prompting method, called self-ask, similar to Chain of Thought with more structure and versatility to improve compositional abilities of large language models. However, a more comprehensive evaluation of this method would be appreciated.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the ability of language models to solve multi-step reasoning tasks from correctly composing the answers to sub-problems. It proposes a metric to evaluate the capability of language models and proposed a new prompt method to help large language models to solve compositional reasoning tasks. 

The reviewers share similar views on the paper – it is a paper with both strength and weakness. It tries to address an interesting problem, and proposed a reasonable evaluation measure plus a sound solution. On the other hand, its technical novelty is somehow limited, and the empirical performance of the proposed method is not very good either. The authors provided their responses, which have not changed the reviewers’ minds. As a result, I am afraid that we could not accept the paper in its current form.

justification_for_why_not_higher_score: The technical novelty of the paper is limited (the proposed prompt scheme is similar to prior works), and the empirical results are not very good (only slight increase over baselines). 
justification_for_why_not_lower_score:  It tries to address an interesting problem, and proposed a reasonable evaluation measure plus a sound solution.
Decision: Reject