Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper uses the information generated by the symbol system to help the neural system model to better reason and test CoT reasoning ability, which creates a Dataset PRONTOQA which aim at testing the reasoning ability of big model, and proposes a new method to measure the correctness of reasoning steps rather than the accuracy of reasoning answers. The experiment reveal some interesting findings of CoT. 

strength_and_weaknesses: Strengths: 
1) The generative process of example in PROTOQA is logical, systematic, interpretable and traceable. The process of logical reasoning is very clear.
2) The interesting experiment is sufficient enough and shows us the reasoning condition of the big model.
3) The concepts in the paper are explained clearly, and examples make it easier for readers to understand
4) This paper allows the neural system and the symbolic system to cooperate in a better way

Weaknesses: 
The context in the dataset may be monotonous to some extent. Will the future work experiment on more diversified natural languages?

clarity,_quality,_novelty_and_reproducibility: Clarity: Excellent.  Most of the definitions in the paper are given with lots of relevant examples, which are very helpful for understanding. The graphs and explanations in the method description and experimental performance are also detailed. So the clarity is excellent.

Quality: Good.  The purpose of the research in this paper is clear (i.e., to propose a form of dataset for evaluating LLM). The proposed approach regarding the generation of thought chains is very interesting and the final experiment is well done. So the quality is good.

Novelty: Excellent.  There are almost no valid assessment datasets to evaluate LLM reasoning ability, and the proposal of this paper has a positive effect on the refinement of LLM reasoning ability later. So the proposal of this paper is novelty.

Reproducibility: Avarage.  The proposed dataset generation logic in this paper is clear and step-by-step. From ontology tree to generating proof and context, and finally generating CoT, quary and label from the proof. where the example generation syntax of PRONTOQA is also relatively simple. However, the large model GPT tested in the experiment is not available to most people. So the reproducibility is average.

summary_of_the_review: The process of example generation in PROTOQA is logical, systematic, interpretable and traceable. The process of logical reasoning is also very clear. This paper uses the information generated by this symbolic system to help neural system models reason better and test CoT reasoning ability. It is very innovative. Very much in line with the ICRL conference requirements. Acceptance of this paper is recommended.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a PrOntoQA dataset and uses it to study the performance of chain-of-thought prompting of large language models. PrOntoQA is a synthetic reasoning dataset where each example is generated from an ontology along with the proof steps. The questions are answerable from repeated applications of the modus ponens deduction rule, which makes the reasoning process fairly simple, with the main driver of complexity coming from the number of deduction steps needed. The dataset is motivated by the desire to easily evaluate the quality of the intermediate chain-of-thought outputs, and to disentangle the reasoning performance that comes from pretraining world knowledge from the pure logical reasoning ability. The authors find that GPT-3 performs very well on individual deduction steps but can fail to output the correct chain of steps needed to solve the problem when there are multiple deduction paths possible. The authors further find that using fictional or false words instead of the true entities significantly hinders performance, thus illustrating the model's reliance on its pretraining knowledge and understanding of the entities to correctly perform the reasoning.
strength_and_weaknesses: ### Strength:

- The paper introduces a dataset for synthetic reasoning that can be useful for probing the reasoning behavior of LLMs. The authors take care to remove spurious heuristic solutions from the task setup, and restrict the complexity of the dataset while still maintaining a few knobs of question difficulty that one can use. 
- The paper explores several interesting notions of validity, atomicity, and misleading steps, which seems to be a fairly comprehensive perspective for studying reasoning behavior in models.
- The paper makes interesting observations regarding the tendency of the model to skip steps and output misleading steps which are irrelevant to the final answer, and its ability to perform basic deductions well.

### Weakness:

- The paper makes a number of observations that have been similarly made in other papers, such as 1) the model relies on pretrained knowledge to perform reasoning (also discussed in [1] and [2]), 2) longer proofs are more challenging and correct answers can sometimes contain misleading steps (which are well known and have been noted in many papers evaluating LLMs and chain-of-thought on reasoning tasks), and 3) sentence ordering affects performance (also discussed in [3]). This limits the contributions of this work.

- Several of the directions mentioned in future work seems well within the scope of the current paper and should probably be included to strengthen the current paper. For example, seeing if including prompt examples containing misleading steps would improve performance is an immediate follow-up to the observation that most errors come from misleading steps. The beam search of deduction steps is also a reasonable analysis to include in the current work in my opinion.

>Our work shows that relying on CoT prompting is not sufficient for more complex reasoning, such as in mathematical domains.

I don't see how this work shows the CoT prompting is insufficient for reasoning in mathematical domains.

[1] https://arxiv.org/abs/2202.12837

[2] https://arxiv.org/abs/2202.07206

[3] https://arxiv.org/abs/2104.08786
clarity,_quality,_novelty_and_reproducibility: The paper is generally easy to follow, though there are a few issues with clarity:

- It's not super clear what the difference between strictly-valid non-atomic and broadly-valid is, a more detailed explanation may be helpful.

- In Figure 3, how is the accuracy on the y-axis measured? Is it the percentage of deduction steps that are correct, or the percentage of questions where all deduction steps are correct?

- The bars in Figure 4 are quite confusing and it's unclear how to interpret the plot.

The work is low in novelty, but it illuminates some interesting observations about the reasoning abilities of LLMs and introduces a dataset that I think will be useful to the community.
summary_of_the_review: This paper introduces a dataset that I think will be useful to the community, and the analysis illuminates some interesting observations about the reasoning abilities of LLMs. However the observations are not very novel or surprising, and the manuscript could improve in its clarity in certain parts.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper releases a first order logic dataset, PRONTOQA, to evaluate the reasoning capabilities of LLMs using chain-of-thought prompting. The intermediate steps of reasoning, which are called chain-of-thought, can help in formal analysis of proofs. They also analyze results of GPT3 and  INSTRUCTGPT on this dataset.
strength_and_weaknesses: Strengths
● The paper is well structured and easy to read.
● Releasing a first order logic dataset for the community to test LLMs reasoning abilities.

Weakness
● Although GPT-3 is well known in the community, InstructGPT is something not popularly known and it would have helped to have a brief description as to what tasks InstructGPT can perform, along with its pre-training data.
● No mention of the number of entities modeled in the ontology.
● In the results, it would have been nice to know how many samples in the test-bed are used, and how many are valid/atomic/misleading.
● How is this work different from [1] 
● A similar work to test reasoning of LLMs is done in [2], where it was established that LLMs cannot reason, which is in contradiction with the proposal in this paper.

[1]Han, Simeng, et al. "Folio: Natural language reasoning with first-order logic."
arXiv preprint arXiv:2209.00840 (2022).

[2]Valmeekam, Karthik, et al. "Large Language Models Still Can't Plan (A Benchmark for
LLMs on Planning and Reasoning about Change)." arXiv preprint arXiv:2206.10498 (2022).
clarity,_quality,_novelty_and_reproducibility: The work is well written and clear, however, there exists another line of work which released a first order logic dataset [1]. So, it is unclear as to the originality of this paper. A comparison with new literature is needed. 

Analysis of proofs is a novel aspect of this work. However, it is not clear what extent of intermediate steps is needed to generate workable proofs. Fig 2 says that there is a one-to-one correspondence between the conclusion of each proof step and the sentences in the chain-of-thought. This implies that the user is giving proofs or significant hints to generate the proof. 



summary_of_the_review: This is an interesting line of work on improving sequential reasoning. It creates a new dataset to facilitate reasoning and analysis of proofs. Although promising, it is not clear if the burden put on the user for generating prompts at every reasoning step is practical. Further, the generated proof is a validation of the system or the user input. 

The discussion of the reasoning quality with respect to ontology quality is interesting. However, if ontology is available, why not use it directly as a combination of neuro-symbolic reasoning? This aspect is not explored in depth.

The evaluations are well described.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new reasoning dataset for LLMs, and uses it to assess the logical reasoning capabilities of a number of GPT variants, using chain-of-thought prompting, while varying a number of parameters. Using this dataset, the authors show expose a number of these models’ limitations, including their poor performance on reasoning tasks requiring multiple steps.
strength_and_weaknesses: The authors’ dataset is quite similar to the ProofWriter dataset released by Allen AI. But the breakdown into fictional, false and true ontologies is novel and useful, and provides insight into the models’ capabilities.

I have some questions and thoughts.

The authors write: “the most permissive accuracy metric has the highest correlation with label accuracy, suggesting that GPT-3 is indeed performing the reasoning to answer the question, rather than relying on heuristics” (p.6). I don’t see how this conclusion follows. Isn’t this assuming causation from correlation? Also, since “valid proof accuracy” is strictly weaker than “strict proof accuracy”, isn’t it inevitable that the points in the scatter plot below the perfect line will be higher in the former compared to the latter? Wouldn’t this be the case however the model arrived at the answer?

The authors write: “Once InstructGPT encounters a branch where one path at the fork follows the correct proof and the other paths do not, InstructGPT will select the incorrect direction with some frequency and is then not able to return to the correct path. Therefore, it seems that while LLMs are able to produce valid proof steps with high probability, they have difficulty with proof planning/strategizing.” (pp.8-9). This account feels highly speculative. As the authors obviously know, LLMs like GPT don’t work like this at a fundamental level. They are just trying, autoregressively, to predict the next token in the given context. There is no explicit mechanism for “selecting” a branch in a proof tree, let alone for planning or strategizing, so it’s hardly surprising that they struggle with tasks where these things are required (because they are out of distribution). Maybe some cautionary remarks would be useful here.

The authors write: “we can extend the CoT paradigm where after every predicted step in the CoT, we perform beam search decoding to find the top-k most likely values for the next step. We can then perform beam search over the proof steps (i.e., sentences) rather than over the tokens.” (p.9). This has recently been done. See: A.Creswell & M.Shanahan, Faithful Reasoning Using Large Language Models, https://arxiv.org/abs/2208.14271.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. There is some novelty in the new dataset, and in the results presented.
summary_of_the_review: The new reasoning dataset offers some new features compared to similar existing datasets such as ProofWriter, and the paper reports a worthwhile investigation into LLM capabilities. However, the dataset is only a marginal improvement on ProofWriter IMO, and the results are unsurprising. So I don’t feel the paper offers a substantial enough contribution to merit acceptance at ICRL, given the low acceptance rate of the conference. It’s good work though, and could be a cornerstone of a larger research effort (e.g. improving the reasoning capabilities of LLMs), so the authors should be encouraged.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose the dataset, PRONTOQA, a reasoning dataset to evaluation Language model's reasoning capabilities. The authors provide an in-depth analysis of reasoning traces.
strength_and_weaknesses: Strengths:
1. It’s great that the authors evaluate both validity, atomicity and wether a step is misleading. These are very important metrics.
2. Results in Figure 3 are very powerful! 
3. The three observations on page 8 are very informative.
4. If the analysis code is made available, I can imagine this being a very useful tool for people to measure reasoning progress. However, this would be better if it had leveraged existing datasets e.g. ProofWriter.

Weaknesses:
1. The authors ignore a significant chunk of prior work. 
2. They propose a novel dataset, while there already exists a dataset, ProofWriter, that would have been suitable for these experiments and work by Betz et al. 2020 is also relevant (see detailed comments below).
3. The title is somewhat miss-leading. I would suggest only using the second half of the title.
4. Some of the claims are not supported, "Reasoning is an emergent ability".
5. Authors only show results for a single model.

clarity,_quality,_novelty_and_reproducibility: Novelty:

1. The proposed dataset is exactly the same as that presented in ProofWriter simply with differently named predicated. There is no direct comparison to ProofWriter and ProofWriter could have been used to perform these experiments. Proof accuracy was reported in ProofWriter and also [2].

2.  “We use a simple grammar to convert the formal statements of the ontology into the natural language utterances that make up the context.”  The authors should reference [1] since they use similar templates. I would also argue that this is still a synthetic language, not a natural language.

Quality, Clarity and Reproducibility:

3. It is great that the authors purposefully force the model to avoid shortcuts. However, if you “add a distractor sentence by generating a novel concept that is disconnected from the ontology tree” the model could just learn to ignore the rules whose concepts do not come from the ontology.

4. “To achieve this, we parse each sentence of the predicted CoT into logical form” - How is this done? What if the samples are not in the correct format?


5. Why do the authors claim “it is not obvious that this metric, which we call strict proof accuracy, would accurately measure the reasoning ability of the model.”? For reasoning, strict proof accuracy may be very important.

6. While, results in Figure 3 are very powerful! However, it would also be great to have some indication of depth of the reasoning problems. For example, I would imagine for depth-1 reasoning problems the correlation would be very strong and possibly less so for problems of depth-5.

7. It is interesting (if not unsurprising) in Figure 4 that the model appears to be dependant on its priors (i.e. best results with the True Ontology).

8. To make Figure 5 more easy to parse it may be a good idea to plot these as a pie char showing that they are making up some proportion of the error and include in each caption the original proportion of incorrect predictions. 

9. The sub-title “Reasoning is an emergent ability.” Is very misleading. Please rephrase this.

10. For completeness please provide the prompt used for CoT and also include some example samples in the appendix.

11. The authors only show results for a single model. It would be best to evaluate other models too.

Refs:
[1] Critical Thinking Thinking for Language Models, Betz et al. 2020 \
[2] Faithful Reasoning Using Large Language Models, Creswell et al, 2022
summary_of_the_review: Recently there has been a lot of interested in language models (LMs) and their capabilities. Reasoning has emerged as an area of increasing interest and it's important that any capabilities are not over- (or under-) estimated. This paper offers many insights into the failings of Chain-Of-Thought prompting for LMs which I consider to be very valuable to the field. However, the (first half of the) title of the paper is misleading and the authors do not use a benchmark dataset. There are also a few points that need clarifying. 

I am willing to recommend acceptance of the paper, conditional on the points above being satisfactorily addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to investigate Chain of thought reasoning by proposing a new synthetic dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. and using that parse the generated chain-of-thought into symbolic proofs for formal analysis. 

Strength:
- The generative process of example in PROTOQA is logical, systematic, interpretable and traceable. The process of logical reasoning is very clear.
- The experiments are extensive on the reasoning condition of the big models.
- The concepts in the paper are explained clearly, and examples make it easier for readers to understand
- This paper allows the neural system and the symbolic system to cooperate in a better way

Weakness:
- The context in the dataset may be monotonous to some extent.
- The paper makes a number of observations that have been similarly made in other papers


justification_for_why_not_higher_score: The insights are not new or surprising.
justification_for_why_not_lower_score: the systematic study is valuable and well-done.
Decision: Accept: poster